org.apache.spark.sql.execution.streaming

ProgressReporter

trait ProgressReporter extends Logging

Responsible for continually reporting statistics about the amount of data processed as well as latency for a streaming query. This trait is designed to be mixed into the StreamExecution, who is responsible for calling startTrigger and finishTrigger at the appropriate times. Additionally, the status can updated with updateStatusMessage to allow reporting on the streams current state (i.e. "Fetching more data").

Linear Supertypes
Logging, AnyRef, Any
Known Subclasses
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. ProgressReporter
  2. Logging
  3. AnyRef
  4. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Type Members

  1. case class ExecutionStats(inputRows: Map[Source, Long], stateOperators: Seq[StateOperatorProgress], eventTimeStats: Map[String, String]) extends Product with Serializable

Abstract Value Members

  1. abstract def availableOffsets: StreamProgress

    Attributes
    protected
  2. abstract def committedOffsets: StreamProgress

    Attributes
    protected
  3. abstract def currentBatchId: Long

    Attributes
    protected
  4. abstract def id: UUID

    Attributes
    protected
  5. abstract def lastExecution: QueryExecution

    Attributes
    protected
  6. abstract def logicalPlan: LogicalPlan

    Attributes
    protected
  7. abstract def name: String

    Attributes
    protected
  8. abstract def newData: Map[Source, DataFrame]

    Attributes
    protected
  9. abstract def offsetSeqMetadata: OffsetSeqMetadata

    Attributes
    protected
  10. abstract def postEvent(event: Event): Unit

    Attributes
    protected
  11. abstract def runId: UUID

    Attributes
    protected
  12. abstract def sink: Sink

    Attributes
    protected
  13. abstract def sources: Seq[Source]

    Attributes
    protected
  14. abstract def sparkSession: SparkSession

    Attributes
    protected
  15. abstract def triggerClock: Clock

    Attributes
    protected

Concrete Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any
  3. final def ##(): Int

    Definition Classes
    AnyRef → Any
  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any
  6. final def asInstanceOf[T0]: T0

    Definition Classes
    Any
  7. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  8. var currentStatus: StreamingQueryStatus

    Attributes
    protected
  9. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  10. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  11. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  12. def finishTrigger(hasNewData: Boolean): Unit

    Finalizes the query progress and adds it to list of recent status updates.

    Finalizes the query progress and adds it to list of recent status updates.

    Attributes
    protected
  13. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any
  14. def hashCode(): Int

    Definition Classes
    AnyRef → Any
  15. def initializeLogIfNecessary(isInterpreter: Boolean): Unit

    Attributes
    protected
    Definition Classes
    Logging
  16. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any
  17. def isTraceEnabled(): Boolean

    Attributes
    protected
    Definition Classes
    Logging
  18. def lastProgress: StreamingQueryProgress

    Returns the most recent query progress update or null if there were no progress updates.

  19. def log: Logger

    Attributes
    protected
    Definition Classes
    Logging
  20. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging
  21. def logDebug(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging
  22. def logError(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging
  23. def logError(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging
  24. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging
  25. def logInfo(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging
  26. def logName: String

    Attributes
    protected
    Definition Classes
    Logging
  27. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging
  28. def logTrace(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging
  29. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging
  30. def logWarning(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging
  31. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  32. final def notify(): Unit

    Definition Classes
    AnyRef
  33. final def notifyAll(): Unit

    Definition Classes
    AnyRef
  34. def recentProgress: Array[StreamingQueryProgress]

    Returns an array containing the most recent query progress updates.

  35. def reportTimeTaken[T](triggerDetailKey: String)(body: ⇒ T): T

    Records the duration of running body for the next query progress update.

    Records the duration of running body for the next query progress update.

    Attributes
    protected
  36. def startTrigger(): Unit

    Begins recording statistics about query progress for a given trigger.

    Begins recording statistics about query progress for a given trigger.

    Attributes
    protected
  37. def status: StreamingQueryStatus

    Returns the current status of the query.

  38. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef
  39. def toString(): String

    Definition Classes
    AnyRef → Any
  40. def updateStatusMessage(message: String): Unit

    Updates the message returned in status.

    Updates the message returned in status.

    Attributes
    protected
  41. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  42. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  43. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from Logging

Inherited from AnyRef

Inherited from Any

Ungrouped