Packages

abstract class StreamExecution extends StreamingQuery with ProgressReporter with Logging

Manages the execution of a streaming Spark SQL query that is occurring in a separate thread. Unlike a standard query, a streaming query executes repeatedly each time new data arrives at any Source present in the query plan. Whenever new data arrives, a QueryExecution is created and the results are committed transactionally to the given Sink.

Linear Supertypes
ProgressReporter, Logging, StreamingQuery, AnyRef, Any
Known Subclasses
AsyncProgressTrackingMicroBatchExecution, MicroBatchExecution, ContinuousExecution
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. StreamExecution
  2. ProgressReporter
  3. Logging
  4. StreamingQuery
  5. AnyRef
  6. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new StreamExecution(sparkSession: SparkSession, name: String, resolvedCheckpointRoot: String, analyzedPlan: LogicalPlan, sink: Table, trigger: Trigger, triggerClock: Clock, outputMode: OutputMode, deleteCheckpointOnStop: Boolean)

    deleteCheckpointOnStop

    whether to delete the checkpoint if the query is stopped without errors. Checkpoint deletion can be forced with the appropriate Spark configuration.

Type Members

  1. case class ExecutionStats(inputRows: Map[SparkDataStream, Long], stateOperators: Seq[StateOperatorProgress], eventTimeStats: Map[String, String]) extends Product with Serializable
    Definition Classes
    ProgressReporter

Abstract Value Members

  1. abstract def logicalPlan: LogicalPlan
    Definition Classes
    StreamExecution → ProgressReporter
  2. abstract def runActivatedStream(sparkSessionForStream: SparkSession): Unit

    Run the activated stream until stopped.

    Run the activated stream until stopped.

    Attributes
    protected
  3. abstract def sources: Seq[SparkDataStream]
    Attributes
    protected
    Definition Classes
    ProgressReporter
  4. abstract def stop(): Unit

    Stops the execution of this query if it is running.

    Stops the execution of this query if it is running. This waits until the termination of the query execution threads or until a timeout is hit.

    By default stop will block indefinitely. You can configure a timeout by the configuration spark.sql.streaming.stopTimeout. A timeout of 0 (or negative) milliseconds will block indefinitely. If a TimeoutException is thrown, users can retry stopping the stream. If the issue persists, it is advisable to kill the Spark application.

    Definition Classes
    StreamingQuery
    Annotations
    @throws( ... )
    Since

    2.0.0

Concrete Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. val analyzedPlan: LogicalPlan
  5. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  6. var availableOffsets: StreamProgress

    Tracks the offsets that are available to be processed, but have not yet be committed to the sink.

    Tracks the offsets that are available to be processed, but have not yet be committed to the sink. Only the scheduler thread should modify this field, and only in atomic steps. Other threads should make a shallow copy if they are going to access this field more than once, since the field's value may change at any time.

  7. def awaitInitialization(timeoutMs: Long): Unit

    Await until all fields of the query have been initialized.

  8. val awaitProgressLock: ReentrantLock

    A lock used to wait/notify when batches complete.

    A lock used to wait/notify when batches complete. Use a fair lock to avoid thread starvation.

    Attributes
    protected
  9. val awaitProgressLockCondition: Condition
    Attributes
    protected
  10. def awaitTermination(timeoutMs: Long): Boolean

    Waits for the termination of this query, either by query.stop() or by an exception.

    Waits for the termination of this query, either by query.stop() or by an exception. If the query has terminated with an exception, then the exception will be thrown. Otherwise, it returns whether the query has terminated or not within the timeoutMs milliseconds.

    If the query has terminated, then all subsequent calls to this method will either return true immediately (if the query was terminated by stop()), or throw the exception immediately (if the query has terminated with exception).

    Definition Classes
    StreamExecution → StreamingQuery
    Since

    2.0.0

    Exceptions thrown

    StreamingQueryException if the query has terminated with an exception

  11. def awaitTermination(): Unit

    Waits for the termination of this query, either by query.stop() or by an exception.

    Waits for the termination of this query, either by query.stop() or by an exception. If the query has terminated with an exception, then the exception will be thrown.

    If the query has terminated, then all subsequent calls to this method will either return immediately (if the query was terminated by stop()), or throw the exception immediately (if the query has terminated with exception).

    Definition Classes
    StreamExecution → StreamingQuery
    Since

    2.0.0

    Exceptions thrown

    StreamingQueryException if the query has terminated with an exception.

  12. def checkpointFile(name: String): String

    Returns the path of a file with name in the checkpoint directory.

    Returns the path of a file with name in the checkpoint directory.

    Attributes
    protected
  13. def cleanup(): Unit

    Any clean up that needs to happen when the query is stopped or exits

    Any clean up that needs to happen when the query is stopped or exits

    Attributes
    protected
  14. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  15. val commitLog: CommitLog

    A log that records the batch ids that have completed.

    A log that records the batch ids that have completed. This is used to check if a batch was fully processed, and its output was committed to the sink, hence no need to process it again. This is used (for instance) during restart, to help identify which batch to run next.

  16. var committedOffsets: StreamProgress

    Tracks how much data we have processed and committed to the sink or state store from each input source.

    Tracks how much data we have processed and committed to the sink or state store from each input source. Only the scheduler thread should modify this field, and only in atomic steps. Other threads should make a shallow copy if they are going to access this field more than once, since the field's value may change at any time.

  17. def createWrite(table: SupportsWrite, options: Map[String, String], inputPlan: LogicalPlan): Write
    Attributes
    protected
  18. var currentBatchId: Long

    The current batchId or -1 if execution has not yet been initialized.

    The current batchId or -1 if execution has not yet been initialized.

    Attributes
    protected
    Definition Classes
    StreamExecution → ProgressReporter
  19. val currentStatus: StreamingQueryStatus
    Attributes
    protected
    Definition Classes
    ProgressReporter
    Annotations
    @volatile()
  20. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  21. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  22. def exception: Option[StreamingQueryException]

    Returns the StreamingQueryException if the query was terminated by an exception.

    Returns the StreamingQueryException if the query was terminated by an exception.

    Definition Classes
    StreamExecution → StreamingQuery
  23. def explain(): Unit

    Prints the physical plan to the console for debugging purposes.

    Prints the physical plan to the console for debugging purposes.

    Definition Classes
    StreamExecution → StreamingQuery
    Since

    2.0.0

  24. def explain(extended: Boolean): Unit

    Prints the physical plan to the console for debugging purposes.

    Prints the physical plan to the console for debugging purposes.

    extended

    whether to do extended explain or not

    Definition Classes
    StreamExecution → StreamingQuery
    Since

    2.0.0

  25. def explainInternal(extended: Boolean): String

    Expose for tests

  26. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  27. def finishTrigger(hasNewData: Boolean, hasExecuted: Boolean): Unit

    Finalizes the query progress and adds it to list of recent status updates.

    Finalizes the query progress and adds it to list of recent status updates.

    hasNewData

    Whether the sources of this stream had new data for this trigger.

    hasExecuted

    Whether any batch was executed during this trigger. Streaming queries that perform stateful aggregations with timeouts can still run batches even though the sources don't have any new data.

    Attributes
    protected
    Definition Classes
    ProgressReporter
  28. def formatTimestamp(millis: Long): String
    Attributes
    protected
    Definition Classes
    ProgressReporter
  29. def getBatchDescriptionString: String
    Attributes
    protected
  30. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  31. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  32. val id: UUID

    Returns the unique id of this query that persists across restarts from checkpoint data.

    Returns the unique id of this query that persists across restarts from checkpoint data. That is, this id is generated when a query is started for the first time, and will be the same every time it is restarted from checkpoint data. Also see runId.

    Definition Classes
    StreamExecution → ProgressReporter → StreamingQuery
    Since

    2.1.0

  33. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  34. def initializeLogIfNecessary(isInterpreter: Boolean): Unit
    Attributes
    protected
    Definition Classes
    Logging
  35. def interruptAndAwaitExecutionThreadTermination(): Unit

    Interrupts the query execution thread and awaits its termination until until it exceeds the timeout.

    Interrupts the query execution thread and awaits its termination until until it exceeds the timeout. The timeout can be set on "spark.sql.streaming.stopTimeout".

    Attributes
    protected
    Annotations
    @throws( ... )
    Exceptions thrown

    TimeoutException If the thread cannot be stopped within the timeout

  36. def isActive: Boolean

    Whether the query is currently active or not

    Whether the query is currently active or not

    Definition Classes
    StreamExecution → StreamingQuery
  37. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  38. def isTraceEnabled(): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  39. var lastExecution: IncrementalExecution
    Definition Classes
    StreamExecution → ProgressReporter
  40. def lastProgress: StreamingQueryProgress

    Returns the most recent query progress update or null if there were no progress updates.

    Returns the most recent query progress update or null if there were no progress updates.

    Definition Classes
    ProgressReporter
  41. var latestOffsets: StreamProgress

    Tracks the latest offsets for each input source.

    Tracks the latest offsets for each input source. Only the scheduler thread should modify this field, and only in atomic steps. Other threads should make a shallow copy if they are going to access this field more than once, since the field's value may change at any time.

  42. def log: Logger
    Attributes
    protected
    Definition Classes
    Logging
  43. def logDebug(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  44. def logDebug(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  45. def logError(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  46. def logError(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  47. def logInfo(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  48. def logInfo(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  49. def logName: String
    Attributes
    protected
    Definition Classes
    Logging
  50. def logTrace(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  51. def logTrace(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  52. def logWarning(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  53. def logWarning(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  54. val minLogEntriesToMaintain: Int
    Attributes
    protected
  55. val name: String

    Returns the user-specified name of the query, or null if not specified.

    Returns the user-specified name of the query, or null if not specified. This name can be specified in the org.apache.spark.sql.streaming.DataStreamWriter as dataframe.writeStream.queryName("query").start(). This name, if set, must be unique across all active queries.

    Definition Classes
    StreamExecution → ProgressReporter → StreamingQuery
    Since

    2.0.0

  56. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  57. var newData: Map[SparkDataStream, LogicalPlan]

    Holds the most recent input data for each source.

    Holds the most recent input data for each source.

    Attributes
    protected
    Definition Classes
    StreamExecution → ProgressReporter
  58. var noNewData: Boolean

    A flag to indicate that a batch has completed with no new data available.

    A flag to indicate that a batch has completed with no new data available.

    Attributes
    protected
  59. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  60. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  61. val offsetLog: OffsetSeqLog

    A write-ahead-log that records the offsets that are present in each batch.

    A write-ahead-log that records the offsets that are present in each batch. In order to ensure that a given batch will always consist of the same data, we write to this log *before* any processing is done. Thus, the Nth record in this log indicated data that is currently being processed and the N-1th entry indicates which offsets have been durably committed to the sink.

  62. var offsetSeqMetadata: OffsetSeqMetadata

    Metadata associated with the offset seq of a batch in the query.

    Metadata associated with the offset seq of a batch in the query.

    Attributes
    protected
    Definition Classes
    StreamExecution → ProgressReporter
  63. val outputMode: OutputMode
  64. val pollingDelayMs: Long
    Attributes
    protected
  65. def postEvent(event: Event): Unit
    Attributes
    protected
    Definition Classes
    StreamExecution → ProgressReporter
  66. val prettyIdString: String

    Pretty identified string of printing in logs.

    Pretty identified string of printing in logs. Format is If name is set "queryName [id = xyz, runId = abc]" else "[id = xyz, runId = abc]"

    Attributes
    protected
  67. def processAllAvailable(): Unit

    Blocks until all available data in the source has been processed and committed to the sink.

    Blocks until all available data in the source has been processed and committed to the sink. This method is intended for testing. Note that in the case of continually arriving data, this method may block forever. Additionally, this method is only guaranteed to block until data that has been synchronously appended data to a org.apache.spark.sql.execution.streaming.Source prior to invocation. (i.e. getOffset must immediately reflect the addition).

    Definition Classes
    StreamExecution → StreamingQuery
    Since

    2.0.0

  68. def purge(threshold: Long): Unit
    Attributes
    protected
  69. val queryExecutionThread: QueryExecutionThread

    The thread that runs the micro-batches of this stream.

    The thread that runs the micro-batches of this stream. Note that this thread must be org.apache.spark.util.UninterruptibleThread to workaround KAFKA-1894: interrupting a running KafkaConsumer may cause endless loop.

  70. def recentProgress: Array[StreamingQueryProgress]

    Returns an array containing the most recent query progress updates.

    Returns an array containing the most recent query progress updates.

    Definition Classes
    ProgressReporter
  71. def recordTriggerOffsets(from: StreamProgress, to: StreamProgress, latest: StreamProgress): Unit

    Record the offsets range this trigger will process.

    Record the offsets range this trigger will process. Call this before updating committedOffsets in StreamExecution to make sure that the correct range is recorded.

    Attributes
    protected
    Definition Classes
    ProgressReporter
  72. def reportTimeTaken[T](triggerDetailKey: String)(body: ⇒ T): T

    Records the duration of running body for the next query progress update.

    Records the duration of running body for the next query progress update.

    Attributes
    protected
    Definition Classes
    ProgressReporter
  73. val resolvedCheckpointRoot: String
  74. val runId: UUID

    Returns the unique id of this run of the query.

    Returns the unique id of this run of the query. That is, every start/restart of a query will generate a unique runId. Therefore, every time a query is restarted from checkpoint, it will have the same id but different runIds.

    Definition Classes
    StreamExecution → ProgressReporter → StreamingQuery
  75. val sink: Table
    Definition Classes
    StreamExecution → ProgressReporter
  76. var sinkCommitProgress: Option[StreamWriterCommitProgress]
    Definition Classes
    StreamExecution → ProgressReporter
  77. val sparkSession: SparkSession

    Returns the SparkSession associated with this.

    Returns the SparkSession associated with this.

    Definition Classes
    StreamExecution → ProgressReporter → StreamingQuery
    Since

    2.0.0

  78. def start(): Unit

    Starts the execution.

    Starts the execution. This returns only after the thread has started and QueryStartedEvent has been posted to all the listeners.

  79. def startTrigger(): Unit

    Begins recording statistics about query progress for a given trigger.

    Begins recording statistics about query progress for a given trigger.

    Attributes
    protected
    Definition Classes
    ProgressReporter
  80. val state: AtomicReference[State]

    Defines the internal state of execution

    Defines the internal state of execution

    Attributes
    protected
  81. def status: StreamingQueryStatus

    Returns the current status of the query.

    Returns the current status of the query.

    Definition Classes
    ProgressReporter
  82. def stopSources(): Unit

    Stops all streaming sources safely.

    Stops all streaming sources safely.

    Attributes
    protected
  83. var streamDeathCause: StreamingQueryException
    Attributes
    protected
  84. val streamMetadata: StreamMetadata

    Metadata associated with the whole query

    Metadata associated with the whole query

    Attributes
    protected
  85. lazy val streamMetrics: MetricsReporter

    Used to report metrics to coda-hale.

    Used to report metrics to coda-hale. This uses id for easier tracking across restarts.

  86. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  87. def toString(): String
    Definition Classes
    StreamExecution → AnyRef → Any
  88. val trigger: Trigger
  89. val triggerClock: Clock
    Definition Classes
    StreamExecution → ProgressReporter
  90. var uniqueSources: Map[SparkDataStream, ReadLimit]

    A list of unique sources in the query plan.

    A list of unique sources in the query plan. This will be set when generating logical plan.

    Attributes
    protected
  91. def updateStatusMessage(message: String): Unit

    Updates the message returned in status.

    Updates the message returned in status.

    Attributes
    protected
    Definition Classes
    ProgressReporter
  92. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  93. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  94. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  95. val watermarkMsMap: Map[Int, Long]

    A map of current watermarks, keyed by the position of the watermark operator in the physical plan.

    A map of current watermarks, keyed by the position of the watermark operator in the physical plan.

    This state is 'soft state', which does not affect the correctness and semantics of watermarks and is not persisted across query restarts. The fault-tolerant watermark state is in offsetSeqMetadata.

    Attributes
    protected

Inherited from ProgressReporter

Inherited from Logging

Inherited from StreamingQuery

Inherited from AnyRef

Inherited from Any

Ungrouped