Packages

c

org.apache.spark.sql.execution.streaming

StreamingQueryWrapper

class StreamingQueryWrapper extends StreamingQuery with Serializable

Wrap non-serializable StreamExecution to make the query serializable as it's easy for it to get captured with normal usage. It's safe to capture the query but not use it in executors. However, if the user tries to call its methods, it will throw IllegalStateException.

Linear Supertypes
Serializable, Serializable, StreamingQuery, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. StreamingQueryWrapper
  2. Serializable
  3. Serializable
  4. StreamingQuery
  5. AnyRef
  6. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new StreamingQueryWrapper(_streamingQuery: StreamExecution)

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def awaitTermination(timeoutMs: Long): Boolean

    Waits for the termination of this query, either by query.stop() or by an exception.

    Waits for the termination of this query, either by query.stop() or by an exception. If the query has terminated with an exception, then the exception will be thrown. Otherwise, it returns whether the query has terminated or not within the timeoutMs milliseconds.

    If the query has terminated, then all subsequent calls to this method will either return true immediately (if the query was terminated by stop()), or throw the exception immediately (if the query has terminated with exception).

    Definition Classes
    StreamingQueryWrapper → StreamingQuery
    Since

    2.0.0

    Exceptions thrown

    StreamingQueryException if the query has terminated with an exception

  6. def awaitTermination(): Unit

    Waits for the termination of this query, either by query.stop() or by an exception.

    Waits for the termination of this query, either by query.stop() or by an exception. If the query has terminated with an exception, then the exception will be thrown.

    If the query has terminated, then all subsequent calls to this method will either return immediately (if the query was terminated by stop()), or throw the exception immediately (if the query has terminated with exception).

    Definition Classes
    StreamingQueryWrapper → StreamingQuery
    Since

    2.0.0

    Exceptions thrown

    StreamingQueryException if the query has terminated with an exception.

  7. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  8. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  9. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  10. def exception: Option[StreamingQueryException]

    Returns the StreamingQueryException if the query was terminated by an exception.

    Returns the StreamingQueryException if the query was terminated by an exception.

    Definition Classes
    StreamingQueryWrapper → StreamingQuery
    Since

    2.0.0

  11. def explain(extended: Boolean): Unit

    Prints the physical plan to the console for debugging purposes.

    Prints the physical plan to the console for debugging purposes.

    extended

    whether to do extended explain or not

    Definition Classes
    StreamingQueryWrapper → StreamingQuery
    Since

    2.0.0

  12. def explain(): Unit

    Prints the physical plan to the console for debugging purposes.

    Prints the physical plan to the console for debugging purposes.

    Definition Classes
    StreamingQueryWrapper → StreamingQuery
    Since

    2.0.0

  13. def explainInternal(extended: Boolean): String

    This method is called in Python.

    This method is called in Python. Python cannot call "explain" directly as it outputs in the JVM process, which may not be visible in Python process.

  14. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  15. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  16. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  17. def id: UUID

    Returns the unique id of this query that persists across restarts from checkpoint data.

    Returns the unique id of this query that persists across restarts from checkpoint data. That is, this id is generated when a query is started for the first time, and will be the same every time it is restarted from checkpoint data. Also see runId.

    Definition Classes
    StreamingQueryWrapper → StreamingQuery
    Since

    2.1.0

  18. def isActive: Boolean

    Returns true if this query is actively running.

    Returns true if this query is actively running.

    Definition Classes
    StreamingQueryWrapper → StreamingQuery
    Since

    2.0.0

  19. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  20. def lastProgress: StreamingQueryProgress

    Returns the most recent StreamingQueryProgress update of this streaming query.

    Returns the most recent StreamingQueryProgress update of this streaming query.

    Definition Classes
    StreamingQueryWrapper → StreamingQuery
    Since

    2.1.0

  21. def name: String

    Returns the user-specified name of the query, or null if not specified.

    Returns the user-specified name of the query, or null if not specified. This name can be specified in the org.apache.spark.sql.streaming.DataStreamWriter as dataframe.writeStream.queryName("query").start(). This name, if set, must be unique across all active queries.

    Definition Classes
    StreamingQueryWrapper → StreamingQuery
    Since

    2.0.0

  22. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  23. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  24. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  25. def processAllAvailable(): Unit

    Blocks until all available data in the source has been processed and committed to the sink.

    Blocks until all available data in the source has been processed and committed to the sink. This method is intended for testing. Note that in the case of continually arriving data, this method may block forever. Additionally, this method is only guaranteed to block until data that has been synchronously appended data to a org.apache.spark.sql.execution.streaming.Source prior to invocation. (i.e. getOffset must immediately reflect the addition).

    Definition Classes
    StreamingQueryWrapper → StreamingQuery
    Since

    2.0.0

  26. def recentProgress: Array[StreamingQueryProgress]

    Returns an array of the most recent StreamingQueryProgress updates for this query.

    Returns an array of the most recent StreamingQueryProgress updates for this query. The number of progress updates retained for each stream is configured by Spark session configuration spark.sql.streaming.numRecentProgressUpdates.

    Definition Classes
    StreamingQueryWrapper → StreamingQuery
    Since

    2.1.0

  27. def runId: UUID

    Returns the unique id of this run of the query.

    Returns the unique id of this run of the query. That is, every start/restart of a query will generate a unique runId. Therefore, every time a query is restarted from checkpoint, it will have the same id but different runIds.

    Definition Classes
    StreamingQueryWrapper → StreamingQuery
  28. def sparkSession: SparkSession

    Returns the SparkSession associated with this.

    Returns the SparkSession associated with this.

    Definition Classes
    StreamingQueryWrapper → StreamingQuery
    Since

    2.0.0

  29. def status: StreamingQueryStatus

    Returns the current status of the query.

    Returns the current status of the query.

    Definition Classes
    StreamingQueryWrapper → StreamingQuery
    Since

    2.0.2

  30. def stop(): Unit

    Stops the execution of this query if it is running.

    Stops the execution of this query if it is running. This waits until the termination of the query execution threads or until a timeout is hit.

    By default stop will block indefinitely. You can configure a timeout by the configuration spark.sql.streaming.stopTimeout. A timeout of 0 (or negative) milliseconds will block indefinitely. If a TimeoutException is thrown, users can retry stopping the stream. If the issue persists, it is advisable to kill the Spark application.

    Definition Classes
    StreamingQueryWrapper → StreamingQuery
    Since

    2.0.0

  31. def streamingQuery: StreamExecution
  32. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  33. def toString(): String
    Definition Classes
    AnyRef → Any
  34. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  35. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  36. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from Serializable

Inherited from Serializable

Inherited from StreamingQuery

Inherited from AnyRef

Inherited from Any

Ungrouped