streaming

Type Members

1. class BatchCommitLog extends HDFSMetadataLog[String]

   Used to write log files that represent batch commit points in structured streaming.

2. abstract class CompactibleFileStreamLog[T <: AnyRef] extends HDFSMetadataLog[Array[T]]

   An abstract class for compactible metadata logs.

3. class ConsoleSink extends Sink with Logging

4. class ConsoleSinkProvider extends StreamSinkProvider with DataSourceRegister

5. case class EventTimeStats(max: Long, min: Long, avg: Double, count: Long) extends Product with Serializable

   Class for collecting event time stats with an accumulator

6. class EventTimeStatsAccum extends AccumulatorV2[Long, EventTimeStats]

   Accumulator that collects stats on event time in a batch.

7. case class EventTimeWatermarkExec(eventTime: Attribute, delay: CalendarInterval, child: SparkPlan) extends SparkPlan with Product with Serializable

   Used to mark a column as the containing the event time for a given record.

8. class FileStreamOptions extends Logging

   User specified options for file streams.

9. class FileStreamSink extends Sink with Logging

   A sink that writes out results to parquet files.

10. class FileStreamSinkLog extends CompactibleFileStreamLog[SinkFileStatus]

    A special log for FileStreamSink.

11. class FileStreamSource extends Source with Logging

    A very simple source that reads files from the given directory as they appear.

12. class FileStreamSourceLog extends CompactibleFileStreamLog[FileEntry]

13. case class FileStreamSourceOffset(logOffset: Long) extends Offset with Product with Serializable

    Offset for the FileStreamSource.

14. case class FlatMapGroupsWithStateExec(func: (Any, Iterator[Any], LogicalGroupState[Any]) ⇒ Iterator[Any], keyDeserializer: Expression, valueDeserializer: Expression, groupingAttributes: Seq[Attribute], dataAttributes: Seq[Attribute], outputObjAttr: Attribute, stateId: Option[OperatorStateId], stateEncoder: ExpressionEncoder[Any], outputMode: OutputMode, timeoutConf: GroupStateTimeout, batchTimestampMs: Option[Long], eventTimeWatermark: Option[Long], child: SparkPlan) extends SparkPlan with UnaryExecNode with ObjectProducerExec with StateStoreWriter with WatermarkSupport with Product with Serializable

    Physical operator for executing FlatMapGroupsWithState.

15. class ForeachSink[T] extends Sink with Serializable

    A Sink that forwards all data into ForeachWriter according to the contract defined by ForeachWriter.

16. class HDFSMetadataLog[T <: AnyRef] extends MetadataLog[T] with Logging

    A MetadataLog implementation based on HDFS.

17. class IncrementalExecution extends QueryExecution with Logging

    A variant of QueryExecution that allows the execution of the given LogicalPlan plan incrementally.

18. case class LongOffset(offset: Long) extends Offset with Product with Serializable

    A simple offset for sources that produce a single linear stream of data.

19. class ManifestFileCommitProtocol extends FileCommitProtocol with Serializable with Logging

    A FileCommitProtocol that tracks the list of valid files in a manifest file, used in structured streaming.

20. case class MemoryPlan(sink: MemorySink, output: Seq[Attribute]) extends LeafNode with Product with Serializable

    Used to query the data that has been written into a MemorySink.

21. class MemorySink extends Sink with Logging

    A sink that stores the results in memory.

22. case class MemoryStream[A](id: Int, sqlContext: SQLContext)(implicit evidence$2: Encoder[A]) extends Source with Logging with Product with Serializable

    A Source that produces value stored in memory as they are added by the user.

23. trait MetadataLog[T] extends AnyRef

    A general MetadataLog that supports the following features:

24. class MetadataLogFileIndex extends PartitioningAwareFileIndex

    A FileIndex that generates the list of files to processing by reading them from the metadata log files generated by the FileStreamSink.

25. class MetricsReporter extends metrics.source.Source with Logging

    Serves metrics from a org.apache.spark.sql.streaming.StreamingQuery to Codahale/DropWizard metrics

26. abstract class Offset extends AnyRef

    An offset is a monotonically increasing metric used to track progress in the computation of a stream.

27. case class OffsetSeq(offsets: Seq[Option[Offset]], metadata: Option[OffsetSeqMetadata] = scala.None) extends Product with Serializable

    An ordered collection of offsets, used to track the progress of processing data from one or more Sources that are present in a streaming query.

28. class OffsetSeqLog extends HDFSMetadataLog[OffsetSeq]

    This class is used to log offsets to persistent files in HDFS.

29. case class OffsetSeqMetadata(batchWatermarkMs: Long = 0, batchTimestampMs: Long = 0, conf: Map[String, String] = ...) extends Product with Serializable

    Contains metadata associated with a OffsetSeq.

30. case class OneTimeExecutor() extends TriggerExecutor with Product with Serializable

    A trigger executor that runs a single batch only, then terminates.

31. case class OperatorStateId(checkpointLocation: String, operatorId: Long, batchId: Long) extends Product with Serializable

    Used to identify the state store for a given operator.

32. case class ProcessingTimeExecutor(processingTime: ProcessingTime, clock: Clock = ...) extends TriggerExecutor with Logging with Product with Serializable

    A trigger executor that runs a batch every intervalMs milliseconds.

33. trait ProgressReporter extends Logging

    Responsible for continually reporting statistics about the amount of data processed as well as latency for a streaming query.

34. class RateSourceProvider extends StreamSourceProvider with DataSourceRegister

    A source that generates increment long values with timestamps.

35. class RateStreamSource extends Source with Logging

36. case class SerializedOffset(json: String) extends Offset with Product with Serializable

    Used when loading a JSON serialized offset from external storage.

37. trait Sink extends AnyRef

    An interface for systems that can collect the results of a streaming query.

38. case class SinkFileStatus(path: String, size: Long, isDir: Boolean, modificationTime: Long, blockReplication: Int, blockSize: Long, action: String) extends Product with Serializable

    The status of a file outputted by FileStreamSink.

39. trait Source extends AnyRef

    A source of continually arriving data for a streaming query.

40. trait State extends AnyRef

    States for StreamExecution's lifecycle.

41. trait StateStoreReader extends SparkPlan with StatefulOperator

    An operator that reads from a StateStore.

42. case class StateStoreRestoreExec(keyExpressions: Seq[Attribute], stateId: Option[OperatorStateId], child: SparkPlan) extends SparkPlan with UnaryExecNode with StateStoreReader with Product with Serializable

    For each input tuple, the key is calculated and the value from the StateStore is added to the stream (in addition to the input tuple) if present.

43. case class StateStoreSaveExec(keyExpressions: Seq[Attribute], stateId: Option[OperatorStateId] = scala.None, outputMode: Option[OutputMode] = scala.None, eventTimeWatermark: Option[Long] = scala.None, child: SparkPlan) extends SparkPlan with UnaryExecNode with StateStoreWriter with WatermarkSupport with Product with Serializable

    For each input tuple, the key is calculated and the tuple is put into the StateStore.

44. trait StateStoreWriter extends SparkPlan with StatefulOperator

    An operator that writes to a StateStore.

45. trait StatefulOperator extends SparkPlan

    An operator that reads or writes state from the StateStore.

46. class StreamExecution extends StreamingQuery with ProgressReporter with Logging

    Manages the execution of a streaming Spark SQL query that is occurring in a separate thread.

47. abstract class StreamExecutionThread extends UninterruptibleThread

    A special thread to run the stream query.

48. case class StreamMetadata(id: String) extends Product with Serializable

    Contains metadata associated with a StreamingQuery.

49. class StreamProgress extends Map[Source, Offset]

    A helper class that looks like a Map[Source, Offset].

50. case class StreamingDeduplicateExec(keyExpressions: Seq[Attribute], child: SparkPlan, stateId: Option[OperatorStateId] = scala.None, eventTimeWatermark: Option[Long] = scala.None) extends SparkPlan with UnaryExecNode with StateStoreWriter with WatermarkSupport with Product with Serializable

    Physical operator for executing streaming Deduplicate.

51. case class StreamingExecutionRelation(source: Source, output: Seq[Attribute]) extends LeafNode with Product with Serializable

    Used to link a streaming Source of data into a org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.

52. class StreamingQueryListenerBus extends SparkListener with ListenerBus[StreamingQueryListener, Event]

    A bus to forward events to StreamingQueryListeners.

53. class StreamingQueryWrapper extends StreamingQuery with Serializable

    Wrap non-serializable StreamExecution to make the query serializable as it's easy to for it to get captured with normal usage.

54. case class StreamingRelation(dataSource: DataSource, sourceName: String, output: Seq[Attribute]) extends LeafNode with Product with Serializable

    Used to link a streaming DataSource into a org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.

55. case class StreamingRelationExec(sourceName: String, output: Seq[Attribute]) extends SparkPlan with LeafExecNode with Product with Serializable

    A dummy physical plan for StreamingRelation to support org.apache.spark.sql.Dataset.explain

56. class TextSocketSource extends Source with Logging

    A source that reads text lines through a TCP socket, designed only for tutorials and debugging.

57. class TextSocketSourceProvider extends StreamSourceProvider with DataSourceRegister with Logging

58. trait TriggerExecutor extends AnyRef

59. trait WatermarkSupport extends SparkPlan with UnaryExecNode

    An operator that supports watermark.