Packages

trait Source extends SparkDataStream

A source of continually arriving data for a streaming query. A Source must have a monotonically increasing notion of progress that can be represented as an Offset. Spark will regularly query each Source to see if any more data is available.

Note that, we extends SparkDataStream here, to make the v1 streaming source API be compatible with data source v2.

Linear Supertypes
SparkDataStream, AnyRef, Any
Known Subclasses
AvailableNowSourceWrapper, FileStreamSource
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. Source
  2. SparkDataStream
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Abstract Value Members

  1. abstract def getBatch(start: Option[Offset], end: Offset): DataFrame

    Returns the data that is between the offsets (start, end].

    Returns the data that is between the offsets (start, end]. When start is None, then the batch should begin with the first record. This method must always return the same data for a particular start and end pair; even after the Source has been restarted on a different node.

    Higher layers will always call this method with a value of start greater than or equal to the last value passed to commit and a value of end less than or equal to the last value returned by getOffset

    It is possible for the Offset type to be a SerializedOffset when it was obtained from the log. Moreover, StreamExecution only compares the Offset JSON representation to determine if the two objects are equal. This could have ramifications when upgrading Offset JSON formats i.e., two equivalent Offset objects could differ between version. Consequently, StreamExecution may call this method with two such equivalent Offset objects. In which case, the Source should return an empty DataFrame

  2. abstract def getOffset: Option[Offset]

    Returns the maximum available offset for this source.

    Returns the maximum available offset for this source. Returns None if this source has never received any data.

  3. abstract def schema: StructType

    Returns the schema of the data from this source

  4. abstract def stop(): Unit
    Definition Classes
    SparkDataStream

Concrete Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  6. def commit(end: connector.read.streaming.Offset): Unit
    Definition Classes
    Source → SparkDataStream
  7. def commit(end: Offset): Unit

    Informs the source that Spark has completed processing all data for offsets less than or equal to end and will only request offsets greater than end in the future.

  8. def deserializeOffset(json: String): connector.read.streaming.Offset
    Definition Classes
    Source → SparkDataStream
  9. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  10. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  11. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  12. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  13. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  14. def initialOffset(): connector.read.streaming.Offset
    Definition Classes
    Source → SparkDataStream
  15. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  16. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  17. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  18. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  19. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  20. def toString(): String
    Definition Classes
    AnyRef → Any
  21. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  22. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  23. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from SparkDataStream

Inherited from AnyRef

Inherited from Any

Ungrouped