Packages

trait Sink extends Table

An interface for systems that can collect the results of a streaming query. In order to preserve exactly once semantics a sink must be idempotent in the face of multiple attempts to add the same batch.

Note that, we extends Table here, to make the v1 streaming sink API be compatible with data source v2.

Linear Supertypes
Table, AnyRef, Any
Known Subclasses
FileStreamSink, ForeachBatchSink
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. Sink
  2. Table
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Abstract Value Members

  1. abstract def addBatch(batchId: Long, data: DataFrame): Unit

    Adds a batch of data to this sink.

    Adds a batch of data to this sink. The data for a given batchId is deterministic and if this method is called more than once with the same batchId (which will happen in the case of failures), then data should only be added once.

    Note 1: You cannot apply any operators on data except consuming it (e.g., collect/foreach). Otherwise, you may get a wrong result.

    Note 2: The method is supposed to be executed synchronously, i.e. the method should only return after data is consumed by sink successfully.

Concrete Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def capabilities(): Set[TableCapability]
    Definition Classes
    Sink → Table
  6. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  7. def columns(): Array[connector.catalog.Column]
    Definition Classes
    Table
  8. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  9. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  10. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  11. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  12. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  13. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  14. def name(): String
    Definition Classes
    Sink → Table
  15. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  16. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  17. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  18. def partitioning(): Array[Transform]
    Definition Classes
    Table
  19. def properties(): Map[String, String]
    Definition Classes
    Table
  20. def schema(): StructType
    Definition Classes
    Sink → Table
  21. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  22. def toString(): String
    Definition Classes
    AnyRef → Any
  23. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  24. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  25. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from Table

Inherited from AnyRef

Inherited from Any

Ungrouped