Class/Object

org.apache.spark.sql.execution.streaming

OffsetSeqLog

Related Docs: object OffsetSeqLog | package streaming

Permalink

class OffsetSeqLog extends HDFSMetadataLog[OffsetSeq]

This class is used to log offsets to persistent files in HDFS. Each file corresponds to a specific batch of offsets. The file format contain a version string in the first line, followed by a the JSON string representation of the offsets separated by a newline character. If a source offset is missing, then that line will contain a string value defined in the SERIALIZED_VOID_OFFSET variable in OffsetSeqLog companion object. For instance, when dealing with LongOffset types: v1 // version 1 metadata {0} // LongOffset 0 {3} // LongOffset 3

Linear Supertypes
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. OffsetSeqLog
  2. HDFSMetadataLog
  3. Logging
  4. MetadataLog
  5. AnyRef
  6. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new OffsetSeqLog(sparkSession: SparkSession, path: String)

    Permalink

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  4. def add(batchId: Long, metadata: OffsetSeq): Boolean

    Permalink

    Store the metadata for the specified batchId and return true if successful.

    Store the metadata for the specified batchId and return true if successful. If the batchId's metadata has already been stored, this method will return false.

    Definition Classes
    HDFSMetadataLogMetadataLog
  5. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  6. val batchFilesFilter: PathFilter

    Permalink

    A PathFilter to filter only batch files

    A PathFilter to filter only batch files

    Attributes
    protected
    Definition Classes
    HDFSMetadataLog
  7. def batchIdToPath(batchId: Long): Path

    Permalink
    Attributes
    protected
    Definition Classes
    HDFSMetadataLog
  8. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  9. def deserialize(in: InputStream): OffsetSeq

    Permalink
    Attributes
    protected
    Definition Classes
    OffsetSeqLogHDFSMetadataLog
  10. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  11. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  12. val fileManager: FileManager

    Permalink
    Attributes
    protected
    Definition Classes
    HDFSMetadataLog
  13. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  14. def get(startId: Option[Long], endId: Option[Long]): Array[(Long, OffsetSeq)]

    Permalink

    Return metadata for batches between startId (inclusive) and endId (inclusive).

    Return metadata for batches between startId (inclusive) and endId (inclusive). If startId is None, just return all batches before endId (inclusive).

    Definition Classes
    HDFSMetadataLogMetadataLog
  15. def get(batchId: Long): Option[OffsetSeq]

    Permalink

    Return the metadata for the specified batchId if it's stored.

    Return the metadata for the specified batchId if it's stored. Otherwise, return None.

    Definition Classes
    HDFSMetadataLogMetadataLog
  16. def get(batchFile: Path): Option[OffsetSeq]

    Permalink

    returns

    the deserialized metadata in a batch file, or None if file not exist.

    Definition Classes
    HDFSMetadataLog
    Exceptions thrown

    IllegalArgumentException when path does not point to a batch file.

  17. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
  18. def getLatest(): Option[(Long, OffsetSeq)]

    Permalink

    Return the latest batch Id and its metadata if exist.

    Return the latest batch Id and its metadata if exist.

    Definition Classes
    HDFSMetadataLogMetadataLog
  19. def getOrderedBatchFiles(): Array[FileStatus]

    Permalink

    Get an array of [FileStatus] referencing batch files.

    Get an array of [FileStatus] referencing batch files. The array is sorted by most recent batch file first to oldest batch file.

    Definition Classes
    HDFSMetadataLog
  20. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  21. def initializeLogIfNecessary(isInterpreter: Boolean): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  22. def isBatchFile(path: Path): Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    HDFSMetadataLog
  23. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  24. def isTraceEnabled(): Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  25. def log: Logger

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  26. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  27. def logDebug(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  28. def logError(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  29. def logError(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  30. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  31. def logInfo(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  32. def logName: String

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  33. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  34. def logTrace(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  35. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  36. def logWarning(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  37. val metadataPath: Path

    Permalink
    Definition Classes
    HDFSMetadataLog
  38. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  39. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
  40. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
  41. def pathToBatchId(path: Path): Long

    Permalink
    Attributes
    protected
    Definition Classes
    HDFSMetadataLog
  42. def purge(thresholdBatchId: Long): Unit

    Permalink

    Removes all the log entry earlier than thresholdBatchId (exclusive).

    Removes all the log entry earlier than thresholdBatchId (exclusive).

    Definition Classes
    HDFSMetadataLogMetadataLog
  43. def serialize(offsetSeq: OffsetSeq, out: OutputStream): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    OffsetSeqLogHDFSMetadataLog
  44. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  45. def toString(): String

    Permalink
    Definition Classes
    AnyRef → Any
  46. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  47. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  48. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from HDFSMetadataLog[OffsetSeq]

Inherited from Logging

Inherited from MetadataLog[OffsetSeq]

Inherited from AnyRef

Inherited from Any

Ungrouped