org.apache.spark.sql.execution.streaming

OffsetSeqLog

class OffsetSeqLog extends HDFSMetadataLog[OffsetSeq]

This class is used to log offsets to persistent files in HDFS. Each file corresponds to a specific batch of offsets. The file format contain a version string in the first line, followed by a the JSON string representation of the offsets separated by a newline character. If a source offset is missing, then that line will contain a string value defined in the SERIALIZED_VOID_OFFSET variable in OffsetSeqLog companion object. For instance, when dealing with LongOffset types: v1 // version 1 metadata {0} // LongOffset 0 {3} // LongOffset 3

Linear Supertypes
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. OffsetSeqLog
  2. HDFSMetadataLog
  3. Logging
  4. MetadataLog
  5. AnyRef
  6. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Instance Constructors

  1. new OffsetSeqLog(sparkSession: SparkSession, path: String)

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any
  3. final def ##(): Int

    Definition Classes
    AnyRef → Any
  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any
  6. def add(batchId: Long, metadata: OffsetSeq): Boolean

    Store the metadata for the specified batchId and return true if successful.

    Store the metadata for the specified batchId and return true if successful. If the batchId's metadata has already been stored, this method will return false.

    Definition Classes
    HDFSMetadataLogMetadataLog
  7. final def asInstanceOf[T0]: T0

    Definition Classes
    Any
  8. val batchFilesFilter: PathFilter

    A PathFilter to filter only batch files

    A PathFilter to filter only batch files

    Attributes
    protected
    Definition Classes
    HDFSMetadataLog
  9. def batchIdToPath(batchId: Long): Path

    Attributes
    protected
    Definition Classes
    HDFSMetadataLog
  10. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  11. def deserialize(in: InputStream): OffsetSeq

    Attributes
    protected
    Definition Classes
    OffsetSeqLogHDFSMetadataLog
  12. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  13. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  14. val fileManager: FileManager

    Attributes
    protected
    Definition Classes
    HDFSMetadataLog
  15. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  16. def get(startId: Option[Long], endId: Option[Long]): Array[(Long, OffsetSeq)]

    Return metadata for batches between startId (inclusive) and endId (inclusive).

    Return metadata for batches between startId (inclusive) and endId (inclusive). If startId is None, just return all batches before endId (inclusive).

    Definition Classes
    HDFSMetadataLogMetadataLog
  17. def get(batchId: Long): Option[OffsetSeq]

    Return the metadata for the specified batchId if it's stored.

    Return the metadata for the specified batchId if it's stored. Otherwise, return None.

    Definition Classes
    HDFSMetadataLogMetadataLog
  18. def get(batchFile: Path): Option[OffsetSeq]

    returns

    the deserialized metadata in a batch file, or None if file not exist.

    Definition Classes
    HDFSMetadataLog
    Exceptions thrown
    IllegalArgumentException

    when path does not point to a batch file.

  19. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any
  20. def getLatest(): Option[(Long, OffsetSeq)]

    Return the latest batch Id and its metadata if exist.

    Return the latest batch Id and its metadata if exist.

    Definition Classes
    HDFSMetadataLogMetadataLog
  21. def getOrderedBatchFiles(): Array[FileStatus]

    Get an array of [FileStatus] referencing batch files.

    Get an array of [FileStatus] referencing batch files. The array is sorted by most recent batch file first to oldest batch file.

    Definition Classes
    HDFSMetadataLog
  22. def hashCode(): Int

    Definition Classes
    AnyRef → Any
  23. def initializeLogIfNecessary(isInterpreter: Boolean): Unit

    Attributes
    protected
    Definition Classes
    Logging
  24. def isBatchFile(path: Path): Boolean

    Attributes
    protected
    Definition Classes
    HDFSMetadataLog
  25. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any
  26. def isTraceEnabled(): Boolean

    Attributes
    protected
    Definition Classes
    Logging
  27. def log: Logger

    Attributes
    protected
    Definition Classes
    Logging
  28. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging
  29. def logDebug(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging
  30. def logError(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging
  31. def logError(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging
  32. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging
  33. def logInfo(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging
  34. def logName: String

    Attributes
    protected
    Definition Classes
    Logging
  35. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging
  36. def logTrace(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging
  37. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging
  38. def logWarning(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging
  39. val metadataPath: Path

    Definition Classes
    HDFSMetadataLog
  40. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  41. final def notify(): Unit

    Definition Classes
    AnyRef
  42. final def notifyAll(): Unit

    Definition Classes
    AnyRef
  43. def pathToBatchId(path: Path): Long

    Attributes
    protected
    Definition Classes
    HDFSMetadataLog
  44. def purge(thresholdBatchId: Long): Unit

    Removes all the log entry earlier than thresholdBatchId (exclusive).

    Removes all the log entry earlier than thresholdBatchId (exclusive).

    Definition Classes
    HDFSMetadataLogMetadataLog
  45. def serialize(offsetSeq: OffsetSeq, out: OutputStream): Unit

    Attributes
    protected
    Definition Classes
    OffsetSeqLogHDFSMetadataLog
  46. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef
  47. def toString(): String

    Definition Classes
    AnyRef → Any
  48. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  49. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  50. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from HDFSMetadataLog[OffsetSeq]

Inherited from Logging

Inherited from MetadataLog[OffsetSeq]

Inherited from AnyRef

Inherited from Any

Ungrouped