kafka.log

OffsetIndex

class OffsetIndex extends Logging

An index that maps offsets to physical file locations for a particular log segment. This index may be sparse: that is it may not hold an entry for all messages in the log.

The index is stored in a file that is pre-allocated to hold a fixed maximum number of 8-byte entries.

The index supports lookups against a memory-map of this file. These lookups are done using a simple binary search variant to locate the offset/location pair for the greatest offset less than or equal to the target offset.

Index files can be opened in two ways: either as an empty, mutable index that allows appends or an immutable read-only index file that has previously been populated. The makeReadOnly method will turn a mutable file into an immutable one and truncate off any extra bytes. This is done when the index file is rolled over.

No attempt is made to checksum the contents of this file, in the event of a crash it is rebuilt.

The file format is a series of entries. The physical format is a 4 byte "relative" offset and a 4 byte file location for the message with that offset. The offset stored is relative to the base offset of the index file. So, for example, if the base offset was 50, then the offset 55 would be stored as 5. Using relative offsets in this way let's us use only 4 bytes for the offset.

The frequency of entries is up to the user of this class.

All external APIs translate from relative offsets to full offsets, so users of this class do not interact with the internal storage format.

Linear Supertypes
Logging, AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. OffsetIndex
  2. Logging
  3. AnyRef
  4. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Instance Constructors

  1. new OffsetIndex(file: File, baseOffset: Long, maxIndexSize: Int = -1)

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any
  3. final def ##(): Int

    Definition Classes
    AnyRef → Any
  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any
  6. def append(offset: Long, position: Int): Unit

    Append entry for the given offset/location pair to the index.

    Append entry for the given offset/location pair to the index. This entry must have a larger offset than all subsequent entries.

  7. final def asInstanceOf[T0]: T0

    Definition Classes
    Any
  8. val baseOffset: Long

  9. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  10. def close(): Unit

    Close the index

  11. def debug(msg: ⇒ String, e: ⇒ Throwable): Unit

    Definition Classes
    Logging
  12. def debug(e: ⇒ Throwable): Any

    Definition Classes
    Logging
  13. def debug(msg: ⇒ String): Unit

    Definition Classes
    Logging
  14. def delete(): Boolean

    Delete this index file

  15. def entries(): Int

    The number of entries in this index

  16. def entry(n: Int): OffsetPosition

    Get the nth offset mapping from the index

  17. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  18. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  19. def error(msg: ⇒ String, e: ⇒ Throwable): Unit

    Definition Classes
    Logging
  20. def error(e: ⇒ Throwable): Any

    Definition Classes
    Logging
  21. def error(msg: ⇒ String): Unit

    Definition Classes
    Logging
  22. def fatal(msg: ⇒ String, e: ⇒ Throwable): Unit

    Definition Classes
    Logging
  23. def fatal(e: ⇒ Throwable): Any

    Definition Classes
    Logging
  24. def fatal(msg: ⇒ String): Unit

    Definition Classes
    Logging
  25. val file: File

  26. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  27. def flush(): Unit

    Flush the data in the index to disk

  28. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any
  29. def hashCode(): Int

    Definition Classes
    AnyRef → Any
  30. def info(msg: ⇒ String, e: ⇒ Throwable): Unit

    Definition Classes
    Logging
  31. def info(e: ⇒ Throwable): Any

    Definition Classes
    Logging
  32. def info(msg: ⇒ String): Unit

    Definition Classes
    Logging
  33. def isFull: Boolean

    True iff there are no more slots available in this index

  34. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any
  35. var lastOffset: Long

  36. var logIdent: String

    Attributes
    protected
    Definition Classes
    Logging
  37. lazy val logger: Logger

    Definition Classes
    Logging
  38. val loggerName: String

    Definition Classes
    Logging
  39. def lookup(targetOffset: Long): OffsetPosition

    Find the largest offset less than or equal to the given targetOffset and return a pair holding this offset and it's corresponding physical file position.

    Find the largest offset less than or equal to the given targetOffset and return a pair holding this offset and it's corresponding physical file position. If the target offset is smaller than the least entry in the index (or the index is empty), the pair (baseOffset, 0) is returned.

  40. def maxEntries: Int

  41. val maxIndexSize: Int

  42. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  43. final def notify(): Unit

    Definition Classes
    AnyRef
  44. final def notifyAll(): Unit

    Definition Classes
    AnyRef
  45. def resize(newSize: Int): Unit

    Reset the size of the memory map and the underneath file.

    Reset the size of the memory map and the underneath file. This is used in two kinds of cases: (1) in trimToValidSize() which is called at closing the segment or new segment being rolled; (2) at loading segments from disk or truncating back to an old segment where a new log segment became active; we want to reset the index size to maximum index size to avoid rolling new segment.

  46. def swallow(action: ⇒ Unit): Unit

    Definition Classes
    Logging
  47. def swallowDebug(action: ⇒ Unit): Unit

    Definition Classes
    Logging
  48. def swallowError(action: ⇒ Unit): Unit

    Definition Classes
    Logging
  49. def swallowInfo(action: ⇒ Unit): Unit

    Definition Classes
    Logging
  50. def swallowTrace(action: ⇒ Unit): Unit

    Definition Classes
    Logging
  51. def swallowWarn(action: ⇒ Unit): Unit

    Definition Classes
    Logging
  52. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef
  53. def toString(): String

    Definition Classes
    AnyRef → Any
  54. def trace(msg: ⇒ String, e: ⇒ Throwable): Unit

    Definition Classes
    Logging
  55. def trace(e: ⇒ Throwable): Any

    Definition Classes
    Logging
  56. def trace(msg: ⇒ String): Unit

    Definition Classes
    Logging
  57. def trimToValidSize(): Unit

    Trim this segment to fit just the valid entries, deleting all trailing unwritten bytes from the file.

  58. def truncate(): Unit

    Truncate the entire index

  59. def truncateTo(offset: Long): Unit

    Remove all entries from the index which have an offset greater than or equal to the given offset.

    Remove all entries from the index which have an offset greater than or equal to the given offset. Truncating to an offset larger than the largest in the index has no effect.

  60. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  61. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  62. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  63. def warn(msg: ⇒ String, e: ⇒ Throwable): Unit

    Definition Classes
    Logging
  64. def warn(e: ⇒ Throwable): Any

    Definition Classes
    Logging
  65. def warn(msg: ⇒ String): Unit

    Definition Classes
    Logging

Inherited from Logging

Inherited from AnyRef

Inherited from Any

Ungrouped