Package org.graylog2.shared.journal

Class LocalKafkaJournal

java.lang.Object
com.google.common.util.concurrent.AbstractIdleService
org.graylog2.shared.journal.LocalKafkaJournal
All Implemented Interfaces:
com.google.common.util.concurrent.Service, Journal
@Singleton public class LocalKafkaJournal extends com.google.common.util.concurrent.AbstractIdleService implements Journal

  • Field Details

  • Constructor Details

    • LocalKafkaJournal

      @Inject public LocalKafkaJournal(@Named("message_journal_dir") Path journalDirectory, @Named("scheduler") ScheduledExecutorService scheduler, @Named("message_journal_segment_size") com.github.joschi.jadconfig.util.Size segmentSize, @Named("message_journal_segment_age") org.joda.time.Duration segmentAge, @Named("message_journal_max_size") com.github.joschi.jadconfig.util.Size retentionSize, @Named("message_journal_max_age") org.joda.time.Duration retentionAge, @Named("message_journal_flush_interval") long flushInterval, @Named("message_journal_flush_age") org.joda.time.Duration flushAge, @Named("lb_throttle_threshold_percentage") int throttleThresholdPercentage, com.codahale.metrics.MetricRegistry metricRegistry, ServerStatus serverStatus)

    • LocalKafkaJournal

      public LocalKafkaJournal(Path journalDirectory, ScheduledExecutorService scheduler, com.github.joschi.jadconfig.util.Size segmentSize, org.joda.time.Duration segmentAge, com.github.joschi.jadconfig.util.Size retentionSize, org.joda.time.Duration retentionAge, long flushInterval, org.joda.time.Duration flushAge, int throttleThresholdPercentage, com.codahale.metrics.MetricRegistry metricRegistry, ServerStatus serverStatus, String metricPrefix)
      Parameters:
      throttleThresholdPercentage - The journal utilization percent at which throttling will be triggered. Expressed as an integer between 1 and 100. The value -1 disables throttling.

  • Method Details

    • flush

      public void flush()
      Specified by:
      flush in interface Journal

    • getPurgedSegmentsInLastRetention

      public int getPurgedSegmentsInLastRetention()

    • createEntry

      public Journal.Entry createEntry(byte[] idBytes, byte[] messageBytes)
      Creates an opaque object which can be passed to write(java.util.List) for a bulk journal write.
      Specified by:
      createEntry in interface Journal
      Parameters:
      idBytes - a byte array which represents the key for the entry
      messageBytes - the journal entry's payload, i.e. the message itself
      Returns:
      a journal entry to be passed to write(java.util.List)

    • write

      public long write(List<Journal.Entry> entries)
      Writes the list of entries to the journal.
      Specified by:
      write in interface Journal
      Parameters:
      entries - journal entries to be written
      Returns:
      the last position written to in the journal

    • write

      public long write(byte[] idBytes, byte[] messageBytes)
      Writes a single message to the journal and returns the new write position
      Specified by:
      write in interface Journal
      Parameters:
      idBytes - byte array congaing the message id
      messageBytes - encoded message payload
      Returns:
      the last position written to in the journal

    • read

      public List<Journal.JournalReadEntry> read(long requestedMaximumCount)
      Specified by:
      read in interface Journal

    • readNext

      public List<Journal.JournalReadEntry> readNext(long startOffset, long requestedMaximumCount)
      Read next messages from the journal, starting at the given offset. If the underlying journal implementation returns an empty list of entries, but we know there are more entries in the journal, we'll try to skip the problematic offset(s) until we find entries again.
      Parameters:
      startOffset - Offset to start reading at
      requestedMaximumCount - Maximum number of entries to return.
      Returns:
      A list of entries

    • read

      public List<Journal.JournalReadEntry> read(long readOffset, long requestedMaximumCount)
      Read from the journal, starting at the given offset. If the underlying journal implementation returns an empty list of entries, it will be returned even if we know there are more entries in the journal.
      Parameters:
      readOffset - Offset to start reading at
      requestedMaximumCount - Maximum number of entries to return.
      Returns:
      A list of entries

    • markJournalOffsetCommitted

      public void markJournalOffsetCommitted(long offset)
      Upon fully processing, and persistently storing, a batch of messages, the system should mark the message with the highest offset as committed. A background job will write the last position to disk periodically.
      Specified by:
      markJournalOffsetCommitted in interface Journal
      Parameters:
      offset - the offset of the latest committed message

    • flushDirtyLogs

      protected void flushDirtyLogs()
      A Java transliteration of what the scala implementation does, which unfortunately is declared as private

    • getCommittedOffset

      public long getCommittedOffset()

    • getNextReadOffset

      public long getNextReadOffset()

    • startUp

      protected void startUp() throws Exception
      Specified by:
      startUp in class com.google.common.util.concurrent.AbstractIdleService
      Throws:
      Exception

    • shutDown

      protected void shutDown() throws Exception
      Specified by:
      shutDown in class com.google.common.util.concurrent.AbstractIdleService
      Throws:
      Exception

    • cleanupLogs

      public int cleanupLogs()

    • getSegments

      public Iterable<org.graylog.shaded.kafka09.log.LogSegment> getSegments()

    • size

      public long size()
      Returns the journal size in bytes, exluding index files.
      Returns:
      journal size in bytes

    • numberOfSegments

      public int numberOfSegments()
      Returns the number of segments this journal consists of.
      Returns:
      number of segments

    • getCommittedReadOffset

      public long getCommittedReadOffset()
      Returns the highest journal offset that has been writting to persistent storage by Graylog.

      Every message at an offset prior to this one can be considered as processed and does not need to be held in the journal any longer. By default Graylog will try to aggressively flush the journal to consume a smaller amount of disk space.

      Returns:
      the offset of the last message which has been successfully processed.

    • truncateTo

      public void truncateTo(long offset)
      Discards all data in the journal prior to the given offset.
      Parameters:
      offset - offset to truncate to, so that no offset in the journal is larger than this.

    • getLogStartOffset

      public long getLogStartOffset()
      Returns the first valid offset in the entire journal.
      Returns:
      first offset

    • getLogEndOffset

      public long getLogEndOffset()
      returns the offset for the next value to be inserted in the entire journal.
      Returns:
      the next offset value (last valid offset is this number - 1)

    • getThrottleState

      public ThrottleState getThrottleState()
      For informational purposes this method provides access to the current state of the journal.
      Returns:
      the journal state for throttling purposes

    • setThrottleState

      public void setThrottleState(ThrottleState state)