Package org.graylog2.shared.journal
Class LocalKafkaJournal
java.lang.Object
com.google.common.util.concurrent.AbstractIdleService
org.graylog2.shared.journal.LocalKafkaJournal
- All Implemented Interfaces:
com.google.common.util.concurrent.Service
,Journal
@Singleton
public class LocalKafkaJournal
extends com.google.common.util.concurrent.AbstractIdleService
implements Journal
-
Nested Class Summary
Modifier and TypeClassDescriptionclass
class
Java implementation of the Kafka log retention cleaner.class
class
Nested classes/interfaces inherited from interface org.graylog2.shared.journal.Journal
Journal.Entry, Journal.JournalReadEntry
Nested classes/interfaces inherited from interface com.google.common.util.concurrent.Service
com.google.common.util.concurrent.Service.Listener, com.google.common.util.concurrent.Service.State
-
Field Summary
-
Constructor Summary
ConstructorDescriptionLocalKafkaJournal
(Path journalDirectory, ScheduledExecutorService scheduler, com.github.joschi.jadconfig.util.Size segmentSize, org.joda.time.Duration segmentAge, com.github.joschi.jadconfig.util.Size retentionSize, org.joda.time.Duration retentionAge, long flushInterval, org.joda.time.Duration flushAge, int throttleThresholdPercentage, com.codahale.metrics.MetricRegistry metricRegistry, ServerStatus serverStatus) LocalKafkaJournal
(Path journalDirectory, ScheduledExecutorService scheduler, com.github.joschi.jadconfig.util.Size segmentSize, org.joda.time.Duration segmentAge, com.github.joschi.jadconfig.util.Size retentionSize, org.joda.time.Duration retentionAge, long flushInterval, org.joda.time.Duration flushAge, int throttleThresholdPercentage, com.codahale.metrics.MetricRegistry metricRegistry, ServerStatus serverStatus, String metricPrefix) -
Method Summary
Modifier and TypeMethodDescriptionint
createEntry
(byte[] idBytes, byte[] messageBytes) Creates an opaque object which can be passed towrite(java.util.List)
for a bulk journal write.void
flush()
protected void
A Java transliteration of what the scala implementation does, which unfortunately is declared as privatelong
long
Returns the highest journal offset that has been writting to persistent storage by Graylog.long
returns the offset for the next value to be inserted in the entire journal.long
Returns the first valid offset in the entire journal.long
int
Iterable<org.graylog.shaded.kafka09.log.LogSegment>
For informational purposes this method provides access to the current state of the journal.void
markJournalOffsetCommitted
(long offset) Upon fully processing, and persistently storing, a batch of messages, the system should mark the message with the highest offset as committed.int
Returns the number of segments this journal consists of.read
(long requestedMaximumCount) read
(long readOffset, long requestedMaximumCount) Read from the journal, starting at the given offset.readNext
(long startOffset, long requestedMaximumCount) Read next messages from the journal, starting at the given offset.void
setThrottleState
(ThrottleState state) protected void
shutDown()
long
size()
Returns the journal size in bytes, exluding index files.protected void
startUp()
void
truncateTo
(long offset) Discards all data in the journal prior to the given offset.long
write
(byte[] idBytes, byte[] messageBytes) Writes a single message to the journal and returns the new write positionlong
write
(List<Journal.Entry> entries) Writes the list of entries to the journal.Methods inherited from class com.google.common.util.concurrent.AbstractIdleService
addListener, awaitRunning, awaitRunning, awaitRunning, awaitTerminated, awaitTerminated, awaitTerminated, executor, failureCause, isRunning, serviceName, startAsync, state, stopAsync, toString
-
Field Details
-
DEFAULT_COMMITTED_OFFSET
public static final long DEFAULT_COMMITTED_OFFSET- See Also:
-
NOTIFY_ON_UTILIZATION_PERCENTAGE
public static final int NOTIFY_ON_UTILIZATION_PERCENTAGE- See Also:
-
THRESHOLD_THROTTLING_DISABLED
public static final int THRESHOLD_THROTTLING_DISABLED- See Also:
-
METER_WRITTEN_MESSAGES
- See Also:
-
METER_READ_MESSAGES
- See Also:
-
GAUGE_UNCOMMITTED_MESSAGES
- See Also:
-
-
Constructor Details
-
LocalKafkaJournal
@Inject public LocalKafkaJournal(@Named("message_journal_dir") Path journalDirectory, @Named("scheduler") ScheduledExecutorService scheduler, @Named("message_journal_segment_size") com.github.joschi.jadconfig.util.Size segmentSize, @Named("message_journal_segment_age") org.joda.time.Duration segmentAge, @Named("message_journal_max_size") com.github.joschi.jadconfig.util.Size retentionSize, @Named("message_journal_max_age") org.joda.time.Duration retentionAge, @Named("message_journal_flush_interval") long flushInterval, @Named("message_journal_flush_age") org.joda.time.Duration flushAge, @Named("lb_throttle_threshold_percentage") int throttleThresholdPercentage, com.codahale.metrics.MetricRegistry metricRegistry, ServerStatus serverStatus) -
LocalKafkaJournal
public LocalKafkaJournal(Path journalDirectory, ScheduledExecutorService scheduler, com.github.joschi.jadconfig.util.Size segmentSize, org.joda.time.Duration segmentAge, com.github.joschi.jadconfig.util.Size retentionSize, org.joda.time.Duration retentionAge, long flushInterval, org.joda.time.Duration flushAge, int throttleThresholdPercentage, com.codahale.metrics.MetricRegistry metricRegistry, ServerStatus serverStatus, String metricPrefix) - Parameters:
throttleThresholdPercentage
- The journal utilization percent at which throttling will be triggered. Expressed as an integer between 1 and 100. The value -1 disables throttling.
-
-
Method Details
-
flush
public void flush() -
getPurgedSegmentsInLastRetention
public int getPurgedSegmentsInLastRetention() -
createEntry
Creates an opaque object which can be passed towrite(java.util.List)
for a bulk journal write.- Specified by:
createEntry
in interfaceJournal
- Parameters:
idBytes
- a byte array which represents the key for the entrymessageBytes
- the journal entry's payload, i.e. the message itself- Returns:
- a journal entry to be passed to
write(java.util.List)
-
write
Writes the list of entries to the journal. -
write
public long write(byte[] idBytes, byte[] messageBytes) Writes a single message to the journal and returns the new write position -
read
-
readNext
Read next messages from the journal, starting at the given offset. If the underlying journal implementation returns an empty list of entries, but we know there are more entries in the journal, we'll try to skip the problematic offset(s) until we find entries again.- Parameters:
startOffset
- Offset to start reading atrequestedMaximumCount
- Maximum number of entries to return.- Returns:
- A list of entries
-
read
Read from the journal, starting at the given offset. If the underlying journal implementation returns an empty list of entries, it will be returned even if we know there are more entries in the journal.- Parameters:
readOffset
- Offset to start reading atrequestedMaximumCount
- Maximum number of entries to return.- Returns:
- A list of entries
-
markJournalOffsetCommitted
public void markJournalOffsetCommitted(long offset) Upon fully processing, and persistently storing, a batch of messages, the system should mark the message with the highest offset as committed. A background job will write the last position to disk periodically.- Specified by:
markJournalOffsetCommitted
in interfaceJournal
- Parameters:
offset
- the offset of the latest committed message
-
flushDirtyLogs
protected void flushDirtyLogs()A Java transliteration of what the scala implementation does, which unfortunately is declared as private -
getCommittedOffset
public long getCommittedOffset() -
getNextReadOffset
public long getNextReadOffset() -
startUp
- Specified by:
startUp
in classcom.google.common.util.concurrent.AbstractIdleService
- Throws:
Exception
-
shutDown
- Specified by:
shutDown
in classcom.google.common.util.concurrent.AbstractIdleService
- Throws:
Exception
-
cleanupLogs
public int cleanupLogs() -
getSegments
-
size
public long size()Returns the journal size in bytes, exluding index files.- Returns:
- journal size in bytes
-
numberOfSegments
public int numberOfSegments()Returns the number of segments this journal consists of.- Returns:
- number of segments
-
getCommittedReadOffset
public long getCommittedReadOffset()Returns the highest journal offset that has been writting to persistent storage by Graylog.Every message at an offset prior to this one can be considered as processed and does not need to be held in the journal any longer. By default Graylog will try to aggressively flush the journal to consume a smaller amount of disk space.
- Returns:
- the offset of the last message which has been successfully processed.
-
truncateTo
public void truncateTo(long offset) Discards all data in the journal prior to the given offset.- Parameters:
offset
- offset to truncate to, so that no offset in the journal is larger than this.
-
getLogStartOffset
public long getLogStartOffset()Returns the first valid offset in the entire journal.- Returns:
- first offset
-
getLogEndOffset
public long getLogEndOffset()returns the offset for the next value to be inserted in the entire journal.- Returns:
- the next offset value (last valid offset is this number - 1)
-
getThrottleState
For informational purposes this method provides access to the current state of the journal.- Returns:
- the journal state for throttling purposes
-
setThrottleState
-