public class KCVSLog extends Object implements Log, BackendOperation.TransactionalProvider
Log
wrapped around a KeyColumnValueStore
. Each message is written as a column-value pair (Entry
)
into a timeslice slot. A timeslice slot is uniquely identified by:
GraphDatabaseConfiguration.LOG_NUM_BUCKETS
. Messages are written to the buckets
in round-robin fashion and each bucket is identified by a bucket id.
Having multiple buckets per timeslice allows for load balancing across multiple keys in the storage backend.TIMESLICE_INTERVAL
microseconds long. And all messages that are added between
start-time and start-time+TIMESLICE_INTERVAL
end up in the same timeslice. For high throughput logs that might be more messages
than the underlying storage backend can handle per key. In that case, ensure that (2^(partition-bit-width) x (num-buckets) is large enough
to distribute the load.KCVSLogManager
), and the
message id (which is auto-incrementing). These three data points comprise the column of a log message. The actual content of the message
is written into the value.
When MessageReader
are registered, one reader thread per partition id and bucket is created which periodically (as configured) checks for
new messages in the storage backend and invokes the reader.
Read-markers are maintained (for each partition-id & bucket id combination) under a dedicated key in the same KeyColumnValueStoreManager
as the
log messages. The read markers are updated to the current position before each new iteration of reading messages from the log. If the system fails
while reading a batch of messages, a subsequently restarted log reader may therefore read messages twice. Hence, MessageReader
implementations
should exhibit correct behavior for the (rare) circumstance that messages are read twice.
Note: All time values in this class are in microseconds. Hence, there are many cases where milliseconds are converted to microseconds.
Modifier and Type | Field and Description |
---|---|
static ConfigOption<Boolean> |
LOG_KEY_CONSISTENT |
static ConfigOption<Duration> |
LOG_MAX_READ_TIME |
static ConfigOption<Duration> |
LOG_MAX_WRITE_TIME |
static ConfigOption<Duration> |
LOG_READ_LAG_TIME |
static long |
TIMESLICE_INTERVAL
The time period that is stored under one key in the underlying KCVS.
|
Constructor and Description |
---|
KCVSLog(String name,
KCVSLogManager manager,
KeyColumnValueStore store,
Configuration config) |
Modifier and Type | Method and Description |
---|---|
Future<Message> |
add(StaticBuffer content)
###################################
Message Sending
###################################
|
Future<Message> |
add(StaticBuffer content,
StaticBuffer key)
Attempts to add the given content to the log and returns a
Future for this action. |
Future<Message> |
add(StaticBuffer content,
StaticBuffer key,
ExternalPersistor persistor) |
void |
close()
Closes the log by terminating all threads and waiting for their termination.
|
String |
getName()
Returns the name of this log
|
StoreTransaction |
openTx() |
void |
registerReader(ReadMarker readMarker,
MessageReader... reader)
###################################
Message Reading
###################################
|
void |
registerReaders(ReadMarker readMarker,
Iterable<MessageReader> readers)
Registers the given readers with this log.
|
boolean |
unregisterReader(MessageReader reader)
Removes the given reader from the list of registered readers and returns whether this reader was registered in the
first place.
|
public static final ConfigOption<Duration> LOG_MAX_WRITE_TIME
public static final ConfigOption<Duration> LOG_MAX_READ_TIME
public static final ConfigOption<Duration> LOG_READ_LAG_TIME
public static final ConfigOption<Boolean> LOG_KEY_CONSISTENT
public static final long TIMESLICE_INTERVAL
public KCVSLog(String name, KCVSLogManager manager, KeyColumnValueStore store, Configuration config)
public void close() throws BackendException
close
in interface Log
close
in interface BackendOperation.TransactionalProvider
BackendException
public StoreTransaction openTx() throws BackendException
openTx
in interface BackendOperation.TransactionalProvider
BackendException
public Future<Message> add(StaticBuffer content)
public Future<Message> add(StaticBuffer content, StaticBuffer key)
Log
Future
for this action.
In addition, a key is provided to signal the recipient of the log message in partitioned logging systems.
If the log is configured for immediate sending, then any exception encountered during this process is thrown by this method. Otherwise, encountered exceptions are attached to the returned future.
public Future<Message> add(StaticBuffer content, StaticBuffer key, ExternalPersistor persistor)
public void registerReader(ReadMarker readMarker, MessageReader... reader)
registerReader
in interface Log
readMarker
- Indicates where to start reading from the log once message readers are registeredreader
- The readers to register (all at once)Log.registerReaders(ReadMarker, Iterable)
public void registerReaders(ReadMarker readMarker, Iterable<MessageReader> readers)
Log
ReadMarker
.
If no previous readers were registered, invoking this method triggers reader threads to be instantiated.
If readers have been previously registered, then the provided ReadMarker
must be compatible with the
previous ReadMarker
or an exception will be thrown.
registerReaders
in interface Log
readMarker
- Indicates where to start reading from the log once message readers are registeredreaders
- The readers to register (all at once)public boolean unregisterReader(MessageReader reader)
Log
Log.close()
instead.unregisterReader
in interface Log
Copyright © 2012–2024. All rights reserved.