java.io.Closeable
, java.lang.AutoCloseable
, IndexShardComponent
public class Translog extends AbstractIndexShardComponent implements IndexShardComponent, java.io.Closeable
InternalEngine
. The engine
records the current translog generation getGeneration()
in it's commit metadata using TRANSLOG_GENERATION_KEY
to reference the generation that contains all operations that have not yet successfully been committed to the engines lucene index.
Additionally, since Elasticsearch 2.0 the engine also records a TRANSLOG_UUID_KEY
with each commit to ensure a strong association
between the lucene index an the transaction log file. This UUID is used to prevent accidental recovery from a transaction log that belongs to a
different engine.
Each Translog has only one translog file open for writes at any time referenced by a translog generation ID. This ID is written to a translog.ckp file that is designed to fit in a single disk block such that a write of the file is atomic. The checkpoint file is written on each fsync operation of the translog and records the number of operations written, the current translog's file generation, its fsynced offset in bytes, and other important statistics.
When the current translog file reaches a certain size (IndexSettings.INDEX_TRANSLOG_GENERATION_THRESHOLD_SIZE_SETTING
, or when
a clear separation between old and new operations (upon change in primary term), the current file is reopened for read only and a new
write only file is created. Any non-current, read only translog file always has a translog-${gen}.ckp associated with it
which is an fsynced copy of its last translog.ckp such that in disaster recovery last fsynced offsets, number of
operation etc. are still preserved.
Modifier and Type | Class | Description |
---|---|---|
static class |
Translog.Delete |
|
static class |
Translog.Durability |
|
static class |
Translog.Index |
|
static class |
Translog.Location |
|
static class |
Translog.NoOp |
|
static interface |
Translog.Operation |
A generic interface representing an operation performed on the transaction log.
|
static interface |
Translog.Snapshot |
A snapshot of the transaction log, allows to iterate over all the transaction log operations.
|
static class |
Translog.Source |
|
static class |
Translog.TranslogGeneration |
References a transaction log generation
|
Modifier and Type | Field | Description |
---|---|---|
static java.lang.String |
CHECKPOINT_FILE_NAME |
|
static java.lang.String |
CHECKPOINT_SUFFIX |
|
protected ReleasableLock |
readLock |
|
static java.lang.String |
TRANSLOG_FILE_PREFIX |
|
static java.lang.String |
TRANSLOG_FILE_SUFFIX |
|
static java.lang.String |
TRANSLOG_GENERATION_KEY |
|
static java.lang.String |
TRANSLOG_UUID_KEY |
|
protected ReleasableLock |
writeLock |
deprecationLogger, indexSettings, logger, shardId
Constructor | Description |
---|---|
Translog(TranslogConfig config,
java.lang.String expectedTranslogUUID,
TranslogDeletionPolicy deletionPolicy,
java.util.function.LongSupplier globalCheckpointSupplier) |
Creates a new Translog instance.
|
Modifier and Type | Method | Description |
---|---|---|
java.io.Closeable |
acquireRetentionLock() |
Acquires a lock on the translog files, preventing them from being trimmed
|
Translog.Location |
add(Translog.Operation operation) |
Adds an operation to the transaction log.
|
void |
close() |
|
long |
currentFileGeneration() |
Returns the generation of the current transaction log.
|
boolean |
ensureSynced(java.util.stream.Stream<Translog.Location> locations) |
Ensures that all locations in the given stream have been synced / written to the underlying storage.
|
boolean |
ensureSynced(Translog.Location location) |
Ensures that the given location has be synced / written to the underlying storage.
|
int |
estimateTotalOperationsFromMinSeq(long minSeqNo) |
Returns the number of operations in the transaction files that contain operations with seq# above the given number.
|
TranslogConfig |
getConfig() |
|
TranslogDeletionPolicy |
getDeletionPolicy() |
|
static java.lang.String |
getFilename(long generation) |
package private for testing
|
Translog.TranslogGeneration |
getGeneration() |
Returns the current generation of this translog.
|
long |
getLastSyncedGlobalCheckpoint() |
The last synced checkpoint for this translog.
|
Translog.Location |
getLastWriteLocation() |
The a Translog.Location that will sort after the Translog.Location returned by the last write but before any locations which
can be returned by the next write.
|
long |
getMinFileGeneration() |
Returns the minimum file generation referenced by the translog
|
Translog.TranslogGeneration |
getMinGenerationForSeqNo(long seqNo) |
Gets the minimum generation that could contain any sequence number after the specified sequence number, or the current generation if
there is no generation that could any such sequence number.
|
java.lang.Exception |
getTragicException() |
If this
Translog was closed as a side-effect of a tragic exception,
e.g. |
java.lang.String |
getTranslogUUID() |
Returns the translog uuid used to associate a lucene index with a translog.
|
boolean |
isCurrent(Translog.TranslogGeneration generation) |
Returns
true iff the given generation is the current generation of this translog |
boolean |
isOpen() |
Returns
true if this Translog is still open. |
java.nio.file.Path |
location() |
Returns all translog locations as absolute paths.
|
Translog.Snapshot |
newSnapshot() |
Snapshots the current transaction log allowing to safely iterate over the snapshot.
|
Translog.Snapshot |
newSnapshotFromGen(long minGeneration) |
|
Translog.Snapshot |
newSnapshotFromMinSeqNo(long minSeqNo) |
|
static long |
parseIdFromFileName(java.nio.file.Path translogFile) |
Extracts the translog generation from a file name.
|
static long |
readGlobalCheckpoint(java.nio.file.Path location,
java.lang.String expectedTranslogUUID) |
Reads the sequence numbers global checkpoint from the translog checkpoint.
|
static long |
readMinReferencedGen(java.nio.file.Path location) |
Reads the minimum translog generation that referenced by translog from the translog checkpoint.
|
static java.util.List<Translog.Operation> |
readOperations(StreamInput input) |
Reads a list of operations written with
writeOperations(StreamOutput, List) |
void |
rollGeneration() |
Roll the current translog generation into a new generation.
|
boolean |
shouldRollGeneration() |
Tests whether or not the translog generation should be rolled to a new generation.
|
long |
sizeInBytes() |
Returns the size in bytes of the v files
|
long |
sizeOfGensAboveSeqNoInBytes(long minSeqNo) |
Returns the size in bytes of the translog files with ops above the given seqNo
|
TranslogStats |
stats() |
return stats
|
void |
sync() |
Sync's the translog.
|
boolean |
syncNeeded() |
Returns
true if an fsync is required to ensure durability of the translogs operations or it's metadata. |
int |
totalOperations() |
Returns the number of operations in the translog files
|
void |
trimUnreferencedReaders() |
Trims unreferenced translog generations by asking
TranslogDeletionPolicy for the minimum
required generation |
int |
uncommittedOperations() |
Returns the number of operations in the translog files that aren't committed to lucene.
|
long |
uncommittedSizeInBytes() |
Returns the size in bytes of the translog files that aren't committed to lucene.
|
static void |
writeOperationNoSize(BufferedChecksumStreamOutput out,
Translog.Operation op) |
|
static void |
writeOperations(StreamOutput outStream,
java.util.List<Translog.Operation> toWrite) |
Writes all operations in the given iterable to the given output stream including the size of the array
use
readOperations(StreamInput) to read it back. |
indexSettings, nodeName, shardId
indexSettings, shardId
public static final java.lang.String TRANSLOG_GENERATION_KEY
public static final java.lang.String TRANSLOG_UUID_KEY
public static final java.lang.String TRANSLOG_FILE_PREFIX
public static final java.lang.String TRANSLOG_FILE_SUFFIX
public static final java.lang.String CHECKPOINT_SUFFIX
public static final java.lang.String CHECKPOINT_FILE_NAME
protected final ReleasableLock readLock
protected final ReleasableLock writeLock
public Translog(TranslogConfig config, java.lang.String expectedTranslogUUID, TranslogDeletionPolicy deletionPolicy, java.util.function.LongSupplier globalCheckpointSupplier) throws java.io.IOException
Translog.TranslogGeneration
is
null
. If the generation is null
this method is destructive and will delete all files in the translog path given. If
the generation is not null
, this method tries to open the given translog generation. The generation is treated as the last
generation referenced from already committed data. This means all operations that have not yet been committed should be in the
translog file referenced by this generation. The translog creation will fail if this generation can't be opened.config
- the configuration of this translogexpectedTranslogUUID
- the translog uuid to open, null for a new translogdeletionPolicy
- an instance of TranslogDeletionPolicy
that controls when a translog file can be safely
deletedglobalCheckpointSupplier
- a supplier for the global checkpointjava.io.IOException
public static long parseIdFromFileName(java.nio.file.Path translogFile)
java.lang.IllegalArgumentException
- if the path doesn't match the expected pattern.public boolean isOpen()
true
if this Translog
is still open.public void close() throws java.io.IOException
close
in interface java.lang.AutoCloseable
close
in interface java.io.Closeable
java.io.IOException
public java.nio.file.Path location()
public long currentFileGeneration()
public long getMinFileGeneration()
public int uncommittedOperations()
public long uncommittedSizeInBytes()
public int totalOperations()
public long sizeInBytes()
public int estimateTotalOperationsFromMinSeq(long minSeqNo)
public long sizeOfGensAboveSeqNoInBytes(long minSeqNo)
public Translog.Location add(Translog.Operation operation) throws java.io.IOException
operation
- the operation to addjava.io.IOException
- if adding the operation to the translog resulted in an I/O exceptionpublic boolean shouldRollGeneration()
true
if the current generation should be rolled to a new generationpublic Translog.Location getLastWriteLocation()
public long getLastSyncedGlobalCheckpoint()
public Translog.Snapshot newSnapshot() throws java.io.IOException
java.io.IOException
public Translog.Snapshot newSnapshotFromGen(long minGeneration) throws java.io.IOException
java.io.IOException
public Translog.Snapshot newSnapshotFromMinSeqNo(long minSeqNo) throws java.io.IOException
java.io.IOException
public java.io.Closeable acquireRetentionLock()
public void sync() throws java.io.IOException
java.io.IOException
public boolean syncNeeded()
true
if an fsync is required to ensure durability of the translogs operations or it's metadata.public static java.lang.String getFilename(long generation)
public boolean ensureSynced(Translog.Location location) throws java.io.IOException
true
iff this call caused an actual sync operation otherwise false
java.io.IOException
public boolean ensureSynced(java.util.stream.Stream<Translog.Location> locations) throws java.io.IOException
true
iff this call caused an actual sync operation otherwise false
java.io.IOException
public TranslogStats stats()
public TranslogConfig getConfig()
public TranslogDeletionPolicy getDeletionPolicy()
public static java.util.List<Translog.Operation> readOperations(StreamInput input) throws java.io.IOException
writeOperations(StreamOutput, List)
java.io.IOException
public static void writeOperations(StreamOutput outStream, java.util.List<Translog.Operation> toWrite) throws java.io.IOException
readOperations(StreamInput)
to read it back.java.io.IOException
public static void writeOperationNoSize(BufferedChecksumStreamOutput out, Translog.Operation op) throws java.io.IOException
java.io.IOException
public Translog.TranslogGeneration getMinGenerationForSeqNo(long seqNo)
seqNo
- the sequence numberpublic void rollGeneration() throws java.io.IOException
java.io.IOException
- if an I/O exception occurred during any file operationspublic void trimUnreferencedReaders() throws java.io.IOException
TranslogDeletionPolicy
for the minimum
required generationjava.io.IOException
public Translog.TranslogGeneration getGeneration()
public boolean isCurrent(Translog.TranslogGeneration generation)
true
iff the given generation is the current generation of this translogpublic java.lang.Exception getTragicException()
Translog
was closed as a side-effect of a tragic exception,
e.g. disk full while flushing a new segment, this returns the root cause exception.
Otherwise (no tragic exception has occurred) it returns null.public static long readGlobalCheckpoint(java.nio.file.Path location, java.lang.String expectedTranslogUUID) throws java.io.IOException
location
- the location of the translogjava.io.IOException
- if an I/O exception occurred reading the checkpointTranslogCorruptedException
- if the translog is corrupted or mismatched with the given uuidpublic static long readMinReferencedGen(java.nio.file.Path location) throws java.io.IOException
java.io.IOException
public java.lang.String getTranslogUUID()