Class Store
- java.lang.Object
-
- org.elasticsearch.index.shard.AbstractIndexShardComponent
-
- org.elasticsearch.index.store.Store
-
- All Implemented Interfaces:
java.io.Closeable
,java.lang.AutoCloseable
,RefCounted
,IndexShardComponent
public class Store extends AbstractIndexShardComponent implements java.io.Closeable, RefCounted
A Store provides plain access to files written by an elasticsearch index shard. Each shard has a dedicated store that is uses to access Lucene's Directory which represents the lowest level of file abstraction in Lucene used to read and write Lucene indices. This class also provides access to metadata information like checksums for committed files. A committed file is a file that belongs to a segment written by a Lucene commit. Files that have not been committed ie. created during a merge or a shard refresh / NRT reopen are not considered in the MetadataSnapshot.Note: If you use a store it's reference count should be increased before using it by calling #incRef and a corresponding #decRef must be called in a try/finally block to release the store again ie.:
store.incRef(); try { // use the store... } finally { store.decRef(); }
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description static class
Store.MetadataSnapshot
Represents a snapshot of the current directory build from the latest Lucene commit.static interface
Store.OnClose
A listener that is executed once the store is closed and all references to it are releasedstatic class
Store.RecoveryDiff
A class representing the diff between a recovery source and recovery target
-
Field Summary
Fields Modifier and Type Field Description static java.lang.String
CORRUPTED
static Setting<TimeValue>
INDEX_STORE_STATS_REFRESH_INTERVAL_SETTING
-
Fields inherited from class org.elasticsearch.index.shard.AbstractIndexShardComponent
deprecationLogger, indexSettings, logger, shardId
-
-
Constructor Summary
Constructors Constructor Description Store(ShardId shardId, IndexSettings indexSettings, org.apache.lucene.store.Directory directory, ShardLock shardLock)
Store(ShardId shardId, IndexSettings indexSettings, org.apache.lucene.store.Directory directory, ShardLock shardLock, Store.OnClose onClose)
-
Method Summary
Modifier and Type Method Description void
associateIndexWithNewTranslog(java.lang.String translogUUID)
Force bakes the given translog generation as recovery information in the lucene index.void
bootstrapNewHistory()
Marks an existing lucene index with a new history uuid.void
bootstrapNewHistory(long localCheckpoint, long maxSeqNo)
Marks an existing lucene index with a new history uuid and sets the given local checkpoint as well as the maximum sequence number.org.apache.lucene.index.CheckIndex.Status
checkIndex(java.io.PrintStream out)
Checks and returns the status of the existing index in this store.static void
checkIntegrity(StoreFileMetaData md, org.apache.lucene.store.Directory directory)
boolean
checkIntegrityNoException(StoreFileMetaData md)
static boolean
checkIntegrityNoException(StoreFileMetaData md, org.apache.lucene.store.Directory directory)
void
cleanupAndVerify(java.lang.String reason, Store.MetadataSnapshot sourceMetaData)
This method deletes every file in this store that is not contained in the given source meta data or is a legacy checksum file.void
close()
void
createEmpty()
creates an empty lucene index and a corresponding empty translog.org.apache.lucene.store.IndexOutput
createVerifyingOutput(java.lang.String fileName, StoreFileMetaData metadata, org.apache.lucene.store.IOContext context)
The returned IndexOutput validates the files checksum.void
decRef()
Decreases the refCount of this Store instance.void
deleteQuiet(java.lang.String... files)
static java.lang.String
digestToString(long digest)
Produces a string representation of the given digest value.org.apache.lucene.store.Directory
directory()
boolean
ensureIndexHas6xCommitTags()
A 5.x index does not have either historyUUDID or sequence number markers as these markers are introduced in 6.0+.void
failIfCorrupted()
Store.MetadataSnapshot
getMetadata(org.apache.lucene.index.IndexCommit commit)
Returns a new MetadataSnapshot for the given commit.Store.MetadataSnapshot
getMetadata(org.apache.lucene.index.IndexCommit commit, boolean lockDirectory)
Returns a new MetadataSnapshot for the given commit.void
incRef()
Increments the refCount of this Store instance.static boolean
isAutogenerated(java.lang.String name)
Returns true if the file is auto-generated by the store and shouldn't be deleted during cleanup.boolean
isMarkedCorrupted()
static SequenceNumbers.CommitInfo
loadSeqNoInfo(org.apache.lucene.index.IndexCommit commit)
Loads the maximum sequence number and local checkpoint from the given Lucene commit point or the latest if not provided.void
markStoreCorrupted(java.io.IOException exception)
Marks this store as corrupted.org.apache.lucene.store.IndexInput
openVerifyingInput(java.lang.String filename, org.apache.lucene.store.IOContext context, StoreFileMetaData metadata)
org.apache.lucene.index.SegmentInfos
readLastCommittedSegmentsInfo()
Returns the last committed segments info for this storestatic Store.MetadataSnapshot
readMetadataSnapshot(java.nio.file.Path indexLocation, ShardId shardId, NodeEnvironment.ShardLocker shardLocker, org.apache.logging.log4j.Logger logger)
Reads a MetadataSnapshot from the given index locations or returns an empty snapshot if it can't be read.int
refCount()
Returns the current reference count.void
removeCorruptionMarker()
Deletes all corruption markers from this store.void
renameTempFilesSafe(java.util.Map<java.lang.String,java.lang.String> tempFileMap)
Renames all the given files from the key of the map to the value of the map.StoreStats
stats()
void
trimUnsafeCommits(long lastSyncedGlobalCheckpoint, long minRetainedTranslogGen, Version indexVersionCreated)
Keeping existing unsafe commits when opening an engine can be problematic because these commits are not safe at the recovering time but they can suddenly become safe in the future.boolean
tryIncRef()
Tries to increment the refCount of this Store instance.static void
tryOpenIndex(java.nio.file.Path indexLocation, ShardId shardId, NodeEnvironment.ShardLocker shardLocker, org.apache.logging.log4j.Logger logger)
Tries to open an index for the given location.static void
verify(org.apache.lucene.store.IndexInput input)
static void
verify(org.apache.lucene.store.IndexOutput output)
-
Methods inherited from class org.elasticsearch.index.shard.AbstractIndexShardComponent
indexSettings, nodeName, shardId
-
-
-
-
Field Detail
-
CORRUPTED
public static final java.lang.String CORRUPTED
- See Also:
- Constant Field Values
-
-
Constructor Detail
-
Store
public Store(ShardId shardId, IndexSettings indexSettings, org.apache.lucene.store.Directory directory, ShardLock shardLock)
-
Store
public Store(ShardId shardId, IndexSettings indexSettings, org.apache.lucene.store.Directory directory, ShardLock shardLock, Store.OnClose onClose)
-
-
Method Detail
-
directory
public org.apache.lucene.store.Directory directory()
-
readLastCommittedSegmentsInfo
public org.apache.lucene.index.SegmentInfos readLastCommittedSegmentsInfo() throws java.io.IOException
Returns the last committed segments info for this store- Throws:
java.io.IOException
- if the index is corrupted or the segments file is not present
-
loadSeqNoInfo
public static SequenceNumbers.CommitInfo loadSeqNoInfo(org.apache.lucene.index.IndexCommit commit) throws java.io.IOException
Loads the maximum sequence number and local checkpoint from the given Lucene commit point or the latest if not provided.- Parameters:
commit
- the commit point to load seqno stats, or the last commit in the store if the parameter is null- Returns:
SequenceNumbers.CommitInfo
containing information about the last commit- Throws:
java.io.IOException
- if an I/O exception occurred reading the latest Lucene commit point from disk
-
getMetadata
public Store.MetadataSnapshot getMetadata(org.apache.lucene.index.IndexCommit commit) throws java.io.IOException
Returns a new MetadataSnapshot for the given commit. If the given commit isnull
the latest commit point is used. Note that this method requires the caller verify it has the right to access the store and no concurrent file changes are happening. If in doubt, you probably want to use one of the following:readMetadataSnapshot(Path, ShardId, NodeEnvironment.ShardLocker, Logger)
to read a meta data while lockingIndexShard.snapshotStoreMetadata()
to safely read from an existing shardIndexShard.acquireLastIndexCommit(boolean)
to get anIndexCommit
which is safe to use but has to be freed- Parameters:
commit
- the index commit to read the snapshot from ornull
if the latest snapshot should be read from the directory- Throws:
org.apache.lucene.index.CorruptIndexException
- if the lucene index is corrupted. This can be caused by a checksum mismatch or an unexpected exception when opening the index reading the segments file.org.apache.lucene.index.IndexFormatTooOldException
- if the lucene index is too old to be opened.org.apache.lucene.index.IndexFormatTooNewException
- if the lucene index is too new to be opened.java.io.FileNotFoundException
- if one or more files referenced by a commit are not present.java.nio.file.NoSuchFileException
- if one or more files referenced by a commit are not present.org.apache.lucene.index.IndexNotFoundException
- if the commit point can't be found in this storejava.io.IOException
-
getMetadata
public Store.MetadataSnapshot getMetadata(org.apache.lucene.index.IndexCommit commit, boolean lockDirectory) throws java.io.IOException
Returns a new MetadataSnapshot for the given commit. If the given commit isnull
the latest commit point is used. Note that this method requires the caller verify it has the right to access the store and no concurrent file changes are happening. If in doubt, you probably want to use one of the following:readMetadataSnapshot(Path, ShardId, NodeEnvironment.ShardLocker, Logger)
to read a meta data while lockingIndexShard.snapshotStoreMetadata()
to safely read from an existing shardIndexShard.acquireLastIndexCommit(boolean)
to get anIndexCommit
which is safe to use but has to be freed- Parameters:
commit
- the index commit to read the snapshot from ornull
if the latest snapshot should be read from the directorylockDirectory
- iftrue
the index writer lock will be obtained before reading the snapshot. This should only be used if there is no started shard using this store.- Throws:
org.apache.lucene.index.CorruptIndexException
- if the lucene index is corrupted. This can be caused by a checksum mismatch or an unexpected exception when opening the index reading the segments file.org.apache.lucene.index.IndexFormatTooOldException
- if the lucene index is too old to be opened.org.apache.lucene.index.IndexFormatTooNewException
- if the lucene index is too new to be opened.java.io.FileNotFoundException
- if one or more files referenced by a commit are not present.java.nio.file.NoSuchFileException
- if one or more files referenced by a commit are not present.org.apache.lucene.index.IndexNotFoundException
- if the commit point can't be found in this storejava.io.IOException
-
renameTempFilesSafe
public void renameTempFilesSafe(java.util.Map<java.lang.String,java.lang.String> tempFileMap) throws java.io.IOException
Renames all the given files from the key of the map to the value of the map. All successfully renamed files are removed from the map in-place.- Throws:
java.io.IOException
-
checkIndex
public org.apache.lucene.index.CheckIndex.Status checkIndex(java.io.PrintStream out) throws java.io.IOException
Checks and returns the status of the existing index in this store.- Parameters:
out
- where infoStream messages should go. SeeCheckIndex.setInfoStream(PrintStream)
- Throws:
java.io.IOException
-
stats
public StoreStats stats() throws java.io.IOException
- Throws:
java.io.IOException
-
incRef
public final void incRef()
Increments the refCount of this Store instance. RefCounts are used to determine when a Store can be closed safely, i.e. as soon as there are no more references. Be sure to always call a correspondingdecRef()
, in a finally clause; otherwise the store may never be closed. Note thatclose()
simply calls decRef(), which means that the Store will not really be closed untildecRef()
has been called for all outstanding references.Note: Close can safely be called multiple times.
- Specified by:
incRef
in interfaceRefCounted
- Throws:
org.apache.lucene.store.AlreadyClosedException
- iff the reference counter can not be incremented.- See Also:
decRef()
,tryIncRef()
-
tryIncRef
public final boolean tryIncRef()
Tries to increment the refCount of this Store instance. This method will returntrue
iff the refCount was incremented successfully otherwisefalse
. RefCounts are used to determine when a Store can be closed safely, i.e. as soon as there are no more references. Be sure to always call a correspondingdecRef()
, in a finally clause; otherwise the store may never be closed. Note thatclose()
simply calls decRef(), which means that the Store will not really be closed untildecRef()
has been called for all outstanding references.Note: Close can safely be called multiple times.
- Specified by:
tryIncRef
in interfaceRefCounted
- See Also:
decRef()
,incRef()
-
decRef
public final void decRef()
Decreases the refCount of this Store instance. If the refCount drops to 0, then this store is closed.- Specified by:
decRef
in interfaceRefCounted
- See Also:
incRef()
-
close
public void close()
- Specified by:
close
in interfacejava.lang.AutoCloseable
- Specified by:
close
in interfacejava.io.Closeable
-
readMetadataSnapshot
public static Store.MetadataSnapshot readMetadataSnapshot(java.nio.file.Path indexLocation, ShardId shardId, NodeEnvironment.ShardLocker shardLocker, org.apache.logging.log4j.Logger logger) throws java.io.IOException
Reads a MetadataSnapshot from the given index locations or returns an empty snapshot if it can't be read.- Throws:
java.io.IOException
- if the index we try to read is corrupted
-
tryOpenIndex
public static void tryOpenIndex(java.nio.file.Path indexLocation, ShardId shardId, NodeEnvironment.ShardLocker shardLocker, org.apache.logging.log4j.Logger logger) throws java.io.IOException, ShardLockObtainFailedException
Tries to open an index for the given location. This includes reading the segment infos and possible corruption markers. If the index can not be opened, an exception is thrown- Throws:
java.io.IOException
ShardLockObtainFailedException
-
createVerifyingOutput
public org.apache.lucene.store.IndexOutput createVerifyingOutput(java.lang.String fileName, StoreFileMetaData metadata, org.apache.lucene.store.IOContext context) throws java.io.IOException
The returned IndexOutput validates the files checksum.Note: Checksums are calculated by default since version 4.8.0. This method only adds the verification against the checksum in the given metadata and does not add any significant overhead.
- Throws:
java.io.IOException
-
verify
public static void verify(org.apache.lucene.store.IndexOutput output) throws java.io.IOException
- Throws:
java.io.IOException
-
openVerifyingInput
public org.apache.lucene.store.IndexInput openVerifyingInput(java.lang.String filename, org.apache.lucene.store.IOContext context, StoreFileMetaData metadata) throws java.io.IOException
- Throws:
java.io.IOException
-
verify
public static void verify(org.apache.lucene.store.IndexInput input) throws java.io.IOException
- Throws:
java.io.IOException
-
checkIntegrityNoException
public boolean checkIntegrityNoException(StoreFileMetaData md)
-
checkIntegrityNoException
public static boolean checkIntegrityNoException(StoreFileMetaData md, org.apache.lucene.store.Directory directory)
-
checkIntegrity
public static void checkIntegrity(StoreFileMetaData md, org.apache.lucene.store.Directory directory) throws java.io.IOException
- Throws:
java.io.IOException
-
isMarkedCorrupted
public boolean isMarkedCorrupted() throws java.io.IOException
- Throws:
java.io.IOException
-
removeCorruptionMarker
public void removeCorruptionMarker() throws java.io.IOException
Deletes all corruption markers from this store.- Throws:
java.io.IOException
-
failIfCorrupted
public void failIfCorrupted() throws java.io.IOException
- Throws:
java.io.IOException
-
cleanupAndVerify
public void cleanupAndVerify(java.lang.String reason, Store.MetadataSnapshot sourceMetaData) throws java.io.IOException
This method deletes every file in this store that is not contained in the given source meta data or is a legacy checksum file. After the delete it pulls the latest metadata snapshot from the store and compares it to the given snapshot. If the snapshots are inconsistent an illegal state exception is thrown.- Parameters:
reason
- the reason for this cleanup operation logged for each deleted filesourceMetaData
- the metadata used for cleanup. all files in this metadata should be kept around.- Throws:
java.io.IOException
- if an IOException occursjava.lang.IllegalStateException
- if the latest snapshot in this store differs from the given one after the cleanup.
-
refCount
public int refCount()
Returns the current reference count.
-
isAutogenerated
public static boolean isAutogenerated(java.lang.String name)
Returns true if the file is auto-generated by the store and shouldn't be deleted during cleanup. This includes write lock and checksum files
-
digestToString
public static java.lang.String digestToString(long digest)
Produces a string representation of the given digest value.
-
deleteQuiet
public void deleteQuiet(java.lang.String... files)
-
markStoreCorrupted
public void markStoreCorrupted(java.io.IOException exception) throws java.io.IOException
Marks this store as corrupted. This method writes acorrupted_${uuid}
file containing the given exception message. If a store contains acorrupted_${uuid}
fileisMarkedCorrupted()
will returntrue
.- Throws:
java.io.IOException
-
createEmpty
public void createEmpty() throws java.io.IOException
creates an empty lucene index and a corresponding empty translog. Any existing data will be deleted.- Throws:
java.io.IOException
-
bootstrapNewHistory
public void bootstrapNewHistory() throws java.io.IOException
Marks an existing lucene index with a new history uuid. This is used to make sure no existing shard will recovery from this index using ops based recovery.- Throws:
java.io.IOException
-
bootstrapNewHistory
public void bootstrapNewHistory(long localCheckpoint, long maxSeqNo) throws java.io.IOException
Marks an existing lucene index with a new history uuid and sets the given local checkpoint as well as the maximum sequence number. This is used to make sure no existing shard will recover from this index using ops based recovery.- Throws:
java.io.IOException
- See Also:
SequenceNumbers.LOCAL_CHECKPOINT_KEY
,SequenceNumbers.MAX_SEQ_NO
-
associateIndexWithNewTranslog
public void associateIndexWithNewTranslog(java.lang.String translogUUID) throws java.io.IOException
Force bakes the given translog generation as recovery information in the lucene index. This is used when recovering from a snapshot or peer file based recovery where a new empty translog is created and the existing lucene index needs should be changed to use it.- Throws:
java.io.IOException
-
ensureIndexHas6xCommitTags
public boolean ensureIndexHas6xCommitTags() throws java.io.IOException
A 5.x index does not have either historyUUDID or sequence number markers as these markers are introduced in 6.0+. This method should be called only in local store recovery or file-based recovery to ensure an index has proper historyUUID and sequence number markers before opening an engine.- Returns:
true
if a new commit is flushed, otherwise return false- Throws:
java.io.IOException
-
trimUnsafeCommits
public void trimUnsafeCommits(long lastSyncedGlobalCheckpoint, long minRetainedTranslogGen, Version indexVersionCreated) throws java.io.IOException
Keeping existing unsafe commits when opening an engine can be problematic because these commits are not safe at the recovering time but they can suddenly become safe in the future. The following issues can happen if unsafe commits are kept oninit.1. Replica can use unsafe commit in peer-recovery. This happens when a replica with a safe commit c1(max_seqno=1) and an unsafe commit c2(max_seqno=2) recovers from a primary with c1(max_seqno=1). If a new document(seqno=2) is added without flushing, the global checkpoint is advanced to 2; and the replica recovers again, it will use the unsafe commit c2(max_seqno=2 at most gcp=2) as the starting commit for sequenced-based recovery even the commit c2 contains a stale operation and the document(with seqno=2) will not be replicated to the replica.
2. Min translog gen for recovery can go backwards in peer-recovery. This happens when are replica with a safe commit c1(local_checkpoint=1, recovery_translog_gen=1) and an unsafe commit c2(local_checkpoint=2, recovery_translog_gen=2). The replica recovers from a primary, and keeps c2 as the last commit, then sets last_translog_gen to 2. Flushing a new commit on the replica will cause exception as the new last commit c3 will have recovery_translog_gen=1. The recovery translog generation of a commit is calculated based on the current local checkpoint. The local checkpoint of c3 is 1 while the local checkpoint of c2 is 2.
3. Commit without translog can be used in recovery. An old index, which was created before multiple-commits is introduced (v6.2), may not have a safe commit. If that index has a snapshotted commit without translog and an unsafe commit, the policy can consider the snapshotted commit as a safe commit for recovery even the commit does not have translog.
- Throws:
java.io.IOException
-
-