Class OnlineIndexer
- java.lang.Object
-
- com.apple.foundationdb.record.provider.foundationdb.OnlineIndexer
-
- All Implemented Interfaces:
AutoCloseable
@API(UNSTABLE) public class OnlineIndexer extends Object implements AutoCloseable
Builds an index online, i.e., concurrently with other database operations. In order to minimize the impact that these operations have with other operations, this attempts to minimize the priorities of its transactions. Additionally, it attempts to limit the amount of work it will done in a fashion that will decrease as the number of failures for a given build attempt increases.As ranges of elements are rebuilt, the fact that the range has rebuilt is added to a
RangeSet
associated with the index being built. ThisRangeSet
is used to (a) coordinate work between different builders that might be running on different machines to ensure that the same work isn't duplicated and to (b) make sure that non-idempotent indexes (likeCOUNT
orSUM_LONG
) don't update themselves (or fail to update themselves) incorrectly.Unlike many other features in the Record Layer core, this has a retry loop.
Build an index immediately in the current transaction:
try (OnlineIndexer indexBuilder = OnlineIndexer.forRecordStoreAndIndex(recordStore, "newIndex")) { indexBuilder.rebuildIndex(recordStore); }
Build an index synchronously in the multiple transactions:
try (OnlineIndexer indexBuilder = OnlineIndexer.forRecordStoreAndIndex(recordStore, "newIndex")) { indexBuilder.buildIndex(); }
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description static class
OnlineIndexer.Builder
Builder forOnlineIndexer
.static class
OnlineIndexer.Config
A holder for the mutable configuration parameters needed to rebuild an online index.static class
OnlineIndexer.IndexStatePrecondition
This defines in which situations the index should be built.static class
OnlineIndexer.RecordBuiltRangeException
ThisException
can be thrown in the case that one calls one of the methods that explicitly state that they are building an unbuilt range, i.e., a range of keys that contains no keys which have yet been processed by theOnlineIndexer
during an index build.
-
Field Summary
Fields Modifier and Type Field Description static long
DEFAULT_LEASE_LENGTH_MILLIS
Default length between last access and lease's end time in milliseconds.static int
DEFAULT_LIMIT
Default number of records to attempt to run in a single transaction.static int
DEFAULT_MAX_RETRIES
Default number of times to retry a single range rebuild.static int
DEFAULT_PROGRESS_LOG_INTERVAL
Default interval to be logging successful progress in millis when building across transactions.static int
DEFAULT_RECORDS_PER_SECOND
Default limit to the number of records to attempt in a single second.static int
DEFAULT_WRITE_LIMIT_BYTES
Default transaction write size limit.static int
DO_NOT_RE_INCREASE_LIMIT
IfOnlineIndexer.Builder.getIncreaseLimitAfter()
is this value, the limit will not go back up, no matter how many successes there are.static int
UNLIMITED
Constant indicating that there should be no limit to some usually limited operation.
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Deprecated Methods Modifier and Type Method Description <T> T
asyncToSync(StoreTimer.Wait event, CompletableFuture<T> async)
Wait for an asynchronous task to complete.<T> T
asyncToSync(CompletableFuture<T> buildIndexFuture)
Deprecated.in favor ofasyncToSync(StoreTimer.Wait, CompletableFuture)
CompletableFuture<TupleRange>
buildEndpoints()
Builds (with a retry loop) the endpoints of an index.CompletableFuture<TupleRange>
buildEndpoints(FDBRecordStore store)
Builds (transactionally) the endpoints of an index.void
buildIndex()
Builds an index across multiple transactions.void
buildIndex(boolean markReadable)
Builds an index across multiple transactions.CompletableFuture<Void>
buildIndexAsync()
Builds an index across multiple transactions.CompletableFuture<Void>
buildRange(Key.Evaluated start, Key.Evaluated end)
Builds (with a retry loop) the index by adding records with primary keys within the given range.CompletableFuture<Void>
buildRange(FDBRecordStore store, Key.Evaluated start, Key.Evaluated end)
Builds (transactionally) the index by adding records with primary keys within the given range.CompletableFuture<Key.Evaluated>
buildUnbuiltRange(FDBRecordStore store, Key.Evaluated start, Key.Evaluated end)
Builds (transactionally) the index by adding records with primary keys within the given range.void
close()
static OnlineIndexer
forRecordStoreAndIndex(FDBRecordStore recordStore, String index)
Create an online indexer for the given record store and index.int
getLimit()
Get the current number of records to process in one transaction.protected static Subspace
indexBuildScannedRecordsSubspace(FDBRecordStoreBase<?> store, Index index)
CompletableFuture<Boolean>
markReadable()
Mark the index as readable.CompletableFuture<Boolean>
markReadableIfBuilt()
Mark the index as readable if it is built.static OnlineIndexer.Builder
newBuilder()
Create an online indexer builder.void
rebuildIndex(FDBRecordStore store)
Transactionally rebuild an entire index.CompletableFuture<Void>
rebuildIndexAsync(FDBRecordStore store)
Transactionally rebuild an entire index.List<org.apache.commons.lang3.tuple.Pair<Tuple,Tuple>>
splitIndexBuildRange(int minSplit, int maxSplit)
Split the index build range to support building an index across multiple transactions in parallel if needed.void
stopOngoingOnlineIndexBuilds()
Synchronous/blocking version ofstopOngoingOnlineIndexBuildsAsync()
.static void
stopOngoingOnlineIndexBuilds(FDBRecordStore recordStore, Index index)
Stop any ongoing online index build (only if it usesSynchronizedSession
s) by forcefully releasing the lock.CompletableFuture<Void>
stopOngoingOnlineIndexBuildsAsync()
Stop any ongoing online index build (only if it usesSynchronizedSession
s) by forcefully releasing the lock.
-
-
-
Field Detail
-
DEFAULT_LIMIT
public static final int DEFAULT_LIMIT
Default number of records to attempt to run in a single transaction.- See Also:
- Constant Field Values
-
DEFAULT_WRITE_LIMIT_BYTES
public static final int DEFAULT_WRITE_LIMIT_BYTES
Default transaction write size limit. Note that the actual write might be "a little" bigger.- See Also:
- Constant Field Values
-
DEFAULT_RECORDS_PER_SECOND
public static final int DEFAULT_RECORDS_PER_SECOND
Default limit to the number of records to attempt in a single second.- See Also:
- Constant Field Values
-
DEFAULT_MAX_RETRIES
public static final int DEFAULT_MAX_RETRIES
Default number of times to retry a single range rebuild.- See Also:
- Constant Field Values
-
DEFAULT_PROGRESS_LOG_INTERVAL
public static final int DEFAULT_PROGRESS_LOG_INTERVAL
Default interval to be logging successful progress in millis when building across transactions.-1
means it will not log.- See Also:
- Constant Field Values
-
DEFAULT_LEASE_LENGTH_MILLIS
public static final long DEFAULT_LEASE_LENGTH_MILLIS
Default length between last access and lease's end time in milliseconds.- See Also:
- Constant Field Values
-
UNLIMITED
public static final int UNLIMITED
Constant indicating that there should be no limit to some usually limited operation.- See Also:
- Constant Field Values
-
DO_NOT_RE_INCREASE_LIMIT
public static final int DO_NOT_RE_INCREASE_LIMIT
IfOnlineIndexer.Builder.getIncreaseLimitAfter()
is this value, the limit will not go back up, no matter how many successes there are. This is the default value.- See Also:
- Constant Field Values
-
-
Method Detail
-
getLimit
public int getLimit()
Get the current number of records to process in one transaction. This may go up or down whilerunAsync(Function, BiFunction, BiConsumer, List)
is running, if there are failures committing or repeated successes.- Returns:
- the current number of records to process in one transaction
-
close
public void close()
- Specified by:
close
in interfaceAutoCloseable
-
buildRange
@Nonnull public CompletableFuture<Void> buildRange(@Nonnull FDBRecordStore store, @Nullable Key.Evaluated start, @Nullable Key.Evaluated end)
Builds (transactionally) the index by adding records with primary keys within the given range. This will look for gaps of keys within the given range that haven't yet been rebuilt and then rebuild only those ranges. As a result, if this method is called twice, the first time, it will build whatever needs to be built, and then the second time, it will notice that there are no ranges that need to be built, so it will do nothing. In this way, it is idempotent and thus safe to use in retry loops. This method will fail if there is too much work to be done in a single transaction. If one wants to handle building a range that does not fit in a single transaction, one should use thebuildRange()
function that takes anFDBDatabase
as its first parameter.- Parameters:
store
- the record store in which to rebuild the rangestart
- the (inclusive) beginning primary key of the range to build (ornull
to go to the end)end
- the (exclusive) end primary key of the range to build (ornull
to go to the end)- Returns:
- a future that will be ready when the build has completed
-
buildRange
@Nonnull public CompletableFuture<Void> buildRange(@Nullable Key.Evaluated start, @Nullable Key.Evaluated end)
Builds (with a retry loop) the index by adding records with primary keys within the given range. This will look for gaps of keys within the given range that haven't yet been rebuilt and then rebuild only those ranges. It will also limit each transaction to the number of records specified by thelimit
parameter of this class's constructor. In the case that that limit is too high (i.e., it can't make any progress or errors out on a non-retriable error liketransaction_too_large
, this method will actually decrease the limit so that less work is attempted each transaction. It will also rate limit itself as to not make too many requests per second.Note that it does not have the protections (synchronized sessions and index state precondition) which are imposed on
buildIndexAsync()
(or its variations), but it does use the created synchronized session if abuildIndexAsync()
is running on theOnlineIndexer
simultaneously or this range build is used as part ofbuildIndexAsync()
internally.- Parameters:
start
- the (inclusive) beginning primary key of the range to build (ornull
to go from the beginning)end
- the (exclusive) end primary key of the range to build (ornull
to go to the end)- Returns:
- a future that will be ready when the build has completed
-
buildUnbuiltRange
@Nonnull public CompletableFuture<Key.Evaluated> buildUnbuiltRange(@Nonnull FDBRecordStore store, @Nullable Key.Evaluated start, @Nullable Key.Evaluated end)
Builds (transactionally) the index by adding records with primary keys within the given range. This requires that the range is initially "unbuilt", i.e., no records within the given range have yet been processed by the index build job. It is acceptable if there are records within that range that have already been added to the index because they were added to the store after the index was added in write-only mode but have not yet been processed by the index build job. Note that this function is not idempotent in that if the first time this function runs, if it fails withcommit_unknown_result
but the transaction actually succeeds, running this function again will result in aOnlineIndexer.RecordBuiltRangeException
being thrown the second time. Retry loops used by theOnlineIndexer
class that call this method handle this contingency. For the most part, this method should only be used by those who know what they are doing. It is included because it is less expensive to make this call if one already knows that the range will be unbuilt, but the caller must be ready to handle the circumstance that the range might be built the second time. Most users should use thebuildRange()
method with the same parameters in the case that they want to build a range of keys into the index. That method is idempotent, but it is slightly more costly as it firsts determines what ranges are have not yet been built before building them.- Parameters:
store
- the record store in which to rebuild the rangestart
- the (inclusive) beginning primary key of the range to build (ornull
to start from the beginning)end
- the (exclusive) end primary key of the range to build (ornull
to go to the end)- Returns:
- a future with the key of the first record not processed by this range rebuild
- Throws:
OnlineIndexer.RecordBuiltRangeException
- if the given range contains keys already processed by the index build
-
rebuildIndexAsync
@Nonnull public CompletableFuture<Void> rebuildIndexAsync(@Nonnull FDBRecordStore store)
Transactionally rebuild an entire index. This will (1) delete any data in the index that is already there and (2) rebuild the entire key range for the given index. It will attempt to do this within a single transaction, and it may fail if there are too many records, so this is only safe to do for small record stores. Many large use-cases should use thebuildIndexAsync()
method along with temporarily changing an index to write-only mode while the index is being rebuilt.- Parameters:
store
- the record store in which to rebuild the index- Returns:
- a future that will be ready when the build has completed
-
rebuildIndex
public void rebuildIndex(@Nonnull FDBRecordStore store)
Transactionally rebuild an entire index. Synchronous version ofrebuildIndexAsync(com.apple.foundationdb.record.provider.foundationdb.FDBRecordStore)
- Parameters:
store
- the record store in which to rebuild the index- See Also:
buildIndex(boolean)
-
buildEndpoints
@Nonnull public CompletableFuture<TupleRange> buildEndpoints(@Nonnull FDBRecordStore store)
Builds (transactionally) the endpoints of an index. What this means is that builds everything from the beginning of the key space to the first record and everything from the last record to the end of the key space. There won't be any records within these ranges (except for the last record of the record store), but it does mean that any records in the future that get added to these ranges will correctly update the index. This means, e.g., that if the workload primarily adds records to the record store after the current last record (because perhaps the primary key is based off of an atomic counter or the current time), running this method will be highly contentious, but once it completes, the rest of the index build should happen without any more conflicts. This will return a (possibly null)TupleRange
that contains the primary keys of the first and last records within the record store. This can then be used to either build the range right away or to then divy-up the remaining ranges between multiple agents working in parallel if one desires.- Parameters:
store
- the record store in which to rebuild the index- Returns:
- a future that will contain the range of records in the interior of the record store
-
buildEndpoints
@Nonnull public CompletableFuture<TupleRange> buildEndpoints()
Builds (with a retry loop) the endpoints of an index. See thebuildEndpoints()
method that takes anFDBRecordStore
as its parameter for more details. This will retry on that function until it gets a non-exceptional result and return the results back.- Returns:
- a future that will contain the range of records in the interior of the record store
-
stopOngoingOnlineIndexBuildsAsync
public CompletableFuture<Void> stopOngoingOnlineIndexBuildsAsync()
Stop any ongoing online index build (only if it usesSynchronizedSession
s) by forcefully releasing the lock.- Returns:
- a future that will be ready when the lock is released
- See Also:
SynchronizedSession.endAnySession(Transaction, Subspace)
-
stopOngoingOnlineIndexBuilds
public void stopOngoingOnlineIndexBuilds()
Synchronous/blocking version ofstopOngoingOnlineIndexBuildsAsync()
.
-
stopOngoingOnlineIndexBuilds
public static void stopOngoingOnlineIndexBuilds(@Nonnull FDBRecordStore recordStore, @Nonnull Index index)
Stop any ongoing online index build (only if it usesSynchronizedSession
s) by forcefully releasing the lock.- Parameters:
recordStore
- record store whose index builds need to be stoppedindex
- the index whose builds need to be stopped
-
buildIndexAsync
@Nonnull public CompletableFuture<Void> buildIndexAsync()
Builds an index across multiple transactions.If it is set to use synchronized sessions, it stops with
SynchronizedSessionLockedException
when there is another runner actively working on the same index. It first checks and updates index states and clear index data respecting theOnlineIndexer.IndexStatePrecondition
being set. It then builds the index across multiple transactions honoring the rate-limiting parameters set in the constructor of this class. It also retries any retriable errors that it encounters while it runs the build. At the end, it marks the index readable in the store.One may consider to set the index state precondition to
OnlineIndexer.IndexStatePrecondition.ERROR_IF_DISABLED_CONTINUE_IF_WRITE_ONLY
andOnlineIndexer.Builder.setUseSynchronizedSession(boolean)
tofalse
, which makes the indexer follow the same behavior as before version 2.8.90.0. But it is not recommended.- Returns:
- a future that will be ready when the build has completed
- Throws:
com.apple.foundationdb.synchronizedsession.SynchronizedSessionLockedException
- the build is stopped because there may be another build running actively on this index.
-
indexBuildScannedRecordsSubspace
@Nonnull protected static Subspace indexBuildScannedRecordsSubspace(@Nonnull FDBRecordStoreBase<?> store, @Nonnull Index index)
-
buildIndex
@Nonnull public void buildIndex(boolean markReadable)
Builds an index across multiple transactions. Synchronous version ofbuildIndexAsync()
.- Parameters:
markReadable
- whether to mark the index as readable after building the index
-
buildIndex
@Nonnull public void buildIndex()
Builds an index across multiple transactions. Synchronous version ofbuildIndexAsync()
.
-
splitIndexBuildRange
@API(EXPERIMENTAL) @Nonnull public List<org.apache.commons.lang3.tuple.Pair<Tuple,Tuple>> splitIndexBuildRange(int minSplit, int maxSplit)
Split the index build range to support building an index across multiple transactions in parallel if needed.It is blocking and should not be called in asynchronous contexts.
- Parameters:
minSplit
- not split if it cannot be split into at leastminSplit
rangesmaxSplit
- the maximum number of splits generated- Returns:
- a list of split primary key ranges (the low endpoint is inclusive and the high endpoint is exclusive)
-
markReadableIfBuilt
@API(EXPERIMENTAL) @Nonnull public CompletableFuture<Boolean> markReadableIfBuilt()
Mark the index as readable if it is built.- Returns:
- a future that will complete to
true
if the index is readable andfalse
otherwise
-
markReadable
@API(EXPERIMENTAL) @Nonnull public CompletableFuture<Boolean> markReadable()
Mark the index as readable.- Returns:
- a future that will either complete exceptionally if the index can not
be made readable or will contain
true
if the store was modified andfalse
otherwise
-
asyncToSync
@API(DEPRECATED) @Deprecated public <T> T asyncToSync(@Nonnull CompletableFuture<T> buildIndexFuture)
Deprecated.in favor ofasyncToSync(StoreTimer.Wait, CompletableFuture)
Wait for an index build to complete. This method has been deprecated in favor ofasyncToSync(StoreTimer.Wait, CompletableFuture)
which gives the user more control over whichStoreTimer.Wait
to instrument.- Type Parameters:
T
- the return type of the asynchronous task- Parameters:
buildIndexFuture
- a task to build an index- Returns:
- the result of
buildIndexFuture
when it completes
-
asyncToSync
@API(INTERNAL) public <T> T asyncToSync(@Nonnull StoreTimer.Wait event, @Nonnull CompletableFuture<T> async)
Wait for an asynchronous task to complete. This returns the result from the future or propagates the error if the future completes exceptionally.- Type Parameters:
T
- the task's return type- Parameters:
event
- the event being waited on (for instrumentation purposes)async
- the asynchronous task to wait on- Returns:
- the result of the asynchronous task
-
newBuilder
@Nonnull public static OnlineIndexer.Builder newBuilder()
Create an online indexer builder.- Returns:
- a new online indexer builder
-
forRecordStoreAndIndex
@Nonnull public static OnlineIndexer forRecordStoreAndIndex(@Nonnull FDBRecordStore recordStore, @Nonnull String index)
Create an online indexer for the given record store and index.- Parameters:
recordStore
- record store in which to indexindex
- name of index to build- Returns:
- a new online indexer
-
-