@API(value=UNSTABLE) public class OnlineIndexer extends Object implements AutoCloseable
As ranges of elements are rebuilt, the fact that the range has rebuilt is added to a RangeSet
associated with the index being built. This RangeSet
is used to (a) coordinate work between
different builders that might be running on different machines to ensure that the same work isn't
duplicated and to (b) make sure that non-idempotent indexes (like COUNT
or SUM_LONG
)
don't update themselves (or fail to update themselves) incorrectly.
Unlike many other features in the Record Layer core, this has a retry loop.
Build an index immediately in the current transaction:
try (OnlineIndexer indexBuilder = OnlineIndexer.forRecordStoreAndIndex(recordStore, "newIndex")) {
indexBuilder.rebuildIndex(recordStore);
}
Build an index synchronously in the multiple transactions:
try (OnlineIndexer indexBuilder = OnlineIndexer.forRecordStoreAndIndex(recordStore, "newIndex")) {
indexBuilder.buildIndex();
}
Modifier and Type | Class and Description |
---|---|
static class |
OnlineIndexer.Builder
Builder for
OnlineIndexer . |
static class |
OnlineIndexer.Config
A holder for the mutable configuration parameters needed to rebuild an online index.
|
static class |
OnlineIndexer.IndexStatePrecondition
This defines in which situations the index should be built.
|
static class |
OnlineIndexer.RecordBuiltRangeException
This
Exception can be thrown in the case that one calls one of the methods
that explicitly state that they are building an unbuilt range, i.e., a range of keys
that contains no keys which have yet been processed by the OnlineIndexer
during an index build. |
Modifier and Type | Field and Description |
---|---|
static long |
DEFAULT_LEASE_LENGTH_MILLIS
Default length between last access and lease's end time in milliseconds.
|
static int |
DEFAULT_LIMIT
Default number of records to attempt to run in a single transaction.
|
static int |
DEFAULT_MAX_RETRIES
Default number of times to retry a single range rebuild.
|
static int |
DEFAULT_PROGRESS_LOG_INTERVAL
Default interval to be logging successful progress in millis when building across transactions.
|
static int |
DEFAULT_RECORDS_PER_SECOND
Default limit to the number of records to attempt in a single second.
|
static int |
DO_NOT_RE_INCREASE_LIMIT
If
OnlineIndexer.Builder.getIncreaseLimitAfter() is this value, the limit will not go back up, no matter how many
successes there are. |
static int |
UNLIMITED
Constant indicating that there should be no limit to some usually limited operation.
|
Modifier and Type | Method and Description |
---|---|
<T> T |
asyncToSync(CompletableFuture<T> buildIndexFuture)
Deprecated.
in favor of
asyncToSync(StoreTimer.Wait, CompletableFuture) |
<T> T |
asyncToSync(StoreTimer.Wait event,
CompletableFuture<T> async)
Wait for an asynchronous task to complete.
|
CompletableFuture<TupleRange> |
buildEndpoints()
Builds (with a retry loop) the endpoints of an index.
|
CompletableFuture<TupleRange> |
buildEndpoints(FDBRecordStore store)
Builds (transactionally) the endpoints of an index.
|
void |
buildIndex()
Builds an index across multiple transactions.
|
void |
buildIndex(boolean markReadable)
Builds an index across multiple transactions.
|
CompletableFuture<Void> |
buildIndexAsync()
Builds an index across multiple transactions.
|
CompletableFuture<Void> |
buildRange(FDBRecordStore store,
Key.Evaluated start,
Key.Evaluated end)
Builds (transactionally) the index by adding records with primary keys within the given range.
|
CompletableFuture<Void> |
buildRange(Key.Evaluated start,
Key.Evaluated end)
Builds (with a retry loop) the index by adding records with primary keys within the given range.
|
CompletableFuture<Key.Evaluated> |
buildUnbuiltRange(FDBRecordStore store,
Key.Evaluated start,
Key.Evaluated end)
Builds (transactionally) the index by adding records with primary keys within the given range.
|
void |
close() |
static OnlineIndexer |
forRecordStoreAndIndex(FDBRecordStore recordStore,
String index)
Create an online indexer for the given record store and index.
|
int |
getLimit()
Get the current number of records to process in one transaction.
|
CompletableFuture<Boolean> |
markReadable()
Mark the index as readable.
|
CompletableFuture<Boolean> |
markReadableIfBuilt()
Mark the index as readable if it is built.
|
static OnlineIndexer.Builder |
newBuilder()
Create an online indexer builder.
|
void |
rebuildIndex(FDBRecordStore store)
Transactionally rebuild an entire index.
|
CompletableFuture<Void> |
rebuildIndexAsync(FDBRecordStore store)
Transactionally rebuild an entire index.
|
List<org.apache.commons.lang3.tuple.Pair<Tuple,Tuple>> |
splitIndexBuildRange(int minSplit,
int maxSplit)
Split the index build range to support building an index across multiple transactions in parallel if needed.
|
public static final int DEFAULT_LIMIT
public static final int DEFAULT_RECORDS_PER_SECOND
public static final int DEFAULT_MAX_RETRIES
public static final int DEFAULT_PROGRESS_LOG_INTERVAL
-1
means it will not log.public static final long DEFAULT_LEASE_LENGTH_MILLIS
public static final int UNLIMITED
public static final int DO_NOT_RE_INCREASE_LIMIT
OnlineIndexer.Builder.getIncreaseLimitAfter()
is this value, the limit will not go back up, no matter how many
successes there are.
This is the default value.public int getLimit()
runAsync(Function, BiFunction, BiConsumer, List)
is running, if there are failures committing or
repeated successes.public void close()
close
in interface AutoCloseable
@Nonnull public CompletableFuture<Void> buildRange(@Nonnull FDBRecordStore store, @Nullable Key.Evaluated start, @Nullable Key.Evaluated end)
buildRange()
function that takes an FDBDatabase
as its first parameter.store
- the record store in which to rebuild the rangestart
- the (inclusive) beginning primary key of the range to build (or null
to go to the end)end
- the (exclusive) end primary key of the range to build (or null
to go to the end)@Nonnull public CompletableFuture<Void> buildRange(@Nullable Key.Evaluated start, @Nullable Key.Evaluated end)
limit
parameter of this class's constructor. In the case that that limit is too high (i.e.,
it can't make any progress or errors out on a non-retriable error like transaction_too_large
,
this method will actually decrease the limit so that less work is attempted each transaction. It will
also rate limit itself as to not make too many requests per second.
Note that it does not have the protections (synchronized sessions and index state precondition) which are imposed
on buildIndexAsync()
(or its variations), but it does use the created synchronized session if a
buildIndexAsync()
is running on the OnlineIndexer
simultaneously or this range build is used as
part of buildIndexAsync()
internally.
start
- the (inclusive) beginning primary key of the range to build (or null
to go from the beginning)end
- the (exclusive) end primary key of the range to build (or null
to go to the end)@Nonnull public CompletableFuture<Key.Evaluated> buildUnbuiltRange(@Nonnull FDBRecordStore store, @Nullable Key.Evaluated start, @Nullable Key.Evaluated end)
commit_unknown_result
but the transaction actually succeeds, running this
function again will result in a OnlineIndexer.RecordBuiltRangeException
being thrown the second
time. Retry loops used by the OnlineIndexer
class that call this method
handle this contingency. For the most part, this method should only be used by those who know
what they are doing. It is included because it is less expensive to make this call if one
already knows that the range will be unbuilt, but the caller must be ready to handle the
circumstance that the range might be built the second time.
Most users should use the
buildRange()
method with the same parameters in the case that they want to build a range of keys into the index. That
method is idempotent, but it is slightly more costly as it firsts determines what ranges are
have not yet been built before building them.store
- the record store in which to rebuild the rangestart
- the (inclusive) beginning primary key of the range to build (or null
to start from the beginning)end
- the (exclusive) end primary key of the range to build (or null
to go to the end)OnlineIndexer.RecordBuiltRangeException
- if the given range contains keys already processed by the index build@Nonnull public CompletableFuture<Void> rebuildIndexAsync(@Nonnull FDBRecordStore store)
buildIndexAsync()
method along with temporarily
changing an index to write-only mode while the index is being rebuilt.store
- the record store in which to rebuild the indexpublic void rebuildIndex(@Nonnull FDBRecordStore store)
rebuildIndexAsync(com.apple.foundationdb.record.provider.foundationdb.FDBRecordStore)
store
- the record store in which to rebuild the indexbuildIndex(boolean)
@Nonnull public CompletableFuture<TupleRange> buildEndpoints(@Nonnull FDBRecordStore store)
TupleRange
that contains the primary keys of the
first and last records within the record store. This can then be used to either build the
range right away or to then divy-up the remaining ranges between multiple agents working
in parallel if one desires.store
- the record store in which to rebuild the index@Nonnull public CompletableFuture<TupleRange> buildEndpoints()
buildEndpoints()
method that takes
an FDBRecordStore
as its parameter for more details. This will retry on that function
until it gets a non-exceptional result and return the results back.@Nonnull public CompletableFuture<Void> buildIndexAsync()
If it is set to use synchronized sessions, it stops with SynchronizedSessionLockedException
when there is another runner actively working on the same index. It first checks and updates index states and
clear index data respecting the OnlineIndexer.IndexStatePrecondition
being set. It then builds the index across
multiple transactions honoring the rate-limiting parameters set in the constructor of this class. It also retries
any retriable errors that it encounters while it runs the build. At the end, it marks the index readable in the
store.
One may consider to set the index state precondition to OnlineIndexer.IndexStatePrecondition.ERROR_IF_DISABLED_CONTINUE_IF_WRITE_ONLY
and OnlineIndexer.Builder.setUseSynchronizedSession(boolean)
to false
, which makes the indexer
follow the same behavior as before version 2.8.90.0. But it is not recommended.
com.apple.foundationdb.synchronizedsession.SynchronizedSessionLockedException
- the build is stopped
because there may be another build running actively on this index.@Nonnull public void buildIndex(boolean markReadable)
buildIndexAsync()
.markReadable
- whether to mark the index as readable after building the index@Nonnull public void buildIndex()
buildIndexAsync()
.@API(value=EXPERIMENTAL) @Nonnull public List<org.apache.commons.lang3.tuple.Pair<Tuple,Tuple>> splitIndexBuildRange(int minSplit, int maxSplit)
It is blocking and should not be called in asynchronous contexts.
minSplit
- not split if it cannot be split into at least minSplit
rangesmaxSplit
- the maximum number of splits generated@API(value=EXPERIMENTAL) @Nonnull public CompletableFuture<Boolean> markReadableIfBuilt()
true
if the index is readable and false
otherwise@API(value=EXPERIMENTAL) @Nonnull public CompletableFuture<Boolean> markReadable()
true
if the store was modified
and false
otherwise@API(value=DEPRECATED) @Deprecated public <T> T asyncToSync(@Nonnull CompletableFuture<T> buildIndexFuture)
asyncToSync(StoreTimer.Wait, CompletableFuture)
asyncToSync(StoreTimer.Wait, CompletableFuture)
which gives the user more
control over which StoreTimer.Wait
to instrument.T
- the return type of the asynchronous taskbuildIndexFuture
- a task to build an indexbuildIndexFuture
when it completes@API(value=INTERNAL) public <T> T asyncToSync(@Nonnull StoreTimer.Wait event, @Nonnull CompletableFuture<T> async)
T
- the task's return typeevent
- the event being waited on (for instrumentation purposes)async
- the asynchronous task to wait on@Nonnull public static OnlineIndexer.Builder newBuilder()
@Nonnull public static OnlineIndexer forRecordStoreAndIndex(@Nonnull FDBRecordStore recordStore, @Nonnull String index)
recordStore
- record store in which to indexindex
- name of index to build