public class ProgressToken extends Object
Modifier and Type | Class and Description |
---|---|
static class |
ProgressToken.BucketEntry
When a bucket has its state kept by the progress token, we need to
discern between active buckets (i.e.
|
static class |
ProgressToken.BucketKeyWrapper
For consistent bucket key ordering, we need to ensure that reverse bucket
IDs that have their MSB set actually are compared as being greater than
those that don't.
|
static class |
ProgressToken.BucketState
Any bucket kept track of by a
ProgressToken instance may
be in one of two states: pending or active. |
Modifier and Type | Field and Description |
---|---|
private long |
activeBucketCount |
private long |
bucketCursor
The bucket cursor (i.e.
|
private TreeMap<ProgressToken.BucketKeyWrapper,ProgressToken.BucketEntry> |
buckets |
private int |
distributionBits
By default, a ProgressToken's distribution bit count is set to the VDS
standard value of 16, but it may be changed via the iterator using it
or by a bucket source when importing an existing progress
|
private TreeMap<com.yahoo.document.BucketId,com.yahoo.document.BucketId> |
failedBuckets |
static com.yahoo.document.BucketId |
FINISHED_BUCKET |
private long |
finishedBucketCount |
private String |
firstErrorMsg |
private boolean |
inconsistentState
Set by the VisitorIterator during a distribution bit change when
the token contains active/pending buckets with different used-bits
|
private static Logger |
log |
static com.yahoo.document.BucketId |
NULL_BUCKET |
private long |
pendingBucketCount |
private long |
totalBucketCount |
Constructor and Description |
---|
ProgressToken()
Creates a progress token.
|
ProgressToken(byte[] serialized) |
ProgressToken(int distributionBits) |
ProgressToken(String serialized) |
Modifier and Type | Method and Description |
---|---|
protected void |
addBucket(com.yahoo.document.BucketId superbucket,
com.yahoo.document.BucketId progress,
ProgressToken.BucketState state)
For use internally by DocumentAPI code only.
|
void |
addFailedBucket(com.yahoo.document.BucketId superbucket,
com.yahoo.document.BucketId progress,
String errorMsg) |
static long |
bucketToKey(long id) |
private static ProgressToken.BucketKeyWrapper |
bucketToKeyWrapper(com.yahoo.document.BucketId bucket) |
protected void |
clearAllBuckets() |
boolean |
containsFailedBuckets() |
long |
getActiveBucketCount() |
long |
getBucketCursor() |
protected TreeMap<ProgressToken.BucketKeyWrapper,ProgressToken.BucketEntry> |
getBuckets()
Get internal progress token bucket state map.
|
int |
getDistributionBitCount() |
Map<com.yahoo.document.BucketId,com.yahoo.document.BucketId> |
getFailedBuckets()
Get all failed buckets and their progress.
|
long |
getFinishedBucketCount() |
String |
getFirstErrorMsg() |
long |
getPendingBucketCount() |
long |
getTotalBucketCount() |
boolean |
hasActive() |
boolean |
hasPending() |
protected boolean |
isBucketFinished(com.yahoo.document.BucketId bucket)
Checks whether or not a given bucket is certain to be finished.
|
boolean |
isEmpty() |
boolean |
isFinished() |
boolean |
isInconsistentState() |
static long |
keyToBucketId(long key) |
static long |
makeNthBucketKey(long n,
int distributionBits)
Directly generate a bucket Id key for the
n th bucket in
reverse sorted order. |
protected void |
mergePendingBucket(com.yahoo.document.BucketId bucket) |
double |
percentFinished()
Calculate an estimate on how far we've managed to iterate over both the
superbuckets and the sub-buckets.
|
double |
progressFraction(com.yahoo.document.BucketId superbucket,
com.yahoo.document.BucketId progress) |
byte[] |
serialize() |
protected void |
setActiveBucketCount(long activeBucketCount) |
protected void |
setAllBucketsToState(ProgressToken.BucketState state) |
protected void |
setBucketCursor(long bucketCursor) |
protected void |
setDistributionBitCount(int distributionBits)
Set the internal number of distribution bits, which wil be used for writing
the progress file and calculating correct percent-wise sub-bucket completion.
|
protected void |
setFinishedBucketCount(long finishedBucketCount)
For use by bucket sources and unit tests only!
|
void |
setInconsistentState(boolean inconsistentState) |
protected void |
setPendingBucketCount(long pendingBucketCount) |
protected void |
setTotalBucketCount(long totalBucketCount)
For use by bucket sources and unit tests only!
|
protected void |
splitPendingBucket(com.yahoo.document.BucketId bucket) |
String |
toString()
The format of the bucket progress output is as follows:
|
protected void |
updateProgress(com.yahoo.document.BucketId superbucket,
com.yahoo.document.BucketId progress)
Updates internal progress state for
bucket , indicating it's currently
at progress . |
private static final Logger log
public static final com.yahoo.document.BucketId NULL_BUCKET
public static final com.yahoo.document.BucketId FINISHED_BUCKET
private int distributionBits
private TreeMap<ProgressToken.BucketKeyWrapper,ProgressToken.BucketEntry> buckets
private long activeBucketCount
private long pendingBucketCount
private long finishedBucketCount
private long totalBucketCount
private TreeMap<com.yahoo.document.BucketId,com.yahoo.document.BucketId> failedBuckets
private String firstErrorMsg
private long bucketCursor
private boolean inconsistentState
public ProgressToken()
public ProgressToken(int distributionBits)
public ProgressToken(String serialized)
public ProgressToken(byte[] serialized)
public byte[] serialize()
public void addFailedBucket(com.yahoo.document.BucketId superbucket, com.yahoo.document.BucketId progress, String errorMsg)
public Map<com.yahoo.document.BucketId,com.yahoo.document.BucketId> getFailedBuckets()
protected void updateProgress(com.yahoo.document.BucketId superbucket, com.yahoo.document.BucketId progress)
bucket
, indicating it's currently
at progress
. Assumes that given a completely finished bucket, this
function will not be called again to further update its progress after
the finished-update.superbucket
- A valid superbucket ID that exists in buckets
progress
- The sub-bucket progress that has been reached in the
superbucketVisitorIterator.update(com.yahoo.document.BucketId, com.yahoo.document.BucketId)
protected void addBucket(com.yahoo.document.BucketId superbucket, com.yahoo.document.BucketId progress, ProgressToken.BucketState state)
superbucket
- Superbucket that will be progress-trackedprogress
- Bucket progress thus farstate
- Initial bucket state. Only pending buckets may be returnedpublic static long makeNthBucketKey(long n, int distributionBits)
n
th bucket in
reverse sorted order.n
- a number in the range [0, 2**distributionBits
)distributionBits
- Distribution bit count for the generated keydistributionBits
BucketId
s with incremental numerical IDs and then sorted
them on their reverse bit-order keys, the returned key would be equal
to the nth element in the resulting sorted sequence.public int getDistributionBitCount()
protected void setDistributionBitCount(int distributionBits)
distributionBits
- new distribution bit valuepublic long getActiveBucketCount()
public long getBucketCursor()
protected void setBucketCursor(long bucketCursor)
public long getFinishedBucketCount()
protected void setFinishedBucketCount(long finishedBucketCount)
finishedBucketCount
- Number of buckets the token has finishedpublic long getTotalBucketCount()
protected void setTotalBucketCount(long totalBucketCount)
totalBucketCount
- Total number of buckets that the progress token spanspublic long getPendingBucketCount()
public boolean hasPending()
public boolean hasActive()
public boolean isFinished()
public boolean isEmpty()
public String getFirstErrorMsg()
public boolean containsFailedBuckets()
public boolean isInconsistentState()
public void setInconsistentState(boolean inconsistentState)
protected TreeMap<ProgressToken.BucketKeyWrapper,ProgressToken.BucketEntry> getBuckets()
protected void setActiveBucketCount(long activeBucketCount)
protected void setPendingBucketCount(long pendingBucketCount)
public String toString()
VDS bucket progress file (n% completed)\n distribution bit count\n current bucket cursor\n number of finished buckets\n total number of buckets\n hex-of-superbucket:hex-of-progress\n ... repeat above line for each pending bucket ...Note that unlike earlier versions of ProgressToken, the bucket IDs are not prefixed with '0x'.
public double percentFinished()
public static long bucketToKey(long id)
private static ProgressToken.BucketKeyWrapper bucketToKeyWrapper(com.yahoo.document.BucketId bucket)
public static long keyToBucketId(long key)
public double progressFraction(com.yahoo.document.BucketId superbucket, com.yahoo.document.BucketId progress)
superbucket
- The superbucket of which progress
is
a sub-bucketprogress
- The sub-bucket for which a fractional progress should
be calculatedprotected boolean isBucketFinished(com.yahoo.document.BucketId bucket)
bucket
- Bucket to check whether or not is finished.true
if bucket
's super-bucket is
finished, false
otherwise.protected void splitPendingBucket(com.yahoo.document.BucketId bucket)
bucket
- BucketId to be split into two buckets. Bucket's used-bits
do not need to match the ProgressToken's current distribution bit count,
as it is assumed the client knows what it's doing and will bring the
token into a consistent state eventually.protected void mergePendingBucket(com.yahoo.document.BucketId bucket)
protected void setAllBucketsToState(ProgressToken.BucketState state)
protected void clearAllBuckets()
Copyright © 2018. All rights reserved.