Package com.yahoo.documentapi
Class ProgressToken
- java.lang.Object
-
- com.yahoo.documentapi.ProgressToken
-
public class ProgressToken extends Object
Token to use to keep track of progress for visiting. Can be used to resume visiting if visiting has been aborted for any reason.- Author:
- Thomas Gundersen, vekterli
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description static class
ProgressToken.BucketEntry
When a bucket has its state kept by the progress token, we need to discern between active buckets (i.e.static class
ProgressToken.BucketKeyWrapper
For consistent bucket key ordering, we need to ensure that reverse bucket IDs that have their MSB set actually are compared as being greater than those that don't.static class
ProgressToken.BucketState
Any bucket kept track of by aProgressToken
instance may be in one of two states: pending or active.
-
Field Summary
Fields Modifier and Type Field Description static com.yahoo.document.BucketId
FINISHED_BUCKET
static com.yahoo.document.BucketId
NULL_BUCKET
-
Constructor Summary
Constructors Constructor Description ProgressToken()
Creates a progress token.ProgressToken(byte[] serialized)
ProgressToken(int distributionBits)
ProgressToken(String serialized)
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description protected void
addBucket(com.yahoo.document.BucketId superbucket, com.yahoo.document.BucketId progress, ProgressToken.BucketState state)
For use internally by DocumentAPI code only.void
addFailedBucket(com.yahoo.document.BucketId superbucket, com.yahoo.document.BucketId progress, String errorMsg)
static long
bucketToKey(long id)
protected void
clearAllBuckets()
boolean
containsFailedBuckets()
static ProgressToken
fromSerializedString(String serializedString)
long
getActiveBucketCount()
long
getBucketCursor()
protected TreeMap<ProgressToken.BucketKeyWrapper,ProgressToken.BucketEntry>
getBuckets()
Get internal progress token bucket state map.int
getDistributionBitCount()
Map<com.yahoo.document.BucketId,com.yahoo.document.BucketId>
getFailedBuckets()
Get all failed buckets and their progress.long
getFinishedBucketCount()
String
getFirstErrorMsg()
long
getPendingBucketCount()
long
getTotalBucketCount()
boolean
hasActive()
boolean
hasPending()
protected boolean
isBucketFinished(com.yahoo.document.BucketId bucket)
Checks whether or not a given bucket is certain to be finished.boolean
isEmpty()
boolean
isFinished()
boolean
isInconsistentState()
static long
keyToBucketId(long key)
static long
makeNthBucketKey(long n, int distributionBits)
Directly generate a bucket Id key for then
th bucket in reverse sorted order.protected void
mergePendingBucket(com.yahoo.document.BucketId bucket)
double
percentFinished()
Calculate an estimate on how far we've managed to iterate over both the superbuckets and the sub-buckets.double
progressFraction(com.yahoo.document.BucketId superbucket, com.yahoo.document.BucketId progress)
byte[]
serialize()
String
serializeToString()
Returns a string (base64) encoding of the serial form of this tokenprotected void
setActiveBucketCount(long activeBucketCount)
protected void
setAllBucketsToState(ProgressToken.BucketState state)
protected void
setBucketCursor(long bucketCursor)
protected void
setDistributionBitCount(int distributionBits)
Set the internal number of distribution bits, which wil be used for writing the progress file and calculating correct percent-wise sub-bucket completion.protected void
setFinishedBucketCount(long finishedBucketCount)
For use by bucket sources and unit tests only!void
setInconsistentState(boolean inconsistentState)
protected void
setPendingBucketCount(long pendingBucketCount)
protected void
setTotalBucketCount(long totalBucketCount)
For use by bucket sources and unit tests only!protected void
splitPendingBucket(com.yahoo.document.BucketId bucket)
String
toString()
The format of the bucket progress output is as follows:protected void
updateProgress(com.yahoo.document.BucketId superbucket, com.yahoo.document.BucketId progress)
Updates internal progress state forbucket
, indicating it's currently atprogress
.
-
-
-
Constructor Detail
-
ProgressToken
public ProgressToken()
Creates a progress token.
-
ProgressToken
public ProgressToken(int distributionBits)
-
ProgressToken
public ProgressToken(String serialized)
-
ProgressToken
public ProgressToken(byte[] serialized)
-
-
Method Detail
-
serialize
public byte[] serialize()
-
serializeToString
public String serializeToString()
Returns a string (base64) encoding of the serial form of this token
-
fromSerializedString
public static ProgressToken fromSerializedString(String serializedString)
-
addFailedBucket
public void addFailedBucket(com.yahoo.document.BucketId superbucket, com.yahoo.document.BucketId progress, String errorMsg)
-
getFailedBuckets
public Map<com.yahoo.document.BucketId,com.yahoo.document.BucketId> getFailedBuckets()
Get all failed buckets and their progress. Not thread safe.- Returns:
- Unmodifiable map of all failed buckets
-
updateProgress
protected void updateProgress(com.yahoo.document.BucketId superbucket, com.yahoo.document.BucketId progress)
Updates internal progress state forbucket
, indicating it's currently atprogress
. Assumes that given a completely finished bucket, this function will not be called again to further update its progress after the finished-update.- Parameters:
superbucket
- A valid superbucket ID that exists inbuckets
progress
- The sub-bucket progress that has been reached in the superbucket- See Also:
VisitorIterator.update(com.yahoo.document.BucketId, com.yahoo.document.BucketId)
-
addBucket
protected void addBucket(com.yahoo.document.BucketId superbucket, com.yahoo.document.BucketId progress, ProgressToken.BucketState state)
For use internally by DocumentAPI code only. Using this method by itself will invariably lead to undefined ProgressToken state unless care is taken. Leave it to the VisitorIterator.- Parameters:
superbucket
- Superbucket that will be progress-trackedprogress
- Bucket progress thus farstate
- Initial bucket state. Only pending buckets may be returned
-
makeNthBucketKey
public static long makeNthBucketKey(long n, int distributionBits)
Directly generate a bucket Id key for then
th bucket in reverse sorted order.- Parameters:
n
- a number in the range [0, 2**distributionBits
)distributionBits
- Distribution bit count for the generated key- Returns:
- A value where, if you had generated 2**
distributionBits
BucketId
s with incremental numerical IDs and then sorted them on their reverse bit-order keys, the returned key would be equal to the nth element in the resulting sorted sequence.
-
getDistributionBitCount
public int getDistributionBitCount()
-
setDistributionBitCount
protected void setDistributionBitCount(int distributionBits)
Set the internal number of distribution bits, which wil be used for writing the progress file and calculating correct percent-wise sub-bucket completion. Note that simply invoking this method on the progress token does not actually change any of its bucket structures/counts! This is the bucket source's responsibility, since only it knows how such a change will affect the progress semantics.- Parameters:
distributionBits
- new distribution bit value
-
getActiveBucketCount
public long getActiveBucketCount()
-
getBucketCursor
public long getBucketCursor()
-
setBucketCursor
protected void setBucketCursor(long bucketCursor)
-
getFinishedBucketCount
public long getFinishedBucketCount()
-
setFinishedBucketCount
protected void setFinishedBucketCount(long finishedBucketCount)
For use by bucket sources and unit tests only!- Parameters:
finishedBucketCount
- Number of buckets the token has finished
-
getTotalBucketCount
public long getTotalBucketCount()
-
setTotalBucketCount
protected void setTotalBucketCount(long totalBucketCount)
For use by bucket sources and unit tests only!- Parameters:
totalBucketCount
- Total number of buckets that the progress token spans
-
getPendingBucketCount
public long getPendingBucketCount()
-
hasPending
public boolean hasPending()
-
hasActive
public boolean hasActive()
-
isFinished
public boolean isFinished()
-
isEmpty
public boolean isEmpty()
-
getFirstErrorMsg
public String getFirstErrorMsg()
-
containsFailedBuckets
public boolean containsFailedBuckets()
-
isInconsistentState
public boolean isInconsistentState()
-
setInconsistentState
public void setInconsistentState(boolean inconsistentState)
-
getBuckets
protected TreeMap<ProgressToken.BucketKeyWrapper,ProgressToken.BucketEntry> getBuckets()
Get internal progress token bucket state map. For internal use only!- Returns:
- Map of superbuckets → sub buckets
-
setActiveBucketCount
protected void setActiveBucketCount(long activeBucketCount)
-
setPendingBucketCount
protected void setPendingBucketCount(long pendingBucketCount)
-
toString
public String toString()
The format of the bucket progress output is as follows:VDS bucket progress file (n% completed)\n distribution bit count\n current bucket cursor\n number of finished buckets\n total number of buckets\n hex-of-superbucket:hex-of-progress\n ... repeat above line for each pending bucket ...
Note that unlike earlier versions of ProgressToken, the bucket IDs are not prefixed with '0x'.
-
percentFinished
public double percentFinished()
Calculate an estimate on how far we've managed to iterate over both the superbuckets and the sub-buckets. Runs in O(n+m) time, where n is the number of active buckets and m is the number of pending buckets. Both these values should be fairly small in practice, however. Method is synchronized, as legacy code treats this as an atomic read.- Returns:
- A value in the range [0, 100] estimating the progress.
-
bucketToKey
public static long bucketToKey(long id)
-
keyToBucketId
public static long keyToBucketId(long key)
-
progressFraction
public double progressFraction(com.yahoo.document.BucketId superbucket, com.yahoo.document.BucketId progress)
- Parameters:
superbucket
- The superbucket of whichprogress
is a sub-bucketprogress
- The sub-bucket for which a fractional progress should be calculated- Returns:
- a value in [0, 1] specifying how far the (sub-bucket) has reached in its superbucket. This is calculated by looking at the bucket's split factor.
-
isBucketFinished
protected boolean isBucketFinished(com.yahoo.document.BucketId bucket)
Checks whether or not a given bucket is certain to be finished. Only looks at the super-bucket part of the given bucket ID, so it's possible that the bucket has in fact finished on a sub-bucket progress level. This does not affect the correctness of the result, however. During a distribution bit change, the token's buckets may be inconsistent. In this scenario, false is always returned since we can't tell for sure if the bucket is still active until the buckets have been made consistent.- Parameters:
bucket
- Bucket to check whether or not is finished.- Returns:
true
ifbucket
's super-bucket is finished,false
otherwise.
-
splitPendingBucket
protected void splitPendingBucket(com.yahoo.document.BucketId bucket)
- Parameters:
bucket
- BucketId to be split into two buckets. Bucket's used-bits do not need to match the ProgressToken's current distribution bit count, as it is assumed the client knows what it's doing and will bring the token into a consistent state eventually.
-
mergePendingBucket
protected void mergePendingBucket(com.yahoo.document.BucketId bucket)
-
setAllBucketsToState
protected void setAllBucketsToState(ProgressToken.BucketState state)
-
clearAllBuckets
protected void clearAllBuckets()
-
-