com.google.cloud.dataflow.sdk.io

Class BoundedSource.AbstractBoundedReader<T>

java.lang.Object

com.google.cloud.dataflow.sdk.io.Source.AbstractReader<T>

com.google.cloud.dataflow.sdk.io.BoundedSource.AbstractBoundedReader<T>

Type Parameters:

T -

All Implemented Interfaces:

BoundedSource.BoundedReader<T>, Source.Reader<T>, java.lang.AutoCloseable

Direct Known Subclasses:

ByteOffsetBasedSource.ByteOffsetBasedReader, DatastoreIO.DatastoreReader

Enclosing class:

BoundedSource<T>

public abstract static class BoundedSource.AbstractBoundedReader<T>
extends Source.AbstractReader<T>
implements BoundedSource.BoundedReader<T>

A base class implementing some optional methods of BoundedSource.BoundedReader in a default way:

• Progress estimation (getFractionConsumed()) is not supported.
• Dynamic splitting (splitAtFraction(double)) is not supported.

Constructor Summary

Constructors

Constructor and Description
BoundedSource.AbstractBoundedReader()

Method Summary

Methods

Modifier and Type	Method and Description
java.lang.Double	getFractionConsumed() Returns a value in [0, 1] representing approximately what fraction of the source (BoundedSource.BoundedReader.getCurrentSource()) this reader has read so far.
BoundedSource<T>	splitAtFraction(double fraction) Tells the reader to narrow the range of the input it's going to read and give up the remainder, so that the new range would contain approximately the given fraction of the amount of data in the current range.

Methods inherited from class com.google.cloud.dataflow.sdk.io.Source.AbstractReader

getCurrentTimestamp

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.google.cloud.dataflow.sdk.io.BoundedSource.BoundedReader

getCurrentSource

Methods inherited from interface com.google.cloud.dataflow.sdk.io.Source.Reader

advance, close, getCurrent, getCurrentTimestamp, start

Constructor Detail

BoundedSource.AbstractBoundedReader

public BoundedSource.AbstractBoundedReader()

Method Detail

getFractionConsumed

public java.lang.Double getFractionConsumed()

Description copied from interface: BoundedSource.BoundedReader

Returns a value in [0, 1] representing approximately what fraction of the source (BoundedSource.BoundedReader.getCurrentSource()) this reader has read so far.

It is recommended that this method should satisfy the following properties:

• Should return 0 before the Source.Reader.start() call.
• Should return 1 after a Source.Reader.start() or Source.Reader.advance() call that returns false.
• The returned values should be non-decreasing (though they don't have to be unique).

Specified by:

getFractionConsumed in interface BoundedSource.BoundedReader<T>

Returns:

A value in [0, 1] representing the fraction of this reader's current input read so far, or null if such an estimate is not available.

splitAtFraction

public BoundedSource<T> splitAtFraction(double fraction)

Description copied from interface: BoundedSource.BoundedReader

Tells the reader to narrow the range of the input it's going to read and give up the remainder, so that the new range would contain approximately the given fraction of the amount of data in the current range. Returns a BoundedSource representing the remainder.

More detailed description: Assuming the following sequence of calls:

BoundedSource<T> initial = reader.getCurrentSource();
   BoundedSource<T> residual = reader.splitAtFraction(fraction);
   BoundedSource<T> primary = reader.getCurrentSource();
 

• The "primary" and "residual" sources, when read, would together cover the same set of records as "initial".
• The current reader should continue to be in a valid state, and continuing to read from it should, together with the records it already read, yield the same records as would have been read by "primary".
• The amount of data read by "primary" should ideally represent approximately the given fraction of the amount of data read by "initial".

For example, a reader that reads a range of offsets [A, B) in a file might implement this method by truncating the current range to [A, A + fraction*(B-A)) and returning a Source representing the range [A + fraction*(B-A), B).

This method should return null if the split cannot be performed for this fraction while satisfying the semantics above. E.g., a reader that reads a range of offsets in a file should return null if it is already past the position in its range corresponding to the given fraction. In this case, the method MUST have no effect (the reader must behave as if the method hadn't been called at all).

Since this method (if successful) affects the reader's source, in subsequent invocations "fraction" should be interpreted relative to the new current source.

Specified by:

splitAtFraction in interface BoundedSource.BoundedReader<T>