com.google.cloud.dataflow.sdk.io

Class OffsetBasedSource.OffsetBasedReader<T>

java.lang.Object

com.google.cloud.dataflow.sdk.io.Source.Reader<T>

com.google.cloud.dataflow.sdk.io.BoundedSource.BoundedReader<T>

com.google.cloud.dataflow.sdk.io.OffsetBasedSource.OffsetBasedReader<T>

All Implemented Interfaces:

AutoCloseable

Direct Known Subclasses:

FileBasedSource.FileBasedReader

Enclosing class:

OffsetBasedSource<T>

public abstract static class OffsetBasedSource.OffsetBasedReader<T>
extends BoundedSource.BoundedReader<T>

A Source.Reader that implements code common to readers of all OffsetBasedSources.

Subclasses have to implement:

• The methods startImpl() and advanceImpl() for reading the first or subsequent records.
• The methods Source.Reader.getCurrent(), getCurrentOffset(), and optionally isAtSplitPoint() and BoundedSource.BoundedReader.getCurrentTimestamp() to access properties of the last record successfully read by startImpl() or advanceImpl().

Constructor Summary

Constructors

Constructor and Description
OffsetBasedReader(OffsetBasedSource<T> source)

Method Summary

Modifier and Type	Method and Description
boolean	advance() Advances the reader to the next valid record.
protected abstract boolean	advanceImpl() Same as BoundedReader#advance, except OffsetBasedSource.OffsetBasedReader base class takes care of coordinating against concurrent calls to splitAtFraction(double).
protected abstract long	getCurrentOffset() Returns the starting offset of the current record, which has been read by the last successful Source.Reader.start() or Source.Reader.advance() call.
OffsetBasedSource<T>	getCurrentSource() Returns a Source describing the same input that this Reader reads (including items already read).
Double	getFractionConsumed() Returns a value in [0, 1] representing approximately what fraction of the source (BoundedSource.BoundedReader.getCurrentSource()) this reader has read so far.
protected boolean	isAtSplitPoint() Returns whether the current record is at a split point (i.e., whether the current record would be the first record to be read by a source with a specified start offset of getCurrentOffset()).
OffsetBasedSource<T>	splitAtFraction(double fraction) Tells the reader to narrow the range of the input it's going to read and give up the remainder, so that the new range would contain approximately the given fraction of the amount of data in the current range.
boolean	start() Initializes the reader and advances the reader to the first record.
protected abstract boolean	startImpl() Same as BoundedReader#start, except OffsetBasedSource.OffsetBasedReader base class takes care of coordinating against concurrent calls to splitAtFraction(double).

Methods inherited from class com.google.cloud.dataflow.sdk.io.BoundedSource.BoundedReader

getCurrentTimestamp

Methods inherited from class com.google.cloud.dataflow.sdk.io.Source.Reader

close, getCurrent

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

OffsetBasedReader

public OffsetBasedReader(OffsetBasedSource<T> source)

Parameters:

source - the OffsetBasedSource to be read by the current reader.

Method Detail

getCurrentOffset

protected abstract long getCurrentOffset()
                                  throws NoSuchElementException

Returns the starting offset of the current record, which has been read by the last successful Source.Reader.start() or Source.Reader.advance() call.

If no such call has been made yet, the return value is unspecified.

See RangeTracker for description of offset semantics.

Throws:

NoSuchElementException

isAtSplitPoint

protected boolean isAtSplitPoint()
                          throws NoSuchElementException

Returns whether the current record is at a split point (i.e., whether the current record would be the first record to be read by a source with a specified start offset of getCurrentOffset()).

See detailed documentation about split points in RangeTracker.

Throws:

NoSuchElementException

start

public final boolean start()
                    throws IOException

Description copied from class: Source.Reader

Initializes the reader and advances the reader to the first record.

This method should be called exactly once. The invocation should occur prior to calling Source.Reader.advance() or Source.Reader.getCurrent(). This method may perform expensive operations that are needed to initialize the reader.

Specified by:

start in class Source.Reader<T>

Returns:

true if a record was read, false if there is no more input available.

Throws:

IOException

advance

public final boolean advance()
                      throws IOException

Description copied from class: Source.Reader

Advances the reader to the next valid record.

Specified by:

advance in class Source.Reader<T>

Returns:

true if a record was read, false if there is no more input available.

Throws:

IOException

startImpl

protected abstract boolean startImpl()
                              throws IOException

Same as BoundedReader#start, except OffsetBasedSource.OffsetBasedReader base class takes care of coordinating against concurrent calls to splitAtFraction(double).

Throws:

IOException

advanceImpl

protected abstract boolean advanceImpl()
                                throws IOException

Same as BoundedReader#advance, except OffsetBasedSource.OffsetBasedReader base class takes care of coordinating against concurrent calls to splitAtFraction(double).

Throws:

IOException

getCurrentSource

public OffsetBasedSource<T> getCurrentSource()

Description copied from class: Source.Reader

Returns a Source describing the same input that this Reader reads (including items already read).

A reader created from the result of getCurrentSource, if consumed, MUST return the same data items as the current reader.

Specified by:

getCurrentSource in class BoundedSource.BoundedReader<T>

getFractionConsumed

public Double getFractionConsumed()

Description copied from class: BoundedSource.BoundedReader

Returns a value in [0, 1] representing approximately what fraction of the source (BoundedSource.BoundedReader.getCurrentSource()) this reader has read so far.

It is recommended that this method should satisfy the following properties:

• Should return 0 before the Source.Reader.start() call.
• Should return 1 after a Source.Reader.start() or Source.Reader.advance() call that returns false.
• The returned values should be non-decreasing (though they don't have to be unique).

By default, returns null to indicate that this cannot be estimated.

Thread safety

If BoundedSource.BoundedReader.splitAtFraction(double) is implemented, this method can be called concurrently to other methods (including itself), and it is therefore critical for it to be implemented in a thread-safe way.

Overrides:

getFractionConsumed in class BoundedSource.BoundedReader<T>

Returns:

A value in [0, 1] representing the fraction of this reader's current input read so far, or null if such an estimate is not available.

splitAtFraction

public final OffsetBasedSource<T> splitAtFraction(double fraction)

Description copied from class: BoundedSource.BoundedReader

Tells the reader to narrow the range of the input it's going to read and give up the remainder, so that the new range would contain approximately the given fraction of the amount of data in the current range.

Returns a BoundedSource representing the remainder.

Detailed description

Assuming the following sequence of calls:

   BoundedSource<T> initial = reader.getCurrentSource();
   BoundedSource<T> residual = reader.splitAtFraction(fraction);
   BoundedSource<T> primary = reader.getCurrentSource();
 

• The "primary" and "residual" sources, when read, should together cover the same set of records as "initial".
• The current reader should continue to be in a valid state, and continuing to read from it should, together with the records it already read, yield the same records as would have been read by "primary".
• The amount of data read by "primary" should ideally represent approximately the given fraction of the amount of data read by "initial".

For example, a reader that reads a range of offsets [A, B) in a file might implement this method by truncating the current range to [A, A + fraction*(B-A)) and returning a Source representing the range [A + fraction*(B-A), B).

This method should return null if the split cannot be performed for this fraction while satisfying the semantics above. E.g., a reader that reads a range of offsets in a file should return null if it is already past the position in its range corresponding to the given fraction. In this case, the method MUST have no effect (the reader must behave as if the method hadn't been called at all).

Statefulness

Since this method (if successful) affects the reader's source, in subsequent invocations "fraction" should be interpreted relative to the new current source.

Thread safety and blocking

This method will be called concurrently to other methods (however there will not be multiple concurrent invocations of this method itself), and it is critical for it to be implemented in a thread-safe way (otherwise data loss is possible).

It is also very important that this method always completes quickly, in particular, it should not perform or wait on any blocking operations such as I/O, RPCs etc. Violating this requirement may stall completion of the work item or even cause it to fail.

E.g. it is incorrect to make both this method and Source.Reader.start()/Source.Reader.advance() synchronized, because those methods can perform blocking operations, and then this method would have to wait for those calls to complete.

RangeTracker makes it easy to implement this method safely and correctly.

By default, returns null to indicate that splitting is not possible.

Overrides:

splitAtFraction in class BoundedSource.BoundedReader<T>