com.google.cloud.dataflow.sdk.util.common.worker

Class Reader<T>

java.lang.Object

java.util.Observable

com.google.cloud.dataflow.sdk.util.common.worker.Reader<T>

Type Parameters:

T - the type of the elements read from the source

Direct Known Subclasses:

AvroByteReader, AvroReader, BigQueryReader, ConcatReader, FileBasedReader, GroupingShuffleReader, InMemoryReader, PartitioningShuffleReader, UngroupedShuffleReader

public abstract class Reader<T>
extends Observable

Abstract base class for readers.

A Source is read from by getting an Iterator-like value and iterating through it.

This class is intended for formats that have built-in support on the Dataflow service. Do not introduce new implementations: for creating new input formats, use Source instead.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	Reader.AbstractReaderIterator<T> An abstract base class for ReaderIterator implementations.
static interface	Reader.DynamicSplitRequest A framework-specific way to specify how Reader.ReaderIterator.requestDynamicSplit(com.google.cloud.dataflow.sdk.util.common.worker.Reader.DynamicSplitRequest) should split the input into a primary and residual part.
static interface	Reader.DynamicSplitResult A framework-specific way to specify how Reader.ReaderIterator.requestDynamicSplit(com.google.cloud.dataflow.sdk.util.common.worker.Reader.DynamicSplitRequest) has split the input into a primary and residual part.
static class	Reader.DynamicSplitResultWithPosition A Reader.DynamicSplitResult that specifies the boundary between the primary and residual parts of the input using a Reader.Position.
static interface	Reader.Position A representation of a position in an iteration through a Reader.
static interface	Reader.Progress A representation of how far a ReaderIterator is through a Reader.
static interface	Reader.ReaderIterator<T> A stateful iterator over the data in a Reader.

Field Summary

Fields

Modifier and Type	Field and Description
protected StateSampler	stateSampler StateSampler object for readers interested in further breaking down of the state space at a finer granularity.
protected String	stateSamplerOperationName Name to be used as a prefix with stateSampler.

Constructor Summary

Constructors

Constructor and Description
Reader()

Method Summary

Modifier and Type	Method and Description
abstract Reader.ReaderIterator<T>	iterator() Returns a ReaderIterator that allows reading from this source.
protected void	notifyElementRead(long byteSize) Utility method to notify observers about a new element, which has been read by this Reader, and its size in bytes.
void	setStateSamplerAndOperationName(StateSampler stateSampler, String stateSamplerOperationName) Sets the state sampler and the state sampler operation name.
boolean	supportsRestart() Returns whether this Reader can be restarted.

Methods inherited from class java.util.Observable

addObserver, clearChanged, countObservers, deleteObserver, deleteObservers, hasChanged, notifyObservers, notifyObservers, setChanged

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

stateSampler

protected StateSampler stateSampler

StateSampler object for readers interested in further breaking down of the state space at a finer granularity.

stateSamplerOperationName

protected String stateSamplerOperationName

Name to be used as a prefix with stateSampler.

Constructor Detail

Reader

public Reader()

Method Detail

setStateSamplerAndOperationName

public void setStateSamplerAndOperationName(StateSampler stateSampler,
                                            String stateSamplerOperationName)

Sets the state sampler and the state sampler operation name.

Parameters:

stateSampler - the StateSampler object

stateSamplerOperationName - the operation name to be used by the state sampler

iterator

public abstract Reader.ReaderIterator<T> iterator()
                                           throws IOException

Returns a ReaderIterator that allows reading from this source.

Throws:

IOException

notifyElementRead

protected void notifyElementRead(long byteSize)

Utility method to notify observers about a new element, which has been read by this Reader, and its size in bytes. Normally, there is only one observer, which is a ReadOperation that encapsules this Reader. Derived classes must call this method whenever they read additional data, even if that element may never be returned from the corresponding source iterator.

supportsRestart

public boolean supportsRestart()

Returns whether this Reader can be restarted.