com.google.cloud.dataflow.sdk.runners.worker

Class FileBasedReader<T>

java.lang.Object

java.util.Observable

com.google.cloud.dataflow.sdk.util.common.worker.Reader<T>

com.google.cloud.dataflow.sdk.runners.worker.FileBasedReader<T>

Type Parameters:

T - the type of the elements read from the source

Direct Known Subclasses:

TextReader

public abstract class FileBasedReader<T>
extends Reader<T>

Abstract base class for sources that read from files.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	FileBasedReader.DecompressingStreamFactory Factory interface for creating a decompressing InputStream.
protected class	FileBasedReader.FileBasedIterator Abstract base class for file-based source iterators.
protected static class	FileBasedReader.FilenameBasedStreamFactory Factory for creating decompressing input streams based on a filename and a TextIO.CompressionType.

Nested classes/interfaces inherited from class com.google.cloud.dataflow.sdk.util.common.worker.Reader

Reader.AbstractReaderIterator<T>, Reader.DynamicSplitRequest, Reader.DynamicSplitResult, Reader.DynamicSplitResultWithPosition, Reader.Position, Reader.Progress, Reader.ReaderIterator<T>

Field Summary

Fields

Modifier and Type	Field and Description
protected static int	BUF_SIZE
protected Coder<T>	coder
protected Long	endPosition
protected String	filename
protected Long	startPosition
protected boolean	useDefaultBufferSize

Fields inherited from class com.google.cloud.dataflow.sdk.util.common.worker.Reader

stateSampler, stateSamplerOperationName

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	FileBasedReader(String filename, Long startPosition, Long endPosition, Coder<T> coder, boolean useDefaultBufferSize)

Method Summary

Modifier and Type	Method and Description
Reader.ReaderIterator<T>	iterator() Returns a ReaderIterator that allows reading from this source.
protected abstract Reader.ReaderIterator<T>	newReaderIteratorForFiles(IOChannelFactory factory, Collection<String> files) Returns a new iterator for elements in the given files.
protected abstract Reader.ReaderIterator<T>	newReaderIteratorForRangeInFile(IOChannelFactory factory, String oneFile, long startPosition, Long endPosition) Returns a new iterator for elements in the given range in the given file.

Methods inherited from class com.google.cloud.dataflow.sdk.util.common.worker.Reader

notifyElementRead, setStateSamplerAndOperationName, supportsRestart

Methods inherited from class java.util.Observable

addObserver, clearChanged, countObservers, deleteObserver, deleteObservers, hasChanged, notifyObservers, notifyObservers, setChanged

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

BUF_SIZE

protected static final int BUF_SIZE

See Also:

Constant Field Values

filename

protected final String filename

startPosition

@Nullable
protected final Long startPosition

endPosition

@Nullable
protected final Long endPosition

coder

protected final Coder<T> coder

useDefaultBufferSize

protected final boolean useDefaultBufferSize

Constructor Detail

FileBasedReader

protected FileBasedReader(String filename,
                          @Nullable
                          Long startPosition,
                          @Nullable
                          Long endPosition,
                          Coder<T> coder,
                          boolean useDefaultBufferSize)

Method Detail

newReaderIteratorForRangeInFile

protected abstract Reader.ReaderIterator<T> newReaderIteratorForRangeInFile(IOChannelFactory factory,
                                                                            String oneFile,
                                                                            long startPosition,
                                                                            @Nullable
                                                                            Long endPosition)
                                                                     throws IOException

Returns a new iterator for elements in the given range in the given file. If the range starts in the middle an element, this element is skipped as it is considered part of the previous range; if the last element that starts in the range finishes beyond the end position, it is still considered part of this range. In other words, the start position and the end position are "rounded up" to element boundaries.

Parameters:

endPosition - offset of the end position; null means end-of-file

Throws:

IOException

newReaderIteratorForFiles

protected abstract Reader.ReaderIterator<T> newReaderIteratorForFiles(IOChannelFactory factory,
                                                                      Collection<String> files)
                                                               throws IOException

Returns a new iterator for elements in the given files. Caller must ensure that the file collection is not empty.

Throws:

IOException

iterator

public Reader.ReaderIterator<T> iterator()
                                  throws IOException

Description copied from class: Reader

Returns a ReaderIterator that allows reading from this source.

Specified by:

iterator in class Reader<T>

Throws:

IOException