com.google.cloud.dataflow.sdk.io

Class OffsetBasedSource<T>

java.lang.Object

com.google.cloud.dataflow.sdk.io.Source<T>

com.google.cloud.dataflow.sdk.io.BoundedSource<T>

com.google.cloud.dataflow.sdk.io.OffsetBasedSource<T>

Type Parameters:

T - Type of records represented by the source.

All Implemented Interfaces:

Serializable

Direct Known Subclasses:

FileBasedSource

public abstract class OffsetBasedSource<T>
extends BoundedSource<T>

A Source that uses offsets to define starting and ending positions.

Extend this class to implement your own offset based custom source. FileBasedSource, which is a subclass of this, adds additional functionality useful for custom sources that are based on files. If possible implementors should start from FileBasedSource instead of OffsetBasedSource.

This is a common base class for all sources that use an offset range. It stores the range and implements splitting into bundles. This should be used for sources that can be cheaply read starting at any given offset.

Consult RangeTracker for important semantics common to all sources defined by a range of positions of a certain type, including the semantics of split points (OffsetBasedSource.OffsetBasedReader.isAtSplitPoint()).

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	OffsetBasedSource.OffsetBasedReader<T> A Source.Reader that implements code common to readers of all OffsetBasedSources.

Nested classes/interfaces inherited from class com.google.cloud.dataflow.sdk.io.BoundedSource

BoundedSource.BoundedReader<T>

Nested classes/interfaces inherited from class com.google.cloud.dataflow.sdk.io.Source

Source.Reader<T>

Constructor Summary

Constructors

Constructor and Description
OffsetBasedSource(long startOffset, long endOffset, long minBundleSize)

Method Summary

Modifier and Type	Method and Description
abstract OffsetBasedSource<T>	createSourceForSubrange(long start, long end) Returns an OffsetBasedSource for a subrange of the current source.
long	getBytesPerOffset() Returns approximately how many bytes of data correspond to a single offset in this source.
long	getEndOffset() Returns the specified ending offset of the source.
long	getEstimatedSizeBytes(PipelineOptions options) An estimate of the total size (in bytes) of the data that would be read from this source.
abstract long	getMaxEndOffset(PipelineOptions options) Returns the exact ending offset of the current source.
long	getMinBundleSize() Returns the minimum bundle size that should be used when splitting the source into sub-sources.
long	getStartOffset() Returns the starting offset of the source.
List<? extends OffsetBasedSource<T>>	splitIntoBundles(long desiredBundleSizeBytes, PipelineOptions options) Splits the source into bundles of approximately given size (in bytes).
String	toString()
void	validate() Checks that this source is valid, before it can be used in a pipeline.

Methods inherited from class com.google.cloud.dataflow.sdk.io.BoundedSource

createReader, producesSortedKeys

Methods inherited from class com.google.cloud.dataflow.sdk.io.Source

getDefaultOutputCoder

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

OffsetBasedSource

public OffsetBasedSource(long startOffset,
                         long endOffset,
                         long minBundleSize)

Parameters:

startOffset - starting offset (inclusive) of the source. Must be non-negative.

endOffset - ending offset (exclusive) of the source. Any offset >= getMaxEndOffset(), e.g., Long.MAX_VALUE, means the same as getMaxEndOffset(). Must be >= startOffset.

minBundleSize - minimum bundle size in offset units that should be used when splitting the source into sub-sources. This will not be respected if the total range of the source is smaller than the specified minBundleSize. Must be non-negative.

Method Detail

getStartOffset

public long getStartOffset()

Returns the starting offset of the source.

getEndOffset

public long getEndOffset()

Returns the specified ending offset of the source. If this is >= getMaxEndOffset(), e.g. Long.MAX_VALUE, this implies getMaxEndOffset().

getMinBundleSize

public long getMinBundleSize()

Returns the minimum bundle size that should be used when splitting the source into sub-sources. This will not be respected if the total range of the source is smaller than the specified minBundleSize.

getEstimatedSizeBytes

public long getEstimatedSizeBytes(PipelineOptions options)
                           throws Exception

Description copied from class: BoundedSource

An estimate of the total size (in bytes) of the data that would be read from this source. This estimate is in terms of external storage size, before any decompression or other processing done by the reader.

Specified by:

getEstimatedSizeBytes in class BoundedSource<T>

Throws:

Exception

splitIntoBundles

public List<? extends OffsetBasedSource<T>> splitIntoBundles(long desiredBundleSizeBytes,
                                                             PipelineOptions options)
                                                      throws Exception

Description copied from class: BoundedSource

Splits the source into bundles of approximately given size (in bytes).

Specified by:

splitIntoBundles in class BoundedSource<T>

Throws:

Exception

validate

public void validate()

Description copied from class: Source

Checks that this source is valid, before it can be used in a pipeline.

It is recommended to use Preconditions for implementing this method.

Specified by:

validate in class Source<T>

toString

public String toString()

Overrides:

toString in class Object

getBytesPerOffset

public long getBytesPerOffset()

Returns approximately how many bytes of data correspond to a single offset in this source. Used for translation between this source's range and methods defined in terms of bytes, such as getEstimatedSizeBytes(com.google.cloud.dataflow.sdk.options.PipelineOptions) and splitIntoBundles(long, com.google.cloud.dataflow.sdk.options.PipelineOptions).

getMaxEndOffset

public abstract long getMaxEndOffset(PipelineOptions options)
                              throws Exception

Returns the exact ending offset of the current source. This will be used if the source was constructed with an endOffset value Long.MAX_VALUE.

Throws:

Exception

createSourceForSubrange

public abstract OffsetBasedSource<T> createSourceForSubrange(long start,
                                                             long end)

Returns an OffsetBasedSource for a subrange of the current source. [start, end) will be within the range [startOffset, endOffset] of the current source.