com.google.cloud.dataflow.sdk.io

Class BoundedSource<T>

java.lang.Object

com.google.cloud.dataflow.sdk.io.Source<T>

com.google.cloud.dataflow.sdk.io.BoundedSource<T>

Type Parameters:

T - Type of records read by the source.

All Implemented Interfaces:

java.io.Serializable

Direct Known Subclasses:

ByteOffsetBasedSource, DatastoreIO.Source

public abstract class BoundedSource<T>
extends Source<T>

A Source that reads a bounded amount of input and, because of that, supports some additional operations, e.g. size estimation, and its reader supports progress estimation.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	BoundedSource.AbstractBoundedReader<T> A base class implementing some optional methods of BoundedSource.BoundedReader in a default way: Progress estimation (BoundedSource.AbstractBoundedReader.getFractionConsumed()) is not supported.
static interface	BoundedSource.BoundedReader<T> A Reader that reads a bounded amount of input and supports some additional operations, such as progress estimation and dynamic work rebalancing.

Nested classes/interfaces inherited from class com.google.cloud.dataflow.sdk.io.Source

Source.AbstractReader<T>, Source.Reader<T>

Constructor Summary

Constructors

Constructor and Description
BoundedSource()

Method Summary

Methods

Modifier and Type	Method and Description
BoundedSource.BoundedReader<T>	createReader(PipelineOptions options, com.google.cloud.dataflow.sdk.util.ExecutionContext executionContext) Creates a reader for this source.
abstract long	getEstimatedSizeBytes(PipelineOptions options) An estimate of the total size (in bytes) of the data that would be read from this source.
abstract boolean	producesSortedKeys(PipelineOptions options) Whether this source is known to produce key/value pairs with the (encoded) keys in lexicographically sorted order.

Methods inherited from class com.google.cloud.dataflow.sdk.io.Source

getDefaultOutputCoder, splitIntoBundles, validate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BoundedSource

public BoundedSource()

Method Detail

getEstimatedSizeBytes

public abstract long getEstimatedSizeBytes(PipelineOptions options)
                                    throws java.lang.Exception

An estimate of the total size (in bytes) of the data that would be read from this source. This estimate is in terms of external storage size, before any decompression or other processing done by the reader.

Throws:

java.lang.Exception

producesSortedKeys

public abstract boolean producesSortedKeys(PipelineOptions options)
                                    throws java.lang.Exception

Whether this source is known to produce key/value pairs with the (encoded) keys in lexicographically sorted order.

Throws:

java.lang.Exception

createReader

public BoundedSource.BoundedReader<T> createReader(PipelineOptions options,
                                          @Nullable
                                          com.google.cloud.dataflow.sdk.util.ExecutionContext executionContext)
                                            throws java.io.IOException

Description copied from class: Source

Creates a reader for this source.

Overrides:

createReader in class Source<T>

Throws:

java.io.IOException