com.google.cloud.dataflow.sdk.io

Class Source<T>

java.lang.Object

com.google.cloud.dataflow.sdk.io.Source<T>

Type Parameters:

T - Type of elements read by the source.

All Implemented Interfaces:

java.io.Serializable

Direct Known Subclasses:

DatastoreIO.Source

public abstract class Source<T>
extends java.lang.Object
implements java.io.Serializable

Base class for defining input formats, with custom logic for splitting the input into shards (parts of the input, each of which may be processed on a different worker) and creating a Source for reading the input.

To use this class for supporting your custom input type, derive your class class from it, and override the abstract methods. Also override either createWindowedReader(com.google.cloud.dataflow.sdk.options.PipelineOptions, com.google.cloud.dataflow.sdk.coders.Coder<com.google.cloud.dataflow.sdk.util.WindowedValue<T>>, com.google.cloud.dataflow.sdk.util.ExecutionContext) if your source supports timestamps and windows, or createBasicReader(com.google.cloud.dataflow.sdk.options.PipelineOptions, com.google.cloud.dataflow.sdk.coders.Coder<T>, com.google.cloud.dataflow.sdk.util.ExecutionContext) otherwise. For an example, see DatastoreIO.

A Source passed to a Read transform must be Serializable. This allows the Source instance created in this "main program" to be sent (in serialized form) to remote worker machines and reconstituted for each batch of elements of the input PCollection being processed or for each source splitting operation. A Source can have instance variable state, and non-transient instance variable state will be serialized in the main program and then deserialized on remote worker machines.

This API is experimental and subject to change.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	Source.Reader<T> The interface which readers of custom input sources must implement.

Constructor Summary

Constructors

Constructor and Description
Source()

Method Summary

Methods

Modifier and Type	Method and Description
protected Source.Reader<T>	createBasicReader(PipelineOptions options, Coder<T> coder, com.google.cloud.dataflow.sdk.util.ExecutionContext executionContext) Creates a basic (non-windowed) reader for this source.
Source.Reader<com.google.cloud.dataflow.sdk.util.WindowedValue<T>>	createWindowedReader(PipelineOptions options, Coder<com.google.cloud.dataflow.sdk.util.WindowedValue<T>> coder, com.google.cloud.dataflow.sdk.util.ExecutionContext executionContext) Creates a windowed reader for this source.
abstract Coder<T>	getDefaultOutputCoder() Returns the default Coder to use for the data read from this source.
abstract long	getEstimatedSizeBytes(PipelineOptions options) An estimate of the total size (in bytes) of the data that would be read from this source.
abstract boolean	producesSortedKeys(PipelineOptions options) Whether this source is known to produce key/value pairs with the (encoded) keys in lexicographically sorted order.
abstract java.util.List<? extends Source<T>>	splitIntoShards(long desiredShardSizeBytes, PipelineOptions options) Splits the source into shards.
abstract void	validate() Checks that this source is valid, before it can be used into a pipeline.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Source

public Source()

Method Detail

splitIntoShards

public abstract java.util.List<? extends Source<T>> splitIntoShards(long desiredShardSizeBytes,
                                                  PipelineOptions options)
                                                             throws java.lang.Exception

Splits the source into shards.

PipelineOptions can be used to get information such as credentials for accessing an external storage.

Throws:

java.lang.Exception

getEstimatedSizeBytes

public abstract long getEstimatedSizeBytes(PipelineOptions options)
                                    throws java.lang.Exception

An estimate of the total size (in bytes) of the data that would be read from this source. This estimate is in terms of external storage size, before any decompression or other processing done by the reader.

Throws:

java.lang.Exception

producesSortedKeys

public abstract boolean producesSortedKeys(PipelineOptions options)
                                    throws java.lang.Exception

Whether this source is known to produce key/value pairs with the (encoded) keys in lexicographically sorted order.

Throws:

java.lang.Exception

createWindowedReader

public Source.Reader<com.google.cloud.dataflow.sdk.util.WindowedValue<T>> createWindowedReader(PipelineOptions options,
                                                                                      Coder<com.google.cloud.dataflow.sdk.util.WindowedValue<T>> coder,
                                                                                      @Nullable
                                                                                      com.google.cloud.dataflow.sdk.util.ExecutionContext executionContext)
                                                                                        throws java.io.IOException

Creates a windowed reader for this source. The default implementation wraps createBasicReader(com.google.cloud.dataflow.sdk.options.PipelineOptions, com.google.cloud.dataflow.sdk.coders.Coder<T>, com.google.cloud.dataflow.sdk.util.ExecutionContext). Override this function if your reader supports timestamps and windows; otherwise, override createBasicReader(com.google.cloud.dataflow.sdk.options.PipelineOptions, com.google.cloud.dataflow.sdk.coders.Coder<T>, com.google.cloud.dataflow.sdk.util.ExecutionContext) instead.

Throws:

java.io.IOException

createBasicReader

protected Source.Reader<T> createBasicReader(PipelineOptions options,
                                 Coder<T> coder,
                                 @Nullable
                                 com.google.cloud.dataflow.sdk.util.ExecutionContext executionContext)
                                      throws java.io.IOException

Creates a basic (non-windowed) reader for this source. If you override this method, each value returned by this reader will be wrapped into the global window.

Throws:

java.io.IOException

validate

public abstract void validate()

Checks that this source is valid, before it can be used into a pipeline. It is recommended to use Preconditions for implementing this method.

getDefaultOutputCoder

public abstract Coder<T> getDefaultOutputCoder()

Returns the default Coder to use for the data read from this source.