com.google.cloud.dataflow.sdk.io

Class DatastoreIO.Source

java.lang.Object

com.google.cloud.dataflow.sdk.io.Source<com.google.api.services.datastore.DatastoreV1.Entity>

com.google.cloud.dataflow.sdk.io.DatastoreIO.Source

All Implemented Interfaces:

java.io.Serializable

Enclosing class:

DatastoreIO

public static class DatastoreIO.Source
extends Source<com.google.api.services.datastore.DatastoreV1.Entity>

A source that reads the result rows of a Datastore query as Entity objects.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class com.google.cloud.dataflow.sdk.io.Source

Source.Reader<T>

Method Summary

Methods

Modifier and Type	Method and Description
Source.Reader<com.google.api.services.datastore.DatastoreV1.Entity>	createBasicReader(PipelineOptions pipelineOptions, Coder<com.google.api.services.datastore.DatastoreV1.Entity> coder, com.google.cloud.dataflow.sdk.util.ExecutionContext executionContext) Creates a basic (non-windowed) reader for this source.
Coder<com.google.api.services.datastore.DatastoreV1.Entity>	getDefaultOutputCoder() Returns the default Coder to use for the data read from this source.
long	getEstimatedSizeBytes(PipelineOptions options) An estimate of the total size (in bytes) of the data that would be read from this source.
boolean	producesSortedKeys(PipelineOptions options) Whether this source is known to produce key/value pairs with the (encoded) keys in lexicographically sorted order.
java.util.List<DatastoreIO.Source>	splitIntoShards(long desiredShardSizeBytes, PipelineOptions options) Splits the source into shards.
void	validate() Checks that this source is valid, before it can be used into a pipeline.
DatastoreIO.Source	withDataset(java.lang.String datasetId)
DatastoreIO.Source	withHost(java.lang.String host)
DatastoreIO.Source	withMockEstimateSizeBytes(com.google.common.base.Supplier<java.lang.Long> estimateSizeBytes) For testing only.
DatastoreIO.Source	withQuery(com.google.api.services.datastore.DatastoreV1.Query query)

Methods inherited from class com.google.cloud.dataflow.sdk.io.Source

createWindowedReader

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

withDataset

public DatastoreIO.Source withDataset(java.lang.String datasetId)

withQuery

public DatastoreIO.Source withQuery(com.google.api.services.datastore.DatastoreV1.Query query)

withHost

public DatastoreIO.Source withHost(java.lang.String host)

getDefaultOutputCoder

public Coder<com.google.api.services.datastore.DatastoreV1.Entity> getDefaultOutputCoder()

Description copied from class: Source

Returns the default Coder to use for the data read from this source.

Specified by:

getDefaultOutputCoder in class Source<com.google.api.services.datastore.DatastoreV1.Entity>

getEstimatedSizeBytes

public long getEstimatedSizeBytes(PipelineOptions options)
                           throws java.lang.Exception

Description copied from class: Source

An estimate of the total size (in bytes) of the data that would be read from this source. This estimate is in terms of external storage size, before any decompression or other processing done by the reader.

Specified by:

getEstimatedSizeBytes in class Source<com.google.api.services.datastore.DatastoreV1.Entity>

Throws:

java.lang.Exception

producesSortedKeys

public boolean producesSortedKeys(PipelineOptions options)

Description copied from class: Source

Whether this source is known to produce key/value pairs with the (encoded) keys in lexicographically sorted order.

Specified by:

producesSortedKeys in class Source<com.google.api.services.datastore.DatastoreV1.Entity>

splitIntoShards

public java.util.List<DatastoreIO.Source> splitIntoShards(long desiredShardSizeBytes,
                                                 PipelineOptions options)
                                                   throws java.lang.Exception

Description copied from class: Source

Splits the source into shards.

PipelineOptions can be used to get information such as credentials for accessing an external storage.

Specified by:

splitIntoShards in class Source<com.google.api.services.datastore.DatastoreV1.Entity>

Throws:

java.lang.Exception

createBasicReader

public Source.Reader<com.google.api.services.datastore.DatastoreV1.Entity> createBasicReader(PipelineOptions pipelineOptions,
                                                                                    Coder<com.google.api.services.datastore.DatastoreV1.Entity> coder,
                                                                                    com.google.cloud.dataflow.sdk.util.ExecutionContext executionContext)
                                                                                      throws java.io.IOException

Description copied from class: Source

Creates a basic (non-windowed) reader for this source. If you override this method, each value returned by this reader will be wrapped into the global window.

Overrides:

createBasicReader in class Source<com.google.api.services.datastore.DatastoreV1.Entity>

Throws:

java.io.IOException

validate

public void validate()

Description copied from class: Source

Checks that this source is valid, before it can be used into a pipeline. It is recommended to use Preconditions for implementing this method.

Specified by:

validate in class Source<com.google.api.services.datastore.DatastoreV1.Entity>

withMockEstimateSizeBytes

public DatastoreIO.Source withMockEstimateSizeBytes(com.google.common.base.Supplier<java.lang.Long> estimateSizeBytes)

For testing only.