public static class DatastoreIO.Source extends Source<com.google.api.services.datastore.DatastoreV1.Entity>
Entity objects.Source.Reader<T>| Modifier and Type | Method and Description |
|---|---|
Source.Reader<com.google.api.services.datastore.DatastoreV1.Entity> |
createBasicReader(PipelineOptions pipelineOptions,
Coder<com.google.api.services.datastore.DatastoreV1.Entity> coder,
com.google.cloud.dataflow.sdk.util.ExecutionContext executionContext)
Creates a basic (non-windowed) reader for this source.
|
Coder<com.google.api.services.datastore.DatastoreV1.Entity> |
getDefaultOutputCoder()
Returns the default
Coder to use for the data read from this source. |
long |
getEstimatedSizeBytes(PipelineOptions options)
An estimate of the total size (in bytes) of the data that would be read from this source.
|
boolean |
producesSortedKeys(PipelineOptions options)
Whether this source is known to produce key/value pairs with the (encoded) keys in
lexicographically sorted order.
|
java.util.List<DatastoreIO.Source> |
splitIntoShards(long desiredShardSizeBytes,
PipelineOptions options)
Splits the source into shards.
|
void |
validate()
Checks that this source is valid, before it can be used into a pipeline.
|
DatastoreIO.Source |
withDataset(java.lang.String datasetId) |
DatastoreIO.Source |
withHost(java.lang.String host) |
DatastoreIO.Source |
withMockEstimateSizeBytes(com.google.common.base.Supplier<java.lang.Long> estimateSizeBytes)
For testing only.
|
DatastoreIO.Source |
withQuery(com.google.api.services.datastore.DatastoreV1.Query query) |
createWindowedReaderpublic DatastoreIO.Source withDataset(java.lang.String datasetId)
public DatastoreIO.Source withQuery(com.google.api.services.datastore.DatastoreV1.Query query)
public DatastoreIO.Source withHost(java.lang.String host)
public Coder<com.google.api.services.datastore.DatastoreV1.Entity> getDefaultOutputCoder()
SourceCoder to use for the data read from this source.getDefaultOutputCoder in class Source<com.google.api.services.datastore.DatastoreV1.Entity>public long getEstimatedSizeBytes(PipelineOptions options) throws java.lang.Exception
SourcegetEstimatedSizeBytes in class Source<com.google.api.services.datastore.DatastoreV1.Entity>java.lang.Exceptionpublic boolean producesSortedKeys(PipelineOptions options)
SourceproducesSortedKeys in class Source<com.google.api.services.datastore.DatastoreV1.Entity>public java.util.List<DatastoreIO.Source> splitIntoShards(long desiredShardSizeBytes, PipelineOptions options) throws java.lang.Exception
Source PipelineOptions can be used to get information such as
credentials for accessing an external storage.
splitIntoShards in class Source<com.google.api.services.datastore.DatastoreV1.Entity>java.lang.Exceptionpublic Source.Reader<com.google.api.services.datastore.DatastoreV1.Entity> createBasicReader(PipelineOptions pipelineOptions, Coder<com.google.api.services.datastore.DatastoreV1.Entity> coder, com.google.cloud.dataflow.sdk.util.ExecutionContext executionContext) throws java.io.IOException
SourcecreateBasicReader in class Source<com.google.api.services.datastore.DatastoreV1.Entity>java.io.IOExceptionpublic void validate()
SourcePreconditions for implementing
this method.public DatastoreIO.Source withMockEstimateSizeBytes(com.google.common.base.Supplier<java.lang.Long> estimateSizeBytes)