Class DatastoreV1.Read
- java.lang.Object
-
- org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PBegin,org.apache.beam.sdk.values.PCollection<com.google.datastore.v1.Entity>>
-
- org.apache.beam.sdk.io.gcp.datastore.DatastoreV1.Read
-
- All Implemented Interfaces:
java.io.Serializable
,org.apache.beam.sdk.transforms.display.HasDisplayData
- Enclosing class:
- DatastoreV1
public abstract static class DatastoreV1.Read extends org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PBegin,org.apache.beam.sdk.values.PCollection<com.google.datastore.v1.Entity>>
APTransform
that reads the result rows of a Cloud Datastore query asEntity
objects.- See Also:
DatastoreIO
, Serialized Form
-
-
Field Summary
Fields Modifier and Type Field Description static int
NUM_QUERY_SPLITS_MAX
An upper bound on the number of splits for a query.
-
Constructor Summary
Constructors Constructor Description Read()
-
Method Summary
All Methods Instance Methods Abstract Methods Concrete Methods Modifier and Type Method Description org.apache.beam.sdk.values.PCollection<com.google.datastore.v1.Entity>
expand(org.apache.beam.sdk.values.PBegin input)
abstract @Nullable org.apache.beam.sdk.options.ValueProvider<java.lang.String>
getLiteralGqlQuery()
abstract @Nullable java.lang.String
getLocalhost()
abstract @Nullable org.apache.beam.sdk.options.ValueProvider<java.lang.String>
getNamespace()
long
getNumEntities(org.apache.beam.sdk.options.PipelineOptions options, java.lang.String ourKind, @Nullable java.lang.String namespace)
Returns Number of entities available for reading.abstract int
getNumQuerySplits()
abstract @Nullable org.apache.beam.sdk.options.ValueProvider<java.lang.String>
getProjectId()
abstract @Nullable com.google.datastore.v1.Query
getQuery()
abstract @Nullable org.joda.time.Instant
getReadTime()
void
populateDisplayData(org.apache.beam.sdk.transforms.display.DisplayData.Builder builder)
abstract java.lang.String
toString()
DatastoreV1.Read
withLiteralGqlQuery(java.lang.String gqlQuery)
Returns a newDatastoreV1.Read
that reads the results of the specified GQL query.DatastoreV1.Read
withLiteralGqlQuery(org.apache.beam.sdk.options.ValueProvider<java.lang.String> gqlQuery)
Same aswithLiteralGqlQuery(String)
but with aValueProvider
.DatastoreV1.Read
withLocalhost(java.lang.String localhost)
Returns a newDatastoreV1.Read
that reads from a Datastore Emulator running at the given localhost address.DatastoreV1.Read
withNamespace(java.lang.String namespace)
Returns a newDatastoreV1.Read
that reads from the given namespace.DatastoreV1.Read
withNamespace(org.apache.beam.sdk.options.ValueProvider<java.lang.String> namespace)
Same aswithNamespace(String)
but with aValueProvider
.DatastoreV1.Read
withNumQuerySplits(int numQuerySplits)
DatastoreV1.Read
withProjectId(java.lang.String projectId)
Returns a newDatastoreV1.Read
that reads from the Cloud Datastore for the specified project.DatastoreV1.Read
withProjectId(org.apache.beam.sdk.options.ValueProvider<java.lang.String> projectId)
Same aswithProjectId(String)
but with aValueProvider
.DatastoreV1.Read
withQuery(com.google.datastore.v1.Query query)
Returns a newDatastoreV1.Read
that reads the results of the specified query.DatastoreV1.Read
withReadTime(org.joda.time.Instant readTime)
Returns a newDatastoreV1.Read
that reads at the specifiedreadTime
.
-
-
-
Field Detail
-
NUM_QUERY_SPLITS_MAX
public static final int NUM_QUERY_SPLITS_MAX
An upper bound on the number of splits for a query.- See Also:
- Constant Field Values
-
-
Method Detail
-
getProjectId
public abstract @Nullable org.apache.beam.sdk.options.ValueProvider<java.lang.String> getProjectId()
-
getQuery
public abstract @Nullable com.google.datastore.v1.Query getQuery()
-
getLiteralGqlQuery
public abstract @Nullable org.apache.beam.sdk.options.ValueProvider<java.lang.String> getLiteralGqlQuery()
-
getNamespace
public abstract @Nullable org.apache.beam.sdk.options.ValueProvider<java.lang.String> getNamespace()
-
getNumQuerySplits
public abstract int getNumQuerySplits()
-
getLocalhost
public abstract @Nullable java.lang.String getLocalhost()
-
getReadTime
public abstract @Nullable org.joda.time.Instant getReadTime()
-
toString
public abstract java.lang.String toString()
- Overrides:
toString
in classorg.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PBegin,org.apache.beam.sdk.values.PCollection<com.google.datastore.v1.Entity>>
-
withProjectId
public DatastoreV1.Read withProjectId(java.lang.String projectId)
Returns a newDatastoreV1.Read
that reads from the Cloud Datastore for the specified project.
-
withProjectId
public DatastoreV1.Read withProjectId(org.apache.beam.sdk.options.ValueProvider<java.lang.String> projectId)
Same aswithProjectId(String)
but with aValueProvider
.
-
withQuery
public DatastoreV1.Read withQuery(com.google.datastore.v1.Query query)
Returns a newDatastoreV1.Read
that reads the results of the specified query.Note: Normally,
DatastoreIO
will read from Cloud Datastore in parallel across many workers. However, when theQuery
is configured with a limit usingQuery.Builder.setLimit(com.google.protobuf.Int32Value)
, then all results will be read by a single worker in order to ensure correct results.
-
withLiteralGqlQuery
public DatastoreV1.Read withLiteralGqlQuery(java.lang.String gqlQuery)
Returns a newDatastoreV1.Read
that reads the results of the specified GQL query. See GQL Reference to know more about GQL grammar.Note: This query is executed with literals allowed, so the users should ensure that the query is originated from trusted sources to avoid any security vulnerabilities via SQL Injection.
Cloud Datastore does not a provide a clean way to translate a gql query string to
Query
, so we end up making a query to the service for translation but this may read the actual data, although it will be a small amount. It needs more validation through production use cases before marking it as stable.
-
withLiteralGqlQuery
public DatastoreV1.Read withLiteralGqlQuery(org.apache.beam.sdk.options.ValueProvider<java.lang.String> gqlQuery)
Same aswithLiteralGqlQuery(String)
but with aValueProvider
.
-
withNamespace
public DatastoreV1.Read withNamespace(java.lang.String namespace)
Returns a newDatastoreV1.Read
that reads from the given namespace.
-
withNamespace
public DatastoreV1.Read withNamespace(org.apache.beam.sdk.options.ValueProvider<java.lang.String> namespace)
Same aswithNamespace(String)
but with aValueProvider
.
-
withNumQuerySplits
public DatastoreV1.Read withNumQuerySplits(int numQuerySplits)
Returns a newDatastoreV1.Read
that reads by splitting the givenquery
intonumQuerySplits
.The semantics for the query splitting is defined below:
- Any value less than or equal to 0 will be ignored, and the number of splits will be chosen dynamically at runtime based on the query data size.
- Any value greater than
NUM_QUERY_SPLITS_MAX
will be capped atNUM_QUERY_SPLITS_MAX
. - If the
query
has a user limit set, thennumQuerySplits
will be ignored and no split will be performed. - Under certain cases Cloud Datastore is unable to split query to the requested number of splits. In such cases we just use whatever the Cloud Datastore returns.
-
withLocalhost
public DatastoreV1.Read withLocalhost(java.lang.String localhost)
Returns a newDatastoreV1.Read
that reads from a Datastore Emulator running at the given localhost address.
-
withReadTime
public DatastoreV1.Read withReadTime(org.joda.time.Instant readTime)
Returns a newDatastoreV1.Read
that reads at the specifiedreadTime
.
-
getNumEntities
public long getNumEntities(org.apache.beam.sdk.options.PipelineOptions options, java.lang.String ourKind, @Nullable java.lang.String namespace)
Returns Number of entities available for reading.
-
expand
public org.apache.beam.sdk.values.PCollection<com.google.datastore.v1.Entity> expand(org.apache.beam.sdk.values.PBegin input)
- Specified by:
expand
in classorg.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PBegin,org.apache.beam.sdk.values.PCollection<com.google.datastore.v1.Entity>>
-
populateDisplayData
public void populateDisplayData(org.apache.beam.sdk.transforms.display.DisplayData.Builder builder)
- Specified by:
populateDisplayData
in interfaceorg.apache.beam.sdk.transforms.display.HasDisplayData
- Overrides:
populateDisplayData
in classorg.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PBegin,org.apache.beam.sdk.values.PCollection<com.google.datastore.v1.Entity>>
-
-