Class BigQueryIO.TypedRead<T>
- java.lang.Object
-
- org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PBegin,org.apache.beam.sdk.values.PCollection<T>>
-
- org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.TypedRead<T>
-
- All Implemented Interfaces:
java.io.Serializable
,org.apache.beam.sdk.schemas.ProjectionProducer<org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PBegin,org.apache.beam.sdk.values.PCollection<T>>>
,org.apache.beam.sdk.transforms.display.HasDisplayData
- Enclosing class:
- BigQueryIO
public abstract static class BigQueryIO.TypedRead<T> extends org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PBegin,org.apache.beam.sdk.values.PCollection<T>> implements org.apache.beam.sdk.schemas.ProjectionProducer<org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PBegin,org.apache.beam.sdk.values.PCollection<T>>>
Implementation ofBigQueryIO.read(SerializableFunction)
.- See Also:
- Serialized Form
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description static class
BigQueryIO.TypedRead.Method
Determines the method used to read data from BigQuery.static class
BigQueryIO.TypedRead.QueryPriority
An enumeration type for the priority of a query.
-
Constructor Summary
Constructors Constructor Description TypedRead()
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PBegin,org.apache.beam.sdk.values.PCollection<T>>
actuateProjectionPushdown(java.util.Map<org.apache.beam.sdk.values.TupleTag<?>,org.apache.beam.sdk.schemas.FieldAccessDescriptor> outputFields)
org.apache.beam.sdk.values.PCollection<T>
expand(org.apache.beam.sdk.values.PBegin input)
BigQueryIO.TypedRead<T>
from(com.google.api.services.bigquery.model.TableReference table)
BigQueryIO.TypedRead<T>
from(java.lang.String tableSpec)
BigQueryIO.TypedRead<T>
from(org.apache.beam.sdk.options.ValueProvider<java.lang.String> tableSpec)
BigQueryIO.TypedRead<T>
fromQuery(java.lang.String query)
BigQueryIO.TypedRead<T>
fromQuery(org.apache.beam.sdk.options.ValueProvider<java.lang.String> query)
@Nullable com.google.api.services.bigquery.model.TableReference
getTable()
@Nullable org.apache.beam.sdk.options.ValueProvider<com.google.api.services.bigquery.model.TableReference>
getTableProvider()
void
populateDisplayData(org.apache.beam.sdk.transforms.display.DisplayData.Builder builder)
boolean
supportsProjectionPushdown()
BigQueryIO.TypedRead<T>
useAvroLogicalTypes()
BigQueryIO.TypedRead<T>
usingStandardSql()
void
validate(org.apache.beam.sdk.options.PipelineOptions options)
BigQueryIO.TypedRead<T>
withBeamRowConverters(org.apache.beam.sdk.values.TypeDescriptor<T> typeDescriptor, org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.TypedRead.ToBeamRowFunction<T> toRowFn, org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.TypedRead.FromBeamRowFunction<T> fromRowFn)
Sets the functions to convert elements to/fromRow
objects.BigQueryIO.TypedRead<T>
withCoder(org.apache.beam.sdk.coders.Coder<T> coder)
Sets aCoder
for the result of the parse function.BigQueryIO.TypedRead<T>
withFormat(com.google.cloud.bigquery.storage.v1.DataFormat format)
SeeDataFormat
.BigQueryIO.TypedRead<T>
withKmsKey(java.lang.String kmsKey)
For query sources, use this Cloud KMS key to encrypt any temporary tables created.BigQueryIO.TypedRead<T>
withMethod(BigQueryIO.TypedRead.Method method)
BigQueryIO.TypedRead<T>
withoutResultFlattening()
BigQueryIO.TypedRead<T>
withoutValidation()
BigQueryIO.TypedRead<T>
withQueryLocation(java.lang.String location)
BigQuery geographic location where the query job will be executed.BigQueryIO.TypedRead<T>
withQueryPriority(BigQueryIO.TypedRead.QueryPriority priority)
BigQueryIO.TypedRead<T>
withQueryTempDataset(java.lang.String queryTempDatasetRef)
Temporary dataset reference when usingfromQuery(String)
.BigQueryIO.TypedRead<T>
withRowRestriction(java.lang.String rowRestriction)
BigQueryIO.TypedRead<T>
withRowRestriction(org.apache.beam.sdk.options.ValueProvider<java.lang.String> rowRestriction)
Read only rows which match the specified filter, which must be a SQL expression compatible with Google standard SQL.BigQueryIO.TypedRead<T>
withSelectedFields(java.util.List<java.lang.String> selectedFields)
BigQueryIO.TypedRead<T>
withSelectedFields(org.apache.beam.sdk.options.ValueProvider<java.util.List<java.lang.String>> selectedFields)
Read only the specified fields (columns) from a BigQuery table.BigQueryIO.TypedRead<T>
withTemplateCompatibility()
BigQueryIO.TypedRead<T>
withTestServices(BigQueryServices testServices)
-
-
-
Method Detail
-
validate
public void validate(org.apache.beam.sdk.options.PipelineOptions options)
- Overrides:
validate
in classorg.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PBegin,org.apache.beam.sdk.values.PCollection<T>>
-
expand
public org.apache.beam.sdk.values.PCollection<T> expand(org.apache.beam.sdk.values.PBegin input)
- Specified by:
expand
in classorg.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PBegin,org.apache.beam.sdk.values.PCollection<T>>
-
populateDisplayData
public void populateDisplayData(org.apache.beam.sdk.transforms.display.DisplayData.Builder builder)
- Specified by:
populateDisplayData
in interfaceorg.apache.beam.sdk.transforms.display.HasDisplayData
- Overrides:
populateDisplayData
in classorg.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PBegin,org.apache.beam.sdk.values.PCollection<T>>
-
getTableProvider
public @Nullable org.apache.beam.sdk.options.ValueProvider<com.google.api.services.bigquery.model.TableReference> getTableProvider()
-
getTable
public @Nullable com.google.api.services.bigquery.model.TableReference getTable()
-
withCoder
public BigQueryIO.TypedRead<T> withCoder(org.apache.beam.sdk.coders.Coder<T> coder)
Sets aCoder
for the result of the parse function. This may be required if a coder can not be inferred automatically.
-
withKmsKey
public BigQueryIO.TypedRead<T> withKmsKey(java.lang.String kmsKey)
For query sources, use this Cloud KMS key to encrypt any temporary tables created.
-
withBeamRowConverters
@Experimental(SCHEMAS) public BigQueryIO.TypedRead<T> withBeamRowConverters(org.apache.beam.sdk.values.TypeDescriptor<T> typeDescriptor, org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.TypedRead.ToBeamRowFunction<T> toRowFn, org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.TypedRead.FromBeamRowFunction<T> fromRowFn)
Sets the functions to convert elements to/fromRow
objects.Setting these conversion functions is necessary to enable
Schema
support.
-
from
public BigQueryIO.TypedRead<T> from(java.lang.String tableSpec)
-
from
public BigQueryIO.TypedRead<T> from(org.apache.beam.sdk.options.ValueProvider<java.lang.String> tableSpec)
-
fromQuery
public BigQueryIO.TypedRead<T> fromQuery(java.lang.String query)
-
fromQuery
public BigQueryIO.TypedRead<T> fromQuery(org.apache.beam.sdk.options.ValueProvider<java.lang.String> query)
-
from
public BigQueryIO.TypedRead<T> from(com.google.api.services.bigquery.model.TableReference table)
-
withoutValidation
public BigQueryIO.TypedRead<T> withoutValidation()
-
withoutResultFlattening
public BigQueryIO.TypedRead<T> withoutResultFlattening()
-
usingStandardSql
public BigQueryIO.TypedRead<T> usingStandardSql()
-
withQueryPriority
public BigQueryIO.TypedRead<T> withQueryPriority(BigQueryIO.TypedRead.QueryPriority priority)
-
withQueryLocation
public BigQueryIO.TypedRead<T> withQueryLocation(java.lang.String location)
BigQuery geographic location where the query job will be executed. If not specified, Beam tries to determine the location by examining the tables referenced by the query. Location must be specified for queries not executed in US or EU, or when you are reading from an authorized view. See BigQuery Jobs: query.
-
withQueryTempDataset
public BigQueryIO.TypedRead<T> withQueryTempDataset(java.lang.String queryTempDatasetRef)
Temporary dataset reference when usingfromQuery(String)
. When reading from a query, BigQuery will create a temporary dataset and a temporary table to store the results of the query. With this option, you can set an existing dataset to create the temporary table. BigQueryIO will create a temporary table in that dataset, and will remove it once it is not needed. No other tables in the dataset will be modified. If your job does not have permissions to create a new dataset, and you want to usefromQuery(String)
(for instance, to read from a view), you should use this option. Remember that the dataset must exist and your job needs permissions to create and remove tables inside that dataset.
-
withMethod
public BigQueryIO.TypedRead<T> withMethod(BigQueryIO.TypedRead.Method method)
-
withFormat
@Experimental(SOURCE_SINK) public BigQueryIO.TypedRead<T> withFormat(com.google.cloud.bigquery.storage.v1.DataFormat format)
SeeDataFormat
.
-
withSelectedFields
public BigQueryIO.TypedRead<T> withSelectedFields(java.util.List<java.lang.String> selectedFields)
-
withSelectedFields
public BigQueryIO.TypedRead<T> withSelectedFields(org.apache.beam.sdk.options.ValueProvider<java.util.List<java.lang.String>> selectedFields)
Read only the specified fields (columns) from a BigQuery table. Fields may not be returned in the order specified. If no value is specified, then all fields are returned.Requires
BigQueryIO.TypedRead.Method.DIRECT_READ
. Not compatible withfromQuery(String)
.
-
withRowRestriction
public BigQueryIO.TypedRead<T> withRowRestriction(java.lang.String rowRestriction)
-
withRowRestriction
public BigQueryIO.TypedRead<T> withRowRestriction(org.apache.beam.sdk.options.ValueProvider<java.lang.String> rowRestriction)
Read only rows which match the specified filter, which must be a SQL expression compatible with Google standard SQL. If no value is specified, then all rows are returned.Requires
BigQueryIO.TypedRead.Method.DIRECT_READ
. Not compatible withfromQuery(String)
.
-
withTemplateCompatibility
public BigQueryIO.TypedRead<T> withTemplateCompatibility()
-
withTestServices
public BigQueryIO.TypedRead<T> withTestServices(BigQueryServices testServices)
-
useAvroLogicalTypes
public BigQueryIO.TypedRead<T> useAvroLogicalTypes()
-
supportsProjectionPushdown
public boolean supportsProjectionPushdown()
- Specified by:
supportsProjectionPushdown
in interfaceorg.apache.beam.sdk.schemas.ProjectionProducer<T>
-
actuateProjectionPushdown
public org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PBegin,org.apache.beam.sdk.values.PCollection<T>> actuateProjectionPushdown(java.util.Map<org.apache.beam.sdk.values.TupleTag<?>,org.apache.beam.sdk.schemas.FieldAccessDescriptor> outputFields)
- Specified by:
actuateProjectionPushdown
in interfaceorg.apache.beam.sdk.schemas.ProjectionProducer<T>
-
-