public class BigQueryUtils
extends java.lang.Object
Modifier and Type | Class and Description |
---|---|
static class |
BigQueryUtils.ConversionOptions
Options for how to convert BigQuery data to Beam data.
|
static class |
BigQueryUtils.SchemaConversionOptions
Options for how to convert BigQuery schemas to Beam schemas.
|
Constructor and Description |
---|
BigQueryUtils() |
Modifier and Type | Method and Description |
---|---|
static java.lang.Object |
convertAvroFormat(org.apache.beam.sdk.schemas.Schema.FieldType beamFieldType,
java.lang.Object avroValue,
BigQueryUtils.ConversionOptions options)
Tries to convert an Avro decoded value to a Beam field value based on the target type of the
Beam field.
|
static com.google.api.services.bigquery.model.TableRow |
convertGenericRecordToTableRow(org.apache.avro.generic.GenericRecord record,
com.google.api.services.bigquery.model.TableSchema tableSchema) |
static org.apache.beam.sdk.schemas.Schema |
fromTableSchema(com.google.api.services.bigquery.model.TableSchema tableSchema)
Convert a BigQuery
TableSchema to a Beam Schema . |
static org.apache.beam.sdk.schemas.Schema |
fromTableSchema(com.google.api.services.bigquery.model.TableSchema tableSchema,
BigQueryUtils.SchemaConversionOptions options)
Convert a BigQuery
TableSchema to a Beam Schema . |
static @Nullable org.apache.beam.runners.core.metrics.ServiceCallMetric |
readCallMetric(@Nullable com.google.api.services.bigquery.model.TableReference tableReference) |
static org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.TypedRead.FromBeamRowFunction<com.google.api.services.bigquery.model.TableRow> |
tableRowFromBeamRow() |
static org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.TypedRead.ToBeamRowFunction<com.google.api.services.bigquery.model.TableRow> |
tableRowToBeamRow() |
static org.apache.beam.sdk.values.Row |
toBeamRow(org.apache.avro.generic.GenericRecord record,
org.apache.beam.sdk.schemas.Schema schema,
BigQueryUtils.ConversionOptions options) |
static org.apache.beam.sdk.values.Row |
toBeamRow(org.apache.beam.sdk.schemas.Schema rowSchema,
com.google.api.services.bigquery.model.TableRow jsonBqRow)
Tries to convert a JSON
TableRow from BigQuery into a Beam Row . |
static org.apache.beam.sdk.values.Row |
toBeamRow(org.apache.beam.sdk.schemas.Schema rowSchema,
com.google.api.services.bigquery.model.TableSchema bqSchema,
com.google.api.services.bigquery.model.TableRow jsonBqRow)
Tries to parse the JSON
TableRow from BigQuery. |
static org.apache.avro.Schema |
toGenericAvroSchema(java.lang.String schemaName,
java.util.List<com.google.api.services.bigquery.model.TableFieldSchema> fieldSchemas)
Convert a list of BigQuery
TableFieldSchema to Avro Schema . |
static @Nullable com.google.api.services.bigquery.model.TableReference |
toTableReference(java.lang.String fullTableId) |
static org.apache.beam.sdk.transforms.SerializableFunction<org.apache.beam.sdk.values.Row,com.google.api.services.bigquery.model.TableRow> |
toTableRow()
Convert a Beam
Row to a BigQuery TableRow . |
static com.google.api.services.bigquery.model.TableRow |
toTableRow(org.apache.beam.sdk.values.Row row)
Convert a Beam Row to a BigQuery TableRow.
|
static <T> org.apache.beam.sdk.transforms.SerializableFunction<T,com.google.api.services.bigquery.model.TableRow> |
toTableRow(org.apache.beam.sdk.transforms.SerializableFunction<T,org.apache.beam.sdk.values.Row> toRow)
Convert a Beam schema type to a BigQuery
TableRow . |
static com.google.api.services.bigquery.model.TableSchema |
toTableSchema(org.apache.beam.sdk.schemas.Schema schema)
Convert a Beam
Schema to a BigQuery TableSchema . |
static @Nullable java.lang.String |
toTableSpec(com.google.api.services.bigquery.model.TableReference tableReference) |
static org.apache.beam.runners.core.metrics.ServiceCallMetric |
writeCallMetric(com.google.api.services.bigquery.model.TableReference tableReference) |
public static com.google.api.services.bigquery.model.TableSchema toTableSchema(org.apache.beam.sdk.schemas.Schema schema)
Schema
to a BigQuery TableSchema
.public static org.apache.beam.sdk.schemas.Schema fromTableSchema(com.google.api.services.bigquery.model.TableSchema tableSchema)
TableSchema
to a Beam Schema
.public static org.apache.beam.sdk.schemas.Schema fromTableSchema(com.google.api.services.bigquery.model.TableSchema tableSchema, BigQueryUtils.SchemaConversionOptions options)
TableSchema
to a Beam Schema
.public static org.apache.avro.Schema toGenericAvroSchema(java.lang.String schemaName, java.util.List<com.google.api.services.bigquery.model.TableFieldSchema> fieldSchemas)
TableFieldSchema
to Avro Schema
.public static final org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.TypedRead.ToBeamRowFunction<com.google.api.services.bigquery.model.TableRow> tableRowToBeamRow()
public static final org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.TypedRead.FromBeamRowFunction<com.google.api.services.bigquery.model.TableRow> tableRowFromBeamRow()
public static org.apache.beam.sdk.transforms.SerializableFunction<org.apache.beam.sdk.values.Row,com.google.api.services.bigquery.model.TableRow> toTableRow()
Row
to a BigQuery TableRow
.public static <T> org.apache.beam.sdk.transforms.SerializableFunction<T,com.google.api.services.bigquery.model.TableRow> toTableRow(org.apache.beam.sdk.transforms.SerializableFunction<T,org.apache.beam.sdk.values.Row> toRow)
TableRow
.public static org.apache.beam.sdk.values.Row toBeamRow(org.apache.avro.generic.GenericRecord record, org.apache.beam.sdk.schemas.Schema schema, BigQueryUtils.ConversionOptions options)
public static com.google.api.services.bigquery.model.TableRow convertGenericRecordToTableRow(org.apache.avro.generic.GenericRecord record, com.google.api.services.bigquery.model.TableSchema tableSchema)
public static com.google.api.services.bigquery.model.TableRow toTableRow(org.apache.beam.sdk.values.Row row)
public static org.apache.beam.sdk.values.Row toBeamRow(org.apache.beam.sdk.schemas.Schema rowSchema, com.google.api.services.bigquery.model.TableRow jsonBqRow)
TableRow
from BigQuery into a Beam Row
.
Only supports basic types and arrays. Doesn't support date types or structs.
public static org.apache.beam.sdk.values.Row toBeamRow(org.apache.beam.sdk.schemas.Schema rowSchema, com.google.api.services.bigquery.model.TableSchema bqSchema, com.google.api.services.bigquery.model.TableRow jsonBqRow)
TableRow
from BigQuery.
Only supports basic types and arrays. Doesn't support date types.
public static java.lang.Object convertAvroFormat(org.apache.beam.sdk.schemas.Schema.FieldType beamFieldType, java.lang.Object avroValue, BigQueryUtils.ConversionOptions options)
For the Avro formats of BigQuery types, see https://cloud.google.com/bigquery/docs/exporting-data#avro_export_details and https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-avro#avro_conversions
public static @Nullable com.google.api.services.bigquery.model.TableReference toTableReference(java.lang.String fullTableId)
fullTableId
- - Is one of the two forms commonly used to refer to bigquery tables in the
beam codebase:
public static @Nullable java.lang.String toTableSpec(com.google.api.services.bigquery.model.TableReference tableReference)
tableReference
- - a BigQueryTableIdentifier that may or may not include the project.public static @Nullable org.apache.beam.runners.core.metrics.ServiceCallMetric readCallMetric(@Nullable com.google.api.services.bigquery.model.TableReference tableReference)
tableReference
- - The table being read from. Can be a temporary BQ table used to read
from a SQL query.public static org.apache.beam.runners.core.metrics.ServiceCallMetric writeCallMetric(com.google.api.services.bigquery.model.TableReference tableReference)
tableReference
- - The table being written to.