com.google.cloud.dataflow.sdk.io

Class BigQueryIO

java.lang.Object

com.google.cloud.dataflow.sdk.io.BigQueryIO

public class BigQueryIO
extends java.lang.Object

Transformations for reading and writing BigQuery tables.

Table References

A fully-qualified BigQuery table name consists of three components:

• projectId: the Cloud project id (defaults to GcpOptions.getProject()).
• datasetId: the BigQuery dataset id, unique within a project.
• tableId: a table id, unique within a dataset.

BigQuery table references are stored as a TableReference, which comes from the BigQuery Java Client API. Tables can be referred to as Strings, with or without the projectId. A helper function is provided (parseTableSpec(String)), which parses the following string forms into a TableReference:

• [project_id]:[dataset_id].[table_id]
• [dataset_id].[table_id]

Reading

To read from a BigQuery table, apply a BigQueryIO.Read transformation. This produces a PCollection<TableRow> as output:

PCollection<TableRow> shakespeare = pipeline.apply(
     BigQueryIO.Read
         .named("Read")
         .from("clouddataflow-readonly:samples.weather_stations");
 

Writing

To write to a BigQuery table, apply a BigQueryIO.Write transformation. This consumes a PCollection<TableRow> as input.

PCollection<TableRow> quotes = ...

 List<TableFieldSchema> fields = new ArrayList<>();
 fields.add(new TableFieldSchema().setName("source").setType("STRING"));
 fields.add(new TableFieldSchema().setName("quote").setType("STRING"));
 TableSchema schema = new TableSchema().setFields(fields);

 quotes.apply(BigQueryIO.Write
     .named("Write")
     .to("my-project:output.output_table")
     .withSchema(schema)
     .withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_TRUNCATE));
 

See BigQueryIO.Write for details on how to specify if a write should append to an existing table, replace the table, or verify that the table is empty. Note that the dataset being written to must already exist.

See Also:

TableRow

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	BigQueryIO.Read A PTransform that reads from a BigQuery table and returns a PCollection<TableRow> containing each of the rows of the table.
static class	BigQueryIO.Write A PTransform that writes a PCollection<TableRow> containing rows to a BigQuery table.

Constructor Summary

Constructors

Constructor and Description
BigQueryIO()

Method Summary

Methods

Modifier and Type	Method and Description
static TableReference	parseTableSpec(java.lang.String tableSpec) Parse a table specification in the form "[project_id]:[dataset_id].[table_id]" or "[dataset_id].[table_id]".
static java.lang.String	toTableSpec(TableReference ref) Returns a canonical string representation of the TableReference.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BigQueryIO

public BigQueryIO()

Method Detail

parseTableSpec

public static TableReference parseTableSpec(java.lang.String tableSpec)

Parse a table specification in the form "[project_id]:[dataset_id].[table_id]" or "[dataset_id].[table_id]".

If the project id is omitted, the default project id is used.

toTableSpec

public static java.lang.String toTableSpec(TableReference ref)

Returns a canonical string representation of the TableReference.