com.google.cloud.dataflow.sdk.io

Class BigQueryIO.Write

java.lang.Object

com.google.cloud.dataflow.sdk.io.BigQueryIO.Write

Enclosing class:

BigQueryIO

public static class BigQueryIO.Write
extends java.lang.Object

A PTransform that writes a PCollection<TableRow> containing rows to a BigQuery table.

By default, tables will be created if they do not exist, which corresponds to a CreateDisposition.CREATE_IF_NEEDED disposition which matches the default of BigQuery's Jobs API. A schema must be provided (via withSchema(com.google.api.services.bigquery.model.TableSchema)), or else the transform may fail at runtime with an IllegalArgumentException.

The dataset being written must already exist.

By default, writes require an empty table, which corresponds to a WriteDisposition.WRITE_EMPTY disposition which matches the default of BigQuery's Jobs API.

Here is a sample transform which produces TableRow values containing "word" and "count" columns:

 static class FormatCountsFn extends DoFnP<KV<String, Long>, TableRow> {
   @Override
   public void processElement(ProcessContext c) {
     TableRow row = new TableRow()
         .set("word", c.element().getKey())
         .set("count", c.element().getValue().intValue());
     c.output(row);
   }
 }
 

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	BigQueryIO.Write.Bound A PTransform that can write either a bounded or unbounded PCollection<TableRow>s to a BigQuery table.
static class	BigQueryIO.Write.CreateDisposition An enumeration type for the BigQuery create disposition strings publicly documented as CREATE_NEVER, and CREATE_IF_NEEDED.
static class	BigQueryIO.Write.WriteDisposition An enumeration type for the BigQuery write disposition strings publicly documented as WRITE_TRUNCATE, WRITE_APPEND, and WRITE_EMPTY.

Constructor Summary

Constructors

Constructor and Description
BigQueryIO.Write()

Method Summary

Methods

Modifier and Type	Method and Description
static BigQueryIO.Write.Bound	named(java.lang.String name) Sets the name associated with this transformation.
static BigQueryIO.Write.Bound	to(java.lang.String tableSpec) Creates a write transformation for the given table specification.
static BigQueryIO.Write.Bound	to(TableReference table) Creates a write transformation for the given table.
static BigQueryIO.Write.Bound	withCreateDisposition(BigQueryIO.Write.CreateDisposition disposition) Specifies options for creating the table.
static BigQueryIO.Write.Bound	withoutValidation() Disables BigQuery table validation which is enabled by default.
static BigQueryIO.Write.Bound	withSchema(TableSchema schema) Specifies a table schema to use in table creation.
static BigQueryIO.Write.Bound	withWriteDisposition(BigQueryIO.Write.WriteDisposition disposition) Specifies options for writing to the table.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BigQueryIO.Write

public BigQueryIO.Write()

Method Detail

named

public static BigQueryIO.Write.Bound named(java.lang.String name)

Sets the name associated with this transformation.

to

public static BigQueryIO.Write.Bound to(java.lang.String tableSpec)

Creates a write transformation for the given table specification.

Refer to BigQueryIO.parseTableSpec(String) for the specification format.

to

public static BigQueryIO.Write.Bound to(TableReference table)

Creates a write transformation for the given table.

withSchema

public static BigQueryIO.Write.Bound withSchema(TableSchema schema)

Specifies a table schema to use in table creation.

The schema is required only if writing to a table which does not already exist, and BigQueryIO.Write.CreateDisposition is set to CREATE_IF_NEEDED.

withCreateDisposition

public static BigQueryIO.Write.Bound withCreateDisposition(BigQueryIO.Write.CreateDisposition disposition)

Specifies options for creating the table.

withWriteDisposition

public static BigQueryIO.Write.Bound withWriteDisposition(BigQueryIO.Write.WriteDisposition disposition)

Specifies options for writing to the table.

withoutValidation

public static BigQueryIO.Write.Bound withoutValidation()

Disables BigQuery table validation which is enabled by default.