com.google.cloud.dataflow.sdk.io

Class BigQueryIO.Write

java.lang.Object

com.google.cloud.dataflow.sdk.io.BigQueryIO.Write

Enclosing class:

BigQueryIO

public static class BigQueryIO.Write
extends Object

A PTransform that writes a PCollection containing TableRows to a BigQuery table.

By default, tables will be created if they do not exist, which corresponds to a CreateDisposition.CREATE_IF_NEEDED disposition that matches the default of BigQuery's Jobs API. A schema must be provided (via withSchema(com.google.api.services.bigquery.model.TableSchema)), or else the transform may fail at runtime with an IllegalArgumentException.

The dataset being written must already exist.

By default, writes require an empty table, which corresponds to a WriteDisposition.WRITE_EMPTY disposition that matches the default of BigQuery's Jobs API.

Here is a sample transform that produces TableRow values containing "word" and "count" columns:

 static class FormatCountsFn extends DoFnP <KV<String, Long>, TableRow> {
    @Override
   public void processElement(ProcessContext c) {
     TableRow row = new TableRow()
         .set("word", c.element().getKey())
         .set("count", c.element().getValue().intValue());
     c.output(row);
   }
 }
 

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	BigQueryIO.Write.Bound A PTransform that can write either a bounded or unbounded PCollection of TableRows to a BigQuery table.
static class	BigQueryIO.Write.CreateDisposition An enumeration type for the BigQuery create disposition strings publicly documented as CREATE_NEVER, and CREATE_IF_NEEDED.
static class	BigQueryIO.Write.WriteDisposition An enumeration type for the BigQuery write disposition strings publicly documented as WRITE_TRUNCATE, WRITE_APPEND, and WRITE_EMPTY.

Constructor Summary

Constructors

Constructor and Description
Write()

Method Summary

Modifier and Type	Method and Description
static BigQueryIO.Write.Bound	named(String name) Sets the name associated with this transformation.
static BigQueryIO.Write.Bound	to(SerializableFunction<BoundedWindow,String> tableSpecFunction) Creates a write transformation from a function that maps windows to table specifications.
static BigQueryIO.Write.Bound	to(String tableSpec) Creates a write transformation for the given table specification.
static BigQueryIO.Write.Bound	to(com.google.api.services.bigquery.model.TableReference table) Creates a write transformation for the given table.
static BigQueryIO.Write.Bound	toTableReference(SerializableFunction<BoundedWindow,com.google.api.services.bigquery.model.TableReference> tableRefFunction) Creates a write transformation from a function that maps windows to TableReference objects.
static BigQueryIO.Write.Bound	withCreateDisposition(BigQueryIO.Write.CreateDisposition disposition) Specifies options for creating the table.
static BigQueryIO.Write.Bound	withoutValidation() Disables BigQuery table validation, which is enabled by default.
static BigQueryIO.Write.Bound	withSchema(com.google.api.services.bigquery.model.TableSchema schema) Specifies a table schema to use in table creation.
static BigQueryIO.Write.Bound	withWriteDisposition(BigQueryIO.Write.WriteDisposition disposition) Specifies options for writing to the table.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Write

public Write()

Method Detail

named

public static BigQueryIO.Write.Bound named(String name)

Sets the name associated with this transformation.

to

public static BigQueryIO.Write.Bound to(String tableSpec)

Creates a write transformation for the given table specification.

Refer to BigQueryIO.parseTableSpec(String) for the specification format.

to

public static BigQueryIO.Write.Bound to(com.google.api.services.bigquery.model.TableReference table)

Creates a write transformation for the given table.

to

public static BigQueryIO.Write.Bound to(SerializableFunction<BoundedWindow,String> tableSpecFunction)

Creates a write transformation from a function that maps windows to table specifications. Each time a new window is encountered, this function will be called and the resulting table will be created. Records within that window will be written to the associated table.

See BigQueryIO.parseTableSpec(String) for the format that tableSpecFunction should return.

tableSpecFunction should be determinstic. When given the same window, it should always return the same table specification.

toTableReference

public static BigQueryIO.Write.Bound toTableReference(SerializableFunction<BoundedWindow,com.google.api.services.bigquery.model.TableReference> tableRefFunction)

Creates a write transformation from a function that maps windows to TableReference objects.

withSchema

public static BigQueryIO.Write.Bound withSchema(com.google.api.services.bigquery.model.TableSchema schema)

Specifies a table schema to use in table creation.

The schema is required only if writing to a table that does not already exist, and BigQueryIO.Write.CreateDisposition is set to CREATE_IF_NEEDED.

withCreateDisposition

public static BigQueryIO.Write.Bound withCreateDisposition(BigQueryIO.Write.CreateDisposition disposition)

Specifies options for creating the table.

withWriteDisposition

public static BigQueryIO.Write.Bound withWriteDisposition(BigQueryIO.Write.WriteDisposition disposition)

Specifies options for writing to the table.

withoutValidation

public static BigQueryIO.Write.Bound withoutValidation()

Disables BigQuery table validation, which is enabled by default.