public static class BigQueryIO.Write extends Object
PTransform that writes a PCollection containing TableRows
to a BigQuery table.
In BigQuery, each table has an encosing dataset. The dataset being written must already exist.
By default, tables will be created if they do not exist, which corresponds to a
BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED disposition that matches the default of BigQuery's
Jobs API. A schema must be provided (via withSchema(TableSchema)),
or else the transform may fail at runtime with an IllegalArgumentException.
By default, writes require an empty table, which corresponds to
a BigQueryIO.Write.WriteDisposition.WRITE_EMPTY disposition that matches the
default of BigQuery's Jobs API.
Here is a sample transform that produces TableRow values containing "word" and "count" columns:
static class FormatCountsFn extends DoFn<KV<String, Long>, TableRow> {
public void processElement(ProcessContext c) {
TableRow row = new TableRow()
.set("word", c.element().getKey())
.set("count", c.element().getValue().intValue());
c.output(row);
}
}| Modifier and Type | Class and Description |
|---|---|
static class |
BigQueryIO.Write.Bound
A
PTransform that can write either a bounded or unbounded
PCollection of TableRows to a BigQuery table. |
static class |
BigQueryIO.Write.CreateDisposition
An enumeration type for the BigQuery create disposition strings.
|
static class |
BigQueryIO.Write.WriteDisposition
An enumeration type for the BigQuery write disposition strings.
|
| Modifier and Type | Method and Description |
|---|---|
static BigQueryIO.Write.Bound |
named(String name)
Creates a write transformation with the given transform name.
|
static BigQueryIO.Write.Bound |
to(SerializableFunction<BoundedWindow,String> tableSpecFunction)
Creates a write transformation from a function that maps windows to table specifications.
|
static BigQueryIO.Write.Bound |
to(String tableSpec)
Creates a write transformation for the given table specification.
|
static BigQueryIO.Write.Bound |
to(TableReference table)
Creates a write transformation for the given table.
|
static BigQueryIO.Write.Bound |
to(ValueProvider<String> tableSpec)
Creates a write transformation for the given table.
|
static BigQueryIO.Write.Bound |
toTableReference(SerializableFunction<BoundedWindow,TableReference> tableRefFunction)
Creates a write transformation from a function that maps windows to
TableReference
objects. |
static BigQueryIO.Write.Bound |
withCreateDisposition(BigQueryIO.Write.CreateDisposition disposition)
Creates a write transformation with the specified options for creating the table.
|
static BigQueryIO.Write.Bound |
withoutValidation()
Creates a write transformation with BigQuery table validation disabled.
|
static BigQueryIO.Write.Bound |
withSchema(TableSchema schema)
Creates a write transformation with the specified schema to use in table creation.
|
static BigQueryIO.Write.Bound |
withSchema(ValueProvider<TableSchema> schema)
Like
withSchema(TableSchema), but with a ValueProvider. |
static BigQueryIO.Write.Bound |
withWriteDisposition(BigQueryIO.Write.WriteDisposition disposition)
Creates a write transformation with the specified options for writing to the table.
|
public static BigQueryIO.Write.Bound named(String name)
public static BigQueryIO.Write.Bound to(String tableSpec)
Refer to BigQueryIO.parseTableSpec(String) for the specification format.
public static BigQueryIO.Write.Bound to(ValueProvider<String> tableSpec)
public static BigQueryIO.Write.Bound to(TableReference table)
public static BigQueryIO.Write.Bound to(SerializableFunction<BoundedWindow,String> tableSpecFunction)
See BigQueryIO.parseTableSpec(String) for the format that tableSpecFunction should
return.
tableSpecFunction should be deterministic. When given the same window, it should
always return the same table specification.
public static BigQueryIO.Write.Bound toTableReference(SerializableFunction<BoundedWindow,TableReference> tableRefFunction)
TableReference
objects.
tableRefFunction should be deterministic. When given the same window, it should
always return the same table reference.
public static BigQueryIO.Write.Bound withSchema(TableSchema schema)
The schema is required only if writing to a table that does not already
exist, and BigQueryIO.Write.CreateDisposition is set to
BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED.
public static BigQueryIO.Write.Bound withSchema(ValueProvider<TableSchema> schema)
withSchema(TableSchema), but with a ValueProvider.public static BigQueryIO.Write.Bound withCreateDisposition(BigQueryIO.Write.CreateDisposition disposition)
public static BigQueryIO.Write.Bound withWriteDisposition(BigQueryIO.Write.WriteDisposition disposition)
public static BigQueryIO.Write.Bound withoutValidation()