Package org.apache.beam.sdk.io.gcp.bigquery

Defines transforms for reading and writing from Google BigQuery.

See Also:

BigQueryIO

Interface Summary

Interface	Description
BigQueryOptions	Properties needed when using Google BigQuery with the Apache Beam SDK.
BigQueryServices	An interface for real, mock, or fake implementations of Cloud BigQuery services.
BigQueryServices.BigQueryServerStream<T>	Container for reading data from streaming endpoints.
BigQueryServices.DatasetService	An interface to get, create and delete Cloud BigQuery datasets and tables.
BigQueryServices.JobService	An interface for the Cloud BigQuery load service.
BigQueryServices.StorageClient	An interface representing a client object for making calls to the BigQuery Storage API.
BigQueryServices.StreamAppendClient	An interface for appending records to a Storage API write stream.
ErrorContainer<T>	ErrorContainer interface.
TestBigQuery.PollingAssertion	Interface to implement a polling assertion.
TestBigQueryOptions	TestPipelineOptions for TestBigQuery.

Class Summary

Class	Description
AvroWriteRequest<T>
BeamRowToStorageApiProto	Utility methods for converting Beam Row objects to dynamic protocol message, for use with the Storage write API.
BigQueryCoderProviderRegistrar	A CoderProviderRegistrar for standard types used with BigQueryIO.
BigQueryDlqProvider
BigQueryHelpers	A set of helper functions and classes used by BigQueryIO.
BigQueryInsertError	Model definition for BigQueryInsertError.
BigQueryInsertErrorCoder	A Coder that encodes BigQuery BigQueryInsertError objects.
BigQueryIO	PTransforms for reading and writing BigQuery tables.
BigQueryIO.Read	Implementation of BigQueryIO.read().
BigQueryIO.TypedRead<T>	Implementation of BigQueryIO.read(SerializableFunction).
BigQueryIO.Write<T>	Implementation of BigQueryIO.write().
BigQuerySchemaIOProvider	An implementation of SchemaIOProvider for reading and writing to BigQuery with BigQueryIO.
BigQuerySchemaTransformReadConfiguration	Configuration for reading from BigQuery.
BigQuerySchemaTransformReadConfiguration.Builder
BigQuerySchemaTransformReadProvider	An implementation of TypedSchemaTransformProvider for BigQuery read jobs configured using BigQuerySchemaTransformReadConfiguration.
BigQuerySchemaTransformWriteConfiguration	Configuration for writing to BigQuery.
BigQuerySchemaTransformWriteConfiguration.Builder
BigQuerySchemaTransformWriteProvider	An implementation of TypedSchemaTransformProvider for BigQuery write jobs configured using BigQuerySchemaTransformWriteConfiguration.
BigQueryStorageApiInsertError
BigQueryStorageApiInsertErrorCoder
BigQueryStorageTableSource<T>	A Source representing reading from a table.
BigQueryUtils	Utility methods for BigQuery related operations.
BigQueryUtils.ConversionOptions	Options for how to convert BigQuery data to Beam data.
BigQueryUtils.ConversionOptions.Builder	Builder for BigQueryUtils.ConversionOptions.
BigQueryUtils.SchemaConversionOptions	Options for how to convert BigQuery schemas to Beam schemas.
BigQueryUtils.SchemaConversionOptions.Builder	Builder for BigQueryUtils.SchemaConversionOptions.
CivilTimeEncoder	Encoder for TIME and DATETIME values, according to civil_time encoding.
CreateTableDestinations<DestinationT,ElementT>	Creates any tables needed before performing writes to the tables.
CreateTableHelpers
CreateTables<DestinationT,ElementT>	Creates any tables needed before performing streaming writes to the tables.
DynamicDestinations<T,DestinationT>	This class provides the most general way of specifying dynamic BigQuery table destinations.
InsertRetryPolicy	A retry policy for streaming BigQuery inserts.
InsertRetryPolicy.Context	Contains information about a failed insert.
PrepareWrite<InputT,DestinationT extends @NonNull java.lang.Object,OutputT>	Prepare an input PCollection for writing to BigQuery.
ReifyAsIterable<T>	This transforms turns a side input into a singleton PCollection that can be used as the main input for another transform.
SchemaAndRecord	A wrapper for a GenericRecord and the TableSchema representing the schema of the table (or query) it was generated from.
StorageApiConvertMessages<DestinationT,ElementT>	A transform that converts messages to protocol buffers in preparation for writing to BigQuery.
StorageApiConvertMessages.ConvertMessagesDoFn<DestinationT extends @NonNull java.lang.Object,ElementT>
StorageApiDynamicDestinationsTableRow<T,DestinationT extends @NonNull java.lang.Object>
StorageApiFlushAndFinalizeDoFn	This DoFn flushes and optionally (if requested) finalizes Storage API streams.
StorageApiLoads<DestinationT,ElementT>	This PTransform manages loads into BigQuery using the Storage API.
StorageApiWritePayload	Class used to wrap elements being sent to the Storage API sinks.
StorageApiWriteRecordsInconsistent<DestinationT,ElementT>	A transform to write sharded records to BigQuery using the Storage API.
StorageApiWritesShardedRecords<DestinationT extends @NonNull java.lang.Object,ElementT>	A transform to write sharded records to BigQuery using the Storage API.
StorageApiWriteUnshardedRecords<DestinationT,ElementT>	Write records to the Storage API using a standard batch approach.
StreamingInserts<DestinationT,ElementT>	PTransform that performs streaming BigQuery write.
StreamingWriteTables<ElementT>	This transform takes in key-value pairs of TableRow entries and the TableDestination it should be written to.
TableDestination	Encapsulates a BigQuery table destination.
TableDestinationCoder	A coder for TableDestination objects.
TableDestinationCoderV2	A Coder for TableDestination that includes time partitioning information.
TableDestinationCoderV3	A Coder for TableDestination that includes time partitioning and clustering information.
TableRowJsonCoder	A Coder that encodes BigQuery TableRow objects in their native JSON format.
TableRowToStorageApiProto	Utility methods for converting JSON TableRow objects to dynamic protocol message, for use with the Storage write API.
TableSchemaCache	An updatable cache for table schemas.
TestBigQuery	Test rule which creates a new table with specified schema, with randomized name and exposes few APIs to work with it.
UpdateSchemaDestination<DestinationT>
WriteResult	The result of a BigQueryIO.Write transform.

Enum Summary

Enum	Description
BigQueryIO.TypedRead.Method	Determines the method used to read data from BigQuery.
BigQueryIO.TypedRead.QueryPriority	An enumeration type for the priority of a query.
BigQueryIO.Write.CreateDisposition	An enumeration type for the BigQuery create disposition strings.
BigQueryIO.Write.Method	Determines the method used to insert data in BigQuery.
BigQueryIO.Write.SchemaUpdateOption	An enumeration type for the BigQuery schema update options strings.
BigQueryIO.Write.WriteDisposition	An enumeration type for the BigQuery write disposition strings.
BigQueryServices.DatasetService.TableMetadataView
BigQueryUtils.ConversionOptions.TruncateTimestamps	Controls whether to truncate timestamps to millisecond precision lossily, or to crash when truncation would result.

Exception Summary

Exception	Description
BigQuerySchemaRetrievalException	Exception to signal that BigQuery schema retrieval failed.
TableRowToStorageApiProto.SchemaConversionException
TableRowToStorageApiProto.SchemaDoesntMatchException
TableRowToStorageApiProto.SchemaTooNarrowException