com.google.cloud.bigquery.storage.v1alpha2

Class StreamWriter

java.lang.Object

com.google.cloud.bigquery.storage.v1alpha2.StreamWriter

All Implemented Interfaces:

AutoCloseable

public class StreamWriter
extends Object
implements AutoCloseable

A BigQuery Stream Writer that can be used to write data into BigQuery Table.

A StreamWrier provides built-in capabilities to: handle batching of messages; controlling memory utilization (through flow control); automatic connection re-establishment and request cleanup (only keeps write schema on first request in the stream).

With customizable options that control:

• Message batching: such as number of messages or max batch byte size, and batching deadline
• Inflight message control: such as number of messages or max batch byte size

StreamWriter will use the credentials set on the channel, which uses application default credentials through GoogleCredentials.getApplicationDefault() by default.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	StreamWriter.Builder A builder of Publishers.

Method Summary

Modifier and Type	Method and Description
com.google.api.core.ApiFuture<Storage.AppendRowsResponse>	append(Storage.AppendRowsRequest message) Schedules the writing of a message.
boolean	awaitTermination(long duration, TimeUnit unit) Wait for all work has completed execution after a shutdown() request, or the timeout occurs, or the current thread is interrupted.
void	close() Close the stream writer.
static long	getApiMaxInflightRequests() The maximum size of in flight requests.
static long	getApiMaxRequestBytes() The maximum size of one request.
com.google.api.gax.batching.BatchingSettings	getBatchingSettings() The batching settings configured on this StreamWriter.
com.google.api.gax.retrying.RetrySettings	getRetrySettings() The retry settings configured on this StreamWriter.
String	getStreamNameString() Stream name we are writing to.
static StreamWriter.Builder	newBuilder(String streamName) Constructs a new StreamWriter.Builder using the given topic.
void	shutdown() Schedules immediate flush of any outstanding messages and waits until all are processed.
void	writeAllOutstanding() Write any outstanding batches if non-empty.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

getApiMaxRequestBytes

public static long getApiMaxRequestBytes()

The maximum size of one request. Defined by the API.

getApiMaxInflightRequests

public static long getApiMaxInflightRequests()

The maximum size of in flight requests. Defined by the API.

getStreamNameString

public String getStreamNameString()

Stream name we are writing to.

append

public com.google.api.core.ApiFuture<Storage.AppendRowsResponse> append(Storage.AppendRowsRequest message)

Schedules the writing of a message. The write of the message may occur immediately or be delayed based on the writer batching options.

Example of writing a message.

 AppendRowsRequest message;
 ApiFuture<AppendRowsResponse> messageIdFuture = writer.append(message);
 ApiFutures.addCallback(messageIdFuture, new ApiFutureCallback<AppendRowsResponse>() {
   public void onSuccess(AppendRowsResponse response) {
     if (response.hasOffset()) {
       System.out.println("written with offset: " + response.getOffset());
     } else {
       System.out.println("received an in stream error: " + response.error().toString());
     }
   }

   public void onFailure(Throwable t) {
     System.out.println("failed to write: " + t);
   }
 }, MoreExecutors.directExecutor());
 

Parameters:

message - the message in serialized format to write to BigQuery.

Returns:

the message ID wrapped in a future.

writeAllOutstanding

public void writeAllOutstanding()

Write any outstanding batches if non-empty. This method sends buffered messages, but does not wait for the send operations to complete. To wait for messages to send, call get on the futures returned from append.

close

public void close()

Close the stream writer. Shut down all resources.

Specified by:

close in interface AutoCloseable

getBatchingSettings

public com.google.api.gax.batching.BatchingSettings getBatchingSettings()

The batching settings configured on this StreamWriter.

getRetrySettings

public com.google.api.gax.retrying.RetrySettings getRetrySettings()

The retry settings configured on this StreamWriter.

shutdown

public void shutdown()

Schedules immediate flush of any outstanding messages and waits until all are processed.

Sends remaining outstanding messages and prevents future calls to publish. This method should be invoked prior to deleting the WriteStream object in order to ensure that no pending messages are lost.

awaitTermination

public boolean awaitTermination(long duration,
                                TimeUnit unit)
                         throws InterruptedException

Wait for all work has completed execution after a shutdown() request, or the timeout occurs, or the current thread is interrupted.

Call this method to make sure all resources are freed properly.

Throws:

InterruptedException

newBuilder

public static StreamWriter.Builder newBuilder(String streamName)

Constructs a new StreamWriter.Builder using the given topic.

Example of creating a WriteStream.

 String table = "projects/my_project/datasets/my_dataset/tables/my_table";
 String stream;
 try (BigQueryWriteClient bigqueryWriteClient = BigQueryWriteClient.create()) {
     CreateWriteStreamRequest request = CreateWriteStreamRequest.newBuilder().setParent(table).build();
     WriteStream response = bigQueryWriteClient.createWriteStream(request);
     stream = response.getName();
 }
 WriteStream writer = WriteStream.newBuilder(stream).build();
 try {
   // ...
 } finally {
   // When finished with the writer, make sure to shutdown to free up resources.
   writer.shutdown();
   writer.awaitTermination(1, TimeUnit.MINUTES);
 }