com.google.cloud.dataflow.sdk.io

Class Sink.Writer<T,WriteT>

java.lang.Object

com.google.cloud.dataflow.sdk.io.Sink.Writer<T,WriteT>

Type Parameters:

T - The type of object to write

WriteT - The writer results type (e.g., the bundle's output filename, as String)

Direct Known Subclasses:

FileBasedSink.FileBasedWriter

Enclosing class:

Sink<T>

public abstract static class Sink.Writer<T,WriteT>
extends Object

A Writer writes a bundle of elements from a PCollection to a sink. open(java.lang.String) is called before writing begins and close() is called after all elements in the bundle have been written. write(T) writes an element to the sink.

Note that any access to static members or methods of a Writer must be thread-safe, as multiple instances of a Writer may be instantiated in different threads on the same worker.

See Sink for more detailed documentation about the process of writing to a Sink.

Constructor Summary

Constructors

Constructor and Description
Writer()

Method Summary

Modifier and Type	Method and Description
abstract WriteT	close() Finishes writing the bundle.
abstract Sink.WriteOperation<T,WriteT>	getWriteOperation() Returns the write operation this writer belongs to.
abstract void	open(String uId) Performs bundle initialization.
abstract void	write(T value) Called for each value in the bundle.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Writer

public Writer()

Method Detail

open

public abstract void open(String uId)
                   throws Exception

Performs bundle initialization. For example, creates a temporary file for writing or initializes any state that will be used across calls to write(T).

The unique id that is given to open should be used to ensure that the writer's output does not interfere with the output of other Writers, as a bundle may be executed many times for fault tolerance. See Sink for more information about bundle ids.

Throws:

Exception

write

public abstract void write(T value)
                    throws Exception

Called for each value in the bundle.

Throws:

Exception

close

public abstract WriteT close()
                      throws Exception

Finishes writing the bundle. Closes any resources used for writing the bundle.

Returns a writer result that will be used in the Sink.WriteOperation's finalization. The result should contain some way to identify the output of this bundle (using the bundle id). Sink.WriteOperation.finalize(java.lang.Iterable<WriteT>, com.google.cloud.dataflow.sdk.options.PipelineOptions) will use the writer result to identify successful writes. See Sink for more information about bundle ids.

Returns:

the writer result

Throws:

Exception

getWriteOperation

public abstract Sink.WriteOperation<T,WriteT> getWriteOperation()

Returns the write operation this writer belongs to.