com.google.cloud.dataflow.sdk.io

Class FileBasedSink.FileBasedWriter<T>

java.lang.Object

com.google.cloud.dataflow.sdk.io.Sink.Writer<T,FileBasedSink.FileResult>

com.google.cloud.dataflow.sdk.io.FileBasedSink.FileBasedWriter<T>

Type Parameters:

T - the type of values to write.

Direct Known Subclasses:

XmlSink.XmlWriter

Enclosing class:

FileBasedSink<T>

public abstract static class FileBasedSink.FileBasedWriter<T>
extends Sink.Writer<T,FileBasedSink.FileResult>

Abstract Sink.Writer that writes a bundle to a FileBasedSink. Subclass implementations provide a method that can write a single value to a WritableByteChannel (Sink.Writer.write(T)).

Subclass implementations may also override methods that write headers and footers before and after the values in a bundle, respectively, as well as provide a MIME type for the output channel.

Multiple FileBasedWriter instances may be created on the same worker, and therefore any access to static members or methods should be thread safe.

Field Summary

Fields

Modifier and Type	Field and Description
protected String	mimeType The MIME type used in the creation of the output channel (if the file system supports it).

Constructor Summary

Constructors

Constructor and Description
FileBasedSink.FileBasedWriter(FileBasedSink.FileBasedWriteOperation<T> writeOperation) Construct a new FileBasedWriter with a base filename.

Method Summary

Methods

Modifier and Type	Method and Description
FileBasedSink.FileResult	close() Closes the channel and return the bundle result.
FileBasedSink.FileBasedWriteOperation<T>	getWriteOperation() Return the FileBasedWriteOperation that this Writer belongs to.
void	open(String uId) Opens the channel.
protected abstract void	prepareWrite(WritableByteChannel channel) Called with the channel that a subclass will write its header, footer, and values to.
protected void	writeFooter() Writes footer at the end of output files.
protected void	writeHeader() Writes header at the beginning of output files.

Methods inherited from class com.google.cloud.dataflow.sdk.io.Sink.Writer

write

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

mimeType

protected String mimeType

The MIME type used in the creation of the output channel (if the file system supports it).

GCS, for example, supports writing files with Content-Type metadata.

May be overridden. Default is MimeTypes.TEXT. See MimeTypes for other options.

Constructor Detail

FileBasedSink.FileBasedWriter

public FileBasedSink.FileBasedWriter(FileBasedSink.FileBasedWriteOperation<T> writeOperation)

Construct a new FileBasedWriter with a base filename.

Method Detail

prepareWrite

protected abstract void prepareWrite(WritableByteChannel channel)
                              throws Exception

Called with the channel that a subclass will write its header, footer, and values to. Subclasses should either keep a reference to the channel provided or create and keep a reference to an appropriate object that they will use to write to it.

Called before any subsequent calls to writeHeader, writeFooter, and write.

Throws:

Exception

writeHeader

protected void writeHeader()
                    throws Exception

Writes header at the beginning of output files. Nothing by default; subclasses may override.

Throws:

Exception

writeFooter

protected void writeFooter()
                    throws Exception

Writes footer at the end of output files. Nothing by default; subclasses may override.

Throws:

Exception

open

public final void open(String uId)
                throws Exception

Opens the channel.

Specified by:

open in class Sink.Writer<T,FileBasedSink.FileResult>

Throws:

Exception

close

public final FileBasedSink.FileResult close()
                                     throws Exception

Closes the channel and return the bundle result.

Specified by:

close in class Sink.Writer<T,FileBasedSink.FileResult>

Returns:

the writer result

Throws:

Exception

getWriteOperation

public FileBasedSink.FileBasedWriteOperation<T> getWriteOperation()

Return the FileBasedWriteOperation that this Writer belongs to.

Specified by:

getWriteOperation in class Sink.Writer<T,FileBasedSink.FileResult>