com.google.cloud.dataflow.sdk.io

Class FileBasedSink<T>

java.lang.Object

com.google.cloud.dataflow.sdk.io.Sink<T>

com.google.cloud.dataflow.sdk.io.FileBasedSink<T>

Type Parameters:

T - the type of values written to the sink.

All Implemented Interfaces:

Serializable

Direct Known Subclasses:

XmlSink.Bound

public abstract class FileBasedSink<T>
extends Sink<T>

Abstract Sink for file-based output. An implementation of FileBasedSink writes file-based output and defines the format of output files (how values are written, headers/footers, MIME type, etc.).

At pipeline construction time, the methods of FileBasedSink are called to validate the sink and to create a Sink.WriteOperation that manages the process of writing to the sink.

The process of writing to file-based sink is as follows:

1. An optional subclass-defined initialization,
2. a parallel write of bundles to temporary files, and finally,
3. these temporary files are renamed with final output filenames.

Supported file systems are those registered with IOChannelUtils.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	FileBasedSink.FileBasedWriteOperation<T> Abstract Sink.WriteOperation that manages the process of writing to a FileBasedSink.
static class	FileBasedSink.FileBasedWriter<T> Abstract Sink.Writer that writes a bundle to a FileBasedSink.
static class	FileBasedSink.FileResult Result of a single bundle write.

Nested classes/interfaces inherited from class com.google.cloud.dataflow.sdk.io.Sink

Sink.WriteOperation<T,WriteT>, Sink.Writer<T,WriteT>

Field Summary

Fields

Modifier and Type	Field and Description
protected String	baseOutputFilename Base filename for final output files.
protected String	extension The extension to be used for the final output files.
protected String	fileNamingTemplate Naming template for output files.

Constructor Summary

Constructors

Constructor and Description
FileBasedSink(String baseOutputFilename, String extension) Construct a FileBasedSink with the given base output filename and extension.
FileBasedSink(String baseOutputFilename, String extension, String fileNamingTemplate) Construct a FileBasedSink with the given base output filename, extension, and file naming template.

Method Summary

Modifier and Type	Method and Description
abstract FileBasedSink.FileBasedWriteOperation<T>	createWriteOperation(PipelineOptions options) Return a subclass of FileBasedSink.FileBasedWriteOperation that will manage the write to the sink.
String	getBaseOutputFilename() Returns the base output filename for this file based sink.
void	validate(PipelineOptions options) Perform pipeline-construction-time validation.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

baseOutputFilename

protected final String baseOutputFilename

Base filename for final output files.

extension

protected final String extension

The extension to be used for the final output files.

fileNamingTemplate

protected final String fileNamingTemplate

Naming template for output files. See ShardNameTemplate for a description of possible naming templates. Default is ShardNameTemplate.INDEX_OF_MAX.

Constructor Detail

FileBasedSink

public FileBasedSink(String baseOutputFilename,
                     String extension)

Construct a FileBasedSink with the given base output filename and extension.

FileBasedSink

public FileBasedSink(String baseOutputFilename,
                     String extension,
                     String fileNamingTemplate)

Construct a FileBasedSink with the given base output filename, extension, and file naming template.

See ShardNameTemplate for a description of file naming templates.

Method Detail

getBaseOutputFilename

public String getBaseOutputFilename()

Returns the base output filename for this file based sink.

validate

public void validate(PipelineOptions options)

Perform pipeline-construction-time validation. The default implementation is a no-op. Subclasses should override to ensure the sink is valid and can be written to. It is recommended to use Preconditions in the implementation of this method.

Specified by:

validate in class Sink<T>

createWriteOperation

public abstract FileBasedSink.FileBasedWriteOperation<T> createWriteOperation(PipelineOptions options)

Return a subclass of FileBasedSink.FileBasedWriteOperation that will manage the write to the sink.

Specified by:

createWriteOperation in class Sink<T>