com.google.cloud.dataflow.sdk.io

Class TextIO.Write

java.lang.Object

com.google.cloud.dataflow.sdk.io.TextIO.Write

Enclosing class:

TextIO

public static class TextIO.Write
extends Object

A PTransform that writes a PCollection to text file (or multiple text files matching a sharding pattern), with each element of the input collection encoded into its own line.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	TextIO.Write.Bound<T> A PTransform that writes a bounded PCollection to a text file (or multiple text files matching a sharding pattern), with each PCollection element being encoded into its own line.

Constructor Summary

Constructors

Constructor and Description
Write()

Method Summary

Modifier and Type	Method and Description
static TextIO.Write.Bound<String>	named(String name) Returns a transform for writing to text files with the given step name.
static TextIO.Write.Bound<String>	to(String prefix) Returns a transform for writing to text files that writes to the file(s) with the given prefix.
static <T> TextIO.Write.Bound<T>	withCoder(Coder<T> coder) Returns a transform for writing to text files that uses the given Coder to encode each of the elements of the input PCollection into an output text line.
static TextIO.Write.Bound<String>	withFooter(String footer) Returns a transform for writing to text files that adds a footer string to the files it writes.
static TextIO.Write.Bound<String>	withHeader(String header) Returns a transform for writing to text files that adds a header string to the files it writes.
static TextIO.Write.Bound<String>	withNumShards(int numShards) Returns a transform for writing to text files that uses the provided shard count.
static TextIO.Write.Bound<String>	withoutSharding() Returns a transform for writing to text files that forces a single file as output.
static TextIO.Write.Bound<String>	withoutValidation() Returns a transform for writing to text files that has GCS path validation on pipeline creation disabled.
static TextIO.Write.Bound<String>	withShardNameTemplate(String shardTemplate) Returns a transform for writing to text files that uses the given shard name template.
static TextIO.Write.Bound<String>	withSuffix(String nameExtension) Returns a transform for writing to text files that appends the specified suffix to the created files.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Write

public Write()

Method Detail

named

public static TextIO.Write.Bound<String> named(String name)

Returns a transform for writing to text files with the given step name.

to

public static TextIO.Write.Bound<String> to(String prefix)

Returns a transform for writing to text files that writes to the file(s) with the given prefix. This can be a local filename (if running locally), or a Google Cloud Storage filename of the form "gs://<bucket>/<filepath>" (if running locally or via the Google Cloud Dataflow service).

The files written will begin with this prefix, followed by a shard identifier (see TextIO.Write.Bound.withNumShards(int), and end in a common extension, if given by TextIO.Write.Bound.withSuffix(String).

withSuffix

public static TextIO.Write.Bound<String> withSuffix(String nameExtension)

Returns a transform for writing to text files that appends the specified suffix to the created files.

withNumShards

public static TextIO.Write.Bound<String> withNumShards(int numShards)

Returns a transform for writing to text files that uses the provided shard count.

Constraining the number of shards is likely to reduce the performance of a pipeline. Setting this value is not recommended unless you require a specific number of output files.

Parameters:

numShards - the number of shards to use, or 0 to let the system decide.

withShardNameTemplate

public static TextIO.Write.Bound<String> withShardNameTemplate(String shardTemplate)

Returns a transform for writing to text files that uses the given shard name template.

See ShardNameTemplate for a description of shard templates.

withoutSharding

public static TextIO.Write.Bound<String> withoutSharding()

Returns a transform for writing to text files that forces a single file as output.

withCoder

public static <T> TextIO.Write.Bound<T> withCoder(Coder<T> coder)

Returns a transform for writing to text files that uses the given Coder to encode each of the elements of the input PCollection into an output text line.

By default, uses StringUtf8Coder, which writes input Java strings directly as output lines.

Type Parameters:

T - the type of the elements of the input PCollection

withoutValidation

public static TextIO.Write.Bound<String> withoutValidation()

Returns a transform for writing to text files that has GCS path validation on pipeline creation disabled.

This can be useful in the case where the GCS output location does not exist at the pipeline creation time, but is expected to be available at execution time.

withHeader

public static TextIO.Write.Bound<String> withHeader(@Nullable
                                                    String header)

Returns a transform for writing to text files that adds a header string to the files it writes. Note that a newline character will be added after the header.

A null value will clear any previously configured header.

Parameters:

header - the string to be added as file header

withFooter

public static TextIO.Write.Bound<String> withFooter(@Nullable
                                                    String footer)

Returns a transform for writing to text files that adds a footer string to the files it writes. Note that a newline character will be added after the header.

A null value will clear any previously configured footer.

Parameters:

footer - the string to be added as file footer