com.google.cloud.dataflow.sdk.io

Class TextIO.Write

java.lang.Object

com.google.cloud.dataflow.sdk.io.TextIO.Write

Enclosing class:

TextIO

public static class TextIO.Write
extends java.lang.Object

A PTransform that writes a PCollection to a text file (or multiple text files matching a sharding pattern), with each PCollection element being encoded into its own line.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	TextIO.Write.Bound<T> A PTransform that writes a bounded PCollection to a text file (or multiple text files matching a sharding pattern), with each PCollection element being encoded into its own line.

Constructor Summary

Constructors

Constructor and Description
TextIO.Write()

Method Summary

Methods

Modifier and Type	Method and Description
static TextIO.Write.Bound<java.lang.String>	named(java.lang.String name) Returns a TextIO.Write PTransform with the given step name.
static TextIO.Write.Bound<java.lang.String>	to(java.lang.String prefix) Returns a TextIO.Write PTransform that writes to the file(s) with the given prefix.
static <T> TextIO.Write.Bound<T>	withCoder(Coder<T> coder) Returns a TextIO.Write PTransform that uses the given Coder<T> to encode each of the elements of the input PCollection<T> into an output text line.
static TextIO.Write.Bound<java.lang.String>	withNumShards(int numShards) Returns a TextIO.Write PTransform that uses the provided shard count.
static TextIO.Write.Bound<java.lang.String>	withoutSharding() Returns a TextIO.Write PTransform that forces a single file as output.
static TextIO.Write.Bound<java.lang.String>	withoutValidation() Returns a TextIO.Write PTransform that has GCS path validation on pipeline creation disabled.
static TextIO.Write.Bound<java.lang.String>	withShardNameTemplate(java.lang.String shardTemplate) Returns a TextIO.Write PTransform that uses the given shard name template.
static TextIO.Write.Bound<java.lang.String>	withSuffix(java.lang.String nameExtension) Returns a TextIO.Write PTransform that writes to the file(s) with the given filename suffix.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TextIO.Write

public TextIO.Write()

Method Detail

named

public static TextIO.Write.Bound<java.lang.String> named(java.lang.String name)

Returns a TextIO.Write PTransform with the given step name.

to

public static TextIO.Write.Bound<java.lang.String> to(java.lang.String prefix)

Returns a TextIO.Write PTransform that writes to the file(s) with the given prefix. This can be a local filename (if running locally), or a Google Cloud Storage filename of the form "gs://<bucket>/<filepath>") (if running locally or via the Google Cloud Dataflow service).

The files written will begin with this prefix, followed by a shard identifier (see TextIO.Write.Bound.withNumShards(int), and end in a common extension, if given by TextIO.Write.Bound.withSuffix(java.lang.String).

withSuffix

public static TextIO.Write.Bound<java.lang.String> withSuffix(java.lang.String nameExtension)

Returns a TextIO.Write PTransform that writes to the file(s) with the given filename suffix.

withNumShards

public static TextIO.Write.Bound<java.lang.String> withNumShards(int numShards)

Returns a TextIO.Write PTransform that uses the provided shard count.

Constraining the number of shards is likely to reduce the performance of a pipeline. Setting this value is not recommended unless you require a specific number of output files.

Parameters:

numShards - the number of shards to use, or 0 to let the system decide.

withShardNameTemplate

public static TextIO.Write.Bound<java.lang.String> withShardNameTemplate(java.lang.String shardTemplate)

Returns a TextIO.Write PTransform that uses the given shard name template.

See ShardNameTemplate for a description of shard templates.

withoutSharding

public static TextIO.Write.Bound<java.lang.String> withoutSharding()

Returns a TextIO.Write PTransform that forces a single file as output.

withCoder

public static <T> TextIO.Write.Bound<T> withCoder(Coder<T> coder)

Returns a TextIO.Write PTransform that uses the given Coder<T> to encode each of the elements of the input PCollection<T> into an output text line.

By default, uses StringUtf8Coder, which writes input Java strings directly as output lines.

Type Parameters:

T - the type of the elements of the input PCollection

withoutValidation

public static TextIO.Write.Bound<java.lang.String> withoutValidation()

Returns a TextIO.Write PTransform that has GCS path validation on pipeline creation disabled.

This can be useful in the case where the GCS output location does not exist at the pipeline creation time, but is expected to be available at execution time.