Class TFRecordIO.Write
- java.lang.Object
-
- org.apache.beam.sdk.transforms.PTransform<PCollection<byte[]>,PDone>
-
- org.apache.beam.sdk.io.TFRecordIO.Write
-
- All Implemented Interfaces:
java.io.Serializable
,HasDisplayData
- Enclosing class:
- TFRecordIO
public abstract static class TFRecordIO.Write extends PTransform<PCollection<byte[]>,PDone>
Implementation ofTFRecordIO.write()
.- See Also:
- Serialized Form
-
-
Field Summary
-
Fields inherited from class org.apache.beam.sdk.transforms.PTransform
name, resourceHints
-
-
Constructor Summary
Constructors Constructor Description Write()
-
Method Summary
All Methods Instance Methods Concrete Methods Deprecated Methods Modifier and Type Method Description PDone
expand(PCollection<byte[]> input)
Override this method to specify how thisPTransform
should be expanded on the givenInputT
.void
populateDisplayData(DisplayData.Builder builder)
Register display data for the given transform or component.TFRecordIO.Write
to(java.lang.String outputPrefix)
Writes TFRecord file(s) with the given output prefix.TFRecordIO.Write
to(ResourceId outputResource)
Writes TFRecord file(s) with a prefix given by the specified resource.TFRecordIO.Write
toResource(ValueProvider<ResourceId> outputResource)
Liketo(ResourceId)
.TFRecordIO.Write
withCompression(Compression compression)
Writes to output files using the specified compression type.TFRecordIO.Write
withCompressionType(TFRecordIO.CompressionType compressionType)
Deprecated.TFRecordIO.Write
withNoSpilling()
TFRecordIO.Write
withNumShards(int numShards)
Writes to the provided number of shards.TFRecordIO.Write
withoutSharding()
Forces a single file as output.TFRecordIO.Write
withShardNameTemplate(java.lang.String shardTemplate)
Uses the given shard name template.TFRecordIO.Write
withSuffix(java.lang.String suffix)
Writes to the file(s) with the given filename suffix.-
Methods inherited from class org.apache.beam.sdk.transforms.PTransform
compose, compose, getAdditionalInputs, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, getResourceHints, setResourceHints, toString, validate, validate
-
-
-
-
Method Detail
-
to
public TFRecordIO.Write to(java.lang.String outputPrefix)
Writes TFRecord file(s) with the given output prefix. Theprefix
will be used as a to generate aResourceId
using any supportedFileSystem
.In addition to their prefix, created files will have a shard identifier (see
withNumShards(int)
), and end in a common suffix, if given bywithSuffix(String)
.For more information on filenames, see
DefaultFilenamePolicy
.
-
to
@Experimental(FILESYSTEM) public TFRecordIO.Write to(ResourceId outputResource)
Writes TFRecord file(s) with a prefix given by the specified resource.In addition to their prefix, created files will have a shard identifier (see
withNumShards(int)
), and end in a common suffix, if given bywithSuffix(String)
.For more information on filenames, see
DefaultFilenamePolicy
.
-
toResource
@Experimental(FILESYSTEM) public TFRecordIO.Write toResource(ValueProvider<ResourceId> outputResource)
Liketo(ResourceId)
.
-
withSuffix
public TFRecordIO.Write withSuffix(java.lang.String suffix)
Writes to the file(s) with the given filename suffix.- See Also:
ShardNameTemplate
-
withNumShards
public TFRecordIO.Write withNumShards(int numShards)
Writes to the provided number of shards.Constraining the number of shards is likely to reduce the performance of a pipeline. Setting this value is not recommended unless you require a specific number of output files.
- Parameters:
numShards
- the number of shards to use, or 0 to let the system decide.- See Also:
ShardNameTemplate
-
withShardNameTemplate
public TFRecordIO.Write withShardNameTemplate(java.lang.String shardTemplate)
Uses the given shard name template.- See Also:
ShardNameTemplate
-
withoutSharding
public TFRecordIO.Write withoutSharding()
Forces a single file as output.Constraining the number of shards is likely to reduce the performance of a pipeline. Using this setting is not recommended unless you truly require a single output file.
This is a shortcut for
.withNumShards(1).withShardNameTemplate("")
-
withCompressionType
@Deprecated public TFRecordIO.Write withCompressionType(TFRecordIO.CompressionType compressionType)
Deprecated.
-
withCompression
public TFRecordIO.Write withCompression(Compression compression)
Writes to output files using the specified compression type.If no compression type is specified, the default is
Compression.UNCOMPRESSED
. SeeTFRecordIO.Read.withCompression(org.apache.beam.sdk.io.Compression)
for more details.
-
withNoSpilling
public TFRecordIO.Write withNoSpilling()
-
expand
public PDone expand(PCollection<byte[]> input)
Description copied from class:PTransform
Override this method to specify how thisPTransform
should be expanded on the givenInputT
.NOTE: This method should not be called directly. Instead apply the
PTransform
should be applied to theInputT
using theapply
method.Composite transforms, which are defined in terms of other transforms, should return the output of one of the composed transforms. Non-composite transforms, which do not apply any transforms internally, should return a new unbound output and register evaluators (via backend-specific registration methods).
- Specified by:
expand
in classPTransform<PCollection<byte[]>,PDone>
-
populateDisplayData
public void populateDisplayData(DisplayData.Builder builder)
Description copied from class:PTransform
Register display data for the given transform or component.populateDisplayData(DisplayData.Builder)
is invoked by Pipeline runners to collect display data viaDisplayData.from(HasDisplayData)
. Implementations may callsuper.populateDisplayData(builder)
in order to register display data in the current namespace, but should otherwise usesubcomponent.populateDisplayData(builder)
to use the namespace of the subcomponent.By default, does not register any display data. Implementors may override this method to provide their own display data.
- Specified by:
populateDisplayData
in interfaceHasDisplayData
- Overrides:
populateDisplayData
in classPTransform<PCollection<byte[]>,PDone>
- Parameters:
builder
- The builder to populate with display data.- See Also:
HasDisplayData
-
-