Class AvroIO.TypedWrite<UserT,DestinationT,OutputT>
- java.lang.Object
-
- org.apache.beam.sdk.transforms.PTransform<PCollection<UserT>,WriteFilesResult<DestinationT>>
-
- org.apache.beam.sdk.io.AvroIO.TypedWrite<UserT,DestinationT,OutputT>
-
- All Implemented Interfaces:
java.io.Serializable
,HasDisplayData
- Enclosing class:
- AvroIO
public abstract static class AvroIO.TypedWrite<UserT,DestinationT,OutputT> extends PTransform<PCollection<UserT>,WriteFilesResult<DestinationT>>
Implementation ofAvroIO.write(java.lang.Class<T>)
.- See Also:
- Serialized Form
-
-
Field Summary
-
Fields inherited from class org.apache.beam.sdk.transforms.PTransform
name, resourceHints
-
-
Constructor Summary
Constructors Constructor Description TypedWrite()
-
Method Summary
All Methods Instance Methods Concrete Methods Deprecated Methods Modifier and Type Method Description WriteFilesResult<DestinationT>
expand(PCollection<UserT> input)
Override this method to specify how thisPTransform
should be expanded on the givenInputT
.void
populateDisplayData(DisplayData.Builder builder)
Register display data for the given transform or component.AvroIO.TypedWrite<UserT,DestinationT,OutputT>
to(java.lang.String outputPrefix)
Writes to file(s) with the given output prefix.<NewDestinationT>
AvroIO.TypedWrite<UserT,NewDestinationT,OutputT>to(DynamicAvroDestinations<UserT,NewDestinationT,OutputT> dynamicDestinations)
Deprecated.UseFileIO.write()
orFileIO.writeDynamic()
instead.AvroIO.TypedWrite<UserT,DestinationT,OutputT>
to(FileBasedSink.FilenamePolicy filenamePolicy)
Writes to files named according to the givenFileBasedSink.FilenamePolicy
.AvroIO.TypedWrite<UserT,DestinationT,OutputT>
to(ResourceId outputPrefix)
Writes to file(s) with the given output prefix.AvroIO.TypedWrite<UserT,DestinationT,OutputT>
to(ValueProvider<java.lang.String> outputPrefix)
Liketo(String)
.AvroIO.TypedWrite<UserT,DestinationT,OutputT>
toResource(ValueProvider<ResourceId> outputPrefix)
Liketo(ResourceId)
.AvroIO.TypedWrite<UserT,DestinationT,OutputT>
withCodec(org.apache.avro.file.CodecFactory codec)
Writes to Avro file(s) compressed using specified codec.AvroIO.TypedWrite<UserT,DestinationT,OutputT>
withDatumWriterFactory(AvroSink.DatumWriterFactory<OutputT> datumWriterFactory)
Specifies aAvroSink.DatumWriterFactory
to use for creatingDatumWriter
instances.AvroIO.TypedWrite<UserT,DestinationT,OutputT>
withFormatFunction(@Nullable SerializableFunction<UserT,OutputT> formatFunction)
Specifies a format function to convertAvroIO.TypedWrite
to the output type.AvroIO.TypedWrite<UserT,DestinationT,OutputT>
withMetadata(java.util.Map<java.lang.String,java.lang.Object> metadata)
Writes to Avro file(s) with the specified metadata.AvroIO.TypedWrite<UserT,DestinationT,OutputT>
withNoSpilling()
AvroIO.TypedWrite<UserT,DestinationT,OutputT>
withNumShards(int numShards)
Configures the number of output shards produced overall (when using unwindowed writes) or per-window (when using windowed writes).AvroIO.TypedWrite<UserT,DestinationT,OutputT>
withoutSharding()
Forces a single file as output and empty shard name template.AvroIO.TypedWrite<UserT,DestinationT,OutputT>
withSchema(org.apache.avro.Schema schema)
Sets the output schema.AvroIO.TypedWrite<UserT,DestinationT,OutputT>
withShardNameTemplate(java.lang.String shardTemplate)
Uses the givenShardNameTemplate
for naming output files.AvroIO.TypedWrite<UserT,DestinationT,OutputT>
withSuffix(java.lang.String filenameSuffix)
Configures the filename suffix for written files.AvroIO.TypedWrite<UserT,DestinationT,OutputT>
withTempDirectory(ResourceId tempDirectory)
Set the base directory used to generate temporary files.AvroIO.TypedWrite<UserT,DestinationT,OutputT>
withTempDirectory(ValueProvider<ResourceId> tempDirectory)
Set the base directory used to generate temporary files.AvroIO.TypedWrite<UserT,DestinationT,OutputT>
withWindowedWrites()
Preserves windowing of input elements and writes them to files based on the element's window.-
Methods inherited from class org.apache.beam.sdk.transforms.PTransform
compose, compose, getAdditionalInputs, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, getResourceHints, setResourceHints, toString, validate, validate
-
-
-
-
Method Detail
-
to
public AvroIO.TypedWrite<UserT,DestinationT,OutputT> to(java.lang.String outputPrefix)
Writes to file(s) with the given output prefix. SeeFileSystems
for information on supported file systems.The name of the output files will be determined by the
FileBasedSink.FilenamePolicy
used.By default, a
DefaultFilenamePolicy
will build output filenames using the specified prefix, a shard name template (seewithShardNameTemplate(String)
, and a common suffix (if supplied usingwithSuffix(String)
). This default can be overridden usingto(FilenamePolicy)
.
-
to
@Experimental(FILESYSTEM) public AvroIO.TypedWrite<UserT,DestinationT,OutputT> to(ResourceId outputPrefix)
Writes to file(s) with the given output prefix. SeeFileSystems
for information on supported file systems. This prefix is used by theDefaultFilenamePolicy
to generate filenames.By default, a
DefaultFilenamePolicy
will build output filenames using the specified prefix, a shard name template (seewithShardNameTemplate(String)
, and a common suffix (if supplied usingwithSuffix(String)
). This default can be overridden usingto(FilenamePolicy)
.This default policy can be overridden using
to(FilenamePolicy)
, in which casewithShardNameTemplate(String)
andwithSuffix(String)
should not be set. Custom filename policies do not automatically see this prefix - you should explicitly pass the prefix into yourFileBasedSink.FilenamePolicy
object if you need this.If
withTempDirectory(org.apache.beam.sdk.options.ValueProvider<org.apache.beam.sdk.io.fs.ResourceId>)
has not been called, this filename prefix will be used to infer a directory for temporary files.
-
to
public AvroIO.TypedWrite<UserT,DestinationT,OutputT> to(ValueProvider<java.lang.String> outputPrefix)
Liketo(String)
.
-
toResource
@Experimental(FILESYSTEM) public AvroIO.TypedWrite<UserT,DestinationT,OutputT> toResource(ValueProvider<ResourceId> outputPrefix)
Liketo(ResourceId)
.
-
to
@Experimental(FILESYSTEM) public AvroIO.TypedWrite<UserT,DestinationT,OutputT> to(FileBasedSink.FilenamePolicy filenamePolicy)
Writes to files named according to the givenFileBasedSink.FilenamePolicy
. A directory for temporary files must be specified usingwithTempDirectory(org.apache.beam.sdk.options.ValueProvider<org.apache.beam.sdk.io.fs.ResourceId>)
.
-
to
@Experimental(FILESYSTEM) @Deprecated public <NewDestinationT> AvroIO.TypedWrite<UserT,NewDestinationT,OutputT> to(DynamicAvroDestinations<UserT,NewDestinationT,OutputT> dynamicDestinations)
Deprecated.UseFileIO.write()
orFileIO.writeDynamic()
instead.Use aDynamicAvroDestinations
object to vendFileBasedSink.FilenamePolicy
objects. These objects can examine the input record when creating aFileBasedSink.FilenamePolicy
. A directory for temporary files must be specified usingwithTempDirectory(org.apache.beam.sdk.options.ValueProvider<org.apache.beam.sdk.io.fs.ResourceId>)
.
-
withSchema
public AvroIO.TypedWrite<UserT,DestinationT,OutputT> withSchema(org.apache.avro.Schema schema)
Sets the output schema. Can only be used when the output type isGenericRecord
and when not usingto(DynamicAvroDestinations)
.
-
withFormatFunction
public AvroIO.TypedWrite<UserT,DestinationT,OutputT> withFormatFunction(@Nullable SerializableFunction<UserT,OutputT> formatFunction)
Specifies a format function to convertAvroIO.TypedWrite
to the output type. Ifto(DynamicAvroDestinations)
is used,FileBasedSink.DynamicDestinations.formatRecord(UserT)
must be used instead.
-
withTempDirectory
@Experimental(FILESYSTEM) public AvroIO.TypedWrite<UserT,DestinationT,OutputT> withTempDirectory(ValueProvider<ResourceId> tempDirectory)
Set the base directory used to generate temporary files.
-
withTempDirectory
@Experimental(FILESYSTEM) public AvroIO.TypedWrite<UserT,DestinationT,OutputT> withTempDirectory(ResourceId tempDirectory)
Set the base directory used to generate temporary files.
-
withShardNameTemplate
public AvroIO.TypedWrite<UserT,DestinationT,OutputT> withShardNameTemplate(java.lang.String shardTemplate)
Uses the givenShardNameTemplate
for naming output files. This option may only be used when using one of the default filename-prefix to() overrides.See
DefaultFilenamePolicy
for how the prefix, shard name template, and suffix are used.
-
withSuffix
public AvroIO.TypedWrite<UserT,DestinationT,OutputT> withSuffix(java.lang.String filenameSuffix)
Configures the filename suffix for written files. This option may only be used when using one of the default filename-prefix to() overrides.See
DefaultFilenamePolicy
for how the prefix, shard name template, and suffix are used.
-
withNumShards
public AvroIO.TypedWrite<UserT,DestinationT,OutputT> withNumShards(int numShards)
Configures the number of output shards produced overall (when using unwindowed writes) or per-window (when using windowed writes).For unwindowed writes, constraining the number of shards is likely to reduce the performance of a pipeline. Setting this value is not recommended unless you require a specific number of output files.
- Parameters:
numShards
- the number of shards to use, or 0 to let the system decide.
-
withoutSharding
public AvroIO.TypedWrite<UserT,DestinationT,OutputT> withoutSharding()
Forces a single file as output and empty shard name template. This option is only compatible with unwindowed writes.For unwindowed writes, constraining the number of shards is likely to reduce the performance of a pipeline. Setting this value is not recommended unless you require a specific number of output files.
This is equivalent to
.withNumShards(1).withShardNameTemplate("")
-
withWindowedWrites
public AvroIO.TypedWrite<UserT,DestinationT,OutputT> withWindowedWrites()
Preserves windowing of input elements and writes them to files based on the element's window.If using
to(FileBasedSink.FilenamePolicy)
. Filenames will be generated usingFileBasedSink.FilenamePolicy.windowedFilename(int, int, org.apache.beam.sdk.transforms.windowing.BoundedWindow, org.apache.beam.sdk.transforms.windowing.PaneInfo, org.apache.beam.sdk.io.FileBasedSink.OutputFileHints)
. See alsoWriteFiles.withWindowedWrites()
.
-
withNoSpilling
public AvroIO.TypedWrite<UserT,DestinationT,OutputT> withNoSpilling()
-
withCodec
public AvroIO.TypedWrite<UserT,DestinationT,OutputT> withCodec(org.apache.avro.file.CodecFactory codec)
Writes to Avro file(s) compressed using specified codec.
-
withDatumWriterFactory
public AvroIO.TypedWrite<UserT,DestinationT,OutputT> withDatumWriterFactory(AvroSink.DatumWriterFactory<OutputT> datumWriterFactory)
Specifies aAvroSink.DatumWriterFactory
to use for creatingDatumWriter
instances.
-
withMetadata
public AvroIO.TypedWrite<UserT,DestinationT,OutputT> withMetadata(java.util.Map<java.lang.String,java.lang.Object> metadata)
Writes to Avro file(s) with the specified metadata.Supported value types are String, Long, and byte[].
-
expand
public WriteFilesResult<DestinationT> expand(PCollection<UserT> input)
Description copied from class:PTransform
Override this method to specify how thisPTransform
should be expanded on the givenInputT
.NOTE: This method should not be called directly. Instead apply the
PTransform
should be applied to theInputT
using theapply
method.Composite transforms, which are defined in terms of other transforms, should return the output of one of the composed transforms. Non-composite transforms, which do not apply any transforms internally, should return a new unbound output and register evaluators (via backend-specific registration methods).
- Specified by:
expand
in classPTransform<PCollection<UserT>,WriteFilesResult<DestinationT>>
-
populateDisplayData
public void populateDisplayData(DisplayData.Builder builder)
Description copied from class:PTransform
Register display data for the given transform or component.populateDisplayData(DisplayData.Builder)
is invoked by Pipeline runners to collect display data viaDisplayData.from(HasDisplayData)
. Implementations may callsuper.populateDisplayData(builder)
in order to register display data in the current namespace, but should otherwise usesubcomponent.populateDisplayData(builder)
to use the namespace of the subcomponent.By default, does not register any display data. Implementors may override this method to provide their own display data.
- Specified by:
populateDisplayData
in interfaceHasDisplayData
- Overrides:
populateDisplayData
in classPTransform<PCollection<UserT>,WriteFilesResult<DestinationT>>
- Parameters:
builder
- The builder to populate with display data.- See Also:
HasDisplayData
-
-