Package org.apache.beam.sdk.io

Class FileBasedSink.DynamicDestinations<UserT,DestinationT,OutputT>

java.lang.Object

org.apache.beam.sdk.io.FileBasedSink.DynamicDestinations<UserT,DestinationT,OutputT>

All Implemented Interfaces:

java.io.Serializable, HasDisplayData

Direct Known Subclasses:

DynamicAvroDestinations

Enclosing class:

FileBasedSink<UserT,DestinationT,OutputT>

@Experimental(FILESYSTEM)
public abstract static class FileBasedSink.DynamicDestinations<UserT,DestinationT,OutputT>
extends java.lang.Object
implements HasDisplayData, java.io.Serializable

A class that allows value-dependent writes in FileBasedSink.

Users can define a custom type to represent destinations, and provide a mapping to turn this destination type into an instance of FileBasedSink.FilenamePolicy.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor	Description
DynamicDestinations()

Method Summary

Modifier and Type	Method	Description
abstract OutputT	formatRecord(UserT record)	Convert an input record type into the output type.
abstract DestinationT	getDefaultDestination()	Returns the default destination.
abstract DestinationT	getDestination(UserT element)	Returns an object that represents at a high level the destination being written to.
@Nullable Coder<DestinationT>	getDestinationCoder()	Returns the coder for FileBasedSink.DynamicDestinations.
abstract FileBasedSink.FilenamePolicy	getFilenamePolicy(DestinationT destination)	Converts a destination into a FileBasedSink.FilenamePolicy.
java.util.List<PCollectionView<?>>	getSideInputs()	Override to specify that this object needs access to one or more side inputs.
void	populateDisplayData(DisplayData.Builder builder)	Populates the display data.
protected <SideInputT> SideInputT	sideInput(PCollectionView<SideInputT> view)	Returns the value of a given side input.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DynamicDestinations

public DynamicDestinations()

Method Detail

getSideInputs

public java.util.List<PCollectionView<?>> getSideInputs()

Override to specify that this object needs access to one or more side inputs. This side inputs must be globally windowed, as they will be accessed from the global window.

sideInput

protected final <SideInputT> SideInputT sideInput(PCollectionView<SideInputT> view)

Returns the value of a given side input. The view must be present in getSideInputs().

formatRecord

public abstract OutputT formatRecord(UserT record)

Convert an input record type into the output type.

getDestination

public abstract DestinationT getDestination(UserT element)

Returns an object that represents at a high level the destination being written to. May not return null. A destination must have deterministic hash and equality methods defined.

getDefaultDestination

public abstract DestinationT getDefaultDestination()

Returns the default destination. This is used for collections that have no elements as the destination to write empty files to.

getDestinationCoder

public @Nullable Coder<DestinationT> getDestinationCoder()

Returns the coder for FileBasedSink.DynamicDestinations. If this is not overridden, then the coder registry will be use to find a suitable coder. This must be a deterministic coder, as FileBasedSink.DynamicDestinations will be used as a key type in a GroupByKey.

getFilenamePolicy

public abstract FileBasedSink.FilenamePolicy getFilenamePolicy(DestinationT destination)

Converts a destination into a FileBasedSink.FilenamePolicy. May not return null.

populateDisplayData

public void populateDisplayData(DisplayData.Builder builder)

Populates the display data.

Specified by:

populateDisplayData in interface HasDisplayData

Parameters:

builder - The builder to populate with display data.

See Also:

HasDisplayData