Package org.apache.beam.sdk.io
Class DefaultFilenamePolicy
- java.lang.Object
-
- org.apache.beam.sdk.io.FileBasedSink.FilenamePolicy
-
- org.apache.beam.sdk.io.DefaultFilenamePolicy
-
- All Implemented Interfaces:
java.io.Serializable
public final class DefaultFilenamePolicy extends FileBasedSink.FilenamePolicy
A defaultFileBasedSink.FilenamePolicy
for windowed and unwindowed files. This policy is constructed using three parameters that together define the output name of a sharded file, in conjunction with the number of shards, index of the particular file, current window and pane information, usingconstructName(org.apache.beam.sdk.io.fs.ResourceId, java.lang.String, java.lang.String, int, int, java.lang.String, java.lang.String)
.Most users will use this
DefaultFilenamePolicy
. For more advanced uses in generating different files for each window and other sharding controls, see theWriteOneFilePerWindow
example pipeline.- See Also:
- Serialized Form
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description static class
DefaultFilenamePolicy.Params
Encapsulates constructor parameters toDefaultFilenamePolicy
.static class
DefaultFilenamePolicy.ParamsCoder
A Coder forDefaultFilenamePolicy.Params
.
-
Field Summary
Fields Modifier and Type Field Description static java.lang.String
DEFAULT_UNWINDOWED_SHARD_TEMPLATE
The default sharding name template.static java.lang.String
DEFAULT_WINDOWED_SHARD_TEMPLATE
The default windowed sharding name template used when writing windowed files.
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description static DefaultFilenamePolicy
fromParams(DefaultFilenamePolicy.Params params)
Construct aDefaultFilenamePolicy
from aDefaultFilenamePolicy.Params
object.static DefaultFilenamePolicy
fromStandardParameters(ValueProvider<ResourceId> baseFilename, @Nullable java.lang.String shardTemplate, @Nullable java.lang.String filenameSuffix, boolean windowedWrites)
Construct aDefaultFilenamePolicy
.void
populateDisplayData(DisplayData.Builder builder)
Populates the display data.@Nullable ResourceId
unwindowedFilename(int shardNumber, int numShards, FileBasedSink.OutputFileHints outputFileHints)
When a sink has not requested windowed or triggered output, this method will be invoked to return the fileresource
to be created given the base output directory and aFileBasedSink.OutputFileHints
containing information about the file, including a suggested (e.g.ResourceId
windowedFilename(int shardNumber, int numShards, BoundedWindow window, PaneInfo paneInfo, FileBasedSink.OutputFileHints outputFileHints)
When a sink has requested windowed or triggered output, this method will be invoked to return the fileresource
to be created given the base output directory and aFileBasedSink.OutputFileHints
containing information about the file, including a suggested extension (e.g.
-
-
-
Field Detail
-
DEFAULT_UNWINDOWED_SHARD_TEMPLATE
public static final java.lang.String DEFAULT_UNWINDOWED_SHARD_TEMPLATE
The default sharding name template.- See Also:
- Constant Field Values
-
DEFAULT_WINDOWED_SHARD_TEMPLATE
public static final java.lang.String DEFAULT_WINDOWED_SHARD_TEMPLATE
The default windowed sharding name template used when writing windowed files. This is used as default in cases when user did not specify shard template to be used and there is a need to write windowed files. In cases when user does specify shard template to be used then provided template will be used for both windowed and non-windowed file names.- See Also:
- Constant Field Values
-
-
Method Detail
-
fromStandardParameters
public static DefaultFilenamePolicy fromStandardParameters(ValueProvider<ResourceId> baseFilename, @Nullable java.lang.String shardTemplate, @Nullable java.lang.String filenameSuffix, boolean windowedWrites)
Construct aDefaultFilenamePolicy
.This is a shortcut for:
DefaultFilenamePolicy.fromParams(new Params() .withBaseFilename(baseFilename) .withShardTemplate(shardTemplate) .withSuffix(filenameSuffix) .withWindowedWrites())
Where the respective
with
methods are invoked only if the value is non-null or true.
-
fromParams
public static DefaultFilenamePolicy fromParams(DefaultFilenamePolicy.Params params)
Construct aDefaultFilenamePolicy
from aDefaultFilenamePolicy.Params
object.
-
unwindowedFilename
public @Nullable ResourceId unwindowedFilename(int shardNumber, int numShards, FileBasedSink.OutputFileHints outputFileHints)
Description copied from class:FileBasedSink.FilenamePolicy
When a sink has not requested windowed or triggered output, this method will be invoked to return the fileresource
to be created given the base output directory and aFileBasedSink.OutputFileHints
containing information about the file, including a suggested (e.g. coming fromCompression
).The shardNumber and numShards parameters, should be used by the policy to generate unique and consistent filenames.
- Specified by:
unwindowedFilename
in classFileBasedSink.FilenamePolicy
-
windowedFilename
public ResourceId windowedFilename(int shardNumber, int numShards, BoundedWindow window, PaneInfo paneInfo, FileBasedSink.OutputFileHints outputFileHints)
Description copied from class:FileBasedSink.FilenamePolicy
When a sink has requested windowed or triggered output, this method will be invoked to return the fileresource
to be created given the base output directory and aFileBasedSink.OutputFileHints
containing information about the file, including a suggested extension (e.g. coming fromCompression
).The policy must return unique and consistent filenames for different windows and panes.
- Specified by:
windowedFilename
in classFileBasedSink.FilenamePolicy
-
populateDisplayData
public void populateDisplayData(DisplayData.Builder builder)
Description copied from class:FileBasedSink.FilenamePolicy
Populates the display data.- Overrides:
populateDisplayData
in classFileBasedSink.FilenamePolicy
-
-