Package org.apache.beam.sdk.io
Class FileIO.MatchConfiguration
- java.lang.Object
-
- org.apache.beam.sdk.io.FileIO.MatchConfiguration
-
- All Implemented Interfaces:
java.io.Serializable
,HasDisplayData
- Enclosing class:
- FileIO
public abstract static class FileIO.MatchConfiguration extends java.lang.Object implements HasDisplayData, java.io.Serializable
Describes configuration for matching filepatterns, such asEmptyMatchTreatment
and continuous watching for matching files.- See Also:
- Serialized Form
-
-
Constructor Summary
Constructors Constructor Description MatchConfiguration()
-
Method Summary
All Methods Static Methods Instance Methods Abstract Methods Concrete Methods Modifier and Type Method Description FileIO.MatchConfiguration
continuously(org.joda.time.Duration interval, Watch.Growth.TerminationCondition<java.lang.String,?> condition)
Continuously watches for new files at the given interval until the given termination condition is reached, where the input to the condition is the filepattern.FileIO.MatchConfiguration
continuously(org.joda.time.Duration interval, Watch.Growth.TerminationCondition<java.lang.String,?> condition, boolean matchUpdatedFiles)
Continuously watches for new files at the given interval until the given termination condition is reached, where the input to the condition is the filepattern.static FileIO.MatchConfiguration
create(EmptyMatchTreatment emptyMatchTreatment)
Creates aFileIO.MatchConfiguration
with the givenEmptyMatchTreatment
.abstract EmptyMatchTreatment
getEmptyMatchTreatment()
abstract boolean
getMatchUpdatedFiles()
abstract @Nullable org.joda.time.Duration
getWatchInterval()
void
populateDisplayData(DisplayData.Builder builder)
Register display data for the given transform or component.FileIO.MatchConfiguration
withEmptyMatchTreatment(EmptyMatchTreatment treatment)
Sets theEmptyMatchTreatment
.
-
-
-
Method Detail
-
create
public static FileIO.MatchConfiguration create(EmptyMatchTreatment emptyMatchTreatment)
Creates aFileIO.MatchConfiguration
with the givenEmptyMatchTreatment
.
-
getEmptyMatchTreatment
public abstract EmptyMatchTreatment getEmptyMatchTreatment()
-
getMatchUpdatedFiles
public abstract boolean getMatchUpdatedFiles()
-
getWatchInterval
public abstract @Nullable org.joda.time.Duration getWatchInterval()
-
withEmptyMatchTreatment
public FileIO.MatchConfiguration withEmptyMatchTreatment(EmptyMatchTreatment treatment)
Sets theEmptyMatchTreatment
.
-
continuously
public FileIO.MatchConfiguration continuously(org.joda.time.Duration interval, Watch.Growth.TerminationCondition<java.lang.String,?> condition, boolean matchUpdatedFiles)
Continuously watches for new files at the given interval until the given termination condition is reached, where the input to the condition is the filepattern.If
matchUpdatedFiles
is set, also watches for files with timestamp change, with the watching frequency given by theinterval
. The pipeline will throw aRuntimeError
if timestamp extraction for the matched file has failed, suggesting the timestamp metadata is not available with the IO connector.
-
continuously
public FileIO.MatchConfiguration continuously(org.joda.time.Duration interval, Watch.Growth.TerminationCondition<java.lang.String,?> condition)
Continuously watches for new files at the given interval until the given termination condition is reached, where the input to the condition is the filepattern. To watch also for updated files, please setmatchUpdatedFiles
astrue
.
-
populateDisplayData
public void populateDisplayData(DisplayData.Builder builder)
Description copied from interface:HasDisplayData
Register display data for the given transform or component.populateDisplayData(DisplayData.Builder)
is invoked by Pipeline runners to collect display data viaDisplayData.from(HasDisplayData)
. Implementations may callsuper.populateDisplayData(builder)
in order to register display data in the current namespace, but should otherwise usesubcomponent.populateDisplayData(builder)
to use the namespace of the subcomponent.- Specified by:
populateDisplayData
in interfaceHasDisplayData
- Parameters:
builder
- The builder to populate with display data.- See Also:
HasDisplayData
-
-