Class AvroIO.Read<T>
- java.lang.Object
-
- org.apache.beam.sdk.transforms.PTransform<PBegin,PCollection<T>>
-
- org.apache.beam.sdk.io.AvroIO.Read<T>
-
- All Implemented Interfaces:
java.io.Serializable
,HasDisplayData
- Enclosing class:
- AvroIO
public abstract static class AvroIO.Read<T> extends PTransform<PBegin,PCollection<T>>
Implementation ofAvroIO.read(java.lang.Class<T>)
andAvroIO.readGenericRecords(org.apache.avro.Schema)
.- See Also:
- Serialized Form
-
-
Field Summary
-
Fields inherited from class org.apache.beam.sdk.transforms.PTransform
name, resourceHints
-
-
Constructor Summary
Constructors Constructor Description Read()
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description PCollection<T>
expand(PBegin input)
Override this method to specify how thisPTransform
should be expanded on the givenInputT
.AvroIO.Read<T>
from(java.lang.String filepattern)
Likefrom(ValueProvider)
.AvroIO.Read<T>
from(ValueProvider<java.lang.String> filepattern)
Reads from the given filename or filepattern.void
populateDisplayData(DisplayData.Builder builder)
Register display data for the given transform or component.AvroIO.Read<T>
watchForNewFiles(org.joda.time.Duration pollInterval, Watch.Growth.TerminationCondition<java.lang.String,?> terminationCondition)
Same aswatchForNewFiles(Duration, TerminationCondition, boolean)
withmatchUpdatedFiles=false
.AvroIO.Read<T>
watchForNewFiles(org.joda.time.Duration pollInterval, Watch.Growth.TerminationCondition<java.lang.String,?> terminationCondition, boolean matchUpdatedFiles)
Continuously watches for new files matching the filepattern, polling it at the given interval, until the given termination condition is reached.AvroIO.Read<T>
withBeamSchemas(boolean withBeamSchemas)
If set to true, a Beam schema will be inferred from the AVRO schema.AvroIO.Read<T>
withEmptyMatchTreatment(EmptyMatchTreatment treatment)
Configures whether or not a filepattern matching no files is allowed.AvroIO.Read<T>
withHintMatchesManyFiles()
Hints that the filepattern specified infrom(String)
matches a very large number of files.AvroIO.Read<T>
withMatchConfiguration(FileIO.MatchConfiguration matchConfiguration)
Sets theFileIO.MatchConfiguration
.-
Methods inherited from class org.apache.beam.sdk.transforms.PTransform
compose, compose, getAdditionalInputs, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, getResourceHints, setResourceHints, toString, validate, validate
-
-
-
-
Method Detail
-
from
public AvroIO.Read<T> from(ValueProvider<java.lang.String> filepattern)
Reads from the given filename or filepattern.If it is known that the filepattern will match a very large number of files (at least tens of thousands), use
withHintMatchesManyFiles()
for better performance and scalability.
-
from
public AvroIO.Read<T> from(java.lang.String filepattern)
Likefrom(ValueProvider)
.
-
withMatchConfiguration
public AvroIO.Read<T> withMatchConfiguration(FileIO.MatchConfiguration matchConfiguration)
Sets theFileIO.MatchConfiguration
.
-
withEmptyMatchTreatment
public AvroIO.Read<T> withEmptyMatchTreatment(EmptyMatchTreatment treatment)
Configures whether or not a filepattern matching no files is allowed.
-
watchForNewFiles
public AvroIO.Read<T> watchForNewFiles(org.joda.time.Duration pollInterval, Watch.Growth.TerminationCondition<java.lang.String,?> terminationCondition, boolean matchUpdatedFiles)
Continuously watches for new files matching the filepattern, polling it at the given interval, until the given termination condition is reached. The returnedPCollection
is unbounded. IfmatchUpdatedFiles
is set, also watches for files with timestamp change.This works only in runners supporting splittable
DoFn
.
-
watchForNewFiles
public AvroIO.Read<T> watchForNewFiles(org.joda.time.Duration pollInterval, Watch.Growth.TerminationCondition<java.lang.String,?> terminationCondition)
Same aswatchForNewFiles(Duration, TerminationCondition, boolean)
withmatchUpdatedFiles=false
.
-
withHintMatchesManyFiles
public AvroIO.Read<T> withHintMatchesManyFiles()
Hints that the filepattern specified infrom(String)
matches a very large number of files.This hint may cause a runner to execute the transform differently, in a way that improves performance for this case, but it may worsen performance if the filepattern matches only a small number of files (e.g., in a runner that supports dynamic work rebalancing, it will happen less efficiently within individual files).
-
withBeamSchemas
@Experimental(SCHEMAS) public AvroIO.Read<T> withBeamSchemas(boolean withBeamSchemas)
If set to true, a Beam schema will be inferred from the AVRO schema. This allows the output to be used by SQL and by the schema-transform library.
-
expand
public PCollection<T> expand(PBegin input)
Description copied from class:PTransform
Override this method to specify how thisPTransform
should be expanded on the givenInputT
.NOTE: This method should not be called directly. Instead apply the
PTransform
should be applied to theInputT
using theapply
method.Composite transforms, which are defined in terms of other transforms, should return the output of one of the composed transforms. Non-composite transforms, which do not apply any transforms internally, should return a new unbound output and register evaluators (via backend-specific registration methods).
- Specified by:
expand
in classPTransform<PBegin,PCollection<T>>
-
populateDisplayData
public void populateDisplayData(DisplayData.Builder builder)
Description copied from class:PTransform
Register display data for the given transform or component.populateDisplayData(DisplayData.Builder)
is invoked by Pipeline runners to collect display data viaDisplayData.from(HasDisplayData)
. Implementations may callsuper.populateDisplayData(builder)
in order to register display data in the current namespace, but should otherwise usesubcomponent.populateDisplayData(builder)
to use the namespace of the subcomponent.By default, does not register any display data. Implementors may override this method to provide their own display data.
- Specified by:
populateDisplayData
in interfaceHasDisplayData
- Overrides:
populateDisplayData
in classPTransform<PBegin,PCollection<T>>
- Parameters:
builder
- The builder to populate with display data.- See Also:
HasDisplayData
-
-