Package org.apache.beam.sdk.io
Class ReadAllViaFileBasedSource<T>
- java.lang.Object
-
- org.apache.beam.sdk.transforms.PTransform<PCollection<FileIO.ReadableFile>,PCollection<T>>
-
- org.apache.beam.sdk.io.ReadAllViaFileBasedSource<T>
-
- All Implemented Interfaces:
java.io.Serializable
,HasDisplayData
@Experimental(SOURCE_SINK) public class ReadAllViaFileBasedSource<T> extends PTransform<PCollection<FileIO.ReadableFile>,PCollection<T>>
Reads each file in the inputPCollection
ofFileIO.ReadableFile
using given parameters for splitting files into offset ranges and for creating aFileBasedSource
for a file. The inputPCollection
must not containdirectories
.To obtain the collection of
FileIO.ReadableFile
from a filepattern, useFileIO.readMatches()
.- See Also:
- Serialized Form
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description static class
ReadAllViaFileBasedSource.ReadFileRangesFnExceptionHandler
A class to handle errors which occur during file reads.
-
Field Summary
Fields Modifier and Type Field Description protected static boolean
DEFAULT_USES_RESHUFFLE
-
Fields inherited from class org.apache.beam.sdk.transforms.PTransform
name, resourceHints
-
-
Constructor Summary
Constructors Constructor Description ReadAllViaFileBasedSource(long desiredBundleSizeBytes, SerializableFunction<java.lang.String,? extends FileBasedSource<T>> createSource, Coder<T> coder)
ReadAllViaFileBasedSource(long desiredBundleSizeBytes, SerializableFunction<java.lang.String,? extends FileBasedSource<T>> createSource, Coder<T> coder, boolean usesReshuffle, ReadAllViaFileBasedSource.ReadFileRangesFnExceptionHandler exceptionHandler)
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description PCollection<T>
expand(PCollection<FileIO.ReadableFile> input)
Override this method to specify how thisPTransform
should be expanded on the givenInputT
.-
Methods inherited from class org.apache.beam.sdk.transforms.PTransform
compose, compose, getAdditionalInputs, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, getResourceHints, populateDisplayData, setResourceHints, toString, validate, validate
-
-
-
-
Field Detail
-
DEFAULT_USES_RESHUFFLE
protected static final boolean DEFAULT_USES_RESHUFFLE
- See Also:
- Constant Field Values
-
-
Constructor Detail
-
ReadAllViaFileBasedSource
public ReadAllViaFileBasedSource(long desiredBundleSizeBytes, SerializableFunction<java.lang.String,? extends FileBasedSource<T>> createSource, Coder<T> coder)
-
ReadAllViaFileBasedSource
public ReadAllViaFileBasedSource(long desiredBundleSizeBytes, SerializableFunction<java.lang.String,? extends FileBasedSource<T>> createSource, Coder<T> coder, boolean usesReshuffle, ReadAllViaFileBasedSource.ReadFileRangesFnExceptionHandler exceptionHandler)
-
-
Method Detail
-
expand
public PCollection<T> expand(PCollection<FileIO.ReadableFile> input)
Description copied from class:PTransform
Override this method to specify how thisPTransform
should be expanded on the givenInputT
.NOTE: This method should not be called directly. Instead apply the
PTransform
should be applied to theInputT
using theapply
method.Composite transforms, which are defined in terms of other transforms, should return the output of one of the composed transforms. Non-composite transforms, which do not apply any transforms internally, should return a new unbound output and register evaluators (via backend-specific registration methods).
- Specified by:
expand
in classPTransform<PCollection<FileIO.ReadableFile>,PCollection<T>>
-
-