Class FlatMapElements<InputT,OutputT>
- java.lang.Object
-
- org.apache.beam.sdk.transforms.PTransform<PCollection<? extends InputT>,PCollection<OutputT>>
-
- org.apache.beam.sdk.transforms.FlatMapElements<InputT,OutputT>
-
- All Implemented Interfaces:
java.io.Serializable
,HasDisplayData
public class FlatMapElements<InputT,OutputT> extends PTransform<PCollection<? extends InputT>,PCollection<OutputT>>
PTransform
s for mapping a simple function that returns iterables over the elements of aPCollection
and merging the results.- See Also:
- Serialized Form
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description static class
FlatMapElements.FlatMapWithFailures<InputT,OutputT,FailureT>
APTransform
that adds exception handling toFlatMapElements
.
-
Field Summary
-
Fields inherited from class org.apache.beam.sdk.transforms.PTransform
name, resourceHints
-
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description <NewFailureT>
FlatMapElements.FlatMapWithFailures<InputT,OutputT,NewFailureT>exceptionsInto(TypeDescriptor<NewFailureT> failureTypeDescriptor)
Returns a newFlatMapElements.FlatMapWithFailures
transform that catches exceptions raised while mapping elements, with the given type descriptor used for the failure collection but the exception handler yet to be specified usingFlatMapElements.FlatMapWithFailures.exceptionsVia(ProcessFunction)
.<FailureT> FlatMapElements.FlatMapWithFailures<InputT,OutputT,FailureT>
exceptionsVia(InferableFunction<WithFailures.ExceptionElement<InputT>,FailureT> exceptionHandler)
Returns a newFlatMapElements.FlatMapWithFailures
transform that catches exceptions raised while mapping elements, passing the raised exception instance and the input element being processed through the givenexceptionHandler
and emitting the result to a failure collection.PCollection<OutputT>
expand(PCollection<? extends InputT> input)
Override this method to specify how thisPTransform
should be expanded on the givenInputT
.static <OutputT> FlatMapElements<?,OutputT>
into(TypeDescriptor<OutputT> outputType)
Returns a newFlatMapElements
transform with the given type descriptor for the output type, but the mapping function yet to be specified usingvia(ProcessFunction)
.void
populateDisplayData(DisplayData.Builder builder)
Register display data for the given transform or component.<NewInputT>
FlatMapElements<NewInputT,OutputT>via(Contextful<Contextful.Fn<NewInputT,java.lang.Iterable<OutputT>>> fn)
Likevia(ProcessFunction)
, but allows access to additional context.static <InputT,OutputT>
FlatMapElements<InputT,OutputT>via(InferableFunction<? super InputT,? extends java.lang.Iterable<OutputT>> fn)
For aInferableFunction<InputT, ? extends Iterable<OutputT>>
fn
, return aPTransform
that appliesfn
to every element of the inputPCollection<InputT>
and outputs all of the elements to the outputPCollection<OutputT>
.<NewInputT>
FlatMapElements<NewInputT,OutputT>via(ProcessFunction<NewInputT,? extends java.lang.Iterable<OutputT>> fn)
For aProcessFunction<InputT, ? extends Iterable<OutputT>>
fn
, returns aPTransform
that appliesfn
to every element of the inputPCollection<InputT>
and outputs all of the elements to the outputPCollection<OutputT>
.<NewInputT>
FlatMapElements<NewInputT,OutputT>via(SerializableFunction<NewInputT,? extends java.lang.Iterable<OutputT>> fn)
Binary compatibility adapter forvia(ProcessFunction)
.static <InputT,OutputT>
FlatMapElements<InputT,OutputT>via(SimpleFunction<? super InputT,? extends java.lang.Iterable<OutputT>> fn)
Binary compatibility adapter forvia(ProcessFunction)
.-
Methods inherited from class org.apache.beam.sdk.transforms.PTransform
compose, compose, getAdditionalInputs, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, getResourceHints, setResourceHints, toString, validate, validate
-
-
-
-
Method Detail
-
via
public static <InputT,OutputT> FlatMapElements<InputT,OutputT> via(InferableFunction<? super InputT,? extends java.lang.Iterable<OutputT>> fn)
For aInferableFunction<InputT, ? extends Iterable<OutputT>>
fn
, return aPTransform
that appliesfn
to every element of the inputPCollection<InputT>
and outputs all of the elements to the outputPCollection<OutputT>
.InferableFunction
has the advantage of providing type descriptor information, but it is generally more convenient to specify output type viainto(TypeDescriptor)
, and provide the mapping as a lambda expression tovia(ProcessFunction)
.Example usage:
PCollection<String> lines = ...; PCollection<String> words = lines.apply(FlatMapElements.via( new InferableFunction<String, List<String>>() { public List<String> apply(String line) throws Exception { return Arrays.asList(line.split(" ")); } });
-
via
public static <InputT,OutputT> FlatMapElements<InputT,OutputT> via(SimpleFunction<? super InputT,? extends java.lang.Iterable<OutputT>> fn)
Binary compatibility adapter forvia(ProcessFunction)
.
-
into
public static <OutputT> FlatMapElements<?,OutputT> into(TypeDescriptor<OutputT> outputType)
Returns a newFlatMapElements
transform with the given type descriptor for the output type, but the mapping function yet to be specified usingvia(ProcessFunction)
.
-
via
public <NewInputT> FlatMapElements<NewInputT,OutputT> via(ProcessFunction<NewInputT,? extends java.lang.Iterable<OutputT>> fn)
For aProcessFunction<InputT, ? extends Iterable<OutputT>>
fn
, returns aPTransform
that appliesfn
to every element of the inputPCollection<InputT>
and outputs all of the elements to the outputPCollection<OutputT>
.Example usage:
PCollection<String> words = lines.apply( FlatMapElements.into(TypeDescriptors.strings()) .via((String line) -> Arrays.asList(line.split(" ")))
-
via
public <NewInputT> FlatMapElements<NewInputT,OutputT> via(SerializableFunction<NewInputT,? extends java.lang.Iterable<OutputT>> fn)
Binary compatibility adapter forvia(ProcessFunction)
.
-
via
@Experimental(CONTEXTFUL) public <NewInputT> FlatMapElements<NewInputT,OutputT> via(Contextful<Contextful.Fn<NewInputT,java.lang.Iterable<OutputT>>> fn)
Likevia(ProcessFunction)
, but allows access to additional context.
-
expand
public PCollection<OutputT> expand(PCollection<? extends InputT> input)
Description copied from class:PTransform
Override this method to specify how thisPTransform
should be expanded on the givenInputT
.NOTE: This method should not be called directly. Instead apply the
PTransform
should be applied to theInputT
using theapply
method.Composite transforms, which are defined in terms of other transforms, should return the output of one of the composed transforms. Non-composite transforms, which do not apply any transforms internally, should return a new unbound output and register evaluators (via backend-specific registration methods).
- Specified by:
expand
in classPTransform<PCollection<? extends InputT>,PCollection<OutputT>>
-
populateDisplayData
public void populateDisplayData(DisplayData.Builder builder)
Description copied from class:PTransform
Register display data for the given transform or component.populateDisplayData(DisplayData.Builder)
is invoked by Pipeline runners to collect display data viaDisplayData.from(HasDisplayData)
. Implementations may callsuper.populateDisplayData(builder)
in order to register display data in the current namespace, but should otherwise usesubcomponent.populateDisplayData(builder)
to use the namespace of the subcomponent.By default, does not register any display data. Implementors may override this method to provide their own display data.
- Specified by:
populateDisplayData
in interfaceHasDisplayData
- Overrides:
populateDisplayData
in classPTransform<PCollection<? extends InputT>,PCollection<OutputT>>
- Parameters:
builder
- The builder to populate with display data.- See Also:
HasDisplayData
-
exceptionsInto
@Experimental(WITH_EXCEPTIONS) public <NewFailureT> FlatMapElements.FlatMapWithFailures<InputT,OutputT,NewFailureT> exceptionsInto(TypeDescriptor<NewFailureT> failureTypeDescriptor)
Returns a newFlatMapElements.FlatMapWithFailures
transform that catches exceptions raised while mapping elements, with the given type descriptor used for the failure collection but the exception handler yet to be specified usingFlatMapElements.FlatMapWithFailures.exceptionsVia(ProcessFunction)
.See
WithFailures
documentation for usage patterns of the returnedWithFailures.Result
.
-
exceptionsVia
@Experimental(WITH_EXCEPTIONS) public <FailureT> FlatMapElements.FlatMapWithFailures<InputT,OutputT,FailureT> exceptionsVia(InferableFunction<WithFailures.ExceptionElement<InputT>,FailureT> exceptionHandler)
Returns a newFlatMapElements.FlatMapWithFailures
transform that catches exceptions raised while mapping elements, passing the raised exception instance and the input element being processed through the givenexceptionHandler
and emitting the result to a failure collection.This method takes advantage of the type information provided by
InferableFunction
, meaning that a call toexceptionsInto(TypeDescriptor)
may not be necessary.See
WithFailures
documentation for usage patterns of the returnedWithFailures.Result
.Example usage:
Result<PCollection<String>, String>> result = words.apply( FlatMapElements .into(TypeDescriptors.strings()) // Could throw ArrayIndexOutOfBoundsException .via((String line) -> Arrays.asList(Arrays.copyOfRange(line.split(" "), 1, 5))) .exceptionsVia(new WithFailures.ExceptionAsMapHandler<String>() {})); PCollection<String> output = result.output(); PCollection<String> failures = result.failures();
-
-