org.apache.beam.sdk.util.construction

Class SplittableParDo<InputT,OutputT,RestrictionT,WatermarkEstimatorStateT>

java.lang.Object

org.apache.beam.sdk.transforms.PTransform<PCollection<InputT>,PCollectionTuple>

org.apache.beam.sdk.util.construction.SplittableParDo<InputT,OutputT,RestrictionT,WatermarkEstimatorStateT>

All Implemented Interfaces:

java.io.Serializable, HasDisplayData

public class SplittableParDo<InputT,OutputT,RestrictionT,WatermarkEstimatorStateT>
extends PTransform<PCollection<InputT>,PCollectionTuple>

A utility transform that executes a splittable DoFn by expanding it into a network of simpler transforms:

1. Pair each element with an initial restriction
2. Split each restriction into sub-restrictions
3. Explode windows, since splitting within each window has to happen independently
4. Assign a unique key to each element/restriction pair
5. Process the keyed element/restriction pairs in a runner-specific way with the splittable DoFn's DoFn.ProcessElement method.

This transform is intended as a helper for internal use by runners when implementing ParDo.of(splittable DoFn), but not for direct use by pipeline writers.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	SplittableParDo.OverrideFactory<InputT,OutputT> A PTransformOverrideFactory that overrides a Splittable DoFn with SplittableParDo.
static class	SplittableParDo.PrimitiveBoundedRead<T> The original primitive based Read.Bounded expansion.
static class	SplittableParDo.PrimitiveUnboundedRead<T> The original primitive based Read.Unbounded expansion.
static class	SplittableParDo.ProcessKeyedElements<InputT,OutputT,RestrictionT,WatermarkEstimatorStateT> Runner-specific primitive PTransform that invokes the DoFn.ProcessElement method for a splittable DoFn on each KV of the input PCollection of KVs keyed with arbitrary but globally unique keys.
static class	SplittableParDo.ProcessKeyedElementsTranslator A translator for SplittableParDo.ProcessKeyedElements.
static class	SplittableParDo.Registrar Registers ReadTranslation.UnboundedReadPayloadTranslator and ReadTranslation.BoundedReadPayloadTranslator.

Field Summary

Fields

Modifier and Type	Field and Description
static PTransformOverride	PRIMITIVE_BOUNDED_READ_OVERRIDE A transform override for Read.Bounded that converts it to a SplittableParDo.PrimitiveBoundedRead.
static PTransformOverride	PRIMITIVE_UNBOUNDED_READ_OVERRIDE A transform override for Read.Unbounded that converts it to a SplittableParDo.PrimitiveUnboundedRead.
static java.lang.String	SPLITTABLE_GBKIKWI_URN
static java.lang.String	SPLITTABLE_PROCESS_URN

Fields inherited from class org.apache.beam.sdk.transforms.PTransform

annotations, displayData, name, resourceHints

Method Summary

Modifier and Type	Method and Description
static void	convertReadBasedSplittableDoFnsToPrimitiveReads(Pipeline pipeline) Converts Read based Splittable DoFn expansions to primitive reads implemented by SplittableParDo.PrimitiveBoundedRead and SplittableParDo.PrimitiveUnboundedRead.
static void	convertReadBasedSplittableDoFnsToPrimitiveReadsIfNecessary(Pipeline pipeline) Converts Read based Splittable DoFn expansions to primitive reads implemented by SplittableParDo.PrimitiveBoundedRead and SplittableParDo.PrimitiveUnboundedRead if either the experiment use_deprecated_read or beam_fn_api_use_deprecated_read are specified.
PCollectionTuple	expand(PCollection<InputT> input) Override this method to specify how this PTransform should be expanded on the given InputT.
static <InputT,OutputT> SplittableParDo<InputT,OutputT,?,?>	forAppliedParDo(AppliedPTransform<PCollection<InputT>,PCollectionTuple,?> parDo) Creates the transform for a ParDo-compatible AppliedPTransform.
java.util.Map<TupleTag<?>,PValue>	getAdditionalInputs() Returns all PValues that are consumed as inputs to this PTransform that are independent of the expansion of the InputT within PTransform.expand(PInput).

Methods inherited from class org.apache.beam.sdk.transforms.PTransform

addAnnotation, compose, compose, getAnnotations, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, getResourceHints, populateDisplayData, setDisplayData, setResourceHints, toString, validate, validate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

SPLITTABLE_PROCESS_URN

public static final java.lang.String SPLITTABLE_PROCESS_URN

See Also:

Constant Field Values

SPLITTABLE_GBKIKWI_URN

public static final java.lang.String SPLITTABLE_GBKIKWI_URN

See Also:

Constant Field Values

PRIMITIVE_BOUNDED_READ_OVERRIDE

public static final PTransformOverride PRIMITIVE_BOUNDED_READ_OVERRIDE

A transform override for Read.Bounded that converts it to a SplittableParDo.PrimitiveBoundedRead.

PRIMITIVE_UNBOUNDED_READ_OVERRIDE

public static final PTransformOverride PRIMITIVE_UNBOUNDED_READ_OVERRIDE

A transform override for Read.Unbounded that converts it to a SplittableParDo.PrimitiveUnboundedRead.

Method Detail

forAppliedParDo

public static <InputT,OutputT> SplittableParDo<InputT,OutputT,?,?> forAppliedParDo(AppliedPTransform<PCollection<InputT>,PCollectionTuple,?> parDo)

Creates the transform for a ParDo-compatible AppliedPTransform.

The input may generally be a deserialized transform so it may not actually be a ParDo. Instead ParDoTranslation will be used to extract fields.

expand

public PCollectionTuple expand(PCollection<InputT> input)

Description copied from class: PTransform

Override this method to specify how this PTransform should be expanded on the given InputT.

NOTE: This method should not be called directly. Instead apply the PTransform should be applied to the InputT using the apply method.

Composite transforms, which are defined in terms of other transforms, should return the output of one of the composed transforms. Non-composite transforms, which do not apply any transforms internally, should return a new unbound output and register evaluators (via backend-specific registration methods).

Specified by:

expand in class PTransform<PCollection<InputT>,PCollectionTuple>

getAdditionalInputs

public java.util.Map<TupleTag<?>,PValue> getAdditionalInputs()

Description copied from class: PTransform

Returns all PValues that are consumed as inputs to this PTransform that are independent of the expansion of the InputT within PTransform.expand(PInput).

For example, this can contain any side input consumed by this PTransform.

Overrides:

getAdditionalInputs in class PTransform<PCollection<InputT>,PCollectionTuple>

convertReadBasedSplittableDoFnsToPrimitiveReadsIfNecessary

public static void convertReadBasedSplittableDoFnsToPrimitiveReadsIfNecessary(Pipeline pipeline)

Converts Read based Splittable DoFn expansions to primitive reads implemented by SplittableParDo.PrimitiveBoundedRead and SplittableParDo.PrimitiveUnboundedRead if either the experiment use_deprecated_read or beam_fn_api_use_deprecated_read are specified.

TODO(https://github.com/apache/beam/issues/20530): Remove the primitive Read and make the splittable DoFn the only option.

convertReadBasedSplittableDoFnsToPrimitiveReads

public static void convertReadBasedSplittableDoFnsToPrimitiveReads(Pipeline pipeline)

Converts Read based Splittable DoFn expansions to primitive reads implemented by SplittableParDo.PrimitiveBoundedRead and SplittableParDo.PrimitiveUnboundedRead.

TODO(https://github.com/apache/beam/issues/20530): Remove the primitive Read and make the splittable DoFn the only option.