org.apache.beam.sdk.util.construction.graph

Interface ExecutableStage

All Known Implementing Classes:

ImmutableExecutableStage

public interface ExecutableStage

A combination of PTransforms that can be executed within a single SDK harness.

Contains only the nodes that specify the processing to perform within the SDK harness, and does not contain any runner-executed nodes.

Within a single RunnerApi.Pipeline, PTransforms and PCollections are permitted to appear in multiple executable stages. However, paths from a root RunnerApi.PTransform to any other RunnerApi.PTransform within that set of stages must be unique.

Field Summary

Fields

Modifier and Type	Field and Description
static java.util.Collection<org.apache.beam.model.pipeline.v1.RunnerApi.ExecutableStagePayload.WireCoderSetting>	DEFAULT_WIRE_CODER_SETTINGS The default wire coder settings which returns an empty list, i.e., the WireCoder for each PCollection and timer will be a WINDOWED_VALUE coder.
static java.lang.String	URN The URN identifying an ExecutableStage that has been converted to a RunnerApi.PTransform.

Method Summary

Modifier and Type	Method and Description
static ExecutableStage	fromPayload(org.apache.beam.model.pipeline.v1.RunnerApi.ExecutableStagePayload payload) Return an ExecutableStage constructed from the provided RunnerApi.FunctionSpec representation.
org.apache.beam.model.pipeline.v1.RunnerApi.Components	getComponents() Return the RunnerApi.Components required to execute this ExecutableStage.
org.apache.beam.model.pipeline.v1.RunnerApi.Environment	getEnvironment() Returns the RunnerApi.Environment this stage executes in.
PipelineNode.PCollectionNode	getInputPCollection() Returns the root PipelineNode.PCollectionNode of this ExecutableStage.
java.util.Collection<PipelineNode.PCollectionNode>	getOutputPCollections() Returns the leaf PCollections of this ExecutableStage.
java.util.Collection<SideInputReference>	getSideInputs() Returns a set of descriptors that will be accessed by this ExecutableStage as side inputs.
java.util.Collection<TimerReference>	getTimers() Returns the set of descriptors that will consume and produce timers by this ExecutableStage.
java.util.Collection<PipelineNode.PTransformNode>	getTransforms() Get the transforms that perform processing within this ExecutableStage.
java.util.Collection<UserStateReference>	getUserStates() Returns the set of descriptors that will consume and produce user state by this ExecutableStage.
java.util.Collection<org.apache.beam.model.pipeline.v1.RunnerApi.ExecutableStagePayload.WireCoderSetting>	getWireCoderSettings() Returns a set of RunnerApi.ExecutableStagePayload.WireCoderSettings this stage executes in.
default org.apache.beam.model.pipeline.v1.RunnerApi.PTransform	toPTransform(java.lang.String uniqueName) Returns a composite RunnerApi.PTransform which is equivalent to this ExecutableStage as follows: The RunnerApi.PTransform.getSubtransformsList() is empty.

Field Detail

URN

static final java.lang.String URN

The URN identifying an ExecutableStage that has been converted to a RunnerApi.PTransform.

See Also:

Constant Field Values

DEFAULT_WIRE_CODER_SETTINGS

static final java.util.Collection<org.apache.beam.model.pipeline.v1.RunnerApi.ExecutableStagePayload.WireCoderSetting> DEFAULT_WIRE_CODER_SETTINGS

The default wire coder settings which returns an empty list, i.e., the WireCoder for each PCollection and timer will be a WINDOWED_VALUE coder.

Method Detail

getComponents

org.apache.beam.model.pipeline.v1.RunnerApi.Components getComponents()

Return the RunnerApi.Components required to execute this ExecutableStage.

This must contain all of the transforms returned by getTransforms() and the closure of all components that those transforms reference.

getEnvironment

org.apache.beam.model.pipeline.v1.RunnerApi.Environment getEnvironment()

Returns the RunnerApi.Environment this stage executes in.

An ExecutableStage consists of PTransforms which can all be executed within a single RunnerApi.Environment. The assumption made here is that runner-implemented transforms will be associated with these subgraphs by the overall graph topology, which will be handled by runners by performing already-required element routing and runner-side processing.

getWireCoderSettings

java.util.Collection<org.apache.beam.model.pipeline.v1.RunnerApi.ExecutableStagePayload.WireCoderSetting> getWireCoderSettings()

Returns a set of RunnerApi.ExecutableStagePayload.WireCoderSettings this stage executes in.

A RunnerApi.ExecutableStagePayload.WireCoderSetting consists of settings which is used to configure the type of the wire coder for a dedicated PCollection.

getInputPCollection

PipelineNode.PCollectionNode getInputPCollection()

Returns the root PipelineNode.PCollectionNode of this ExecutableStage. This ExecutableStage executes by reading elements from a Remote gRPC Read Node.

TODO(BEAM-4658): Add timers as input PCollections to executable stages.

getSideInputs

java.util.Collection<SideInputReference> getSideInputs()

Returns a set of descriptors that will be accessed by this ExecutableStage as side inputs.

getUserStates

java.util.Collection<UserStateReference> getUserStates()

Returns the set of descriptors that will consume and produce user state by this ExecutableStage.

getTimers

java.util.Collection<TimerReference> getTimers()

Returns the set of descriptors that will consume and produce timers by this ExecutableStage.

getOutputPCollections

java.util.Collection<PipelineNode.PCollectionNode> getOutputPCollections()

Returns the leaf PCollections of this ExecutableStage.

All of these PCollections are consumed by a PTransform which is not contained within this executable stage, and must be materialized at execution time by a Remote gRPC Write Transform.

TODO(BEAM-4658): Add timers as output PCollections to executable stages.

getTransforms

java.util.Collection<PipelineNode.PTransformNode> getTransforms()

Get the transforms that perform processing within this ExecutableStage.

toPTransform

default org.apache.beam.model.pipeline.v1.RunnerApi.PTransform toPTransform(java.lang.String uniqueName)

Returns a composite RunnerApi.PTransform which is equivalent to this ExecutableStage as follows:

• The RunnerApi.PTransform.getSubtransformsList() is empty. This ensures that executable stages are treated as primitive transforms.
• The only PCollections in the RunnerApi.PTransform.getInputsMap() is the result of getInputPCollection() and getSideInputs().
• The output PCollections in the values of RunnerApi.PTransform.getOutputsMap() are the PCollections returned by getOutputPCollections().
• The RunnerApi.PTransform.getSpec() contains an RunnerApi.ExecutableStagePayload with inputs and outputs equal to the PTransform's inputs and outputs, and transforms equal to the result of getTransforms().

The executable stage can be reconstructed from the resulting RunnerApi.ExecutableStagePayload via #fromPayload(ExecutableStagePayload).

fromPayload

static ExecutableStage fromPayload(org.apache.beam.model.pipeline.v1.RunnerApi.ExecutableStagePayload payload)

Return an ExecutableStage constructed from the provided RunnerApi.FunctionSpec representation.

See toPTransform(java.lang.String) for how the payload is constructed.

Note: The payload contains some information redundant with the RunnerApi.PTransform it is the payload of. The RunnerApi.ExecutableStagePayload should be sufficiently rich to construct a ProcessBundleDescriptor using only the payload.