com.google.cloud.dataflow.sdk.values

Class PCollectionTuple

java.lang.Object

com.google.cloud.dataflow.sdk.values.PCollectionTuple

All Implemented Interfaces:

PInput, POutput

public class PCollectionTuple
extends Object
implements PInput, POutput

A PCollectionTuple is an immutable tuple of heterogeneously-typed PCollections, "keyed" by TupleTags. A PCollectionTuple can be used as the input or output of a PTransform taking or producing multiple PCollection inputs or outputs that can be of different types, for instance a ParDo with side outputs.

PCollectionTuples can be created and accessed like follows:

 PCollection<String> pc1 = ...;
 PCollection<Integer> pc2 = ...;
 PCollection<Iterable<String>> pc3 = ...;

 // Create TupleTags for each of the PCollections to put in the
 // PCollectionTuple (the type of the TupleTag enables tracking the
 // static type of each of the PCollections in the PCollectionTuple):
 TupleTag<String> tag1 = new TupleTag<>();
 TupleTag<Integer> tag2 = new TupleTag<>();
 TupleTag<Iterable<String>> tag3 = new TupleTag<>();

 // Create a PCollectionTuple with three PCollections:
 PCollectionTuple pcs =
     PCollectionTuple.of(tag1, pc1)
                     .and(tag2, pc2)
                     .and(tag3, pc3);

 // Create an empty PCollectionTuple:
 Pipeline p = ...;
 PCollectionTuple pcs2 = PCollectionTuple.empty(p);

 // Get PCollections out of a PCollectionTuple, using the same tags
 // that were used to put them in:
 PCollection<Integer> pcX = pcs.get(tag2);
 PCollection<String> pcY = pcs.get(tag1);
 PCollection<Iterable<String>> pcZ = pcs.get(tag3);

 // Get a map of all PCollections in a PCollectionTuple:
 Map<TupleTag<?>, PCollection<?>> allPcs = pcs.getAll();
  

Method Summary

Methods

Modifier and Type	Method and Description
<T> PCollectionTuple	and(TupleTag<T> tag, PCollection<T> pc) Returns a new PCollectionTuple that has all the PCollections and tags of this PCollectionTuple plus the given PCollection and tag.
<OutputT extends POutput> OutputT	apply(PTransform<PCollectionTuple,OutputT> t) Applies the given PTransform to this input PCollectionTuple, and returns the PTransform's Output.
static PCollectionTuple	empty(Pipeline pipeline) Returns an empty PCollectionTuple that is part of the given Pipeline.
Collection<? extends PValue>	expand() Expands this PInput into a list of its component input PValues.
void	finishSpecifying() After building, finalizes this PInput to make it ready for being used as an input to a PTransform.
void	finishSpecifyingOutput() As part of applying the producing PTransform, finalizes this output to make it ready for being used as an input and for running.
<T> PCollection<T>	get(TupleTag<T> tag) Returns the PCollection with the given tag in this PCollectionTuple.
Map<TupleTag<?>,PCollection<?>>	getAll() Returns an immutable Map from TupleTag to corresponding PCollection, for all the members of this PCollectionTuple.
Pipeline	getPipeline() Returns the owning Pipeline of this PInput.
<T> boolean	has(TupleTag<T> tag) Returns whether this PCollectionTuple contains a PCollection with the given tag.
static <T> PCollectionTuple	of(TupleTag<T> tag, PCollection<T> pc) Returns a singleton PCollectionTuple containing the given PCollection keyed by the given TupleTag.
static PCollectionTuple	ofPrimitiveOutputsInternal(Pipeline pipeline, TupleTagList outputTags, com.google.cloud.dataflow.sdk.util.WindowingStrategy<?,?> windowingStrategy, PCollection.IsBounded isBounded) Returns a PCollectionTuple with each of the given tags mapping to a new output PCollection.
void	recordAsOutput(AppliedPTransform<?,?,?> transform) Records that this POutput is an output of the given PTransform.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

empty

public static PCollectionTuple empty(Pipeline pipeline)

Returns an empty PCollectionTuple that is part of the given Pipeline.

Longer PCollectionTuples can be created by calling and(com.google.cloud.dataflow.sdk.values.TupleTag<T>, com.google.cloud.dataflow.sdk.values.PCollection<T>) on the result.

of

public static <T> PCollectionTuple of(TupleTag<T> tag,
                      PCollection<T> pc)

Returns a singleton PCollectionTuple containing the given PCollection keyed by the given TupleTag.

Longer PCollectionTuples can be created by calling and(com.google.cloud.dataflow.sdk.values.TupleTag<T>, com.google.cloud.dataflow.sdk.values.PCollection<T>) on the result.

and

public <T> PCollectionTuple and(TupleTag<T> tag,
                       PCollection<T> pc)

Returns a new PCollectionTuple that has all the PCollections and tags of this PCollectionTuple plus the given PCollection and tag.

The given TupleTag should not already be mapped to a PCollection in this PCollectionTuple.

All the PCollections in the resulting PCollectionTuple must be part of the same Pipeline.

has

public <T> boolean has(TupleTag<T> tag)

Returns whether this PCollectionTuple contains a PCollection with the given tag.

get

public <T> PCollection<T> get(TupleTag<T> tag)

Returns the PCollection with the given tag in this PCollectionTuple. Throws IllegalArgumentException if there is no such PCollection, i.e., !has(tag).

getAll

public Map<TupleTag<?>,PCollection<?>> getAll()

Returns an immutable Map from TupleTag to corresponding PCollection, for all the members of this PCollectionTuple.

apply

public <OutputT extends POutput> OutputT apply(PTransform<PCollectionTuple,OutputT> t)

Applies the given PTransform to this input PCollectionTuple, and returns the PTransform's Output.

ofPrimitiveOutputsInternal

public static PCollectionTuple ofPrimitiveOutputsInternal(Pipeline pipeline,
                                          TupleTagList outputTags,
                                          com.google.cloud.dataflow.sdk.util.WindowingStrategy<?,?> windowingStrategy,
                                          PCollection.IsBounded isBounded)

Returns a PCollectionTuple with each of the given tags mapping to a new output PCollection.

For use by primitive transformations only.

getPipeline

public Pipeline getPipeline()

Description copied from interface: PInput

Returns the owning Pipeline of this PInput.

Specified by:

getPipeline in interface PInput

Specified by:

getPipeline in interface POutput

expand

public Collection<? extends PValue> expand()

Description copied from interface: PInput

Expands this PInput into a list of its component input PValues.

A PValue expands to itself.

A tuple or list of PValues (e.g., PCollectionTuple, and PCollectionList) expands to its component PValues.

Not intended to be invoked directly by user code.

Specified by:

expand in interface PInput

Specified by:

expand in interface POutput

recordAsOutput

public void recordAsOutput(AppliedPTransform<?,?,?> transform)

Description copied from interface: POutput

Records that this POutput is an output of the given PTransform.

For a compound POutput, it is advised to call this method on each component POutput.

This is not intended to be invoked by user code, but is automatically invoked as part of applying the producing PTransform.

Specified by:

recordAsOutput in interface POutput

finishSpecifying

public void finishSpecifying()

Description copied from interface: PInput

After building, finalizes this PInput to make it ready for being used as an input to a PTransform.

Automatically invoked whenever apply() is invoked on this PInput, so users do not normally call this explicitly.

Specified by:

finishSpecifying in interface PInput

finishSpecifyingOutput

public void finishSpecifyingOutput()

Description copied from interface: POutput

As part of applying the producing PTransform, finalizes this output to make it ready for being used as an input and for running.

This includes ensuring that all PCollections have Coders specified or defaulted.

Automatically invoked whenever this POutput is used as a PInput to another PTransform, or if never used as a PInput, when Pipeline.run() is called, so users do not normally call this explicitly.

Specified by:

finishSpecifyingOutput in interface POutput