Package org.apache.beam.sdk.values

Class PCollection<T>

java.lang.Object

org.apache.beam.sdk.values.PValueBase

org.apache.beam.sdk.values.PCollection<T>

Type Parameters:

T - the type of the elements of this PCollection

All Implemented Interfaces:

PInput, POutput, PValue

public class PCollection<T>
extends PValueBase
implements PValue

A PCollection<T> is an immutable collection of values of type T. A PCollection can contain either a bounded or unbounded number of elements. Bounded and unbounded PCollections are produced as the output of PTransforms (including root PTransforms like Read and Create), and can be passed as the inputs of other PTransforms.

Some root transforms produce bounded PCollections and others produce unbounded ones. For example, GenerateSequence.from(long) with GenerateSequence.to(long) produces a fixed set of integers, so it produces a bounded PCollection. GenerateSequence.from(long) without a GenerateSequence.to(long) produces all integers as an infinite stream, so it produces an unbounded PCollection.

Each element in a PCollection has an associated timestamp. Readers assign timestamps to elements when they create PCollections, and other PTransforms propagate these timestamps from their input to their output. See the documentation on BoundedSource.BoundedReader and UnboundedSource.UnboundedReader for more information on how these readers produce timestamps and watermarks.

Additionally, a PCollection has an associated WindowFn and each element is assigned to a set of windows. By default, the windowing function is GlobalWindows and all elements are assigned into a single default window. This default can be overridden with the Window PTransform.

See the individual PTransform subclasses for specific information on how they propagate timestamps and windowing.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	PCollection.IsBounded	The enumeration of cases for whether a PCollection is bounded.

Method Summary

Modifier and Type	Method	Description
<OutputT extends POutput> OutputT	apply(java.lang.String name, PTransform<? super PCollection<T>,OutputT> t)	Applies the given PTransform to this input PCollection, using name to identify this specific application of the transform.
<OutputT extends POutput> OutputT	apply(PTransform<? super PCollection<T>,OutputT> t)	of the PTransform.
static <T> PCollection<T>	createPrimitiveOutputInternal(Pipeline pipeline, WindowingStrategy<?,?> windowingStrategy, PCollection.IsBounded isBounded, @Nullable Coder<T> coder)	For internal use only; no backwards-compatibility guarantees.
static <T> PCollection<T>	createPrimitiveOutputInternal(Pipeline pipeline, WindowingStrategy<?,?> windowingStrategy, PCollection.IsBounded isBounded, @Nullable Coder<T> coder, TupleTag<?> tag)	For internal use only; no backwards-compatibility guarantees.
java.util.Map<TupleTag<?>,PValue>	expand()	Expands this POutput into a list of its component output PValues.
void	finishSpecifying(PInput input, PTransform<?,?> transform)	After building, finalizes this PValue to make it ready for running.
void	finishSpecifyingOutput(java.lang.String transformName, PInput input, PTransform<?,?> transform)	As part of applying the producing PTransform, finalizes this output to make it ready for being used as an input and for running.
Coder<T>	getCoder()	Returns the Coder used by this PCollection to encode and decode the values stored in it.
SerializableFunction<Row,T>	getFromRowFunction()	Returns the attached schema's fromRowFunction.
java.lang.String	getName()	Returns the name of this PCollection.
Schema	getSchema()	Returns the attached schema.
SerializableFunction<T,Row>	getToRowFunction()	Returns the attached schema's toRowFunction.
@Nullable TypeDescriptor<T>	getTypeDescriptor()	Returns a TypeDescriptor<T> with some reflective information about T, if possible.
WindowingStrategy<?,?>	getWindowingStrategy()	Returns the WindowingStrategy of this PCollection.
boolean	hasSchema()	Returns whether this PCollection has an attached schema.
PCollection.IsBounded	isBounded()
PCollection<T>	setCoder(Coder<T> coder)	Sets the Coder used by this PCollection to encode and decode the values stored in it.
PCollection<T>	setIsBoundedInternal(PCollection.IsBounded isBounded)	For internal use only; no backwards-compatibility guarantees.
PCollection<T>	setName(java.lang.String name)	Sets the name of this PCollection.
PCollection<T>	setRowSchema(Schema schema)	Sets a schema on this PCollection.
PCollection<T>	setSchema(Schema schema, TypeDescriptor<T> typeDescriptor, SerializableFunction<T,Row> toRowFunction, SerializableFunction<Row,T> fromRowFunction)	Sets a Schema on this PCollection.
PCollection<T>	setTypeDescriptor(TypeDescriptor<T> typeDescriptor)	Sets the TypeDescriptor<T> for this PCollection<T>.
PCollection<T>	setWindowingStrategyInternal(WindowingStrategy<?,?> windowingStrategy)	For internal use only; no backwards-compatibility guarantees.

Methods inherited from class org.apache.beam.sdk.values.PValueBase

getKindString, getPipeline, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.beam.sdk.values.PInput

getPipeline

Methods inherited from interface org.apache.beam.sdk.values.POutput

getPipeline

Method Detail

finishSpecifyingOutput

public void finishSpecifyingOutput(java.lang.String transformName,
                                   PInput input,
                                   PTransform<?,?> transform)

Description copied from interface: POutput

As part of applying the producing PTransform, finalizes this output to make it ready for being used as an input and for running.

This includes ensuring that all PCollections have Coders specified or defaulted.

Automatically invoked whenever this POutput is output, after POutput.finishSpecifyingOutput(String, PInput, PTransform) has been called on each component PValue returned by POutput.expand().

Specified by:

finishSpecifyingOutput in interface POutput

Overrides:

finishSpecifyingOutput in class PValueBase

finishSpecifying

public void finishSpecifying(PInput input,
                             PTransform<?,?> transform)

After building, finalizes this PValue to make it ready for running. Automatically invoked whenever the PValue is "used" (e.g., when apply() is called on it) and when the Pipeline is run (useful if this is a PValue with no consumers).

Specified by:

finishSpecifying in interface PValue

Overrides:

finishSpecifying in class PValueBase

Parameters:

input - the PInput the PTransform was applied to to produce this output

transform - the PTransform that produced this PValue

getTypeDescriptor

public @Nullable TypeDescriptor<T> getTypeDescriptor()

Returns a TypeDescriptor<T> with some reflective information about T, if possible. May return null if no information is available. Subclasses may override this to enable better Coder inference.

getName

public java.lang.String getName()

Returns the name of this PCollection.

By default, the name of a PCollection is based on the name of the PTransform that produces it. It can be specified explicitly by calling setName(java.lang.String).

Specified by:

getName in interface PValue

Overrides:

getName in class PValueBase

Throws:

java.lang.IllegalStateException - if the name hasn't been set yet

expand

public final java.util.Map<TupleTag<?>,PValue> expand()

Description copied from interface: PValue

Expands this POutput into a list of its component output PValues.

• A PValue expands to itself.
• A tuple or list of PValues (such as PCollectionTuple or PCollectionList) expands to its component PValue PValues.

Not intended to be invoked directly by user code..

Specified by:

expand in interface PInput

Specified by:

expand in interface POutput

Specified by:

expand in interface PValue

setName

public PCollection<T> setName(java.lang.String name)

Sets the name of this PCollection. Returns this.

Overrides:

setName in class PValueBase

Throws:

java.lang.IllegalStateException - if this PCollection has already been finalized and may no longer be set. Once apply(org.apache.beam.sdk.transforms.PTransform<? super org.apache.beam.sdk.values.PCollection<T>, OutputT>) has been called, this will be the case.

getCoder

public Coder<T> getCoder()

Returns the Coder used by this PCollection to encode and decode the values stored in it.

Throws:

java.lang.IllegalStateException - if the Coder hasn't been set, and couldn't be inferred.

setCoder

public PCollection<T> setCoder(Coder<T> coder)

Sets the Coder used by this PCollection to encode and decode the values stored in it. Returns this.

Throws:

java.lang.IllegalStateException - if this PCollection has already been finalized and may no longer be set. Once apply(org.apache.beam.sdk.transforms.PTransform<? super org.apache.beam.sdk.values.PCollection<T>, OutputT>) has been called, this will be the case.

setRowSchema

@Experimental(SCHEMAS)
public PCollection<T> setRowSchema(Schema schema)

Sets a schema on this PCollection.

Can only be called on a PCollection.

setSchema

@Experimental(SCHEMAS)
public PCollection<T> setSchema(Schema schema,
                                TypeDescriptor<T> typeDescriptor,
                                SerializableFunction<T,Row> toRowFunction,
                                SerializableFunction<Row,T> fromRowFunction)

Sets a Schema on this PCollection.

hasSchema

@Experimental(SCHEMAS)
public boolean hasSchema()

Returns whether this PCollection has an attached schema.

getSchema

@Experimental(SCHEMAS)
public Schema getSchema()

Returns the attached schema.

getToRowFunction

@Experimental(SCHEMAS)
public SerializableFunction<T,Row> getToRowFunction()

Returns the attached schema's toRowFunction.

getFromRowFunction

@Experimental(SCHEMAS)
public SerializableFunction<Row,T> getFromRowFunction()

Returns the attached schema's fromRowFunction.

apply

public <OutputT extends POutput> OutputT apply(PTransform<? super PCollection<T>,OutputT> t)

of the PTransform.

Returns:

the output of the applied PTransform

apply

public <OutputT extends POutput> OutputT apply(java.lang.String name,
                                               PTransform<? super PCollection<T>,OutputT> t)

Applies the given PTransform to this input PCollection, using name to identify this specific application of the transform. This name is used in various places, including the monitoring UI, logging, and to stably identify this application node in the job graph.

Returns:

the output of the applied PTransform

getWindowingStrategy

public WindowingStrategy<?,?> getWindowingStrategy()

Returns the WindowingStrategy of this PCollection.

isBounded

public PCollection.IsBounded isBounded()

setTypeDescriptor

public PCollection<T> setTypeDescriptor(TypeDescriptor<T> typeDescriptor)

Sets the TypeDescriptor<T> for this PCollection<T>. This may allow the enclosing PCollectionTuple, PCollectionList, or PTransform<?, PCollection<T>>, etc., to provide more detailed reflective information.

setWindowingStrategyInternal

@Internal
public PCollection<T> setWindowingStrategyInternal(WindowingStrategy<?,?> windowingStrategy)

For internal use only; no backwards-compatibility guarantees.

setIsBoundedInternal

@Internal
public PCollection<T> setIsBoundedInternal(PCollection.IsBounded isBounded)

For internal use only; no backwards-compatibility guarantees.

createPrimitiveOutputInternal

@Internal
public static <T> PCollection<T> createPrimitiveOutputInternal(Pipeline pipeline,
                                                               WindowingStrategy<?,?> windowingStrategy,
                                                               PCollection.IsBounded isBounded,
                                                               @Nullable Coder<T> coder)

For internal use only; no backwards-compatibility guarantees.

createPrimitiveOutputInternal

@Internal
public static <T> PCollection<T> createPrimitiveOutputInternal(Pipeline pipeline,
                                                               WindowingStrategy<?,?> windowingStrategy,
                                                               PCollection.IsBounded isBounded,
                                                               @Nullable Coder<T> coder,
                                                               TupleTag<?> tag)

For internal use only; no backwards-compatibility guarantees.