T - the type of the elements of this PCollectionpublic class PCollection<T> extends TypedPValue<T>
PCollection<T> is an immutable collection of values of type
T. A PCollection can contain either a bounded or unbounded
number of elements. Bounded and unbounded PCollections are produced
as the output of PTransforms
(including root PTransforms like
TextIO.Read,
PubsubIO.Read and
Create), and can
be passed as the inputs of other PTransforms.
Some root transforms produce bounded PCollections and others
produce unbounded ones. For example,
TextIO.Read reads a static set
of files, so it produces a bounded PCollection.
PubsubIO.Read, on the other hand,
receives a potentially infinite stream of Pubsub messages, so it produces
an unbounded PCollection.
Each element in a PCollection may have an associated implicit
timestamp. Readers assign timestamps to elements when they create
PCollections, and other PTransforms propagate these
timestamps from their input to their output. For example, PubsubIO.Read
assigns pubsub message timestamps to elements, and TextIO.Read assigns
the default value Long.MIN_VALUE to elements. User code can
explicitly assign timestamps to elements with
DoFn.Context.outputWithTimestamp(OutputT, org.joda.time.Instant).
Additionally, a PCollection has an associated
WindowFn and each element is assigned to a set of windows.
By default, the windowing function is
GlobalWindows
and all elements are assigned into a single default window.
This default can be overridden with the
Window
PTransform. Dataflow pipelines run in classic batch MapReduce style
with the default GlobalWindow strategy if timestamps are ignored.
See the individual PTransform subclasses for specific information
on how they propagate timestamps and windowing.
| Modifier and Type | Class and Description |
|---|---|
static class |
PCollection.IsBounded
The PCollection IsBounded property.
|
| Modifier and Type | Method and Description |
|---|---|
<OutputT extends POutput> |
apply(PTransform<? super PCollection<T>,OutputT> t)
Like
apply(String, PTransform) but defaulting to the name
of the PTransform. |
<OutputT extends POutput> |
apply(String name,
PTransform<? super PCollection<T>,OutputT> t)
Applies the given
PTransform to this input PCollection<T>,
using name to identify this specific application of the transform. |
static <T> PCollection<T> |
createPrimitiveOutputInternal(Pipeline pipeline,
WindowingStrategy<?,?> windowingStrategy,
PCollection.IsBounded isBounded)
Creates and returns a new PCollection for a primitive output.
|
Coder<T> |
getCoder()
Returns the Coder used by this PCollection to encode and decode
the values stored in it.
|
String |
getName()
Returns the name of this PCollection.
|
WindowingStrategy<?,?> |
getWindowingStrategy()
Returns the
WindowingStrategy of this PCollection. |
PCollection.IsBounded |
isBounded() |
PCollection<T> |
setCoder(Coder<T> coder)
Sets the Coder used by this PCollection to encode and decode the
values stored in it.
|
PCollection<T> |
setIsBoundedInternal(PCollection.IsBounded isBounded)
Sets the
PCollection.IsBounded of this PCollection. |
PCollection<T> |
setName(String name)
Sets the name of this PCollection.
|
PCollection<T> |
setTypeDescriptorInternal(TypeDescriptor<T> typeDescriptor)
Sets the
TypeDescriptor<T> for this PCollection<T>, so that
the enclosing PCollectionTuple, PCollectionList<T>,
or PTransform<?, PCollection<T>>, etc., can provide
more detailed reflective information. |
PCollection<T> |
setWindowingStrategyInternal(WindowingStrategy<?,?> windowingStrategy)
Sets the
WindowingStrategy of this PCollection. |
finishSpecifying, getTypeDescriptorexpand, getKindString, isFinishedSpecifyingInternal, recordAsOutput, recordAsOutput, toStringfinishSpecifyingOutput, getPipeline, getProducingTransformInternalclone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, waitgetProducingTransformInternalexpand, finishSpecifyingOutput, getPipeline, recordAsOutputexpand, getPipelinepublic String getName()
By default, the name of a PCollection is based on the name of the
PTransform that produces it. It can be specified explicitly by
calling setName(java.lang.String).
getName in interface PValuegetName in class PValueBaseIllegalStateException - if the name hasn't been set yetpublic PCollection<T> setName(String name)
this.setName in class PValueBaseIllegalStateException - if this PCollection has already been
finalized and is no longer settable, e.g., by having
apply() called on itpublic Coder<T> getCoder()
getCoder in class TypedPValue<T>IllegalStateException - if the Coder hasn't been set, and
couldn't be inferredpublic PCollection<T> setCoder(Coder<T> coder)
this.setCoder in class TypedPValue<T>IllegalStateException - if this PCollection has already
been finalized and is no longer settable, e.g., by having
apply() called on itpublic <OutputT extends POutput> OutputT apply(PTransform<? super PCollection<T>,OutputT> t)
apply(String, PTransform) but defaulting to the name
of the PTransform.public <OutputT extends POutput> OutputT apply(String name, PTransform<? super PCollection<T>,OutputT> t)
PTransform to this input PCollection<T>,
using name to identify this specific application of the transform.
This name is used in various places, including the monitoring UI, logging,
and to stably identify this application node in the job graph.public WindowingStrategy<?,?> getWindowingStrategy()
WindowingStrategy of this PCollection.public PCollection.IsBounded isBounded()
public PCollection<T> setTypeDescriptorInternal(TypeDescriptor<T> typeDescriptor)
TypeDescriptor<T> for this PCollection<T>, so that
the enclosing PCollectionTuple, PCollectionList<T>,
or PTransform<?, PCollection<T>>, etc., can provide
more detailed reflective information.setTypeDescriptorInternal in class TypedPValue<T>public PCollection<T> setWindowingStrategyInternal(WindowingStrategy<?,?> windowingStrategy)
public PCollection<T> setIsBoundedInternal(PCollection.IsBounded isBounded)
public static <T> PCollection<T> createPrimitiveOutputInternal(Pipeline pipeline, WindowingStrategy<?,?> windowingStrategy, PCollection.IsBounded isBounded)
For use by primitive transformations only.