com.google.cloud.dataflow.sdk.transforms

Class Partition<T>

java.lang.Object

com.google.cloud.dataflow.sdk.transforms.PTransform<PCollection<T>,PCollectionList<T>>

com.google.cloud.dataflow.sdk.transforms.Partition<T>

Type Parameters:

T - the type of the elements of the input and output PCollections

All Implemented Interfaces:

java.io.Serializable

public class Partition<T>
extends PTransform<PCollection<T>,PCollectionList<T>>

Partition takes a PCollection<T> and a PartitionFn, uses the PartitionFn to split the elements of the input PCollection into N partitions, and returns a PCollectionList<T> that bundles N PCollection<T>s containing the split elements.

Example of use:

 PCollection<Student> students = ...;
 // Split students up into 10 partitions, by percentile:
 PCollectionList<Student> studentsByPercentile =
     students.apply(Partition.of(10, new PartitionFn<Student>() {
         public int partitionFor(Student student, int numPartitions) {
             return student.getPercentile()  // 0..99
                  * numPartitions / 100;
         }}))
 for (int i = 0; i < 10; i++) {
   PCollection<Student> partition = studentsByPercentile.get(i);
   ...
 }
  

By default, the Coder of each of the PCollections in the output PCollectionList is the same as the Coder of the input PCollection.

Each output element has the same timestamp and is in the same windows as its corresponding input element, and each output PCollection has the same WindowFn associated with it as the input.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	Partition.PartitionFn<T> A function object that chooses an output partition for an element.

Field Summary

Fields inherited from class com.google.cloud.dataflow.sdk.transforms.PTransform

name

Method Summary

Methods

Modifier and Type	Method and Description
PCollectionList<T>	apply(PCollection<T> in) Applies this PTransform on the given Input, and returns its Output.
static <T> Partition<T>	of(int numPartitions, Partition.PartitionFn<? super T> partitionFn) Returns a new Partition PTransform that divides its input PCollection into the given number of partitions, using the given partitioning function.

Methods inherited from class com.google.cloud.dataflow.sdk.transforms.PTransform

finishSpecifying, getCoderRegistry, getDefaultName, getDefaultOutputCoder, getDefaultOutputCoder, getInput, getKindString, getName, getOutput, getPipeline, setName, setPipeline, toString, withName

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Method Detail

of

public static <T> Partition<T> of(int numPartitions,
                  Partition.PartitionFn<? super T> partitionFn)

Returns a new Partition PTransform that divides its input PCollection into the given number of partitions, using the given partitioning function.

Parameters:

numPartitions - the number of partitions to divide the input PCollection into

partitionFn - the function to invoke on each element to choose its output partition

Throws:

java.lang.IllegalArgumentException - if numPartitions <= 0

apply

public PCollectionList<T> apply(PCollection<T> in)

Description copied from class: PTransform

Applies this PTransform on the given Input, and returns its Output.

Composite transforms, which are defined in terms of other transforms, should return the output of one of the composed transforms. Non-composite transforms, which do not apply any transforms internally, should return a new unbound output and register evaluators (via backend-specific registration methods).

The default implementation throws an exception. A derived class must either implement apply, or else each runner must supply a custom implementation via PipelineRunner.apply(com.google.cloud.dataflow.sdk.transforms.PTransform<Input, Output>, Input).

Overrides:

apply in class PTransform<PCollection<T>,PCollectionList<T>>