com.google.cloud.dataflow.sdk.transforms

Class Combine

java.lang.Object

com.google.cloud.dataflow.sdk.transforms.Combine

public class Combine
extends java.lang.Object

PTransforms for combining PCollection elements globally and per-key.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	Combine.AccumulatingCombineFn<VI,VA extends Combine.AccumulatingCombineFn.Accumulator<VI,VA,VO>,VO> A CombineFn that uses a subclass of Combine.AccumulatingCombineFn.Accumulator as its accumulator type.
static class	Combine.CombineFn<VI,VA,VO> A CombineFn<VI, VA, VO> specifies how to combine a collection of input values of type VI into a single output value of type VO.
static class	Combine.Globally<VI,VO> Combine.Globally<VI, VO> takes a PCollection<VI> and returns a PCollection<VO> whose single element is the result of combining all the elements of the input PCollection, using a specified CombineFn.
static class	Combine.GroupedValues<K,VI,VO> GroupedValues<K, VI, VO> takes a PCollection<KV<K, Iterable<VI>>>, such as the result of GroupByKey, applies a specified KeyedCombineFn to each of the input KV<K, Iterable<VI>> elements to produce a combined output KV<K, VO> element, and returns a PCollection<KV<K, VO>> containing all the combined output elements.
static class	Combine.KeyedCombineFn<K,VI,VA,VO> A KeyedCombineFn<K, VI, VA, VO> specifies how to combine a collection of input values of type VI, associated with a key of type K, into a single output value of type VO.
static class	Combine.PerKey<K,VI,VO> PerKey<K, VI, VO> takes a PCollection<KV<K, VI>>, groups it by key, applies a combining function to the VI values associated with each key to produce a combined VO value, and returns a PCollection<KV<K, VO>> representing a map from each distinct key of the input PCollection to the corresponding combined value.

Constructor Summary

Constructors

Constructor and Description
Combine()

Method Summary

Methods

Modifier and Type	Method and Description
static <VI,VO> Combine.Globally<VI,VO>	globally(Combine.CombineFn<? super VI,?,VO> fn) Returns a Combine.Globally PTransform that uses the given CombineFn to combine all the elements of the input PCollection into a singleton PCollection value.
static <V> Combine.Globally<V,V>	globally(SerializableFunction<java.lang.Iterable<V>,V> combiner) Returns a Combine.Globally PTransform that uses the given SerializableFunction to combine all the elements of the input PCollection into a singleton PCollection value.
static <K,VI,VO> Combine.GroupedValues<K,VI,VO>	groupedValues(Combine.CombineFn<? super VI,?,VO> fn) Returns a Combine.GroupedValues PTransform that takes a PCollection of KVs where a key maps to an Iterable of values, e.g., the result of a GroupByKey, then uses the given CombineFn to combine all the values associated with a key, ignoring the key.
static <K,VI,VO> Combine.GroupedValues<K,VI,VO>	groupedValues(Combine.KeyedCombineFn<? super K,? super VI,?,VO> fn) Returns a Combine.GroupedValues PTransform that takes a PCollection of KVs where a key maps to an Iterable of values, e.g., the result of a GroupByKey, then uses the given KeyedCombineFn to combine all the values associated with each key.
static <K,V> Combine.GroupedValues<K,V,V>	groupedValues(SerializableFunction<java.lang.Iterable<V>,V> fn) Returns a Combine.GroupedValues PTransform that takes a PCollection of KVs where a key maps to an Iterable of values, e.g., the result of a GroupByKey, then uses the given SerializableFunction to combine all the values associated with a key, ignoring the key.
static <K,VI,VO> Combine.PerKey<K,VI,VO>	perKey(Combine.CombineFn<? super VI,?,VO> fn) Returns a Combine.PerKey PTransform that first groups its input PCollection of KVs by keys and windows, then invokes the given function on each of the values lists to produce a combined value, and then returns a PCollection of KVs mapping each distinct key to its combined value for each window.
static <K,VI,VO> Combine.PerKey<K,VI,VO>	perKey(Combine.KeyedCombineFn<? super K,? super VI,?,VO> fn) Returns a Combine.PerKey PTransform that first groups its input PCollection of KVs by keys and windows, then invokes the given function on each of the key/values-lists pairs to produce a combined value, and then returns a PCollection of KVs mapping each distinct key to its combined value for each window.
static <K,V> Combine.PerKey<K,V,V>	perKey(SerializableFunction<java.lang.Iterable<V>,V> fn) Returns a Combine.PerKey PTransform that first groups its input PCollection of KVs by keys and windows, then invokes the given function on each of the values lists to produce a combined value, and then returns a PCollection of KVs mapping each distinct key to its combined value for each window.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Combine

public Combine()

Method Detail

globally

public static <V> Combine.Globally<V,V> globally(SerializableFunction<java.lang.Iterable<V>,V> combiner)

Returns a Combine.Globally PTransform that uses the given SerializableFunction to combine all the elements of the input PCollection into a singleton PCollection value. The types of the input elements and the output value must be the same.

If the input PCollection is empty, the ouput will contain a the default value of the combining function if the input is windowed into the GlobalWindows; otherwise, the output will be empty. Note: this behavior is subject to change.

See Combine.Globally for more information.

globally

public static <VI,VO> Combine.Globally<VI,VO> globally(Combine.CombineFn<? super VI,?,VO> fn)

Returns a Combine.Globally PTransform that uses the given CombineFn to combine all the elements of the input PCollection into a singleton PCollection value. The types of the input elements and the output value can differ. If the input PCollection is empty, the ouput will contain a the default value of the combining function if the input is windowed into the GlobalWindows; otherwise, the output will be empty. Note: this behavior is subject to change.

See Combine.Globally for more information.

perKey

public static <K,V> Combine.PerKey<K,V,V> perKey(SerializableFunction<java.lang.Iterable<V>,V> fn)

Returns a Combine.PerKey PTransform that first groups its input PCollection of KVs by keys and windows, then invokes the given function on each of the values lists to produce a combined value, and then returns a PCollection of KVs mapping each distinct key to its combined value for each window.

Each output element is in the window by which its corresponding input was grouped, and has the timestamp of the end of that window. The output PCollection has the same WindowFn as the input.

See Combine.PerKey for more information.

perKey

public static <K,VI,VO> Combine.PerKey<K,VI,VO> perKey(Combine.CombineFn<? super VI,?,VO> fn)

Returns a Combine.PerKey PTransform that first groups its input PCollection of KVs by keys and windows, then invokes the given function on each of the values lists to produce a combined value, and then returns a PCollection of KVs mapping each distinct key to its combined value for each window.

Each output element is in the window by which its corresponding input was grouped, and has the timestamp of the end of that window. The output PCollection has the same WindowFn as the input.

See Combine.PerKey for more information.

perKey

public static <K,VI,VO> Combine.PerKey<K,VI,VO> perKey(Combine.KeyedCombineFn<? super K,? super VI,?,VO> fn)

Returns a Combine.PerKey PTransform that first groups its input PCollection of KVs by keys and windows, then invokes the given function on each of the key/values-lists pairs to produce a combined value, and then returns a PCollection of KVs mapping each distinct key to its combined value for each window.

Each output element is in the window by which its corresponding input was grouped, and has the timestamp of the end of that window. The output PCollection has the same WindowFn as the input.

See Combine.PerKey for more information.

groupedValues

public static <K,V> Combine.GroupedValues<K,V,V> groupedValues(SerializableFunction<java.lang.Iterable<V>,V> fn)

Returns a Combine.GroupedValues PTransform that takes a PCollection of KVs where a key maps to an Iterable of values, e.g., the result of a GroupByKey, then uses the given SerializableFunction to combine all the values associated with a key, ignoring the key. The type of the input and output values must be the same.

Each output element has the same timestamp and is in the same window as its corresponding input element, and the output PCollection has the same WindowFn associated with it as the input.

See Combine.GroupedValues for more information.

Note that perKey(SerializableFunction) is typically more convenient to use than GroupByKey followed by groupedValues(...).

groupedValues

public static <K,VI,VO> Combine.GroupedValues<K,VI,VO> groupedValues(Combine.CombineFn<? super VI,?,VO> fn)

Returns a Combine.GroupedValues PTransform that takes a PCollection of KVs where a key maps to an Iterable of values, e.g., the result of a GroupByKey, then uses the given CombineFn to combine all the values associated with a key, ignoring the key. The types of the input and output values can differ.

Each output element has the same timestamp and is in the same window as its corresponding input element, and the output PCollection has the same WindowFn associated with it as the input.

See Combine.GroupedValues for more information.

Note that perKey(CombineFn) is typically more convenient to use than GroupByKey followed by groupedValues(...).

groupedValues

public static <K,VI,VO> Combine.GroupedValues<K,VI,VO> groupedValues(Combine.KeyedCombineFn<? super K,? super VI,?,VO> fn)

Returns a Combine.GroupedValues PTransform that takes a PCollection of KVs where a key maps to an Iterable of values, e.g., the result of a GroupByKey, then uses the given KeyedCombineFn to combine all the values associated with each key. The combining function is provided the key. The types of the input and output values can differ.

Each output element has the same timestamp and is in the same window as its corresponding input element, and the output PCollection has the same WindowFn associated with it as the input.

See Combine.GroupedValues for more information.

Note that perKey(KeyedCombineFn) is typically more convenient to use than GroupByKey followed by groupedValues(...).