Class Combine.PerKey<K,InputT,OutputT>
- java.lang.Object
-
- org.apache.beam.sdk.transforms.PTransform<PCollection<KV<K,InputT>>,PCollection<KV<K,OutputT>>>
-
- org.apache.beam.sdk.transforms.Combine.PerKey<K,InputT,OutputT>
-
- Type Parameters:
K
- the type of the keys of the input and outputPCollection
sInputT
- the type of the values of the inputPCollection
OutputT
- the type of the values of the outputPCollection
- All Implemented Interfaces:
java.io.Serializable
,HasDisplayData
- Enclosing class:
- Combine
public static class Combine.PerKey<K,InputT,OutputT> extends PTransform<PCollection<KV<K,InputT>>,PCollection<KV<K,OutputT>>>
PerKey<K, InputT, OutputT>
takes aPCollection<KV<K, InputT>>
, groups it by key, applies a combining function to theInputT
values associated with each key to produce a combinedOutputT
value, and returns aPCollection<KV<K, OutputT>>
representing a map from each distinct key of the inputPCollection
to the corresponding combined value.InputT
andOutputT
are often the same.This is a concise shorthand for an application of
GroupByKey
followed by an application ofCombine.GroupedValues
. See those operations for more details on how keys are compared for equality and on the defaultCoder
for the output.Example of use:
PCollection<KV<String, Double>> salesRecords = ...; PCollection<KV<String, Double>> totalSalesPerPerson = salesRecords.apply(Combine.<String, Double, Double>perKey( Sum.ofDoubles()));
Each output element is in the window by which its corresponding input was grouped, and has the timestamp of the end of that window. The output
PCollection
has the sameWindowFn
as the input.- See Also:
- Serialized Form
-
-
Field Summary
-
Fields inherited from class org.apache.beam.sdk.transforms.PTransform
name, resourceHints
-
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description PCollection<KV<K,OutputT>>
expand(PCollection<KV<K,InputT>> input)
Override this method to specify how thisPTransform
should be expanded on the givenInputT
.java.util.Map<TupleTag<?>,PValue>
getAdditionalInputs()
Returns the side inputs of thisCombine
, tagged with the tag of thePCollectionView
.CombineFnBase.GlobalCombineFn<? super InputT,?,OutputT>
getFn()
Returns theCombineFnBase.GlobalCombineFn
used by this Combine operation.protected java.lang.String
getKindString()
Returns the name to use by default for thisPTransform
(not including the names of any enclosingPTransform
s).java.util.List<PCollectionView<?>>
getSideInputs()
Returns the side inputs used by this Combine operation.void
populateDisplayData(DisplayData.Builder builder)
Register display data for the given transform or component.Combine.PerKeyWithHotKeyFanout<K,InputT,OutputT>
withHotKeyFanout(int hotKeyFanout)
LikewithHotKeyFanout(SerializableFunction)
, but returning the given constant value for every key.Combine.PerKeyWithHotKeyFanout<K,InputT,OutputT>
withHotKeyFanout(SerializableFunction<? super K,java.lang.Integer> hotKeyFanout)
If a single key has disproportionately many values, it may become a bottleneck, especially in streaming mode.Combine.PerKey<K,InputT,OutputT>
withSideInputs(java.lang.Iterable<? extends PCollectionView<?>> sideInputs)
Returns aPTransform
identical to this, but with the specified side inputs to use inCombineWithContext.CombineFnWithContext
.Combine.PerKey<K,InputT,OutputT>
withSideInputs(PCollectionView<?>... sideInputs)
Returns aPTransform
identical to this, but with the specified side inputs to use inCombineWithContext.CombineFnWithContext
.-
Methods inherited from class org.apache.beam.sdk.transforms.PTransform
compose, compose, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getName, getResourceHints, setResourceHints, toString, validate, validate
-
-
-
-
Method Detail
-
getKindString
protected java.lang.String getKindString()
Description copied from class:PTransform
Returns the name to use by default for thisPTransform
(not including the names of any enclosingPTransform
s).By default, returns the base name of this
PTransform
's class.The caller is responsible for ensuring that names of applied
PTransform
s are unique, e.g., by adding a uniquifying suffix when needed.- Overrides:
getKindString
in classPTransform<PCollection<KV<K,InputT>>,PCollection<KV<K,OutputT>>>
-
withSideInputs
public Combine.PerKey<K,InputT,OutputT> withSideInputs(PCollectionView<?>... sideInputs)
Returns aPTransform
identical to this, but with the specified side inputs to use inCombineWithContext.CombineFnWithContext
.
-
withSideInputs
public Combine.PerKey<K,InputT,OutputT> withSideInputs(java.lang.Iterable<? extends PCollectionView<?>> sideInputs)
Returns aPTransform
identical to this, but with the specified side inputs to use inCombineWithContext.CombineFnWithContext
.
-
withHotKeyFanout
public Combine.PerKeyWithHotKeyFanout<K,InputT,OutputT> withHotKeyFanout(SerializableFunction<? super K,java.lang.Integer> hotKeyFanout)
If a single key has disproportionately many values, it may become a bottleneck, especially in streaming mode. This returns a new per-key combining transform that inserts an intermediate node to combine "hot" keys partially before performing the full combine.- Parameters:
hotKeyFanout
- a function from keys to an integer N, where the key will be spread among N intermediate nodes for partial combining. If N is less than or equal to 1, this key will not be sent through an intermediate node.
-
withHotKeyFanout
public Combine.PerKeyWithHotKeyFanout<K,InputT,OutputT> withHotKeyFanout(int hotKeyFanout)
LikewithHotKeyFanout(SerializableFunction)
, but returning the given constant value for every key.
-
getFn
public CombineFnBase.GlobalCombineFn<? super InputT,?,OutputT> getFn()
Returns theCombineFnBase.GlobalCombineFn
used by this Combine operation.
-
getSideInputs
public java.util.List<PCollectionView<?>> getSideInputs()
Returns the side inputs used by this Combine operation.
-
getAdditionalInputs
public java.util.Map<TupleTag<?>,PValue> getAdditionalInputs()
Returns the side inputs of thisCombine
, tagged with the tag of thePCollectionView
. The values of the returned map will be equal to the result ofgetSideInputs()
.- Overrides:
getAdditionalInputs
in classPTransform<PCollection<KV<K,InputT>>,PCollection<KV<K,OutputT>>>
-
expand
public PCollection<KV<K,OutputT>> expand(PCollection<KV<K,InputT>> input)
Description copied from class:PTransform
Override this method to specify how thisPTransform
should be expanded on the givenInputT
.NOTE: This method should not be called directly. Instead apply the
PTransform
should be applied to theInputT
using theapply
method.Composite transforms, which are defined in terms of other transforms, should return the output of one of the composed transforms. Non-composite transforms, which do not apply any transforms internally, should return a new unbound output and register evaluators (via backend-specific registration methods).
- Specified by:
expand
in classPTransform<PCollection<KV<K,InputT>>,PCollection<KV<K,OutputT>>>
-
populateDisplayData
public void populateDisplayData(DisplayData.Builder builder)
Description copied from class:PTransform
Register display data for the given transform or component.populateDisplayData(DisplayData.Builder)
is invoked by Pipeline runners to collect display data viaDisplayData.from(HasDisplayData)
. Implementations may callsuper.populateDisplayData(builder)
in order to register display data in the current namespace, but should otherwise usesubcomponent.populateDisplayData(builder)
to use the namespace of the subcomponent.By default, does not register any display data. Implementors may override this method to provide their own display data.
- Specified by:
populateDisplayData
in interfaceHasDisplayData
- Overrides:
populateDisplayData
in classPTransform<PCollection<KV<K,InputT>>,PCollection<KV<K,OutputT>>>
- Parameters:
builder
- The builder to populate with display data.- See Also:
HasDisplayData
-
-