Class GroupByKey<K,V>
- java.lang.Object
-
- org.apache.beam.sdk.transforms.PTransform<PCollection<KV<K,V>>,PCollection<KV<K,java.lang.Iterable<V>>>>
-
- org.apache.beam.sdk.transforms.GroupByKey<K,V>
-
- Type Parameters:
K
- the type of the keys of the input and outputPCollection
sV
- the type of the values of the inputPCollection
and the elements of theIterable
s in the outputPCollection
- All Implemented Interfaces:
java.io.Serializable
,HasDisplayData
public class GroupByKey<K,V> extends PTransform<PCollection<KV<K,V>>,PCollection<KV<K,java.lang.Iterable<V>>>>
GroupByKey<K, V>
takes aPCollection<KV<K, V>>
, groups the values by key and windows, and returns aPCollection<KV<K, Iterable<V>>>
representing a map from each distinct key and window of the inputPCollection
to anIterable
over all the values associated with that key in the input per window. Absent repeatedly-firingtriggering
, each key in the outputPCollection
is unique within each window.GroupByKey
is analogous to converting a multi-map into a uni-map, and related toGROUP BY
in SQL. It corresponds to the "shuffle" step between the Mapper and the Reducer in the MapReduce framework.Two keys of type
K
are compared for equality not by regular JavaObject.equals(java.lang.Object)
, but instead by first encoding each of the keys using theCoder
of the keys of the inputPCollection
, and then comparing the encoded bytes. This admits efficient parallel evaluation. Note that this requires that theCoder
of the keys be deterministic (seeCoder.verifyDeterministic()
). If the keyCoder
is not deterministic, an exception is thrown at pipeline construction time.By default, the
Coder
of the keys of the outputPCollection
is the same as that of the keys of the input, and theCoder
of the elements of theIterable
values of the outputPCollection
is the same as theCoder
of the values of the input.Example of use:
PCollection<KV<String, Doc>> urlDocPairs = ...; PCollection<KV<String, Iterable<Doc>>> urlToDocs = urlDocPairs.apply(GroupByKey.<String, Doc>create()); PCollection<R> results = urlToDocs.apply(ParDo.of(new DoFn<KV<String, Iterable<Doc>>, R>()
{@ProcessElement public void processElement(ProcessContext c) { String url = c.element().getKey(); Iterable<Doc> docsWithThatUrl = c.element().getValue(); ... process all docs having that url ... }
}));GroupByKey
is a key primitive in data-parallel processing, since it is the main way to efficiently bring associated data together into one location. It is also a key determiner of the performance of a data-parallel pipeline.See
CoGroupByKey
for a way to group multiple input PCollections by a common key at once.See
Combine.PerKey
for a common pattern ofGroupByKey
followed byCombine.GroupedValues
.When grouping, windows that can be merged according to the
WindowFn
of the inputPCollection
will be merged together, and a window pane corresponding to the new, merged window will be created. The items in this pane will be emitted when a trigger fires. By default this will be when the input sources estimate there will be no more data for the window. SeeAfterWatermark
for details on the estimation.The timestamp for each emitted pane is determined by the
Window.withTimestampCombiner(TimestampCombiner)
windowing operation}. The outputPCollection
will have the sameWindowFn
as the input.If the input
PCollection
contains late data or therequested TriggerFn
can fire before the watermark, then there may be multiple elements output by aGroupByKey
that correspond to the same key and window.If the
WindowFn
of the input requires merging, it is not valid to apply anotherGroupByKey
without first applying a newWindowFn
or applyingWindow.remerge()
.- See Also:
- Serialized Form
-
-
Field Summary
-
Fields inherited from class org.apache.beam.sdk.transforms.PTransform
name, resourceHints
-
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description static void
applicableTo(PCollection<?> input)
static <K,V>
GroupByKey<K,V>create()
Returns aGroupByKey<K, V>
PTransform
.PCollection<KV<K,java.lang.Iterable<V>>>
expand(PCollection<KV<K,V>> input)
Override this method to specify how thisPTransform
should be expanded on the givenInputT
.boolean
fewKeys()
Returns whether it groups just few keys.static <K,V>
Coder<V>getInputValueCoder(Coder<KV<K,V>> inputCoder)
Returns theCoder
of the values of the input to this transform.static <K,V>
Coder<K>getKeyCoder(Coder<KV<K,V>> inputCoder)
Returns theCoder
of the keys of the input to this transform, which is also used as theCoder
of the keys of the output of this transform.static <K,V>
KvCoder<K,java.lang.Iterable<V>>getOutputKvCoder(Coder<KV<K,V>> inputCoder)
Returns theCoder
of the output of this transform.void
populateDisplayData(DisplayData.Builder builder)
Register display data for the given transform or component.WindowingStrategy<?,?>
updateWindowingStrategy(WindowingStrategy<?,?> inputStrategy)
void
validate(@Nullable PipelineOptions options, java.util.Map<TupleTag<?>,PCollection<?>> inputs, java.util.Map<TupleTag<?>,PCollection<?>> outputs)
Called before running the Pipeline to verify this transform, its inputs, and outputs are fully and correctly specified.-
Methods inherited from class org.apache.beam.sdk.transforms.PTransform
compose, compose, getAdditionalInputs, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, getResourceHints, setResourceHints, toString, validate
-
-
-
-
Method Detail
-
create
public static <K,V> GroupByKey<K,V> create()
Returns aGroupByKey<K, V>
PTransform
.- Type Parameters:
K
- the type of the keys of the input and outputPCollection
sV
- the type of the values of the inputPCollection
and the elements of theIterable
s in the outputPCollection
-
fewKeys
public boolean fewKeys()
Returns whether it groups just few keys.
-
applicableTo
public static void applicableTo(PCollection<?> input)
-
validate
public void validate(@Nullable PipelineOptions options, java.util.Map<TupleTag<?>,PCollection<?>> inputs, java.util.Map<TupleTag<?>,PCollection<?>> outputs)
Description copied from class:PTransform
Called before running the Pipeline to verify this transform, its inputs, and outputs are fully and correctly specified.By default, delegates to
PTransform.validate(PipelineOptions)
.- Overrides:
validate
in classPTransform<PCollection<KV<K,V>>,PCollection<KV<K,java.lang.Iterable<V>>>>
-
updateWindowingStrategy
public WindowingStrategy<?,?> updateWindowingStrategy(WindowingStrategy<?,?> inputStrategy)
-
expand
public PCollection<KV<K,java.lang.Iterable<V>>> expand(PCollection<KV<K,V>> input)
Description copied from class:PTransform
Override this method to specify how thisPTransform
should be expanded on the givenInputT
.NOTE: This method should not be called directly. Instead apply the
PTransform
should be applied to theInputT
using theapply
method.Composite transforms, which are defined in terms of other transforms, should return the output of one of the composed transforms. Non-composite transforms, which do not apply any transforms internally, should return a new unbound output and register evaluators (via backend-specific registration methods).
- Specified by:
expand
in classPTransform<PCollection<KV<K,V>>,PCollection<KV<K,java.lang.Iterable<V>>>>
-
getKeyCoder
public static <K,V> Coder<K> getKeyCoder(Coder<KV<K,V>> inputCoder)
Returns theCoder
of the keys of the input to this transform, which is also used as theCoder
of the keys of the output of this transform.
-
getInputValueCoder
public static <K,V> Coder<V> getInputValueCoder(Coder<KV<K,V>> inputCoder)
Returns theCoder
of the values of the input to this transform.
-
getOutputKvCoder
public static <K,V> KvCoder<K,java.lang.Iterable<V>> getOutputKvCoder(Coder<KV<K,V>> inputCoder)
Returns theCoder
of the output of this transform.
-
populateDisplayData
public void populateDisplayData(DisplayData.Builder builder)
Description copied from class:PTransform
Register display data for the given transform or component.populateDisplayData(DisplayData.Builder)
is invoked by Pipeline runners to collect display data viaDisplayData.from(HasDisplayData)
. Implementations may callsuper.populateDisplayData(builder)
in order to register display data in the current namespace, but should otherwise usesubcomponent.populateDisplayData(builder)
to use the namespace of the subcomponent.By default, does not register any display data. Implementors may override this method to provide their own display data.
- Specified by:
populateDisplayData
in interfaceHasDisplayData
- Overrides:
populateDisplayData
in classPTransform<PCollection<KV<K,V>>,PCollection<KV<K,java.lang.Iterable<V>>>>
- Parameters:
builder
- The builder to populate with display data.- See Also:
HasDisplayData
-
-