Deduplicate (core 2.44.0 API)

java.lang.Object
- org.apache.beam.sdk.transforms.Deduplicate

```
public final class Deduplicate
extends java.lang.Object
```
A set of PTransforms which deduplicate input records over a time domain and threshold. Values in different windows will not be considered duplicates of each other. Deduplication is best effort.
Two values of type T are compared for equality not by regular Java Object.equals(java.lang.Object), but instead by first encoding each of the elements using the PCollection's Coder, and then comparing the encoded bytes. This admits efficient parallel evaluation.
These PTransforms are different then Distinct since Distinct guarantees uniqueness of values within a PCollection but may support a narrower set of windowing strategies or may delay when output is produced.
The durations specified may impose memory and/or storage requirements within a runner and care might need to be used to ensure that the deduplication time limit is long enough to remove duplicates but short enough to not cause performance problems within a runner. Each runner may provide an optimized implementation of their choice using the deduplication time domain and threshold specified.
Does not preserve any order the input PCollection might have had.
Example of use:
```
 PCollection<String> words = ...;
 PCollection<String> deduplicatedWords =
     words.apply(Deduplicate.<String>values());
 
```

Nested Class Summary

Nested Classes
Modifier and Type	Class	Description
`static class`	`Deduplicate.KeyedValues<K,V>`	Deduplicates keyed values using the key over a specified time domain and threshold.
`static class`	`Deduplicate.Values<T>`	Deduplicates values over a specified time domain and threshold.
`static class`	`Deduplicate.WithRepresentativeValues<T,IdT>`	A `PTransform` that uses a `SerializableFunction` to obtain a representative value for each input element used for deduplication.

Field Summary

Fields
Modifier and Type	Field	Description
`static org.joda.time.Duration`	`DEFAULT_DURATION`	The default duration is 10 mins.
`static TimeDomain`	`DEFAULT_TIME_DOMAIN`	The default is the `processing time domain`.

Method Summary

All Methods Static Methods Concrete Methods
Modifier and Type	Method	Description
`static <K,V> Deduplicate.KeyedValues<K,V>`	`keyedValues()`	Returns a deduplication transform that deduplicates keyed values using the key for up to 10 mins within the `processing time domain`.
`static <T> Deduplicate.Values<T>`	`values()`	Returns a deduplication transform that deduplicates values for up to 10 mins within the `processing time domain`.
`static <T,IdT> Deduplicate.WithRepresentativeValues<T,IdT>`	`withRepresentativeValueFn(SerializableFunction<T,IdT> representativeValueFn)`	Returns a deduplication transform that deduplicates values using the supplied representative value for up to 10 mins within the `processing time domain`.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - DEFAULT_TIME_DOMAIN
```
public static final TimeDomain DEFAULT_TIME_DOMAIN
```
    The default is the processing time domain.
  - DEFAULT_DURATION
```
public static final org.joda.time.Duration DEFAULT_DURATION
```
    The default duration is 10 mins.
- Method Detail
  - values
```
public static <T> Deduplicate.Values<T> values()
```
    Returns a deduplication transform that deduplicates values for up to 10 mins within the processing time domain.
  - keyedValues
```
public static <K,V> Deduplicate.KeyedValues<K,V> keyedValues()
```
    Returns a deduplication transform that deduplicates keyed values using the key for up to 10 mins within the processing time domain.
  - withRepresentativeValueFn
```
public static <T,IdT> Deduplicate.WithRepresentativeValues<T,IdT> withRepresentativeValueFn(SerializableFunction<T,IdT> representativeValueFn)
```
    Returns a deduplication transform that deduplicates values using the supplied representative value for up to 10 mins within the processing time domain.