I - the type of the (main) input elementsO - the type of the (main) output elementspublic abstract class DoFn<I,O>
extends java.lang.Object
implements java.io.Serializable
ParDo providing the code to use to process
elements of the input
PCollection.
See ParDo for more explanation, examples of use, and
discussion of constraints on DoFns, including their
serializability, lack of access to global shared mutable state,
requirements for failure tolerance, and benefits of optimization.
DoFns can be tested in the context of a particular
Pipeline by running that Pipeline on sample input
and then checking its output. Unit testing of a DoFn,
separately from any ParDo transform or Pipeline,
can be done via the DoFnTester harness.
| Modifier and Type | Class and Description |
|---|---|
class |
DoFn.Context
Information accessible to all methods in this
DoFn. |
static interface |
DoFn.KeyedState
Interface for interacting with keyed state.
|
class |
DoFn.ProcessContext
Information accessible when running
processElement(com.google.cloud.dataflow.sdk.transforms.DoFn<I, O>.ProcessContext). |
static interface |
DoFn.RequiresKeyedState
Interface for signaling that a
DoFn needs to maintain
per-key state, accessed via
DoFn.ProcessContext.keyedState(). |
| Constructor and Description |
|---|
DoFn() |
| Modifier and Type | Method and Description |
|---|---|
void |
finishBundle(DoFn.Context c)
Finishes processing this batch of elements.
|
Duration |
getAllowedTimestampSkew()
Returns the allowed timestamp skew duration, which is the maximum
duration that timestamps can be shifted backward in
DoFn.Context.outputWithTimestamp(O, org.joda.time.Instant). |
abstract void |
processElement(DoFn.ProcessContext c)
Processes an input element.
|
void |
startBundle(DoFn.Context c)
Prepares this
DoFn instance for processing a batch of elements. |
public Duration getAllowedTimestampSkew()
DoFn.Context.outputWithTimestamp(O, org.joda.time.Instant).
The default value is Duration.ZERO, in which case
timestamps can only be shifted forward to future. For infinite
skew, return Duration.millis(Long.MAX_VALUE).public void startBundle(DoFn.Context c) throws java.lang.Exception
DoFn instance for processing a batch of elements.
By default, does nothing.
java.lang.Exceptionpublic abstract void processElement(DoFn.ProcessContext c) throws java.lang.Exception
java.lang.Exceptionpublic void finishBundle(DoFn.Context c) throws java.lang.Exception
DoFn
instance will be thrown away after this operation returns.
By default, does nothing.
java.lang.Exception