com.google.cloud.dataflow.sdk.transforms

Class DoFn<I,O>

java.lang.Object

com.google.cloud.dataflow.sdk.transforms.DoFn<I,O>

Type Parameters:

I - the type of the (main) input elements

O - the type of the (main) output elements

All Implemented Interfaces:

java.io.Serializable

Direct Known Subclasses:

DirectPipelineRunner.TestCombineDoFn, RateLimiting.RateLimitingDoFn

public abstract class DoFn<I,O>
extends java.lang.Object
implements java.io.Serializable

The argument to ParDo providing the code to use to process elements of the input PCollection.

See ParDo for more explanation, examples of use, and discussion of constraints on DoFns, including their serializability, lack of access to global shared mutable state, requirements for failure tolerance, and benefits of optimization.

DoFns can be tested in the context of a particular Pipeline by running that Pipeline on sample input and then checking its output. Unit testing of a DoFn, separately from any ParDo transform or Pipeline, can be done via the DoFnTester harness.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
class	DoFn.Context Information accessible to all methods in this DoFn.
static interface	DoFn.KeyedState Interface for interacting with keyed state.
class	DoFn.ProcessContext Information accessible when running processElement(com.google.cloud.dataflow.sdk.transforms.DoFn<I, O>.ProcessContext).
static interface	DoFn.RequiresKeyedState Interface for signaling that a DoFn needs to maintain per-key state, accessed via DoFn.ProcessContext.keyedState().

Constructor Summary

Constructors

Constructor and Description
DoFn()

Method Summary

Methods

Modifier and Type	Method and Description
void	finishBundle(DoFn.Context c) Finishes processing this batch of elements.
Duration	getAllowedTimestampSkew() Returns the allowed timestamp skew duration, which is the maximum duration that timestamps can be shifted backward in DoFn.Context.outputWithTimestamp(O, org.joda.time.Instant).
abstract void	processElement(DoFn.ProcessContext c) Processes an input element.
void	startBundle(DoFn.Context c) Prepares this DoFn instance for processing a batch of elements.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DoFn

public DoFn()

Method Detail

getAllowedTimestampSkew

public Duration getAllowedTimestampSkew()

Returns the allowed timestamp skew duration, which is the maximum duration that timestamps can be shifted backward in DoFn.Context.outputWithTimestamp(O, org.joda.time.Instant). The default value is Duration.ZERO, in which case timestamps can only be shifted forward to future. For infinite skew, return Duration.millis(Long.MAX_VALUE).

startBundle

public void startBundle(DoFn.Context c)
                 throws java.lang.Exception

Prepares this DoFn instance for processing a batch of elements.

By default, does nothing.

Throws:

java.lang.Exception

processElement

public abstract void processElement(DoFn.ProcessContext c)
                             throws java.lang.Exception

Processes an input element.

Throws:

java.lang.Exception

finishBundle

public void finishBundle(DoFn.Context c)
                  throws java.lang.Exception

Finishes processing this batch of elements. This DoFn instance will be thrown away after this operation returns.

By default, does nothing.

Throws:

java.lang.Exception