com.google.cloud.dataflow.sdk.io

Class CountingInput

java.lang.Object

com.google.cloud.dataflow.sdk.io.CountingInput

public class CountingInput
extends Object

A PTransform that produces longs. When used to produce a bounded PCollection, CountingInput starts at 0 and counts up to a specified maximum. When used to produce an unbounded PCollection, it counts up to Long.MAX_VALUE and then never produces more output. (In practice, this limit should never be reached.)

The bounded CountingInput is implemented based on OffsetBasedSource and OffsetBasedSource.OffsetBasedReader, so it performs efficient initial splitting and it supports dynamic work rebalancing.

To produce a bounded PCollection<Long>, use upTo(long):

 Pipeline p = ...
 PTransform<PBegin, PCollection<Long>> producer = CountingInput.upTo(1000);
 PCollection<Long> bounded = p.apply(producer);
 

To produce an unbounded PCollection<Long>, use unbounded(), calling CountingInput.UnboundedCountingInput.withTimestampFn(SerializableFunction) to provide values with timestamps other than Instant.now().

 Pipeline p = ...

 // To create an unbounded producer that uses processing time as the element timestamp.
 PCollection<Long> unbounded = p.apply(CountingInput.unbounded());
 // Or, to create an unbounded source that uses a provided function to set the element timestamp.
 PCollection<Long> unboundedWithTimestamps =
     p.apply(CountingInput.unbounded().withTimestampFn(someFn));
 

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	CountingInput.BoundedCountingInput A PTransform that will produce a specified number of Longs starting from 0.
static class	CountingInput.UnboundedCountingInput A PTransform that will produce numbers starting from 0 up to Long.MAX_VALUE.

Constructor Summary

Constructors

Constructor and Description
CountingInput()

Method Summary

Modifier and Type	Method and Description
static CountingInput.UnboundedCountingInput	unbounded() Creates an CountingInput.UnboundedCountingInput that will produce numbers starting from 0 up to Long.MAX_VALUE.
static CountingInput.BoundedCountingInput	upTo(long numElements) Creates a CountingInput.BoundedCountingInput that will produce the specified number of elements, from 0 to numElements - 1.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

CountingInput

public CountingInput()

Method Detail

upTo

public static CountingInput.BoundedCountingInput upTo(long numElements)

Creates a CountingInput.BoundedCountingInput that will produce the specified number of elements, from 0 to numElements - 1.

unbounded

public static CountingInput.UnboundedCountingInput unbounded()

Creates an CountingInput.UnboundedCountingInput that will produce numbers starting from 0 up to Long.MAX_VALUE.

After Long.MAX_VALUE, the transform never produces more output. (In practice, this limit should never be reached.)

Elements in the resulting PCollection<Long> will by default have timestamps corresponding to processing time at element generation, provided by Instant.now(). Use the transform returned by CountingInput.UnboundedCountingInput.withTimestampFn(SerializableFunction) to control the output timestamps.