com.google.cloud.dataflow.sdk.io

Class CountingSource

java.lang.Object

com.google.cloud.dataflow.sdk.io.CountingSource

public class CountingSource
extends Object

A source that produces longs. When used as a BoundedSource, CountingSource starts at 0 and counts up to a specified maximum. When used as an UnboundedSource, it counts up to Long.MAX_VALUE and then never produces more output. (In practice, this limit should never be reached.)

The bounded CountingSource is implemented based on OffsetBasedSource and OffsetBasedSource.OffsetBasedReader, so it performs efficient initial splitting and it supports dynamic work rebalancing.

To produce a bounded PCollection<Long>, use upTo(long):

 Pipeline p = ...
 BoundedSource<Long> source = CountingSource.upTo(1000);
 PCollection<Long> bounded = p.apply(Read.from(source));
 

To produce an unbounded PCollection<Long>, use unbounded() or unboundedWithTimestampFn(com.google.cloud.dataflow.sdk.transforms.SerializableFunction<java.lang.Long, org.joda.time.Instant>):

 Pipeline p = ...

 // To create an unbounded source that uses processing time as the element timestamp.
 UnboundedSource<Long, CounterMark> source = CountingSource.unbounded();
 // Or, to create an unbounded source that uses a provided function to set the element timestamp.
 UnboundedSource<Long, CounterMark> source = CountingSource.unboundedWithTimestampFn(someFn);

 PCollection<Long> unbounded = p.apply(Read.from(source));
 

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	CountingSource.CounterMark The checkpoint for an unbounded CountingSource is simply the last value produced.

Method Summary

Modifier and Type	Method and Description
static UnboundedSource<Long,CountingSource.CounterMark>	unbounded() Creates an UnboundedSource that will produce numbers starting from 0 up to Long.MAX_VALUE.
static UnboundedSource<Long,CountingSource.CounterMark>	unboundedWithTimestampFn(SerializableFunction<Long,Instant> timestampFn) Creates an UnboundedSource that will produce numbers starting from 0 up to Long.MAX_VALUE, with element timestamps supplied by the specified function.
static BoundedSource<Long>	upTo(long numElements) Creates a BoundedSource that will produce the specified number of elements, from 0 to numElements - 1.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

upTo

public static BoundedSource<Long> upTo(long numElements)

Creates a BoundedSource that will produce the specified number of elements, from 0 to numElements - 1.

unbounded

public static UnboundedSource<Long,CountingSource.CounterMark> unbounded()

Creates an UnboundedSource that will produce numbers starting from 0 up to Long.MAX_VALUE.

After Long.MAX_VALUE, the source never produces more output. (In practice, this limit should never be reached.)

Elements in the resulting PCollection<Long> will have timestamps corresponding to processing time at element generation, provided by Instant.now().

unboundedWithTimestampFn

public static UnboundedSource<Long,CountingSource.CounterMark> unboundedWithTimestampFn(SerializableFunction<Long,Instant> timestampFn)

Creates an UnboundedSource that will produce numbers starting from 0 up to Long.MAX_VALUE, with element timestamps supplied by the specified function.

After Long.MAX_VALUE, the source never produces more output. (In practice, this limit should never be reached.)

Note that the timestamps produced by timestampFn may not decrease.