Package org.apache.beam.sdk.io

Class GenerateSequence

  • All Implemented Interfaces:
    java.io.Serializable, HasDisplayData
    public abstract class GenerateSequence
extends PTransform<PBegin,​PCollection<java.lang.Long>>
    A PTransform that produces longs starting from the given value, and either up to the given limit or until Long.MAX_VALUE / until the given time elapses.

    The bounded GenerateSequence is implemented based on OffsetBasedSource and OffsetBasedSource.OffsetBasedReader, so it performs efficient initial splitting and it supports dynamic work rebalancing.

    To produce a bounded PCollection<Long>: 

    
 Pipeline p = ...
 PCollection<Long> bounded = p.apply(GenerateSequence.from(0).to(1000));

    To produce an unbounded PCollection<Long>, simply do not specify to(long), calling withTimestampFn(SerializableFunction) to provide values with timestamps other than Instant.now(). 

    
 Pipeline p = ...

 // To use processing time as the element timestamp.
 PCollection<Long> unbounded = p.apply(GenerateSequence.from(0));
 // Or, to use a provided function to set the element timestamp.
 PCollection<Long> unboundedWithTimestamps =
     p.apply(GenerateSequence.from(0).withTimestampFn(someFn));

    In all cases, the sequence of numbers is generated in parallel, so there is no inherent ordering between the generated values - it is only guaranteed that all values in the given range will be present in the resulting PCollection.

    See Also:
    Serialized Form

    • Constructor Detail

      • GenerateSequence

        public GenerateSequence()

    • Method Detail

      • from

        public static GenerateSequence from​(long from)
        Specifies the minimum number to generate (inclusive).

      • to

        public GenerateSequence to​(long to)
        Specifies the maximum number to generate (exclusive).

      • withTimestampFn

        public GenerateSequence withTimestampFn​(SerializableFunction<java.lang.Long,​org.joda.time.Instant> timestampFn)
        Specifies the function to use to assign timestamps to the elements.

      • withRate

        public GenerateSequence withRate​(long numElements,
                                 org.joda.time.Duration periodLength)
        Specifies to generate at most a given number of elements per a given period.

      • withMaxReadTime

        public GenerateSequence withMaxReadTime​(org.joda.time.Duration maxReadTime)
        Specifies to stop generating elements after the given time.

      • expand

        public PCollection<java.lang.Long> expand​(PBegin input)
        Description copied from class: PTransform
        Override this method to specify how this PTransform should be expanded on the given InputT.

        NOTE: This method should not be called directly. Instead apply the PTransform should be applied to the InputT using the apply method.

        Composite transforms, which are defined in terms of other transforms, should return the output of one of the composed transforms. Non-composite transforms, which do not apply any transforms internally, should return a new unbound output and register evaluators (via backend-specific registration methods).

        Specified by:
        expand in class PTransform<PBegin,​PCollection<java.lang.Long>>

      • populateDisplayData

        public void populateDisplayData​(DisplayData.Builder builder)
        Description copied from class: PTransform
        Register display data for the given transform or component.

        populateDisplayData(DisplayData.Builder) is invoked by Pipeline runners to collect display data via DisplayData.from(HasDisplayData). Implementations may call super.populateDisplayData(builder) in order to register display data in the current namespace, but should otherwise use subcomponent.populateDisplayData(builder) to use the namespace of the subcomponent.

        By default, does not register any display data. Implementors may override this method to provide their own display data.

        Specified by:
        populateDisplayData in interface HasDisplayData
        Overrides:
        populateDisplayData in class PTransform<PBegin,​PCollection<java.lang.Long>>
        Parameters:
        builder - The builder to populate with display data.
        See Also:
        HasDisplayData