Package org.apache.beam.sdk.transforms

Class Wait

  • @Experimental
public class Wait
extends java.lang.Object
    Delays processing of each window in a PCollection until signaled.

    Given a main PCollection and a signal PCollection, produces output identical to its main input, but all elements for a window are produced only once that window is closed in the signal PCollection.

    To express the pattern "apply T to X after Y is ready", use X.apply(Wait.on(Y)).apply(T).

    In particular: returns a PCollection with contents identical to the input, but delays producing elements of the output in window W until the signal's window W closes (i.e. signal's watermark passes W.end + signal.allowedLateness).

    In other words, an element of the output at timestamp "t" will be produced only after no more elements of the signal can appear with a timestamp below "t".

    Example usage: write a PCollection to one database and then to another database, making sure that writing a window of data to the second database starts only after the respective window has been fully written to the first database. 

    
 PCollection<Void> firstWriteResults = data.apply(ParDo.of(...write to first database...));
 data.apply(Wait.on(firstWriteResults))
     // Windows of this intermediate PCollection will be processed no earlier than when
     // the respective window of firstWriteResults closes.
     .apply(ParDo.of(...write to second database...));

    Notes:

    • If signal is globally windowed, main input must also be. This typically would be useful only in a batch pipeline, because the global window of an infinite PCollection never closes, so the wait signal will never be ready.
    • Beware that if the signal has large allowed lateness, the wait signal will fire only after that lateness elapses, i.e. after the watermark of the signal passes end of the window plus allowed lateness. In other words: do not use this with signals that specify a large allowed lateness.

    • Constructor Detail

      • Wait

        public Wait()

    • Method Detail

      • on

        public static <T> Wait.OnSignal<T> on​(java.util.List<PCollection<?>> signals)
        Waits on the given signal collections.