com.google.cloud.dataflow.sdk.transforms

Class RateLimiting

java.lang.Object

com.google.cloud.dataflow.sdk.transforms.RateLimiting

public class RateLimiting
extends java.lang.Object

Provides rate-limiting of user functions, using threaded execution and a RateLimiter to process elements at the desired rate.

For example, to limit each worker to 10 requests per second:

PCollection<T> data = ...;
 data.apply(
   RateLimiting.perWorker(new MyDoFn())
               .withRateLimit(10)));
 

An uncaught exception from the wrapped DoFn will result in the exception being rethrown in later calls to RateLimiting.RateLimitingDoFn.processElement(com.google.cloud.dataflow.sdk.transforms.DoFn<I, O>.ProcessContext) or a call to RateLimiting.RateLimitingDoFn.finishBundle(com.google.cloud.dataflow.sdk.transforms.DoFn<I, O>.Context).

Rate limiting is provided as a PTransform (RateLimiting.RateLimitingTransform), and also as a DoFn (RateLimiting.RateLimitingDoFn).

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	RateLimiting.RateLimitingDoFn<I,O> A rate-limiting DoFn wrapper.
static class	RateLimiting.RateLimitingTransform<I,O> A PTransform which applies rate limiting to a DoFn.

Constructor Summary

Constructors

Constructor and Description
RateLimiting()

Method Summary

Methods

Modifier and Type	Method and Description
static <I,O> RateLimiting.RateLimitingTransform<I,O>	perWorker(DoFn<I,O> doFn) Creates a new per-worker rate-limiting transform for the given DoFn.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

RateLimiting

public RateLimiting()

Method Detail

perWorker

public static <I,O> RateLimiting.RateLimitingTransform<I,O> perWorker(DoFn<I,O> doFn)

Creates a new per-worker rate-limiting transform for the given DoFn.

The default behavior is to process elements with multiple threads, but no rate limit is applied.

Use RateLimiting.RateLimitingTransform.withRateLimit(double) to limit the processing rate, and RateLimiting.RateLimitingTransform.withMaxParallelism(int) to control the maximum concurrent processing limit.

Aside from the above, the DoFn will be executed in the same manner as in ParDo.

Rate limiting is applied independently per-worker.