Package io.confluent.parallelconsumer

Class ParallelConsumerOptions.ParallelConsumerOptionsBuilder<K,V>

java.lang.Object

io.confluent.parallelconsumer.ParallelConsumerOptions.ParallelConsumerOptionsBuilder<K,V>

Enclosing class:

ParallelConsumerOptions<K,V>

public static class ParallelConsumerOptions.ParallelConsumerOptionsBuilder<K,V>
extends Object

Method Summary

Modifier and Type	Method	Description
ParallelConsumerOptions.ParallelConsumerOptionsBuilder<K,V>	allowEagerProcessingDuringTransactionCommit(boolean allowEagerProcessingDuringTransactionCommit)	When using ParallelConsumerOptions.CommitMode.PERIODIC_TRANSACTIONAL_PRODUCER, allows new records to be processed UP UNTIL the result record SENDING (Producer.send(org.apache.kafka.clients.producer.ProducerRecord<K, V>)) step, potentially while a transaction is being committed.
ParallelConsumerOptions.ParallelConsumerOptionsBuilder<K,V>	batchSize(Integer batchSize)	The maximum number of messages to attempt to pass into the user functions.
ParallelConsumerOptions<K,V>	build()
ParallelConsumerOptions.ParallelConsumerOptionsBuilder<K,V>	commitInterval(Duration commitInterval)	Time between commits.
ParallelConsumerOptions.ParallelConsumerOptionsBuilder<K,V>	commitLockAcquisitionTimeout(Duration commitLockAcquisitionTimeout)	Time to allow for acquiring the commit lock.
ParallelConsumerOptions.ParallelConsumerOptionsBuilder<K,V>	commitMode(ParallelConsumerOptions.CommitMode commitMode)	The ParallelConsumerOptions.CommitMode to be used
ParallelConsumerOptions.ParallelConsumerOptionsBuilder<K,V>	consumer(org.apache.kafka.clients.consumer.Consumer<K,V> consumer)	Required parameter for all use.
ParallelConsumerOptions.ParallelConsumerOptionsBuilder<K,V>	defaultMessageRetryDelay(Duration defaultMessageRetryDelay)	Deprecated. will be renamed to static retry delay
ParallelConsumerOptions.ParallelConsumerOptionsBuilder<K,V>	managedExecutorService(String managedExecutorService)	Path to Managed executor service for Java EE
ParallelConsumerOptions.ParallelConsumerOptionsBuilder<K,V>	managedThreadFactory(String managedThreadFactory)	Path to Managed thread factory for Java EE
ParallelConsumerOptions.ParallelConsumerOptionsBuilder<K,V>	maxConcurrency(int maxConcurrency)	Controls the maximum degree of concurrency to occur.
ParallelConsumerOptions.ParallelConsumerOptionsBuilder<K,V>	maxFailureHistory(int maxFailureHistory)
ParallelConsumerOptions.ParallelConsumerOptionsBuilder<K,V>	offsetCommitTimeout(Duration offsetCommitTimeout)	Controls how long to block while waiting for offsets to be committed.
ParallelConsumerOptions.ParallelConsumerOptionsBuilder<K,V>	ordering(ParallelConsumerOptions.ProcessingOrder ordering)	The ParallelConsumerOptions.ProcessingOrder type to use
ParallelConsumerOptions.ParallelConsumerOptionsBuilder<K,V>	produceLockAcquisitionTimeout(Duration produceLockAcquisitionTimeout)	Time to allow for acquiring the produce lock.
ParallelConsumerOptions.ParallelConsumerOptionsBuilder<K,V>	producer(org.apache.kafka.clients.producer.Producer<K,V> producer)	Supplying a producer is only needed if using the produce flows.
ParallelConsumerOptions.ParallelConsumerOptionsBuilder<K,V>	retryDelayProvider(Function<RecordContext<K,V>,Duration> retryDelayProvider)	When present, use this to generate a dynamic retry delay, instead of a static one with ParallelConsumerOptions.getDefaultMessageRetryDelay().
ParallelConsumerOptions.ParallelConsumerOptionsBuilder<K,V>	sendTimeout(Duration sendTimeout)	Controls how long to block while waiting for the Producer.send(org.apache.kafka.clients.producer.ProducerRecord<K, V>) to complete for any ProducerRecords returned from the user-function.
ParallelConsumerOptions.ParallelConsumerOptionsBuilder<K,V>	thresholdForTimeSpendInQueueWarning(Duration thresholdForTimeSpendInQueueWarning)	Configure the amount of delay a record experiences, before a warning is logged.
String	toString()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Method Details

consumer

public ParallelConsumerOptions.ParallelConsumerOptionsBuilder<K,V> consumer(org.apache.kafka.clients.consumer.Consumer<K,V> consumer)

Required parameter for all use.

Returns:

this.

producer

public ParallelConsumerOptions.ParallelConsumerOptionsBuilder<K,V> producer(org.apache.kafka.clients.producer.Producer<K,V> producer)

Supplying a producer is only needed if using the produce flows.

Returns:

this.

See Also:

• ParallelStreamProcessor

managedExecutorService

public ParallelConsumerOptions.ParallelConsumerOptionsBuilder<K,V> managedExecutorService(String managedExecutorService)

Path to Managed executor service for Java EE

Returns:

this.

managedThreadFactory

public ParallelConsumerOptions.ParallelConsumerOptionsBuilder<K,V> managedThreadFactory(String managedThreadFactory)

Path to Managed thread factory for Java EE

Returns:

this.

allowEagerProcessingDuringTransactionCommit

public ParallelConsumerOptions.ParallelConsumerOptionsBuilder<K,V> allowEagerProcessingDuringTransactionCommit(boolean allowEagerProcessingDuringTransactionCommit)

When using ParallelConsumerOptions.CommitMode.PERIODIC_TRANSACTIONAL_PRODUCER, allows new records to be processed UP UNTIL the result record SENDING (Producer.send(org.apache.kafka.clients.producer.ProducerRecord<K, V>)) step, potentially while a transaction is being committed. Disabled by default as to prevent replay side effects when records need to be retried in some scenarios.

Doesn't interfere with the transaction itself, just reduces side effects.

Recommended to leave this off to avoid side effect duplicates upon rebalances after a crash. Enabling could improve performance as the produce lock will only be taken right before it's needed (optimistic locking) to produce the result record, instead of pessimistically locking.

Returns:

this.

commitLockAcquisitionTimeout

public ParallelConsumerOptions.ParallelConsumerOptionsBuilder<K,V> commitLockAcquisitionTimeout(Duration commitLockAcquisitionTimeout)

Time to allow for acquiring the commit lock. If record processing or producing takes a long time, you may need to increase this. If this fails, the system will shut down (fail fast) and attempt to commit once more.

Returns:

this.

produceLockAcquisitionTimeout

public ParallelConsumerOptions.ParallelConsumerOptionsBuilder<K,V> produceLockAcquisitionTimeout(Duration produceLockAcquisitionTimeout)

Time to allow for acquiring the produce lock. If transaction committing a long time, you may need to increase this. If this fails, the record will be returned to the processing queue for later retry.

Returns:

this.

commitInterval

public ParallelConsumerOptions.ParallelConsumerOptionsBuilder<K,V> commitInterval(Duration commitInterval)

Time between commits. Using a higher frequency (a lower value) will put more load on the brokers.

Returns:

this.

ordering

public ParallelConsumerOptions.ParallelConsumerOptionsBuilder<K,V> ordering(ParallelConsumerOptions.ProcessingOrder ordering)

The ParallelConsumerOptions.ProcessingOrder type to use

Returns:

this.

commitMode

public ParallelConsumerOptions.ParallelConsumerOptionsBuilder<K,V> commitMode(ParallelConsumerOptions.CommitMode commitMode)

The ParallelConsumerOptions.CommitMode to be used

Returns:

this.

maxConcurrency

public ParallelConsumerOptions.ParallelConsumerOptionsBuilder<K,V> maxConcurrency(int maxConcurrency)

Controls the maximum degree of concurrency to occur. Used to limit concurrent calls to external systems to a maximum to prevent overloading them or to a degree, using up quotas.

When using ParallelConsumerOptions.getBatchSize(), this is over and above the batch size setting. So for example, a ParallelConsumerOptions.getMaxConcurrency() of 2 and a batch size of 3 would result in at most 15 records being processed at once.

A note on quotas - if your quota is expressed as maximum concurrent calls, this works well. If it's limited in total requests / sec, this may still overload the system. See towards the distributed rate limiting feature for this to be properly addressed: https://github.com/confluentinc/parallel-consumer/issues/24 Add distributed rate limiting support #24.

In the core module, this sets the number of threads to use in the core's thread pool.

It's recommended to set this quite high, much higher than core count, as it's expected that these threads will spend most of their time blocked waiting for IO. For automatic setting of this variable, look out for issue https://github.com/confluentinc/parallel-consumer/issues/21 Dynamic concurrency control with flow control or tcp congestion control theory #21.

Returns:

this.

defaultMessageRetryDelay

@Deprecated
public ParallelConsumerOptions.ParallelConsumerOptionsBuilder<K,V> defaultMessageRetryDelay(Duration defaultMessageRetryDelay)

Deprecated.

will be renamed to static retry delay

When a message fails, how long the system should wait before trying that message again. Note that this will not be exact, and is just a target.

Returns:

this.

retryDelayProvider

public ParallelConsumerOptions.ParallelConsumerOptionsBuilder<K,V> retryDelayProvider(Function<RecordContext<K,V>,Duration> retryDelayProvider)

When present, use this to generate a dynamic retry delay, instead of a static one with ParallelConsumerOptions.getDefaultMessageRetryDelay().

Overrides defaultMessageRetryDelay(java.time.Duration), even if it's set.

Returns:

this.

sendTimeout

public ParallelConsumerOptions.ParallelConsumerOptionsBuilder<K,V> sendTimeout(Duration sendTimeout)

Controls how long to block while waiting for the Producer.send(org.apache.kafka.clients.producer.ProducerRecord<K, V>) to complete for any ProducerRecords returned from the user-function. Only relevant if using one of the produce-flows and providing a ParallelConsumerOptions.producer. If the timeout occurs the record will be re-processed in the user-function.

Consider aligning the value with the ParallelConsumerOptions.producer-options to avoid unnecessary re-processing and duplicates on slow Producer.send(org.apache.kafka.clients.producer.ProducerRecord<K, V>) calls.

Returns:

this.

See Also:

• ProducerConfig.DELIVERY_TIMEOUT_MS_CONFIG

offsetCommitTimeout

public ParallelConsumerOptions.ParallelConsumerOptionsBuilder<K,V> offsetCommitTimeout(Duration offsetCommitTimeout)

Controls how long to block while waiting for offsets to be committed. Only relevant if using ParallelConsumerOptions.CommitMode.PERIODIC_CONSUMER_SYNC commit-mode.

Returns:

this.

batchSize

public ParallelConsumerOptions.ParallelConsumerOptionsBuilder<K,V> batchSize(Integer batchSize)

The maximum number of messages to attempt to pass into the user functions.

Batch sizes may sometimes be less than this size, but will never be more.

The system will treat the messages as a set, so if an error is thrown by the user code, then all messages will be marked as failed and be retried (Note that when they are retried, there is no guarantee they will all be in the same batch again). So if you're going to process messages individually, then don't set a batch size.

Otherwise, if you're going to process messages in sub sets from this batch, it's better to instead adjust the ParallelConsumerOptions.getBatchSize() instead to the actual desired size, and process them as a whole.

Note that there is no relationship between the ConsumerConfig setting of ConsumerConfig.MAX_POLL_RECORDS_CONFIG and this configured batch size, as this library introduces a large layer of indirection between the managed consumer, and the managed queues we use.

This indirection effectively disconnects the processing of messages from "polling" them from the managed client, as we do not wait to process them before calling poll again. We simply call poll as much as we need to, in order to keep our queues full of enough work to satisfy demand.

If we have enough, then we actively manage pausing our subscription so that we can continue calling poll without pulling in even more messages.

Returns:

this.

See Also:

• ParallelConsumerOptions.getBatchSize()

thresholdForTimeSpendInQueueWarning

public ParallelConsumerOptions.ParallelConsumerOptionsBuilder<K,V> thresholdForTimeSpendInQueueWarning(Duration thresholdForTimeSpendInQueueWarning)

Configure the amount of delay a record experiences, before a warning is logged.

Returns:

this.

maxFailureHistory

public ParallelConsumerOptions.ParallelConsumerOptionsBuilder<K,V> maxFailureHistory(int maxFailureHistory)

Returns:

this.

build

public ParallelConsumerOptions<K,V> build()

toString

public String toString()

Overrides:

toString in class Object