zio-kafka
API
Generated with
zio-kafka/zio.kafka/zio.kafka.consumer/ConsumerSettings

ConsumerSettings

zio.kafka.consumer.ConsumerSettings
See theConsumerSettings companion object
final case class ConsumerSettings(properties: Map[String, AnyRef], closeTimeout: Duration, pollTimeout: Duration, commitTimeout: Duration, offsetRetrieval: OffsetRetrieval, rebalanceListener: RebalanceListener, rebalanceSafeCommits: Boolean, maxRebalanceDuration: Option[Duration], fetchStrategy: FetchStrategy, metricLabels: Set[MetricLabel], runloopMetricsSchedule: Schedule[Any, Unit, Long], authErrorRetrySchedule: Schedule[Any, Throwable, Any], maxStreamPullIntervalOption: Option[Duration], diagnostics: ConsumerDiagnostics)

Settings for the consumer.

To stay source compatible with future releases, you are recommended to construct the settings as follows:

 ConsumerSettings(bootstrapServers)
   .withGroupId(groupId)
   .withProperties(properties)
   .... etc.

Attributes

Companion
object
Graph
Supertypes
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all

Members list

Value members

Concrete methods

def driverSettings: Map[String, AnyRef]
def tuneForHighThroughput: ConsumerSettings

Tunes the consumer for high throughput.

Tunes the consumer for high throughput.

Sets poll timeout to 500 ms, max.poll.records to 2000 and partitionPreFetchBufferLimit to 4096.

Attributes

See also
def tuneForLowLatency: ConsumerSettings

Tunes the consumer for low latency.

Tunes the consumer for low latency.

Sets poll timeout to 50 ms, max.poll.records to 100 and partitionPreFetchBufferLimit to 512.

Attributes

See also
def withAuthErrorRetrySchedule(authErrorRetrySchedule: Schedule[Any, Throwable, Any]): ConsumerSettings

Value parameters

authErrorRetrySchedule

The schedule at which the consumer will retry polling the broker for more records, even though a poll fails with an org.apache.kafka.common.errors.AuthorizationException or org.apache.kafka.common.errors.AuthenticationException. This setting helps with failed polls due to too slow authorization or authentication in the broker. You may also consider increasing pollTimeout to reduce auth-work on the broker. Set to Schedule.stop to fail the consumer on the first auth error. The default is

Schedule.recurs(5) && Schedule.spaced(500.millis)

which is, to retry 5 times, spaced by 500ms.

Attributes

def withCommitTimeout(timeout: Duration): ConsumerSettings

The maximum time that zio-kafka waits for a commit to complete.

The maximum time that zio-kafka waits for a commit to complete.

Note: this configuration also affects the duration in which records should be processed when rebalanceSafeCommits is enabled.

Attributes

def withDiagnostics(diagnostics: ConsumerDiagnostics): ConsumerSettings

Value parameters

diagnostics

an optional callback for key events in the consumer life-cycle. The callbacks will be executed in a separate fiber. Since the events are queued, failure to handle these events leads to out of memory errors.

Attributes

def withFetchStrategy(fetchStrategy: FetchStrategy): ConsumerSettings

Value parameters

fetchStrategy

The fetch strategy which selects which partitions can fetch data in the next poll. The default is to use the zio.kafka.consumer.fetch.QueueSizeBasedFetchStrategy with a partitionPreFetchBufferLimit parameter of 1024, which is calculated by taking 2 * the default max.poll.records of 500, rounded to the nearest power of 2.

Attributes

def withMaxPollInterval(maxPollInterval: Duration): ConsumerSettings

Set Kafka's max.poll.interval.ms configuration. See https://kafka.apache.org/documentation/#consumerconfigs_max.poll.interval.ms for more information.

Set Kafka's max.poll.interval.ms configuration. See https://kafka.apache.org/documentation/#consumerconfigs_max.poll.interval.ms for more information.

The default is 5 minutes. Make sure that all records from a single poll can be processed in this interval. See also the maxPollRecords configuration.

Attributes

def withMaxPollRecords(maxPollRecords: Int): ConsumerSettings

Set Kafka's max.poll.records configuration. See https://kafka.apache.org/documentation/#consumerconfigs_max.poll.records for more information.

Set Kafka's max.poll.records configuration. See https://kafka.apache.org/documentation/#consumerconfigs_max.poll.records for more information.

The default is 500.

Attributes

def withMaxRebalanceDuration(value: Duration): ConsumerSettings

Value parameters

value

Maximum time spent in the rebalance callback when rebalanceSafeCommits is enabled. In this time zio-kafka awaits processing of records and the completion of commits. By default, this value is set to 3/5 of maxPollInterval which by default calculates to 3 minutes. Only values between commitTimeout and maxPollInterval are useful. Lower values will make the rebalance callback be done immediately, higher values lead to lost partitions. See withRebalanceSafeCommits for more information.

Attributes

def withMaxStreamPullInterval(maxStreamPullInterval: Duration): ConsumerSettings

The maximum time a stream may run without pulling a chunk of records.

The maximum time a stream may run without pulling a chunk of records.

Zio-kafka uses this value to determine whether a stream stopped processing. This is to safeguard against alive consumers in the consumer group which hold partition assignments but make no progress. If no chunks are pulled by user code from a partition stream for this interval (while data is available) we consider the stream to be halted. When this happens we interrupt the stream with a failure. In addition, the entire consumer is shutdown. In future versions of zio-kafka we may (instead of a shutdown) stop only the affected subscription.

Make sure that all records from a single poll (see maxPollRecords) can be processed in this interval, even when there is no concurrency because the records are all in the same partition.

The default is equal to maxPollInterval.

Attributes

def withMetricsLabels(metricLabels: Set[MetricLabel]): ConsumerSettings

Value parameters

metricLabels

The labels given to all metrics collected by the zio-kafka consumer. By default, no labels are set. For applications with multiple consumers it is recommended to set some metric labels. For example, if one is used, the consumer group id could be used as a label:

 consumerSettings.withMetricLabels(Set(MetricLabel("group-id", groupId)))

Attributes

def withOffsetRetrieval(retrieval: OffsetRetrieval): ConsumerSettings

Which offset to start consuming from for new partitions.

Which offset to start consuming from for new partitions.

The options are:

 import zio.kafka.consumer.Consumer._
 OffsetRetrieval.Auto(AutoOffsetStrategy.Latest) // the default
 OffsetRetrieval.Auto(AutoOffsetStrategy.Earliest)
 OffsetRetrieval.Auto(AutoOffsetStrategy.None)
 OffsetRetrieval.Manual(getOffsets, defaultStrategy)

The Auto options make consuming start from the latest committed offset. When no committed offset is available, the given offset strategy is used and consuming starts from the Latest offset (the default), the Earliest offset, or results in an error for None.

The Manual option allows fine grained control over which offset to consume from. The provided getOffsets function should return an offset for each topic-partition that is being assigned. When the returned offset is smaller than the log start offset or larger than the log end offset, the defaultStrategy is used and consuming starts from the Latest offset (the default), the Earliest offset, or results in an error for None.

When the returned map does ''not'' contain an entry for a topic-partition, the consumer will continue from the last committed offset. When no committed offset is available, the defaultStrategy is used and consuming starts from the Latest offset (the default), the Earliest offset, or results in an error for None.

This configuration applies to both subscribed and assigned partitions.

This method sets the auto.offset.reset Kafka configuration. See https://kafka.apache.org/documentation/#consumerconfigs_auto.offset.reset for more information.

Attributes

def withPartitionPreFetchBufferLimit(partitionPreFetchBufferLimit: Int): ConsumerSettings

Value parameters

partitionPreFetchBufferLimit

The queue size at or below which more records are fetched and buffered (per partition). This buffer improves throughput and supports varying downstream message processing time, while maintaining some backpressure. Large values effectively disable backpressure at the cost of high memory usage, low values will effectively disable prefetching in favor of low memory consumption. The number of records that is fetched on every poll is controlled by the max.poll.records setting, the number of records fetched for every partition is somewhere between 0 and max.poll.records. The default value for this parameter is 1024. It is calculated by taking 2 * the default max.poll.records of 500, rounded to the nearest power of 2. The value 0 disables pre-fetching.

Attributes

def withPollTimeout(timeout: Duration): ConsumerSettings

The maximum time to block while polling the Kafka consumer. The Kafka consumer will return earlier when the maximum number of record to poll (see https://kafka.apache.org/documentation/#consumerconfigs_max.poll.records) is collected.

The maximum time to block while polling the Kafka consumer. The Kafka consumer will return earlier when the maximum number of record to poll (see https://kafka.apache.org/documentation/#consumerconfigs_max.poll.records) is collected.

The default is 50ms which is good for low latency applications. Set this higher, e.g. 500ms for better throughput.

Attributes

def withProperties(kvs: (String, AnyRef)*): ConsumerSettings
def withProperty(key: String, value: AnyRef): ConsumerSettings
def withReadCommitted(readCommitted: Boolean): ConsumerSettings

Controls how to consume records produced transactionally.

Controls how to consume records produced transactionally.

Value parameters

readCommitted

when true, only consume records which have been committed, when false, consume all records, even records which are part of an aborted transaction. Non-transactional records will be consumed unconditionally in either mode. Note that Kafka's default is to read all records (readCommitted = false).

Attributes

def withRebalanceSafeCommits(value: Boolean): ConsumerSettings

Value parameters

value

Whether to hold up a rebalance until all offsets of consumed messages have been committed. The default is false, but the recommended value is true as it prevents duplicate messages. Use false when:

  • your streams do not commit, or
  • your streams require access to the consumer (the consumer is not available until the rebalance is done), or
  • when it is okay to process records twice (possibly concurrently), for example, because processing is idempotent. When true, messages consumed from revoked partitions must be committed before we allow the rebalance to continue. When a partition is revoked, consuming the messages will be taken over by another consumer. The other consumer will continue from the committed offset. It is therefore important that this consumer commits offsets of all consumed messages. Therefore, by holding up the rebalance until these commits are done, we ensure that the new consumer will start from the correct offset. During a rebalance no new messages can be received for any stream. Therefore, all streams are deprived of new messages until the revoked streams are ready committing. Rebalances are held up for at most 3/5 of maxPollInterval (see withMaxPollInterval), by default this calculates to 3 minutes but this value can be changed with withMaxRebalanceDuration. In this time your program should be able to process up to maxPollRecords records, including commits! Commits take time as well. Zio-kafka assumes withCommitTimeout as worse case (defaults to 15 seconds). Therefore, with all default settings, your program has 2 minutes and 45 seconds to process and commit the records. External commits (that is, commits to an external system, e.g. a relational database) must be registered to the consumer with Consumer.registerExternalCommits. When this consumer is coupled to a TransactionalProducer, rebalanceSafeCommits must be enabled. When false, streams for revoked partitions may continue to run even though the rebalance is not held up. Any offset commits from these streams have a high chance of being delayed (commits are not possible during some phases of a rebalance). The consumer that takes over the partition will likely not see these delayed commits and will start from an earlier offset. The result is that some messages are processed twice and concurrently.

Attributes

def withRunloopMetricsSchedule(runloopMetricsSchedule: Schedule[Any, Unit, Long]): ConsumerSettings

Value parameters

runloopMetricsSchedule

The schedule at which the runloop metrics are measured. Example runloop metrics are queue sizes and number of outstanding commits. The default is to measure every 500ms.

Attributes

def withoutPartitionPreFetching: ConsumerSettings

Disables partition record pre-fetching.

Disables partition record pre-fetching.

Attributes

Inherited methods

def productElementNames: Iterator[String]

Attributes

Inherited from:
Product
def productIterator: Iterator[Any]

Attributes

Inherited from:
Product
Generated with