zio-kafka
API
Generated with
zio-kafka/zio.kafka/zio.kafka.consumer/ConsumerSettings

ConsumerSettings

zio.kafka.consumer.ConsumerSettings
See theConsumerSettings companion object
final case class ConsumerSettings(properties: Map[String, AnyRef], closeTimeout: Duration, pollTimeout: Duration, commitTimeout: Duration, offsetRetrieval: OffsetRetrieval, rebalanceListener: RebalanceListener, restartStreamOnRebalancing: Boolean, rebalanceSafeCommits: Boolean, maxRebalanceDuration: Option[Duration], fetchStrategy: FetchStrategy)

Settings for the consumer.

To stay source compatible with future releases, you are recommended to construct the settings as follows:

 ConsumerSettings(bootstrapServers)
   .withGroupId(groupId)
   .withProperties(properties)
   .... etc.

Value parameters

bootstrapServers

the Kafka bootstrap servers

Attributes

Companion
object
Graph
Supertypes
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all

Members list

Value members

Concrete methods

def driverSettings: Map[String, AnyRef]
def tuneForHighThroughput: ConsumerSettings

Tunes the consumer for high throughput.

Tunes the consumer for high throughput.

Sets poll timeout to 500 ms, max.poll.records to 2000 and partitionPreFetchBufferLimit to 4096.

Attributes

See also
def tuneForLowLatency: ConsumerSettings

Tunes the consumer for low latency.

Tunes the consumer for low latency.

Sets poll timeout to 50 ms, max.poll.records to 100 and partitionPreFetchBufferLimit to 512.

Attributes

See also
def withFetchStrategy(fetchStrategy: FetchStrategy): ConsumerSettings

WARNING: zio.kafka.consumer.fetch.FetchStrategy is an EXPERIMENTAL API and may change in an incompatible way without notice in any zio-kafka version.

WARNING: zio.kafka.consumer.fetch.FetchStrategy is an EXPERIMENTAL API and may change in an incompatible way without notice in any zio-kafka version.

Value parameters

fetchStrategy

The fetch strategy which selects which partitions can fetch data in the next poll. The default is to use the zio.kafka.consumer.fetch.QueueSizeBasedFetchStrategy with a partitionPreFetchBufferLimit parameter of 1024, which is calculated by taking 2 * the default max.poll.records of 500, rounded to the nearest power of 2.

Attributes

def withMaxPollInterval(maxPollInterval: Duration): ConsumerSettings

Set Kafka's max.poll.interval.ms configuration. See https://kafka.apache.org/documentation/#consumerconfigs_max.poll.interval.ms for more information.

Set Kafka's max.poll.interval.ms configuration. See https://kafka.apache.org/documentation/#consumerconfigs_max.poll.interval.ms for more information.

Zio-kafka uses this value also to determine whether a stream stopped processing. If no chunks are pulled from a stream for this interval (while data is available) we consider the stream to be halted. When this happens we interrupt the stream with a failure. In addition the entire consumer is shutdown. In future versions of zio-kafka we may (instead of a shutdown) stop only the affected subscription.

The default is 5 minutes. Make sure that all records from a single poll can be processed in this interval. The maximum number of records in a single poll is configured with the max.poll.records configuration (see https://kafka.apache.org/documentation/#consumerconfigs_max.poll.records and withMaxPollRecords).

Attributes

def withMaxPollRecords(maxPollRecords: Int): ConsumerSettings

Set Kafka's max.poll.records configuration. See https://kafka.apache.org/documentation/#consumerconfigs_max.poll.records for more information.

Set Kafka's max.poll.records configuration. See https://kafka.apache.org/documentation/#consumerconfigs_max.poll.records for more information.

Attributes

def withMaxRebalanceDuration(value: Duration): ConsumerSettings

NOTE: 'rebalanceSafeCommits' is an EXPERIMENTAL feature. It is not recommended for production use yet.

NOTE: 'rebalanceSafeCommits' is an EXPERIMENTAL feature. It is not recommended for production use yet.

Value parameters

value

Maximum time spent in the rebalance callback when rebalanceSafeCommits is enabled. In this time zio-kafka awaits processing of records and the completion of commits. By default this value is set to 3/5 of maxPollInterval which by default calculates to 3 minutes. Only values between commitTimeout and maxPollInterval are useful. Lower values will make the rebalance callback be done immediately, higher values lead to lost partitions. See withRebalanceSafeCommits for more information.

Attributes

def withPartitionPreFetchBufferLimit(partitionPreFetchBufferLimit: Int): ConsumerSettings

Value parameters

partitionPreFetchBufferLimit

The queue size at or below which more records are fetched and buffered (per partition). This buffer improves throughput and supports varying downstream message processing time, while maintaining some backpressure. Large values effectively disable backpressure at the cost of high memory usage, low values will effectively disable prefetching in favor of low memory consumption. The number of records that is fetched on every poll is controlled by the max.poll.records setting, the number of records fetched for every partition is somewhere between 0 and max.poll.records. The default value for this parameter is 1024. It is calculated by taking 2 * the default max.poll.records of 500, rounded to the nearest power of 2. The value 0 disables pre-fetching.

Attributes

def withPollTimeout(timeout: Duration): ConsumerSettings

The maximum time to block while polling the Kafka consumer. The Kafka consumer will return earlier when the maximum number of record to poll (see https://kafka.apache.org/documentation/#consumerconfigs_max.poll.records) is collected.

The maximum time to block while polling the Kafka consumer. The Kafka consumer will return earlier when the maximum number of record to poll (see https://kafka.apache.org/documentation/#consumerconfigs_max.poll.records) is collected.

The default is 50ms which is good for low latency applications. Set this higher, e.g. 500ms for better throughput.

Attributes

def withProperties(kvs: (String, AnyRef)*): ConsumerSettings
def withProperty(key: String, value: AnyRef): ConsumerSettings
def withRebalanceSafeCommits(value: Boolean): ConsumerSettings

NOTE: 'rebalanceSafeCommits' is an EXPERIMENTAL feature. It is not recommended for production use yet.

NOTE: 'rebalanceSafeCommits' is an EXPERIMENTAL feature. It is not recommended for production use yet.

Value parameters

value

Whether to hold up a rebalance until all offsets of consumed messages have been committed. The default is false, but the recommended value is true as it prevents duplicate messages. Use false when:

  • your streams do not commit, or
  • your streams require access to the consumer (the consumer is not available until the rebalance is done), or
  • when it is okay to process records twice (possibly concurrently), for example, because processing is idempotent. When true, messages consumed from revoked partitions must be committed before we allow the rebalance to continue. When a partition is revoked, consuming the messages will be taken over by another consumer. The other consumer will continue from the committed offset. It is therefore important that this consumer commits offsets of all consumed messages. Therefore, by holding up the rebalance until these commits are done, we ensure that the new consumer will start from the correct offset. During a rebalance no new messages can be received for any stream. Therefore, all streams are deprived of new messages until the revoked streams are ready committing. Rebalances are held up for at most 3/5 of maxPollInterval (see withMaxPollInterval), by default this calculates to 3 minutes. See withMaxRebalanceDuration to change the default. When false, streams for revoked partitions may continue to run even though the rebalance is not held up. Any offset commits from these streams have a high chance of being delayed (commits are not possible during some phases of a rebalance). The consumer that takes over the partition will likely not see these delayed commits and will start from an earlier offset. The result is that some messages are processed twice and concurrently.

Attributes

def withRestartStreamOnRebalancing(value: Boolean): ConsumerSettings

Value parameters

value

When true all streams are restarted during a rebalance, including those streams that are not revoked. The default is false.

Attributes

def withoutPartitionPreFetching: ConsumerSettings

Disables partition record pre-fetching.

Disables partition record pre-fetching.

Attributes

Deprecated methods

def withMaxPartitionQueueSize(partitionPreFetchBufferLimit: Int): ConsumerSettings

Attributes

Deprecated
true

Inherited methods

def productElementNames: Iterator[String]

Attributes

Inherited from:
Product
def productIterator: Iterator[Any]

Attributes

Inherited from:
Product
Generated with