zio-kafka
API
Generated with
zio-kafka/zio.kafka/zio.kafka.consumer/ConsumerSettings

ConsumerSettings

zio.kafka.consumer.ConsumerSettings
See theConsumerSettings companion object
final case class ConsumerSettings(properties: Map[String, AnyRef], closeTimeout: Duration, pollTimeout: Duration, commitTimeout: Duration, offsetRetrieval: OffsetRetrieval, rebalanceListener: RebalanceListener, restartStreamOnRebalancing: Boolean, fetchStrategy: FetchStrategy)

Settings for the consumer.

To stay source compatible with future releases, you are recommended to construct the settings as follows:

 ConsumerSettings(bootstrapServers)
   .withGroupId(groupId)
   .withProperties(properties)
   .... etc.

Value parameters

bootstrapServers

the Kafka bootstrap servers

Attributes

Companion
object
Graph
Supertypes
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all

Members list

Value members

Concrete methods

def driverSettings: Map[String, AnyRef]
def tuneForHighThroughput: ConsumerSettings

Tunes the consumer for high throughput.

Tunes the consumer for high throughput.

Sets poll timeout to 500 ms, max.poll.records to 2000 and partitionPreFetchBufferLimit to 4096.

Attributes

See also
def tuneForLowLatency: ConsumerSettings

Tunes the consumer for low latency.

Tunes the consumer for low latency.

Sets poll timeout to 50 ms, max.poll.records to 100 and partitionPreFetchBufferLimit to 512.

Attributes

See also
def withFetchStrategy(fetchStrategy: FetchStrategy): ConsumerSettings

WARNING: zio.kafka.consumer.fetch.FetchStrategy is an EXPERIMENTAL API and may change in an incompatible way without notice in any zio-kafka version.

WARNING: zio.kafka.consumer.fetch.FetchStrategy is an EXPERIMENTAL API and may change in an incompatible way without notice in any zio-kafka version.

Value parameters

fetchStrategy

The fetch strategy which selects which partitions can fetch data in the next poll. The default is to use the zio.kafka.consumer.fetch.QueueSizeBasedFetchStrategy with a partitionPreFetchBufferLimit parameter of 1024, which is calculated by taking 2 * the default max.poll.records of 500, rounded to the nearest power of 2.

Attributes

def withMaxPollInterval(maxPollInterval: Duration): ConsumerSettings

Set Kafka's max.poll.interval.ms configuration. See https://kafka.apache.org/documentation/#consumerconfigs_max.poll.interval.ms for more information.

Set Kafka's max.poll.interval.ms configuration. See https://kafka.apache.org/documentation/#consumerconfigs_max.poll.interval.ms for more information.

Zio-kafka uses this value also to determine whether a stream stopped processing. If no chunks are pulled from a stream for this interval (while data is available) we consider the stream to be halted. When this happens we interrupt the stream with a failure. In addition the entire consumer is shutdown. In future versions of zio-kafka we may (instead of a shutdown) stop only the affected subscription.

The default is 5 minutes. Make sure that all records from a single poll can be processed in this interval. The maximum number of records in a single poll is configured with the max.poll.records configuration (see https://kafka.apache.org/documentation/#consumerconfigs_max.poll.records and withMaxPollRecords).

Attributes

def withMaxPollRecords(maxPollRecords: Int): ConsumerSettings

Set Kafka's max.poll.records configuration. See https://kafka.apache.org/documentation/#consumerconfigs_max.poll.records for more information.

Set Kafka's max.poll.records configuration. See https://kafka.apache.org/documentation/#consumerconfigs_max.poll.records for more information.

Attributes

def withPartitionPreFetchBufferLimit(partitionPreFetchBufferLimit: Int): ConsumerSettings

Value parameters

partitionPreFetchBufferLimit

The queue size below which more records are fetched and buffered (per partition). This buffer improves throughput and supports varying downstream message processing time, while maintaining some backpressure. Large values effectively disable backpressure at the cost of high memory usage, low values will effectively disable prefetching in favour of low memory consumption. The number of records that is fetched on every poll is controlled by the max.poll.records setting, the number of records fetched for every partition is somewhere between 0 and max.poll.records. A value that is a power of 2 offers somewhat better queueing performance. The default value for this parameter is 1024. It is calculated by taking 2 * the default max.poll.records of 500, rounded to the nearest power of 2.

Attributes

def withPollTimeout(timeout: Duration): ConsumerSettings

The maximum time to block while polling the Kafka consumer. The Kafka consumer will return earlier when the maximum number of record to poll (see https://kafka.apache.org/documentation/#consumerconfigs_max.poll.records) is collected.

The maximum time to block while polling the Kafka consumer. The Kafka consumer will return earlier when the maximum number of record to poll (see https://kafka.apache.org/documentation/#consumerconfigs_max.poll.records) is collected.

The default is 50ms which is good for low latency applications. Set this higher, e.g. 500ms for better throughput.

Attributes

def withProperties(kvs: (String, AnyRef)*): ConsumerSettings
def withProperty(key: String, value: AnyRef): ConsumerSettings
def withRestartStreamOnRebalancing(value: Boolean): ConsumerSettings

Value parameters

value

When true all streams are restarted during a rebalance, including those streams that are not revoked. The default is false.

Attributes

Deprecated methods

def withMaxPartitionQueueSize(partitionPreFetchBufferLimit: Int): ConsumerSettings

Attributes

Deprecated
true

Inherited methods

def productElementNames: Iterator[String]

Attributes

Inherited from:
Product
def productIterator: Iterator[Any]

Attributes

Inherited from:
Product
Generated with