org.apache.kafka.streams

Class StreamsConfig

java.lang.Object

org.apache.kafka.common.config.AbstractConfig

org.apache.kafka.streams.StreamsConfig

public class StreamsConfig
extends org.apache.kafka.common.config.AbstractConfig

Configuration for a KafkaStreams instance. Can also be use to configure the Kafka Streams internal KafkaConsumer and KafkaProducer. To avoid consumer/producer property conflicts, you should prefix those properties using consumerPrefix(String) and producerPrefix(String), respectively.

Example:

// potentially wrong: sets "metadata.max.age.ms" to 1 minute for producer AND consumer
 Properties streamsProperties = new Properties();
 streamsProperties.put(ConsumerConfig.METADATA_MAX_AGE_CONFIG, 60000);
 // or
 streamsProperties.put(ProducerConfig.METADATA_MAX_AGE_CONFIG, 60000);

 // suggested:
 Properties streamsProperties = new Properties();
 // sets "metadata.max.age.ms" to 1 minute for consumer only
 streamsProperties.put(StreamsConfig.consumerPrefix(ConsumerConfig.METADATA_MAX_AGE_CONFIG), 60000);
 // sets "metadata.max.age.ms" to 1 minute for producer only
 streamsProperties.put(StreamsConfig.producerPrefix(ProducerConfig.METADATA_MAX_AGE_CONFIG), 60000);

 StreamsConfig streamsConfig = new StreamsConfig(streamsProperties);
 

Kafka Streams required to set at least properties "application.id" and "bootstrap.servers". Furthermore, it is not allowed to enable "enable.auto.commit" that is disabled by Kafka Streams by default.

See Also:

KafkaStreams.KafkaStreams(org.apache.kafka.streams.processor.TopologyBuilder, StreamsConfig), ConsumerConfig, ProducerConfig

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	StreamsConfig.InternalConfig

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	APPLICATION_ID_CONFIG application.id
static java.lang.String	APPLICATION_SERVER_CONFIG user.endpoint
static java.lang.String	BOOTSTRAP_SERVERS_CONFIG bootstrap.servers
static java.lang.String	BUFFERED_RECORDS_PER_PARTITION_CONFIG buffered.records.per.partition
static java.lang.String	CACHE_MAX_BYTES_BUFFERING_CONFIG cache.max.bytes.buffering
static java.lang.String	CLIENT_ID_CONFIG client.id
static java.lang.String	COMMIT_INTERVAL_MS_CONFIG commit.interval.ms
static java.lang.String	CONNECTIONS_MAX_IDLE_MS_CONFIG
static java.lang.String	CONSUMER_PREFIX Prefix used to isolate consumer configs from producer configs.
static java.lang.String	DEFAULT_SECURITY_PROTOCOL
static java.lang.String	KEY_SERDE_CLASS_CONFIG key.serde
static java.lang.String	METADATA_MAX_AGE_CONFIG
static java.lang.String	METRIC_REPORTER_CLASSES_CONFIG metric.reporters
static java.lang.String	METRICS_NUM_SAMPLES_CONFIG metrics.num.samples
static java.lang.String	METRICS_RECORDING_LEVEL_CONFIG metrics.record.level
static java.lang.String	METRICS_SAMPLE_WINDOW_MS_CONFIG metrics.sample.window.ms
static java.lang.String	NUM_STANDBY_REPLICAS_CONFIG num.standby.replicas
static java.lang.String	NUM_STREAM_THREADS_CONFIG num.stream.threads
static java.lang.String	PARTITION_GROUPER_CLASS_CONFIG partition.grouper
static java.lang.String	POLL_MS_CONFIG poll.ms
static java.lang.String	PRODUCER_PREFIX Prefix used to isolate producer configs from consumer configs.
static java.lang.String	RECEIVE_BUFFER_CONFIG
static java.lang.String	RECONNECT_BACKOFF_MS_CONFIG
static java.lang.String	REPLICATION_FACTOR_CONFIG replication.factor
static java.lang.String	REQUEST_TIMEOUT_MS_CONFIG
static java.lang.String	RETRY_BACKOFF_MS_CONFIG
static java.lang.String	ROCKSDB_CONFIG_SETTER_CLASS_CONFIG rocksdb.config.setter
static java.lang.String	SECURITY_PROTOCOL_CONFIG
static java.lang.String	SEND_BUFFER_CONFIG
static java.lang.String	STATE_CLEANUP_DELAY_MS_CONFIG state.cleanup.delay
static java.lang.String	STATE_DIR_CONFIG state.dir
static java.lang.String	TIMESTAMP_EXTRACTOR_CLASS_CONFIG timestamp.extractor
static java.lang.String	UPGRADE_FROM_0100 Config value for parameter "upgrade.from" for upgrading an application from version 0.10.0.x.
static java.lang.String	UPGRADE_FROM_CONFIG upgrade.from
static java.lang.String	UPGRADE_FROM_DOC
static java.lang.String	VALUE_SERDE_CLASS_CONFIG value.serde
static java.lang.String	WINDOW_STORE_CHANGE_LOG_ADDITIONAL_RETENTION_MS_CONFIG windowstore.changelog.additional.retention.ms
static java.lang.String	ZOOKEEPER_CONNECT_CONFIG Deprecated. Kakfa Streams does not use Zookeeper anymore and this parameter will be ignored.

Constructor Summary

Constructors

Constructor and Description
StreamsConfig(java.util.Map<?,?> props) Create a new StreamsConfig using the given properties.

Method Summary

Methods

Modifier and Type	Method and Description
static org.apache.kafka.common.config.ConfigDef	configDef() Return a copy of the config definition.
static java.lang.String	consumerPrefix(java.lang.String consumerProp) Prefix a property with CONSUMER_PREFIX.
java.util.Map<java.lang.String,java.lang.Object>	getConsumerConfigs(org.apache.kafka.streams.processor.internals.StreamThread streamThread, java.lang.String groupId, java.lang.String clientId) Get the configs to the consumer.
java.util.Map<java.lang.String,java.lang.Object>	getProducerConfigs(java.lang.String clientId) Get the configs for the producer.
java.util.Map<java.lang.String,java.lang.Object>	getRestoreConsumerConfigs(java.lang.String clientId) Get the configs for the restore-consumer.
org.apache.kafka.common.serialization.Serde	keySerde() Return an configured instance of key Serde class.
static void	main(java.lang.String[] args)
static java.lang.String	producerPrefix(java.lang.String producerProp) Prefix a property with PRODUCER_PREFIX.
org.apache.kafka.common.serialization.Serde	valueSerde() Return an configured instance of value Serde class.

Methods inherited from class org.apache.kafka.common.config.AbstractConfig

equals, get, getBoolean, getClass, getConfiguredInstance, getConfiguredInstances, getConfiguredInstances, getDouble, getInt, getList, getLong, getPassword, getShort, getString, hashCode, ignore, logUnused, originals, originalsStrings, originalsWithPrefix, unused, values

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait

Field Detail

CONSUMER_PREFIX

public static final java.lang.String CONSUMER_PREFIX

Prefix used to isolate consumer configs from producer configs. It is recommended to use consumerPrefix(String) to add this prefix to consumer properties.

See Also:

Constant Field Values

PRODUCER_PREFIX

public static final java.lang.String PRODUCER_PREFIX

Prefix used to isolate producer configs from consumer configs. It is recommended to use producerPrefix(String) to add this prefix to producer properties.

See Also:

Constant Field Values

UPGRADE_FROM_0100

public static final java.lang.String UPGRADE_FROM_0100

Config value for parameter "upgrade.from" for upgrading an application from version 0.10.0.x.

See Also:

Constant Field Values

UPGRADE_FROM_CONFIG

public static final java.lang.String UPGRADE_FROM_CONFIG

upgrade.from

See Also:

Constant Field Values

UPGRADE_FROM_DOC

public static final java.lang.String UPGRADE_FROM_DOC

See Also:

Constant Field Values

STATE_DIR_CONFIG

public static final java.lang.String STATE_DIR_CONFIG

state.dir

See Also:

Constant Field Values

ZOOKEEPER_CONNECT_CONFIG

@Deprecated
public static final java.lang.String ZOOKEEPER_CONNECT_CONFIG

Deprecated. Kakfa Streams does not use Zookeeper anymore and this parameter will be ignored.

zookeeper.connect

See Also:

Constant Field Values

COMMIT_INTERVAL_MS_CONFIG

public static final java.lang.String COMMIT_INTERVAL_MS_CONFIG

commit.interval.ms

See Also:

Constant Field Values

POLL_MS_CONFIG

public static final java.lang.String POLL_MS_CONFIG

poll.ms

See Also:

Constant Field Values

NUM_STREAM_THREADS_CONFIG

public static final java.lang.String NUM_STREAM_THREADS_CONFIG

num.stream.threads

See Also:

Constant Field Values

NUM_STANDBY_REPLICAS_CONFIG

public static final java.lang.String NUM_STANDBY_REPLICAS_CONFIG

num.standby.replicas

See Also:

Constant Field Values

BUFFERED_RECORDS_PER_PARTITION_CONFIG

public static final java.lang.String BUFFERED_RECORDS_PER_PARTITION_CONFIG

buffered.records.per.partition

See Also:

Constant Field Values

STATE_CLEANUP_DELAY_MS_CONFIG

public static final java.lang.String STATE_CLEANUP_DELAY_MS_CONFIG

state.cleanup.delay

See Also:

Constant Field Values

TIMESTAMP_EXTRACTOR_CLASS_CONFIG

public static final java.lang.String TIMESTAMP_EXTRACTOR_CLASS_CONFIG

timestamp.extractor

See Also:

Constant Field Values

PARTITION_GROUPER_CLASS_CONFIG

public static final java.lang.String PARTITION_GROUPER_CLASS_CONFIG

partition.grouper

See Also:

Constant Field Values

APPLICATION_ID_CONFIG

public static final java.lang.String APPLICATION_ID_CONFIG

application.id

See Also:

Constant Field Values

REPLICATION_FACTOR_CONFIG

public static final java.lang.String REPLICATION_FACTOR_CONFIG

replication.factor

See Also:

Constant Field Values

KEY_SERDE_CLASS_CONFIG

public static final java.lang.String KEY_SERDE_CLASS_CONFIG

key.serde

See Also:

Constant Field Values

VALUE_SERDE_CLASS_CONFIG

public static final java.lang.String VALUE_SERDE_CLASS_CONFIG

value.serde

See Also:

Constant Field Values

APPLICATION_SERVER_CONFIG

public static final java.lang.String APPLICATION_SERVER_CONFIG

user.endpoint

See Also:

Constant Field Values

METRICS_SAMPLE_WINDOW_MS_CONFIG

public static final java.lang.String METRICS_SAMPLE_WINDOW_MS_CONFIG

metrics.sample.window.ms

See Also:

Constant Field Values

METRICS_NUM_SAMPLES_CONFIG

public static final java.lang.String METRICS_NUM_SAMPLES_CONFIG

metrics.num.samples

See Also:

Constant Field Values

METRICS_RECORDING_LEVEL_CONFIG

public static final java.lang.String METRICS_RECORDING_LEVEL_CONFIG

metrics.record.level

See Also:

Constant Field Values

METRIC_REPORTER_CLASSES_CONFIG

public static final java.lang.String METRIC_REPORTER_CLASSES_CONFIG

metric.reporters

See Also:

Constant Field Values

BOOTSTRAP_SERVERS_CONFIG

public static final java.lang.String BOOTSTRAP_SERVERS_CONFIG

bootstrap.servers

See Also:

Constant Field Values

CLIENT_ID_CONFIG

public static final java.lang.String CLIENT_ID_CONFIG

client.id

See Also:

Constant Field Values

ROCKSDB_CONFIG_SETTER_CLASS_CONFIG

public static final java.lang.String ROCKSDB_CONFIG_SETTER_CLASS_CONFIG

rocksdb.config.setter

See Also:

Constant Field Values

WINDOW_STORE_CHANGE_LOG_ADDITIONAL_RETENTION_MS_CONFIG

public static final java.lang.String WINDOW_STORE_CHANGE_LOG_ADDITIONAL_RETENTION_MS_CONFIG

windowstore.changelog.additional.retention.ms

See Also:

Constant Field Values

CACHE_MAX_BYTES_BUFFERING_CONFIG

public static final java.lang.String CACHE_MAX_BYTES_BUFFERING_CONFIG

cache.max.bytes.buffering

See Also:

Constant Field Values

SECURITY_PROTOCOL_CONFIG

public static final java.lang.String SECURITY_PROTOCOL_CONFIG

See Also:

Constant Field Values

DEFAULT_SECURITY_PROTOCOL

public static final java.lang.String DEFAULT_SECURITY_PROTOCOL

See Also:

Constant Field Values

CONNECTIONS_MAX_IDLE_MS_CONFIG

public static final java.lang.String CONNECTIONS_MAX_IDLE_MS_CONFIG

See Also:

Constant Field Values

RETRY_BACKOFF_MS_CONFIG

public static final java.lang.String RETRY_BACKOFF_MS_CONFIG

See Also:

Constant Field Values

METADATA_MAX_AGE_CONFIG

public static final java.lang.String METADATA_MAX_AGE_CONFIG

See Also:

Constant Field Values

RECONNECT_BACKOFF_MS_CONFIG

public static final java.lang.String RECONNECT_BACKOFF_MS_CONFIG

See Also:

Constant Field Values

SEND_BUFFER_CONFIG

public static final java.lang.String SEND_BUFFER_CONFIG

See Also:

Constant Field Values

RECEIVE_BUFFER_CONFIG

public static final java.lang.String RECEIVE_BUFFER_CONFIG

See Also:

Constant Field Values

REQUEST_TIMEOUT_MS_CONFIG

public static final java.lang.String REQUEST_TIMEOUT_MS_CONFIG

See Also:

Constant Field Values

Constructor Detail

StreamsConfig

public StreamsConfig(java.util.Map<?,?> props)

Create a new StreamsConfig using the given properties.

Parameters:

props - properties that specify Kafka Streams and internal consumer/producer configuration

Method Detail

consumerPrefix

public static java.lang.String consumerPrefix(java.lang.String consumerProp)

Prefix a property with CONSUMER_PREFIX. This is used to isolate consumer configs from producer configs.

Parameters:

consumerProp - the consumer property to be masked

Returns:

CONSUMER_PREFIX + consumerProp

producerPrefix

public static java.lang.String producerPrefix(java.lang.String producerProp)

Prefix a property with PRODUCER_PREFIX. This is used to isolate producer configs from consumer configs.

Parameters:

producerProp - the producer property to be masked

Returns:

PRODUCER_PREFIX + producerProp

configDef

public static org.apache.kafka.common.config.ConfigDef configDef()

Return a copy of the config definition.

Returns:

a copy of the config definition

getConsumerConfigs

public java.util.Map<java.lang.String,java.lang.Object> getConsumerConfigs(org.apache.kafka.streams.processor.internals.StreamThread streamThread,
                                                                  java.lang.String groupId,
                                                                  java.lang.String clientId)
                                                                    throws org.apache.kafka.common.config.ConfigException

Get the configs to the consumer. Properties using the prefix CONSUMER_PREFIX will be used in favor over their non-prefixed versions except in the case of ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG where we always use the non-prefixed version as we only support reading/writing from/to the same Kafka Cluster.

Parameters:

streamThread - the StreamThread creating a consumer

groupId - consumer groupId

clientId - clientId

Returns:

Map of the consumer configuration.

Throws:

org.apache.kafka.common.config.ConfigException - if "enable.auto.commit" was set to false by the user

getRestoreConsumerConfigs

public java.util.Map<java.lang.String,java.lang.Object> getRestoreConsumerConfigs(java.lang.String clientId)
                                                                           throws org.apache.kafka.common.config.ConfigException

Get the configs for the restore-consumer. Properties using the prefix CONSUMER_PREFIX will be used in favor over their non-prefixed versions except in the case of ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG where we always use the non-prefixed version as we only support reading/writing from/to the same Kafka Cluster.

Parameters:

clientId - clientId

Returns:

Map of the consumer configuration.

Throws:

org.apache.kafka.common.config.ConfigException - if "enable.auto.commit" was set to false by the user

getProducerConfigs

public java.util.Map<java.lang.String,java.lang.Object> getProducerConfigs(java.lang.String clientId)

Get the configs for the producer. Properties using the prefix PRODUCER_PREFIX will be used in favor over their non-prefixed versions except in the case of ProducerConfig.BOOTSTRAP_SERVERS_CONFIG where we always use the non-prefixed version as we only support reading/writing from/to the same Kafka Cluster.

Parameters:

clientId - clientId

Returns:

Map of the producer configuration.

keySerde

public org.apache.kafka.common.serialization.Serde keySerde()

Return an configured instance of key Serde class.

Returns:

an configured instance of key Serde class

valueSerde

public org.apache.kafka.common.serialization.Serde valueSerde()

Return an configured instance of value Serde class.

Returns:

an configured instance of value Serde class

main

public static void main(java.lang.String[] args)