org.springframework.integration.kafka.inbound

Class KafkaMessageSource<K,V>

java.lang.Object

org.springframework.integration.util.AbstractExpressionEvaluator

org.springframework.integration.endpoint.AbstractMessageSource<java.lang.Object>

org.springframework.integration.kafka.inbound.KafkaMessageSource<K,V>

Type Parameters:

K - the key type.

V - the value type.

All Implemented Interfaces:

org.springframework.beans.factory.Aware, org.springframework.beans.factory.BeanFactoryAware, org.springframework.beans.factory.BeanNameAware, org.springframework.beans.factory.DisposableBean, org.springframework.beans.factory.InitializingBean, org.springframework.context.Lifecycle, org.springframework.integration.core.MessageSource<java.lang.Object>, org.springframework.integration.core.Pausable, org.springframework.integration.IntegrationPattern, org.springframework.integration.support.context.NamedComponent, org.springframework.integration.support.management.IntegrationInboundManagement, org.springframework.integration.support.management.IntegrationManagement, org.springframework.integration.support.management.ManageableLifecycle

public class KafkaMessageSource<K,V>
extends org.springframework.integration.endpoint.AbstractMessageSource<java.lang.Object>
implements org.springframework.integration.core.Pausable

Polled message source for kafka. Only one thread can poll for data (or acknowledge a message) at a time.

NOTE: If the application acknowledges messages out of order, the acks will be deferred until all messages prior to the offset are ack'd. If multiple records are retrieved and an earlier offset is requeued, records from the subsequent offsets will be redelivered - even if they were processed successfully. Applications should therefore implement idempotency.

Starting with version 3.1.2, this source implements Pausable which allows you to pause and resume the Consumer. While the consumer is paused, you must continue to call AbstractMessageSource.receive() within max.poll.interval.ms, to prevent a rebalance.

Since:

5.4

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	KafkaMessageSource.KafkaAckCallback<K,V> AcknowledgmentCallback for Kafka.
static class	KafkaMessageSource.KafkaAckCallbackFactory<K,V> AcknowledgmentCallbackFactory for KafkaAckInfo.
static interface	KafkaMessageSource.KafkaAckInfo<K,V> Information for building an KafkaAckCallback.
class	KafkaMessageSource.KafkaAckInfoImpl Information for building an KafkaAckCallback.

Nested classes/interfaces inherited from interface org.springframework.integration.support.management.IntegrationManagement

org.springframework.integration.support.management.IntegrationManagement.ManagementOverrides

Field Summary

Fields

Modifier and Type	Field and Description
boolean	newAssignment
static java.lang.String	REMAINING_RECORDS The number of records remaining from the previous poll.

Fields inherited from class org.springframework.integration.util.AbstractExpressionEvaluator

EXPRESSION_PARSER, logger

Fields inherited from interface org.springframework.integration.support.management.IntegrationManagement

METER_PREFIX, RECEIVE_COUNTER_NAME, SEND_TIMER_NAME

Constructor Summary

Constructors

Constructor and Description
KafkaMessageSource(org.springframework.kafka.core.ConsumerFactory<K,V> consumerFactory, org.springframework.kafka.listener.ConsumerProperties consumerProperties) Construct an instance with the supplied parameters.
KafkaMessageSource(org.springframework.kafka.core.ConsumerFactory<K,V> consumerFactory, org.springframework.kafka.listener.ConsumerProperties consumerProperties, boolean allowMultiFetch) Construct an instance with the supplied parameters.
KafkaMessageSource(org.springframework.kafka.core.ConsumerFactory<K,V> consumerFactory, org.springframework.kafka.listener.ConsumerProperties consumerProperties, KafkaMessageSource.KafkaAckCallbackFactory<K,V> ackCallbackFactory) Construct an instance with the supplied parameters.
KafkaMessageSource(org.springframework.kafka.core.ConsumerFactory<K,V> consumerFactory, org.springframework.kafka.listener.ConsumerProperties consumerProperties, KafkaMessageSource.KafkaAckCallbackFactory<K,V> ackCallbackFactory, boolean allowMultiFetch) Construct an instance with the supplied parameters.

Method Summary

Modifier and Type	Method and Description
protected void	createConsumer()
void	destroy()
protected java.lang.Object	doReceive()
java.util.Collection<org.apache.kafka.common.TopicPartition>	getAssignedPartitions() Return the currently assigned partitions.
protected java.lang.String	getClientId()
protected java.time.Duration	getCommitTimeout()
java.lang.String	getComponentType()
org.springframework.kafka.listener.ConsumerProperties	getConsumerProperties() Get a reference to the configured consumer properties; allows further customization of the properties before the source is started.
protected java.lang.String	getGroupId()
protected org.springframework.kafka.support.converter.RecordMessageConverter	getMessageConverter()
protected java.lang.Class<?>	getPayloadType()
protected long	getPollTimeout()
protected org.apache.kafka.clients.consumer.ConsumerRebalanceListener	getRebalanceListener()
boolean	isPaused()
protected boolean	isRawMessageHeader()
boolean	isRunning()
protected void	onInit()
void	pause()
void	resume()
void	setCloseTimeout(java.time.Duration closeTimeout) Set the close timeout - default 30 seconds.
void	setMessageConverter(org.springframework.kafka.support.converter.RecordMessageConverter messageConverter) Set the message converter to replace the default MessagingMessageConverter.
void	setPayloadType(java.lang.Class<?> payloadType) Set the payload type.
void	setRawMessageHeader(boolean rawMessageHeader) Set to true to include the raw ConsumerRecord as headers with keys KafkaHeaders.RAW_DATA and IntegrationMessageHeaderAccessor.SOURCE_DATA.
void	start()
void	stop()

Methods inherited from class org.springframework.integration.endpoint.AbstractMessageSource

buildMessage, getBeanName, getComponentName, getManagedName, getManagedType, getOverrides, isLoggingEnabled, receive, registerMetricsCaptor, setBeanName, setHeaderExpressions, setLoggingEnabled, setManagedName, setManagedType

Methods inherited from class org.springframework.integration.util.AbstractExpressionEvaluator

afterPropertiesSet, evaluateExpression, evaluateExpression, evaluateExpression, evaluateExpression, evaluateExpression, evaluateExpression, evaluateExpression, getBeanFactory, getEvaluationContext, getEvaluationContext, getMessageBuilderFactory, setBeanFactory, setConversionService

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.springframework.integration.core.MessageSource

getIntegrationPatternType

Methods inherited from interface org.springframework.integration.support.management.IntegrationManagement

getThisAs

Field Detail

REMAINING_RECORDS

public static final java.lang.String REMAINING_RECORDS

The number of records remaining from the previous poll.

Since:

3.2

See Also:

Constant Field Values

newAssignment

public boolean newAssignment

Constructor Detail

KafkaMessageSource

public KafkaMessageSource(org.springframework.kafka.core.ConsumerFactory<K,V> consumerFactory,
                          org.springframework.kafka.listener.ConsumerProperties consumerProperties)

Construct an instance with the supplied parameters. Fetching multiple records per poll will be disabled.

Parameters:

consumerFactory - the consumer factory.

consumerProperties - the consumer properties.

Since:

3.2

See Also:

KafkaMessageSource(ConsumerFactory, ConsumerProperties, KafkaAckCallbackFactory, boolean)

KafkaMessageSource

public KafkaMessageSource(org.springframework.kafka.core.ConsumerFactory<K,V> consumerFactory,
                          org.springframework.kafka.listener.ConsumerProperties consumerProperties,
                          boolean allowMultiFetch)

Construct an instance with the supplied parameters. Set 'allowMultiFetch' to true to allow up to max.poll.records to be fetched on each poll. When false (default) max.poll.records is coerced to 1 if the consumer factory is a DefaultKafkaConsumerFactory or otherwise rejected with an IllegalArgumentException. IMPORTANT: When true, you must call AbstractMessageSource.receive() at a sufficient rate to consume the number of records received within max.poll.interval.ms. When false, you must call AbstractMessageSource.receive() within max.poll.interval.ms. pause() will not take effect until the records from the previous poll are consumed.

Parameters:

consumerFactory - the consumer factory.

consumerProperties - the consumer properties.

allowMultiFetch - true to allow max.poll.records > 1.

Since:

3.2

KafkaMessageSource

public KafkaMessageSource(org.springframework.kafka.core.ConsumerFactory<K,V> consumerFactory,
                          org.springframework.kafka.listener.ConsumerProperties consumerProperties,
                          KafkaMessageSource.KafkaAckCallbackFactory<K,V> ackCallbackFactory)

Construct an instance with the supplied parameters. Fetching multiple records per poll will be disabled.

Parameters:

consumerFactory - the consumer factory.

consumerProperties - the consumer properties.

ackCallbackFactory - the ack callback factory.

Since:

3.2

See Also:

KafkaMessageSource(ConsumerFactory, ConsumerProperties, KafkaAckCallbackFactory, boolean)

KafkaMessageSource

public KafkaMessageSource(org.springframework.kafka.core.ConsumerFactory<K,V> consumerFactory,
                          org.springframework.kafka.listener.ConsumerProperties consumerProperties,
                          KafkaMessageSource.KafkaAckCallbackFactory<K,V> ackCallbackFactory,
                          boolean allowMultiFetch)

Construct an instance with the supplied parameters. Set 'allowMultiFetch' to true to allow up to max.poll.records to be fetched on each poll. When false (default) max.poll.records is coerced to 1 if the consumer factory is a DefaultKafkaConsumerFactory or otherwise rejected with an IllegalArgumentException. IMPORTANT: When true, you must call AbstractMessageSource.receive() at a sufficient rate to consume the number of records received within max.poll.interval.ms. When false, you must call AbstractMessageSource.receive() within max.poll.interval.ms. pause() will not take effect until the records from the previous poll are consumed.

Parameters:

consumerFactory - the consumer factory.

consumerProperties - the consumer properties.

ackCallbackFactory - the ack callback factory.

allowMultiFetch - true to allow max.poll.records > 1.

Since:

3.2

Method Detail

getAssignedPartitions

public java.util.Collection<org.apache.kafka.common.TopicPartition> getAssignedPartitions()

Return the currently assigned partitions.

Returns:

the partitions.

Since:

3.2.2

onInit

protected void onInit()

Overrides:

onInit in class org.springframework.integration.util.AbstractExpressionEvaluator

getConsumerProperties

public org.springframework.kafka.listener.ConsumerProperties getConsumerProperties()

Get a reference to the configured consumer properties; allows further customization of the properties before the source is started.

Returns:

the properties.

Since:

3.2

getGroupId

protected java.lang.String getGroupId()

getClientId

protected java.lang.String getClientId()

getPollTimeout

protected long getPollTimeout()

getMessageConverter

protected org.springframework.kafka.support.converter.RecordMessageConverter getMessageConverter()

setMessageConverter

public void setMessageConverter(org.springframework.kafka.support.converter.RecordMessageConverter messageConverter)

Set the message converter to replace the default MessagingMessageConverter.

Parameters:

messageConverter - the converter.

getPayloadType

protected java.lang.Class<?> getPayloadType()

setPayloadType

public void setPayloadType(java.lang.Class<?> payloadType)

Set the payload type. Only applies if a type-aware message converter is provided.

Parameters:

payloadType - the type to convert to.

getRebalanceListener

protected org.apache.kafka.clients.consumer.ConsumerRebalanceListener getRebalanceListener()

getComponentType

public java.lang.String getComponentType()

Specified by:

getComponentType in interface org.springframework.integration.support.context.NamedComponent

isRawMessageHeader

protected boolean isRawMessageHeader()

setRawMessageHeader

public void setRawMessageHeader(boolean rawMessageHeader)

Set to true to include the raw ConsumerRecord as headers with keys KafkaHeaders.RAW_DATA and IntegrationMessageHeaderAccessor.SOURCE_DATA. enabling callers to have access to the record to process errors.

Parameters:

rawMessageHeader - true to include the header.

getCommitTimeout

protected java.time.Duration getCommitTimeout()

setCloseTimeout

public void setCloseTimeout(java.time.Duration closeTimeout)

Set the close timeout - default 30 seconds.

Parameters:

closeTimeout - the close timeout.

isRunning

public boolean isRunning()

Specified by:

isRunning in interface org.springframework.context.Lifecycle

Specified by:

isRunning in interface org.springframework.integration.support.management.ManageableLifecycle

start

public void start()

Specified by:

start in interface org.springframework.context.Lifecycle

Specified by:

start in interface org.springframework.integration.support.management.ManageableLifecycle

stop

public void stop()

Specified by:

stop in interface org.springframework.context.Lifecycle

Specified by:

stop in interface org.springframework.integration.support.management.ManageableLifecycle

pause

public void pause()

Specified by:

pause in interface org.springframework.integration.core.Pausable

resume

public void resume()

Specified by:

resume in interface org.springframework.integration.core.Pausable

isPaused

public boolean isPaused()

Specified by:

isPaused in interface org.springframework.integration.core.Pausable

doReceive

protected java.lang.Object doReceive()

Specified by:

doReceive in class org.springframework.integration.endpoint.AbstractMessageSource<java.lang.Object>

createConsumer

protected void createConsumer()

destroy

public void destroy()

Specified by:

destroy in interface org.springframework.beans.factory.DisposableBean

Specified by:

destroy in interface org.springframework.integration.support.management.IntegrationManagement

Overrides:

destroy in class org.springframework.integration.endpoint.AbstractMessageSource<java.lang.Object>