Package io.debezium.relational.history

Class KafkaDatabaseHistory

java.lang.Object

io.debezium.relational.history.AbstractDatabaseHistory

io.debezium.relational.history.KafkaDatabaseHistory

All Implemented Interfaces:

DatabaseHistory

@NotThreadSafe
public class KafkaDatabaseHistory
extends AbstractDatabaseHistory

A DatabaseHistory implementation that records schema changes as normal SourceRecords on the specified topic, and that recovers the history by establishing a Kafka Consumer re-processing all messages on that topic.

Author:

Randall Hauch

Field Summary

Fields

Modifier and Type	Field	Description
static Field.Set	ALL_FIELDS
static final Field	BOOTSTRAP_SERVERS
private ExecutorService	checkTopicSettingsExecutor
private static final String	CLEANUP_POLICY_NAME
private static final String	CLEANUP_POLICY_VALUE
private static final String	CONSUMER_PREFIX
private Configuration	consumerConfig
private static final short	DEFAULT_TOPIC_REPLICATION_FACTOR	The default replication factor for the history topic which is used in case the value couldn't be retrieved from the broker.
private static final String	DEFAULT_TOPIC_REPLICATION_FACTOR_PROP_NAME	The name of broker property defining default replication factor for topics without the explicit setting.
static final Field	INTERNAL_CONNECTOR_CLASS
static final Field	INTERNAL_CONNECTOR_ID
static final Field	KAFKA_QUERY_TIMEOUT_MS
private Duration	kafkaQueryTimeout
private static final org.slf4j.Logger	LOGGER
private int	maxRecoveryAttempts
private static final Integer	PARTITION	The one and only partition of the history topic.
private static final int	PARTITION_COUNT
private Duration	pollInterval
private org.apache.kafka.clients.producer.KafkaProducer<String,String>	producer
private static final String	PRODUCER_PREFIX
private Configuration	producerConfig
private final DocumentReader	reader
static final Field	RECOVERY_POLL_ATTEMPTS
static final Field	RECOVERY_POLL_INTERVAL_MS
private static final String	RETENTION_BYTES_NAME
private static final long	RETENTION_MS_MAX
private static final long	RETENTION_MS_MIN
private static final String	RETENTION_MS_NAME
static final Field	TOPIC
private String	topicName
private static final int	UNLIMITED_VALUE
private static final boolean	USE_KAFKA_24_NEW_TOPIC_CONSTRUCTOR

Fields inherited from class io.debezium.relational.history.AbstractDatabaseHistory

config, INTERNAL_PREFER_DDL, logger

Fields inherited from interface io.debezium.relational.history.DatabaseHistory

CONFIGURATION_FIELD_PREFIX_STRING, DDL_FILTER, NAME, SKIP_UNPARSEABLE_DDL_STATEMENTS, STORE_ONLY_CAPTURED_TABLES_DDL

Constructor Summary

Constructors

Constructor	Description
KafkaDatabaseHistory()

Method Summary

Modifier and Type	Method	Description
private void	checkTopicSettings(String topicName)
void	configure(Configuration config, HistoryRecordComparator comparator, DatabaseHistoryListener listener, boolean useCatalogBeforeSchema)	Configure this instance.
protected static String	consumerConfigPropertyName(String kafkaConsumerPropertyName)
boolean	exists()	Determines if the database history entity exists; i.e.
private static Field.Validator	forKafka(Field.Validator validator)
private short	getDefaultTopicReplicationFactor(org.apache.kafka.clients.admin.AdminClient admin)
private Long	getEndOffsetOfDbHistoryTopic(Long previousEndOffset, org.apache.kafka.clients.consumer.KafkaConsumer<String,String> historyConsumer)
private org.apache.kafka.clients.admin.Config	getKafkaBrokerConfig(org.apache.kafka.clients.admin.AdminClient admin)
private static boolean	hasNewTopicConstructorWithOptionals()
void	initializeStorage()	Called to initialize permanent storage of the history.
protected void	recoverRecords(Consumer<HistoryRecord> records)
void	start()	Start the history.
void	stop()	Stop recording history and release any resources acquired since DatabaseHistory.configure(Configuration, HistoryRecordComparator, DatabaseHistoryListener, boolean).
private void	stopCheckTopicSettingsExecutor()
boolean	storageExists()	Determines if the underlying storage exists (e.g.
protected void	storeRecord(HistoryRecord record)
String	toString()

Methods inherited from class io.debezium.relational.history.AbstractDatabaseHistory

record, record, recover, skipUnparseableDdlStatements, storeOnlyCapturedTables

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface io.debezium.relational.history.DatabaseHistory

recover, recover

Field Details

LOGGER

private static final org.slf4j.Logger LOGGER

CLEANUP_POLICY_NAME

private static final String CLEANUP_POLICY_NAME

See Also:

• Constant Field Values

CLEANUP_POLICY_VALUE

private static final String CLEANUP_POLICY_VALUE

See Also:

• Constant Field Values

RETENTION_MS_NAME

private static final String RETENTION_MS_NAME

See Also:

• Constant Field Values

RETENTION_MS_MAX

private static final long RETENTION_MS_MAX

See Also:

• Constant Field Values

RETENTION_MS_MIN

private static final long RETENTION_MS_MIN

RETENTION_BYTES_NAME

private static final String RETENTION_BYTES_NAME

See Also:

• Constant Field Values

UNLIMITED_VALUE

private static final int UNLIMITED_VALUE

See Also:

• Constant Field Values

PARTITION_COUNT

private static final int PARTITION_COUNT

See Also:

• Constant Field Values

DEFAULT_TOPIC_REPLICATION_FACTOR_PROP_NAME

private static final String DEFAULT_TOPIC_REPLICATION_FACTOR_PROP_NAME

The name of broker property defining default replication factor for topics without the explicit setting.

See Also:

• kafka.server.KafkaConfig.DefaultReplicationFactorProp
• Constant Field Values

DEFAULT_TOPIC_REPLICATION_FACTOR

private static final short DEFAULT_TOPIC_REPLICATION_FACTOR

The default replication factor for the history topic which is used in case the value couldn't be retrieved from the broker.

See Also:

• Constant Field Values

TOPIC

public static final Field TOPIC

BOOTSTRAP_SERVERS

public static final Field BOOTSTRAP_SERVERS

RECOVERY_POLL_INTERVAL_MS

public static final Field RECOVERY_POLL_INTERVAL_MS

RECOVERY_POLL_ATTEMPTS

public static final Field RECOVERY_POLL_ATTEMPTS

INTERNAL_CONNECTOR_CLASS

public static final Field INTERNAL_CONNECTOR_CLASS

INTERNAL_CONNECTOR_ID

public static final Field INTERNAL_CONNECTOR_ID

KAFKA_QUERY_TIMEOUT_MS

public static final Field KAFKA_QUERY_TIMEOUT_MS

ALL_FIELDS

public static Field.Set ALL_FIELDS

CONSUMER_PREFIX

private static final String CONSUMER_PREFIX

See Also:

• Constant Field Values

PRODUCER_PREFIX

private static final String PRODUCER_PREFIX

See Also:

• Constant Field Values

PARTITION

private static final Integer PARTITION

The one and only partition of the history topic.

reader

private final DocumentReader reader

topicName

private String topicName

consumerConfig

private Configuration consumerConfig

producerConfig

private Configuration producerConfig

producer

private volatile org.apache.kafka.clients.producer.KafkaProducer<String,String> producer

maxRecoveryAttempts

private int maxRecoveryAttempts

pollInterval

private Duration pollInterval

checkTopicSettingsExecutor

private ExecutorService checkTopicSettingsExecutor

kafkaQueryTimeout

private Duration kafkaQueryTimeout

USE_KAFKA_24_NEW_TOPIC_CONSTRUCTOR

private static final boolean USE_KAFKA_24_NEW_TOPIC_CONSTRUCTOR

Constructor Details

KafkaDatabaseHistory

public KafkaDatabaseHistory()

Method Details

hasNewTopicConstructorWithOptionals

private static boolean hasNewTopicConstructorWithOptionals()

configure

public void configure(Configuration config,
 HistoryRecordComparator comparator,
 DatabaseHistoryListener listener,
 boolean useCatalogBeforeSchema)

Description copied from interface: DatabaseHistory

Configure this instance.

Specified by:

configure in interface DatabaseHistory

Overrides:

configure in class AbstractDatabaseHistory

Parameters:

config - the configuration for this history store

comparator - the function that should be used to compare history records during recovery; may be null if the default comparator is to be used

listener - TODO

useCatalogBeforeSchema - true if the parsed string for a table contains only 2 items and the first should be used as the catalog and the second as the table name, or false if the first should be used as the schema and the second as the table name

start

public void start()

Description copied from interface: DatabaseHistory

Start the history.

Specified by:

start in interface DatabaseHistory

Overrides:

start in class AbstractDatabaseHistory

storeRecord

protected void storeRecord(HistoryRecord record)
                    throws DatabaseHistoryException

Specified by:

storeRecord in class AbstractDatabaseHistory

Throws:

DatabaseHistoryException

recoverRecords

protected void recoverRecords(Consumer<HistoryRecord> records)

Specified by:

recoverRecords in class AbstractDatabaseHistory

getEndOffsetOfDbHistoryTopic

private Long getEndOffsetOfDbHistoryTopic(Long previousEndOffset,
 org.apache.kafka.clients.consumer.KafkaConsumer<String,String> historyConsumer)

storageExists

public boolean storageExists()

Description copied from interface: DatabaseHistory

Determines if the underlying storage exists (e.g. a Kafka topic, file or similar). Note: storage may exist while history entities not yet written, see DatabaseHistory.exists()

exists

public boolean exists()

Description copied from interface: DatabaseHistory

Determines if the database history entity exists; i.e. the storage must have been initialized and the history must have been populated.

checkTopicSettings

private void checkTopicSettings(String topicName)

stop

public void stop()

Description copied from interface: DatabaseHistory

Stop recording history and release any resources acquired since DatabaseHistory.configure(Configuration, HistoryRecordComparator, DatabaseHistoryListener, boolean).

Specified by:

stop in interface DatabaseHistory

Overrides:

stop in class AbstractDatabaseHistory

stopCheckTopicSettingsExecutor

private void stopCheckTopicSettingsExecutor()

toString

public String toString()

Overrides:

toString in class Object

consumerConfigPropertyName

protected static String consumerConfigPropertyName(String kafkaConsumerPropertyName)

initializeStorage

public void initializeStorage()

Description copied from interface: DatabaseHistory

Called to initialize permanent storage of the history.

Specified by:

initializeStorage in interface DatabaseHistory

Overrides:

initializeStorage in class AbstractDatabaseHistory

getDefaultTopicReplicationFactor

private short getDefaultTopicReplicationFactor(org.apache.kafka.clients.admin.AdminClient admin)
                                        throws Exception

Throws:

Exception

getKafkaBrokerConfig

private org.apache.kafka.clients.admin.Config getKafkaBrokerConfig(org.apache.kafka.clients.admin.AdminClient admin)
                                                            throws Exception

Throws:

Exception

forKafka

private static Field.Validator forKafka(Field.Validator validator)