Package io.debezium.relational

Class HistorizedRelationalDatabaseConnectorConfig

java.lang.Object

io.debezium.config.CommonConnectorConfig

io.debezium.relational.RelationalDatabaseConnectorConfig

io.debezium.relational.HistorizedRelationalDatabaseConnectorConfig

public abstract class HistorizedRelationalDatabaseConnectorConfig
extends RelationalDatabaseConnectorConfig

Configuration options shared across the relational CDC connectors which use a persistent database schema history.

Author:

Gunnar Morling

Nested Class Summary

Nested classes/interfaces inherited from class io.debezium.relational.RelationalDatabaseConnectorConfig

RelationalDatabaseConnectorConfig.DecimalHandlingMode, RelationalDatabaseConnectorConfig.SnapshotTablesRowCountOrder

Nested classes/interfaces inherited from class io.debezium.config.CommonConnectorConfig

CommonConnectorConfig.BinaryHandlingMode, CommonConnectorConfig.EventConvertingFailureHandlingMode, CommonConnectorConfig.EventProcessingFailureHandlingMode, CommonConnectorConfig.FieldNameAdjustmentMode, CommonConnectorConfig.SchemaNameAdjustmentMode, CommonConnectorConfig.SnapshotQueryMode, CommonConnectorConfig.Version, CommonConnectorConfig.WatermarkStrategy

Field Summary

Fields

Modifier and Type	Field	Description
protected static final ConfigDefinition	CONFIG_DEFINITION
private final Class<? extends org.apache.kafka.connect.source.SourceConnector>	connectorClass
private final Predicate<String>	ddlFilter
private static final String	DEFAULT_SCHEMA_HISTORY
protected static final int	DEFAULT_SNAPSHOT_FETCH_SIZE
private final boolean	multiPartitionMode
static final Field	SCHEMA_HISTORY	The database schema history class is hidden in the invalid reference #configDef() since that is designed to work with a user interface, and in these situations using Kafka is the only way to go.
static final Field	SKIP_UNPARSEABLE_DDL_STATEMENTS
protected boolean	skipUnparseableDDL
static final Field	STORE_ONLY_CAPTURED_DATABASES_DDL
static final Field	STORE_ONLY_CAPTURED_TABLES_DDL
protected boolean	storeOnlyCapturedDatabasesDdl
protected boolean	storeOnlyCapturedTablesDdl
private final boolean	useCatalogBeforeSchema

Fields inherited from class io.debezium.relational.RelationalDatabaseConnectorConfig

COLUMN_EXCLUDE_LIST, COLUMN_INCLUDE_LIST, COLUMN_INCLUDE_LIST_ALREADY_SPECIFIED_ERROR_MSG, DATABASE_EXCLUDE_LIST, DATABASE_EXCLUDE_LIST_NAME, DATABASE_INCLUDE_LIST, DATABASE_INCLUDE_LIST_ALREADY_SPECIFIED_ERROR_MSG, DATABASE_INCLUDE_LIST_NAME, DATABASE_NAME, DECIMAL_HANDLING_MODE, DEFAULT_SNAPSHOT_LOCK_TIMEOUT_MILLIS, DEFAULT_UNAVAILABLE_VALUE_PLACEHOLDER, HOSTNAME, HOSTNAME_PATTERN, INCLUDE_SCHEMA_CHANGES, INCLUDE_SCHEMA_COMMENTS, MASK_COLUMN, MASK_COLUMN_WITH_HASH, MSG_KEY_COLUMNS, PASSWORD, PORT, PROPAGATE_COLUMN_SOURCE_TYPE, PROPAGATE_DATATYPE_SOURCE_TYPE, QUERY_TIMEOUT_MS, SCHEMA_EXCLUDE_LIST, SCHEMA_EXCLUDE_LIST_NAME, SCHEMA_INCLUDE_LIST, SCHEMA_INCLUDE_LIST_ALREADY_SPECIFIED_ERROR_MSG, SCHEMA_INCLUDE_LIST_NAME, SNAPSHOT_FULL_COLUMN_SCAN_FORCE, SNAPSHOT_LOCK_TIMEOUT_MS, SNAPSHOT_SELECT_STATEMENT_OVERRIDES_BY_TABLE, SNAPSHOT_TABLES_ORDER_BY_ROW_COUNT, TABLE_EXCLUDE_LIST, TABLE_EXCLUDE_LIST_NAME, TABLE_IGNORE_BUILTIN, TABLE_INCLUDE_LIST, TABLE_INCLUDE_LIST_ALREADY_SPECIFIED_ERROR_MSG, TABLE_INCLUDE_LIST_NAME, TIME_PRECISION_MODE, TRUNCATE_COLUMN, UNAVAILABLE_VALUE_PLACEHOLDER, USER

Fields inherited from class io.debezium.config.CommonConnectorConfig

ADVANCED_METRICS_ENABLE, beanRegistry, BINARY_HANDLING_MODE, CUSTOM_CONVERTERS, CUSTOM_METRIC_TAGS, CUSTOM_POST_PROCESSORS, CUSTOM_RETRIABLE_EXCEPTION, DATABASE_CONFIG_PREFIX, DEFAULT_MAX_BATCH_SIZE, DEFAULT_MAX_QUEUE_SIZE, DEFAULT_MAX_QUEUE_SIZE_IN_BYTES, DEFAULT_MAX_RETRIES, DEFAULT_POLL_INTERVAL_MILLIS, DEFAULT_QUERY_FETCH_SIZE, DEFAULT_RETRIABLE_RESTART_WAIT, DRIVER_CONFIG_PREFIX, ERRORS_MAX_RETRIES, EVENT_CONVERTING_FAILURE_HANDLING_MODE, EVENT_PROCESSING_FAILURE_HANDLING_MODE, EXECUTOR_SHUTDOWN_TIMEOUT_SEC, FIELD_NAME_ADJUSTMENT_MODE, INCREMENTAL_SNAPSHOT_ALLOW_SCHEMA_CHANGES, INCREMENTAL_SNAPSHOT_CHUNK_SIZE, INCREMENTAL_SNAPSHOT_WATERMARKING_STRATEGY, isAdvancedMetricsEnabled, isLogPositionCheckEnabled, LOG_POSITION_CHECK_ENABLED, logicalName, MAX_BATCH_SIZE, MAX_QUEUE_SIZE, MAX_QUEUE_SIZE_IN_BYTES, MAX_RETRIES_ON_ERROR, MULTI_PARTITION_MODE, NOTIFICATION_CONFIGURATION_FIELD_PREFIX_STRING, NOTIFICATION_ENABLED_CHANNELS, POLL_INTERVAL_MS, PROVIDE_TRANSACTION_METADATA, QUERY_FETCH_SIZE, RETRIABLE_RESTART_WAIT, SCHEMA_NAME_ADJUSTMENT_MODE, serviceRegistry, SIGNAL_DATA_COLLECTION, SIGNAL_ENABLED_CHANNELS, SIGNAL_POLL_INTERVAL_MS, SKIP_MESSAGES_WITHOUT_CHANGE, SKIPPED_OPERATIONS, SNAPSHOT_DELAY_MS, SNAPSHOT_FETCH_SIZE, SNAPSHOT_LOCKING_MODE_CUSTOM_NAME, SNAPSHOT_LOCKING_MODE_PROPERTY_NAME, SNAPSHOT_MAX_THREADS, SNAPSHOT_MODE_CONFIGURATION_BASED_SNAPSHOT_DATA, SNAPSHOT_MODE_CONFIGURATION_BASED_SNAPSHOT_ON_DATA_ERROR, SNAPSHOT_MODE_CONFIGURATION_BASED_SNAPSHOT_ON_SCHEMA_ERROR, SNAPSHOT_MODE_CONFIGURATION_BASED_SNAPSHOT_SCHEMA, SNAPSHOT_MODE_CONFIGURATION_BASED_START_STREAM, SNAPSHOT_MODE_CUSTOM_NAME, SNAPSHOT_MODE_PROPERTY_NAME, SNAPSHOT_MODE_TABLES, SNAPSHOT_QUERY_MODE, SNAPSHOT_QUERY_MODE_CUSTOM_NAME, snapshotLockingModeCustomName, snapshotModeConfigurationBasedSnapshotData, snapshotModeConfigurationBasedSnapshotOnDataError, snapshotModeConfigurationBasedSnapshotOnSchemaError, snapshotModeConfigurationBasedSnapshotSchema, snapshotModeConfigurationBasedStream, snapshotQueryMode, snapshotQueryModeCustomName, SOURCE_INFO_STRUCT_MAKER, STREAMING_DELAY_MS, TASK_ID, TOMBSTONES_ON_DELETE, TOPIC_NAME_PATTERN, TOPIC_NAMING_STRATEGY, TOPIC_PREFIX, TRANSACTION_METADATA_FACTORY

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	HistorizedRelationalDatabaseConnectorConfig(Class<? extends org.apache.kafka.connect.source.SourceConnector> connectorClass, Configuration config, Tables.TableFilter systemTablesFilter, boolean useCatalogBeforeSchema, int defaultSnapshotFetchSize, ColumnFilterMode columnFilterMode, boolean multiPartitionMode)
protected	HistorizedRelationalDatabaseConnectorConfig(Class<? extends org.apache.kafka.connect.source.SourceConnector> connectorClass, Configuration config, Tables.TableFilter systemTablesFilter, Selectors.TableIdToStringMapper tableIdMapper, boolean useCatalogBeforeSchema, int defaultSnapshotFetchSize, ColumnFilterMode columnFilterMode, boolean multiPartitionMode)
protected	HistorizedRelationalDatabaseConnectorConfig(Class<? extends org.apache.kafka.connect.source.SourceConnector> connectorClass, Configuration config, Tables.TableFilter systemTablesFilter, Selectors.TableIdToStringMapper tableIdMapper, boolean useCatalogBeforeSchema, ColumnFilterMode columnFilterMode, boolean multiPartitionMode)

Method Summary

Modifier and Type	Method	Description
private Predicate<String>	createDdlFilter(Configuration config)
Predicate<String>	ddlFilter()
protected abstract HistoryRecordComparator	getHistoryRecordComparator()	Returns a comparator to be used when recovering records from the schema history, making sure no history entries newer than the offset we resume from are recovered (which could happen when restarting a connector after history records have been persisted but no new offset has been committed yet).
SchemaHistory	getSchemaHistory()	Returns a configured (but not yet started) instance of the database schema history.
boolean	multiPartitionMode()
boolean	skipUnparseableDdlStatements()
boolean	storeOnlyCapturedDatabases()
boolean	storeOnlyCapturedTables()
boolean	useCatalogBeforeSchema()

Methods inherited from class io.debezium.relational.RelationalDatabaseConnectorConfig

createHeartbeat, getColumnFilter, getDecimalMode, getFieldNamer, getHeartbeatActionQuery, getJdbcConfig, getKeyMapper, getSnapshotSelectOverridesByTable, getTableFilters, getTableIdMapper, getTemporalPrecisionMode, getUnavailableValuePlaceholder, isColumnsFiltered, isFullColumnScanRequired, isSchemaChangesHistoryEnabled, isSchemaCommentsHistoryEnabled, schemaExcludeList, schemaIncludeList, snapshotLockTimeout, snapshotOrderByRowCount, tableExcludeList, tableIncludeList

Methods inherited from class io.debezium.config.CommonConnectorConfig

binaryHandlingMode, createCustomMetricTags, customConverterRegistry, customRetriableException, fieldNameAdjuster, getBeanRegistry, getConfig, getConnectorName, getContextName, getCustomMetricTags, getDataCollectionsToBeSnapshotted, getEnabledChannels, getEnabledNotificationChannels, getEventConvertingFailureHandlingMode, getEventProcessingFailureHandlingMode, getHeartbeatInterval, getHeartbeatTopicsPrefix, getIncrementalSnapshotChunkSize, getIncrementalSnapshotWatermarkingStrategy, getLogicalName, getMaxBatchSize, getMaxQueueSize, getMaxQueueSizeInBytes, getMaxRetriesOnError, getNotificationTopic, getPollInterval, getQueryFetchSize, getServiceRegistry, getSignalingDataCollectionId, getSignalPollInterval, getSkippedOperations, getSnapshotDelay, getSnapshotFetchSize, getSnapshotLockingMode, getSnapshotMaxThreads, getSnapshotMode, getSnapshotModeCustomName, getSourceInfoStructMaker, getSourceInfoStructMaker, getSourceInfoStructMaker, getStreamingDelay, getTaskId, getTopicNamingStrategy, getTopicNamingStrategy, getTransactionMetadataFactory, getTransactionMetadataFactory, isAdvancedMetricsEnabled, isEmitTombstoneOnDelete, isIncrementalSnapshotSchemaChangesEnabled, isLogPositionCheckEnabled, isSignalDataCollection, parseSignallingMessage, schemaNameAdjuster, shouldProvideTransactionMetadata, skipMessagesWithoutChange, snapshotLockingModeCustomName, snapshotModeConfigurationBasedSnapshotData, snapshotModeConfigurationBasedSnapshotOnDataError, snapshotModeConfigurationBasedSnapshotOnSchemaError, snapshotModeConfigurationBasedSnapshotSchema, snapshotModeConfigurationBasedStream, snapshotQueryMode, snapshotQueryModeCustomName, supportsOperationFiltering, supportsSchemaChangesDuringIncrementalSnapshot, validate, validateAndRecord, validateSkippedOperation, validateTopicName

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

DEFAULT_SNAPSHOT_FETCH_SIZE

protected static final int DEFAULT_SNAPSHOT_FETCH_SIZE

See Also:

• Constant Field Values

DEFAULT_SCHEMA_HISTORY

private static final String DEFAULT_SCHEMA_HISTORY

See Also:

• Constant Field Values

useCatalogBeforeSchema

private final boolean useCatalogBeforeSchema

connectorClass

private final Class<? extends org.apache.kafka.connect.source.SourceConnector> connectorClass

multiPartitionMode

private final boolean multiPartitionMode

ddlFilter

private final Predicate<String> ddlFilter

skipUnparseableDDL

protected boolean skipUnparseableDDL

storeOnlyCapturedTablesDdl

protected boolean storeOnlyCapturedTablesDdl

storeOnlyCapturedDatabasesDdl

protected boolean storeOnlyCapturedDatabasesDdl

SCHEMA_HISTORY

public static final Field SCHEMA_HISTORY

The database schema history class is hidden in the

invalid reference

#configDef()

since that is designed to work with a user interface, and in these situations using Kafka is the only way to go.

SKIP_UNPARSEABLE_DDL_STATEMENTS

public static final Field SKIP_UNPARSEABLE_DDL_STATEMENTS

STORE_ONLY_CAPTURED_TABLES_DDL

public static final Field STORE_ONLY_CAPTURED_TABLES_DDL

STORE_ONLY_CAPTURED_DATABASES_DDL

public static final Field STORE_ONLY_CAPTURED_DATABASES_DDL

CONFIG_DEFINITION

protected static final ConfigDefinition CONFIG_DEFINITION

Constructor Details

HistorizedRelationalDatabaseConnectorConfig

protected HistorizedRelationalDatabaseConnectorConfig(Class<? extends org.apache.kafka.connect.source.SourceConnector> connectorClass,
 Configuration config,
 Tables.TableFilter systemTablesFilter,
 boolean useCatalogBeforeSchema,
 int defaultSnapshotFetchSize,
 ColumnFilterMode columnFilterMode,
 boolean multiPartitionMode)

HistorizedRelationalDatabaseConnectorConfig

protected HistorizedRelationalDatabaseConnectorConfig(Class<? extends org.apache.kafka.connect.source.SourceConnector> connectorClass,
 Configuration config,
 Tables.TableFilter systemTablesFilter,
 Selectors.TableIdToStringMapper tableIdMapper,
 boolean useCatalogBeforeSchema,
 ColumnFilterMode columnFilterMode,
 boolean multiPartitionMode)

HistorizedRelationalDatabaseConnectorConfig

protected HistorizedRelationalDatabaseConnectorConfig(Class<? extends org.apache.kafka.connect.source.SourceConnector> connectorClass,
 Configuration config,
 Tables.TableFilter systemTablesFilter,
 Selectors.TableIdToStringMapper tableIdMapper,
 boolean useCatalogBeforeSchema,
 int defaultSnapshotFetchSize,
 ColumnFilterMode columnFilterMode,
 boolean multiPartitionMode)

Method Details

getSchemaHistory

public SchemaHistory getSchemaHistory()

Returns a configured (but not yet started) instance of the database schema history.

useCatalogBeforeSchema

public boolean useCatalogBeforeSchema()

multiPartitionMode

public boolean multiPartitionMode()

createDdlFilter

private Predicate<String> createDdlFilter(Configuration config)

ddlFilter

public Predicate<String> ddlFilter()

skipUnparseableDdlStatements

public boolean skipUnparseableDdlStatements()

storeOnlyCapturedTables

public boolean storeOnlyCapturedTables()

storeOnlyCapturedDatabases

public boolean storeOnlyCapturedDatabases()

getHistoryRecordComparator

protected abstract HistoryRecordComparator getHistoryRecordComparator()

Returns a comparator to be used when recovering records from the schema history, making sure no history entries newer than the offset we resume from are recovered (which could happen when restarting a connector after history records have been persisted but no new offset has been committed yet).