Package io.debezium.connector.mysql

Class MySqlSnapshotChangeEventSource

java.lang.Object

io.debezium.pipeline.source.AbstractSnapshotChangeEventSource<P,O>

io.debezium.relational.RelationalSnapshotChangeEventSource<MySqlPartition,MySqlOffsetContext>

io.debezium.connector.mysql.MySqlSnapshotChangeEventSource

All Implemented Interfaces:

ChangeEventSource, SnapshotChangeEventSource<MySqlPartition,MySqlOffsetContext>, AutoCloseable

public class MySqlSnapshotChangeEventSource
extends RelationalSnapshotChangeEventSource<MySqlPartition,MySqlOffsetContext>

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
private static class	MySqlSnapshotChangeEventSource.MySqlSnapshotContext	Mutable context which is populated in the course of snapshotting.

Nested classes/interfaces inherited from class io.debezium.relational.RelationalSnapshotChangeEventSource

RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P extends Partition,O extends OffsetContext>

Nested classes/interfaces inherited from class io.debezium.pipeline.source.AbstractSnapshotChangeEventSource

AbstractSnapshotChangeEventSource.SnapshotContext<P extends Partition,O extends OffsetContext>, AbstractSnapshotChangeEventSource.SnapshottingTask

Nested classes/interfaces inherited from interface io.debezium.pipeline.source.spi.ChangeEventSource

ChangeEventSource.ChangeEventSourceContext

Field Summary

Fields

Modifier and Type	Field	Description
private final MySqlConnection	connection
private final MySqlConnectorConfig	connectorConfig
private final MySqlDatabaseSchema	databaseSchema
private Set<TableId>	delayedSchemaSnapshotTables
private final RelationalTableFilters	filters
private long	globalLockAcquiredAt
private final BlockingConsumer<Function<org.apache.kafka.connect.source.SourceRecord,org.apache.kafka.connect.source.SourceRecord>>	lastEventProcessor
private static final org.slf4j.Logger	LOGGER
private final MySqlSnapshotChangeEventSourceMetrics	metrics
private final List<SchemaChangeEvent>	schemaEvents
private long	tableLockAcquiredAt

Fields inherited from class io.debezium.relational.RelationalSnapshotChangeEventSource

clock, dispatcher, SELECT_ALL_PATTERN

Fields inherited from class io.debezium.pipeline.source.AbstractSnapshotChangeEventSource

LOG_INTERVAL

Constructor Summary

Constructors

Constructor

Description

MySqlSnapshotChangeEventSource(MySqlConnectorConfig connectorConfig, MainConnectionProvidingConnectionFactory<MySqlConnection> connectionFactory, MySqlDatabaseSchema schema, EventDispatcher<MySqlPartition,TableId> dispatcher, Clock clock, MySqlSnapshotChangeEventSourceMetrics metrics, BlockingConsumer<Function<org.apache.kafka.connect.source.SourceRecord,org.apache.kafka.connect.source.SourceRecord>> lastEventProcessor)

Method Summary

Modifier and Type	Method	Description
private void	addSchemaEvent(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlPartition,MySqlOffsetContext> snapshotContext, String database, String ddl)
protected MySqlOffsetContext	copyOffset(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlPartition,MySqlOffsetContext> snapshotContext)
protected void	createSchemaChangeEventsForTables(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlPartition,MySqlOffsetContext> snapshotContext, AbstractSnapshotChangeEventSource.SnapshottingTask snapshottingTask)
(package private) void	createSchemaEventsForTables(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlPartition,MySqlOffsetContext> snapshotContext, Collection<TableId> tablesToRead, boolean firstPhase)
private Statement	createStatementWithLargeResultSet(MySqlConnection connection)	Create a JDBC statement that can be used for large result sets.
protected void	determineSnapshotOffset(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlPartition,MySqlOffsetContext> ctx, MySqlOffsetContext previousOffset)
protected Set<TableId>	getAllTableIds(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlPartition,MySqlOffsetContext> ctx)
protected SchemaChangeEvent	getCreateTableEvent(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlPartition,MySqlOffsetContext> snapshotContext, Table table)
protected Optional<String>	getSnapshotConnectionFirstSelect(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlPartition,MySqlOffsetContext> snapshotContext, TableId tableId)
protected Optional<String>	getSnapshotSelect(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlPartition,MySqlOffsetContext> snapshotContext, TableId tableId, List<String> columns)	Generate a valid MySQL query string for the specified table and columns
private String	getSnapshotSelect(TableId tableId, List<String> columns)
protected AbstractSnapshotChangeEventSource.SnapshottingTask	getSnapshottingTask(MySqlPartition partition, MySqlOffsetContext previousOffset)
private void	globalLock()
private void	globalUnlock()
private boolean	isGloballyLocked()
private boolean	isTablesLocked()
protected void	lockTablesForSchemaSnapshot(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlPartition,MySqlOffsetContext> snapshotContext)
protected void	postSnapshot()
protected AbstractSnapshotChangeEventSource.SnapshotContext<MySqlPartition,MySqlOffsetContext>	prepare(MySqlPartition partition)
private String	quote(TableId id)
private String	quote(String dbOrTableName)
protected Statement	readTableStatement(JdbcConnection jdbcConnection, OptionalLong rowCount)
protected void	readTableStructure(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlPartition,MySqlOffsetContext> snapshotContext, MySqlOffsetContext offsetContext)
protected void	releaseDataSnapshotLocks(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlPartition,MySqlOffsetContext> snapshotContext)
protected void	releaseSchemaSnapshotLocks(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlPartition,MySqlOffsetContext> snapshotContext)
protected OptionalLong	rowCountForTable(TableId tableId)
private void	tableLock(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlPartition,MySqlOffsetContext> snapshotContext)
private void	tableUnlock()
private boolean	twoPhaseSchemaSnapshot()

Methods inherited from class io.debezium.relational.RelationalSnapshotChangeEventSource

additionalColumnFilter, connectionCreated, connectionPoolConnectionCreated, createSnapshotConnection, doExecute, enhanceOverriddenSelect, getChangeRecordEmitter, getClock, getPreparedColumnNames, getSnapshotSelectOverridesByTable, getSnapshotSourceTimestamp, getTablesForSchemaChange, lastSnapshotRecord, tryStartingSnapshot

Methods inherited from class io.debezium.pipeline.source.AbstractSnapshotChangeEventSource

aborted, close, completed, delaySnapshotIfNeeded, determineDataCollectionsToBeSnapshotted, execute

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

LOGGER

private static final org.slf4j.Logger LOGGER

connectorConfig

private final MySqlConnectorConfig connectorConfig

connection

private final MySqlConnection connection

globalLockAcquiredAt

private long globalLockAcquiredAt

tableLockAcquiredAt

private long tableLockAcquiredAt

filters

private final RelationalTableFilters filters

metrics

private final MySqlSnapshotChangeEventSourceMetrics metrics

databaseSchema

private final MySqlDatabaseSchema databaseSchema

schemaEvents

private final List<SchemaChangeEvent> schemaEvents

delayedSchemaSnapshotTables

private Set<TableId> delayedSchemaSnapshotTables

lastEventProcessor

private final BlockingConsumer<Function<org.apache.kafka.connect.source.SourceRecord,org.apache.kafka.connect.source.SourceRecord>> lastEventProcessor

Constructor Details

MySqlSnapshotChangeEventSource

public MySqlSnapshotChangeEventSource(MySqlConnectorConfig connectorConfig,
 MainConnectionProvidingConnectionFactory<MySqlConnection> connectionFactory,
 MySqlDatabaseSchema schema,
 EventDispatcher<MySqlPartition,TableId> dispatcher,
 Clock clock,
 MySqlSnapshotChangeEventSourceMetrics metrics,
 BlockingConsumer<Function<org.apache.kafka.connect.source.SourceRecord,org.apache.kafka.connect.source.SourceRecord>> lastEventProcessor)

Method Details

getSnapshottingTask

protected AbstractSnapshotChangeEventSource.SnapshottingTask getSnapshottingTask(MySqlPartition partition,
 MySqlOffsetContext previousOffset)

Specified by:

getSnapshottingTask in class AbstractSnapshotChangeEventSource<MySqlPartition,MySqlOffsetContext>

prepare

protected AbstractSnapshotChangeEventSource.SnapshotContext<MySqlPartition,MySqlOffsetContext> prepare(MySqlPartition partition)
                                                                                                throws Exception

Specified by:

prepare in class AbstractSnapshotChangeEventSource<MySqlPartition,MySqlOffsetContext>

Throws:

Exception

getAllTableIds

protected Set<TableId> getAllTableIds(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlPartition,MySqlOffsetContext> ctx)
                               throws Exception

Specified by:

getAllTableIds in class RelationalSnapshotChangeEventSource<MySqlPartition,MySqlOffsetContext>

Throws:

Exception

lockTablesForSchemaSnapshot

protected void lockTablesForSchemaSnapshot(ChangeEventSource.ChangeEventSourceContext sourceContext,
 RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlPartition,MySqlOffsetContext> snapshotContext)
                                    throws SQLException

Specified by:

lockTablesForSchemaSnapshot in class RelationalSnapshotChangeEventSource<MySqlPartition,MySqlOffsetContext>

Throws:

SQLException

releaseSchemaSnapshotLocks

protected void releaseSchemaSnapshotLocks(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlPartition,MySqlOffsetContext> snapshotContext)
                                   throws SQLException

Specified by:

releaseSchemaSnapshotLocks in class RelationalSnapshotChangeEventSource<MySqlPartition,MySqlOffsetContext>

Throws:

SQLException

releaseDataSnapshotLocks

protected void releaseDataSnapshotLocks(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlPartition,MySqlOffsetContext> snapshotContext)
                                 throws Exception

Overrides:

releaseDataSnapshotLocks in class RelationalSnapshotChangeEventSource<MySqlPartition,MySqlOffsetContext>

Throws:

Exception

determineSnapshotOffset

protected void determineSnapshotOffset(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlPartition,MySqlOffsetContext> ctx,
 MySqlOffsetContext previousOffset)
                                throws Exception

Specified by:

determineSnapshotOffset in class RelationalSnapshotChangeEventSource<MySqlPartition,MySqlOffsetContext>

Throws:

Exception

addSchemaEvent

private void addSchemaEvent(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlPartition,MySqlOffsetContext> snapshotContext,
 String database,
 String ddl)

readTableStructure

protected void readTableStructure(ChangeEventSource.ChangeEventSourceContext sourceContext,
 RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlPartition,MySqlOffsetContext> snapshotContext,
 MySqlOffsetContext offsetContext)
                           throws Exception

Specified by:

readTableStructure in class RelationalSnapshotChangeEventSource<MySqlPartition,MySqlOffsetContext>

Throws:

Exception

createSchemaEventsForTables

void createSchemaEventsForTables(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlPartition,MySqlOffsetContext> snapshotContext,
 Collection<TableId> tablesToRead,
 boolean firstPhase)
                          throws SQLException

Throws:

SQLException

twoPhaseSchemaSnapshot

private boolean twoPhaseSchemaSnapshot()

getCreateTableEvent

protected SchemaChangeEvent getCreateTableEvent(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlPartition,MySqlOffsetContext> snapshotContext,
 Table table)
                                         throws SQLException

Specified by:

getCreateTableEvent in class RelationalSnapshotChangeEventSource<MySqlPartition,MySqlOffsetContext>

Throws:

SQLException

getSnapshotSelect

protected Optional<String> getSnapshotSelect(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlPartition,MySqlOffsetContext> snapshotContext,
 TableId tableId,
 List<String> columns)

Generate a valid MySQL query string for the specified table and columns

Specified by:

getSnapshotSelect in class RelationalSnapshotChangeEventSource<MySqlPartition,MySqlOffsetContext>

Parameters:

tableId - the table to generate a query for

Returns:

a valid query string

getSnapshotSelect

private String getSnapshotSelect(TableId tableId,
 List<String> columns)

getSnapshotConnectionFirstSelect

protected Optional<String> getSnapshotConnectionFirstSelect(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlPartition,MySqlOffsetContext> snapshotContext,
 TableId tableId)

Overrides:

getSnapshotConnectionFirstSelect in class RelationalSnapshotChangeEventSource<MySqlPartition,MySqlOffsetContext>

isGloballyLocked

private boolean isGloballyLocked()

isTablesLocked

private boolean isTablesLocked()

globalLock

private void globalLock()
                 throws SQLException

Throws:

SQLException

globalUnlock

private void globalUnlock()
                   throws SQLException

Throws:

SQLException

tableLock

private void tableLock(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlPartition,MySqlOffsetContext> snapshotContext)
                throws SQLException

Throws:

SQLException

tableUnlock

private void tableUnlock()
                  throws SQLException

Throws:

SQLException

quote

private String quote(String dbOrTableName)

quote

private String quote(TableId id)

rowCountForTable

protected OptionalLong rowCountForTable(TableId tableId)

Overrides:

rowCountForTable in class RelationalSnapshotChangeEventSource<MySqlPartition,MySqlOffsetContext>

readTableStatement

protected Statement readTableStatement(JdbcConnection jdbcConnection,
 OptionalLong rowCount)
                                throws SQLException

Overrides:

readTableStatement in class RelationalSnapshotChangeEventSource<MySqlPartition,MySqlOffsetContext>

Throws:

SQLException

createStatementWithLargeResultSet

private Statement createStatementWithLargeResultSet(MySqlConnection connection)
                                             throws SQLException

Create a JDBC statement that can be used for large result sets.

By default, the MySQL Connector/J driver retrieves all rows for ResultSets and stores them in memory. In most cases this is the most efficient way to operate and, due to the design of the MySQL network protocol, is easier to implement. However, when ResultSets that have a large number of rows or large values, the driver may not be able to allocate heap space in the JVM and may result in an OutOfMemoryError. See DBZ-94 for details.

This method handles such cases using the recommended technique for MySQL by creating the JDBC Statement with forward-only cursor and read-only concurrency flags, and with a minimum value fetch size hint.

Returns:

the statement; never null

Throws:

SQLException - if there is a problem creating the statement

createSchemaChangeEventsForTables

protected void createSchemaChangeEventsForTables(ChangeEventSource.ChangeEventSourceContext sourceContext,
 RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlPartition,MySqlOffsetContext> snapshotContext,
 AbstractSnapshotChangeEventSource.SnapshottingTask snapshottingTask)
                                          throws Exception

Overrides:

createSchemaChangeEventsForTables in class RelationalSnapshotChangeEventSource<MySqlPartition,MySqlOffsetContext>

Throws:

Exception

postSnapshot

protected void postSnapshot()
                     throws InterruptedException

Overrides:

postSnapshot in class RelationalSnapshotChangeEventSource<MySqlPartition,MySqlOffsetContext>

Throws:

InterruptedException

copyOffset

protected MySqlOffsetContext copyOffset(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlPartition,MySqlOffsetContext> snapshotContext)

Specified by:

copyOffset in class RelationalSnapshotChangeEventSource<MySqlPartition,MySqlOffsetContext>