Package io.debezium.connector.mysql

Class MySqlSnapshotChangeEventSource

java.lang.Object

io.debezium.pipeline.source.AbstractSnapshotChangeEventSource<P,O>

io.debezium.relational.RelationalSnapshotChangeEventSource<MySqlPartition,MySqlOffsetContext>

io.debezium.connector.mysql.MySqlSnapshotChangeEventSource

All Implemented Interfaces:

ChangeEventSource, SnapshotChangeEventSource<MySqlPartition,MySqlOffsetContext>, AutoCloseable

public class MySqlSnapshotChangeEventSource
extends RelationalSnapshotChangeEventSource<MySqlPartition,MySqlOffsetContext>

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
private static class	MySqlSnapshotChangeEventSource.MySqlSnapshotContext	Mutable context which is populated in the course of snapshotting.

Nested classes/interfaces inherited from class io.debezium.relational.RelationalSnapshotChangeEventSource

RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P extends Partition,O extends OffsetContext>

Nested classes/interfaces inherited from class io.debezium.pipeline.source.AbstractSnapshotChangeEventSource

AbstractSnapshotChangeEventSource.SnapshotContext<P extends Partition,O extends OffsetContext>

Nested classes/interfaces inherited from interface io.debezium.pipeline.source.spi.ChangeEventSource

ChangeEventSource.ChangeEventSourceContext

Field Summary

Fields

Modifier and Type	Field	Description
private final AbstractConnectorConnection	connection
private final MySqlConnectorConfig	connectorConfig
private final MySqlDatabaseSchema	databaseSchema
private Set<TableId>	delayedSchemaSnapshotTables
private final RelationalTableFilters	filters
private long	globalLockAcquiredAt
private final BlockingConsumer<Function<org.apache.kafka.connect.source.SourceRecord,org.apache.kafka.connect.source.SourceRecord>>	lastEventProcessor
private static final org.slf4j.Logger	LOGGER
private final MySqlSnapshotChangeEventSourceMetrics	metrics
private final Runnable	preSnapshotAction
private final List<SchemaChangeEvent>	schemaEvents
private long	tableLockAcquiredAt

Fields inherited from class io.debezium.relational.RelationalSnapshotChangeEventSource

clock, connectionPool, dispatcher, MATCH_ALL_PATTERN, SELECT_ALL_PATTERN

Fields inherited from class io.debezium.pipeline.source.AbstractSnapshotChangeEventSource

LOG_INTERVAL, notificationService

Constructor Summary

Constructors

Constructor

Description

MySqlSnapshotChangeEventSource(MySqlConnectorConfig connectorConfig, MainConnectionProvidingConnectionFactory<AbstractConnectorConnection> connectionFactory, MySqlDatabaseSchema schema, EventDispatcher<MySqlPartition,TableId> dispatcher, Clock clock, MySqlSnapshotChangeEventSourceMetrics metrics, BlockingConsumer<Function<org.apache.kafka.connect.source.SourceRecord,org.apache.kafka.connect.source.SourceRecord>> lastEventProcessor, Runnable preSnapshotAction, NotificationService<MySqlPartition,MySqlOffsetContext> notificationService)

Method Summary

Modifier and Type	Method	Description
private void	addSchemaEvent(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlPartition,MySqlOffsetContext> snapshotContext, String database, String ddl)
protected MySqlOffsetContext	copyOffset(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlPartition,MySqlOffsetContext> snapshotContext)
private Callable<Map<TableId,String>>	createDdlForTableCallable(TableId tableId, Queue<JdbcConnection> connectionPool)
protected void	createSchemaChangeEventsForTables(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlPartition,MySqlOffsetContext> snapshotContext, SnapshottingTask snapshottingTask)
private void	createSchemaEventsForTables(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlPartition,MySqlOffsetContext> snapshotContext, Collection<TableId> tablesToRead, boolean firstPhase)
private void	createSchemaEventsForTables(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlPartition,MySqlOffsetContext> snapshotContext, Collection<TableId> tablesToRead, boolean firstPhase, ExecutorService executorService)
private Statement	createStatementWithLargeResultSet(AbstractConnectorConnection connection)	Create a JDBC statement that can be used for large result sets.
protected void	determineSnapshotOffset(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlPartition,MySqlOffsetContext> ctx, MySqlOffsetContext previousOffset)
protected Set<TableId>	getAllTableIds(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlPartition,MySqlOffsetContext> ctx)
protected SchemaChangeEvent	getCreateTableEvent(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlPartition,MySqlOffsetContext> snapshotContext, Table table)
protected Optional<String>	getSnapshotConnectionFirstSelect(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlPartition,MySqlOffsetContext> snapshotContext, TableId tableId)
protected Optional<String>	getSnapshotSelect(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlPartition,MySqlOffsetContext> snapshotContext, TableId tableId, List<String> columns)	Generate a valid MySQL query string for the specified table and columns
private String	getSnapshotSelect(TableId tableId, List<String> columns)
SnapshottingTask	getSnapshottingTask(MySqlPartition partition, MySqlOffsetContext previousOffset)
private void	globalLock()
private void	globalUnlock()
private boolean	isGloballyLocked()
private boolean	isTablesLocked()
protected void	lockTablesForSchemaSnapshot(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlPartition,MySqlOffsetContext> snapshotContext)
protected void	postSnapshot()
protected AbstractSnapshotChangeEventSource.SnapshotContext<MySqlPartition,MySqlOffsetContext>	prepare(MySqlPartition partition)
protected void	preSnapshot()
private String	quote(TableId id)
private String	quote(String dbOrTableName)
protected Statement	readTableStatement(JdbcConnection jdbcConnection, OptionalLong rowCount)
protected void	readTableStructure(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlPartition,MySqlOffsetContext> snapshotContext, MySqlOffsetContext offsetContext, SnapshottingTask snapshottingTask)
protected void	releaseDataSnapshotLocks(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlPartition,MySqlOffsetContext> snapshotContext)
protected void	releaseSchemaSnapshotLocks(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlPartition,MySqlOffsetContext> snapshotContext)
protected OptionalLong	rowCountForTable(TableId tableId)
private void	tableLock(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlPartition,MySqlOffsetContext> snapshotContext)
private void	tableUnlock()
private boolean	twoPhaseSchemaSnapshot()

Methods inherited from class io.debezium.relational.RelationalSnapshotChangeEventSource

additionalColumnFilter, connectionCreated, connectionPoolConnectionCreated, createSnapshotConnection, doExecute, enhanceOverriddenSelect, getChangeRecordEmitter, getClock, getPreparedColumnNames, getSignalDataCollectionPattern, getSnapshotSelectOverridesByTable, getSnapshotSourceTimestamp, getTablesForSchemaChange, lastSnapshotRecord, resultSetForDataEvents, tryStartingSnapshot

Methods inherited from class io.debezium.pipeline.source.AbstractSnapshotChangeEventSource

aborted, close, completed, delaySnapshotIfNeeded, determineDataCollectionsToBeSnapshotted, execute, getDataCollectionPattern, getOffsets

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface io.debezium.pipeline.source.spi.SnapshotChangeEventSource

getBlockingSnapshottingTask

Field Details

LOGGER

private static final org.slf4j.Logger LOGGER

connectorConfig

private final MySqlConnectorConfig connectorConfig

connection

private final AbstractConnectorConnection connection

globalLockAcquiredAt

private long globalLockAcquiredAt

tableLockAcquiredAt

private long tableLockAcquiredAt

filters

private final RelationalTableFilters filters

metrics

private final MySqlSnapshotChangeEventSourceMetrics metrics

databaseSchema

private final MySqlDatabaseSchema databaseSchema

schemaEvents

private final List<SchemaChangeEvent> schemaEvents

delayedSchemaSnapshotTables

private Set<TableId> delayedSchemaSnapshotTables

lastEventProcessor

private final BlockingConsumer<Function<org.apache.kafka.connect.source.SourceRecord,org.apache.kafka.connect.source.SourceRecord>> lastEventProcessor

preSnapshotAction

private final Runnable preSnapshotAction

Constructor Details

MySqlSnapshotChangeEventSource

public MySqlSnapshotChangeEventSource(MySqlConnectorConfig connectorConfig,
 MainConnectionProvidingConnectionFactory<AbstractConnectorConnection> connectionFactory,
 MySqlDatabaseSchema schema,
 EventDispatcher<MySqlPartition,TableId> dispatcher,
 Clock clock,
 MySqlSnapshotChangeEventSourceMetrics metrics,
 BlockingConsumer<Function<org.apache.kafka.connect.source.SourceRecord,org.apache.kafka.connect.source.SourceRecord>> lastEventProcessor,
 Runnable preSnapshotAction,
 NotificationService<MySqlPartition,MySqlOffsetContext> notificationService)

Method Details

getSnapshottingTask

public SnapshottingTask getSnapshottingTask(MySqlPartition partition,
 MySqlOffsetContext previousOffset)

Specified by:

getSnapshottingTask in interface SnapshotChangeEventSource<MySqlPartition,MySqlOffsetContext>

Specified by:

getSnapshottingTask in class AbstractSnapshotChangeEventSource<MySqlPartition,MySqlOffsetContext>

prepare

protected AbstractSnapshotChangeEventSource.SnapshotContext<MySqlPartition,MySqlOffsetContext> prepare(MySqlPartition partition)
                                                                                                throws Exception

Specified by:

prepare in class AbstractSnapshotChangeEventSource<MySqlPartition,MySqlOffsetContext>

Throws:

Exception

getAllTableIds

protected Set<TableId> getAllTableIds(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlPartition,MySqlOffsetContext> ctx)
                               throws Exception

Specified by:

getAllTableIds in class RelationalSnapshotChangeEventSource<MySqlPartition,MySqlOffsetContext>

Throws:

Exception

lockTablesForSchemaSnapshot

protected void lockTablesForSchemaSnapshot(ChangeEventSource.ChangeEventSourceContext sourceContext,
 RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlPartition,MySqlOffsetContext> snapshotContext)
                                    throws SQLException

Specified by:

lockTablesForSchemaSnapshot in class RelationalSnapshotChangeEventSource<MySqlPartition,MySqlOffsetContext>

Throws:

SQLException

releaseSchemaSnapshotLocks

protected void releaseSchemaSnapshotLocks(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlPartition,MySqlOffsetContext> snapshotContext)
                                   throws SQLException

Specified by:

releaseSchemaSnapshotLocks in class RelationalSnapshotChangeEventSource<MySqlPartition,MySqlOffsetContext>

Throws:

SQLException

releaseDataSnapshotLocks

protected void releaseDataSnapshotLocks(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlPartition,MySqlOffsetContext> snapshotContext)
                                 throws Exception

Overrides:

releaseDataSnapshotLocks in class RelationalSnapshotChangeEventSource<MySqlPartition,MySqlOffsetContext>

Throws:

Exception

determineSnapshotOffset

protected void determineSnapshotOffset(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlPartition,MySqlOffsetContext> ctx,
 MySqlOffsetContext previousOffset)
                                throws Exception

Specified by:

determineSnapshotOffset in class RelationalSnapshotChangeEventSource<MySqlPartition,MySqlOffsetContext>

Throws:

Exception

addSchemaEvent

private void addSchemaEvent(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlPartition,MySqlOffsetContext> snapshotContext,
 String database,
 String ddl)

readTableStructure

protected void readTableStructure(ChangeEventSource.ChangeEventSourceContext sourceContext,
 RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlPartition,MySqlOffsetContext> snapshotContext,
 MySqlOffsetContext offsetContext,
 SnapshottingTask snapshottingTask)
                           throws Exception

Specified by:

readTableStructure in class RelationalSnapshotChangeEventSource<MySqlPartition,MySqlOffsetContext>

Throws:

Exception

createSchemaEventsForTables

private void createSchemaEventsForTables(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlPartition,MySqlOffsetContext> snapshotContext,
 Collection<TableId> tablesToRead,
 boolean firstPhase)
                                  throws Exception

Throws:

Exception

createSchemaEventsForTables

private void createSchemaEventsForTables(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlPartition,MySqlOffsetContext> snapshotContext,
 Collection<TableId> tablesToRead,
 boolean firstPhase,
 ExecutorService executorService)
                                  throws Exception

Throws:

Exception

createDdlForTableCallable

private Callable<Map<TableId,String>> createDdlForTableCallable(TableId tableId,
 Queue<JdbcConnection> connectionPool)

twoPhaseSchemaSnapshot

private boolean twoPhaseSchemaSnapshot()

getCreateTableEvent

protected SchemaChangeEvent getCreateTableEvent(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlPartition,MySqlOffsetContext> snapshotContext,
 Table table)

Specified by:

getCreateTableEvent in class RelationalSnapshotChangeEventSource<MySqlPartition,MySqlOffsetContext>

getSnapshotSelect

protected Optional<String> getSnapshotSelect(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlPartition,MySqlOffsetContext> snapshotContext,
 TableId tableId,
 List<String> columns)

Generate a valid MySQL query string for the specified table and columns

Specified by:

getSnapshotSelect in class RelationalSnapshotChangeEventSource<MySqlPartition,MySqlOffsetContext>

Parameters:

tableId - the table to generate a query for

Returns:

a valid query string

getSnapshotSelect

private String getSnapshotSelect(TableId tableId,
 List<String> columns)

getSnapshotConnectionFirstSelect

protected Optional<String> getSnapshotConnectionFirstSelect(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlPartition,MySqlOffsetContext> snapshotContext,
 TableId tableId)

Overrides:

getSnapshotConnectionFirstSelect in class RelationalSnapshotChangeEventSource<MySqlPartition,MySqlOffsetContext>

isGloballyLocked

private boolean isGloballyLocked()

isTablesLocked

private boolean isTablesLocked()

globalLock

private void globalLock()
                 throws SQLException

Throws:

SQLException

globalUnlock

private void globalUnlock()
                   throws SQLException

Throws:

SQLException

tableLock

private void tableLock(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlPartition,MySqlOffsetContext> snapshotContext)
                throws SQLException

Throws:

SQLException

tableUnlock

private void tableUnlock()
                  throws SQLException

Throws:

SQLException

quote

private String quote(String dbOrTableName)

quote

private String quote(TableId id)

rowCountForTable

protected OptionalLong rowCountForTable(TableId tableId)

Overrides:

rowCountForTable in class RelationalSnapshotChangeEventSource<MySqlPartition,MySqlOffsetContext>

readTableStatement

protected Statement readTableStatement(JdbcConnection jdbcConnection,
 OptionalLong rowCount)
                                throws SQLException

Overrides:

readTableStatement in class RelationalSnapshotChangeEventSource<MySqlPartition,MySqlOffsetContext>

Throws:

SQLException

createStatementWithLargeResultSet

private Statement createStatementWithLargeResultSet(AbstractConnectorConnection connection)
                                             throws SQLException

Create a JDBC statement that can be used for large result sets.

By default, the MySQL Connector/J driver retrieves all rows for ResultSets and stores them in memory. In most cases this is the most efficient way to operate and, due to the design of the MySQL network protocol, is easier to implement. However, when ResultSets that have a large number of rows or large values, the driver may not be able to allocate heap space in the JVM and may result in an OutOfMemoryError. See DBZ-94 for details.

This method handles such cases using the recommended technique for MySQL by creating the JDBC Statement with forward-only cursor and read-only concurrency flags, and with a minimum value fetch size hint.

Returns:

the statement; never null

Throws:

SQLException - if there is a problem creating the statement

createSchemaChangeEventsForTables

protected void createSchemaChangeEventsForTables(ChangeEventSource.ChangeEventSourceContext sourceContext,
 RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlPartition,MySqlOffsetContext> snapshotContext,
 SnapshottingTask snapshottingTask)
                                          throws Exception

Overrides:

createSchemaChangeEventsForTables in class RelationalSnapshotChangeEventSource<MySqlPartition,MySqlOffsetContext>

Throws:

Exception

postSnapshot

protected void postSnapshot()
                     throws InterruptedException

Overrides:

postSnapshot in class RelationalSnapshotChangeEventSource<MySqlPartition,MySqlOffsetContext>

Throws:

InterruptedException

preSnapshot

protected void preSnapshot()
                    throws InterruptedException

Overrides:

preSnapshot in class RelationalSnapshotChangeEventSource<MySqlPartition,MySqlOffsetContext>

Throws:

InterruptedException

copyOffset

protected MySqlOffsetContext copyOffset(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlPartition,MySqlOffsetContext> snapshotContext)

Specified by:

copyOffset in class RelationalSnapshotChangeEventSource<MySqlPartition,MySqlOffsetContext>