Package io.debezium.connector.mysql
Class MySqlSnapshotChangeEventSource
- java.lang.Object
-
- io.debezium.pipeline.source.AbstractSnapshotChangeEventSource<O>
-
- io.debezium.relational.RelationalSnapshotChangeEventSource<MySqlOffsetContext>
-
- io.debezium.connector.mysql.MySqlSnapshotChangeEventSource
-
- All Implemented Interfaces:
ChangeEventSource
,SnapshotChangeEventSource<MySqlOffsetContext>
public class MySqlSnapshotChangeEventSource extends RelationalSnapshotChangeEventSource<MySqlOffsetContext>
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description private static class
MySqlSnapshotChangeEventSource.MySqlSnapshotContext
Mutable context which is populated in the course of snapshotting.-
Nested classes/interfaces inherited from class io.debezium.relational.RelationalSnapshotChangeEventSource
RelationalSnapshotChangeEventSource.RelationalSnapshotContext<O extends OffsetContext>
-
Nested classes/interfaces inherited from class io.debezium.pipeline.source.AbstractSnapshotChangeEventSource
AbstractSnapshotChangeEventSource.SnapshotContext<O extends OffsetContext>, AbstractSnapshotChangeEventSource.SnapshottingTask
-
Nested classes/interfaces inherited from interface io.debezium.pipeline.source.spi.ChangeEventSource
ChangeEventSource.ChangeEventSourceContext
-
-
Field Summary
Fields Modifier and Type Field Description private MySqlConnection
connection
private MySqlConnectorConfig
connectorConfig
private MySqlDatabaseSchema
databaseSchema
private Set<TableId>
delayedSchemaSnapshotTables
private RelationalTableFilters
filters
private long
globalLockAcquiredAt
private BlockingConsumer<Function<org.apache.kafka.connect.source.SourceRecord,org.apache.kafka.connect.source.SourceRecord>>
lastEventProcessor
private static org.slf4j.Logger
LOGGER
private MySqlSnapshotChangeEventSourceMetrics
metrics
private List<SchemaChangeEvent>
schemaEvents
private long
tableLockAcquiredAt
-
Fields inherited from class io.debezium.relational.RelationalSnapshotChangeEventSource
clock, dispatcher, LOG_INTERVAL
-
-
Constructor Summary
Constructors Constructor Description MySqlSnapshotChangeEventSource(MySqlConnectorConfig connectorConfig, MySqlConnection connection, MySqlDatabaseSchema schema, EventDispatcher<TableId> dispatcher, Clock clock, MySqlSnapshotChangeEventSourceMetrics metrics, BlockingConsumer<Function<org.apache.kafka.connect.source.SourceRecord,org.apache.kafka.connect.source.SourceRecord>> lastEventProcessor)
-
Method Summary
-
Methods inherited from class io.debezium.relational.RelationalSnapshotChangeEventSource
createSnapshotConnection, doExecute, enhanceOverriddenSelect, getChangeRecordEmitter, getClock, getColumnValue, schema, tryStartingSnapshot
-
Methods inherited from class io.debezium.pipeline.source.AbstractSnapshotChangeEventSource
delaySnapshotIfNeeded, determineDataCollectionsToBeSnapshotted, execute
-
-
-
-
Field Detail
-
LOGGER
private static final org.slf4j.Logger LOGGER
-
connectorConfig
private final MySqlConnectorConfig connectorConfig
-
connection
private final MySqlConnection connection
-
globalLockAcquiredAt
private long globalLockAcquiredAt
-
tableLockAcquiredAt
private long tableLockAcquiredAt
-
filters
private final RelationalTableFilters filters
-
metrics
private final MySqlSnapshotChangeEventSourceMetrics metrics
-
databaseSchema
private final MySqlDatabaseSchema databaseSchema
-
schemaEvents
private final List<SchemaChangeEvent> schemaEvents
-
lastEventProcessor
private final BlockingConsumer<Function<org.apache.kafka.connect.source.SourceRecord,org.apache.kafka.connect.source.SourceRecord>> lastEventProcessor
-
-
Constructor Detail
-
MySqlSnapshotChangeEventSource
public MySqlSnapshotChangeEventSource(MySqlConnectorConfig connectorConfig, MySqlConnection connection, MySqlDatabaseSchema schema, EventDispatcher<TableId> dispatcher, Clock clock, MySqlSnapshotChangeEventSourceMetrics metrics, BlockingConsumer<Function<org.apache.kafka.connect.source.SourceRecord,org.apache.kafka.connect.source.SourceRecord>> lastEventProcessor)
-
-
Method Detail
-
getSnapshottingTask
protected AbstractSnapshotChangeEventSource.SnapshottingTask getSnapshottingTask(MySqlOffsetContext previousOffset)
- Specified by:
getSnapshottingTask
in classAbstractSnapshotChangeEventSource<MySqlOffsetContext>
-
prepare
protected AbstractSnapshotChangeEventSource.SnapshotContext<MySqlOffsetContext> prepare(ChangeEventSource.ChangeEventSourceContext context) throws Exception
- Specified by:
prepare
in classAbstractSnapshotChangeEventSource<MySqlOffsetContext>
- Throws:
Exception
-
connectionCreated
protected void connectionCreated(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlOffsetContext> snapshotContext) throws Exception
- Overrides:
connectionCreated
in classRelationalSnapshotChangeEventSource<MySqlOffsetContext>
- Throws:
Exception
-
getAllTableIds
protected Set<TableId> getAllTableIds(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlOffsetContext> ctx) throws Exception
- Specified by:
getAllTableIds
in classRelationalSnapshotChangeEventSource<MySqlOffsetContext>
- Throws:
Exception
-
lockTablesForSchemaSnapshot
protected void lockTablesForSchemaSnapshot(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlOffsetContext> snapshotContext) throws SQLException, InterruptedException
- Specified by:
lockTablesForSchemaSnapshot
in classRelationalSnapshotChangeEventSource<MySqlOffsetContext>
- Throws:
SQLException
InterruptedException
-
releaseSchemaSnapshotLocks
protected void releaseSchemaSnapshotLocks(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlOffsetContext> snapshotContext) throws SQLException
- Specified by:
releaseSchemaSnapshotLocks
in classRelationalSnapshotChangeEventSource<MySqlOffsetContext>
- Throws:
SQLException
-
releaseDataSnapshotLocks
protected void releaseDataSnapshotLocks(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlOffsetContext> snapshotContext) throws Exception
- Overrides:
releaseDataSnapshotLocks
in classRelationalSnapshotChangeEventSource<MySqlOffsetContext>
- Throws:
Exception
-
determineSnapshotOffset
protected void determineSnapshotOffset(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlOffsetContext> ctx, MySqlOffsetContext previousOffset) throws Exception
- Specified by:
determineSnapshotOffset
in classRelationalSnapshotChangeEventSource<MySqlOffsetContext>
- Throws:
Exception
-
addSchemaEvent
private void addSchemaEvent(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlOffsetContext> snapshotContext, String database, String ddl)
-
readTableStructure
protected void readTableStructure(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlOffsetContext> snapshotContext, MySqlOffsetContext offsetContext) throws Exception
- Specified by:
readTableStructure
in classRelationalSnapshotChangeEventSource<MySqlOffsetContext>
- Throws:
Exception
-
createSchemaEventsForTables
void createSchemaEventsForTables(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlOffsetContext> snapshotContext, Collection<TableId> tablesToRead, boolean firstPhase) throws SQLException
- Throws:
SQLException
-
twoPhaseSchemaSnapshot
private boolean twoPhaseSchemaSnapshot()
-
getCreateTableEvent
protected SchemaChangeEvent getCreateTableEvent(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlOffsetContext> snapshotContext, Table table) throws SQLException
- Specified by:
getCreateTableEvent
in classRelationalSnapshotChangeEventSource<MySqlOffsetContext>
- Throws:
SQLException
-
complete
protected void complete(AbstractSnapshotChangeEventSource.SnapshotContext<MySqlOffsetContext> snapshotContext)
- Overrides:
complete
in classAbstractSnapshotChangeEventSource<MySqlOffsetContext>
-
getSnapshotSelect
protected Optional<String> getSnapshotSelect(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlOffsetContext> snapshotContext, TableId tableId)
Generate a valid sqlserver query string for the specified table- Specified by:
getSnapshotSelect
in classRelationalSnapshotChangeEventSource<MySqlOffsetContext>
- Parameters:
tableId
- the table to generate a query for- Returns:
- a valid query string
-
isGloballyLocked
private boolean isGloballyLocked()
-
isTablesLocked
private boolean isTablesLocked()
-
globalLock
private void globalLock() throws SQLException
- Throws:
SQLException
-
globalUnlock
private void globalUnlock() throws SQLException
- Throws:
SQLException
-
tableLock
private void tableLock(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlOffsetContext> snapshotContext) throws SQLException
- Throws:
SQLException
-
tableUnlock
private void tableUnlock() throws SQLException
- Throws:
SQLException
-
rowCountForTable
protected OptionalLong rowCountForTable(TableId tableId)
- Overrides:
rowCountForTable
in classRelationalSnapshotChangeEventSource<MySqlOffsetContext>
-
readTableStatement
protected Statement readTableStatement(OptionalLong rowCount) throws SQLException
- Overrides:
readTableStatement
in classRelationalSnapshotChangeEventSource<MySqlOffsetContext>
- Throws:
SQLException
-
createStatementWithLargeResultSet
private Statement createStatementWithLargeResultSet() throws SQLException
Create a JDBC statement that can be used for large result sets.By default, the MySQL Connector/J driver retrieves all rows for ResultSets and stores them in memory. In most cases this is the most efficient way to operate and, due to the design of the MySQL network protocol, is easier to implement. However, when ResultSets that have a large number of rows or large values, the driver may not be able to allocate heap space in the JVM and may result in an
OutOfMemoryError
. See DBZ-94 for details.This method handles such cases using the recommended technique for MySQL by creating the JDBC
Statement
withforward-only
cursor andread-only concurrency
flags, and with aminimum value
fetch size hint
.- Returns:
- the statement; never null
- Throws:
SQLException
- if there is a problem creating the statement
-
createSchemaChangeEventsForTables
protected void createSchemaChangeEventsForTables(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlOffsetContext> snapshotContext, AbstractSnapshotChangeEventSource.SnapshottingTask snapshottingTask) throws Exception
- Overrides:
createSchemaChangeEventsForTables
in classRelationalSnapshotChangeEventSource<MySqlOffsetContext>
- Throws:
Exception
-
lastSnapshotRecord
protected void lastSnapshotRecord(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<MySqlOffsetContext> snapshotContext)
- Overrides:
lastSnapshotRecord
in classRelationalSnapshotChangeEventSource<MySqlOffsetContext>
-
postSnapshot
protected void postSnapshot() throws InterruptedException
- Overrides:
postSnapshot
in classRelationalSnapshotChangeEventSource<MySqlOffsetContext>
- Throws:
InterruptedException
-
-