Package io.debezium.relational
Class RelationalSnapshotChangeEventSource<O extends OffsetContext>
- java.lang.Object
-
- io.debezium.pipeline.source.AbstractSnapshotChangeEventSource<O>
-
- io.debezium.relational.RelationalSnapshotChangeEventSource<O>
-
- All Implemented Interfaces:
ChangeEventSource
,SnapshotChangeEventSource<O>
public abstract class RelationalSnapshotChangeEventSource<O extends OffsetContext> extends AbstractSnapshotChangeEventSource<O>
Base class forSnapshotChangeEventSource
for relational databases with or without a schema history.A transaction is managed by this base class, sub-classes shouldn't rollback or commit this transaction. They are free to use nested transactions or savepoints, though.
- Author:
- Gunnar Morling
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description static class
RelationalSnapshotChangeEventSource.RelationalSnapshotContext<O extends OffsetContext>
Mutable context which is populated in the course of snapshotting.-
Nested classes/interfaces inherited from class io.debezium.pipeline.source.AbstractSnapshotChangeEventSource
AbstractSnapshotChangeEventSource.SnapshotContext<O extends OffsetContext>, AbstractSnapshotChangeEventSource.SnapshottingTask
-
Nested classes/interfaces inherited from interface io.debezium.pipeline.source.spi.ChangeEventSource
ChangeEventSource.ChangeEventSourceContext
-
-
Field Summary
Fields Modifier and Type Field Description protected Clock
clock
private RelationalDatabaseConnectorConfig
connectorConfig
protected EventDispatcher<TableId>
dispatcher
private JdbcConnection
jdbcConnection
static Duration
LOG_INTERVAL
Interval for showing a log statement with the progress while scanning a single table.private static org.slf4j.Logger
LOGGER
private HistorizedRelationalDatabaseSchema
schema
private SnapshotProgressListener
snapshotProgressListener
-
Constructor Summary
Constructors Constructor Description RelationalSnapshotChangeEventSource(RelationalDatabaseConnectorConfig connectorConfig, JdbcConnection jdbcConnection, EventDispatcher<TableId> dispatcher, Clock clock, SnapshotProgressListener snapshotProgressListener)
RelationalSnapshotChangeEventSource(RelationalDatabaseConnectorConfig connectorConfig, JdbcConnection jdbcConnection, HistorizedRelationalDatabaseSchema schema, EventDispatcher<TableId> dispatcher, Clock clock, SnapshotProgressListener snapshotProgressListener)
-
Method Summary
All Methods Instance Methods Abstract Methods Concrete Methods Modifier and Type Method Description protected void
connectionCreated(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<O> snapshotContext)
Executes steps which have to be taken just after the database connection is created.private void
createDataEvents(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.RelationalSnapshotContext<O> snapshotContext)
private void
createDataEventsForTable(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.RelationalSnapshotContext<O> snapshotContext, EventDispatcher.SnapshotReceiver snapshotReceiver, Table table, int tableOrder, int tableCount)
Dispatches the data change events for the records of a single table.protected void
createSchemaChangeEventsForTables(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.RelationalSnapshotContext<O> snapshotContext, AbstractSnapshotChangeEventSource.SnapshottingTask snapshottingTask)
Connection
createSnapshotConnection()
private void
determineCapturedTables(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<O> ctx)
protected abstract void
determineSnapshotOffset(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<O> snapshotContext, O previousOffset)
Determines the current offset (MySQL binlog position, Oracle SCN etc.), storing it into the passed context object.private Optional<String>
determineSnapshotSelect(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<O> snapshotContext, TableId tableId)
Returns a valid query string for the specified table, either given by the user via snapshot select overrides or defaulting to a statement provided by the DB-specific change event source.SnapshotResult<O>
doExecute(ChangeEventSource.ChangeEventSourceContext context, O previousOffset, AbstractSnapshotChangeEventSource.SnapshotContext<O> snapshotContext, AbstractSnapshotChangeEventSource.SnapshottingTask snapshottingTask)
Executes this source.protected String
enhanceOverriddenSelect(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<O> snapshotContext, String overriddenSelect, TableId tableId)
This method is overridden for Oracle to implement "as of SCN" predicateprotected abstract Set<TableId>
getAllTableIds(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<O> snapshotContext)
Returns all candidate tables; the current filter configuration will be applied to the result set, resulting in the effective set of captured tables.protected ChangeRecordEmitter
getChangeRecordEmitter(AbstractSnapshotChangeEventSource.SnapshotContext<O> snapshotContext, TableId tableId, Object[] row)
Returns aChangeRecordEmitter
producing the change records for the given table row.protected Clock
getClock()
protected Object
getColumnValue(ResultSet rs, int columnIndex, Column column, Table table)
protected abstract SchemaChangeEvent
getCreateTableEvent(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<O> snapshotContext, Table table)
Creates aSchemaChangeEvent
representing the creation of the given table.protected abstract Optional<String>
getSnapshotSelect(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<O> snapshotContext, TableId tableId)
Returns the SELECT statement to be used for scanning the given table or empty value if the table will be streamed from but not snapshottedprivate Threads.Timer
getTableScanLogTimer()
protected void
lastSnapshotRecord(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<O> snapshotContext)
protected abstract void
lockTablesForSchemaSnapshot(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.RelationalSnapshotContext<O> snapshotContext)
Locks all tables to be captured, so that no concurrent schema changes can be applied to them.protected void
postSnapshot()
protected Statement
readTableStatement(OptionalLong tableSize)
Allow per-connector query creation to override for best database performance depending on the table size.protected abstract void
readTableStructure(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.RelationalSnapshotContext<O> snapshotContext, O offsetContext)
Reads the structure of all the captured tables, writing it toRelationalSnapshotChangeEventSource.RelationalSnapshotContext.tables
.protected void
releaseDataSnapshotLocks(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<O> snapshotContext)
Releases all locks established in order to create a consistent data snapshot.protected abstract void
releaseSchemaSnapshotLocks(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<O> snapshotContext)
Releases all locks established in order to create a consistent schema snapshot.private void
rollbackTransaction(Connection connection)
protected OptionalLong
rowCountForTable(TableId tableId)
If connector is able to provide statistics-based number of records per table.protected RelationalDatabaseSchema
schema()
private Set<TableId>
sort(Set<TableId> capturedTables)
private Stream<TableId>
toTableIds(Set<TableId> tableIds, Pattern pattern)
protected void
tryStartingSnapshot(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<O> snapshotContext)
-
Methods inherited from class io.debezium.pipeline.source.AbstractSnapshotChangeEventSource
complete, delaySnapshotIfNeeded, determineDataCollectionsToBeSnapshotted, execute, getSnapshottingTask, prepare
-
-
-
-
Field Detail
-
LOGGER
private static final org.slf4j.Logger LOGGER
-
LOG_INTERVAL
public static final Duration LOG_INTERVAL
Interval for showing a log statement with the progress while scanning a single table.
-
connectorConfig
private final RelationalDatabaseConnectorConfig connectorConfig
-
jdbcConnection
private final JdbcConnection jdbcConnection
-
schema
private final HistorizedRelationalDatabaseSchema schema
-
dispatcher
protected final EventDispatcher<TableId> dispatcher
-
clock
protected final Clock clock
-
snapshotProgressListener
private final SnapshotProgressListener snapshotProgressListener
-
-
Constructor Detail
-
RelationalSnapshotChangeEventSource
public RelationalSnapshotChangeEventSource(RelationalDatabaseConnectorConfig connectorConfig, JdbcConnection jdbcConnection, HistorizedRelationalDatabaseSchema schema, EventDispatcher<TableId> dispatcher, Clock clock, SnapshotProgressListener snapshotProgressListener)
-
RelationalSnapshotChangeEventSource
public RelationalSnapshotChangeEventSource(RelationalDatabaseConnectorConfig connectorConfig, JdbcConnection jdbcConnection, EventDispatcher<TableId> dispatcher, Clock clock, SnapshotProgressListener snapshotProgressListener)
-
-
Method Detail
-
doExecute
public SnapshotResult<O> doExecute(ChangeEventSource.ChangeEventSourceContext context, O previousOffset, AbstractSnapshotChangeEventSource.SnapshotContext<O> snapshotContext, AbstractSnapshotChangeEventSource.SnapshottingTask snapshottingTask) throws Exception
Description copied from class:AbstractSnapshotChangeEventSource
Executes this source. Implementations should regularly check via the given context if they should stop. If that's the case, they should abort their processing and perform any clean-up needed, such as rolling back pending transactions, releasing locks, etc.- Specified by:
doExecute
in classAbstractSnapshotChangeEventSource<O extends OffsetContext>
- Parameters:
context
- contextual information for this source's executionpreviousOffset
- previous offset restored from KafkasnapshotContext
- mutable context information populated throughout the snapshot processsnapshottingTask
- immutable information about what tasks should be performed during snapshot- Returns:
- an indicator to the position at which the snapshot was taken
- Throws:
Exception
-
createSnapshotConnection
public Connection createSnapshotConnection() throws SQLException
- Throws:
SQLException
-
connectionCreated
protected void connectionCreated(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<O> snapshotContext) throws Exception
Executes steps which have to be taken just after the database connection is created.- Throws:
Exception
-
determineCapturedTables
private void determineCapturedTables(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<O> ctx) throws Exception
- Throws:
Exception
-
getAllTableIds
protected abstract Set<TableId> getAllTableIds(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<O> snapshotContext) throws Exception
Returns all candidate tables; the current filter configuration will be applied to the result set, resulting in the effective set of captured tables.- Throws:
Exception
-
lockTablesForSchemaSnapshot
protected abstract void lockTablesForSchemaSnapshot(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.RelationalSnapshotContext<O> snapshotContext) throws Exception
Locks all tables to be captured, so that no concurrent schema changes can be applied to them.- Throws:
Exception
-
determineSnapshotOffset
protected abstract void determineSnapshotOffset(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<O> snapshotContext, O previousOffset) throws Exception
Determines the current offset (MySQL binlog position, Oracle SCN etc.), storing it into the passed context object. Subsequently, the DB's schema (and data) will be be read at this position. Once the snapshot is completed, aStreamingChangeEventSource
will be set up with this initial position to continue with stream reading from there.- Throws:
Exception
-
readTableStructure
protected abstract void readTableStructure(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.RelationalSnapshotContext<O> snapshotContext, O offsetContext) throws Exception
Reads the structure of all the captured tables, writing it toRelationalSnapshotChangeEventSource.RelationalSnapshotContext.tables
.- Throws:
Exception
-
releaseSchemaSnapshotLocks
protected abstract void releaseSchemaSnapshotLocks(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<O> snapshotContext) throws Exception
Releases all locks established in order to create a consistent schema snapshot.- Throws:
Exception
-
releaseDataSnapshotLocks
protected void releaseDataSnapshotLocks(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<O> snapshotContext) throws Exception
Releases all locks established in order to create a consistent data snapshot.- Throws:
Exception
-
createSchemaChangeEventsForTables
protected void createSchemaChangeEventsForTables(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.RelationalSnapshotContext<O> snapshotContext, AbstractSnapshotChangeEventSource.SnapshottingTask snapshottingTask) throws Exception
- Throws:
Exception
-
getCreateTableEvent
protected abstract SchemaChangeEvent getCreateTableEvent(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<O> snapshotContext, Table table) throws Exception
Creates aSchemaChangeEvent
representing the creation of the given table.- Throws:
Exception
-
createDataEvents
private void createDataEvents(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.RelationalSnapshotContext<O> snapshotContext) throws Exception
- Throws:
Exception
-
tryStartingSnapshot
protected void tryStartingSnapshot(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<O> snapshotContext)
-
createDataEventsForTable
private void createDataEventsForTable(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.RelationalSnapshotContext<O> snapshotContext, EventDispatcher.SnapshotReceiver snapshotReceiver, Table table, int tableOrder, int tableCount) throws InterruptedException
Dispatches the data change events for the records of a single table.- Throws:
InterruptedException
-
lastSnapshotRecord
protected void lastSnapshotRecord(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<O> snapshotContext)
-
rowCountForTable
protected OptionalLong rowCountForTable(TableId tableId)
If connector is able to provide statistics-based number of records per table.
-
getTableScanLogTimer
private Threads.Timer getTableScanLogTimer()
-
getChangeRecordEmitter
protected ChangeRecordEmitter getChangeRecordEmitter(AbstractSnapshotChangeEventSource.SnapshotContext<O> snapshotContext, TableId tableId, Object[] row)
Returns aChangeRecordEmitter
producing the change records for the given table row.
-
determineSnapshotSelect
private Optional<String> determineSnapshotSelect(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<O> snapshotContext, TableId tableId)
Returns a valid query string for the specified table, either given by the user via snapshot select overrides or defaulting to a statement provided by the DB-specific change event source.- Parameters:
tableId
- the table to generate a query for- Returns:
- a valid query string or empty if table will not be snapshotted
-
enhanceOverriddenSelect
protected String enhanceOverriddenSelect(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<O> snapshotContext, String overriddenSelect, TableId tableId)
This method is overridden for Oracle to implement "as of SCN" predicate- Parameters:
snapshotContext
- snapshot context, used for getting offset SCNoverriddenSelect
- conditional snapshot select- Returns:
- enhanced select statement. By default it just returns original select statements.
-
getSnapshotSelect
protected abstract Optional<String> getSnapshotSelect(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<O> snapshotContext, TableId tableId)
Returns the SELECT statement to be used for scanning the given table or empty value if the table will be streamed from but not snapshotted
-
schema
protected RelationalDatabaseSchema schema()
-
getColumnValue
protected Object getColumnValue(ResultSet rs, int columnIndex, Column column, Table table) throws SQLException
- Throws:
SQLException
-
readTableStatement
protected Statement readTableStatement(OptionalLong tableSize) throws SQLException
Allow per-connector query creation to override for best database performance depending on the table size.- Throws:
SQLException
-
rollbackTransaction
private void rollbackTransaction(Connection connection)
-
getClock
protected Clock getClock()
-
postSnapshot
protected void postSnapshot() throws InterruptedException
- Throws:
InterruptedException
-
-