Package io.debezium.relational
Class RelationalSnapshotChangeEventSource<P extends Partition,O extends OffsetContext>
java.lang.Object
io.debezium.pipeline.source.AbstractSnapshotChangeEventSource<P,O>
io.debezium.relational.RelationalSnapshotChangeEventSource<P,O>
- All Implemented Interfaces:
ChangeEventSource
,SnapshotChangeEventSource<P,
,O> AutoCloseable
public abstract class RelationalSnapshotChangeEventSource<P extends Partition,O extends OffsetContext>
extends AbstractSnapshotChangeEventSource<P,O>
Base class for
SnapshotChangeEventSource
for relational databases with or without a schema history.
A transaction is managed by this base class, sub-classes shouldn't rollback or commit this transaction. They are free to use nested transactions or savepoints, though.
- Author:
- Gunnar Morling
-
Nested Class Summary
Modifier and TypeClassDescriptionstatic class
RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P extends Partition,
O extends OffsetContext> Mutable context which is populated in the course of snapshotting.Nested classes/interfaces inherited from class io.debezium.pipeline.source.AbstractSnapshotChangeEventSource
AbstractSnapshotChangeEventSource.SnapshotContext<P extends Partition,
O extends OffsetContext> Nested classes/interfaces inherited from interface io.debezium.pipeline.source.spi.ChangeEventSource
ChangeEventSource.ChangeEventSourceContext
-
Field Summary
Modifier and TypeFieldDescriptionprotected final Clock
protected Queue<JdbcConnection>
private final RelationalDatabaseConnectorConfig
protected final EventDispatcher<P,
TableId> private final JdbcConnection
private final MainConnectionProvidingConnectionFactory<? extends JdbcConnection>
private static final org.slf4j.Logger
static final Pattern
private final RelationalDatabaseSchema
static final Pattern
private final SnapshotProgressListener<P>
Fields inherited from class io.debezium.pipeline.source.AbstractSnapshotChangeEventSource
LOG_INTERVAL, notificationService
-
Constructor Summary
ConstructorDescriptionRelationalSnapshotChangeEventSource
(RelationalDatabaseConnectorConfig connectorConfig, MainConnectionProvidingConnectionFactory<? extends JdbcConnection> jdbcConnectionFactory, RelationalDatabaseSchema schema, EventDispatcher<P, TableId> dispatcher, Clock clock, SnapshotProgressListener<P> snapshotProgressListener, NotificationService<P, O> notificationService) -
Method Summary
Modifier and TypeMethodDescriptionprotected boolean
additionalColumnFilter
(P partition, TableId tableId, String columnName) Additional filter handling for preparing column names for snapshot selectaddSignalingCollectionAndSort
(Set<TableId> capturedTables) protected void
connectionCreated
(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P, O> snapshotContext) Executes steps which have to be taken just after the database connection is created.protected void
connectionPoolConnectionCreated
(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P, O> snapshotContext, JdbcConnection connection) Executes steps which have to be taken just after a connection pool connection is created.protected abstract O
copyOffset
(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P, O> snapshotContext) private Queue<JdbcConnection>
private void
createDataEvents
(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P, O> snapshotContext, Queue<JdbcConnection> connectionPool, Map<DataCollectionId, String> snapshotSelectOverridesByTable, SnapshottingTask snapshottingTask) createDataEventsForTableCallable
(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P, O> snapshotContext, EventDispatcher.SnapshotReceiver<P> snapshotReceiver, Table table, boolean firstTable, boolean lastTable, int tableOrder, int tableCount, String selectStatement, OptionalLong rowCount, Queue<JdbcConnection> connectionPool, Queue<O> offsets) protected void
createSchemaChangeEventsForTables
(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P, O> snapshotContext, SnapshottingTask snapshottingTask) private void
determineCapturedTables
(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P, O> ctx, Set<Pattern> dataCollectionsToBeSnapshotted, SnapshottingTask snapshottingTask) protected abstract void
determineSnapshotOffset
(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P, O> snapshotContext, O previousOffset) Determines the current offset (MySQL binlog position, Oracle SCN etc.), storing it into the passed context object.determineSnapshotSelect
(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P, O> snapshotContext, TableId tableId, Map<DataCollectionId, String> snapshotSelectOverridesByTable) Returns a valid query string for the specified table, either given by the user via snapshot select overrides or defaulting to a statement provided by the DB-specific change event source.private void
doCreateDataEventsForTable
(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P, O> snapshotContext, O offset, EventDispatcher.SnapshotReceiver<P> snapshotReceiver, Table table, boolean firstTable, boolean lastTable, int tableOrder, int tableCount, String selectStatement, OptionalLong rowCount, JdbcConnection jdbcConnection) doExecute
(ChangeEventSource.ChangeEventSourceContext context, O previousOffset, AbstractSnapshotChangeEventSource.SnapshotContext<P, O> snapshotContext, SnapshottingTask snapshottingTask) Executes this source.protected String
enhanceOverriddenSelect
(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P, O> snapshotContext, String overriddenSelect, TableId tableId) This method is overridden for Oracle to implement "as of SCN" predicategetAllTableIds
(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P, O> snapshotContext) Returns all candidate tables; the current filter configuration will be applied to the result set, resulting in the effective set of captured tables.protected ChangeRecordEmitter<P>
Returns aChangeRecordEmitter
producing the change records for the given table row.protected Clock
getClock()
protected abstract SchemaChangeEvent
getCreateTableEvent
(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P, O> snapshotContext, Table table) Creates aSchemaChangeEvent
representing the creation of the given table.getPreparedColumnNames
(P partition, Table table) Prepares a list of columns to be used in the snapshot select.getSignalDataCollectionPattern
(String signalingDataCollection) getSnapshotConnectionFirstSelect
(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P, O> snapshotContext, TableId tableId) getSnapshotSelect
(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P, O> snapshotContext, TableId tableId, List<String> columns) Returns the SELECT statement to be used for scanning the given table or empty value if the table will be streamed from but not snapshottedprotected String
getSnapshotSelectOverridesByTable
(TableId tableId, Map<DataCollectionId, String> snapshotSelectOverrides) protected Instant
getSnapshotSourceTimestamp
(JdbcConnection jdbcConnection, O offset, TableId tableId) For the given table gets source.ts_ms value from the database for snapshot data! For Postgresql its globally static for all tables since postgresql snapshot process setting auto commit off.private Threads.Timer
protected Collection<TableId>
getTablesForSchemaChange
(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P, O> snapshotContext) protected void
lastSnapshotRecord
(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P, O> snapshotContext) protected abstract void
lockTablesForSchemaSnapshot
(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P, O> snapshotContext) Locks all tables to be captured, so that no concurrent schema changes can be applied to them.protected void
protected void
protected Statement
readTableStatement
(JdbcConnection jdbcConnection, OptionalLong tableSize) Allow per-connector query creation to override for best database performance depending on the table size.protected abstract void
readTableStructure
(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P, O> snapshotContext, O offsetContext, SnapshottingTask snapshottingTask) Reads the structure of all the captured tables, writing it toRelationalSnapshotChangeEventSource.RelationalSnapshotContext.tables
.protected void
releaseDataSnapshotLocks
(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P, O> snapshotContext) Releases all locks established in order to create a consistent data snapshot.protected abstract void
releaseSchemaSnapshotLocks
(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P, O> snapshotContext) Releases all locks established in order to create a consistent schema snapshot.protected ResultSet
resultSetForDataEvents
(String selectStatement, Statement statement) private void
rollbackTransaction
(Connection connection) protected OptionalLong
rowCountForTable
(TableId tableId) If connector is able to provide statistics-based number of records per table.private void
setSnapshotMarker
(OffsetContext offset, boolean firstTable, boolean lastTable, boolean firstRecordInTable, boolean lastRecordInTable) toTableIds
(Set<TableId> tableIds, Pattern pattern) protected void
tryStartingSnapshot
(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P, O> snapshotContext) Methods inherited from class io.debezium.pipeline.source.AbstractSnapshotChangeEventSource
aborted, close, completed, delaySnapshotIfNeeded, determineDataCollectionsToBeSnapshotted, execute, getDataCollectionPattern, getOffsets, getSnapshottingTask, prepare
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
Methods inherited from interface io.debezium.pipeline.source.spi.SnapshotChangeEventSource
getBlockingSnapshottingTask
-
Field Details
-
LOGGER
private static final org.slf4j.Logger LOGGER -
SELECT_ALL_PATTERN
-
MATCH_ALL_PATTERN
-
connectorConfig
-
jdbcConnection
-
jdbcConnectionFactory
private final MainConnectionProvidingConnectionFactory<? extends JdbcConnection> jdbcConnectionFactory -
schema
-
dispatcher
-
clock
-
snapshotProgressListener
-
connectionPool
-
-
Constructor Details
-
RelationalSnapshotChangeEventSource
public RelationalSnapshotChangeEventSource(RelationalDatabaseConnectorConfig connectorConfig, MainConnectionProvidingConnectionFactory<? extends JdbcConnection> jdbcConnectionFactory, RelationalDatabaseSchema schema, EventDispatcher<P, TableId> dispatcher, Clock clock, SnapshotProgressListener<P> snapshotProgressListener, NotificationService<P, O> notificationService)
-
-
Method Details
-
doExecute
public SnapshotResult<O> doExecute(ChangeEventSource.ChangeEventSourceContext context, O previousOffset, AbstractSnapshotChangeEventSource.SnapshotContext<P, O> snapshotContext, SnapshottingTask snapshottingTask) throws ExceptionDescription copied from class:AbstractSnapshotChangeEventSource
Executes this source. Implementations should regularly check via the given context if they should stop. If that's the case, they should abort their processing and perform any clean-up needed, such as rolling back pending transactions, releasing locks, etc.- Specified by:
doExecute
in classAbstractSnapshotChangeEventSource<P extends Partition,
O extends OffsetContext> - Parameters:
context
- contextual information for this source's executionpreviousOffset
- previous offset restored from KafkasnapshotContext
- mutable context information populated throughout the snapshot processsnapshottingTask
- immutable information about what tasks should be performed during snapshot- Returns:
- an indicator to the position at which the snapshot was taken
- Throws:
Exception
-
createConnectionPool
private Queue<JdbcConnection> createConnectionPool(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P, O> ctx) throws SQLException- Throws:
SQLException
-
createSnapshotConnection
- Throws:
SQLException
-
connectionCreated
protected void connectionCreated(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P, O> snapshotContext) throws ExceptionExecutes steps which have to be taken just after the database connection is created.- Throws:
Exception
-
connectionPoolConnectionCreated
protected void connectionPoolConnectionCreated(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P, O> snapshotContext, JdbcConnection connection) throws SQLExceptionExecutes steps which have to be taken just after a connection pool connection is created.- Throws:
SQLException
-
getSignalDataCollectionPattern
-
toTableIds
-
addSignalingCollectionAndSort
-
determineCapturedTables
private void determineCapturedTables(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P, O> ctx, Set<Pattern> dataCollectionsToBeSnapshotted, SnapshottingTask snapshottingTask) throws Exception- Throws:
Exception
-
getAllTableIds
protected abstract Set<TableId> getAllTableIds(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P, O> snapshotContext) throws ExceptionReturns all candidate tables; the current filter configuration will be applied to the result set, resulting in the effective set of captured tables.- Throws:
Exception
-
lockTablesForSchemaSnapshot
protected abstract void lockTablesForSchemaSnapshot(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P, O> snapshotContext) throws ExceptionLocks all tables to be captured, so that no concurrent schema changes can be applied to them.- Throws:
Exception
-
determineSnapshotOffset
protected abstract void determineSnapshotOffset(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P, O> snapshotContext, O previousOffset) throws ExceptionDetermines the current offset (MySQL binlog position, Oracle SCN etc.), storing it into the passed context object. Subsequently, the DB's schema (and data) will be be read at this position. Once the snapshot is completed, aStreamingChangeEventSource
will be set up with this initial position to continue with stream reading from there.- Throws:
Exception
-
readTableStructure
protected abstract void readTableStructure(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P, O> snapshotContext, O offsetContext, SnapshottingTask snapshottingTask) throws ExceptionReads the structure of all the captured tables, writing it toRelationalSnapshotChangeEventSource.RelationalSnapshotContext.tables
.- Throws:
Exception
-
releaseSchemaSnapshotLocks
protected abstract void releaseSchemaSnapshotLocks(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P, O> snapshotContext) throws ExceptionReleases all locks established in order to create a consistent schema snapshot.- Throws:
Exception
-
releaseDataSnapshotLocks
protected void releaseDataSnapshotLocks(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P, O> snapshotContext) throws ExceptionReleases all locks established in order to create a consistent data snapshot.- Throws:
Exception
-
createSchemaChangeEventsForTables
protected void createSchemaChangeEventsForTables(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P, O> snapshotContext, SnapshottingTask snapshottingTask) throws Exception- Throws:
Exception
-
getTablesForSchemaChange
protected Collection<TableId> getTablesForSchemaChange(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P, O> snapshotContext) -
getCreateTableEvent
protected abstract SchemaChangeEvent getCreateTableEvent(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P, O> snapshotContext, Table table) throws ExceptionCreates aSchemaChangeEvent
representing the creation of the given table.- Throws:
Exception
-
createDataEvents
private void createDataEvents(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P, O> snapshotContext, Queue<JdbcConnection> connectionPool, Map<DataCollectionId, throws ExceptionString> snapshotSelectOverridesByTable, SnapshottingTask snapshottingTask) - Throws:
Exception
-
copyOffset
protected abstract O copyOffset(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P, O> snapshotContext) -
tryStartingSnapshot
protected void tryStartingSnapshot(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P, O> snapshotContext) -
getSnapshotSourceTimestamp
protected Instant getSnapshotSourceTimestamp(JdbcConnection jdbcConnection, O offset, TableId tableId) For the given table gets source.ts_ms value from the database for snapshot data! For Postgresql its globally static for all tables since postgresql snapshot process setting auto commit off. For Mysql its static per table and might be ~second behind of the select statements start ts. -
createDataEventsForTableCallable
private Callable<Void> createDataEventsForTableCallable(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P, O> snapshotContext, EventDispatcher.SnapshotReceiver<P> snapshotReceiver, Table table, boolean firstTable, boolean lastTable, int tableOrder, int tableCount, String selectStatement, OptionalLong rowCount, Queue<JdbcConnection> connectionPool, Queue<O> offsets) -
doCreateDataEventsForTable
private void doCreateDataEventsForTable(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P, O> snapshotContext, O offset, EventDispatcher.SnapshotReceiver<P> snapshotReceiver, Table table, boolean firstTable, boolean lastTable, int tableOrder, int tableCount, String selectStatement, OptionalLong rowCount, JdbcConnection jdbcConnection) throws InterruptedException- Throws:
InterruptedException
-
resultSetForDataEvents
protected ResultSet resultSetForDataEvents(String selectStatement, Statement statement) throws SQLException - Throws:
SQLException
-
setSnapshotMarker
private void setSnapshotMarker(OffsetContext offset, boolean firstTable, boolean lastTable, boolean firstRecordInTable, boolean lastRecordInTable) -
lastSnapshotRecord
protected void lastSnapshotRecord(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P, O> snapshotContext) -
rowCountForTable
If connector is able to provide statistics-based number of records per table. -
getTableScanLogTimer
-
getChangeRecordEmitter
protected ChangeRecordEmitter<P> getChangeRecordEmitter(P partition, O offset, TableId tableId, Object[] row, Instant timestamp) Returns aChangeRecordEmitter
producing the change records for the given table row. -
determineSnapshotSelect
private Optional<String> determineSnapshotSelect(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P, O> snapshotContext, TableId tableId, Map<DataCollectionId, String> snapshotSelectOverridesByTable) Returns a valid query string for the specified table, either given by the user via snapshot select overrides or defaulting to a statement provided by the DB-specific change event source.- Parameters:
tableId
- the table to generate a query forsnapshotSelectOverridesByTable
- the select overrides by table- Returns:
- a valid query string or empty if table will not be snapshotted
-
getSnapshotSelectOverridesByTable
protected String getSnapshotSelectOverridesByTable(TableId tableId, Map<DataCollectionId, String> snapshotSelectOverrides) -
getPreparedColumnNames
Prepares a list of columns to be used in the snapshot select. The selected columns are based on the column include/exclude filters and if all columns are excluded, the list will contain all the primary key columns.- Returns:
- list of snapshot select columns
-
additionalColumnFilter
Additional filter handling for preparing column names for snapshot select -
enhanceOverriddenSelect
protected String enhanceOverriddenSelect(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P, O> snapshotContext, String overriddenSelect, TableId tableId) This method is overridden for Oracle to implement "as of SCN" predicate- Parameters:
snapshotContext
- snapshot context, used for getting offset SCNoverriddenSelect
- conditional snapshot select- Returns:
- enhanced select statement. By default it just returns original select statements.
-
getSnapshotSelect
protected abstract Optional<String> getSnapshotSelect(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P, O> snapshotContext, TableId tableId, List<String> columns) Returns the SELECT statement to be used for scanning the given table or empty value if the table will be streamed from but not snapshotted -
getSnapshotConnectionFirstSelect
-
readTableStatement
protected Statement readTableStatement(JdbcConnection jdbcConnection, OptionalLong tableSize) throws SQLException Allow per-connector query creation to override for best database performance depending on the table size.- Throws:
SQLException
-
rollbackTransaction
-
getClock
-
postSnapshot
- Throws:
InterruptedException
-
preSnapshot
- Throws:
InterruptedException
-