Class AbstractIncrementalSnapshotChangeEventSource<P extends Partition,T extends DataCollectionId>
java.lang.Object
io.debezium.pipeline.source.snapshot.incremental.AbstractIncrementalSnapshotChangeEventSource<P,T>
- All Implemented Interfaces:
IncrementalSnapshotChangeEventSource<P,
T>
- Direct Known Subclasses:
SignalBasedIncrementalSnapshotChangeEventSource
@NotThreadSafe
public abstract class AbstractIncrementalSnapshotChangeEventSource<P extends Partition,T extends DataCollectionId>
extends Object
implements IncrementalSnapshotChangeEventSource<P,T>
An incremental snapshot change event source that emits events from a DB log interleaved with snapshot events.
-
Field Summary
Modifier and TypeFieldDescriptionprivate final Clock
private final RelationalDatabaseConnectorConfig
protected IncrementalSnapshotContext<T>
private Table
private final RelationalDatabaseSchema
private final DataChangeEventListener<P>
protected EventDispatcher<P,
T> protected JdbcConnection
private static final org.slf4j.Logger
private final SnapshotProgressListener<P>
private long
-
Constructor Summary
ConstructorDescriptionAbstractIncrementalSnapshotChangeEventSource
(RelationalDatabaseConnectorConfig config, JdbcConnection jdbcConnection, EventDispatcher<P, T> dispatcher, DatabaseSchema<?> databaseSchema, Clock clock, SnapshotProgressListener<P> progressListener, DataChangeEventListener<P> dataChangeEventListener) -
Method Summary
Modifier and TypeMethodDescriptionvoid
addDataCollectionNamesToSnapshot
(P partition, List<String> dataCollectionIds, OffsetContext offsetContext) protected void
addKeyColumnsToCondition
(Table table, StringBuilder sql, String predicate) private void
addLowerBound
(Table table, StringBuilder sql) protected String
buildChunkQuery
(Table table) protected String
buildChunkQuery
(Table table, int limit) protected String
buildMaxPrimaryKeyQuery
(Table table) private boolean
void
closeWindow
(P partition, String id, OffsetContext offsetContext) private boolean
createDataEventsForTable
(P partition) Dispatches the data change events for the records of a single table.protected void
deduplicateWindow
(DataCollectionId dataCollectionId, Object key) protected abstract void
emitWindowClose
(P partition) Update high watermark for the incremental snapshot chunkprotected abstract void
Update low watermark for the incremental snapshot chunkprotected ChangeRecordEmitter<P>
getChangeRecordEmitter
(P partition, T dataCollectionId, OffsetContext offsetContext, Object[] row) Returns aChangeRecordEmitter
producing the change records for the given table row.private Key.KeyMapper
protected String
getSignalTableName
(String dataCollectionId) private Table
private Threads.Timer
private void
incrementTableRowsScanned
(P partition, long rows) void
init
(P partition, OffsetContext offsetContext) private boolean
isTableInvalid
(P partition) private Object[]
keyFromRow
(Object[] row) private void
nextDataCollection
(P partition) protected void
protected void
postReadChunk
(IncrementalSnapshotContext<T> context) protected void
preReadChunk
(IncrementalSnapshotContext<T> context) void
processSchemaChange
(P partition, DataCollectionId dataCollectionId) protected void
private Table
protected PreparedStatement
protected Table
refreshTableSchema
(Table table) void
rereadChunk
(P partition) private boolean
Verifies that in-memory representation of the table’s schema is up to date with the table's schema in the database.protected void
sendEvent
(P partition, EventDispatcher<P, T> dispatcher, OffsetContext offsetContext, Object[] row) protected void
sendWindowEvents
(P partition, OffsetContext offsetContext) protected void
setContext
(IncrementalSnapshotContext<T> context) private void
tableScanCompleted
(P partition) private void
Verifies that table's schema in the database has not changed since it was captured in the previous windowMethods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
Methods inherited from interface io.debezium.pipeline.source.snapshot.incremental.IncrementalSnapshotChangeEventSource
processFilteredEvent, processHeartbeat, processMessage, processTransactionCommittedEvent, processTransactionStartedEvent
-
Field Details
-
LOGGER
private static final org.slf4j.Logger LOGGER -
connectorConfig
-
clock
-
databaseSchema
-
progressListener
-
dataListener
-
totalRowsScanned
private long totalRowsScanned -
currentTable
-
dispatcher
-
context
-
jdbcConnection
-
window
-
-
Constructor Details
-
AbstractIncrementalSnapshotChangeEventSource
public AbstractIncrementalSnapshotChangeEventSource(RelationalDatabaseConnectorConfig config, JdbcConnection jdbcConnection, EventDispatcher<P, T> dispatcher, DatabaseSchema<?> databaseSchema, Clock clock, SnapshotProgressListener<P> progressListener, DataChangeEventListener<P> dataChangeEventListener)
-
-
Method Details
-
closeWindow
public void closeWindow(P partition, String id, OffsetContext offsetContext) throws InterruptedException - Specified by:
closeWindow
in interfaceIncrementalSnapshotChangeEventSource<P extends Partition,
T extends DataCollectionId> - Throws:
InterruptedException
-
processSchemaChange
public void processSchemaChange(P partition, DataCollectionId dataCollectionId) throws InterruptedException - Specified by:
processSchemaChange
in interfaceIncrementalSnapshotChangeEventSource<P extends Partition,
T extends DataCollectionId> - Throws:
InterruptedException
-
rereadChunk
- Throws:
InterruptedException
-
getSignalTableName
-
sendWindowEvents
protected void sendWindowEvents(P partition, OffsetContext offsetContext) throws InterruptedException - Throws:
InterruptedException
-
sendEvent
protected void sendEvent(P partition, EventDispatcher<P, T> dispatcher, OffsetContext offsetContext, Object[] row) throws InterruptedException- Throws:
InterruptedException
-
getChangeRecordEmitter
protected ChangeRecordEmitter<P> getChangeRecordEmitter(P partition, T dataCollectionId, OffsetContext offsetContext, Object[] row) Returns aChangeRecordEmitter
producing the change records for the given table row. -
deduplicateWindow
-
emitWindowOpen
Update low watermark for the incremental snapshot chunk- Throws:
SQLException
-
emitWindowClose
Update high watermark for the incremental snapshot chunk- Throws:
SQLException
InterruptedException
-
buildChunkQuery
-
buildChunkQuery
-
addLowerBound
-
buildMaxPrimaryKeyQuery
-
init
- Specified by:
init
in interfaceIncrementalSnapshotChangeEventSource<P extends Partition,
T extends DataCollectionId>
-
readChunk
- Throws:
InterruptedException
-
isTableInvalid
-
schemaHistoryIsUpToDate
private boolean schemaHistoryIsUpToDate()Verifies that in-memory representation of the table’s schema is up to date with the table's schema in the database.Verification is a two step process:
- Save table's schema from the database to the context
- Verify schema hasn't changed in the following window. If schema has changed repeat the process
Verification is done at the beginning of the incremental snapshot and on every schema change during the snapshotting.
-
verifySchemaUnchanged
private void verifySchemaUnchanged()Verifies that table's schema in the database has not changed since it was captured in the previous window -
readSchema
-
nextDataCollection
-
addDataCollectionNamesToSnapshot
public void addDataCollectionNamesToSnapshot(P partition, List<String> dataCollectionIds, OffsetContext offsetContext) throws InterruptedException - Specified by:
addDataCollectionNamesToSnapshot
in interfaceIncrementalSnapshotChangeEventSource<P extends Partition,
T extends DataCollectionId> - Throws:
InterruptedException
-
addKeyColumnsToCondition
-
createDataEventsForTable
Dispatches the data change events for the records of a single table. -
checkSchemaChanges
- Throws:
SQLException
-
getTable
- Throws:
SQLException
-
incrementTableRowsScanned
-
tableScanCompleted
-
readTableChunkStatement
- Throws:
SQLException
-
getTableScanLogTimer
-
keyFromRow
-
setContext
-
preReadChunk
-
postReadChunk
-
postIncrementalSnapshotCompleted
protected void postIncrementalSnapshotCompleted() -
refreshTableSchema
- Throws:
SQLException
-
getKeyMapper
-