Class SnapshotReader
- java.lang.Object
-
- io.debezium.connector.mysql.legacy.AbstractReader
-
- io.debezium.connector.mysql.legacy.SnapshotReader
-
- All Implemented Interfaces:
Reader
public class SnapshotReader extends AbstractReader
A component that performs a snapshot of a MySQL server, and records the schema changes inMySqlSchema
.- Author:
- Randall Hauch
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description protected static interface
SnapshotReader.RecordRecorder
-
Nested classes/interfaces inherited from class io.debezium.connector.mysql.legacy.AbstractReader
AbstractReader.AcceptAllPredicate
-
Nested classes/interfaces inherited from interface io.debezium.connector.mysql.legacy.Reader
Reader.State
-
-
Field Summary
Fields Modifier and Type Field Description private ExecutorService
executorService
private boolean
includeData
private SnapshotReaderMetrics
metrics
private MysqlFieldReader
mysqlFieldReader
private SnapshotReader.RecordRecorder
recorder
private MySqlConnectorConfig.SnapshotLockingMode
snapshotLockingMode
private boolean
useGlobalLock
-
Fields inherited from class io.debezium.connector.mysql.legacy.AbstractReader
changeEventQueueMetrics, connectionContext, context, logger
-
-
Constructor Summary
Constructors Constructor Description SnapshotReader(String name, MySqlTaskContext context)
Create a snapshot reader.SnapshotReader(String name, MySqlTaskContext context, boolean useGlobalLock)
Create a snapshot reader that can use global locking only optionally.
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description private Statement
createStatement(Connection connection)
private Statement
createStatementWithLargeResultSet(Connection connection)
Create a JDBC statement that can be used for large result sets.protected void
doCleanup()
The reader has completed all processing and allenqueued records
have beenconsumed
, so this reader should clean up any resources that might remain.void
doDestroy()
The reader has been requested to de-initialize resources after stopping.protected void
doInitialize()
The reader has been requested to initialize resources prior to starting.protected void
doStart()
Start the snapshot and return immediately.protected void
doStop()
The reader has been requested to stop, so perform any work required to stop the reader's resources that were previouslystarted
.protected void
enqueueSchemaChanges(String dbName, Set<TableId> tables, String ddlStatement)
protected void
execute()
Perform the snapshot using the same logic as the "mysqldump" utility.SnapshotReader
generateReadEvents()
Set this reader'sexecution
to produce aEnvelope.Operation.READ
event for each row.private Filters
getCreateTableFilters(Filters filters)
Get the filters for table creation.private void
logRolesForCurrentUser(JdbcConnection mysql)
private void
logServerInformation(JdbcConnection mysql)
protected String
quote(TableId id)
protected String
quote(String dbOrTableName)
protected void
readBinlogPosition(int step, SourceInfo source, JdbcConnection mysql, AtomicReference<String> sql)
private void
readTableSchema(AtomicReference<String> sql, JdbcConnection mysql, MySqlSchema schema, SourceInfo source, String dbName, TableId tableId)
protected void
recordRowAsInsert(RecordMakers.RecordsForTable recordMaker, Object[] row, Instant ts)
protected void
recordRowAsRead(RecordMakers.RecordsForTable recordMaker, Object[] row, Instant ts)
protected org.apache.kafka.connect.source.SourceRecord
replaceOffsetAndSource(org.apache.kafka.connect.source.SourceRecord record)
Utility method to replace the offset and the source in the given record with the latest.private boolean
shouldRecordTableSchema(MySqlSchema schema, Filters filters, TableId id)
Whether DDL for the given table should be recorded.-
Methods inherited from class io.debezium.connector.mysql.legacy.AbstractReader
cleanupResources, completeSuccessfully, destroy, enqueueRecord, failed, failed, initialize, isRunning, name, poll, pollComplete, start, state, stop, toString, uponCompletion, wrap
-
-
-
-
Field Detail
-
includeData
private final boolean includeData
-
recorder
private SnapshotReader.RecordRecorder recorder
-
metrics
private final SnapshotReaderMetrics metrics
-
executorService
private ExecutorService executorService
-
useGlobalLock
private final boolean useGlobalLock
-
mysqlFieldReader
private final MysqlFieldReader mysqlFieldReader
-
snapshotLockingMode
private final MySqlConnectorConfig.SnapshotLockingMode snapshotLockingMode
-
-
Constructor Detail
-
SnapshotReader
public SnapshotReader(String name, MySqlTaskContext context)
Create a snapshot reader.- Parameters:
name
- the name of this reader; may not be nullcontext
- the task context in which this reader is running; may not be null
-
SnapshotReader
SnapshotReader(String name, MySqlTaskContext context, boolean useGlobalLock)
Create a snapshot reader that can use global locking only optionally. Used mostly for testing.- Parameters:
name
- the name of this reader; may not be nullcontext
- the task context in which this reader is running; may not be nulluseGlobalLock
-false
to simulate cloud (Amazon RDS) restrictions
-
-
Method Detail
-
generateReadEvents
public SnapshotReader generateReadEvents()
Set this reader'sexecution
to produce aEnvelope.Operation.READ
event for each row.- Returns:
- this object for method chaining; never null
-
doInitialize
protected void doInitialize()
Description copied from class:AbstractReader
The reader has been requested to initialize resources prior to starting. This should only be called once beforeAbstractReader.doStart()
.- Overrides:
doInitialize
in classAbstractReader
-
doDestroy
public void doDestroy()
Description copied from class:AbstractReader
The reader has been requested to de-initialize resources after stopping. This should only be called once afterAbstractReader.doStop()
.- Overrides:
doDestroy
in classAbstractReader
-
doStart
protected void doStart()
Start the snapshot and return immediately. Once started, the records read from the database can be retrieved usingAbstractReader.poll()
until that method returnsnull
.- Specified by:
doStart
in classAbstractReader
-
doStop
protected void doStop()
Description copied from class:AbstractReader
The reader has been requested to stop, so perform any work required to stop the reader's resources that were previouslystarted
.This method is always called when
AbstractReader.stop()
is called, and the first timeAbstractReader.isRunning()
will returntrue
the first time andfalse
for any subsequent calls.- Specified by:
doStop
in classAbstractReader
-
doCleanup
protected void doCleanup()
Description copied from class:AbstractReader
The reader has completed all processing and allenqueued records
have beenconsumed
, so this reader should clean up any resources that might remain.- Specified by:
doCleanup
in classAbstractReader
-
execute
protected void execute()
Perform the snapshot using the same logic as the "mysqldump" utility.
-
readTableSchema
private void readTableSchema(AtomicReference<String> sql, JdbcConnection mysql, MySqlSchema schema, SourceInfo source, String dbName, TableId tableId) throws SQLException
- Throws:
SQLException
-
shouldRecordTableSchema
private boolean shouldRecordTableSchema(MySqlSchema schema, Filters filters, TableId id)
Whether DDL for the given table should be recorded.
-
readBinlogPosition
protected void readBinlogPosition(int step, SourceInfo source, JdbcConnection mysql, AtomicReference<String> sql) throws SQLException
- Throws:
SQLException
-
getCreateTableFilters
private Filters getCreateTableFilters(Filters filters)
Get the filters for table creation. Depending on the configuration, this may not be the default filter set.- Parameters:
filters
- the default filters of thisSnapshotReader
- Returns:
Filters
that represent all the tables that this snapshot reader should CREATE
-
createStatementWithLargeResultSet
private Statement createStatementWithLargeResultSet(Connection connection) throws SQLException
Create a JDBC statement that can be used for large result sets.By default, the MySQL Connector/J driver retrieves all rows for ResultSets and stores them in memory. In most cases this is the most efficient way to operate and, due to the design of the MySQL network protocol, is easier to implement. However, when ResultSets that have a large number of rows or large values, the driver may not be able to allocate heap space in the JVM and may result in an
OutOfMemoryError
. See DBZ-94 for details.This method handles such cases using the recommended technique for MySQL by creating the JDBC
Statement
withforward-only
cursor andread-only concurrency
flags, and with aminimum value
fetch size hint
.- Parameters:
connection
- the JDBC connection; may not be null- Returns:
- the statement; never null
- Throws:
SQLException
- if there is a problem creating the statement
-
createStatement
private Statement createStatement(Connection connection) throws SQLException
- Throws:
SQLException
-
logServerInformation
private void logServerInformation(JdbcConnection mysql)
-
logRolesForCurrentUser
private void logRolesForCurrentUser(JdbcConnection mysql)
-
replaceOffsetAndSource
protected org.apache.kafka.connect.source.SourceRecord replaceOffsetAndSource(org.apache.kafka.connect.source.SourceRecord record)
Utility method to replace the offset and the source in the given record with the latest. This is used on the last record produced during the snapshot.- Parameters:
record
- the record- Returns:
- the updated record
-
enqueueSchemaChanges
protected void enqueueSchemaChanges(String dbName, Set<TableId> tables, String ddlStatement)
-
recordRowAsRead
protected void recordRowAsRead(RecordMakers.RecordsForTable recordMaker, Object[] row, Instant ts) throws InterruptedException
- Throws:
InterruptedException
-
recordRowAsInsert
protected void recordRowAsInsert(RecordMakers.RecordsForTable recordMaker, Object[] row, Instant ts) throws InterruptedException
- Throws:
InterruptedException
-
-