io.debezium.connector.mysql

Class SnapshotReader

java.lang.Object

io.debezium.connector.mysql.AbstractReader

io.debezium.connector.mysql.SnapshotReader

All Implemented Interfaces:

Reader

public class SnapshotReader
extends AbstractReader

A component that performs a snapshot of a MySQL server, and records the schema changes in MySqlSchema.

Author:

Randall Hauch

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static interface	SnapshotReader.RecordRecorder

Nested classes/interfaces inherited from interface io.debezium.connector.mysql.Reader

Reader.State

Field Summary

Fields

Modifier and Type	Field and Description
private ExecutorService	executorService
private boolean	includeData
private SnapshotReaderMetrics	metrics
private SnapshotReader.RecordRecorder	recorder
private MySqlConnectorConfig.SnapshotLockingMode	snapshotLockingMode
private static Pattern	TIME_FIELD_PATTERN Used to parse values of TIME columns.

Fields inherited from class io.debezium.connector.mysql.AbstractReader

connectionContext, context, logger

Constructor Summary

Constructors

Constructor and Description
SnapshotReader(String name, MySqlTaskContext context) Create a snapshot reader.

Method Summary

Modifier and Type	Method and Description
private Statement	createStatement(Connection connection)
private Statement	createStatementWithLargeResultSet(Connection connection) Create a JDBC statement that can be used for large result sets.
protected void	doCleanup() The reader has completed all processing and all enqueued records have been consumed, so this reader should clean up any resources that might remain.
void	doDestroy() The reader has been requested to de-initialize resources after stopping.
protected void	doInitialize() The reader has been requested to initialize resources prior to starting.
protected void	doStart() Start the snapshot and return immediately.
protected void	doStop() The reader has been requested to stop, so perform any work required to stop the reader's resources that were previously started.
protected void	enqueueSchemaChanges(String dbName, String ddlStatement)
protected void	execute() Perform the snapshot using the same logic as the "mysqldump" utility.
SnapshotReader	generateInsertEvents() Set this reader's execution to produce a Envelope.Operation.CREATE event for each row.
SnapshotReader	generateReadEvents() Set this reader's execution to produce a Envelope.Operation.READ event for each row.
private Map<TableId,String>	getSnapshotSelectOverridesByTable() Returns any SELECT overrides, if present.
private void	logRolesForCurrentUser(JdbcConnection mysql)
private void	logServerInformation(JdbcConnection mysql)
protected String	quote(String dbOrTableName)
protected String	quote(TableId id)
protected void	readBinlogPosition(int step, SourceInfo source, JdbcConnection mysql, AtomicReference<String> sql)
protected Object	readField(ResultSet rs, int fieldNo, Column actualColumn)
private Object	readTimeField(ResultSet rs, int fieldNo) As MySQL connector/J implementation is broken for MySQL type "TIME" we have to use a binary-ish workaround
protected void	recordRowAsInsert(RecordMakers.RecordsForTable recordMaker, Object[] row, long ts)
protected void	recordRowAsRead(RecordMakers.RecordsForTable recordMaker, Object[] row, long ts)
protected org.apache.kafka.connect.source.SourceRecord	replaceOffset(org.apache.kafka.connect.source.SourceRecord record) Utility method to replace the offset in the given record with the latest.
private String	rightPad(String input, int length, char c)

Methods inherited from class io.debezium.connector.mysql.AbstractReader

cleanupResources, completeSuccessfully, destroy, enqueueRecord, failed, failed, initialize, isRunning, name, poll, pollComplete, start, state, stop, uponCompletion, wrap

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

TIME_FIELD_PATTERN

private static final Pattern TIME_FIELD_PATTERN

Used to parse values of TIME columns. Format: 000:00:00.000000.

includeData

private final boolean includeData

recorder

private SnapshotReader.RecordRecorder recorder

metrics

private final SnapshotReaderMetrics metrics

executorService

private ExecutorService executorService

snapshotLockingMode

private final MySqlConnectorConfig.SnapshotLockingMode snapshotLockingMode

Constructor Detail

SnapshotReader

public SnapshotReader(String name,
                      MySqlTaskContext context)

Create a snapshot reader.

Parameters:

name - the name of this reader; may not be null

context - the task context in which this reader is running; may not be null

Method Detail

generateReadEvents

public SnapshotReader generateReadEvents()

Set this reader's execution to produce a Envelope.Operation.READ event for each row.

Returns:

this object for method chaining; never null

generateInsertEvents

public SnapshotReader generateInsertEvents()

Set this reader's execution to produce a Envelope.Operation.CREATE event for each row.

Returns:

this object for method chaining; never null

doInitialize

protected void doInitialize()

Description copied from class: AbstractReader

The reader has been requested to initialize resources prior to starting. This should only be called once before AbstractReader.doStart().

Overrides:

doInitialize in class AbstractReader

doDestroy

public void doDestroy()

Description copied from class: AbstractReader

The reader has been requested to de-initialize resources after stopping. This should only be called once after AbstractReader.doStop().

Overrides:

doDestroy in class AbstractReader

doStart

protected void doStart()

Start the snapshot and return immediately. Once started, the records read from the database can be retrieved using AbstractReader.poll() until that method returns null.

Specified by:

doStart in class AbstractReader

doStop

protected void doStop()

Description copied from class: AbstractReader

The reader has been requested to stop, so perform any work required to stop the reader's resources that were previously started.

This method is always called when AbstractReader.stop() is called, and the first time AbstractReader.isRunning() will return true the first time and false for any subsequent calls.

Specified by:

doStop in class AbstractReader

doCleanup

protected void doCleanup()

Description copied from class: AbstractReader

The reader has completed all processing and all enqueued records have been consumed, so this reader should clean up any resources that might remain.

Specified by:

doCleanup in class AbstractReader

readField

protected Object readField(ResultSet rs,
                           int fieldNo,
                           Column actualColumn)
                    throws SQLException

Throws:

SQLException

readTimeField

private Object readTimeField(ResultSet rs,
                             int fieldNo)
                      throws SQLException

As MySQL connector/J implementation is broken for MySQL type "TIME" we have to use a binary-ish workaround

Throws:

SQLException

See Also:

https://issues.jboss.org/browse/DBZ-342

rightPad

private String rightPad(String input,
                        int length,
                        char c)

execute

protected void execute()

Perform the snapshot using the same logic as the "mysqldump" utility.

readBinlogPosition

protected void readBinlogPosition(int step,
                                  SourceInfo source,
                                  JdbcConnection mysql,
                                  AtomicReference<String> sql)
                           throws SQLException

Throws:

SQLException

quote

protected String quote(String dbOrTableName)

quote

protected String quote(TableId id)

createStatementWithLargeResultSet

private Statement createStatementWithLargeResultSet(Connection connection)
                                             throws SQLException

Create a JDBC statement that can be used for large result sets.

By default, the MySQL Connector/J driver retrieves all rows for ResultSets and stores them in memory. In most cases this is the most efficient way to operate and, due to the design of the MySQL network protocol, is easier to implement. However, when ResultSets that have a large number of rows or large values, the driver may not be able to allocate heap space in the JVM and may result in an OutOfMemoryError. See DBZ-94 for details.

This method handles such cases using the recommended technique for MySQL by creating the JDBC Statement with forward-only cursor and read-only concurrency flags, and with a minimum value fetch size hint.

Parameters:

connection - the JDBC connection; may not be null

Returns:

the statement; never null

Throws:

SQLException - if there is a problem creating the statement

createStatement

private Statement createStatement(Connection connection)
                           throws SQLException

Throws:

SQLException

logServerInformation

private void logServerInformation(JdbcConnection mysql)

logRolesForCurrentUser

private void logRolesForCurrentUser(JdbcConnection mysql)

replaceOffset

protected org.apache.kafka.connect.source.SourceRecord replaceOffset(org.apache.kafka.connect.source.SourceRecord record)

Utility method to replace the offset in the given record with the latest. This is used on the last record produced during the snapshot.

Parameters:

record - the record

Returns:

the updated record

enqueueSchemaChanges

protected void enqueueSchemaChanges(String dbName,
                                    String ddlStatement)

recordRowAsRead

protected void recordRowAsRead(RecordMakers.RecordsForTable recordMaker,
                               Object[] row,
                               long ts)
                        throws InterruptedException

Throws:

InterruptedException

recordRowAsInsert

protected void recordRowAsInsert(RecordMakers.RecordsForTable recordMaker,
                                 Object[] row,
                                 long ts)
                          throws InterruptedException

Throws:

InterruptedException

getSnapshotSelectOverridesByTable

private Map<TableId,String> getSnapshotSelectOverridesByTable()

Returns any SELECT overrides, if present.