io.debezium.connector.mysql

Class BinlogReader

java.lang.Object

io.debezium.connector.mysql.AbstractReader

io.debezium.connector.mysql.BinlogReader

All Implemented Interfaces:

Reader

public class BinlogReader
extends AbstractReader

A component that reads the binlog of a MySQL server, and records any schema changes in MySqlSchema.

Author:

Randall Hauch

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	BinlogReader.BinlogPosition
protected class	BinlogReader.ReaderThreadLifecycleListener

Nested classes/interfaces inherited from class io.debezium.connector.mysql.AbstractReader

AbstractReader.AcceptAllPredicate

Nested classes/interfaces inherited from interface io.debezium.connector.mysql.Reader

Reader.State

Field Summary

Fields

Modifier and Type	Field and Description
private Set<Thread>	binaryLogClientThreads
private com.github.shyiko.mysql.binlog.BinaryLogClient	client
private Clock	clock
private MySqlJdbcContext	connectionContext
private CommonConnectorConfig.EventProcessingFailureHandlingMode	eventDeserializationFailureHandlingMode
private EnumMap<com.github.shyiko.mysql.binlog.event.EventType,BlockingConsumer<com.github.shyiko.mysql.binlog.event.Event>>	eventHandlers
private Predicate<String>	gtidDmlSourceFilter
private com.github.shyiko.mysql.binlog.GtidSet	gtidSet
private Heartbeat	heartbeat
private float	heartbeatIntervalFactor
private boolean	ignoreDmlEventByGtidSource
private CommonConnectorConfig.EventProcessingFailureHandlingMode	inconsistentSchemaHandlingMode
private static long	INITIAL_POLL_PERIOD_IN_MILLIS
private long	initialEventsToSkip
private static String	KEEPALIVE_THREAD_NAME
private Map<String,?>	lastOffset
private static long	MAX_POLL_PERIOD_IN_MILLIS
private BinlogReaderMetrics	metrics
private ElapsedTimeStrategy	pollOutputDelay
private long	previousOutputMillis
private long	recordCounter
private RecordMakers	recordMakers
private boolean	recordSchemaChangesInSourceRecords
private boolean	skipEvent
private SourceInfo	source
private int	startingRowNumber
private AtomicLong	totalRecordCounter

Fields inherited from class io.debezium.connector.mysql.AbstractReader

changeEventQueueMetrics, context, logger

Constructor Summary

Constructors

Constructor and Description
BinlogReader(String name, MySqlTaskContext context, HaltingPredicate acceptAndContinue) Create a binlog reader.
BinlogReader(String name, MySqlTaskContext context, HaltingPredicate acceptAndContinue, long serverId) Create a binlog reader.

Method Summary

Modifier and Type	Method and Description
protected void	doCleanup() The reader has completed all processing and all enqueued records have been consumed, so this reader should clean up any resources that might remain.
void	doDestroy() The reader has been requested to de-initialize resources after stopping.
protected void	doInitialize() The reader has been requested to initialize resources prior to starting.
protected void	doStart() The reader has been requested to start, so initialize any un-initialized resources required by the reader.
protected void	doStop() The reader has been requested to stop, so perform any work required to stop the reader's resources that were previously started.
protected com.github.shyiko.mysql.binlog.BinaryLogClient	getBinlogClient()
private com.github.shyiko.mysql.binlog.network.SSLSocketFactory	getBinlogSslSocketFactory(MySqlJdbcContext connectionContext)
BinlogReader.BinlogPosition	getCurrentBinlogPosition()
Map<String,?>	getLastOffset()
protected BinlogReaderMetrics	getMetrics()
protected void	handleDelete(com.github.shyiko.mysql.binlog.event.Event event) Generate source records for the supplied event with an DeleteRowsEventData.
protected void	handleEvent(com.github.shyiko.mysql.binlog.event.Event event)
protected void	handleGtidEvent(com.github.shyiko.mysql.binlog.event.Event event) Handle the supplied event with a GtidEventData that signals the beginning of a GTID transaction.
protected void	handleInsert(com.github.shyiko.mysql.binlog.event.Event event) Generate source records for the supplied event with an WriteRowsEventData.
protected void	handleQueryEvent(com.github.shyiko.mysql.binlog.event.Event event) Handle the supplied event with an QueryEventData by possibly recording the DDL statements as changes in the MySQL schemas.
protected void	handleRotateLogsEvent(com.github.shyiko.mysql.binlog.event.Event event) Handle the supplied event with a RotateEventData that signals the logs are being rotated.
protected void	handleRowsQuery(com.github.shyiko.mysql.binlog.event.Event event) Handle the supplied event with an RowsQueryEventData by recording the original SQL query that generated the event.
protected void	handleServerHeartbeat(com.github.shyiko.mysql.binlog.event.Event event) Handle the supplied event that is sent by a master to a slave to let the slave know that the master is still alive.
protected void	handleServerIncident(com.github.shyiko.mysql.binlog.event.Event event) Handle the supplied event that signals that an out of the ordinary event that occurred on the master.
protected void	handleServerStop(com.github.shyiko.mysql.binlog.event.Event event) Handle the supplied event that signals that mysqld has stopped.
private void	handleTransactionCompletion(com.github.shyiko.mysql.binlog.event.Event event)
protected void	handleUpdate(com.github.shyiko.mysql.binlog.event.Event event) Generate source records for the supplied event with an UpdateRowsEventData.
protected void	handleUpdateTableMetadata(com.github.shyiko.mysql.binlog.event.Event event) Handle a change in the table metadata.
protected void	ignoreEvent(com.github.shyiko.mysql.binlog.event.Event event)
private void	informAboutUnknownTableIfRequired(com.github.shyiko.mysql.binlog.event.Event event, TableId tableId, String typeToLog) If we receive an event for a table that is monitored but whose metadata we don't know, either ignore that event or raise a warning or error as per the MySqlConnectorConfig.INCONSISTENT_SCHEMA_HANDLING_MODE configuration.
protected void	logEvent(com.github.shyiko.mysql.binlog.event.Event event)
private void	logReaderState()
private void	logReaderState(org.slf4j.event.Level severity)
protected void	onEvent(com.github.shyiko.mysql.binlog.event.Event event)
protected void	pollComplete(List<org.apache.kafka.connect.source.SourceRecord> batch) Method called when AbstractReader.poll() completes sending a non-zero-sized batch of records.
protected void	prepareTransaction(com.github.shyiko.mysql.binlog.event.Event event) Handle a EventType.XA_PREPARE event.
protected void	rewindBinaryLogClient(BinlogReader.BinlogPosition position)
protected static com.github.shyiko.mysql.binlog.network.SSLMode	sslModeFor(MySqlConnectorConfig.SecureConnectionMode mode)
protected <T extends com.github.shyiko.mysql.binlog.event.EventData> T	unwrapData(com.github.shyiko.mysql.binlog.event.Event event)
protected void	viewChange(com.github.shyiko.mysql.binlog.event.Event event) Handle a EventType.VIEW_CHANGE event.

Methods inherited from class io.debezium.connector.mysql.AbstractReader

cleanupResources, completeSuccessfully, destroy, enqueueRecord, failed, failed, initialize, isRunning, name, poll, start, state, stop, toString, uponCompletion, wrap

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

INITIAL_POLL_PERIOD_IN_MILLIS

private static final long INITIAL_POLL_PERIOD_IN_MILLIS

MAX_POLL_PERIOD_IN_MILLIS

private static final long MAX_POLL_PERIOD_IN_MILLIS

KEEPALIVE_THREAD_NAME

private static final String KEEPALIVE_THREAD_NAME

See Also:

Constant Field Values

recordSchemaChangesInSourceRecords

private final boolean recordSchemaChangesInSourceRecords

recordMakers

private final RecordMakers recordMakers

source

private final SourceInfo source

eventHandlers

private final EnumMap<com.github.shyiko.mysql.binlog.event.EventType,BlockingConsumer<com.github.shyiko.mysql.binlog.event.Event>> eventHandlers

client

private final com.github.shyiko.mysql.binlog.BinaryLogClient client

metrics

private final BinlogReaderMetrics metrics

clock

private final Clock clock

pollOutputDelay

private final ElapsedTimeStrategy pollOutputDelay

eventDeserializationFailureHandlingMode

private final CommonConnectorConfig.EventProcessingFailureHandlingMode eventDeserializationFailureHandlingMode

inconsistentSchemaHandlingMode

private final CommonConnectorConfig.EventProcessingFailureHandlingMode inconsistentSchemaHandlingMode

startingRowNumber

private int startingRowNumber

recordCounter

private long recordCounter

previousOutputMillis

private long previousOutputMillis

initialEventsToSkip

private long initialEventsToSkip

skipEvent

private boolean skipEvent

ignoreDmlEventByGtidSource

private boolean ignoreDmlEventByGtidSource

gtidDmlSourceFilter

private final Predicate<String> gtidDmlSourceFilter

totalRecordCounter

private final AtomicLong totalRecordCounter

lastOffset

private volatile Map<String,?> lastOffset

gtidSet

private com.github.shyiko.mysql.binlog.GtidSet gtidSet

heartbeat

private Heartbeat heartbeat

connectionContext

private MySqlJdbcContext connectionContext

heartbeatIntervalFactor

private final float heartbeatIntervalFactor

See Also:

Constant Field Values

binaryLogClientThreads

private final Set<Thread> binaryLogClientThreads

Constructor Detail

BinlogReader

public BinlogReader(String name,
                    MySqlTaskContext context,
                    HaltingPredicate acceptAndContinue)

Create a binlog reader.

Parameters:

name - the name of this reader; may not be null

context - the task context in which this reader is running; may not be null

acceptAndContinue - see AbstractReader#AbstractReader(String, MySqlTaskContext, Predicate)

BinlogReader

public BinlogReader(String name,
                    MySqlTaskContext context,
                    HaltingPredicate acceptAndContinue,
                    long serverId)

Create a binlog reader.

Parameters:

name - the name of this reader; may not be null

context - the task context in which this reader is running; may not be null

acceptAndContinue - see AbstractReader#AbstractReader(String, MySqlTaskContext, Predicate)

serverId - the server id to use for the BinaryLogClient

Method Detail

doInitialize

protected void doInitialize()

Description copied from class: AbstractReader

The reader has been requested to initialize resources prior to starting. This should only be called once before AbstractReader.doStart().

Overrides:

doInitialize in class AbstractReader

doDestroy

public void doDestroy()

Description copied from class: AbstractReader

The reader has been requested to de-initialize resources after stopping. This should only be called once after AbstractReader.doStop().

Overrides:

doDestroy in class AbstractReader

doStart

protected void doStart()

Description copied from class: AbstractReader

The reader has been requested to start, so initialize any un-initialized resources required by the reader.

Specified by:

doStart in class AbstractReader

rewindBinaryLogClient

protected void rewindBinaryLogClient(BinlogReader.BinlogPosition position)

getLastOffset

public Map<String,?> getLastOffset()

Returns:

a copy of the last offset of this reader, or null if this reader has not completed a poll.

doStop

protected void doStop()

Description copied from class: AbstractReader

The reader has been requested to stop, so perform any work required to stop the reader's resources that were previously started.

This method is always called when AbstractReader.stop() is called, and the first time AbstractReader.isRunning() will return true the first time and false for any subsequent calls.

Specified by:

doStop in class AbstractReader

doCleanup

protected void doCleanup()

Description copied from class: AbstractReader

The reader has completed all processing and all enqueued records have been consumed, so this reader should clean up any resources that might remain.

Specified by:

doCleanup in class AbstractReader

pollComplete

protected void pollComplete(List<org.apache.kafka.connect.source.SourceRecord> batch)

Description copied from class: AbstractReader

Method called when AbstractReader.poll() completes sending a non-zero-sized batch of records.

Overrides:

pollComplete in class AbstractReader

Parameters:

batch - the batch of records being recorded

logEvent

protected void logEvent(com.github.shyiko.mysql.binlog.event.Event event)

onEvent

protected void onEvent(com.github.shyiko.mysql.binlog.event.Event event)

ignoreEvent

protected void ignoreEvent(com.github.shyiko.mysql.binlog.event.Event event)

handleEvent

protected void handleEvent(com.github.shyiko.mysql.binlog.event.Event event)

unwrapData

protected <T extends com.github.shyiko.mysql.binlog.event.EventData> T unwrapData(com.github.shyiko.mysql.binlog.event.Event event)

handleServerStop

protected void handleServerStop(com.github.shyiko.mysql.binlog.event.Event event)

Handle the supplied event that signals that mysqld has stopped.

Parameters:

event - the server stopped event to be processed; may not be null

handleServerHeartbeat

protected void handleServerHeartbeat(com.github.shyiko.mysql.binlog.event.Event event)

Handle the supplied event that is sent by a master to a slave to let the slave know that the master is still alive. Not written to a binary log.

Parameters:

event - the server stopped event to be processed; may not be null

handleServerIncident

protected void handleServerIncident(com.github.shyiko.mysql.binlog.event.Event event)

Handle the supplied event that signals that an out of the ordinary event that occurred on the master. It notifies the slave that something happened on the master that might cause data to be in an inconsistent state.

Parameters:

event - the server stopped event to be processed; may not be null

handleRotateLogsEvent

protected void handleRotateLogsEvent(com.github.shyiko.mysql.binlog.event.Event event)

Handle the supplied event with a RotateEventData that signals the logs are being rotated. This means that either the server was restarted, or the binlog has transitioned to a new file. In either case, subsequent table numbers will be different than those seen to this point, so we need to discard the cache of record makers.

Parameters:

event - the database change data event to be processed; may not be null

handleGtidEvent

protected void handleGtidEvent(com.github.shyiko.mysql.binlog.event.Event event)

Handle the supplied event with a GtidEventData that signals the beginning of a GTID transaction. We don't yet know whether this transaction contains any events we're interested in, but we have to record it so that we know the position of this event and know we've processed the binlog to this point.

Note that this captures the current GTID and complete GTID set, regardless of whether the connector is filtering the GTID set upon connection. We do this because we actually want to capture all GTID set values found in the binlog, whether or not we process them. However, only when we connect do we actually want to pass to MySQL only those GTID ranges that are applicable per the configuration.

Parameters:

event - the GTID event to be processed; may not be null

handleRowsQuery

protected void handleRowsQuery(com.github.shyiko.mysql.binlog.event.Event event)

Handle the supplied event with an RowsQueryEventData by recording the original SQL query that generated the event.

Parameters:

event - the database change data event to be processed; may not be null

handleQueryEvent

protected void handleQueryEvent(com.github.shyiko.mysql.binlog.event.Event event)
                         throws InterruptedException

Handle the supplied event with an QueryEventData by possibly recording the DDL statements as changes in the MySQL schemas.

Parameters:

event - the database change data event to be processed; may not be null

Throws:

InterruptedException - if this thread is interrupted while recording the DDL statements

handleTransactionCompletion

private void handleTransactionCompletion(com.github.shyiko.mysql.binlog.event.Event event)

handleUpdateTableMetadata

protected void handleUpdateTableMetadata(com.github.shyiko.mysql.binlog.event.Event event)

Handle a change in the table metadata.

This method should be called whenever we consume a TABLE_MAP event, and every transaction in the log should include one of these for each table affected by the transaction. Each table map event includes a monotonically-increasing numeric identifier, and this identifier is used within subsequent events within the same transaction. This table identifier can change when:

1. the table structure is modified (e.g., via an ALTER TABLE ... command); or
2. MySQL rotates to a new binary log file, even if the table structure does not change.

Parameters:

event - the update event; never null

informAboutUnknownTableIfRequired

private void informAboutUnknownTableIfRequired(com.github.shyiko.mysql.binlog.event.Event event,
                                               TableId tableId,
                                               String typeToLog)

If we receive an event for a table that is monitored but whose metadata we don't know, either ignore that event or raise a warning or error as per the MySqlConnectorConfig.INCONSISTENT_SCHEMA_HANDLING_MODE configuration.

handleInsert

protected void handleInsert(com.github.shyiko.mysql.binlog.event.Event event)
                     throws InterruptedException

Generate source records for the supplied event with an WriteRowsEventData.

Parameters:

event - the database change data event to be processed; may not be null

Throws:

InterruptedException - if this thread is interrupted while blocking

handleUpdate

protected void handleUpdate(com.github.shyiko.mysql.binlog.event.Event event)
                     throws InterruptedException

Generate source records for the supplied event with an UpdateRowsEventData.

Parameters:

event - the database change data event to be processed; may not be null

Throws:

InterruptedException - if this thread is interrupted while blocking

handleDelete

protected void handleDelete(com.github.shyiko.mysql.binlog.event.Event event)
                     throws InterruptedException

Generate source records for the supplied event with an DeleteRowsEventData.

Parameters:

event - the database change data event to be processed; may not be null

Throws:

InterruptedException - if this thread is interrupted while blocking

viewChange

protected void viewChange(com.github.shyiko.mysql.binlog.event.Event event)
                   throws InterruptedException

Handle a EventType.VIEW_CHANGE event.

Parameters:

event - the database change data event to be processed; may not be null

Throws:

InterruptedException - if this thread is interrupted while blocking

prepareTransaction

protected void prepareTransaction(com.github.shyiko.mysql.binlog.event.Event event)
                           throws InterruptedException

Handle a EventType.XA_PREPARE event.

Parameters:

event - the database change data event to be processed; may not be null

Throws:

InterruptedException - if this thread is interrupted while blocking

sslModeFor

protected static com.github.shyiko.mysql.binlog.network.SSLMode sslModeFor(MySqlConnectorConfig.SecureConnectionMode mode)

logReaderState

private void logReaderState()

logReaderState

private void logReaderState(org.slf4j.event.Level severity)

getMetrics

protected BinlogReaderMetrics getMetrics()

getBinlogClient

protected com.github.shyiko.mysql.binlog.BinaryLogClient getBinlogClient()

getCurrentBinlogPosition

public BinlogReader.BinlogPosition getCurrentBinlogPosition()

getBinlogSslSocketFactory

private com.github.shyiko.mysql.binlog.network.SSLSocketFactory getBinlogSslSocketFactory(MySqlJdbcContext connectionContext)