Class MySqlStreamingChangeEventSource
- java.lang.Object
-
- io.debezium.connector.mysql.MySqlStreamingChangeEventSource
-
- All Implemented Interfaces:
ChangeEventSource
,StreamingChangeEventSource
public class MySqlStreamingChangeEventSource extends Object implements StreamingChangeEventSource
- Author:
- Jiri Pechanec
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description private static interface
MySqlStreamingChangeEventSource.BinlogChangeEmitter<T>
static class
MySqlStreamingChangeEventSource.BinlogPosition
protected class
MySqlStreamingChangeEventSource.ReaderThreadLifecycleListener
private static interface
MySqlStreamingChangeEventSource.RowsProvider<E extends com.github.shyiko.mysql.binlog.event.EventData,U>
private static interface
MySqlStreamingChangeEventSource.TableIdProvider<E extends com.github.shyiko.mysql.binlog.event.EventData>
-
Nested classes/interfaces inherited from interface io.debezium.pipeline.source.spi.ChangeEventSource
ChangeEventSource.ChangeEventSourceContext
-
-
Field Summary
-
Constructor Summary
Constructors Constructor Description MySqlStreamingChangeEventSource(MySqlConnectorConfig connectorConfig, MySqlOffsetContext offsetContext, MySqlConnection connection, EventDispatcher<TableId> dispatcher, ErrorHandler errorHandler, Clock clock, MySqlTaskContext taskContext, MySqlStreamingChangeEventSourceMetrics metrics)
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description void
execute(ChangeEventSource.ChangeEventSourceContext context)
GtidSet
filterGtidSet(GtidSet availableServerGtidSet, GtidSet purgedServerGtid)
Apply the include/exclude GTID source filters to the currentGTID set
and merge them onto the currently available GTID set from a MySQL server.private com.github.shyiko.mysql.binlog.network.SSLSocketFactory
getBinlogSslSocketFactory(MySqlConnectorConfig connectorConfig, MySqlConnection connection)
(package private) MySqlStreamingChangeEventSource.BinlogPosition
getCurrentBinlogPosition()
(package private) MySqlStreamingChangeEventSourceMetrics
getMetrics()
private <T extends com.github.shyiko.mysql.binlog.event.EventData,U>
voidhandleChange(com.github.shyiko.mysql.binlog.event.Event event, String changeType, Class<T> eventDataClass, MySqlStreamingChangeEventSource.TableIdProvider<T> tableIdProvider, MySqlStreamingChangeEventSource.RowsProvider<T,U> rowsProvider, MySqlStreamingChangeEventSource.BinlogChangeEmitter<U> changeEmitter)
protected void
handleDelete(com.github.shyiko.mysql.binlog.event.Event event)
Generate source records for the supplied event with anDeleteRowsEventData
.protected void
handleEvent(com.github.shyiko.mysql.binlog.event.Event event)
protected void
handleGtidEvent(com.github.shyiko.mysql.binlog.event.Event event)
Handle the supplied event with aGtidEventData
that signals the beginning of a GTID transaction.protected void
handleInsert(com.github.shyiko.mysql.binlog.event.Event event)
Generate source records for the supplied event with anWriteRowsEventData
.protected void
handleQueryEvent(com.github.shyiko.mysql.binlog.event.Event event)
Handle the supplied event with anQueryEventData
by possibly recording the DDL statements as changes in the MySQL schemas.protected void
handleRotateLogsEvent(com.github.shyiko.mysql.binlog.event.Event event)
Handle the supplied event with aRotateEventData
that signals the logs are being rotated.protected void
handleRowsQuery(com.github.shyiko.mysql.binlog.event.Event event)
Handle the supplied event with anRowsQueryEventData
by recording the original SQL query that generated the event.protected void
handleServerHeartbeat(com.github.shyiko.mysql.binlog.event.Event event)
Handle the supplied event that is sent by a primary to a replica to let the replica know that the primary is still alive.protected void
handleServerIncident(com.github.shyiko.mysql.binlog.event.Event event)
Handle the supplied event that signals that an out of the ordinary event that occurred on the master.protected void
handleServerStop(com.github.shyiko.mysql.binlog.event.Event event)
Handle the supplied event that signals that mysqld has stopped.private void
handleTransactionCompletion(com.github.shyiko.mysql.binlog.event.Event event)
protected void
handleUpdate(com.github.shyiko.mysql.binlog.event.Event event)
Generate source records for the supplied event with anUpdateRowsEventData
.protected void
handleUpdateTableMetadata(com.github.shyiko.mysql.binlog.event.Event event)
Handle a change in the table metadata.protected void
ignoreEvent(com.github.shyiko.mysql.binlog.event.Event event)
private void
informAboutUnknownTableIfRequired(com.github.shyiko.mysql.binlog.event.Event event, TableId tableId, String typeToLog)
If we receive an event for a table that is monitored but whose metadata we don't know, either ignore that event or raise a warning or error as per theMySqlConnectorConfig.INCONSISTENT_SCHEMA_HANDLING_MODE
configuration.protected void
logEvent(com.github.shyiko.mysql.binlog.event.Event event)
private void
logStreamingSourceState()
private void
logStreamingSourceState(org.slf4j.event.Level severity)
protected void
onEvent(com.github.shyiko.mysql.binlog.event.Event event)
protected void
prepareTransaction(com.github.shyiko.mysql.binlog.event.Event event)
Handle aEventType.XA_PREPARE
event.(package private) void
rewindBinaryLogClient(ChangeEventSource.ChangeEventSourceContext context, MySqlStreamingChangeEventSource.BinlogPosition position)
private com.github.shyiko.mysql.binlog.network.SSLMode
sslModeFor(MySqlConnectorConfig.SecureConnectionMode mode)
protected <T extends com.github.shyiko.mysql.binlog.event.EventData>
TunwrapData(com.github.shyiko.mysql.binlog.event.Event event)
protected void
viewChange(com.github.shyiko.mysql.binlog.event.Event event)
Handle aEventType.VIEW_CHANGE
event.protected io.debezium.DebeziumException
wrap(Throwable error)
Wraps the specified exception in aDebeziumException
, ensuring that all useful state is captured inside the new exception's message.-
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
-
Methods inherited from interface io.debezium.pipeline.source.spi.StreamingChangeEventSource
commitOffset
-
-
-
-
Field Detail
-
LOGGER
private static final org.slf4j.Logger LOGGER
-
KEEPALIVE_THREAD_NAME
private static final String KEEPALIVE_THREAD_NAME
- See Also:
- Constant Field Values
-
eventHandlers
private final EnumMap<com.github.shyiko.mysql.binlog.event.EventType,BlockingConsumer<com.github.shyiko.mysql.binlog.event.Event>> eventHandlers
-
client
private final com.github.shyiko.mysql.binlog.BinaryLogClient client
-
metrics
private final MySqlStreamingChangeEventSourceMetrics metrics
-
clock
private final Clock clock
-
eventDeserializationFailureHandlingMode
private final CommonConnectorConfig.EventProcessingFailureHandlingMode eventDeserializationFailureHandlingMode
-
inconsistentSchemaHandlingMode
private final CommonConnectorConfig.EventProcessingFailureHandlingMode inconsistentSchemaHandlingMode
-
startingRowNumber
private int startingRowNumber
-
initialEventsToSkip
private long initialEventsToSkip
-
skipEvent
private boolean skipEvent
-
ignoreDmlEventByGtidSource
private boolean ignoreDmlEventByGtidSource
-
totalRecordCounter
private final AtomicLong totalRecordCounter
-
gtidSet
private com.github.shyiko.mysql.binlog.GtidSet gtidSet
-
heartbeatIntervalFactor
private final float heartbeatIntervalFactor
- See Also:
- Constant Field Values
-
taskContext
private final MySqlTaskContext taskContext
-
connectorConfig
private final MySqlConnectorConfig connectorConfig
-
connection
private final MySqlConnection connection
-
eventDispatcher
private final EventDispatcher<TableId> eventDispatcher
-
offsetContext
private final MySqlOffsetContext offsetContext
-
errorHandler
private final ErrorHandler errorHandler
-
eventTimestamp
private Instant eventTimestamp
-
-
Constructor Detail
-
MySqlStreamingChangeEventSource
public MySqlStreamingChangeEventSource(MySqlConnectorConfig connectorConfig, MySqlOffsetContext offsetContext, MySqlConnection connection, EventDispatcher<TableId> dispatcher, ErrorHandler errorHandler, Clock clock, MySqlTaskContext taskContext, MySqlStreamingChangeEventSourceMetrics metrics)
-
-
Method Detail
-
onEvent
protected void onEvent(com.github.shyiko.mysql.binlog.event.Event event)
-
ignoreEvent
protected void ignoreEvent(com.github.shyiko.mysql.binlog.event.Event event)
-
handleEvent
protected void handleEvent(com.github.shyiko.mysql.binlog.event.Event event)
-
unwrapData
protected <T extends com.github.shyiko.mysql.binlog.event.EventData> T unwrapData(com.github.shyiko.mysql.binlog.event.Event event)
-
handleServerStop
protected void handleServerStop(com.github.shyiko.mysql.binlog.event.Event event)
Handle the supplied event that signals that mysqld has stopped.- Parameters:
event
- the server stopped event to be processed; may not be null
-
handleServerHeartbeat
protected void handleServerHeartbeat(com.github.shyiko.mysql.binlog.event.Event event)
Handle the supplied event that is sent by a primary to a replica to let the replica know that the primary is still alive. Not written to a binary log.- Parameters:
event
- the server stopped event to be processed; may not be null
-
handleServerIncident
protected void handleServerIncident(com.github.shyiko.mysql.binlog.event.Event event)
Handle the supplied event that signals that an out of the ordinary event that occurred on the master. It notifies the replica that something happened on the primary that might cause data to be in an inconsistent state.- Parameters:
event
- the server stopped event to be processed; may not be null
-
handleRotateLogsEvent
protected void handleRotateLogsEvent(com.github.shyiko.mysql.binlog.event.Event event)
Handle the supplied event with aRotateEventData
that signals the logs are being rotated. This means that either the server was restarted, or the binlog has transitioned to a new file. In either case, subsequent table numbers will be different than those seen to this point.- Parameters:
event
- the database change data event to be processed; may not be null
-
handleGtidEvent
protected void handleGtidEvent(com.github.shyiko.mysql.binlog.event.Event event)
Handle the supplied event with aGtidEventData
that signals the beginning of a GTID transaction. We don't yet know whether this transaction contains any events we're interested in, but we have to record it so that we know the position of this event and know we've processed the binlog to this point.Note that this captures the current GTID and complete GTID set, regardless of whether the connector is
filtering
the GTID set upon connection. We do this because we actually want to capture all GTID set values found in the binlog, whether or not we process them. However, only when we connect do we actually want to pass to MySQL only those GTID ranges that are applicable per the configuration.- Parameters:
event
- the GTID event to be processed; may not be null
-
handleRowsQuery
protected void handleRowsQuery(com.github.shyiko.mysql.binlog.event.Event event)
Handle the supplied event with anRowsQueryEventData
by recording the original SQL query that generated the event.- Parameters:
event
- the database change data event to be processed; may not be null
-
handleQueryEvent
protected void handleQueryEvent(com.github.shyiko.mysql.binlog.event.Event event) throws InterruptedException
Handle the supplied event with anQueryEventData
by possibly recording the DDL statements as changes in the MySQL schemas.- Parameters:
event
- the database change data event to be processed; may not be null- Throws:
InterruptedException
- if this thread is interrupted while recording the DDL statements
-
handleTransactionCompletion
private void handleTransactionCompletion(com.github.shyiko.mysql.binlog.event.Event event) throws InterruptedException
- Throws:
InterruptedException
-
handleUpdateTableMetadata
protected void handleUpdateTableMetadata(com.github.shyiko.mysql.binlog.event.Event event)
Handle a change in the table metadata.This method should be called whenever we consume a TABLE_MAP event, and every transaction in the log should include one of these for each table affected by the transaction. Each table map event includes a monotonically-increasing numeric identifier, and this identifier is used within subsequent events within the same transaction. This table identifier can change when:
- the table structure is modified (e.g., via an
ALTER TABLE ...
command); or - MySQL rotates to a new binary log file, even if the table structure does not change.
- Parameters:
event
- the update event; never null
- the table structure is modified (e.g., via an
-
informAboutUnknownTableIfRequired
private void informAboutUnknownTableIfRequired(com.github.shyiko.mysql.binlog.event.Event event, TableId tableId, String typeToLog)
If we receive an event for a table that is monitored but whose metadata we don't know, either ignore that event or raise a warning or error as per theMySqlConnectorConfig.INCONSISTENT_SCHEMA_HANDLING_MODE
configuration.
-
handleInsert
protected void handleInsert(com.github.shyiko.mysql.binlog.event.Event event) throws InterruptedException
Generate source records for the supplied event with anWriteRowsEventData
.- Parameters:
event
- the database change data event to be processed; may not be null- Throws:
InterruptedException
- if this thread is interrupted while blocking
-
handleUpdate
protected void handleUpdate(com.github.shyiko.mysql.binlog.event.Event event) throws InterruptedException
Generate source records for the supplied event with anUpdateRowsEventData
.- Parameters:
event
- the database change data event to be processed; may not be null- Throws:
InterruptedException
- if this thread is interrupted while blocking
-
handleDelete
protected void handleDelete(com.github.shyiko.mysql.binlog.event.Event event) throws InterruptedException
Generate source records for the supplied event with anDeleteRowsEventData
.- Parameters:
event
- the database change data event to be processed; may not be null- Throws:
InterruptedException
- if this thread is interrupted while blocking
-
handleChange
private <T extends com.github.shyiko.mysql.binlog.event.EventData,U> void handleChange(com.github.shyiko.mysql.binlog.event.Event event, String changeType, Class<T> eventDataClass, MySqlStreamingChangeEventSource.TableIdProvider<T> tableIdProvider, MySqlStreamingChangeEventSource.RowsProvider<T,U> rowsProvider, MySqlStreamingChangeEventSource.BinlogChangeEmitter<U> changeEmitter) throws InterruptedException
- Throws:
InterruptedException
-
viewChange
protected void viewChange(com.github.shyiko.mysql.binlog.event.Event event) throws InterruptedException
Handle aEventType.VIEW_CHANGE
event.- Parameters:
event
- the database change data event to be processed; may not be null- Throws:
InterruptedException
- if this thread is interrupted while blocking
-
prepareTransaction
protected void prepareTransaction(com.github.shyiko.mysql.binlog.event.Event event) throws InterruptedException
Handle aEventType.XA_PREPARE
event.- Parameters:
event
- the database change data event to be processed; may not be null- Throws:
InterruptedException
- if this thread is interrupted while blocking
-
sslModeFor
private com.github.shyiko.mysql.binlog.network.SSLMode sslModeFor(MySqlConnectorConfig.SecureConnectionMode mode)
-
execute
public void execute(ChangeEventSource.ChangeEventSourceContext context) throws InterruptedException
- Specified by:
execute
in interfaceStreamingChangeEventSource
- Throws:
InterruptedException
-
getBinlogSslSocketFactory
private com.github.shyiko.mysql.binlog.network.SSLSocketFactory getBinlogSslSocketFactory(MySqlConnectorConfig connectorConfig, MySqlConnection connection)
-
logStreamingSourceState
private void logStreamingSourceState()
-
logEvent
protected void logEvent(com.github.shyiko.mysql.binlog.event.Event event)
-
logStreamingSourceState
private void logStreamingSourceState(org.slf4j.event.Level severity)
-
filterGtidSet
public GtidSet filterGtidSet(GtidSet availableServerGtidSet, GtidSet purgedServerGtid)
Apply the include/exclude GTID source filters to the currentGTID set
and merge them onto the currently available GTID set from a MySQL server. The merging behavior of this method might seem a bit strange at first. It's required in order for Debezium to consume a MySQL binlog that has multi-source replication enabled, if a failover has to occur. In such a case, the server that Debezium is failed over to might have a different set of sources, but still include the sources required for Debezium to continue to function. MySQL does not allow downstream replicas to connect if the GTID set does not contain GTIDs for all channels that the server is replicating from, even if the server does have the data needed by the client. To get around this, we can have Debezium merge its GTID set with whatever is on the server, so that MySQL will allow it to connect. See DBZ-143 for details. This method does not mutate any state in the context.- Parameters:
availableServerGtidSet
- the GTID set currently available in the MySQL serverpurgedServerGtid
- the GTID set already purged by the MySQL server- Returns:
- A GTID set meant for consuming from a MySQL binlog; may return null if the SourceInfo has no GTIDs and therefore none were filtered
-
getMetrics
MySqlStreamingChangeEventSourceMetrics getMetrics()
-
rewindBinaryLogClient
void rewindBinaryLogClient(ChangeEventSource.ChangeEventSourceContext context, MySqlStreamingChangeEventSource.BinlogPosition position)
-
getCurrentBinlogPosition
MySqlStreamingChangeEventSource.BinlogPosition getCurrentBinlogPosition()
-
wrap
protected io.debezium.DebeziumException wrap(Throwable error)
Wraps the specified exception in aDebeziumException
, ensuring that all useful state is captured inside the new exception's message.- Parameters:
error
- the exception; may not be null- Returns:
- the wrapped Kafka Connect exception
-
-