Package io.debezium.connector.mysql.legacy

Class BinlogReader

  • All Implemented Interfaces:
    Reader
    public class BinlogReader
extends AbstractReader
    A component that reads the binlog of a MySQL server, and records any schema changes in MySqlSchema.
    Author:
    Randall Hauch

    • Field Detail

      • INITIAL_POLL_PERIOD_IN_MILLIS

        private static final long INITIAL_POLL_PERIOD_IN_MILLIS

      • MAX_POLL_PERIOD_IN_MILLIS

        private static final long MAX_POLL_PERIOD_IN_MILLIS

      • recordSchemaChangesInSourceRecords

        private final boolean recordSchemaChangesInSourceRecords

      • eventHandlers

        private final EnumMap<com.github.shyiko.mysql.binlog.event.EventType,​BlockingConsumer<com.github.shyiko.mysql.binlog.event.Event>> eventHandlers

      • client

        private final com.github.shyiko.mysql.binlog.BinaryLogClient client

      • clock

        private final Clock clock

      • startingRowNumber

        private int startingRowNumber

      • recordCounter

        private long recordCounter

      • previousOutputMillis

        private long previousOutputMillis

      • initialEventsToSkip

        private long initialEventsToSkip

      • skipEvent

        private boolean skipEvent

      • ignoreDmlEventByGtidSource

        private boolean ignoreDmlEventByGtidSource

      • totalRecordCounter

        private final AtomicLong totalRecordCounter

      • lastOffset

        private volatile Map<String,​?> lastOffset

      • gtidSet

        private com.github.shyiko.mysql.binlog.GtidSet gtidSet

      • binaryLogClientThreads

        private final Map<String,​Thread> binaryLogClientThreads

    • Constructor Detail

      • BinlogReader

        public BinlogReader​(String name,
                    MySqlTaskContext context,
                    HaltingPredicate acceptAndContinue)
        Create a binlog reader.
        Parameters:
        name - the name of this reader; may not be null
        context - the task context in which this reader is running; may not be null
        acceptAndContinue - see AbstractReader#AbstractReader(String, MySqlTaskContext, Predicate)

      • BinlogReader

        public BinlogReader​(String name,
                    MySqlTaskContext context,
                    HaltingPredicate acceptAndContinue,
                    long serverId)
        Create a binlog reader.
        Parameters:
        name - the name of this reader; may not be null
        context - the task context in which this reader is running; may not be null
        acceptAndContinue - see AbstractReader#AbstractReader(String, MySqlTaskContext, Predicate)
        serverId - the server id to use for the BinaryLogClient

    • Method Detail

      • doStart

        protected void doStart()
        Description copied from class: AbstractReader
        The reader has been requested to start, so initialize any un-initialized resources required by the reader.
        Specified by:
        doStart in class AbstractReader

      • getLastOffset

        public Map<String,​?> getLastOffset()
        Returns:
        a copy of the last offset of this reader, or null if this reader has not completed a poll.

      • pollComplete

        protected void pollComplete​(List<org.apache.kafka.connect.source.SourceRecord> batch)
        Description copied from class: AbstractReader
        Method called when AbstractReader.poll() completes sending a non-zero-sized batch of records.
        Overrides:
        pollComplete in class AbstractReader
        Parameters:
        batch - the batch of records being recorded

      • logEvent

        protected void logEvent​(com.github.shyiko.mysql.binlog.event.Event event)

      • onEvent

        protected void onEvent​(com.github.shyiko.mysql.binlog.event.Event event)

      • ignoreEvent

        protected void ignoreEvent​(com.github.shyiko.mysql.binlog.event.Event event)

      • handleEvent

        protected void handleEvent​(com.github.shyiko.mysql.binlog.event.Event event)

      • unwrapData

        protected <T extends com.github.shyiko.mysql.binlog.event.EventData> T unwrapData​(com.github.shyiko.mysql.binlog.event.Event event)

      • handleServerStop

        protected void handleServerStop​(com.github.shyiko.mysql.binlog.event.Event event)
        Handle the supplied event that signals that mysqld has stopped.
        Parameters:
        event - the server stopped event to be processed; may not be null

      • handleServerHeartbeat

        protected void handleServerHeartbeat​(com.github.shyiko.mysql.binlog.event.Event event)
        Handle the supplied event that is sent by a primary to a replica to let the replica know that the primary is still alive. Not written to a binary log.
        Parameters:
        event - the server stopped event to be processed; may not be null

      • handleServerIncident

        protected void handleServerIncident​(com.github.shyiko.mysql.binlog.event.Event event)
        Handle the supplied event that signals that an out of the ordinary event that occurred on the master. It notifies the replica that something happened on the primary that might cause data to be in an inconsistent state.
        Parameters:
        event - the server stopped event to be processed; may not be null

      • handleRotateLogsEvent

        protected void handleRotateLogsEvent​(com.github.shyiko.mysql.binlog.event.Event event)
        Handle the supplied event with a RotateEventData that signals the logs are being rotated. This means that either the server was restarted, or the binlog has transitioned to a new file. In either case, subsequent table numbers will be different than those seen to this point, so we need to discard the cache of record makers.
        Parameters:
        event - the database change data event to be processed; may not be null

      • handleGtidEvent

        protected void handleGtidEvent​(com.github.shyiko.mysql.binlog.event.Event event)
        Handle the supplied event with a GtidEventData that signals the beginning of a GTID transaction. We don't yet know whether this transaction contains any events we're interested in, but we have to record it so that we know the position of this event and know we've processed the binlog to this point.

        Note that this captures the current GTID and complete GTID set, regardless of whether the connector is filtering the GTID set upon connection. We do this because we actually want to capture all GTID set values found in the binlog, whether or not we process them. However, only when we connect do we actually want to pass to MySQL only those GTID ranges that are applicable per the configuration.

        Parameters:
        event - the GTID event to be processed; may not be null

      • handleRowsQuery

        protected void handleRowsQuery​(com.github.shyiko.mysql.binlog.event.Event event)
        Handle the supplied event with an RowsQueryEventData by recording the original SQL query that generated the event.
        Parameters:
        event - the database change data event to be processed; may not be null

      • handleQueryEvent

        protected void handleQueryEvent​(com.github.shyiko.mysql.binlog.event.Event event)
                         throws InterruptedException
        Handle the supplied event with an QueryEventData by possibly recording the DDL statements as changes in the MySQL schemas.
        Parameters:
        event - the database change data event to be processed; may not be null
        Throws:
        InterruptedException - if this thread is interrupted while recording the DDL statements

      • handleTransactionCompletion

        private void handleTransactionCompletion​(com.github.shyiko.mysql.binlog.event.Event event)

      • handleUpdateTableMetadata

        protected void handleUpdateTableMetadata​(com.github.shyiko.mysql.binlog.event.Event event)
        Handle a change in the table metadata.

        This method should be called whenever we consume a TABLE_MAP event, and every transaction in the log should include one of these for each table affected by the transaction. Each table map event includes a monotonically-increasing numeric identifier, and this identifier is used within subsequent events within the same transaction. This table identifier can change when:

        1. the table structure is modified (e.g., via an ALTER TABLE ... command); or
        2. MySQL rotates to a new binary log file, even if the table structure does not change.
        Parameters:
        event - the update event; never null

      • informAboutUnknownTableIfRequired

        private void informAboutUnknownTableIfRequired​(com.github.shyiko.mysql.binlog.event.Event event,
                                               TableId tableId,
                                               String typeToLog)
        If we receive an event for a table that is monitored but whose metadata we don't know, either ignore that event or raise a warning or error as per the MySqlConnectorConfig.INCONSISTENT_SCHEMA_HANDLING_MODE configuration.

      • handleInsert

        protected void handleInsert​(com.github.shyiko.mysql.binlog.event.Event event)
                     throws InterruptedException
        Generate source records for the supplied event with an WriteRowsEventData.
        Parameters:
        event - the database change data event to be processed; may not be null
        Throws:
        InterruptedException - if this thread is interrupted while blocking

      • handleUpdate

        protected void handleUpdate​(com.github.shyiko.mysql.binlog.event.Event event)
                     throws InterruptedException
        Generate source records for the supplied event with an UpdateRowsEventData.
        Parameters:
        event - the database change data event to be processed; may not be null
        Throws:
        InterruptedException - if this thread is interrupted while blocking

      • handleDelete

        protected void handleDelete​(com.github.shyiko.mysql.binlog.event.Event event)
                     throws InterruptedException
        Generate source records for the supplied event with an DeleteRowsEventData.
        Parameters:
        event - the database change data event to be processed; may not be null
        Throws:
        InterruptedException - if this thread is interrupted while blocking

      • viewChange

        protected void viewChange​(com.github.shyiko.mysql.binlog.event.Event event)
                   throws InterruptedException
        Handle a EventType.VIEW_CHANGE event.
        Parameters:
        event - the database change data event to be processed; may not be null
        Throws:
        InterruptedException - if this thread is interrupted while blocking

      • prepareTransaction

        protected void prepareTransaction​(com.github.shyiko.mysql.binlog.event.Event event)
                           throws InterruptedException
        Handle a EventType.XA_PREPARE event.
        Parameters:
        event - the database change data event to be processed; may not be null
        Throws:
        InterruptedException - if this thread is interrupted while blocking

      • logReaderState

        private void logReaderState()

      • logReaderState

        private void logReaderState​(org.slf4j.event.Level severity)

      • getBinlogClient

        protected com.github.shyiko.mysql.binlog.BinaryLogClient getBinlogClient()

      • getBinlogSslSocketFactory

        private com.github.shyiko.mysql.binlog.network.SSLSocketFactory getBinlogSslSocketFactory​(MySqlJdbcContext connectionContext)