Package com.mongodb.kafka.connect.source

Class MongoSourceTask

java.lang.Object
org.apache.kafka.connect.source.SourceTask
com.mongodb.kafka.connect.source.MongoSourceTask
All Implemented Interfaces:
org.apache.kafka.connect.connector.Task
public final class MongoSourceTask extends org.apache.kafka.connect.source.SourceTask
A Kafka Connect source task that uses change streams to broadcast changes to the collection, database or client.

Copy Existing Data

If configured the connector will copy the existing data from the collection, database or client. All namespaces that exist at the time of starting the task will be broadcast onto the topic as insert operations. Only when all the data from all namespaces have been broadcast will the change stream cursor start broadcasting new changes. The logic for copying existing data is as follows:

  1. Get the latest resumeToken from MongoDB
  2. Create insert events for all configured namespaces using multiple threads. This step is completed only after all collections are successfully copied.
  3. Start a change stream cursor from the saved resumeToken

It should be noted that the reading of all the data during the copy and then the subsequent change stream events may produce duplicated events. During the copy, clients can make changes to the data in MongoDB, which may be represented both by the copying process and the change stream. However, as the change stream events are idempotent the changes can be applied so that the data is eventually consistent.

It should also be noted renaming a collection during the copying process is not supported.

Restarts

Restarting the connector during the copying phase, will cause the whole copy process to restart. Restarts after the copying process will resume from the last seen resumeToken.

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    static final String
    ID_FIELD
     

    Fields inherited from class org.apache.kafka.connect.source.SourceTask

    context

  • Constructor Summary

    Constructors
    Constructor
    Description
    MongoSourceTask()
     

  • Method Summary

    Modifier and Type
    Method
    Description
    void
    commitRecord(org.apache.kafka.connect.source.SourceRecord record, org.apache.kafka.clients.producer.RecordMetadata metadata)
     
    List<org.apache.kafka.connect.source.SourceRecord>
    poll()
     
    void
    start(Map<String,String> props)
     
    void
    stop()
     
    String
    version()
     

    Methods inherited from class org.apache.kafka.connect.source.SourceTask

    commit, commitRecord, initialize

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Field Details

  • Constructor Details

    • MongoSourceTask

      public MongoSourceTask()

  • Method Details

    • version

      public String version()

    • start

      public void start(Map<String,String> props)
      Specified by:
      start in interface org.apache.kafka.connect.connector.Task
      Specified by:
      start in class org.apache.kafka.connect.source.SourceTask

    • poll

      public List<org.apache.kafka.connect.source.SourceRecord> poll()
      Specified by:
      poll in class org.apache.kafka.connect.source.SourceTask

    • stop

      public void stop()
      Specified by:
      stop in interface org.apache.kafka.connect.connector.Task
      Specified by:
      stop in class org.apache.kafka.connect.source.SourceTask

    • commitRecord

      public void commitRecord(org.apache.kafka.connect.source.SourceRecord record, org.apache.kafka.clients.producer.RecordMetadata metadata)
      Overrides:
      commitRecord in class org.apache.kafka.connect.source.SourceTask