Package no.mnemonic.messaging.documentchannel.kafka

Class KafkaDocumentSource<T>

java.lang.Object
no.mnemonic.messaging.documentchannel.kafka.KafkaDocumentSource<T>
Type Parameters:
T - document type
All Implemented Interfaces:
AutoCloseable, no.mnemonic.commons.metrics.MetricAspect, DocumentSource<T>
public class KafkaDocumentSource<T> extends Object implements DocumentSource<T>, no.mnemonic.commons.metrics.MetricAspect
Kafka version of a document channel source. The source is configured with a kafka cluster, topic and groupID. Multiple sources may be configured to the same topic and groupID, which will load-balance the incoming documents between the active sources.

Multiple sources with different groupIDs will receive individual copies of each document on the topic.

  • Method Details

    • seek

      public void seek(String cursor) throws KafkaInvalidSeekException
      Seek to cursor
      Parameters:
      cursor - A string representation of a cursor as returned in a KafkaDocument. A null cursor will not change the offset of the consumer.
      Throws:
      KafkaInvalidSeekException - if the cursor is invalid, or points to a timestamp which is not reachable

    • getMetrics

      public no.mnemonic.commons.metrics.Metrics getMetrics() throws no.mnemonic.commons.metrics.MetricException
      Specified by:
      getMetrics in interface no.mnemonic.commons.metrics.MetricAspect
      Throws:
      no.mnemonic.commons.metrics.MetricException

    • waitForAssignment

      public boolean waitForAssignment(Duration maxWait) throws InterruptedException
      Let client wait for partition assignment to complete. Method is reentrant, so calling this AFTER assignment is already done will immediately return true.
      Parameters:
      maxWait - max duration to wait before returning false
      Returns:
      true if partitions are assigned, or false if no partitions were assigned before the timeout
      Throws:
      InterruptedException

    • getCursor

      public String getCursor() throws InterruptedException
      Fetch the current cursor of this source. The method will block up to maxAssignmentWait milliseconds waiting for partitions assignment.

      Note; this cursor is not thread safe; if there is a separate consumer thread polling, you should rather use the cursor of the KafkaDocument.

      Returns:
      the String cursor for the current position of this source
      Throws:
      InterruptedException - if interrupted while waiting for Kafka to assign partitions
      See Also:

    • createDocumentSubscription

      public DocumentChannelSubscription createDocumentSubscription(DocumentChannelListener<T> listener)
      Specified by:
      createDocumentSubscription in interface DocumentSource<T>

    • poll

      public KafkaDocumentBatch<T> poll(Duration duration)
      Specified by:
      poll in interface DocumentSource<T>

    • close

      public void close()
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface DocumentSource<T>

    • getKafkaConsumer

      public org.apache.kafka.clients.consumer.KafkaConsumer<String,T> getKafkaConsumer()

    • setMaxAssignmentWait

      public KafkaDocumentSource<T> setMaxAssignmentWait(long maxAssignmentWait)
      This parameter only applies to getCursor(), and should probably never need to be changed.
      Parameters:
      maxAssignmentWait - max millis to wait for assignments.

    • builder

      public static <T> KafkaDocumentSource.Builder<T> builder()