kafka

Type Members

1. class KafkaSink extends DataSink

   

   kafka sink connectors that invokes org.apache.kafka.clients.producer.KafkaProducer to send messages to kafka queue

2. class KafkaSource extends TimeReplayableSource

   

   Kafka source connectors that pulls a batch of messages (kafka.consumer.emit.batch.size) from multiple Kafka TopicAndPartition in a round-robin way.

   Kafka source connectors that pulls a batch of messages (kafka.consumer.emit.batch.size) from multiple Kafka TopicAndPartition in a round-robin way.

   This is a TimeReplayableSource which is able to replay messages given a start time. Each kafka message is tagged with a timestamp by io.gearpump.streaming.transaction.api.MessageDecoder and the (offset, timestamp) mapping is stored to a OffsetStorage. On recovery, we could retrieve the previously stored offset from the OffsetStorage by timestamp and start to read from there.

   kafka message is wrapped into gearpump Message and further filtered by a TimeStampFilter such that obsolete messages are dropped.

3. class KafkaStorage extends OffsetStorage

   

   this stores offset-timestamp mapping to kafka

4. class KafkaStorageFactory extends OffsetStorageFactory

   

   factory that builds KafkaStorage