kafka010

Type Members

1. trait CanCommitOffsets extends AnyRef

   

   :: Experimental :: Represents any object that can commit a collection of OffsetRanges.

   :: Experimental :: Represents any object that can commit a collection of OffsetRanges. The direct Kafka DStream implements this interface (see KafkaUtils.createDirectStream).

   val stream = KafkaUtils.createDirectStream(...)
     ...
   stream.asInstanceOf[CanCommitOffsets].commitAsync(offsets, new OffsetCommitCallback() {
     def onComplete(m: java.util.Map[TopicPartition, OffsetAndMetadata], e: Exception) {
        if (null != e) {
           // error
        } else {
         // success
       }
     }
   })

   Annotations

   @Experimental()

2. abstract class ConsumerStrategy[K, V] extends AnyRef

   

   :: Experimental :: Choice of how to create and configure underlying Kafka Consumers on driver and executors.

   :: Experimental :: Choice of how to create and configure underlying Kafka Consumers on driver and executors. See ConsumerStrategies to obtain instances. Kafka 0.10 consumers can require additional, sometimes complex, setup after object instantiation. This interface encapsulates that process, and allows it to be checkpointed.

   K

   type of Kafka message key

   V

   type of Kafka message value

   Annotations

   @Experimental()

3. trait HasOffsetRanges extends AnyRef

   

   Represents any object that has a collection of OffsetRanges.

   Represents any object that has a collection of OffsetRanges. This can be used to access the offset ranges in RDDs generated by the direct Kafka DStream (see KafkaUtils.createDirectStream).

   KafkaUtils.createDirectStream(...).foreachRDD { rdd =>
      val offsetRanges = rdd.asInstanceOf[HasOffsetRanges].offsetRanges
      ...
   }

4. sealed abstract class LocationStrategy extends AnyRef

   

   :: Experimental :: Choice of how to schedule consumers for a given TopicPartition on an executor.

   :: Experimental :: Choice of how to schedule consumers for a given TopicPartition on an executor. See LocationStrategies to obtain instances. Kafka 0.10 consumers prefetch messages, so it's important for performance to keep cached consumers on appropriate executors, not recreate them for every partition. Choice of location is only a preference, not an absolute; partitions may be scheduled elsewhere.

   Annotations

   @Experimental()

5. final class OffsetRange extends Serializable

   

   Represents a range of offsets from a single Kafka TopicPartition.

   Represents a range of offsets from a single Kafka TopicPartition. Instances of this class can be created with OffsetRange.create().