Package com.github.ddth.queue.impl

Class KafkaQueue<ID,​DATA>

  • All Implemented Interfaces:
    IQueue<ID,​DATA>, AutoCloseable
    Direct Known Subclasses:
    BaseUniversalKafkaQueue
    public abstract class KafkaQueue<ID,​DATA>
extends AbstractQueue<ID,​DATA>
    (Experimental) Kafka implementation of IQueue.
    • Queue-size support: no
    • Ephemeral storage support: no
    Since:
    0.3.2
    Author:
    Thanh Ba Nguyen

    • Field Detail

      • DEFAULT_PRODUCER_TYPE

        public static final com.github.ddth.kafka.KafkaClient.ProducerType DEFAULT_PRODUCER_TYPE

    • Constructor Detail

      • KafkaQueue

        public KafkaQueue()

    • Method Detail

      • isSendAsync

        public boolean isSendAsync()
        Should messages sent to Kafka asynchronously (default true)?
        Returns:
        Since:
        0.5.0

      • setSendAsync

        public KafkaQueue<ID,​DATA> setSendAsync​(boolean value)
        Should messages sent to Kafka asynchronously (default true)?
        Parameters:
        value -
        Returns:

      • getProducerType

        public com.github.ddth.kafka.KafkaClient.ProducerType getProducerType()
        Kafka's producer type, used to send messages (default KafkaClient.ProducerType.LEADER_ACK).
        Returns:

      • setProducerType

        public KafkaQueue<ID,​DATA> setProducerType​(com.github.ddth.kafka.KafkaClient.ProducerType producerType)
        Kafka's producer type, used to send messages (default KafkaClient.ProducerType.LEADER_ACK).
        Parameters:
        producerType -
        Returns:

      • getKafkaBootstrapServers

        public String getKafkaBootstrapServers()
        Kafka bootstrap server list (format host1:9092,host2:port2,host3:port3).
        Returns:
        Since:
        0.4.0

      • setKafkaBootstrapServers

        public KafkaQueue<ID,​DATA> setKafkaBootstrapServers​(String kafkaBootstrapServers)
        Kafka bootstrap server list (format host1:9092,host2:port2,host3:port3).
        Parameters:
        kafkaBootstrapServers -
        Returns:
        Since:
        0.4.0

      • getKafkaProducerProperties

        public Properties getKafkaProducerProperties()
        Custom configuration properties for Kafka producer.
        Returns:
        Since:
        0.4.0

      • setKafkaProducerProperties

        public KafkaQueue<ID,​DATA> setKafkaProducerProperties​(Properties kafkaProducerConfigs)
        Custom configuration properties for Kafka producer.
        Parameters:
        kafkaProducerConfigs -
        Returns:
        Since:
        0.4.0

      • getKafkaConsumerProperties

        public Properties getKafkaConsumerProperties()
        Custom configuration properties for Kafka consumer.
        Returns:
        Since:
        0.4.0

      • setKafkaConsumerProperties

        public KafkaQueue<ID,​DATA> setKafkaConsumerProperties​(Properties kafkaConsumerConfigs)
        Custom configuration properties for Kafka consumer.
        Parameters:
        kafkaConsumerConfigs -
        Returns:
        Since:
        0.4.0

      • getTopicName

        public String getTopicName()
        Name of Kafka topic to store queue messages.
        Returns:

      • setTopicName

        public KafkaQueue<ID,​DATA> setTopicName​(String topicName)
        Name of Kafka topic to store queue messages.
        Parameters:
        topicName -
        Returns:

      • getConsumerGroupId

        public String getConsumerGroupId()
        Kafka group-id to consume messages.
        Returns:

      • setConsumerGroupId

        public KafkaQueue<ID,​DATA> setConsumerGroupId​(String consumerGroupId)
        Kafka group-id to consume messages.
        Parameters:
        consumerGroupId -
        Returns:

      • getKafkaClient

        protected com.github.ddth.kafka.KafkaClient getKafkaClient()
        An external KafkaClient can be used. If not set, KafkaQueue will automatically create a KafkaClient for its own use.
        Returns:

      • setKafkaClient

        protected KafkaQueue<ID,​DATA> setKafkaClient​(com.github.ddth.kafka.KafkaClient kafkaClient,
                                                   boolean setMyOwnKafkaClient)
        An external KafkaClient can be used. If not set, KafkaQueue will automatically create a KafkaClient for its own use.
        Parameters:
        kafkaClient -
        setMyOwnKafkaClient -
        Returns:

      • setKafkaClient

        public KafkaQueue<ID,​DATA> setKafkaClient​(com.github.ddth.kafka.KafkaClient kafkaClient)
        An external KafkaClient can be used. If not set, KafkaQueue will automatically create a KafkaClient for its own use.
        Parameters:
        kafkaClient -
        Returns:

      • buildKafkaClient

        protected com.github.ddth.kafka.KafkaClient buildKafkaClient()

      • takeFromQueue

        protected IQueueMessage<ID,​DATA> takeFromQueue()
        Take a message from Kafka queue.
        Returns:
        Since:
        0.3.3

      • finish

        public void finish​(IQueueMessage<ID,​DATA> msg)
        Called when finish processing the message to cleanup ephemeral storage.

        Implementation flow:

        • Remove message from ephemeral storage.

        Note: ephemeral storage implementation is optional, depends on implementation.

      • take

        public IQueueMessage<ID,​DATA> take()
        Take a message out of queue.

        Implementation flow:

        • Read message from head of queue storage.
        • Write message to ephemeral storage.
        • Remove message from queue storage.

        Note: ephemeral storage implementation is optional, depends on implementation.

        Returns:

      • getOrphanMessages

        public Collection<IQueueMessage<ID,​DATA>> getOrphanMessages​(long thresholdTimestampMs)
        Get all orphan messages (messages that were left in ephemeral storage for a long time).

        This method throws QueueException.OperationNotSupported

        Parameters:
        thresholdTimestampMs - message is orphan if message's timestampMillis + thresholdTimestampMs < now . Which means getOrphanMessages(10000) will return orphan messages that have stayed in ephemeral storage for more than 10000 milliseconds.
        Returns:
        null or empty collection if there is no orphan message

      • queueSize

        public int queueSize()
        Get number of items currently in queue storage.
        Returns:
        negative number if queue size can not be queried

      • ephemeralSize

        public int ephemeralSize()
        Get number of items currently in ephemeral storage.

        Note: ephemeral storage implementation is optional, depends on implementation.

        Returns:
        negative number if ephemeral size can not be queried