Package com.privalia.qa.utils

Class KafkaUtils

  • public class KafkaUtils
extends Object
    Generic utilities for operations over Kafka.

    • Field Detail

      • avroRecords

        public Map<String,​org.apache.avro.generic.GenericRecord> avroRecords

    • Constructor Detail

      • KafkaUtils

        public KafkaUtils()
        Generic contructor of KafkaUtils.

    • Method Detail

      • getPropsConsumer

        public Properties getPropsConsumer()

      • connect

        public void connect()
        Connect to Kafka.

      • getAvroRecords

        public Map<String,​org.apache.avro.generic.GenericRecord> getAvroRecords()

      • getZkUtils

        public kafka.utils.ZkUtils getZkUtils()

      • getPartitions

        public int getPartitions​(String topicName)
        Returns the number of partitions for the given topic
        Parameters:
        topicName - Name of the topic
        Returns:
        Number of partitions for the topic

      • createTopic

        public boolean createTopic​(String topicName)
                    throws kafka.common.KafkaException
        Create a Kafka topic.
        Parameters:
        topicName - name of topic.
        Returns:
        true if the topic has been created and false if the topic has not been created.
        Throws:
        kafka.common.KafkaException - the kafka exception

      • deleteTopic

        public boolean deleteTopic​(String topicName)
                    throws kafka.common.KafkaException,
                           kafka.common.TopicAlreadyMarkedForDeletionException
        Delete a Kafka topic.
        Parameters:
        topicName - name of topic.
        Returns:
        true if the topic has been deleted and false if the topic has not been deleted.
        Throws:
        kafka.common.KafkaException - the kafka exception
        kafka.common.TopicAlreadyMarkedForDeletionException - the topic already marked for deletion exception

      • listTopics

        public List<String> listTopics()
                        throws kafka.common.KafkaException
        List all Kafka topics.
        Returns:
        list of topics.
        Throws:
        kafka.common.KafkaException - the kafka exception

      • modifyTopicPartitioning

        public void modifyTopicPartitioning​(String topicName,
                                    int numPartitions)
                             throws kafka.common.KafkaException
        Modify number of partition of a Kafka topic.
        Parameters:
        topicName - name of topic.
        numPartitions - the num partitions
        Throws:
        kafka.common.KafkaException - the kafka exception

      • sendMessage

        @Deprecated
public void sendMessage​(String message,
                        String topicName)
        Deprecated.
        Send a message to a Kafka topic.
        Parameters:
        message - the message
        topicName - name of topic.

      • sendAndConfirmMessage

        public void sendAndConfirmMessage​(String message,
                                  String key,
                                  String topicName,
                                  long timeoutSeconds)
                           throws InterruptedException,
                                  ExecutionException,
                                  TimeoutException
        Send (and confirm) a message. The system automatically infers the type of the key and value based on the producer properties for key and value serializer types
        Parameters:
        message - The message to be sent
        key - the key
        topicName - name of topic
        timeoutSeconds - Number of seconds to wait for acknowledgement by Kafka
        Throws:
        InterruptedException - InterruptedException
        ExecutionException - ExecutionException
        TimeoutException - TimeoutException

      • readTopicFromBeginning

        public Map<Object,​Object> readTopicFromBeginning​(String topic)
        Fetch messages from the given partition using poll(). Since there is no warranty that the pool will return messages, the function is enclosed in a loop for 5 seconds
        Parameters:
        topic - Name of the topic from which retrieve messages
        Returns:
        List of messages in the topic

      • readTopicFromBeginning

        public <K,​V> Map readTopicFromBeginning​(String topic,
                                              K keyClass,
                                              V valueClass)

      • setSchemaRegistryUrl

        public void setSchemaRegistryUrl​(String host)
        Set remote schema registry url and port for all future requests
        Parameters:
        host - host (defaults to http://0.0.0.0:8081)

      • getSchemaRegistryUrl

        public String getSchemaRegistryUrl()

      • registerNewSchema

        public okhttp3.Response registerNewSchema​(String subject,
                                          String schema)
                                   throws IOException
        Publish a new version of the schema under the given subject
        Parameters:
        subject - Name of the subject
        schema - Schema object as string
        Returns:
        response object from the schema server
        Throws:
        IOException - IOException

      • getSchemaFromRegistry

        public String getSchemaFromRegistry​(String subject,
                                    String version)
                             throws IOException
        Fetch version of the schema registered under the specified subject in the registry
        Parameters:
        subject - Subject name
        version - Version of the schema to fetch
        Returns:
        Json encoded string of the schema
        Throws:
        IOException - IOException

      • modifyProducerProperties

        public void modifyProducerProperties​(String key,
                                     String value)
        Modify a single property of the producer
        Parameters:
        key - Property name
        value - Property new value

      • modifyConsumerProperties

        public void modifyConsumerProperties​(String key,
                                     String value)
        Modify a single property of the producer
        Parameters:
        key - Property name
        value - Property new value

      • createGenericRecord

        public void createGenericRecord​(String key,
                                String json,
                                String schema)
                         throws IOException
        given a json representation of the data, creates a generic record with the given schema
        Parameters:
        key - Name of the generic record
        json - Json string with data
        schema - Schema to be used to serialize the object
        Throws:
        IOException - IOException

      • createGenericRecord

        public void createGenericRecord​(String key,
                                Map<String,​String> propertyList,
                                String schema)
        Creates a GenericRecord to be sent through kafka using avro serializers. The message is stored in an internal map using the specified key for later use.
        Parameters:
        key - Name of the generic record
        propertyList - List of properties and values
        schema - Schema to be used to serialize the object