Package com.privalia.qa.utils
Class KafkaUtils
- java.lang.Object
-
- com.privalia.qa.utils.KafkaUtils
-
public class KafkaUtils extends Object
Generic utilities for operations over Kafka.
-
-
Field Summary
Fields Modifier and Type Field Description Map<String,org.apache.avro.generic.GenericRecord>
avroRecords
-
Constructor Summary
Constructors Constructor Description KafkaUtils()
Generic contructor of KafkaUtils.
-
Method Summary
All Methods Instance Methods Concrete Methods Deprecated Methods Modifier and Type Method Description void
connect()
Connect to Kafka.void
createGenericRecord(String key, String json, String schema)
given a json representation of the data, creates a generic record with the given schemavoid
createGenericRecord(String key, Map<String,String> propertyList, String schema)
Creates aGenericRecord
to be sent through kafka using avro serializers.boolean
createTopic(String topicName)
Create a Kafka topic.boolean
deleteTopic(String topicName)
Delete a Kafka topic.Map<String,org.apache.avro.generic.GenericRecord>
getAvroRecords()
int
getPartitions(String topicName)
Returns the number of partitions for the given topicProperties
getPropsConsumer()
String
getSchemaFromRegistry(String subject, String version)
Fetch version of the schema registered under the specified subject in the registryString
getSchemaRegistryUrl()
kafka.utils.ZkUtils
getZkUtils()
List<String>
listTopics()
List all Kafka topics.void
modifyConsumerProperties(String key, String value)
Modify a single property of the producervoid
modifyProducerProperties(String key, String value)
Modify a single property of the producervoid
modifyTopicPartitioning(String topicName, int numPartitions)
Modify number of partition of a Kafka topic.Map<Object,Object>
readTopicFromBeginning(String topic)
Fetch messages from the given partition using poll().<K,V>
MapreadTopicFromBeginning(String topic, K keyClass, V valueClass)
okhttp3.Response
registerNewSchema(String subject, String schema)
Publish a new version of the schema under the given subjectvoid
sendAndConfirmMessage(String message, String key, String topicName, long timeoutSeconds)
Send (and confirm) a message.void
sendMessage(String message, String topicName)
Deprecated.void
setSchemaRegistryUrl(String host)
Set remote schema registry url and port for all future requestsvoid
setZkHost(String host, String port, String zkPath)
-
-
-
Method Detail
-
getPropsConsumer
public Properties getPropsConsumer()
-
connect
public void connect()
Connect to Kafka.
-
getZkUtils
public kafka.utils.ZkUtils getZkUtils()
-
getPartitions
public int getPartitions(String topicName)
Returns the number of partitions for the given topic- Parameters:
topicName
- Name of the topic- Returns:
- Number of partitions for the topic
-
createTopic
public boolean createTopic(String topicName) throws kafka.common.KafkaException
Create a Kafka topic.- Parameters:
topicName
- name of topic.- Returns:
- true if the topic has been created and false if the topic has not been created.
- Throws:
kafka.common.KafkaException
- the kafka exception
-
deleteTopic
public boolean deleteTopic(String topicName) throws kafka.common.KafkaException, kafka.common.TopicAlreadyMarkedForDeletionException
Delete a Kafka topic.- Parameters:
topicName
- name of topic.- Returns:
- true if the topic has been deleted and false if the topic has not been deleted.
- Throws:
kafka.common.KafkaException
- the kafka exceptionkafka.common.TopicAlreadyMarkedForDeletionException
- the topic already marked for deletion exception
-
listTopics
public List<String> listTopics() throws kafka.common.KafkaException
List all Kafka topics.- Returns:
- list of topics.
- Throws:
kafka.common.KafkaException
- the kafka exception
-
modifyTopicPartitioning
public void modifyTopicPartitioning(String topicName, int numPartitions) throws kafka.common.KafkaException
Modify number of partition of a Kafka topic.- Parameters:
topicName
- name of topic.numPartitions
- the num partitions- Throws:
kafka.common.KafkaException
- the kafka exception
-
sendMessage
@Deprecated public void sendMessage(String message, String topicName)
Deprecated.Send a message to a Kafka topic.- Parameters:
message
- the messagetopicName
- name of topic.
-
sendAndConfirmMessage
public void sendAndConfirmMessage(String message, String key, String topicName, long timeoutSeconds) throws InterruptedException, ExecutionException, TimeoutException
Send (and confirm) a message. The system automatically infers the type of the key and value based on the producer properties for key and value serializer types- Parameters:
message
- The message to be sentkey
- the keytopicName
- name of topictimeoutSeconds
- Number of seconds to wait for acknowledgement by Kafka- Throws:
InterruptedException
- InterruptedExceptionExecutionException
- ExecutionExceptionTimeoutException
- TimeoutException
-
readTopicFromBeginning
public Map<Object,Object> readTopicFromBeginning(String topic)
Fetch messages from the given partition using poll(). Since there is no warranty that the pool will return messages, the function is enclosed in a loop for 5 seconds- Parameters:
topic
- Name of the topic from which retrieve messages- Returns:
- List of messages in the topic
-
readTopicFromBeginning
public <K,V> Map readTopicFromBeginning(String topic, K keyClass, V valueClass)
-
setSchemaRegistryUrl
public void setSchemaRegistryUrl(String host)
Set remote schema registry url and port for all future requests- Parameters:
host
- host (defaults to http://0.0.0.0:8081)
-
getSchemaRegistryUrl
public String getSchemaRegistryUrl()
-
registerNewSchema
public okhttp3.Response registerNewSchema(String subject, String schema) throws IOException
Publish a new version of the schema under the given subject- Parameters:
subject
- Name of the subjectschema
- Schema object as string- Returns:
- response object from the schema server
- Throws:
IOException
- IOException
-
getSchemaFromRegistry
public String getSchemaFromRegistry(String subject, String version) throws IOException
Fetch version of the schema registered under the specified subject in the registry- Parameters:
subject
- Subject nameversion
- Version of the schema to fetch- Returns:
- Json encoded string of the schema
- Throws:
IOException
- IOException
-
modifyProducerProperties
public void modifyProducerProperties(String key, String value)
Modify a single property of the producer- Parameters:
key
- Property namevalue
- Property new value
-
modifyConsumerProperties
public void modifyConsumerProperties(String key, String value)
Modify a single property of the producer- Parameters:
key
- Property namevalue
- Property new value
-
createGenericRecord
public void createGenericRecord(String key, String json, String schema) throws IOException
given a json representation of the data, creates a generic record with the given schema- Parameters:
key
- Name of the generic recordjson
- Json string with dataschema
- Schema to be used to serialize the object- Throws:
IOException
- IOException
-
createGenericRecord
public void createGenericRecord(String key, Map<String,String> propertyList, String schema)
Creates aGenericRecord
to be sent through kafka using avro serializers. The message is stored in an internal map using the specified key for later use.- Parameters:
key
- Name of the generic recordpropertyList
- List of properties and valuesschema
- Schema to be used to serialize the object
-
-