com.github.garyaiki.dendrites.kafka.stream
KafkaSink
Related Docs:
object KafkaSink
| package stream
com.github.garyaiki.dendrites.kafka.stream
sets up Supervision with exponential backoff
sets up Supervision with exponential backoff
extends KafkaProducer with key, value, topic, and backoff fields
implicit LoggingAdapter
Sink stage that writes to Kafka
KafkaSink is initialized with a wrapped KafkaProducer. It includes topic, key, Key, Value types for the topic, and retry backoff. KafkaProducer is heavy weight, multi-threaded, may serve other topics and long lived. If a RetryableException is thrown while writing it's retried with backoff If a write throws a Stop Exception the stage fails and the stream is stopped.
KafkaProducer's send returns a Java/Guava ListenableFuture, not a nice Scala Future. The ListenableFuture passes either a RecordMetadata or an Exception to a Callback.
A RecordMetadata is passed on success and an Akka Stream AsynCallback is called that pulls a record from upstream. If an exception was returned a different AsynCallback is called that handles Kafka RetriableExceptions and Stop Exceptions.
RetriableException means the error may not recur, so retry the message. Retries use exponential backoff until success or the configured maxBackoff is reached and then the stream is shutdown.
ProducerRecord key
Type of serialized object received from stream and ProducerRecord value