Package io.debezium.connector.base
Class ChangeEventQueue<T>
- java.lang.Object
-
- io.debezium.connector.base.ChangeEventQueue<T>
-
- Type Parameters:
T
- the type of events in this queue. UsuallySourceRecord
is used, but in cases where additional metadata must be passed from producers to the consumer, a custom type wrapping source records may be used.
- All Implemented Interfaces:
ChangeEventQueueMetrics
@ThreadSafe public class ChangeEventQueue<T> extends Object implements ChangeEventQueueMetrics
A queue which serves as handover point between producer threads (e.g. MySQL's binlog reader thread) and the Kafka Connect polling loop.The queue is configurable in different aspects, e.g. its maximum size and the time to sleep (block) between two subsequent poll calls. See the
ChangeEventQueue.Builder
for the different options. The queue applies back-pressure semantics, i.e. if it holds the maximum number of elements, subsequent calls toenqueue(Object)
will block until elements have been removed from the queue.If an exception occurs on the producer side, the producer should make that exception known by calling
producerException(RuntimeException)
before stopping its operation. Upon the next call topoll()
, that exception will be raised, causing Kafka Connect to stop the connector and mark it asFAILED
.- Author:
- Gunnar Morling
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description static class
ChangeEventQueue.Builder<T>
-
Field Summary
Fields Modifier and Type Field Description private T
bufferedEvent
private boolean
buffering
private long
currentQueueSizeInBytes
private static org.slf4j.Logger
LOGGER
private Supplier<LoggingContext.PreviousContext>
loggingContextSupplier
private int
maxBatchSize
private int
maxQueueSize
private long
maxQueueSizeInBytes
private Duration
pollInterval
private RuntimeException
producerException
private Queue<T>
queue
private Queue<Long>
sizeInBytesQueue
-
Constructor Summary
Constructors Modifier Constructor Description private
ChangeEventQueue(Duration pollInterval, int maxQueueSize, int maxBatchSize, Supplier<LoggingContext.PreviousContext> loggingContextSupplier, long maxQueueSizeInBytes, boolean buffering)
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description long
currentQueueSizeInBytes()
void
disableBuffering()
Disable buffering for the queueprotected void
doEnqueue(T record)
private long
drainRecords(List<T> records, int maxElements)
void
enqueue(T record)
Enqueues a record so that it can be obtained viapoll()
.void
flushBuffer(Function<T,T> recordModifier)
Applies a function to the event and the buffer and adds it to the queue.long
maxQueueSizeInBytes()
List<T>
poll()
Returns the next batch of elements from this queue.void
producerException(RuntimeException producerException)
int
remainingCapacity()
private void
throwProducerExceptionIfPresent()
int
totalCapacity()
-
-
-
Field Detail
-
LOGGER
private static final org.slf4j.Logger LOGGER
-
pollInterval
private final Duration pollInterval
-
maxBatchSize
private final int maxBatchSize
-
maxQueueSize
private final int maxQueueSize
-
maxQueueSizeInBytes
private final long maxQueueSizeInBytes
-
loggingContextSupplier
private final Supplier<LoggingContext.PreviousContext> loggingContextSupplier
-
currentQueueSizeInBytes
private long currentQueueSizeInBytes
-
buffering
private boolean buffering
-
bufferedEvent
private T bufferedEvent
-
producerException
private volatile RuntimeException producerException
-
-
Constructor Detail
-
ChangeEventQueue
private ChangeEventQueue(Duration pollInterval, int maxQueueSize, int maxBatchSize, Supplier<LoggingContext.PreviousContext> loggingContextSupplier, long maxQueueSizeInBytes, boolean buffering)
-
-
Method Detail
-
enqueue
public void enqueue(T record) throws InterruptedException
Enqueues a record so that it can be obtained viapoll()
. This method will block if the queue is full.- Parameters:
record
- the record to be enqueued- Throws:
InterruptedException
- if this thread has been interrupted
-
flushBuffer
public void flushBuffer(Function<T,T> recordModifier) throws InterruptedException
Applies a function to the event and the buffer and adds it to the queue. Buffer is emptied.- Parameters:
recordModifier
-- Throws:
InterruptedException
-
disableBuffering
public void disableBuffering()
Disable buffering for the queue
-
doEnqueue
protected void doEnqueue(T record) throws InterruptedException
- Throws:
InterruptedException
-
poll
public List<T> poll() throws InterruptedException
Returns the next batch of elements from this queue. May be empty in case no elements have arrived in the maximum waiting time.- Throws:
InterruptedException
- if this thread has been interrupted while waiting for more elements to arrive
-
producerException
public void producerException(RuntimeException producerException)
-
throwProducerExceptionIfPresent
private void throwProducerExceptionIfPresent()
-
totalCapacity
public int totalCapacity()
- Specified by:
totalCapacity
in interfaceChangeEventQueueMetrics
-
remainingCapacity
public int remainingCapacity()
- Specified by:
remainingCapacity
in interfaceChangeEventQueueMetrics
-
maxQueueSizeInBytes
public long maxQueueSizeInBytes()
- Specified by:
maxQueueSizeInBytes
in interfaceChangeEventQueueMetrics
-
currentQueueSizeInBytes
public long currentQueueSizeInBytes()
- Specified by:
currentQueueSizeInBytes
in interfaceChangeEventQueueMetrics
-
-