Package io.debezium.connector.base
Class ChangeEventQueue<T extends Sizeable>
java.lang.Object
io.debezium.connector.base.ChangeEventQueue<T>
- Type Parameters:
T
- the type of events in this queue. UsuallySizeable
is used, but in cases where additional metadata must be passed from producers to the consumer, a custom type extending source records may be used.
- All Implemented Interfaces:
ChangeEventQueueMetrics
@ThreadSafe
public class ChangeEventQueue<T extends Sizeable>
extends Object
implements ChangeEventQueueMetrics
A queue which serves as handover point between producer threads (e.g. MySQL's
binlog reader thread) and the Kafka Connect polling loop.
The queue is configurable in different aspects, e.g. its maximum size and the
time to sleep (block) between two subsequent poll calls. See the
ChangeEventQueue.Builder
for the different options. The queue applies back-pressure
semantics, i.e. if it holds the maximum number of elements, subsequent calls
to enqueue(T)
will block until elements have been removed from
the queue.
If an exception occurs on the producer side, the producer should make that
exception known by calling producerException(RuntimeException)
before stopping its
operation. Upon the next call to poll()
, that exception will be
raised, causing Kafka Connect to stop the connector and mark it as
FAILED
.
- Author:
- Gunnar Morling
-
Nested Class Summary
-
Field Summary
Modifier and TypeFieldDescriptionprivate final AtomicReference<T>
private boolean
private long
private final Condition
private final Condition
private final Lock
private static final org.slf4j.Logger
private final Supplier<LoggingContext.PreviousContext>
private final int
private final int
private final long
private final Duration
private RuntimeException
-
Constructor Summary
ModifierConstructorDescriptionprivate
ChangeEventQueue
(Duration pollInterval, int maxQueueSize, int maxBatchSize, Supplier<LoggingContext.PreviousContext> loggingContextSupplier, long maxQueueSizeInBytes, boolean buffering) -
Method Summary
Modifier and TypeMethodDescriptionlong
void
Disable buffering for the queueprotected void
private long
drainRecords
(List<T> records, int maxElements) void
Enqueues a record so that it can be obtained viapoll()
.void
flushBuffer
(Function<T, T> recordModifier) Applies a function to the event and the buffer and adds it to the queue.boolean
long
poll()
Returns the next batch of elements from this queue.void
producerException
(RuntimeException producerException) int
private void
int
-
Field Details
-
LOGGER
private static final org.slf4j.Logger LOGGER -
pollInterval
-
maxBatchSize
private final int maxBatchSize -
maxQueueSize
private final int maxQueueSize -
maxQueueSizeInBytes
private final long maxQueueSizeInBytes -
lock
-
isFull
-
isNotFull
-
queue
-
loggingContextSupplier
-
sizeInBytesQueue
-
currentQueueSizeInBytes
private long currentQueueSizeInBytes -
buffering
private boolean buffering -
bufferedEvent
-
producerException
-
-
Constructor Details
-
ChangeEventQueue
private ChangeEventQueue(Duration pollInterval, int maxQueueSize, int maxBatchSize, Supplier<LoggingContext.PreviousContext> loggingContextSupplier, long maxQueueSizeInBytes, boolean buffering)
-
-
Method Details
-
enqueue
Enqueues a record so that it can be obtained viapoll()
. This method will block if the queue is full.- Parameters:
record
- the record to be enqueued- Throws:
InterruptedException
- if this thread has been interrupted
-
flushBuffer
Applies a function to the event and the buffer and adds it to the queue. Buffer is emptied.- Parameters:
recordModifier
-- Throws:
InterruptedException
-
disableBuffering
public void disableBuffering()Disable buffering for the queue -
doEnqueue
- Throws:
InterruptedException
-
poll
Returns the next batch of elements from this queue. May be empty in case no elements have arrived in the maximum waiting time.- Throws:
InterruptedException
- if this thread has been interrupted while waiting for more elements to arrive
-
drainRecords
-
producerException
-
throwProducerExceptionIfPresent
private void throwProducerExceptionIfPresent() -
totalCapacity
public int totalCapacity()- Specified by:
totalCapacity
in interfaceChangeEventQueueMetrics
-
remainingCapacity
public int remainingCapacity()- Specified by:
remainingCapacity
in interfaceChangeEventQueueMetrics
-
maxQueueSizeInBytes
public long maxQueueSizeInBytes()- Specified by:
maxQueueSizeInBytes
in interfaceChangeEventQueueMetrics
-
currentQueueSizeInBytes
public long currentQueueSizeInBytes()- Specified by:
currentQueueSizeInBytes
in interfaceChangeEventQueueMetrics
-
isBuffered
public boolean isBuffered()
-