Class EmbeddedEngine
- All Implemented Interfaces:
io.debezium.engine.DebeziumEngine<org.apache.kafka.connect.source.SourceRecord>
,Closeable
,AutoCloseable
,Runnable
SourceConnector
within an application's process. An embedded connector
is entirely standalone and only talks with the source system; no Kafka, Kafka Connect, or Zookeeper processes are needed.
Applications using an embedded connector simply set one up and supply a consumer function
to which the
connector will pass all SourceRecord
s containing database change events.
With an embedded connector, the application that runs the connector assumes all responsibility for fault tolerance, scalability, and durability. Additionally, applications must specify how the connector can store its relational database schema history and offsets. By default, this information will be stored in memory and will thus be lost upon application restart.
Embedded connectors are designed to be submitted to an Executor
or ExecutorService
for execution by a single
thread, and a running connector can be stopped either by calling stop()
from another thread or by interrupting
the running thread (e.g., as is the case with ExecutorService.shutdownNow()
).
- Author:
- Randall Hauch
-
Nested Class Summary
Modifier and TypeClassDescriptionstatic interface
Deprecated.static final class
static interface
Deprecated.static interface
Deprecated.static class
A callback function to be notified when the connector completes.static interface
Deprecated.protected static class
static interface
Deprecated.protected class
Implementation ofDebeziumEngine.Offsets
which can be used to construct aSourceRecord
with its offsets.Nested classes/interfaces inherited from interface io.debezium.engine.DebeziumEngine
io.debezium.engine.DebeziumEngine.BuilderFactory, io.debezium.engine.DebeziumEngine.Offsets
-
Field Summary
Modifier and TypeFieldDescriptionprotected static final Field.Set
The array of all exposed fields.private final ClassLoader
private final Clock
private final io.debezium.engine.DebeziumEngine.CompletionCallback
private final EmbeddedEngine.CompletionResult
private final Configuration
static final Field
A required field for an embedded connector that specifies the name of the normal Debezium connector's Java class.static final Field.Set
The array of fields that are required by each connectors.private final io.debezium.engine.DebeziumEngine.ConnectorCallback
static final Field
A required field for an embedded connector that specifies the unique name for the connector instance.private final io.debezium.engine.DebeziumEngine.ChangeConsumer<org.apache.kafka.connect.source.SourceRecord>
protected static final Field
protected static final Field
private final org.apache.kafka.connect.storage.Converter
private final VariableLatch
private static final org.slf4j.Logger
static final Field
static final Field
An optional advanced field that specifies the maximum amount of time that the embedded connector should wait for an offset commit to complete.static final Field
An optional advanced field that specifies the maximum amount of time that the embedded connector should wait for an offset commit to complete.static final Field
An optional field that specifies the name of the class that implements theOffsetBackingStore
interface, and that will be used to store offsets recorded by the connector.static final Field
An optional field that specifies the file location for theFileOffsetBackingStore
.static final Field
An optional field that specifies the number of partitions for theKafkaOffsetBackingStore
.static final Field
An optional field that specifies the replication factor for theKafkaOffsetBackingStore
.static final Field
An optional field that specifies the topic name for theKafkaOffsetBackingStore
.private io.debezium.engine.spi.OffsetCommitPolicy
private long
private final AtomicReference<Thread>
private org.apache.kafka.connect.source.SourceTask
private long
private final Transformations
static final Field
A list of SMTs to be applied on the messages generated by the engine.private final org.apache.kafka.connect.storage.Converter
private static final Duration
How long we wait before forcefully stopping the connector thread when shutting down.private static final String
private final org.apache.kafka.connect.runtime.WorkerConfig
Fields inherited from interface io.debezium.engine.DebeziumEngine
OFFSET_FLUSH_INTERVAL_MS_PROP
-
Constructor Summary
ModifierConstructorDescriptionprivate
EmbeddedEngine
(Configuration config, ClassLoader classLoader, Clock clock, io.debezium.engine.DebeziumEngine.ChangeConsumer<org.apache.kafka.connect.source.SourceRecord> handler, io.debezium.engine.DebeziumEngine.CompletionCallback completionCallback, io.debezium.engine.DebeziumEngine.ConnectorCallback connectorCallback, io.debezium.engine.spi.OffsetCommitPolicy offsetCommitPolicy) -
Method Summary
Modifier and TypeMethodDescriptionboolean
Wait for the connector to complete processing.private static EmbeddedEngine.ChangeConsumer
buildDefaultChangeConsumer
(Consumer<org.apache.kafka.connect.source.SourceRecord> consumer) protected EmbeddedEngine.RecordCommitter
buildRecordCommitter
(org.apache.kafka.connect.storage.OffsetStorageWriter offsetWriter, org.apache.kafka.connect.source.SourceTask task, Duration commitTimeout) Creates a new RecordCommitter that is responsible for informing the engine about the updates to the given batchvoid
close()
protected void
commitOffsets
(org.apache.kafka.connect.storage.OffsetStorageWriter offsetWriter, Duration commitTimeout, org.apache.kafka.connect.source.SourceTask task) Flush offsets to storage.protected void
completedFlush
(Throwable error, Void result) static EmbeddedEngine.Builder
create()
Deprecated.private void
private void
boolean
Determine if this embedded connector is currently running.protected void
maybeFlush
(org.apache.kafka.connect.storage.OffsetStorageWriter offsetWriter, io.debezium.engine.spi.OffsetCommitPolicy policy, Duration commitTimeout, org.apache.kafka.connect.source.SourceTask task) Determine if we should flush offsets to storage, and if so then attempt to flush offsets.void
run()
Run this embedded connector and deliver database changes to the registeredConsumer
.void
runWithTask
(Consumer<org.apache.kafka.connect.source.SourceTask> consumer) boolean
stop()
Stop the execution of this embedded connector.private void
toString()
-
Field Details
-
ENGINE_NAME
A required field for an embedded connector that specifies the unique name for the connector instance. -
CONNECTOR_CLASS
A required field for an embedded connector that specifies the name of the normal Debezium connector's Java class. -
OFFSET_STORAGE
An optional field that specifies the name of the class that implements theOffsetBackingStore
interface, and that will be used to store offsets recorded by the connector. -
OFFSET_STORAGE_FILE_FILENAME
An optional field that specifies the file location for theFileOffsetBackingStore
.- See Also:
-
OFFSET_STORAGE_KAFKA_TOPIC
An optional field that specifies the topic name for theKafkaOffsetBackingStore
.- See Also:
-
OFFSET_STORAGE_KAFKA_PARTITIONS
An optional field that specifies the number of partitions for theKafkaOffsetBackingStore
.- See Also:
-
OFFSET_STORAGE_KAFKA_REPLICATION_FACTOR
An optional field that specifies the replication factor for theKafkaOffsetBackingStore
.- See Also:
-
OFFSET_FLUSH_INTERVAL_MS
An optional advanced field that specifies the maximum amount of time that the embedded connector should wait for an offset commit to complete. -
OFFSET_COMMIT_TIMEOUT_MS
An optional advanced field that specifies the maximum amount of time that the embedded connector should wait for an offset commit to complete. -
OFFSET_COMMIT_POLICY
-
INTERNAL_KEY_CONVERTER_CLASS
-
INTERNAL_VALUE_CONVERTER_CLASS
-
TRANSFORMS
A list of SMTs to be applied on the messages generated by the engine. -
CONNECTOR_FIELDS
The array of fields that are required by each connectors. -
ALL_FIELDS
The array of all exposed fields. -
WAIT_FOR_COMPLETION_BEFORE_INTERRUPT_DEFAULT
How long we wait before forcefully stopping the connector thread when shutting down. Must be longer thanChangeEventSourceCoordinator.SHUTDOWN_WAIT_TIMEOUT
* 2. -
WAIT_FOR_COMPLETION_BEFORE_INTERRUPT_PROP
- See Also:
-
LOGGER
private static final org.slf4j.Logger LOGGER -
config
-
clock
-
classLoader
-
handler
private final io.debezium.engine.DebeziumEngine.ChangeConsumer<org.apache.kafka.connect.source.SourceRecord> handler -
completionCallback
private final io.debezium.engine.DebeziumEngine.CompletionCallback completionCallback -
connectorCallback
private final io.debezium.engine.DebeziumEngine.ConnectorCallback connectorCallback -
runningThread
-
latch
-
keyConverter
private final org.apache.kafka.connect.storage.Converter keyConverter -
valueConverter
private final org.apache.kafka.connect.storage.Converter valueConverter -
workerConfig
private final org.apache.kafka.connect.runtime.WorkerConfig workerConfig -
completionResult
-
recordsSinceLastCommit
private long recordsSinceLastCommit -
timeOfLastCommitMillis
private long timeOfLastCommitMillis -
offsetCommitPolicy
private io.debezium.engine.spi.OffsetCommitPolicy offsetCommitPolicy -
task
private org.apache.kafka.connect.source.SourceTask task -
transformations
-
-
Constructor Details
-
EmbeddedEngine
private EmbeddedEngine(Configuration config, ClassLoader classLoader, Clock clock, io.debezium.engine.DebeziumEngine.ChangeConsumer<org.apache.kafka.connect.source.SourceRecord> handler, io.debezium.engine.DebeziumEngine.CompletionCallback completionCallback, io.debezium.engine.DebeziumEngine.ConnectorCallback connectorCallback, io.debezium.engine.spi.OffsetCommitPolicy offsetCommitPolicy)
-
-
Method Details
-
buildDefaultChangeConsumer
private static EmbeddedEngine.ChangeConsumer buildDefaultChangeConsumer(Consumer<org.apache.kafka.connect.source.SourceRecord> consumer) -
create
Deprecated.Obtain a newEmbeddedEngine.Builder
instance that can be used to construct runnableEmbeddedEngine
instances.- Returns:
- the new builder; never null
-
isRunning
public boolean isRunning()Determine if this embedded connector is currently running.- Returns:
true
if running, orfalse
otherwise
-
fail
-
fail
-
succeed
-
run
public void run()Run this embedded connector and deliver database changes to the registeredConsumer
. This method blocks until the connector is stopped.First, the method checks to see if this instance is currently
running
, and if so immediately returns.If the configuration is valid, this method starts the connector and starts polling the connector for change events. All messages are delivered in batches to the
Consumer
registered with this embedded connector. The batch size, polling frequency, and other parameters are controlled via configuration settings. This continues until this connector isstopped
.Note that there are two ways to stop a connector running on a thread: calling
stop()
from another thread, or interrupting the thread (e.g., viaExecutorService.shutdownNow()
).This method can be called repeatedly as needed.
-
buildRecordCommitter
protected EmbeddedEngine.RecordCommitter buildRecordCommitter(org.apache.kafka.connect.storage.OffsetStorageWriter offsetWriter, org.apache.kafka.connect.source.SourceTask task, Duration commitTimeout) Creates a new RecordCommitter that is responsible for informing the engine about the updates to the given batch- Parameters:
offsetWriter
- the offsetWriter current in usetask
- the sourcetaskcommitTimeout
- the time in ms until a commit times out- Returns:
- the new recordCommitter to be used for a given batch
-
maybeFlush
protected void maybeFlush(org.apache.kafka.connect.storage.OffsetStorageWriter offsetWriter, io.debezium.engine.spi.OffsetCommitPolicy policy, Duration commitTimeout, org.apache.kafka.connect.source.SourceTask task) throws InterruptedException Determine if we should flush offsets to storage, and if so then attempt to flush offsets.- Parameters:
offsetWriter
- the offset storage writer; may not be nullpolicy
- the offset commit policy; may not be nullcommitTimeout
- the timeout to wait for commit resultstask
- the task which produced the records for which the offsets have been committed- Throws:
InterruptedException
-
commitOffsets
protected void commitOffsets(org.apache.kafka.connect.storage.OffsetStorageWriter offsetWriter, Duration commitTimeout, org.apache.kafka.connect.source.SourceTask task) throws InterruptedException Flush offsets to storage.- Parameters:
offsetWriter
- the offset storage writer; may not be nullcommitTimeout
- the timeout to wait for commit resultstask
- the task which produced the records for which the offsets have been committed- Throws:
InterruptedException
-
completedFlush
-
stop
public boolean stop()Stop the execution of this embedded connector. This method does not block until the connector is stopped; useawait(long, TimeUnit)
for this purpose.- Returns:
true
if the connector wasrunning
and will eventually stop, orfalse
if it was not running when this method is called- See Also:
-
close
- Specified by:
close
in interfaceAutoCloseable
- Specified by:
close
in interfaceCloseable
- Throws:
IOException
-
await
Wait for the connector to complete processing. If the processor is not running, this method returns immediately; however, if the processor isstopped
and restarted before this method is called, this method will return only when it completes the second time.- Parameters:
timeout
- the maximum amount of time to wait before returningunit
- the unit of time; may not be null- Returns:
true
if the connector completed within the timeout (or was not running), orfalse
if it is still running when the timeout occurred- Throws:
InterruptedException
- if this thread is interrupted while waiting for the completion of the connector
-
toString
-
runWithTask
-