Class EmbeddedEngine
- All Implemented Interfaces:
EmbeddedEngineConfig
,io.debezium.engine.DebeziumEngine<org.apache.kafka.connect.source.SourceRecord>
,Closeable
,AutoCloseable
,Runnable
SourceConnector
within an application's process. An embedded connector
is entirely standalone and only talks with the source system; no Kafka, Kafka Connect, or Zookeeper processes are needed.
Applications using an embedded connector simply set one up and supply a consumer function
to which the
connector will pass all SourceRecord
s containing database change events.
With an embedded connector, the application that runs the connector assumes all responsibility for fault tolerance, scalability, and durability. Additionally, applications must specify how the connector can store its relational database schema history and offsets. By default, this information will be stored in memory and will thus be lost upon application restart.
Embedded connectors are designed to be submitted to an Executor
or ExecutorService
for execution by a single
thread, and a running connector can be stopped either by calling stop()
from another thread or by interrupting
the running thread (e.g., as is the case with ExecutorService.shutdownNow()
).
- Author:
- Randall Hauch
-
Nested Class Summary
Modifier and TypeClassDescriptionstatic class
A callback function to be notified when the connector completes.protected static class
private static class
static final class
private class
protected class
Implementation ofDebeziumEngine.Offsets
which can be used to construct aSourceRecord
with its offsets.Nested classes/interfaces inherited from interface io.debezium.engine.DebeziumEngine
io.debezium.engine.DebeziumEngine.Builder<R extends Object>, io.debezium.engine.DebeziumEngine.BuilderFactory, io.debezium.engine.DebeziumEngine.ChangeConsumer<R extends Object>, io.debezium.engine.DebeziumEngine.CompletionCallback, io.debezium.engine.DebeziumEngine.ConnectorCallback, io.debezium.engine.DebeziumEngine.Offsets, io.debezium.engine.DebeziumEngine.RecordCommitter<R extends Object>
-
Field Summary
Modifier and TypeFieldDescriptionprivate final ClassLoader
private final Clock
private final io.debezium.engine.DebeziumEngine.CompletionCallback
private final EmbeddedEngine.CompletionResult
private final Configuration
private final io.debezium.engine.DebeziumEngine.ConnectorCallback
private final io.debezium.engine.DebeziumEngine.ChangeConsumer<org.apache.kafka.connect.source.SourceRecord>
private final org.apache.kafka.connect.storage.Converter
private final VariableLatch
private static final org.slf4j.Logger
private io.debezium.engine.spi.OffsetCommitPolicy
private long
private final AtomicReference<Thread>
private org.apache.kafka.connect.source.SourceTask
private long
private final Transformations
private final org.apache.kafka.connect.storage.Converter
private final org.apache.kafka.connect.runtime.WorkerConfig
Fields inherited from interface io.debezium.engine.DebeziumEngine
OFFSET_FLUSH_INTERVAL_MS_PROP
Fields inherited from interface io.debezium.embedded.EmbeddedEngineConfig
ALL_FIELDS, CONNECTOR_CLASS, CONNECTOR_FIELDS, DEFAULT_ERROR_MAX_RETRIES, ENGINE_NAME, ERRORS_MAX_RETRIES, ERRORS_RETRY_DELAY_INITIAL_MS, ERRORS_RETRY_DELAY_MAX_MS, OFFSET_COMMIT_POLICY, OFFSET_COMMIT_TIMEOUT_MS, OFFSET_FLUSH_INTERVAL_MS, OFFSET_STORAGE, OFFSET_STORAGE_FILE_FILENAME, OFFSET_STORAGE_KAFKA_PARTITIONS, OFFSET_STORAGE_KAFKA_REPLICATION_FACTOR, OFFSET_STORAGE_KAFKA_TOPIC, PREDICATES, TRANSFORMS, WAIT_FOR_COMPLETION_BEFORE_INTERRUPT_MS
-
Constructor Summary
ModifierConstructorDescriptionprivate
EmbeddedEngine
(Configuration config, ClassLoader classLoader, Clock clock, io.debezium.engine.DebeziumEngine.ChangeConsumer<org.apache.kafka.connect.source.SourceRecord> handler, io.debezium.engine.DebeziumEngine.CompletionCallback completionCallback, io.debezium.engine.DebeziumEngine.ConnectorCallback connectorCallback, io.debezium.engine.spi.OffsetCommitPolicy offsetCommitPolicy) -
Method Summary
Modifier and TypeMethodDescriptionboolean
Wait for the connector to complete processing.private static io.debezium.engine.DebeziumEngine.ChangeConsumer<org.apache.kafka.connect.source.SourceRecord>
buildDefaultChangeConsumer
(Consumer<org.apache.kafka.connect.source.SourceRecord> consumer) protected io.debezium.engine.DebeziumEngine.RecordCommitter
buildRecordCommitter
(org.apache.kafka.connect.storage.OffsetStorageWriter offsetWriter, org.apache.kafka.connect.source.SourceTask task, Duration commitTimeout) Creates a new RecordCommitter that is responsible for informing the engine about the updates to the given batchvoid
close()
protected void
commitOffsets
(org.apache.kafka.connect.storage.OffsetStorageWriter offsetWriter, Duration commitTimeout, org.apache.kafka.connect.source.SourceTask task) Flush offsets to storage.protected void
completedFlush
(Throwable error, Void result) private org.apache.kafka.connect.source.SourceTask
createSourceTask
(org.apache.kafka.connect.source.SourceConnector connector, List<Map<String, String>> taskConfigs, Class<? extends org.apache.kafka.connect.connector.Task> taskClass) private DelayStrategy
delayStrategy
(Configuration config) private void
private void
private void
failAndThrow
(String msg, Throwable error) getConnectorConfig
(org.apache.kafka.connect.source.SourceConnector connector, String connectorClassName) private int
private Throwable
handleRetries
(org.apache.kafka.connect.errors.RetriableException e, List<Map<String, String>> taskConfigs) private void
initializeConnector
(org.apache.kafka.connect.source.SourceConnector connector, org.apache.kafka.connect.storage.OffsetStorageReader offsetReader) private org.apache.kafka.connect.storage.OffsetBackingStore
initializeOffsetStore
(Map<String, String> connectorConfig) Determines, which offset backing store should be used, instantiate it and start the offset store.private org.apache.kafka.connect.source.SourceConnector
instantiateConnector
(String connectorClassName) boolean
Determine if this embedded connector is currently running.protected void
maybeFlush
(org.apache.kafka.connect.storage.OffsetStorageWriter offsetWriter, io.debezium.engine.spi.OffsetCommitPolicy policy, Duration commitTimeout, org.apache.kafka.connect.source.SourceTask task) Determine if we should flush offsets to storage, and if so then attempt to flush offsets.private void
pollRecords
(List<Map<String, String>> taskConfigs, io.debezium.engine.DebeziumEngine.RecordCommitter committer, EmbeddedEngine.HandlerErrors errors) void
run()
Run this embedded connector and deliver database changes to the registeredConsumer
.void
runWithTask
(Consumer<org.apache.kafka.connect.source.SourceTask> consumer) private void
setCompletionResult
(String connectorClassName, EmbeddedEngine.HandlerErrors errors) private void
private void
startSourceTask
(List<Map<String, String>> taskConfigs, org.apache.kafka.connect.storage.OffsetStorageReader offsetReader) boolean
stop()
Stop the execution of this embedded connector.private void
stopOffsetStoreAndConnector
(org.apache.kafka.connect.source.SourceConnector connector, String connectorClassName, org.apache.kafka.connect.storage.OffsetBackingStore offsetStore, Optional<io.debezium.engine.DebeziumEngine.ConnectorCallback> connectorCallback) private void
private void
stopTaskAndCommitOffset
(org.apache.kafka.connect.storage.OffsetStorageWriter offsetWriter, Duration commitTimeout, Optional<io.debezium.engine.DebeziumEngine.ConnectorCallback> connectorCallback) private void
toString()
-
Field Details
-
LOGGER
private static final org.slf4j.Logger LOGGER -
config
-
clock
-
classLoader
-
handler
private final io.debezium.engine.DebeziumEngine.ChangeConsumer<org.apache.kafka.connect.source.SourceRecord> handler -
completionCallback
private final io.debezium.engine.DebeziumEngine.CompletionCallback completionCallback -
connectorCallback
private final io.debezium.engine.DebeziumEngine.ConnectorCallback connectorCallback -
runningThread
-
latch
-
keyConverter
private final org.apache.kafka.connect.storage.Converter keyConverter -
valueConverter
private final org.apache.kafka.connect.storage.Converter valueConverter -
workerConfig
private final org.apache.kafka.connect.runtime.WorkerConfig workerConfig -
completionResult
-
recordsSinceLastCommit
private long recordsSinceLastCommit -
timeOfLastCommitMillis
private long timeOfLastCommitMillis -
offsetCommitPolicy
private io.debezium.engine.spi.OffsetCommitPolicy offsetCommitPolicy -
task
private org.apache.kafka.connect.source.SourceTask task -
transformations
-
-
Constructor Details
-
EmbeddedEngine
private EmbeddedEngine(Configuration config, ClassLoader classLoader, Clock clock, io.debezium.engine.DebeziumEngine.ChangeConsumer<org.apache.kafka.connect.source.SourceRecord> handler, io.debezium.engine.DebeziumEngine.CompletionCallback completionCallback, io.debezium.engine.DebeziumEngine.ConnectorCallback connectorCallback, io.debezium.engine.spi.OffsetCommitPolicy offsetCommitPolicy)
-
-
Method Details
-
buildDefaultChangeConsumer
private static io.debezium.engine.DebeziumEngine.ChangeConsumer<org.apache.kafka.connect.source.SourceRecord> buildDefaultChangeConsumer(Consumer<org.apache.kafka.connect.source.SourceRecord> consumer) -
isRunning
public boolean isRunning()Determine if this embedded connector is currently running.- Returns:
true
if running, orfalse
otherwise
-
fail
-
fail
-
failAndThrow
private void failAndThrow(String msg, Throwable error) throws EmbeddedEngine.EmbeddedEngineRuntimeException -
succeed
-
run
public void run()Run this embedded connector and deliver database changes to the registeredConsumer
. This method blocks until the connector is stopped.First, the method checks to see if this instance is currently
running
, and if so immediately returns.If the configuration is valid, this method starts the connector and starts polling the connector for change events. All messages are delivered in batches to the
Consumer
registered with this embedded connector. The batch size, polling frequency, and other parameters are controlled via configuration settings. This continues until this connector isstopped
.Note that there are two ways to stop a connector running on a thread: calling
stop()
from another thread, or interrupting the thread (e.g., viaExecutorService.shutdownNow()
).This method can be called repeatedly as needed.
-
instantiateConnector
private org.apache.kafka.connect.source.SourceConnector instantiateConnector(String connectorClassName) throws EmbeddedEngine.EmbeddedEngineRuntimeException -
getConnectorConfig
private Map<String,String> getConnectorConfig(org.apache.kafka.connect.source.SourceConnector connector, String connectorClassName) throws EmbeddedEngine.EmbeddedEngineRuntimeException -
initializeOffsetStore
private org.apache.kafka.connect.storage.OffsetBackingStore initializeOffsetStore(Map<String, String> connectorConfig) throws EmbeddedEngine.EmbeddedEngineRuntimeExceptionDetermines, which offset backing store should be used, instantiate it and start the offset store. -
setOffsetCommitPolicy
-
initializeConnector
private void initializeConnector(org.apache.kafka.connect.source.SourceConnector connector, org.apache.kafka.connect.storage.OffsetStorageReader offsetReader) -
createSourceTask
private org.apache.kafka.connect.source.SourceTask createSourceTask(org.apache.kafka.connect.source.SourceConnector connector, List<Map<String, String>> taskConfigs, Class<? extends org.apache.kafka.connect.connector.Task> taskClass) throws EmbeddedEngine.EmbeddedEngineRuntimeException, NoSuchMethodException, InvocationTargetException -
startSourceTask
-
stopSourceTask
private void stopSourceTask() -
handleRetries
-
pollRecords
private void pollRecords(List<Map<String, String>> taskConfigs, io.debezium.engine.DebeziumEngine.RecordCommitter committer, EmbeddedEngine.HandlerErrors errors) throws Throwable- Throws:
Throwable
-
setCompletionResult
-
stopTaskAndCommitOffset
-
stopOffsetStoreAndConnector
-
getErrorsMaxRetries
private int getErrorsMaxRetries() -
buildRecordCommitter
protected io.debezium.engine.DebeziumEngine.RecordCommitter buildRecordCommitter(org.apache.kafka.connect.storage.OffsetStorageWriter offsetWriter, org.apache.kafka.connect.source.SourceTask task, Duration commitTimeout) Creates a new RecordCommitter that is responsible for informing the engine about the updates to the given batch- Parameters:
offsetWriter
- the offsetWriter current in usetask
- the sourcetaskcommitTimeout
- the time in ms until a commit times out- Returns:
- the new recordCommitter to be used for a given batch
-
maybeFlush
protected void maybeFlush(org.apache.kafka.connect.storage.OffsetStorageWriter offsetWriter, io.debezium.engine.spi.OffsetCommitPolicy policy, Duration commitTimeout, org.apache.kafka.connect.source.SourceTask task) throws InterruptedException Determine if we should flush offsets to storage, and if so then attempt to flush offsets.- Parameters:
offsetWriter
- the offset storage writer; may not be nullpolicy
- the offset commit policy; may not be nullcommitTimeout
- the timeout to wait for commit resultstask
- the task which produced the records for which the offsets have been committed- Throws:
InterruptedException
-
commitOffsets
protected void commitOffsets(org.apache.kafka.connect.storage.OffsetStorageWriter offsetWriter, Duration commitTimeout, org.apache.kafka.connect.source.SourceTask task) throws InterruptedException Flush offsets to storage.- Parameters:
offsetWriter
- the offset storage writer; may not be nullcommitTimeout
- the timeout to wait for commit resultstask
- the task which produced the records for which the offsets have been committed- Throws:
InterruptedException
-
completedFlush
-
stop
public boolean stop()Stop the execution of this embedded connector. This method does not block until the connector is stopped; useawait(long, TimeUnit)
for this purpose.- Returns:
true
if the connector wasrunning
and will eventually stop, orfalse
if it was not running when this method is called- See Also:
-
close
- Specified by:
close
in interfaceAutoCloseable
- Specified by:
close
in interfaceCloseable
- Throws:
IOException
-
await
Wait for the connector to complete processing. If the processor is not running, this method returns immediately; however, if the processor isstopped
and restarted before this method is called, this method will return only when it completes the second time.- Parameters:
timeout
- the maximum amount of time to wait before returningunit
- the unit of time; may not be null- Returns:
true
if the connector completed within the timeout (or was not running), orfalse
if it is still running when the timeout occurred- Throws:
InterruptedException
- if this thread is interrupted while waiting for the completion of the connector
-
toString
-
runWithTask
-
delayStrategy
-