Package io.debezium.connector.common

Class BaseSourceTask<P extends Partition,O extends OffsetContext>

java.lang.Object

org.apache.kafka.connect.source.SourceTask

io.debezium.connector.common.BaseSourceTask<P,O>

All Implemented Interfaces:

org.apache.kafka.connect.connector.Task

public abstract class BaseSourceTask<P extends Partition,O extends OffsetContext>
extends org.apache.kafka.connect.source.SourceTask

Base class for Debezium's CDC SourceTask implementations. Provides functionality common to all connectors, such as validation of the configuration.

Author:

Gunnar Morling

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
protected static class	BaseSourceTask.State

Field Summary

Fields

Modifier and Type	Field	Description
private Clock	clock
private ChangeEventSourceCoordinator<P,O>	coordinator	The change event source coordinator for those connectors adhering to the new framework structure, null for legacy-style connectors.
private static long	INITIAL_POLL_PERIOD_IN_MILLIS
private Map<String,?>	lastOffset	The latest offset that has been acknowledged by the Kafka producer.
private static org.slf4j.Logger	LOGGER
private static long	MAX_POLL_PERIOD_IN_MILLIS
private ElapsedTimeStrategy	pollOutputDelay
private Instant	previousOutputInstant
private Map<String,String>	props	Raw connector properties, kept here so they can be passed again in case of a restart.
private long	recordCounter
private ElapsedTimeStrategy	restartDelay
private Duration	retriableRestartWait
private AtomicReference<BaseSourceTask.State>	state
private ReentrantLock	stateLock	Used to ensure that start(), stop() and commitRecord() calls are serialized.

Fields inherited from class org.apache.kafka.connect.source.SourceTask

context

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	BaseSourceTask()

Method Summary

Modifier and Type	Method	Description
void	commit()
void	commitRecord(org.apache.kafka.connect.source.SourceRecord record)
protected abstract List<org.apache.kafka.connect.source.SourceRecord>	doPoll()	Returns the next batch of source records, if any are available.
protected abstract void	doStop()
protected abstract Iterable<Field>	getAllConfigurationFields()	Returns all configuration Field supported by this source task.
protected Offsets<P,O>	getPreviousOffsets(Partition.Provider<P> provider, OffsetContext.Loader<O> loader)	Loads the connector's persistent offsets (if present) via the given loader.
(package private) void	logStatistics(List<org.apache.kafka.connect.source.SourceRecord> records)
List<org.apache.kafka.connect.source.SourceRecord>	poll()
protected abstract ChangeEventSourceCoordinator<P,O>	start(Configuration config)	Called once when starting this source task.
void	start(Map<String,String> props)
private boolean	startIfNeededAndPossible()	Starts this connector in case it has been stopped after a retriable error, and the backoff period has passed.
void	stop()
private void	stop(boolean restart)

Methods inherited from class org.apache.kafka.connect.source.SourceTask

commitRecord, initialize

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.kafka.connect.connector.Task

version

Field Detail

LOGGER

private static final org.slf4j.Logger LOGGER

INITIAL_POLL_PERIOD_IN_MILLIS

private static final long INITIAL_POLL_PERIOD_IN_MILLIS

MAX_POLL_PERIOD_IN_MILLIS

private static final long MAX_POLL_PERIOD_IN_MILLIS

state

private final AtomicReference<BaseSourceTask.State> state

stateLock

private final ReentrantLock stateLock

Used to ensure that start(), stop() and commitRecord() calls are serialized.

restartDelay

private volatile ElapsedTimeStrategy restartDelay

props

private volatile Map<String,String> props

Raw connector properties, kept here so they can be passed again in case of a restart.

coordinator

private ChangeEventSourceCoordinator<P extends Partition,O extends OffsetContext> coordinator

The change event source coordinator for those connectors adhering to the new framework structure, null for legacy-style connectors.

lastOffset

private volatile Map<String,?> lastOffset

The latest offset that has been acknowledged by the Kafka producer. Will be acknowledged with the source database in commit() (which may be a no-op depending on the connector).

retriableRestartWait

private Duration retriableRestartWait

pollOutputDelay

private final ElapsedTimeStrategy pollOutputDelay

clock

private final Clock clock

recordCounter

private long recordCounter

previousOutputInstant

private Instant previousOutputInstant

Constructor Detail

BaseSourceTask

protected BaseSourceTask()

Method Detail

start

public final void start(Map<String,String> props)

Specified by:

start in interface org.apache.kafka.connect.connector.Task

Specified by:

start in class org.apache.kafka.connect.source.SourceTask

start

protected abstract ChangeEventSourceCoordinator<P,O> start(Configuration config)

Called once when starting this source task.

Parameters:

config - the task configuration; implementations should wrap it in a dedicated implementation of CommonConnectorConfig and work with typed access to configuration properties that way

poll

public final List<org.apache.kafka.connect.source.SourceRecord> poll()
                                                              throws InterruptedException

Specified by:

poll in class org.apache.kafka.connect.source.SourceTask

Throws:

InterruptedException

logStatistics

void logStatistics(List<org.apache.kafka.connect.source.SourceRecord> records)

doPoll

protected abstract List<org.apache.kafka.connect.source.SourceRecord> doPoll()
                                                                      throws InterruptedException

Returns the next batch of source records, if any are available.

Throws:

InterruptedException

startIfNeededAndPossible

private boolean startIfNeededAndPossible()

Starts this connector in case it has been stopped after a retriable error, and the backoff period has passed.

stop

public final void stop()

Specified by:

stop in interface org.apache.kafka.connect.connector.Task

Specified by:

stop in class org.apache.kafka.connect.source.SourceTask

stop

private void stop(boolean restart)

doStop

protected abstract void doStop()

commitRecord

public void commitRecord(org.apache.kafka.connect.source.SourceRecord record)
                  throws InterruptedException

Overrides:

commitRecord in class org.apache.kafka.connect.source.SourceTask

Throws:

InterruptedException

commit

public void commit()
            throws InterruptedException

Overrides:

commit in class org.apache.kafka.connect.source.SourceTask

Throws:

InterruptedException

getAllConfigurationFields

protected abstract Iterable<Field> getAllConfigurationFields()

Returns all configuration Field supported by this source task.

getPreviousOffsets

protected Offsets<P,O> getPreviousOffsets(Partition.Provider<P> provider,
                                                OffsetContext.Loader<O> loader)

Loads the connector's persistent offsets (if present) via the given loader.