Package io.debezium.transforms

Class ExtractNewRecordState<R extends org.apache.kafka.connect.connector.ConnectRecord<R>>

java.lang.Object

io.debezium.transforms.ExtractNewRecordState<R>

Type Parameters:

R - the subtype of ConnectRecord on which this transformation will operate

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.kafka.common.Configurable, org.apache.kafka.connect.transforms.Transformation<R>

public class ExtractNewRecordState<R extends org.apache.kafka.connect.connector.ConnectRecord<R>>
extends Object
implements org.apache.kafka.connect.transforms.Transformation<R>

Debezium generates CDC (Envelope) records that are struct of values containing values before and after change. Sink connectors usually are not able to work with a complex structure so a user use this SMT to extract after value and send it down unwrapped in Envelope.

The functionality is similar to ExtractField SMT but has a special semantics for handling delete events; when delete event is emitted by database then Debezium emits two messages: a delete message and a tombstone message that serves as a signal to Kafka compaction process.

The SMT by default drops the tombstone message created by Debezium and converts the delete message into a tombstone message that can be dropped, too, if required.

The SMT also has the option to insert fields from the original record (e.g. 'op' or 'source.ts_ms') into the unwrapped record or add them as header attributes.

Author:

Jiri Pechanec

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
private static class	ExtractNewRecordState.FieldReference	Represents a field that should be added to the outgoing record as a header attribute or struct field.
private static class	ExtractNewRecordState.NewRecordValueMetadata

Field Summary

Fields

Modifier and Type	Field	Description
private List<ExtractNewRecordState.FieldReference>	additionalFields
private List<ExtractNewRecordState.FieldReference>	additionalHeaders
private final org.apache.kafka.connect.transforms.ExtractField<R>	afterDelegate
private final org.apache.kafka.connect.transforms.ExtractField<R>	beforeDelegate
private static final Field	DROP_FIELDS_FROM_KEY
private static final Field	DROP_FIELDS_HEADER
private static final Field	DROP_FIELDS_KEEP_SCHEMA_COMPATIBLE
private boolean	dropFieldsFromKey
private String	dropFieldsHeaderName
private boolean	dropFieldsKeepSchemaCompatible
private boolean	dropTombstones
private static final String	EXCLUDE
private static final Pattern	FIELD_SEPARATOR
private ExtractNewRecordStateConfigDefinition.DeleteHandling	handleDeletes
private static final org.slf4j.Logger	LOGGER
private static final Pattern	NEW_FIELD_SEPARATOR
private static final String	PURPOSE
private final org.apache.kafka.connect.transforms.InsertField<R>	removedDelegate
private String	routeByField
private static final int	SCHEMA_CACHE_SIZE
private BoundedConcurrentHashMap<ExtractNewRecordState.NewRecordValueMetadata,org.apache.kafka.connect.data.Schema>	schemaUpdateCache
private SmtManager<R>	smtManager
private final org.apache.kafka.connect.transforms.InsertField<R>	updatedDelegate

Constructor Summary

Constructors

Constructor	Description
ExtractNewRecordState()

Method Summary

Modifier and Type	Method	Description
private R	addFields(List<ExtractNewRecordState.FieldReference> additionalFields, R originalRecord, R unwrappedRecord)
R	apply(R record)
private ExtractNewRecordState.NewRecordValueMetadata	buildCacheKey(org.apache.kafka.connect.data.Struct value, R originalRecord)
void	close()
org.apache.kafka.common.config.ConfigDef	config()
void	configure(Map<String,?> configs)
private R	dropFields(R record)
private R	dropKeyFields(R record, List<String> fieldNames)
private R	dropValueFields(R record, List<String> fieldNames)
private List<String>	getFieldsToDropFromSchema(org.apache.kafka.connect.data.Schema schema, List<String> fieldNames)
private org.apache.kafka.connect.header.Header	getHeaderByName(R record, String headerName)
private org.apache.kafka.connect.header.Headers	makeHeaders(List<ExtractNewRecordState.FieldReference> additionalHeaders, org.apache.kafka.connect.data.Struct originalRecordValue)	Create an Headers object which contains the headers to be added.
private org.apache.kafka.connect.data.Schema	makeUpdatedSchema(List<ExtractNewRecordState.FieldReference> additionalFields, org.apache.kafka.connect.data.Schema schema, org.apache.kafka.connect.data.Struct originalRecordValue)
private R	setTopic(String updatedTopicValue, R record)
private org.apache.kafka.connect.data.SchemaBuilder	updateSchema(ExtractNewRecordState.FieldReference fieldReference, org.apache.kafka.connect.data.SchemaBuilder builder, org.apache.kafka.connect.data.Schema fieldSchema)
private org.apache.kafka.connect.data.Struct	updateValue(ExtractNewRecordState.FieldReference fieldReference, org.apache.kafka.connect.data.Struct updatedValue, org.apache.kafka.connect.data.Struct struct)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

LOGGER

private static final org.slf4j.Logger LOGGER

PURPOSE

private static final String PURPOSE

See Also:

• Constant Field Values

EXCLUDE

private static final String EXCLUDE

See Also:

• Constant Field Values

SCHEMA_CACHE_SIZE

private static final int SCHEMA_CACHE_SIZE

See Also:

• Constant Field Values

FIELD_SEPARATOR

private static final Pattern FIELD_SEPARATOR

NEW_FIELD_SEPARATOR

private static final Pattern NEW_FIELD_SEPARATOR

DROP_FIELDS_HEADER

private static final Field DROP_FIELDS_HEADER

DROP_FIELDS_FROM_KEY

private static final Field DROP_FIELDS_FROM_KEY

DROP_FIELDS_KEEP_SCHEMA_COMPATIBLE

private static final Field DROP_FIELDS_KEEP_SCHEMA_COMPATIBLE

dropTombstones

private boolean dropTombstones

dropFieldsHeaderName

private String dropFieldsHeaderName

dropFieldsFromKey

private boolean dropFieldsFromKey

dropFieldsKeepSchemaCompatible

private boolean dropFieldsKeepSchemaCompatible

handleDeletes

private ExtractNewRecordStateConfigDefinition.DeleteHandling handleDeletes

additionalHeaders

private List<ExtractNewRecordState.FieldReference> additionalHeaders

additionalFields

private List<ExtractNewRecordState.FieldReference> additionalFields

routeByField

private String routeByField

afterDelegate

private final org.apache.kafka.connect.transforms.ExtractField<R extends org.apache.kafka.connect.connector.ConnectRecord<R>> afterDelegate

beforeDelegate

private final org.apache.kafka.connect.transforms.ExtractField<R extends org.apache.kafka.connect.connector.ConnectRecord<R>> beforeDelegate

removedDelegate

private final org.apache.kafka.connect.transforms.InsertField<R extends org.apache.kafka.connect.connector.ConnectRecord<R>> removedDelegate

updatedDelegate

private final org.apache.kafka.connect.transforms.InsertField<R extends org.apache.kafka.connect.connector.ConnectRecord<R>> updatedDelegate

schemaUpdateCache

private BoundedConcurrentHashMap<ExtractNewRecordState.NewRecordValueMetadata,org.apache.kafka.connect.data.Schema> schemaUpdateCache

smtManager

private SmtManager<R extends org.apache.kafka.connect.connector.ConnectRecord<R>> smtManager

Constructor Details

ExtractNewRecordState

public ExtractNewRecordState()

Method Details

configure

public void configure(Map<String,?> configs)

Specified by:

configure in interface org.apache.kafka.common.Configurable

apply

public R apply(R record)

Specified by:

apply in interface org.apache.kafka.connect.transforms.Transformation<R extends org.apache.kafka.connect.connector.ConnectRecord<R>>

setTopic

private R setTopic(String updatedTopicValue,
 R record)

makeHeaders

private org.apache.kafka.connect.header.Headers makeHeaders(List<ExtractNewRecordState.FieldReference> additionalHeaders,
 org.apache.kafka.connect.data.Struct originalRecordValue)

Create an Headers object which contains the headers to be added.

addFields

private R addFields(List<ExtractNewRecordState.FieldReference> additionalFields,
 R originalRecord,
 R unwrappedRecord)

buildCacheKey

private ExtractNewRecordState.NewRecordValueMetadata buildCacheKey(org.apache.kafka.connect.data.Struct value,
 R originalRecord)

dropFields

private R dropFields(R record)

dropKeyFields

private R dropKeyFields(R record,
 List<String> fieldNames)

dropValueFields

private R dropValueFields(R record,
 List<String> fieldNames)

getFieldsToDropFromSchema

private List<String> getFieldsToDropFromSchema(org.apache.kafka.connect.data.Schema schema,
 List<String> fieldNames)

getHeaderByName

private org.apache.kafka.connect.header.Header getHeaderByName(R record,
 String headerName)

makeUpdatedSchema

private org.apache.kafka.connect.data.Schema makeUpdatedSchema(List<ExtractNewRecordState.FieldReference> additionalFields,
 org.apache.kafka.connect.data.Schema schema,
 org.apache.kafka.connect.data.Struct originalRecordValue)

updateSchema

private org.apache.kafka.connect.data.SchemaBuilder updateSchema(ExtractNewRecordState.FieldReference fieldReference,
 org.apache.kafka.connect.data.SchemaBuilder builder,
 org.apache.kafka.connect.data.Schema fieldSchema)

updateValue

private org.apache.kafka.connect.data.Struct updateValue(ExtractNewRecordState.FieldReference fieldReference,
 org.apache.kafka.connect.data.Struct updatedValue,
 org.apache.kafka.connect.data.Struct struct)

config

public org.apache.kafka.common.config.ConfigDef config()

Specified by:

config in interface org.apache.kafka.connect.transforms.Transformation<R extends org.apache.kafka.connect.connector.ConnectRecord<R>>

close

public void close()

Specified by:

close in interface AutoCloseable

Specified by:

close in interface Closeable

Specified by:

close in interface org.apache.kafka.connect.transforms.Transformation<R extends org.apache.kafka.connect.connector.ConnectRecord<R>>