Class ByLogicalTableRouter<R extends org.apache.kafka.connect.connector.ConnectRecord<R>>
- java.lang.Object
-
- io.debezium.transforms.ByLogicalTableRouter<R>
-
- Type Parameters:
R
- the subtype ofConnectRecord
on which this transformation will operate
- All Implemented Interfaces:
Closeable
,AutoCloseable
,org.apache.kafka.common.Configurable
,org.apache.kafka.connect.transforms.Transformation<R>
public class ByLogicalTableRouter<R extends org.apache.kafka.connect.connector.ConnectRecord<R>> extends Object implements org.apache.kafka.connect.transforms.Transformation<R>
A logical table consists of one or more physical tables with the same schema. A common use case is sharding -- the two physical tables `db_shard1.my_table` and `db_shard2.my_table` together form one logical table.This Transformation allows us to change a record's topic name and send change events from multiple physical tables to one topic. For instance, we might choose to send the two tables from the above example to the topic `db_shard.my_table`. The config options
TOPIC_REGEX
andTOPIC_REPLACEMENT
are used to change the record's topic.Now that multiple physical tables can share a topic, the event's key may need to be augmented to include fields other than just those for the record's primary/unique key, since these are not guaranteed to be unique across tables. We need some identifier added to the key that distinguishes the different physical tables. The field name specified by the config option
KEY_FIELD_NAME
is added to the key schema for this purpose. By default, its value will be the old topic name, but if a custom value is desired, the config optionsKEY_FIELD_REGEX
andKEY_FIELD_REPLACEMENT
may be used to change it. For instance, in our above example, we might choose to make the identifier `db_shard1` and `db_shard2` respectively.- Author:
- David Leibovic, Mario Mueller
-
-
Field Summary
Fields Modifier and Type Field Description private org.apache.kafka.common.cache.Cache<org.apache.kafka.connect.data.Schema,org.apache.kafka.connect.data.Schema>
envelopeSchemaUpdateCache
private static Field
KEY_ENFORCE_UNIQUENESS
private static Field
KEY_FIELD_NAME
private static Field
KEY_FIELD_REGEX
private static Field
KEY_FIELD_REPLACEMENT
private boolean
keyEnforceUniqueness
private String
keyFieldName
private Pattern
keyFieldRegex
private String
keyFieldReplacement
private org.apache.kafka.common.cache.Cache<String,String>
keyRegexReplaceCache
private org.apache.kafka.common.cache.Cache<org.apache.kafka.connect.data.Schema,org.apache.kafka.connect.data.Schema>
keySchemaUpdateCache
private static org.slf4j.Logger
LOGGER
private SchemaNameAdjuster
schemaNameAdjuster
private SmtManager<R>
smtManager
private static Field
TOPIC_REGEX
private static Field
TOPIC_REPLACEMENT
private Pattern
topicRegex
private org.apache.kafka.common.cache.Cache<String,String>
topicRegexReplaceCache
private String
topicReplacement
-
Constructor Summary
Constructors Constructor Description ByLogicalTableRouter()
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description R
apply(R record)
void
close()
org.apache.kafka.common.config.ConfigDef
config()
void
configure(Map<String,?> props)
private org.apache.kafka.connect.data.SchemaBuilder
copySchemaExcludingName(org.apache.kafka.connect.data.Schema source, org.apache.kafka.connect.data.SchemaBuilder builder)
private org.apache.kafka.connect.data.SchemaBuilder
copySchemaExcludingName(org.apache.kafka.connect.data.Schema source, org.apache.kafka.connect.data.SchemaBuilder builder, boolean copyFields)
private String
determineNewTopic(String oldTopic)
Determine the new topic name.private org.apache.kafka.connect.data.Struct
updateEnvelope(org.apache.kafka.connect.data.Schema newEnvelopeSchema, org.apache.kafka.connect.data.Struct oldEnvelope)
private org.apache.kafka.connect.data.Schema
updateEnvelopeSchema(org.apache.kafka.connect.data.Schema oldEnvelopeSchema, String newTopicName)
private org.apache.kafka.connect.data.Struct
updateKey(org.apache.kafka.connect.data.Schema newKeySchema, org.apache.kafka.connect.data.Struct oldKey, String oldTopic)
private org.apache.kafka.connect.data.Schema
updateKeySchema(org.apache.kafka.connect.data.Schema oldKeySchema, String newTopicName)
private org.apache.kafka.connect.data.Struct
updateValue(org.apache.kafka.connect.data.Schema newValueSchema, org.apache.kafka.connect.data.Struct oldValue)
private static int
validateKeyFieldReplacement(Configuration config, Field field, Field.ValidationOutput problems)
If KEY_FIELD_REGEX has a value that is really a regex, then the KEY_FIELD_REPLACEMENT must be a non-empty value.
-
-
-
Field Detail
-
TOPIC_REGEX
private static final Field TOPIC_REGEX
-
TOPIC_REPLACEMENT
private static final Field TOPIC_REPLACEMENT
-
KEY_ENFORCE_UNIQUENESS
private static final Field KEY_ENFORCE_UNIQUENESS
-
KEY_FIELD_REGEX
private static final Field KEY_FIELD_REGEX
-
KEY_FIELD_NAME
private static final Field KEY_FIELD_NAME
-
KEY_FIELD_REPLACEMENT
private static final Field KEY_FIELD_REPLACEMENT
-
LOGGER
private static final org.slf4j.Logger LOGGER
-
schemaNameAdjuster
private final SchemaNameAdjuster schemaNameAdjuster
-
topicRegex
private Pattern topicRegex
-
topicReplacement
private String topicReplacement
-
keyFieldRegex
private Pattern keyFieldRegex
-
keyEnforceUniqueness
private boolean keyEnforceUniqueness
-
keyFieldReplacement
private String keyFieldReplacement
-
keyFieldName
private String keyFieldName
-
keySchemaUpdateCache
private final org.apache.kafka.common.cache.Cache<org.apache.kafka.connect.data.Schema,org.apache.kafka.connect.data.Schema> keySchemaUpdateCache
-
envelopeSchemaUpdateCache
private final org.apache.kafka.common.cache.Cache<org.apache.kafka.connect.data.Schema,org.apache.kafka.connect.data.Schema> envelopeSchemaUpdateCache
-
keyRegexReplaceCache
private final org.apache.kafka.common.cache.Cache<String,String> keyRegexReplaceCache
-
topicRegexReplaceCache
private final org.apache.kafka.common.cache.Cache<String,String> topicRegexReplaceCache
-
smtManager
private SmtManager<R extends org.apache.kafka.connect.connector.ConnectRecord<R>> smtManager
-
-
Method Detail
-
validateKeyFieldReplacement
private static int validateKeyFieldReplacement(Configuration config, Field field, Field.ValidationOutput problems)
If KEY_FIELD_REGEX has a value that is really a regex, then the KEY_FIELD_REPLACEMENT must be a non-empty value.
-
configure
public void configure(Map<String,?> props)
- Specified by:
configure
in interfaceorg.apache.kafka.common.Configurable
-
close
public void close()
-
config
public org.apache.kafka.common.config.ConfigDef config()
-
determineNewTopic
private String determineNewTopic(String oldTopic)
Determine the new topic name.- Parameters:
oldTopic
- the name of the old topic- Returns:
- return the new topic name, if the regex applies. Otherwise, return null.
-
updateKeySchema
private org.apache.kafka.connect.data.Schema updateKeySchema(org.apache.kafka.connect.data.Schema oldKeySchema, String newTopicName)
-
updateKey
private org.apache.kafka.connect.data.Struct updateKey(org.apache.kafka.connect.data.Schema newKeySchema, org.apache.kafka.connect.data.Struct oldKey, String oldTopic)
-
updateEnvelopeSchema
private org.apache.kafka.connect.data.Schema updateEnvelopeSchema(org.apache.kafka.connect.data.Schema oldEnvelopeSchema, String newTopicName)
-
updateEnvelope
private org.apache.kafka.connect.data.Struct updateEnvelope(org.apache.kafka.connect.data.Schema newEnvelopeSchema, org.apache.kafka.connect.data.Struct oldEnvelope)
-
updateValue
private org.apache.kafka.connect.data.Struct updateValue(org.apache.kafka.connect.data.Schema newValueSchema, org.apache.kafka.connect.data.Struct oldValue)
-
copySchemaExcludingName
private org.apache.kafka.connect.data.SchemaBuilder copySchemaExcludingName(org.apache.kafka.connect.data.Schema source, org.apache.kafka.connect.data.SchemaBuilder builder)
-
copySchemaExcludingName
private org.apache.kafka.connect.data.SchemaBuilder copySchemaExcludingName(org.apache.kafka.connect.data.Schema source, org.apache.kafka.connect.data.SchemaBuilder builder, boolean copyFields)
-
-