Class ByLogicalTableRouter<R extends org.apache.kafka.connect.connector.ConnectRecord<R>>
- Type Parameters:
R
- the subtype ofConnectRecord
on which this transformation will operate
- All Implemented Interfaces:
Closeable
,AutoCloseable
,org.apache.kafka.common.Configurable
,org.apache.kafka.connect.transforms.Transformation<R>
This Transformation allows us to change a record's topic name and send change events from multiple physical tables to
one topic. For instance, we might choose to send the two tables from the above example to the topic
`db_shard.my_table`. The config options TOPIC_REGEX
and TOPIC_REPLACEMENT
are used
to change the record's topic.
Now that multiple physical tables can share a topic, the event's key may need to be augmented to include fields other
than just those for the record's primary/unique key, since these are not guaranteed to be unique across tables. We
need some identifier added to the key that distinguishes the different physical tables. The field name specified by
the config option KEY_FIELD_NAME
is added to the key schema for this purpose. By default, its value will
be the old topic name, but if a custom value is desired, the config options KEY_FIELD_REGEX
and
KEY_FIELD_REPLACEMENT
may be used to change it. For instance, in our above example, we might choose to
make the identifier `db_shard1` and `db_shard2` respectively.
- Author:
- David Leibovic, Mario Mueller
-
Field Summary
Modifier and TypeFieldDescriptionprivate org.apache.kafka.common.cache.Cache<org.apache.kafka.connect.data.Schema,
org.apache.kafka.connect.data.Schema> private static final Field
private static final Field
private static final Field
private static final Field
private boolean
private String
private Pattern
private String
private org.apache.kafka.common.cache.Cache<org.apache.kafka.connect.data.Schema,
org.apache.kafka.connect.data.Schema> private static final org.slf4j.Logger
private static final Field
private static final Field
private SchemaNameAdjuster
private SmtManager<R>
private static final Field
private static final Field
private Pattern
private String
-
Constructor Summary
-
Method Summary
Modifier and TypeMethodDescriptionvoid
close()
org.apache.kafka.common.config.ConfigDef
config()
void
private org.apache.kafka.connect.data.SchemaBuilder
copySchemaExcludingName
(org.apache.kafka.connect.data.Schema source, org.apache.kafka.connect.data.SchemaBuilder builder) private org.apache.kafka.connect.data.SchemaBuilder
copySchemaExcludingName
(org.apache.kafka.connect.data.Schema source, org.apache.kafka.connect.data.SchemaBuilder builder, boolean copyFields) private String
determineNewTopic
(String oldTopic) Determine the new topic name.private org.apache.kafka.connect.data.Struct
updateEnvelope
(org.apache.kafka.connect.data.Schema newEnvelopeSchema, org.apache.kafka.connect.data.Struct oldEnvelope) private org.apache.kafka.connect.data.Schema
updateEnvelopeSchema
(org.apache.kafka.connect.data.Schema oldEnvelopeSchema, String newTopicName) private org.apache.kafka.connect.data.Struct
updateKey
(org.apache.kafka.connect.data.Schema newKeySchema, org.apache.kafka.connect.data.Struct oldKey, String oldTopic) private org.apache.kafka.connect.data.Schema
updateKeySchema
(org.apache.kafka.connect.data.Schema oldKeySchema, String newTopicName) private org.apache.kafka.connect.data.Struct
updateValue
(org.apache.kafka.connect.data.Schema newValueSchema, org.apache.kafka.connect.data.Struct oldValue) private static int
validateKeyFieldReplacement
(Configuration config, Field field, Field.ValidationOutput problems) If KEY_FIELD_REGEX has a value that is really a regex, then the KEY_FIELD_REPLACEMENT must be a non-empty value.
-
Field Details
-
TOPIC_REGEX
-
TOPIC_REPLACEMENT
-
KEY_ENFORCE_UNIQUENESS
-
KEY_FIELD_REGEX
-
KEY_FIELD_NAME
-
KEY_FIELD_REPLACEMENT
-
SCHEMA_NAME_ADJUSTMENT_MODE
-
LOGICAL_TABLE_CACHE_SIZE
-
LOGGER
private static final org.slf4j.Logger LOGGER -
schemaNameAdjuster
-
topicRegex
-
topicReplacement
-
keyFieldRegex
-
keyEnforceUniqueness
private boolean keyEnforceUniqueness -
keyFieldReplacement
-
keyFieldName
-
keySchemaUpdateCache
private org.apache.kafka.common.cache.Cache<org.apache.kafka.connect.data.Schema,org.apache.kafka.connect.data.Schema> keySchemaUpdateCache -
envelopeSchemaUpdateCache
private org.apache.kafka.common.cache.Cache<org.apache.kafka.connect.data.Schema,org.apache.kafka.connect.data.Schema> envelopeSchemaUpdateCache -
keyRegexReplaceCache
-
topicRegexReplaceCache
-
smtManager
-
-
Constructor Details
-
ByLogicalTableRouter
public ByLogicalTableRouter()
-
-
Method Details
-
validateKeyFieldReplacement
private static int validateKeyFieldReplacement(Configuration config, Field field, Field.ValidationOutput problems) If KEY_FIELD_REGEX has a value that is really a regex, then the KEY_FIELD_REPLACEMENT must be a non-empty value. -
configure
- Specified by:
configure
in interfaceorg.apache.kafka.common.Configurable
-
apply
-
close
public void close() -
config
public org.apache.kafka.common.config.ConfigDef config() -
determineNewTopic
Determine the new topic name.- Parameters:
oldTopic
- the name of the old topic- Returns:
- return the new topic name, if the regex applies. Otherwise, return null.
-
updateKeySchema
private org.apache.kafka.connect.data.Schema updateKeySchema(org.apache.kafka.connect.data.Schema oldKeySchema, String newTopicName) -
updateKey
private org.apache.kafka.connect.data.Struct updateKey(org.apache.kafka.connect.data.Schema newKeySchema, org.apache.kafka.connect.data.Struct oldKey, String oldTopic) -
updateEnvelopeSchema
private org.apache.kafka.connect.data.Schema updateEnvelopeSchema(org.apache.kafka.connect.data.Schema oldEnvelopeSchema, String newTopicName) -
updateEnvelope
private org.apache.kafka.connect.data.Struct updateEnvelope(org.apache.kafka.connect.data.Schema newEnvelopeSchema, org.apache.kafka.connect.data.Struct oldEnvelope) -
updateValue
private org.apache.kafka.connect.data.Struct updateValue(org.apache.kafka.connect.data.Schema newValueSchema, org.apache.kafka.connect.data.Struct oldValue) -
copySchemaExcludingName
private org.apache.kafka.connect.data.SchemaBuilder copySchemaExcludingName(org.apache.kafka.connect.data.Schema source, org.apache.kafka.connect.data.SchemaBuilder builder) -
copySchemaExcludingName
private org.apache.kafka.connect.data.SchemaBuilder copySchemaExcludingName(org.apache.kafka.connect.data.Schema source, org.apache.kafka.connect.data.SchemaBuilder builder, boolean copyFields)
-