Package io.debezium.transforms

Class ByLogicalTableRouter<R extends org.apache.kafka.connect.connector.ConnectRecord<R>>

java.lang.Object
io.debezium.transforms.ByLogicalTableRouter<R>
Type Parameters:
R - the subtype of ConnectRecord on which this transformation will operate
All Implemented Interfaces:
Closeable, AutoCloseable, org.apache.kafka.common.Configurable, org.apache.kafka.connect.transforms.Transformation<R>
public class ByLogicalTableRouter<R extends org.apache.kafka.connect.connector.ConnectRecord<R>> extends Object implements org.apache.kafka.connect.transforms.Transformation<R>
A logical table consists of one or more physical tables with the same schema. A common use case is sharding -- the two physical tables `db_shard1.my_table` and `db_shard2.my_table` together form one logical table.

This Transformation allows us to change a record's topic name and send change events from multiple physical tables to one topic. For instance, we might choose to send the two tables from the above example to the topic `db_shard.my_table`. The config options TOPIC_REGEX and TOPIC_REPLACEMENT are used to change the record's topic.

Now that multiple physical tables can share a topic, the event's key may need to be augmented to include fields other than just those for the record's primary/unique key, since these are not guaranteed to be unique across tables. We need some identifier added to the key that distinguishes the different physical tables. The field name specified by the config option KEY_FIELD_NAME is added to the key schema for this purpose. By default, its value will be the old topic name, but if a custom value is desired, the config options KEY_FIELD_REGEX and KEY_FIELD_REPLACEMENT may be used to change it. For instance, in our above example, we might choose to make the identifier `db_shard1` and `db_shard2` respectively.

Author:
David Leibovic, Mario Mueller

  • Field Details

    • TOPIC_REGEX

      private static final Field TOPIC_REGEX

    • TOPIC_REPLACEMENT

      private static final Field TOPIC_REPLACEMENT

    • KEY_ENFORCE_UNIQUENESS

      private static final Field KEY_ENFORCE_UNIQUENESS

    • KEY_FIELD_REGEX

      private static final Field KEY_FIELD_REGEX

    • KEY_FIELD_NAME

      private static final Field KEY_FIELD_NAME

    • KEY_FIELD_REPLACEMENT

      private static final Field KEY_FIELD_REPLACEMENT

    • SCHEMA_NAME_ADJUSTMENT_MODE

      private static final Field SCHEMA_NAME_ADJUSTMENT_MODE

    • LOGICAL_TABLE_CACHE_SIZE

      private static final Field LOGICAL_TABLE_CACHE_SIZE

    • LOGGER

      private static final org.slf4j.Logger LOGGER

    • schemaNameAdjuster

      private SchemaNameAdjuster schemaNameAdjuster

    • topicRegex

      private Pattern topicRegex

    • topicReplacement

      private String topicReplacement

    • keyFieldRegex

      private Pattern keyFieldRegex

    • keyEnforceUniqueness

      private boolean keyEnforceUniqueness

    • keyFieldReplacement

      private String keyFieldReplacement

    • keyFieldName

      private String keyFieldName

    • keySchemaUpdateCache

      private org.apache.kafka.common.cache.Cache<org.apache.kafka.connect.data.Schema,org.apache.kafka.connect.data.Schema> keySchemaUpdateCache

    • envelopeSchemaUpdateCache

      private org.apache.kafka.common.cache.Cache<org.apache.kafka.connect.data.Schema,org.apache.kafka.connect.data.Schema> envelopeSchemaUpdateCache

    • keyRegexReplaceCache

      private org.apache.kafka.common.cache.Cache<String,String> keyRegexReplaceCache

    • topicRegexReplaceCache

      private org.apache.kafka.common.cache.Cache<String,String> topicRegexReplaceCache

    • smtManager

      private SmtManager<R extends org.apache.kafka.connect.connector.ConnectRecord<R>> smtManager

  • Constructor Details

    • ByLogicalTableRouter

      public ByLogicalTableRouter()

  • Method Details

    • validateKeyFieldReplacement

      private static int validateKeyFieldReplacement(Configuration config, Field field, Field.ValidationOutput problems)
      If KEY_FIELD_REGEX has a value that is really a regex, then the KEY_FIELD_REPLACEMENT must be a non-empty value.

    • configure

      public void configure(Map<String,?> props)
      Specified by:
      configure in interface org.apache.kafka.common.Configurable

    • apply

      public R apply(R record)
      Specified by:
      apply in interface org.apache.kafka.connect.transforms.Transformation<R extends org.apache.kafka.connect.connector.ConnectRecord<R>>

    • close

      public void close()
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable
      Specified by:
      close in interface org.apache.kafka.connect.transforms.Transformation<R extends org.apache.kafka.connect.connector.ConnectRecord<R>>

    • config

      public org.apache.kafka.common.config.ConfigDef config()
      Specified by:
      config in interface org.apache.kafka.connect.transforms.Transformation<R extends org.apache.kafka.connect.connector.ConnectRecord<R>>

    • determineNewTopic

      private String determineNewTopic(String oldTopic)
      Determine the new topic name.
      Parameters:
      oldTopic - the name of the old topic
      Returns:
      return the new topic name, if the regex applies. Otherwise, return null.

    • updateKeySchema

      private org.apache.kafka.connect.data.Schema updateKeySchema(org.apache.kafka.connect.data.Schema oldKeySchema, String newTopicName)

    • updateKey

      private org.apache.kafka.connect.data.Struct updateKey(org.apache.kafka.connect.data.Schema newKeySchema, org.apache.kafka.connect.data.Struct oldKey, String oldTopic)

    • updateEnvelopeSchema

      private org.apache.kafka.connect.data.Schema updateEnvelopeSchema(org.apache.kafka.connect.data.Schema oldEnvelopeSchema, String newTopicName)

    • updateEnvelope

      private org.apache.kafka.connect.data.Struct updateEnvelope(org.apache.kafka.connect.data.Schema newEnvelopeSchema, org.apache.kafka.connect.data.Struct oldEnvelope)

    • updateValue

      private org.apache.kafka.connect.data.Struct updateValue(org.apache.kafka.connect.data.Schema newValueSchema, org.apache.kafka.connect.data.Struct oldValue)

    • copySchemaExcludingName

      private org.apache.kafka.connect.data.SchemaBuilder copySchemaExcludingName(org.apache.kafka.connect.data.Schema source, org.apache.kafka.connect.data.SchemaBuilder builder)

    • copySchemaExcludingName

      private org.apache.kafka.connect.data.SchemaBuilder copySchemaExcludingName(org.apache.kafka.connect.data.Schema source, org.apache.kafka.connect.data.SchemaBuilder builder, boolean copyFields)