Class CloudEventsConverter
- java.lang.Object
-
- io.debezium.converters.CloudEventsConverter
-
- All Implemented Interfaces:
org.apache.kafka.connect.storage.Converter
public class CloudEventsConverter extends Object implements org.apache.kafka.connect.storage.Converter
Implementation of Converter that express schemas and objects with CloudEvents specification. The serialization format can be Json or Avro.The serialization format of CloudEvents is configured with
cloudevents.serializer.type
option.The serialization format of the data attribute in CloudEvents is configured with
cloudevents.data.serializer.type
option.Configuration options of the underlying converters can be passed through using the
json
andavro
prefixes, respectively.There are two modes for transferring CloudEvents as Kafka messages: structured and binary. In the structured content mode, event metadata attributes and event data are placed into the Kafka message value section using an event format. In the binary content mode, the value of the event data is placed into the Kafka message's value section as-is, with the content-type header value declaring its media type; all other event attributes are mapped to the Kafka message's header section.
Since Kafka converters has not support headers yet, right now CloudEvents converter use structured mode as the default.
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description static interface
CloudEventsConverter.CESchemaBuilder
Builder of a CloudEvents envelope schema.static interface
CloudEventsConverter.CEValueBuilder
Builder of a CloudEvents value.
-
Field Summary
Fields Modifier and Type Field Description private static String
APICURIO_AVRO_CONVERTER_CLASS
private static String
APICURIO_SCHEMA_REGISTRY_URL_CONFIG
private org.apache.kafka.connect.storage.Converter
avroConverter
private SerializerType
ceSerializerType
private static String
CONFLUENT_AVRO_CONVERTER_CLASS
Instantiated reflectively to avoid hard dependency to Avro converter.private static String
CONFLUENT_SCHEMA_REGISTRY_URL_CONFIG
private static Method
CONVERT_TO_CONNECT_METHOD
private static String
DATA_SCHEMA_SUFFIX
Suffix appended to schema names of data schema in case of Avro/Avro, to keep data schema and CE schema apart from each otherprivate SerializerType
dataSerializerType
private boolean
enableJsonSchemas
private static String
EXTENSION_NAME_PREFIX
private org.apache.kafka.connect.json.JsonConverter
jsonCloudEventsConverter
private org.apache.kafka.connect.json.JsonConverter
jsonDataConverter
private org.apache.kafka.connect.json.JsonDeserializer
jsonDeserializer
private static org.slf4j.Logger
LOGGER
private static Map<String,CloudEventsProvider>
providers
private List<String>
schemaRegistryUrls
private static String
TX_ATTRIBUTE_PREFIX
-
Constructor Summary
Constructors Constructor Description CloudEventsConverter()
CloudEventsConverter(org.apache.kafka.connect.storage.Converter avroConverter)
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description (package private) static String
adjustExtensionName(String original)
Adjust the name of CloudEvents attributes for Debezium events, following CloudEvents attribute naming convention as follows: prefixed withEXTENSION_NAME_PREFIX
CloudEvents attribute names MUST consist of lower-case letters ('a' to 'z') or digits ('0' to '9') from the ASCII character set, so any other characters are removedprivate void
ceSchemaFromSchema(org.apache.kafka.connect.data.Schema schema, CloudEventsConverter.CESchemaBuilder ceSchemaBuilder, Function<String,String> nameMapper, boolean alwaysOptional)
private void
ceValueFromStruct(org.apache.kafka.connect.data.Struct struct, org.apache.kafka.connect.data.Schema schema, CloudEventsConverter.CEValueBuilder ceValueBuilder, Function<String,String> nameMapper)
void
configure(Map<String,?> configs, boolean isKey)
private org.apache.kafka.connect.data.Schema
convertToCeExtensionSchema(org.apache.kafka.connect.data.Schema schema)
private org.apache.kafka.connect.data.Schema
convertToCeExtensionSchema(org.apache.kafka.connect.data.Schema schema, boolean alwaysOptional)
Converts the given source attribute schema into a corresponding CE extension schema.private org.apache.kafka.connect.data.SchemaAndValue
convertToCloudEventsFormat(RecordParser parser, CloudEventsMaker maker, org.apache.kafka.connect.data.Schema dataSchemaType, String dataSchema, Object serializedData)
private org.apache.kafka.connect.data.SchemaAndValue
convertToCloudEventsFormatWithDataAsAvro(String topic, RecordParser parser, CloudEventsMaker maker)
Creates a CloudEvents wrapper, converting the "data" to Avro.private static CloudEventsConverter.CESchemaBuilder
defineSchema()
byte[]
fromConnectData(String topic, org.apache.kafka.connect.data.Schema schema, Object value)
private String
getSchemaIdFromAvroMessage(byte[] serializedData)
Obtains the schema id from the given Avro record.private static boolean
isValidExtensionNameCharacter(char c)
private static CloudEventsProvider
lookupCloudEventsProvider(org.apache.kafka.connect.data.Struct record)
Lookup the CloudEventsProvider implementation for the source connector.private org.apache.kafka.connect.data.SchemaAndValue
reconvertData(String topic, byte[] serializedData, SerializerType dataType, Boolean enableSchemas)
org.apache.kafka.connect.data.SchemaAndValue
toConnectData(String topic, byte[] value)
private static String
txExtensionName(String name)
private static CloudEventsConverter.CEValueBuilder
withValue(org.apache.kafka.connect.data.Schema schema)
-
-
-
Field Detail
-
EXTENSION_NAME_PREFIX
private static final String EXTENSION_NAME_PREFIX
- See Also:
- Constant Field Values
-
TX_ATTRIBUTE_PREFIX
private static final String TX_ATTRIBUTE_PREFIX
- See Also:
- Constant Field Values
-
CONFLUENT_AVRO_CONVERTER_CLASS
private static final String CONFLUENT_AVRO_CONVERTER_CLASS
Instantiated reflectively to avoid hard dependency to Avro converter.- See Also:
- Constant Field Values
-
CONFLUENT_SCHEMA_REGISTRY_URL_CONFIG
private static final String CONFLUENT_SCHEMA_REGISTRY_URL_CONFIG
- See Also:
- Constant Field Values
-
APICURIO_AVRO_CONVERTER_CLASS
private static String APICURIO_AVRO_CONVERTER_CLASS
-
APICURIO_SCHEMA_REGISTRY_URL_CONFIG
private static final String APICURIO_SCHEMA_REGISTRY_URL_CONFIG
- See Also:
- Constant Field Values
-
DATA_SCHEMA_SUFFIX
private static final String DATA_SCHEMA_SUFFIX
Suffix appended to schema names of data schema in case of Avro/Avro, to keep data schema and CE schema apart from each other- See Also:
- Constant Field Values
-
LOGGER
private static final org.slf4j.Logger LOGGER
-
CONVERT_TO_CONNECT_METHOD
private static Method CONVERT_TO_CONNECT_METHOD
-
providers
@Immutable private static Map<String,CloudEventsProvider> providers
-
ceSerializerType
private SerializerType ceSerializerType
-
dataSerializerType
private SerializerType dataSerializerType
-
jsonCloudEventsConverter
private final org.apache.kafka.connect.json.JsonConverter jsonCloudEventsConverter
-
jsonDataConverter
private final org.apache.kafka.connect.json.JsonConverter jsonDataConverter
-
enableJsonSchemas
private boolean enableJsonSchemas
-
jsonDeserializer
private final org.apache.kafka.connect.json.JsonDeserializer jsonDeserializer
-
avroConverter
private org.apache.kafka.connect.storage.Converter avroConverter
-
-
Method Detail
-
configure
public void configure(Map<String,?> configs, boolean isKey)
- Specified by:
configure
in interfaceorg.apache.kafka.connect.storage.Converter
-
fromConnectData
public byte[] fromConnectData(String topic, org.apache.kafka.connect.data.Schema schema, Object value)
- Specified by:
fromConnectData
in interfaceorg.apache.kafka.connect.storage.Converter
-
lookupCloudEventsProvider
private static CloudEventsProvider lookupCloudEventsProvider(org.apache.kafka.connect.data.Struct record)
Lookup the CloudEventsProvider implementation for the source connector.
-
convertToCloudEventsFormatWithDataAsAvro
private org.apache.kafka.connect.data.SchemaAndValue convertToCloudEventsFormatWithDataAsAvro(String topic, RecordParser parser, CloudEventsMaker maker)
Creates a CloudEvents wrapper, converting the "data" to Avro.
-
getSchemaIdFromAvroMessage
private String getSchemaIdFromAvroMessage(byte[] serializedData)
Obtains the schema id from the given Avro record. They are prefixed by one magic byte, followed by an int for the schem id.
-
toConnectData
public org.apache.kafka.connect.data.SchemaAndValue toConnectData(String topic, byte[] value)
- Specified by:
toConnectData
in interfaceorg.apache.kafka.connect.storage.Converter
-
reconvertData
private org.apache.kafka.connect.data.SchemaAndValue reconvertData(String topic, byte[] serializedData, SerializerType dataType, Boolean enableSchemas)
-
convertToCloudEventsFormat
private org.apache.kafka.connect.data.SchemaAndValue convertToCloudEventsFormat(RecordParser parser, CloudEventsMaker maker, org.apache.kafka.connect.data.Schema dataSchemaType, String dataSchema, Object serializedData)
-
ceValueFromStruct
private void ceValueFromStruct(org.apache.kafka.connect.data.Struct struct, org.apache.kafka.connect.data.Schema schema, CloudEventsConverter.CEValueBuilder ceValueBuilder, Function<String,String> nameMapper)
-
ceSchemaFromSchema
private void ceSchemaFromSchema(org.apache.kafka.connect.data.Schema schema, CloudEventsConverter.CESchemaBuilder ceSchemaBuilder, Function<String,String> nameMapper, boolean alwaysOptional)
-
convertToCeExtensionSchema
private org.apache.kafka.connect.data.Schema convertToCeExtensionSchema(org.apache.kafka.connect.data.Schema schema, boolean alwaysOptional)
Converts the given source attribute schema into a corresponding CE extension schema. The types supported there are limited, e.g. int64 can only be represented as string.
-
convertToCeExtensionSchema
private org.apache.kafka.connect.data.Schema convertToCeExtensionSchema(org.apache.kafka.connect.data.Schema schema)
-
defineSchema
private static CloudEventsConverter.CESchemaBuilder defineSchema()
-
withValue
private static CloudEventsConverter.CEValueBuilder withValue(org.apache.kafka.connect.data.Schema schema)
-
adjustExtensionName
@VisibleForTesting static String adjustExtensionName(String original)
Adjust the name of CloudEvents attributes for Debezium events, following CloudEvents attribute naming convention as follows:- prefixed with
EXTENSION_NAME_PREFIX
- CloudEvents attribute names MUST consist of lower-case letters ('a' to 'z') or digits ('0' to '9') from the ASCII character set, so any other characters are removed
- Parameters:
original
- the original field name- Returns:
- the valid extension attribute name
- prefixed with
-
isValidExtensionNameCharacter
private static boolean isValidExtensionNameCharacter(char c)
-
-