Class CloudEventsConverter
- All Implemented Interfaces:
org.apache.kafka.connect.storage.Converter
The serialization format of CloudEvents is configured with
cloudevents.serializer.type
option.
The serialization format of the data attribute in CloudEvents is configured with
cloudevents.data.serializer.type
option.
Configuration options of the underlying converters can be passed through using the json
and avro
prefixes, respectively.
There are two modes for transferring CloudEvents as Kafka messages: structured and binary. In the structured content mode, event metadata attributes and event data are placed into the Kafka message value section using an event format. In the binary content mode, the value of the event data is placed into the Kafka message's value section as-is, with the content-type header value declaring its media type; all other event attributes are mapped to the Kafka message's header section.
Since Kafka converters has not support headers yet, right now CloudEvents converter use structured mode as the default.
-
Nested Class Summary
Modifier and TypeClassDescriptionstatic interface
Builder of a CloudEvents envelope schema.static interface
Builder of a CloudEvents value. -
Field Summary
Modifier and TypeFieldDescriptionprivate static String
private static final String
private org.apache.kafka.connect.storage.Converter
private SerializerType
private static final String
Instantiated reflectively to avoid hard dependency to Avro converter.private static final String
private static Method
private static final String
Suffix appended to schema names of data schema in case of Avro/Avro, to keep data schema and CE schema apart from each otherprivate SerializerType
private boolean
private static final String
private final org.apache.kafka.connect.json.JsonConverter
private final org.apache.kafka.connect.json.JsonConverter
private final org.apache.kafka.connect.json.JsonDeserializer
private static final org.slf4j.Logger
private static Map<String,
CloudEventsProvider> private SchemaNameAdjuster
private static final String
-
Constructor Summary
ConstructorDescriptionCloudEventsConverter
(org.apache.kafka.connect.storage.Converter avroConverter) -
Method Summary
Modifier and TypeMethodDescription(package private) static String
adjustExtensionName
(String original) Adjust the name of CloudEvents attributes for Debezium events, following CloudEvents attribute naming convention as follows: prefixed withEXTENSION_NAME_PREFIX
CloudEvents attribute names MUST consist of lower-case letters ('a' to 'z') or digits ('0' to '9') from the ASCII character set, so any other characters are removedprivate void
ceSchemaFromSchema
(org.apache.kafka.connect.data.Schema schema, CloudEventsConverter.CESchemaBuilder ceSchemaBuilder, Function<String, String> nameMapper, boolean alwaysOptional) private void
ceValueFromStruct
(org.apache.kafka.connect.data.Struct struct, org.apache.kafka.connect.data.Schema schema, CloudEventsConverter.CEValueBuilder ceValueBuilder, Function<String, String> nameMapper) void
private org.apache.kafka.connect.data.Schema
convertToCeExtensionSchema
(org.apache.kafka.connect.data.Schema schema) private org.apache.kafka.connect.data.Schema
convertToCeExtensionSchema
(org.apache.kafka.connect.data.Schema schema, boolean alwaysOptional) Converts the given source attribute schema into a corresponding CE extension schema.private org.apache.kafka.connect.data.SchemaAndValue
convertToCloudEventsFormat
(RecordParser parser, CloudEventsMaker maker, org.apache.kafka.connect.data.Schema dataSchemaType, String dataSchema, Object serializedData) private org.apache.kafka.connect.data.SchemaAndValue
convertToCloudEventsFormatWithDataAsAvro
(String topic, RecordParser parser, CloudEventsMaker maker) Creates a CloudEvents wrapper, converting the "data" to Avro.private static CloudEventsConverter.CESchemaBuilder
byte[]
fromConnectData
(String topic, org.apache.kafka.connect.data.Schema schema, Object value) private String
getSchemaIdFromAvroMessage
(byte[] serializedData) Obtains the schema id from the given Avro record.private static boolean
isValidExtensionNameCharacter
(char c) private static CloudEventsProvider
lookupCloudEventsProvider
(org.apache.kafka.connect.data.Struct record) Lookup the CloudEventsProvider implementation for the source connector.private org.apache.kafka.connect.data.SchemaAndValue
reconvertData
(String topic, byte[] serializedData, SerializerType dataType, Boolean enableSchemas) org.apache.kafka.connect.data.SchemaAndValue
toConnectData
(String topic, byte[] value) private static String
txExtensionName
(String name) private static CloudEventsConverter.CEValueBuilder
withValue
(org.apache.kafka.connect.data.Schema schema) Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
Methods inherited from interface org.apache.kafka.connect.storage.Converter
config, fromConnectData, toConnectData
-
Field Details
-
EXTENSION_NAME_PREFIX
- See Also:
-
TX_ATTRIBUTE_PREFIX
- See Also:
-
CONFLUENT_AVRO_CONVERTER_CLASS
Instantiated reflectively to avoid hard dependency to Avro converter.- See Also:
-
CONFLUENT_SCHEMA_REGISTRY_URL_CONFIG
- See Also:
-
APICURIO_AVRO_CONVERTER_CLASS
-
APICURIO_SCHEMA_REGISTRY_URL_CONFIG
- See Also:
-
DATA_SCHEMA_SUFFIX
Suffix appended to schema names of data schema in case of Avro/Avro, to keep data schema and CE schema apart from each other- See Also:
-
LOGGER
private static final org.slf4j.Logger LOGGER -
CONVERT_TO_CONNECT_METHOD
-
providers
-
ceSerializerType
-
dataSerializerType
-
jsonCloudEventsConverter
private final org.apache.kafka.connect.json.JsonConverter jsonCloudEventsConverter -
jsonDataConverter
private final org.apache.kafka.connect.json.JsonConverter jsonDataConverter -
enableJsonSchemas
private boolean enableJsonSchemas -
jsonDeserializer
private final org.apache.kafka.connect.json.JsonDeserializer jsonDeserializer -
avroConverter
private org.apache.kafka.connect.storage.Converter avroConverter -
schemaRegistryUrls
-
schemaNameAdjuster
-
-
Constructor Details
-
CloudEventsConverter
public CloudEventsConverter() -
CloudEventsConverter
public CloudEventsConverter(org.apache.kafka.connect.storage.Converter avroConverter)
-
-
Method Details
-
configure
- Specified by:
configure
in interfaceorg.apache.kafka.connect.storage.Converter
-
fromConnectData
public byte[] fromConnectData(String topic, org.apache.kafka.connect.data.Schema schema, Object value) - Specified by:
fromConnectData
in interfaceorg.apache.kafka.connect.storage.Converter
-
lookupCloudEventsProvider
private static CloudEventsProvider lookupCloudEventsProvider(org.apache.kafka.connect.data.Struct record) Lookup the CloudEventsProvider implementation for the source connector. -
convertToCloudEventsFormatWithDataAsAvro
private org.apache.kafka.connect.data.SchemaAndValue convertToCloudEventsFormatWithDataAsAvro(String topic, RecordParser parser, CloudEventsMaker maker) Creates a CloudEvents wrapper, converting the "data" to Avro. -
getSchemaIdFromAvroMessage
Obtains the schema id from the given Avro record. They are prefixed by one magic byte, followed by an int for the schem id. -
toConnectData
- Specified by:
toConnectData
in interfaceorg.apache.kafka.connect.storage.Converter
-
reconvertData
private org.apache.kafka.connect.data.SchemaAndValue reconvertData(String topic, byte[] serializedData, SerializerType dataType, Boolean enableSchemas) -
convertToCloudEventsFormat
private org.apache.kafka.connect.data.SchemaAndValue convertToCloudEventsFormat(RecordParser parser, CloudEventsMaker maker, org.apache.kafka.connect.data.Schema dataSchemaType, String dataSchema, Object serializedData) -
ceValueFromStruct
private void ceValueFromStruct(org.apache.kafka.connect.data.Struct struct, org.apache.kafka.connect.data.Schema schema, CloudEventsConverter.CEValueBuilder ceValueBuilder, Function<String, String> nameMapper) -
ceSchemaFromSchema
private void ceSchemaFromSchema(org.apache.kafka.connect.data.Schema schema, CloudEventsConverter.CESchemaBuilder ceSchemaBuilder, Function<String, String> nameMapper, boolean alwaysOptional) -
convertToCeExtensionSchema
private org.apache.kafka.connect.data.Schema convertToCeExtensionSchema(org.apache.kafka.connect.data.Schema schema, boolean alwaysOptional) Converts the given source attribute schema into a corresponding CE extension schema. The types supported there are limited, e.g. int64 can only be represented as string. -
convertToCeExtensionSchema
private org.apache.kafka.connect.data.Schema convertToCeExtensionSchema(org.apache.kafka.connect.data.Schema schema) -
defineSchema
-
withValue
private static CloudEventsConverter.CEValueBuilder withValue(org.apache.kafka.connect.data.Schema schema) -
adjustExtensionName
Adjust the name of CloudEvents attributes for Debezium events, following CloudEvents attribute naming convention as follows:- prefixed with
EXTENSION_NAME_PREFIX
- CloudEvents attribute names MUST consist of lower-case letters ('a' to 'z') or digits ('0' to '9') from the ASCII character set, so any other characters are removed
- Parameters:
original
- the original field name- Returns:
- the valid extension attribute name
- prefixed with
-
txExtensionName
-
isValidExtensionNameCharacter
private static boolean isValidExtensionNameCharacter(char c)
-