it.agilelab.bigdata.wasp.consumers.spark.plugins.kafka
Creates a streaming DataFrame from a Kafka streaming source.
Creates a streaming DataFrame from a Kafka streaming source.
If all the input topics share the same schema the returned DataFrame will contain a column named "kafkaMetadata" with message metadata and the message contents either as a single column named "value" or as multiple columns named after the value fields depending on the topic datatype. If the input topics do not share the same schema the returned Dataframe will contain a column named "kafkaMetadata" with message metadata and each topic content on a column named after the topic name, previously escaped calling the function MultiTopicModel.topicNameToColumnName(). This means that if 5 topic models with different schema are read, the output dataframe will contain 6 columns, and of these 6 columns only the kafkaMetadata and the topic related to that message one, will have a value different from null, like the following:
+--------------------+--------------------+-------------------------+ | kafkaMetadata| test_json_topic|testcheckpoint_avro_topic| +--------------------+--------------------+-------------------------+ |[45, [], test_jso...|[45, 45, [field1_...| null| |[12, [], testchec...| null| [12, 77, [field1_..| +--------------------+--------------------+-------------------------+
The "kafkaMetadata" column contains the following: - key: bytes - headers: array of {headerKey: string, headerValue: bytes} - topic: string - partition: int - offset: long - timestamp: timestamp - timestampType: int
The behaviour for message contents column(s) is the following: - the "avro" and "json" topic data types will output the columns specified by their schemas - the "plaintext" and "bytes" topic data types output a "value" column with the contents as string or bytes respectively