Definition of date partition column to extract formatted timestamp into column.
DataObject of type KafkaTopic.
DataObject of type KafkaTopic. Provides details to an action to read from Kafka Topics using either org.apache.spark.sql.DataFrameReader or org.apache.spark.sql.streaming.DataStreamReader
The name of the topic to read
Optional type the key column should be converted to. If none is given it will remain a bytearray / binary.
Optional type the value column should be converted to. If none is given it will remain a bytearray / binary.
An optional, minimal schema that this DataObject must have to pass schema validation on reading and writing.
Columns to be selected when reading the DataFrame. Available columns are key, value, topic, partition, offset, timestamp, timestampType. If key/valueType is AvroSchemaRegistry the key/value column are convert to a complex type according to the avro schema. To expand it select "value.*". Default is to select key and value.
definition of date partition column to extract formatted timestamp into column. This is used to list existing partition and is added as additional column on batch read.
Set to true if consecutive partitions should be combined as one range of offsets when batch reading from topic. This results in less tasks but can be a performance problem when reading many partitions. (default=false)
Set number of offsets per Spark task when batch reading from topic.
Options for the Kafka stream reader (see https://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html). These options override connection.kafkaOptions.
Definition of date partition column to extract formatted timestamp into column.
date partition column name to extract timestamp into column on batch read
time format for timestamp in date partition column, definition according to java DateTimeFormatter. Default is "yyyyMMdd".
time unit for timestamp in date partition column, definition according to java ChronoUnit. Default is "days".
time zone used for date logic. If not specified, java system default is used.