it.agilelab.bigdata.wasp.consumers.spark.strategies.cdc
Eventual enrichment function of the final DF.
Eventual enrichment function of the final DF.
final output DF
Eventual preparation function of the initial DF an example can be the removal of fields that are not required to be mapped in the table or a mapping with a default value.
Eventual preparation function of the initial DF an example can be the removal of fields that are not required to be mapped in the table or a mapping with a default value.
final output DF
Strategy that read data from Kafka, transform the format to make it compliant with the DataLake format that the writer expect.
Strategy that read data from Kafka, transform the format to make it compliant with the DataLake format that the writer expect.
the dataframe that need to be transformed
a dataframe transformed that can be sent to the CDCWriter
Strategy that enable to map a flat mutation model to be mapped to an insert/update/delete object that can be sent to the CDC plugin that writes on DeltaLake. So having has input the raw flat mutations coming from a goldengate topic it will produce in output a dataframe composed of rows that has the shape accepted in input by the cdc plugin.
NB:
in this case you need to insert the configuration the following line:
goldengate.key.fields=["CUST_CODE", "ORDER_DATE", "PRODUCT_CODE", "ORDER_ID"]"