Purpose of LogFlattenerJob is to unpack serialized Avro data from online requests and flatten each field
(both keys and values) into individual columns and save to an offline "flattened" log table.
Steps:
1. determine unfilled range and pull raw logs from partitioned log table
2. fetch joinCodecs for all unique schema_hash present in the logs
3. build a merged schema from all schema versions, which will be used as output schema
4. unpack each row and adhere to the output schema
5. save the schema info in the flattened log table properties (cumulatively)
Purpose of LogFlattenerJob is to unpack serialized Avro data from online requests and flatten each field (both keys and values) into individual columns and save to an offline "flattened" log table.
Steps: 1. determine unfilled range and pull raw logs from partitioned log table 2. fetch joinCodecs for all unique schema_hash present in the logs 3. build a merged schema from all schema versions, which will be used as output schema 4. unpack each row and adhere to the output schema 5. save the schema info in the flattened log table properties (cumulatively)