Close the writer
Close the writer
an error if the close fails or unit
Write data incrementally
Write data incrementally
the data to be written
the current progress of processing in the subject file
the writer instance or an error message with the checkpoint if the write fails
Incremental functional writer class for parquet
Overview
Allows large files to be written to parquet in smaller incements. Closes the underlying writer and creates a new file at a defined writer count.
Also keeps a checkpoint of when the last file was sucessfully flushed to the filesystem so in the event of an error processing can be restarted from that point.
type of data to be written