Returns the current caching storage level.
Turns off caching that happens every time a checkpoint is generated.
Turns on caching that happens every time a checkpoint is generated (default behavior).
Turns on caching that happens every time a checkpoint is generated (default behavior).
Atum uses the default storage level for Spark 2.4 and later. But you can specify a different Spark storage level to use for caching. (See the definition of org.apache.spark.storage.StorageLevel for the details).
It can be one of the following: NONE, DISK_ONLY, DISK_ONLY_2, MEMORY_ONLY, MEMORY_ONLY_2, MEMORY_ONLY_SER, MEMORY_ONLY_SER_2, MEMORY_AND_DISK, MEMORY_AND_DISK_2, MEMORY_AND_DISK_SER, MEMORY_AND_DISK_SER_2, MEMORY_AND_DISK_SER_2, OFF_HEAP.
A caching storage level.
Returns the current control measures object containing all the checkpoints up to the current point.
Sets an additional info in metadata of the _INFO file
Allows Atum to automatically unpersist all cached datasets it creates before calculating a checkpoint.
Allows Atum to automatically unpersist all cached datasets it creates before calculating a checkpoint. This may lead to performance improvement,
This feature is experimental and so it is disabled by default.
Same as enableCaching()
with cacheStorageLevel
being a mandatory parameter.
Same as enableCaching()
with cacheStorageLevel
being a mandatory parameter.
The storage level is passed as a String
here for forward compatibility.
Same as enableCaching()
with cacheStorageLevel
being a mandatory parameter.
Set's an unique id of a run stored in the _INFO file
Returns the current control measures object containing all the checkpoints up to the current point.
Returns the current control measures object containing all the checkpoints up to the current point.
This method is deprecated. Please use getControlMeasure