Package com.exasol.spark.s3

Class S3CleanupListener

java.lang.Object

org.apache.spark.scheduler.SparkListener

com.exasol.spark.s3.S3CleanupListener

All Implemented Interfaces:

org.apache.spark.scheduler.SparkListenerInterface

public final class S3CleanupListener
extends org.apache.spark.scheduler.SparkListener

A SparkListener class that cleans up S3 intermediate location at the end of job run.

Constructor Summary

Constructors

Constructor	Description
S3CleanupListener(com.exasol.spark.common.ExasolOptions options, String bucketKey)	Creates an instance of S3CleanupListener.

Method Summary

Modifier and Type	Method	Description
void	onJobEnd(org.apache.spark.scheduler.SparkListenerJobEnd jobEnd)

Methods inherited from class org.apache.spark.scheduler.SparkListener

onApplicationEnd, onApplicationStart, onBlockManagerAdded, onBlockManagerRemoved, onBlockUpdated, onEnvironmentUpdate, onExecutorAdded, onExecutorBlacklisted, onExecutorBlacklistedForStage, onExecutorExcluded, onExecutorExcludedForStage, onExecutorMetricsUpdate, onExecutorRemoved, onExecutorUnblacklisted, onExecutorUnexcluded, onJobStart, onNodeBlacklisted, onNodeBlacklistedForStage, onNodeExcluded, onNodeExcludedForStage, onNodeUnblacklisted, onNodeUnexcluded, onOtherEvent, onResourceProfileAdded, onSpeculativeTaskSubmitted, onStageCompleted, onStageExecutorMetrics, onStageSubmitted, onTaskEnd, onTaskGettingResult, onTaskStart, onUnpersistRDD, onUnschedulableTaskSetAdded, onUnschedulableTaskSetRemoved

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

S3CleanupListener

public S3CleanupListener(com.exasol.spark.common.ExasolOptions options,
                         String bucketKey)

Creates an instance of S3CleanupListener.

Parameters:

options - user provided options

bucketKey - bucketKey inside the user provided bucket

Method Detail

onJobEnd

public void onJobEnd(org.apache.spark.scheduler.SparkListenerJobEnd jobEnd)

Specified by:

onJobEnd in interface org.apache.spark.scheduler.SparkListenerInterface

Overrides:

onJobEnd in class org.apache.spark.scheduler.SparkListener