org.apache.hadoop.fs.s3a.commit.magic

Class MagicS3GuardCommitter

java.lang.Object

org.apache.hadoop.mapreduce.OutputCommitter

org.apache.hadoop.mapreduce.lib.output.PathOutputCommitter

org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter

org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter

All Implemented Interfaces:

org.apache.hadoop.fs.statistics.IOStatisticsSource

@InterfaceAudience.Public
 @InterfaceStability.Unstable
public class MagicS3GuardCommitter
extends AbstractS3ACommitter

This is a dedicated committer which requires the "magic" directory feature of the S3A Filesystem to be enabled; it then uses paths for task and job attempts in magic paths, so as to ensure that the final output goes direct to the destination directory.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter

AbstractS3ACommitter.ActiveCommit, AbstractS3ACommitter.JobUUIDSource

Field Summary

Fields

Modifier and Type	Field and Description
static String	NAME Name: "magic".

Fields inherited from class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter

E_SELF_GENERATED_JOB_UUID, THREAD_PREFIX

Constructor Summary

Constructors

Constructor and Description
MagicS3GuardCommitter(org.apache.hadoop.fs.Path outputPath, org.apache.hadoop.mapreduce.TaskAttemptContext context) Create a task committer.

Method Summary

Modifier and Type	Method and Description
void	abortTask(org.apache.hadoop.mapreduce.TaskAttemptContext context) Abort a task.
void	cleanupStagingDirs() Delete the magic directory.
void	commitTask(org.apache.hadoop.mapreduce.TaskAttemptContext context)
protected org.apache.hadoop.fs.Path	getBaseTaskAttemptPath(org.apache.hadoop.mapreduce.TaskAttemptContext context) Compute the base path where the output of a task attempt is written.
protected org.apache.hadoop.fs.Path	getJobAttemptPath(int appAttemptId) Compute the path where the output of a given job attempt will be placed.
protected org.apache.hadoop.fs.Path	getJobPath() Compute the path under which all job attempts will be placed.
String	getName() Get the name of this committer.
org.apache.hadoop.fs.Path	getTaskAttemptPath(org.apache.hadoop.mapreduce.TaskAttemptContext context) Compute the path where the output of a task attempt is stored until that task is committed.
org.apache.hadoop.fs.Path	getTempTaskAttemptPath(org.apache.hadoop.mapreduce.TaskAttemptContext context) Get a temporary directory for data.
protected AbstractS3ACommitter.ActiveCommit	listPendingUploadsToCommit(CommitContext commitContext) Get the list of pending uploads for this job attempt, by listing all .pendingset files in the job attempt directory.
protected PendingSet	loadPendingCommits(org.apache.hadoop.mapreduce.TaskAttemptContext context) Loads pending commits from either memory or from the remote store (S3) based on the config.
boolean	needsTaskCommit(org.apache.hadoop.mapreduce.TaskAttemptContext context) Did this task write any files in the work directory? Probes for a task existing by looking to see if the attempt dir exists.
protected boolean	requiresDelayedCommitOutputInFileSystem() Require magic paths in the FS client.
void	setupJob(org.apache.hadoop.mapreduce.JobContext context) Base job setup (optionally) deletes the success marker and always creates the destination directory.
String	toString()

Methods inherited from class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter

abortJob, abortJobInternal, abortPendingUploads, abortPendingUploads, abortPendingUploadsInCleanup, buildJobUUID, cleanup, cleanupJob, commitJob, commitJobInternal, commitPendingUploads, deleteTaskAttemptPathQuietly, getAuditSpanSource, getCommitOperations, getConf, getDestFS, getDestinationFS, getDestS3AFS, getIOStatistics, getJobAttemptPath, getJobContext, getOutputPath, getRole, getTaskAttemptFilesystem, getUUID, getUUIDSource, getWorkPath, initiateJobOperation, initiateTaskOperation, initOutput, jobCompleted, maybeCreateSuccessMarker, maybeCreateSuccessMarkerFromCommits, maybeIgnore, maybeIgnore, precommitCheckPendingFiles, preCommitJob, recoverTask, setConf, setDestFS, setOutputPath, setupTask, setWorkPath, startOperation, updateCommonContext, warnOnActiveUploads

Methods inherited from class org.apache.hadoop.mapreduce.lib.output.PathOutputCommitter

hasOutputPath

Methods inherited from class org.apache.hadoop.mapreduce.OutputCommitter

isCommitJobRepeatable, isRecoverySupported, isRecoverySupported

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

NAME

public static final String NAME

Name: "magic".

See Also:

Constant Field Values

Constructor Detail

MagicS3GuardCommitter

public MagicS3GuardCommitter(org.apache.hadoop.fs.Path outputPath,
                             org.apache.hadoop.mapreduce.TaskAttemptContext context)
                      throws IOException

Create a task committer.

Parameters:

outputPath - the job's output path

context - the task's context

Throws:

IOException - on a failure

Method Detail

getName

public String getName()

Description copied from class: AbstractS3ACommitter

Get the name of this committer.

Specified by:

getName in class AbstractS3ACommitter

Returns:

the committer name.

requiresDelayedCommitOutputInFileSystem

protected boolean requiresDelayedCommitOutputInFileSystem()

Require magic paths in the FS client.

Overrides:

requiresDelayedCommitOutputInFileSystem in class AbstractS3ACommitter

Returns:

true, always.

setupJob

public void setupJob(org.apache.hadoop.mapreduce.JobContext context)
              throws IOException

Description copied from class: AbstractS3ACommitter

Base job setup (optionally) deletes the success marker and always creates the destination directory. When objects are committed that dest dir marker will inevitably be deleted; creating it now ensures there is something at the end while the job is in progress -and if nothing is created, that it is still there.

The option InternalCommitterConstants.FS_S3A_COMMITTER_UUID is set to the job UUID; if generated locally InternalCommitterConstants.SPARK_WRITE_UUID is also patched. The field AbstractS3ACommitter.jobSetup is set to true to note that this specific committer instance was used to set up a job.

Overrides:

setupJob in class AbstractS3ACommitter

Parameters:

context - context

Throws:

IOException - IO failure

listPendingUploadsToCommit

protected AbstractS3ACommitter.ActiveCommit listPendingUploadsToCommit(CommitContext commitContext)
                                                                throws IOException

Get the list of pending uploads for this job attempt, by listing all .pendingset files in the job attempt directory.

Specified by:

listPendingUploadsToCommit in class AbstractS3ACommitter

Parameters:

commitContext - job context

Returns:

a list of pending commits.

Throws:

IOException - Any IO failure

cleanupStagingDirs

public void cleanupStagingDirs()

Delete the magic directory.

Specified by:

cleanupStagingDirs in class AbstractS3ACommitter

needsTaskCommit

public boolean needsTaskCommit(org.apache.hadoop.mapreduce.TaskAttemptContext context)
                        throws IOException

Did this task write any files in the work directory? Probes for a task existing by looking to see if the attempt dir exists. This adds more HTTP requests to the call. It may be better just to return true and rely on the commit task doing the work.

Specified by:

needsTaskCommit in class org.apache.hadoop.mapreduce.OutputCommitter

Parameters:

context - the task's context

Returns:

true if the attempt path exists

Throws:

IOException - failure to list the path

commitTask

public void commitTask(org.apache.hadoop.mapreduce.TaskAttemptContext context)
                throws IOException

Specified by:

commitTask in class org.apache.hadoop.mapreduce.OutputCommitter

Throws:

IOException

loadPendingCommits

protected PendingSet loadPendingCommits(org.apache.hadoop.mapreduce.TaskAttemptContext context)
                                 throws IOException

Loads pending commits from either memory or from the remote store (S3) based on the config.

Parameters:

context - TaskAttemptContext

Returns:

All pending commit data for the given TaskAttemptContext

Throws:

IOException - if there is an error trying to read the commit data

abortTask

public void abortTask(org.apache.hadoop.mapreduce.TaskAttemptContext context)
               throws IOException

Abort a task. Attempt load then abort all pending files, then try to delete the task attempt path. This method may be called on the job committer, rather than the task one (such as in the MapReduce AM after a task container failure). It must extract all paths and state from the passed in context.

Specified by:

abortTask in class org.apache.hadoop.mapreduce.OutputCommitter

Parameters:

context - task context

Throws:

IOException - if there was some problem querying the path other than it not actually existing.

getJobPath

protected org.apache.hadoop.fs.Path getJobPath()

Compute the path under which all job attempts will be placed.

Specified by:

getJobPath in class AbstractS3ACommitter

Returns:

the path to store job attempt data.

getJobAttemptPath

protected final org.apache.hadoop.fs.Path getJobAttemptPath(int appAttemptId)

Compute the path where the output of a given job attempt will be placed. For the magic committer, the path includes the job UUID.

Specified by:

getJobAttemptPath in class AbstractS3ACommitter

Parameters:

appAttemptId - the ID of the application attempt for this job.

Returns:

the path to store job attempt data.

getTaskAttemptPath

public final org.apache.hadoop.fs.Path getTaskAttemptPath(org.apache.hadoop.mapreduce.TaskAttemptContext context)

Compute the path where the output of a task attempt is stored until that task is committed.

Overrides:

getTaskAttemptPath in class AbstractS3ACommitter

Parameters:

context - the context of the task attempt.

Returns:

the path where a task attempt should be stored.

getBaseTaskAttemptPath

protected final org.apache.hadoop.fs.Path getBaseTaskAttemptPath(org.apache.hadoop.mapreduce.TaskAttemptContext context)

Description copied from class: AbstractS3ACommitter

Compute the base path where the output of a task attempt is written. This is the path which will be deleted when a task is cleaned up and aborted.

Specified by:

getBaseTaskAttemptPath in class AbstractS3ACommitter

Parameters:

context - the context of the task attempt.

Returns:

the path where a task attempt should be stored.

getTempTaskAttemptPath

public org.apache.hadoop.fs.Path getTempTaskAttemptPath(org.apache.hadoop.mapreduce.TaskAttemptContext context)

Get a temporary directory for data. When a task is aborted/cleaned up, the contents of this directory are all deleted.

Specified by:

getTempTaskAttemptPath in class AbstractS3ACommitter

Parameters:

context - task context

Returns:

a path for temporary data.

toString

public String toString()

Overrides:

toString in class AbstractS3ACommitter