org.apache.hadoop.fs.s3a.commit.magic

Class MagicS3GuardCommitter

java.lang.Object

org.apache.hadoop.mapreduce.OutputCommitter

org.apache.hadoop.mapreduce.lib.output.PathOutputCommitter

org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter

org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter

@InterfaceAudience.Public
 @InterfaceStability.Unstable
public class MagicS3GuardCommitter
extends AbstractS3ACommitter

This is a dedicated committer which requires the "magic" directory feature of the S3A Filesystem to be enabled; it then uses paths for task and job attempts in magic paths, so as to ensure that the final output goes direct to the destination directory.

Field Summary

Fields

Modifier and Type	Field and Description
static String	NAME Name: "magic".

Constructor Summary

Constructors

Constructor and Description
MagicS3GuardCommitter(org.apache.hadoop.fs.Path outputPath, org.apache.hadoop.mapreduce.TaskAttemptContext context) Create a task committer.

Method Summary

Modifier and Type	Method and Description
void	abortTask(org.apache.hadoop.mapreduce.TaskAttemptContext context) Abort a task.
void	cleanupStagingDirs() Delete the magic directory.
void	commitTask(org.apache.hadoop.mapreduce.TaskAttemptContext context)
protected org.apache.hadoop.fs.Path	getBaseTaskAttemptPath(org.apache.hadoop.mapreduce.TaskAttemptContext context) Compute the base path where the output of a task attempt is written.
protected org.apache.hadoop.fs.Path	getJobAttemptPath(int appAttemptId) Compute the path where the output of a given job attempt will be placed.
String	getName() Get the name of this committer.
org.apache.hadoop.fs.Path	getTaskAttemptPath(org.apache.hadoop.mapreduce.TaskAttemptContext context) Compute the path where the output of a task attempt is stored until that task is committed.
org.apache.hadoop.fs.Path	getTempTaskAttemptPath(org.apache.hadoop.mapreduce.TaskAttemptContext context) Get a temporary directory for data.
protected List<SinglePendingCommit>	listPendingUploadsToCommit(org.apache.hadoop.mapreduce.JobContext context) Get the list of pending uploads for this job attempt, by listing all .pendingset files in the job attempt directory.
boolean	needsTaskCommit(org.apache.hadoop.mapreduce.TaskAttemptContext context) Did this task write any files in the work directory? Probes for a task existing by looking to see if the attempt dir exists.
protected boolean	requiresDelayedCommitOutputInFileSystem() Require magic paths in the FS client.
void	setupJob(org.apache.hadoop.mapreduce.JobContext context)
void	setupTask(org.apache.hadoop.mapreduce.TaskAttemptContext context) Base job setup deletes the success marker.
String	toString()

Methods inherited from class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter

abortJob, abortJobInternal, abortPendingUploads, abortPendingUploadsInCleanup, buildThreadPool, cleanup, cleanupJob, commitJob, commitJobInternal, commitPendingUploads, deleteTaskAttemptPathQuietly, getCommitOperations, getConf, getDestFS, getDestinationFS, getDestS3AFS, getJobAttemptPath, getJobContext, getOutputPath, getRole, getTaskAttemptFilesystem, getWorkPath, initOutput, jobCompleted, loadPendingsetFiles, maybeCreateSuccessMarker, maybeCreateSuccessMarkerFromCommits, maybeIgnore, maybeIgnore, preCommitJob, recoverTask, setConf, setDestFS, setOutputPath, setWorkPath

Methods inherited from class org.apache.hadoop.mapreduce.lib.output.PathOutputCommitter

hasOutputPath

Methods inherited from class org.apache.hadoop.mapreduce.OutputCommitter

isCommitJobRepeatable, isRecoverySupported, isRecoverySupported

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

NAME

public static final String NAME

Name: "magic".

See Also:

Constant Field Values

Constructor Detail

MagicS3GuardCommitter

public MagicS3GuardCommitter(org.apache.hadoop.fs.Path outputPath,
                             org.apache.hadoop.mapreduce.TaskAttemptContext context)
                      throws IOException

Create a task committer.

Parameters:

outputPath - the job's output path

context - the task's context

Throws:

IOException - on a failure

Method Detail

getName

public String getName()

Description copied from class: AbstractS3ACommitter

Get the name of this committer.

Specified by:

getName in class AbstractS3ACommitter

Returns:

the committer name.

requiresDelayedCommitOutputInFileSystem

protected boolean requiresDelayedCommitOutputInFileSystem()

Require magic paths in the FS client.

Overrides:

requiresDelayedCommitOutputInFileSystem in class AbstractS3ACommitter

Returns:

true, always.

setupJob

public void setupJob(org.apache.hadoop.mapreduce.JobContext context)
              throws IOException

Specified by:

setupJob in class org.apache.hadoop.mapreduce.OutputCommitter

Throws:

IOException

listPendingUploadsToCommit

protected List<SinglePendingCommit> listPendingUploadsToCommit(org.apache.hadoop.mapreduce.JobContext context)
                                                        throws IOException

Get the list of pending uploads for this job attempt, by listing all .pendingset files in the job attempt directory.

Specified by:

listPendingUploadsToCommit in class AbstractS3ACommitter

Parameters:

context - job context

Returns:

a list of pending commits.

Throws:

IOException - Any IO failure

cleanupStagingDirs

public void cleanupStagingDirs()

Delete the magic directory.

Specified by:

cleanupStagingDirs in class AbstractS3ACommitter

setupTask

public void setupTask(org.apache.hadoop.mapreduce.TaskAttemptContext context)
               throws IOException

Description copied from class: AbstractS3ACommitter

Base job setup deletes the success marker. TODO: Do we need this?

Overrides:

setupTask in class AbstractS3ACommitter

Parameters:

context - context

Throws:

IOException - IO failure

needsTaskCommit

public boolean needsTaskCommit(org.apache.hadoop.mapreduce.TaskAttemptContext context)
                        throws IOException

Did this task write any files in the work directory? Probes for a task existing by looking to see if the attempt dir exists. This adds more HTTP requests to the call. It may be better just to return true and rely on the commit task doing the work.

Specified by:

needsTaskCommit in class org.apache.hadoop.mapreduce.OutputCommitter

Parameters:

context - the task's context

Returns:

true if the attempt path exists

Throws:

IOException - failure to list the path

commitTask

public void commitTask(org.apache.hadoop.mapreduce.TaskAttemptContext context)
                throws IOException

Specified by:

commitTask in class org.apache.hadoop.mapreduce.OutputCommitter

Throws:

IOException

abortTask

public void abortTask(org.apache.hadoop.mapreduce.TaskAttemptContext context)
               throws IOException

Abort a task. Attempt load then abort all pending files, then try to delete the task attempt path. This method may be called on the job committer, rather than the task one (such as in the MapReduce AM after a task container failure). It must extract all paths and state from the passed in context.

Specified by:

abortTask in class org.apache.hadoop.mapreduce.OutputCommitter

Parameters:

context - task context

Throws:

IOException - if there was some problem querying the path other than it not actually existing.

getJobAttemptPath

protected org.apache.hadoop.fs.Path getJobAttemptPath(int appAttemptId)

Compute the path where the output of a given job attempt will be placed.

Specified by:

getJobAttemptPath in class AbstractS3ACommitter

Parameters:

appAttemptId - the ID of the application attempt for this job.

Returns:

the path to store job attempt data.

getTaskAttemptPath

public org.apache.hadoop.fs.Path getTaskAttemptPath(org.apache.hadoop.mapreduce.TaskAttemptContext context)

Compute the path where the output of a task attempt is stored until that task is committed.

Overrides:

getTaskAttemptPath in class AbstractS3ACommitter

Parameters:

context - the context of the task attempt.

Returns:

the path where a task attempt should be stored.

getBaseTaskAttemptPath

protected org.apache.hadoop.fs.Path getBaseTaskAttemptPath(org.apache.hadoop.mapreduce.TaskAttemptContext context)

Description copied from class: AbstractS3ACommitter

Compute the base path where the output of a task attempt is written. This is the path which will be deleted when a task is cleaned up and aborted.

Specified by:

getBaseTaskAttemptPath in class AbstractS3ACommitter

Parameters:

context - the context of the task attempt.

Returns:

the path where a task attempt should be stored.

getTempTaskAttemptPath

public org.apache.hadoop.fs.Path getTempTaskAttemptPath(org.apache.hadoop.mapreduce.TaskAttemptContext context)

Get a temporary directory for data. When a task is aborted/cleaned up, the contents of this directory are all deleted.

Specified by:

getTempTaskAttemptPath in class AbstractS3ACommitter

Parameters:

context - task context

Returns:

a path for temporary data.

toString

public String toString()

Overrides:

toString in class AbstractS3ACommitter