org.apache.hadoop.fs.s3a.commit

Class AbstractS3ACommitter

java.lang.Object

org.apache.hadoop.mapreduce.OutputCommitter

org.apache.hadoop.mapreduce.lib.output.PathOutputCommitter

org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter

Direct Known Subclasses:

MagicS3GuardCommitter, StagingCommitter

public abstract class AbstractS3ACommitter
extends org.apache.hadoop.mapreduce.lib.output.PathOutputCommitter

Abstract base class for S3A committers; allows for any commonality between different architectures. Although the committer APIs allow for a committer to be created without an output path, this is not supported in this class or its subclasses: a destination must be supplied. It is left to the committer factory to handle the creation of a committer when the destination is unknown. Requiring an output directory simplifies coding and testing.

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	AbstractS3ACommitter(org.apache.hadoop.fs.Path outputPath, org.apache.hadoop.mapreduce.TaskAttemptContext context) Create a committer.

Method Summary

Modifier and Type	Method and Description
void	abortJob(org.apache.hadoop.mapreduce.JobContext context, org.apache.hadoop.mapreduce.JobStatus.State state)
protected void	abortJobInternal(org.apache.hadoop.mapreduce.JobContext context, boolean suppressExceptions) The internal job abort operation; can be overridden in tests.
protected void	abortPendingUploads(org.apache.hadoop.mapreduce.JobContext context, List<SinglePendingCommit> pending, boolean suppressExceptions) Abort all pending uploads in the list.
protected void	abortPendingUploadsInCleanup(boolean suppressExceptions) Abort all pending uploads to the destination directory during job cleanup operations.
protected ExecutorService	buildThreadPool(org.apache.hadoop.mapreduce.JobContext context) Returns an ExecutorService for parallel tasks.
protected void	cleanup(org.apache.hadoop.mapreduce.JobContext context, boolean suppressExceptions) Cleanup the job context, including aborting anything pending.
void	cleanupJob(org.apache.hadoop.mapreduce.JobContext context)
abstract void	cleanupStagingDirs() Clean up any staging directories.
void	commitJob(org.apache.hadoop.mapreduce.JobContext context) Commit work.
protected void	commitJobInternal(org.apache.hadoop.mapreduce.JobContext context, List<SinglePendingCommit> pending) Internal Job commit operation: where the S3 requests are made (potentially in parallel).
protected void	commitPendingUploads(org.apache.hadoop.mapreduce.JobContext context, List<SinglePendingCommit> pending) Commit a list of pending uploads.
protected void	deleteTaskAttemptPathQuietly(org.apache.hadoop.mapreduce.TaskAttemptContext context) Delete the task attempt path without raising any errors.
protected abstract org.apache.hadoop.fs.Path	getBaseTaskAttemptPath(org.apache.hadoop.mapreduce.TaskAttemptContext context) Compute the base path where the output of a task attempt is written.
protected CommitOperations	getCommitOperations() Get the commit actions instance.
org.apache.hadoop.conf.Configuration	getConf()
org.apache.hadoop.fs.FileSystem	getDestFS() Get the destination FS, creating it on demand if needed.
protected org.apache.hadoop.fs.FileSystem	getDestinationFS(org.apache.hadoop.fs.Path out, org.apache.hadoop.conf.Configuration config) Get the destination filesystem from the output path and the configuration.
S3AFileSystem	getDestS3AFS() Get the destination as an S3A Filesystem; casting it.
protected abstract org.apache.hadoop.fs.Path	getJobAttemptPath(int appAttemptId) Compute the path where the output of a given job attempt will be placed.
org.apache.hadoop.fs.Path	getJobAttemptPath(org.apache.hadoop.mapreduce.JobContext context) Compute the path where the output of a given job attempt will be placed.
org.apache.hadoop.mapreduce.JobContext	getJobContext() Get the job/task context this committer was instantiated with.
abstract String	getName() Get the name of this committer.
org.apache.hadoop.fs.Path	getOutputPath() Final path of output, in the destination FS.
protected String	getRole() Used in logging and reporting to help disentangle messages.
protected org.apache.hadoop.fs.FileSystem	getTaskAttemptFilesystem(org.apache.hadoop.mapreduce.TaskAttemptContext context) Get the task attempt path filesystem.
org.apache.hadoop.fs.Path	getTaskAttemptPath(org.apache.hadoop.mapreduce.TaskAttemptContext context) Compute the path where the output of a task attempt is stored until that task is committed.
abstract org.apache.hadoop.fs.Path	getTempTaskAttemptPath(org.apache.hadoop.mapreduce.TaskAttemptContext context) Get a temporary directory for data.
org.apache.hadoop.fs.Path	getWorkPath() This is the critical method for FileOutputFormat; it declares the path for work.
protected void	initOutput(org.apache.hadoop.fs.Path out) Init the output filesystem and path.
protected void	jobCompleted(boolean success) Job completion outcome; this may be subclassed in tests.
protected abstract List<SinglePendingCommit>	listPendingUploadsToCommit(org.apache.hadoop.mapreduce.JobContext context) Get the list of pending uploads for this job attempt.
protected List<SinglePendingCommit>	loadPendingsetFiles(org.apache.hadoop.mapreduce.JobContext context, boolean suppressExceptions, org.apache.hadoop.fs.FileSystem fs, Iterable<? extends org.apache.hadoop.fs.FileStatus> pendingCommitFiles) Try to read every pendingset file and build a list of them/ In the case of a failure to read the file, exceptions are held until all reads have been attempted.
protected void	maybeCreateSuccessMarker(org.apache.hadoop.mapreduce.JobContext context, List<String> filenames) if the job requires a success marker on a successful job, create the file CommitConstants._SUCCESS.
protected void	maybeCreateSuccessMarkerFromCommits(org.apache.hadoop.mapreduce.JobContext context, List<SinglePendingCommit> pending) if the job requires a success marker on a successful job, create the file CommitConstants._SUCCESS.
protected void	maybeIgnore(boolean suppress, String action, Invoker.VoidOperation operation) Execute an operation; maybe suppress any raised IOException.
protected void	maybeIgnore(boolean suppress, String action, IOException ex) Log or rethrow a caught IOException.
protected void	preCommitJob(org.apache.hadoop.mapreduce.JobContext context, List<SinglePendingCommit> pending) Subclass-specific pre commit actions.
void	recoverTask(org.apache.hadoop.mapreduce.TaskAttemptContext taskContext) Task recovery considered unsupported: Warn and fail.
protected boolean	requiresDelayedCommitOutputInFileSystem() Flag to indicate whether or not the destination filesystem needs to be configured to support magic paths where the output isn't immediately visible.
protected void	setConf(org.apache.hadoop.conf.Configuration conf)
protected void	setDestFS(org.apache.hadoop.fs.FileSystem destFS) Set the destination FS: the FS of the final output.
protected void	setOutputPath(org.apache.hadoop.fs.Path outputPath) Set the output path.
void	setupTask(org.apache.hadoop.mapreduce.TaskAttemptContext context) Base job setup deletes the success marker.
protected void	setWorkPath(org.apache.hadoop.fs.Path workPath) Set the work path for this committer.
String	toString()

Methods inherited from class org.apache.hadoop.mapreduce.lib.output.PathOutputCommitter

hasOutputPath

Methods inherited from class org.apache.hadoop.mapreduce.OutputCommitter

abortTask, commitTask, isCommitJobRepeatable, isRecoverySupported, isRecoverySupported, needsTaskCommit, setupJob

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

AbstractS3ACommitter

protected AbstractS3ACommitter(org.apache.hadoop.fs.Path outputPath,
                               org.apache.hadoop.mapreduce.TaskAttemptContext context)
                        throws IOException

Create a committer. This constructor binds the destination directory and configuration, but does not update the work path: That must be calculated by the implementation; It is omitted here to avoid subclass methods being called too early.

Parameters:

outputPath - the job's output path: MUST NOT be null.

context - the task's context

Throws:

IOException - on a failure

Method Detail

initOutput

protected void initOutput(org.apache.hadoop.fs.Path out)
                   throws IOException

Init the output filesystem and path. TESTING ONLY; allows mock FS to cheat.

Parameters:

out - output path

Throws:

IOException - failure to create the FS.

getJobContext

public final org.apache.hadoop.mapreduce.JobContext getJobContext()

Get the job/task context this committer was instantiated with.

Returns:

the context.

getOutputPath

public final org.apache.hadoop.fs.Path getOutputPath()

Final path of output, in the destination FS.

Specified by:

getOutputPath in class org.apache.hadoop.mapreduce.lib.output.PathOutputCommitter

Returns:

the path

setOutputPath

protected final void setOutputPath(org.apache.hadoop.fs.Path outputPath)

Set the output path.

Parameters:

outputPath - new value

getWorkPath

public org.apache.hadoop.fs.Path getWorkPath()

This is the critical method for FileOutputFormat; it declares the path for work.

Specified by:

getWorkPath in class org.apache.hadoop.mapreduce.lib.output.PathOutputCommitter

Returns:

the working path.

setWorkPath

protected void setWorkPath(org.apache.hadoop.fs.Path workPath)

Set the work path for this committer.

Parameters:

workPath - the work path to use.

getConf

public org.apache.hadoop.conf.Configuration getConf()

setConf

protected void setConf(org.apache.hadoop.conf.Configuration conf)

getDestFS

public org.apache.hadoop.fs.FileSystem getDestFS()
                                          throws IOException

Get the destination FS, creating it on demand if needed.

Returns:

the filesystem; requires the output path to be set up

Throws:

IOException - if the FS cannot be instantiated.

getDestS3AFS

public S3AFileSystem getDestS3AFS()
                           throws IOException

Get the destination as an S3A Filesystem; casting it.

Returns:

the dest S3A FS.

Throws:

IOException - if the FS cannot be instantiated.

setDestFS

protected void setDestFS(org.apache.hadoop.fs.FileSystem destFS)

Set the destination FS: the FS of the final output.

Parameters:

destFS - destination FS.

getJobAttemptPath

public org.apache.hadoop.fs.Path getJobAttemptPath(org.apache.hadoop.mapreduce.JobContext context)

Compute the path where the output of a given job attempt will be placed.

Parameters:

context - the context of the job. This is used to get the application attempt ID.

Returns:

the path to store job attempt data.

getJobAttemptPath

protected abstract org.apache.hadoop.fs.Path getJobAttemptPath(int appAttemptId)

Compute the path where the output of a given job attempt will be placed.

Parameters:

appAttemptId - the ID of the application attempt for this job.

Returns:

the path to store job attempt data.

getTaskAttemptPath

public org.apache.hadoop.fs.Path getTaskAttemptPath(org.apache.hadoop.mapreduce.TaskAttemptContext context)

Compute the path where the output of a task attempt is stored until that task is committed. This may be the normal Task attempt path or it may be a subdirectory. The default implementation returns the value of getBaseTaskAttemptPath(TaskAttemptContext); subclasses may return different values.

Parameters:

context - the context of the task attempt.

Returns:

the path where a task attempt should be stored.

getBaseTaskAttemptPath

protected abstract org.apache.hadoop.fs.Path getBaseTaskAttemptPath(org.apache.hadoop.mapreduce.TaskAttemptContext context)

Compute the base path where the output of a task attempt is written. This is the path which will be deleted when a task is cleaned up and aborted.

Parameters:

context - the context of the task attempt.

Returns:

the path where a task attempt should be stored.

getTempTaskAttemptPath

public abstract org.apache.hadoop.fs.Path getTempTaskAttemptPath(org.apache.hadoop.mapreduce.TaskAttemptContext context)

Get a temporary directory for data. When a task is aborted/cleaned up, the contents of this directory are all deleted.

Parameters:

context - task context

Returns:

a path for temporary data.

getName

public abstract String getName()

Get the name of this committer.

Returns:

the committer name.

toString

public String toString()

Overrides:

toString in class org.apache.hadoop.mapreduce.lib.output.PathOutputCommitter

getDestinationFS

protected org.apache.hadoop.fs.FileSystem getDestinationFS(org.apache.hadoop.fs.Path out,
                                                           org.apache.hadoop.conf.Configuration config)
                                                    throws IOException

Get the destination filesystem from the output path and the configuration.

Parameters:

out - output path

config - job/task config

Returns:

the associated FS

Throws:

PathCommitException - output path isn't to an S3A FS instance.

IOException - failure to instantiate the FS.

requiresDelayedCommitOutputInFileSystem

protected boolean requiresDelayedCommitOutputInFileSystem()

Flag to indicate whether or not the destination filesystem needs to be configured to support magic paths where the output isn't immediately visible. If the committer returns true, then committer setup will fail if the FS doesn't have the capability. Base implementation returns false.

Returns:

what the requirements of the committer are of the filesystem.

recoverTask

public void recoverTask(org.apache.hadoop.mapreduce.TaskAttemptContext taskContext)
                 throws IOException

Task recovery considered unsupported: Warn and fail.

Overrides:

recoverTask in class org.apache.hadoop.mapreduce.OutputCommitter

Parameters:

taskContext - Context of the task whose output is being recovered

Throws:

IOException - always.

maybeCreateSuccessMarkerFromCommits

protected void maybeCreateSuccessMarkerFromCommits(org.apache.hadoop.mapreduce.JobContext context,
                                                   List<SinglePendingCommit> pending)
                                            throws IOException

if the job requires a success marker on a successful job, create the file CommitConstants._SUCCESS. While the classic committers create a 0-byte file, the S3Guard committers PUT up a the contents of a SuccessData file.

Parameters:

context - job context

pending - the pending commits

Throws:

IOException - IO failure

maybeCreateSuccessMarker

protected void maybeCreateSuccessMarker(org.apache.hadoop.mapreduce.JobContext context,
                                        List<String> filenames)
                                 throws IOException

if the job requires a success marker on a successful job, create the file CommitConstants._SUCCESS. While the classic committers create a 0-byte file, the S3Guard committers PUT up a the contents of a SuccessData file.

Parameters:

context - job context

filenames - list of filenames.

Throws:

IOException - IO failure

setupTask

public void setupTask(org.apache.hadoop.mapreduce.TaskAttemptContext context)
               throws IOException

Base job setup deletes the success marker. TODO: Do we need this?

Specified by:

setupTask in class org.apache.hadoop.mapreduce.OutputCommitter

Parameters:

context - context

Throws:

IOException - IO failure

getTaskAttemptFilesystem

protected org.apache.hadoop.fs.FileSystem getTaskAttemptFilesystem(org.apache.hadoop.mapreduce.TaskAttemptContext context)
                                                            throws IOException

Get the task attempt path filesystem. This may not be the same as the final destination FS, and so may not be an S3A FS.

Parameters:

context - task attempt

Returns:

the filesystem

Throws:

IOException - failure to instantiate

commitPendingUploads

protected void commitPendingUploads(org.apache.hadoop.mapreduce.JobContext context,
                                    List<SinglePendingCommit> pending)
                             throws IOException

Commit a list of pending uploads.

Parameters:

context - job context

pending - list of pending uploads

Throws:

IOException - on any failure

loadPendingsetFiles

protected List<SinglePendingCommit> loadPendingsetFiles(org.apache.hadoop.mapreduce.JobContext context,
                                                        boolean suppressExceptions,
                                                        org.apache.hadoop.fs.FileSystem fs,
                                                        Iterable<? extends org.apache.hadoop.fs.FileStatus> pendingCommitFiles)
                                                 throws IOException

Try to read every pendingset file and build a list of them/ In the case of a failure to read the file, exceptions are held until all reads have been attempted.

Parameters:

context - job context

suppressExceptions - whether to suppress exceptions.

fs - job attempt fs

pendingCommitFiles - list of files found in the listing scan

Returns:

the list of commits

Throws:

IOException - on a failure when suppressExceptions is false.

commitJobInternal

protected void commitJobInternal(org.apache.hadoop.mapreduce.JobContext context,
                                 List<SinglePendingCommit> pending)
                          throws IOException

Internal Job commit operation: where the S3 requests are made (potentially in parallel).

Parameters:

context - job context

pending - pending request

Throws:

IOException - any failure

abortJob

public void abortJob(org.apache.hadoop.mapreduce.JobContext context,
                     org.apache.hadoop.mapreduce.JobStatus.State state)
              throws IOException

Overrides:

abortJob in class org.apache.hadoop.mapreduce.OutputCommitter

Throws:

IOException

abortJobInternal

protected void abortJobInternal(org.apache.hadoop.mapreduce.JobContext context,
                                boolean suppressExceptions)
                         throws IOException

The internal job abort operation; can be overridden in tests. This must clean up operations; it is called when a commit fails, as well as in an abortJob(JobContext, JobStatus.State) call. The base implementation calls cleanup(JobContext, boolean)

Parameters:

context - job context

suppressExceptions - should exceptions be suppressed?

Throws:

IOException - any IO problem raised when suppressExceptions is false.

abortPendingUploadsInCleanup

protected void abortPendingUploadsInCleanup(boolean suppressExceptions)
                                     throws IOException

Abort all pending uploads to the destination directory during job cleanup operations.

Parameters:

suppressExceptions - should exceptions be suppressed

Throws:

IOException - IO problem

preCommitJob

protected void preCommitJob(org.apache.hadoop.mapreduce.JobContext context,
                            List<SinglePendingCommit> pending)
                     throws IOException

Subclass-specific pre commit actions.

Parameters:

context - job context

pending - the pending operations

Throws:

IOException - any failure

commitJob

public void commitJob(org.apache.hadoop.mapreduce.JobContext context)
               throws IOException

Commit work. This consists of two stages: precommit and commit.

Precommit: identify pending uploads, then allow subclasses to validate the state of the destination and the pending uploads. Any failure here triggers an abort of all pending uploads.

Commit internal: do the final commit sequence.

The final commit action is to build the __SUCCESS file entry.

Overrides:

commitJob in class org.apache.hadoop.mapreduce.OutputCommitter

Parameters:

context - job context

Throws:

IOException - any failure

jobCompleted

protected void jobCompleted(boolean success)

Job completion outcome; this may be subclassed in tests.

Parameters:

success - did the job succeed.

cleanupStagingDirs

public abstract void cleanupStagingDirs()

Clean up any staging directories. IOEs must be caught and swallowed.

listPendingUploadsToCommit

protected abstract List<SinglePendingCommit> listPendingUploadsToCommit(org.apache.hadoop.mapreduce.JobContext context)
                                                                 throws IOException

Get the list of pending uploads for this job attempt.

Parameters:

context - job context

Returns:

a list of pending uploads.

Throws:

IOException - Any IO failure

cleanup

protected void cleanup(org.apache.hadoop.mapreduce.JobContext context,
                       boolean suppressExceptions)
                throws IOException

Cleanup the job context, including aborting anything pending.

Parameters:

context - job context

suppressExceptions - should exceptions be suppressed?

Throws:

IOException - any failure if exceptions were not suppressed.

cleanupJob

public void cleanupJob(org.apache.hadoop.mapreduce.JobContext context)
                throws IOException

Overrides:

cleanupJob in class org.apache.hadoop.mapreduce.OutputCommitter

Throws:

IOException

maybeIgnore

protected void maybeIgnore(boolean suppress,
                           String action,
                           Invoker.VoidOperation operation)
                    throws IOException

Execute an operation; maybe suppress any raised IOException.

Parameters:

suppress - should raised IOEs be suppressed?

action - action (for logging when the IOE is supressed.

operation - operation

Throws:

IOException - if operation raised an IOE and suppress == false

maybeIgnore

protected void maybeIgnore(boolean suppress,
                           String action,
                           IOException ex)
                    throws IOException

Log or rethrow a caught IOException.

Parameters:

suppress - should raised IOEs be suppressed?

action - action (for logging when the IOE is suppressed.

ex - exception

Throws:

IOException - if suppress == false

getCommitOperations

protected CommitOperations getCommitOperations()

Get the commit actions instance. Subclasses may provide a mock version of this.

Returns:

the commit actions instance to use for operations.

getRole

protected String getRole()

Used in logging and reporting to help disentangle messages.

Returns:

the committer's role.

buildThreadPool

protected final ExecutorService buildThreadPool(org.apache.hadoop.mapreduce.JobContext context)

Returns an ExecutorService for parallel tasks. The number of threads in the thread-pool is set by s3.multipart.committer.num-threads. If num-threads is 0, this will return null;

Parameters:

context - the JobContext for this commit

Returns:

an ExecutorService or null for the number of threads

deleteTaskAttemptPathQuietly

protected void deleteTaskAttemptPathQuietly(org.apache.hadoop.mapreduce.TaskAttemptContext context)

Delete the task attempt path without raising any errors.

Parameters:

context - task context

abortPendingUploads

protected void abortPendingUploads(org.apache.hadoop.mapreduce.JobContext context,
                                   List<SinglePendingCommit> pending,
                                   boolean suppressExceptions)
                            throws IOException

Abort all pending uploads in the list.

Parameters:

context - job context

pending - pending uploads

suppressExceptions - should exceptions be suppressed

Throws:

IOException - any exception raised