org.apache.hadoop.fs.s3a.commit.impl

Class CommitOperations

java.lang.Object

org.apache.hadoop.fs.s3a.impl.AbstractStoreOperation

org.apache.hadoop.fs.s3a.commit.impl.CommitOperations

All Implemented Interfaces:

org.apache.hadoop.fs.statistics.IOStatisticsSource

public class CommitOperations
extends AbstractStoreOperation
implements org.apache.hadoop.fs.statistics.IOStatisticsSource

The implementation of the various actions a committer needs. This doesn't implement the protocol/binding to a specific execution engine, just the operations needed to to build one. When invoking FS operations, it assumes that the underlying FS is handling retries and exception translation: it does not attempt to duplicate that work.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	CommitOperations.MaybeIOE A holder for a possible IOException; the call CommitOperations.MaybeIOE.maybeRethrow() will throw any exception passed into the constructor, and be a no-op if none was.

Field Summary

Fields

Modifier and Type	Field and Description
static org.apache.hadoop.fs.PathFilter	PENDING_FILTER Filter to find all {code .pending} files.
static org.apache.hadoop.fs.PathFilter	PENDINGSET_FILTER Filter to find all {code .pendingset} files.

Constructor Summary

Constructors

Constructor and Description
CommitOperations(S3AFileSystem fs) Instantiate.
CommitOperations(S3AFileSystem fs, CommitterStatistics committerStatistics, String outputPath) Instantiate.

Method Summary

Modifier and Type	Method and Description
CommitOperations.MaybeIOE	abortAllSinglePendingCommits(org.apache.hadoop.fs.Path pendingDir, CommitContext commitContext, boolean recursive) Enumerate all pending files in a dir/tree, abort.
void	abortMultipartCommit(String destKey, String uploadId) Create an AbortMultipartUpload request and POST it to S3, incrementing statistics afterwards.
int	abortPendingUploadsUnderPath(org.apache.hadoop.fs.Path dest) Abort all pending uploads to the destination FS under a path.
void	abortSingleCommit(SinglePendingCommit commit) Abort the multipart commit supplied.
void	addFileSystemStatistics(Map<String,Long> dest) Add the filesystem statistics to the map; overwriting anything with the same name.
CommitOperations.MaybeIOE	commit(SinglePendingCommit commit, String origin) Commit a single pending commit; exceptions are caught and converted to an outcome.
void	commitOrFail(SinglePendingCommit commit) Commit the operation, throwing an exception on any failure.
CommitContext	createCommitContext(org.apache.hadoop.mapreduce.JobContext context, org.apache.hadoop.fs.Path path, int committerThreads, org.apache.hadoop.fs.statistics.IOStatisticsContext ioStatisticsContext) Create a commit context for a job or task.
CommitContext	createCommitContextForTesting(org.apache.hadoop.fs.Path path, String jobId, int committerThreads) Create a stub commit context for tests.
void	createSuccessMarker(org.apache.hadoop.fs.Path outputPath, SuccessData successData, boolean addMetrics) Save the success data to the _SUCCESS file.
void	deleteSuccessMarker(org.apache.hadoop.fs.Path outputPath) Delete any existing _SUCCESS file.
static Optional<Long>	extractMagicFileLength(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path) Get the magic file length of a file.
org.apache.hadoop.fs.statistics.IOStatistics	getIOStatistics()
protected CommitterStatistics	getStatistics()
void	jobCompleted(boolean success) Note that a job has completed.
List<software.amazon.awssdk.services.s3.model.MultipartUpload>	listPendingUploadsUnderPath(org.apache.hadoop.fs.Path dest) List all pending uploads to the destination FS under a path.
org.apache.commons.lang3.tuple.Pair<PendingSet,List<org.apache.commons.lang3.tuple.Pair<org.apache.hadoop.fs.LocatedFileStatus,IOException>>>	loadSinglePendingCommits(org.apache.hadoop.fs.Path pendingDir, boolean recursive, CommitContext commitContext) Load all single pending commits in the directory, using the outer submitter.
org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus>	locateAllSinglePendingCommits(org.apache.hadoop.fs.Path pendingDir, boolean recursive) Locate all files with the pending suffix under a directory.
protected org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus>	ls(org.apache.hadoop.fs.Path path, boolean recursive) List files.
IOException	makeIOE(String key, Exception ex) Convert any exception to an IOE, if needed.
void	revertCommit(SinglePendingCommit commit) Revert a pending commit by deleting the destination.
void	taskCompleted(boolean success) Note that a task has completed.
static List<software.amazon.awssdk.services.s3.model.CompletedPart>	toPartEtags(List<String> tagIds) Convert an ordered list of strings to a list of index etag parts.
String	toString()
SinglePendingCommit	uploadFileToPendingCommit(File localFile, org.apache.hadoop.fs.Path destPath, String partition, long uploadPartSize, org.apache.hadoop.util.Progressable progress) Upload all the data in the local file, returning the information needed to commit the work.

Methods inherited from class org.apache.hadoop.fs.s3a.impl.AbstractStoreOperation

activateAuditSpan, getAuditSpan, getStoreContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

PENDINGSET_FILTER

public static final org.apache.hadoop.fs.PathFilter PENDINGSET_FILTER

Filter to find all {code .pendingset} files.

PENDING_FILTER

public static final org.apache.hadoop.fs.PathFilter PENDING_FILTER

Filter to find all {code .pending} files.

Constructor Detail

CommitOperations

public CommitOperations(S3AFileSystem fs)
                 throws IOException

Instantiate.

Parameters:

fs - FS to bind to

Throws:

IOException - failure to bind.

CommitOperations

public CommitOperations(S3AFileSystem fs,
                        CommitterStatistics committerStatistics,
                        String outputPath)
                 throws IOException

Instantiate. This creates a new audit span for the commit operations.

Parameters:

fs - FS to bind to

committerStatistics - committer statistics

outputPath - destination of work.

Throws:

IOException - failure to bind.

Method Detail

toPartEtags

public static List<software.amazon.awssdk.services.s3.model.CompletedPart> toPartEtags(List<String> tagIds)

Convert an ordered list of strings to a list of index etag parts.

Parameters:

tagIds - list of tags

Returns:

same list, now in numbered tuples

toString

public String toString()

Overrides:

toString in class Object

getStatistics

protected CommitterStatistics getStatistics()

Returns:

statistics.

getIOStatistics

public org.apache.hadoop.fs.statistics.IOStatistics getIOStatistics()

Specified by:

getIOStatistics in interface org.apache.hadoop.fs.statistics.IOStatisticsSource

commitOrFail

public void commitOrFail(SinglePendingCommit commit)
                  throws IOException

Commit the operation, throwing an exception on any failure.

Parameters:

commit - commit to execute

Throws:

IOException - on a failure

commit

public CommitOperations.MaybeIOE commit(SinglePendingCommit commit,
                                        String origin)

Commit a single pending commit; exceptions are caught and converted to an outcome.

Parameters:

commit - entry to commit

origin - origin path/string for outcome text

Returns:

the outcome

locateAllSinglePendingCommits

public org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus> locateAllSinglePendingCommits(org.apache.hadoop.fs.Path pendingDir,
                                                                                                                 boolean recursive)
                                                                                                          throws IOException

Locate all files with the pending suffix under a directory.

Parameters:

pendingDir - directory

recursive - recursive listing?

Returns:

iterator of all located entries

Throws:

IOException - if there is a problem listing the path.

loadSinglePendingCommits

public org.apache.commons.lang3.tuple.Pair<PendingSet,List<org.apache.commons.lang3.tuple.Pair<org.apache.hadoop.fs.LocatedFileStatus,IOException>>> loadSinglePendingCommits(org.apache.hadoop.fs.Path pendingDir,
                                                                                                                                                                              boolean recursive,
                                                                                                                                                                              CommitContext commitContext)
                                                                                                                                                                       throws IOException

Load all single pending commits in the directory, using the outer submitter. All load failures are logged and then added to list of files which would not load.

Parameters:

pendingDir - directory containing commits

recursive - do a recursive scan?

commitContext - commit context

Returns:

tuple of loaded entries and those pending files which would not load/validate.

Throws:

IOException - on a failure to list the files.

makeIOE

public IOException makeIOE(String key,
                           Exception ex)

Convert any exception to an IOE, if needed.

Parameters:

key - key to use in a path exception

ex - exception

Returns:

an IOE, either the passed in value or a new one wrapping the other exception.

abortSingleCommit

public void abortSingleCommit(SinglePendingCommit commit)
                       throws IOException

Abort the multipart commit supplied. This is the lower level operation which doesn't generate an outcome, instead raising an exception.

Parameters:

commit - pending commit to abort

Throws:

FileNotFoundException - if the abort ID is unknown

IOException - on any failure

abortMultipartCommit

public void abortMultipartCommit(String destKey,
                                 String uploadId)
                          throws IOException

Create an AbortMultipartUpload request and POST it to S3, incrementing statistics afterwards.

Parameters:

destKey - destination key

uploadId - upload to cancel

Throws:

FileNotFoundException - if the abort ID is unknown

IOException - on any failure

abortAllSinglePendingCommits

public CommitOperations.MaybeIOE abortAllSinglePendingCommits(org.apache.hadoop.fs.Path pendingDir,
                                                              CommitContext commitContext,
                                                              boolean recursive)
                                                       throws IOException

Enumerate all pending files in a dir/tree, abort.

Parameters:

pendingDir - directory of pending operations

commitContext - commit context

recursive - recurse?

Returns:

the outcome of all the abort operations

Throws:

IOException - if there is a problem listing the path.

ls

protected org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus> ls(org.apache.hadoop.fs.Path path,
                                                                                         boolean recursive)
                                                                                  throws IOException

List files.

Parameters:

path - path

recursive - recursive listing?

Returns:

iterator

Throws:

IOException - failure

listPendingUploadsUnderPath

public List<software.amazon.awssdk.services.s3.model.MultipartUpload> listPendingUploadsUnderPath(org.apache.hadoop.fs.Path dest)
                                                                                           throws IOException

List all pending uploads to the destination FS under a path.

Parameters:

dest - destination path

Returns:

A list of the pending uploads to any directory under that path.

Throws:

IOException - IO failure

abortPendingUploadsUnderPath

public int abortPendingUploadsUnderPath(org.apache.hadoop.fs.Path dest)
                                 throws IOException

Abort all pending uploads to the destination FS under a path.

Parameters:

dest - destination path

Returns:

a count of the number of uploads aborted.

Throws:

IOException - IO failure

deleteSuccessMarker

public void deleteSuccessMarker(org.apache.hadoop.fs.Path outputPath)
                         throws IOException

Delete any existing _SUCCESS file.

Parameters:

outputPath - output directory

Throws:

IOException - IO problem

createSuccessMarker

public void createSuccessMarker(org.apache.hadoop.fs.Path outputPath,
                                SuccessData successData,
                                boolean addMetrics)
                         throws IOException

Save the success data to the _SUCCESS file.

Parameters:

outputPath - output directory

successData - success data to save.

addMetrics - should the FS metrics be added?

Throws:

IOException - IO problem

revertCommit

public void revertCommit(SinglePendingCommit commit)
                  throws IOException

Revert a pending commit by deleting the destination.

Parameters:

commit - pending commit

Throws:

IOException - failure

uploadFileToPendingCommit

public SinglePendingCommit uploadFileToPendingCommit(File localFile,
                                                     org.apache.hadoop.fs.Path destPath,
                                                     String partition,
                                                     long uploadPartSize,
                                                     org.apache.hadoop.util.Progressable progress)
                                              throws IOException

Upload all the data in the local file, returning the information needed to commit the work.

Parameters:

localFile - local file (be a file)

destPath - destination path

partition - partition/subdir. Not used

uploadPartSize - size of upload

progress - progress callback

Returns:

a pending upload entry

Throws:

IOException - failure

addFileSystemStatistics

public void addFileSystemStatistics(Map<String,Long> dest)

Add the filesystem statistics to the map; overwriting anything with the same name.

Parameters:

dest - destination map

taskCompleted

public void taskCompleted(boolean success)

Note that a task has completed.

Parameters:

success - success flag

jobCompleted

public void jobCompleted(boolean success)

Note that a job has completed.

Parameters:

success - success flag

createCommitContext

public CommitContext createCommitContext(org.apache.hadoop.mapreduce.JobContext context,
                                         org.apache.hadoop.fs.Path path,
                                         int committerThreads,
                                         org.apache.hadoop.fs.statistics.IOStatisticsContext ioStatisticsContext)
                                  throws IOException

Create a commit context for a job or task.

Parameters:

context - job context

path - path for all work.

committerThreads - thread pool size

ioStatisticsContext - IOStatistics context of current thread

Returns:

the commit context to pass in.

Throws:

IOException - failure.

createCommitContextForTesting

public CommitContext createCommitContextForTesting(org.apache.hadoop.fs.Path path,
                                                   @Nullable
                                                   String jobId,
                                                   int committerThreads)
                                            throws IOException

Create a stub commit context for tests. There's no job context.

Parameters:

path - path for all work.

jobId - job ID; if null a random UUID is generated.

committerThreads - number of committer threads.

Returns:

the commit context to pass in.

Throws:

IOException - failure.

extractMagicFileLength

public static Optional<Long> extractMagicFileLength(org.apache.hadoop.fs.FileSystem fs,
                                                    org.apache.hadoop.fs.Path path)
                                             throws IOException

Get the magic file length of a file. If the FS doesn't support the API, the attribute is missing or the parse to long fails, then Optional.empty() is returned. Static for some easier testability.

Parameters:

fs - filesystem

path - path

Returns:

either a length or None.

Throws:

IOException - on error