org.apache.hadoop.fs.s3a.commit

Class CommitOperations

java.lang.Object

org.apache.hadoop.fs.s3a.commit.CommitOperations

public class CommitOperations
extends Object

The implementation of the various actions a committer needs. This doesn't implement the protocol/binding to a specific execution engine, just the operations needed to to build one. When invoking FS operations, it assumes that the underlying FS is handling retries and exception translation: it does not attempt to duplicate that work.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	CommitOperations.MaybeIOE A holder for a possible IOException; the call CommitOperations.MaybeIOE.maybeRethrow() will throw any exception passed into the constructor, and be a no-op if none was.

Field Summary

Fields

Modifier and Type	Field and Description
static org.apache.hadoop.fs.PathFilter	PENDING_FILTER Filter to find all {code .pending} files.
static org.apache.hadoop.fs.PathFilter	PENDINGSET_FILTER Filter to find all {code .pendingset} files.

Constructor Summary

Constructors

Constructor and Description
CommitOperations(S3AFileSystem fs) Instantiate.

Method Summary

Modifier and Type	Method and Description
CommitOperations.MaybeIOE	abortAllSinglePendingCommits(org.apache.hadoop.fs.Path pendingDir, boolean recursive) Enumerate all pending files in a dir/tree, abort.
void	abortMultipartCommit(String destKey, String uploadId) Create an AbortMultipartUpload request and POST it to S3, incrementing statistics afterwards.
int	abortPendingUploadsUnderPath(org.apache.hadoop.fs.Path dest) Abort all pending uploads to the destination FS under a path.
void	abortSingleCommit(SinglePendingCommit commit) Abort the multipart commit supplied.
void	addFileSystemStatistics(Map<String,Long> dest) Add the filesystem statistics to the map; overwriting anything with the same name.
CommitOperations.MaybeIOE	commit(SinglePendingCommit commit, String origin) Commit a single pending commit; exceptions are caught and converted to an outcome.
void	commitOrFail(SinglePendingCommit commit) Commit the operation, throwing an exception on any failure.
void	createSuccessMarker(org.apache.hadoop.fs.Path outputPath, SuccessData successData, boolean addMetrics) Save the success data to the _SUCCESS file.
void	deleteSuccessMarker(org.apache.hadoop.fs.Path outputPath) Delete any existing _SUCCESS file.
protected S3AInstrumentation.CommitterStatistics	getStatistics()
void	jobCompleted(boolean success) Note that a job has completed.
List<com.amazonaws.services.s3.model.MultipartUpload>	listPendingUploadsUnderPath(org.apache.hadoop.fs.Path dest) List all pending uploads to the destination FS under a path.
org.apache.commons.lang3.tuple.Pair<PendingSet,List<org.apache.commons.lang3.tuple.Pair<org.apache.hadoop.fs.LocatedFileStatus,IOException>>>	loadSinglePendingCommits(org.apache.hadoop.fs.Path pendingDir, boolean recursive) Load all single pending commits in the directory.
List<org.apache.hadoop.fs.LocatedFileStatus>	locateAllSinglePendingCommits(org.apache.hadoop.fs.Path pendingDir, boolean recursive) Locate all files with the pending suffix under a directory.
protected org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus>	ls(org.apache.hadoop.fs.Path path, boolean recursive) List files.
IOException	makeIOE(String key, Exception ex) Convert any exception to an IOE, if needed.
void	revertCommit(SinglePendingCommit commit) Revert a pending commit by deleting the destination.
void	taskCompleted(boolean success) Note that a task has completed.
static List<com.amazonaws.services.s3.model.PartETag>	toPartEtags(List<String> tagIds) Convert an ordered list of strings to a list of index etag parts.
String	toString()
SinglePendingCommit	uploadFileToPendingCommit(File localFile, org.apache.hadoop.fs.Path destPath, String partition, long uploadPartSize) Upload all the data in the local file, returning the information needed to commit the work.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

PENDINGSET_FILTER

public static final org.apache.hadoop.fs.PathFilter PENDINGSET_FILTER

Filter to find all {code .pendingset} files.

PENDING_FILTER

public static final org.apache.hadoop.fs.PathFilter PENDING_FILTER

Filter to find all {code .pending} files.

Constructor Detail

CommitOperations

public CommitOperations(S3AFileSystem fs)

Instantiate.

Parameters:

fs - FS to bind to

Method Detail

toPartEtags

public static List<com.amazonaws.services.s3.model.PartETag> toPartEtags(List<String> tagIds)

Convert an ordered list of strings to a list of index etag parts.

Parameters:

tagIds - list of tags

Returns:

same list, now in numbered tuples

toString

public String toString()

Overrides:

toString in class Object

getStatistics

protected S3AInstrumentation.CommitterStatistics getStatistics()

Returns:

statistics.

commitOrFail

public void commitOrFail(SinglePendingCommit commit)
                  throws IOException

Commit the operation, throwing an exception on any failure.

Parameters:

commit - commit to execute

Throws:

IOException - on a failure

commit

public CommitOperations.MaybeIOE commit(SinglePendingCommit commit,
                                        String origin)

Commit a single pending commit; exceptions are caught and converted to an outcome.

Parameters:

commit - entry to commit

origin - origin path/string for outcome text

Returns:

the outcome

locateAllSinglePendingCommits

public List<org.apache.hadoop.fs.LocatedFileStatus> locateAllSinglePendingCommits(org.apache.hadoop.fs.Path pendingDir,
                                                                                  boolean recursive)
                                                                           throws IOException

Locate all files with the pending suffix under a directory.

Parameters:

pendingDir - directory

recursive - recursive listing?

Returns:

the list of all located entries

Throws:

IOException - if there is a problem listing the path.

loadSinglePendingCommits

public org.apache.commons.lang3.tuple.Pair<PendingSet,List<org.apache.commons.lang3.tuple.Pair<org.apache.hadoop.fs.LocatedFileStatus,IOException>>> loadSinglePendingCommits(org.apache.hadoop.fs.Path pendingDir,
                                                                                                                                                                              boolean recursive)
                                                                                                                                                                       throws IOException

Load all single pending commits in the directory. All load failures are logged and then added to list of files which would not load.

Parameters:

pendingDir - directory containing commits

recursive - do a recursive scan?

Returns:

tuple of loaded entries and those pending files which would not load/validate.

Throws:

IOException - on a failure to list the files.

makeIOE

public IOException makeIOE(String key,
                           Exception ex)

Convert any exception to an IOE, if needed.

Parameters:

key - key to use in a path exception

ex - exception

Returns:

an IOE, either the passed in value or a new one wrapping the other exception.

abortSingleCommit

public void abortSingleCommit(SinglePendingCommit commit)
                       throws IOException

Abort the multipart commit supplied. This is the lower level operation which doesn't generate an outcome, instead raising an exception.

Parameters:

commit - pending commit to abort

Throws:

FileNotFoundException - if the abort ID is unknown

IOException - on any failure

abortMultipartCommit

public void abortMultipartCommit(String destKey,
                                 String uploadId)
                          throws IOException

Create an AbortMultipartUpload request and POST it to S3, incrementing statistics afterwards.

Parameters:

destKey - destination key

uploadId - upload to cancel

Throws:

FileNotFoundException - if the abort ID is unknown

IOException - on any failure

abortAllSinglePendingCommits

public CommitOperations.MaybeIOE abortAllSinglePendingCommits(org.apache.hadoop.fs.Path pendingDir,
                                                              boolean recursive)
                                                       throws IOException

Enumerate all pending files in a dir/tree, abort.

Parameters:

pendingDir - directory of pending operations

recursive - recurse?

Returns:

the outcome of all the abort operations

Throws:

IOException - if there is a problem listing the path.

ls

protected org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus> ls(org.apache.hadoop.fs.Path path,
                                                                                         boolean recursive)
                                                                                  throws IOException

List files.

Parameters:

path - path

recursive - recursive listing?

Returns:

iterator

Throws:

IOException - failure

listPendingUploadsUnderPath

public List<com.amazonaws.services.s3.model.MultipartUpload> listPendingUploadsUnderPath(org.apache.hadoop.fs.Path dest)
                                                                                  throws IOException

List all pending uploads to the destination FS under a path.

Parameters:

dest - destination path

Returns:

A list of the pending uploads to any directory under that path.

Throws:

IOException - IO failure

abortPendingUploadsUnderPath

public int abortPendingUploadsUnderPath(org.apache.hadoop.fs.Path dest)
                                 throws IOException

Abort all pending uploads to the destination FS under a path.

Parameters:

dest - destination path

Returns:

a count of the number of uploads aborted.

Throws:

IOException - IO failure

deleteSuccessMarker

public void deleteSuccessMarker(org.apache.hadoop.fs.Path outputPath)
                         throws IOException

Delete any existing _SUCCESS file.

Parameters:

outputPath - output directory

Throws:

IOException - IO problem

createSuccessMarker

public void createSuccessMarker(org.apache.hadoop.fs.Path outputPath,
                                SuccessData successData,
                                boolean addMetrics)
                         throws IOException

Save the success data to the _SUCCESS file.

Parameters:

outputPath - output directory

successData - success data to save.

addMetrics - should the FS metrics be added?

Throws:

IOException - IO problem

revertCommit

public void revertCommit(SinglePendingCommit commit)
                  throws IOException

Revert a pending commit by deleting the destination.

Parameters:

commit - pending commit

Throws:

IOException - failure

uploadFileToPendingCommit

public SinglePendingCommit uploadFileToPendingCommit(File localFile,
                                                     org.apache.hadoop.fs.Path destPath,
                                                     String partition,
                                                     long uploadPartSize)
                                              throws IOException

Upload all the data in the local file, returning the information needed to commit the work.

Parameters:

localFile - local file (be a file)

destPath - destination path

partition - partition/subdir. Not used

uploadPartSize - size of upload

Returns:

a pending upload entry

Throws:

IOException - failure

addFileSystemStatistics

public void addFileSystemStatistics(Map<String,Long> dest)

Add the filesystem statistics to the map; overwriting anything with the same name.

Parameters:

dest - destination map

taskCompleted

public void taskCompleted(boolean success)

Note that a task has completed.

Parameters:

success - success flag

jobCompleted

public void jobCompleted(boolean success)

Note that a job has completed.

Parameters:

success - success flag