org.apache.hadoop.fs.s3a.commit

Class CommitOperations

java.lang.Object

org.apache.hadoop.fs.s3a.commit.CommitOperations

All Implemented Interfaces:

org.apache.hadoop.fs.statistics.IOStatisticsSource

public class CommitOperations
extends Object
implements org.apache.hadoop.fs.statistics.IOStatisticsSource

The implementation of the various actions a committer needs. This doesn't implement the protocol/binding to a specific execution engine, just the operations needed to to build one. When invoking FS operations, it assumes that the underlying FS is handling retries and exception translation: it does not attempt to duplicate that work.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
class	CommitOperations.CommitContext Commit context.
static class	CommitOperations.MaybeIOE A holder for a possible IOException; the call CommitOperations.MaybeIOE.maybeRethrow() will throw any exception passed into the constructor, and be a no-op if none was.

Field Summary

Fields

Modifier and Type	Field and Description
static org.apache.hadoop.fs.PathFilter	PENDING_FILTER Filter to find all {code .pending} files.
static org.apache.hadoop.fs.PathFilter	PENDINGSET_FILTER Filter to find all {code .pendingset} files.

Constructor Summary

Constructors

Constructor and Description
CommitOperations(S3AFileSystem fs) Instantiate.
CommitOperations(S3AFileSystem fs, CommitterStatistics committerStatistics) Instantiate.

Method Summary

Modifier and Type	Method and Description
CommitOperations.MaybeIOE	abortAllSinglePendingCommits(org.apache.hadoop.fs.Path pendingDir, boolean recursive) Enumerate all pending files in a dir/tree, abort.
int	abortPendingUploadsUnderPath(org.apache.hadoop.fs.Path dest) Abort all pending uploads to the destination FS under a path.
void	addFileSystemStatistics(Map<String,Long> dest) Add the filesystem statistics to the map; overwriting anything with the same name.
void	createSuccessMarker(org.apache.hadoop.fs.Path outputPath, SuccessData successData, boolean addMetrics) Save the success data to the _SUCCESS file.
void	deleteSuccessMarker(org.apache.hadoop.fs.Path outputPath) Delete any existing _SUCCESS file.
static Optional<Long>	extractMagicFileLength(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path) Get the magic file length of a file.
org.apache.hadoop.fs.statistics.IOStatistics	getIOStatistics()
protected CommitterStatistics	getStatistics()
CommitOperations.CommitContext	initiateCommitOperation(org.apache.hadoop.fs.Path path) Begin the final commit.
void	jobCompleted(boolean success) Note that a job has completed.
List<com.amazonaws.services.s3.model.MultipartUpload>	listPendingUploadsUnderPath(org.apache.hadoop.fs.Path dest) List all pending uploads to the destination FS under a path.
org.apache.commons.lang3.tuple.Pair<PendingSet,List<org.apache.commons.lang3.tuple.Pair<org.apache.hadoop.fs.LocatedFileStatus,IOException>>>	loadSinglePendingCommits(org.apache.hadoop.fs.Path pendingDir, boolean recursive) Load all single pending commits in the directory.
List<org.apache.hadoop.fs.LocatedFileStatus>	locateAllSinglePendingCommits(org.apache.hadoop.fs.Path pendingDir, boolean recursive) Locate all files with the pending suffix under a directory.
protected org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus>	ls(org.apache.hadoop.fs.Path path, boolean recursive) List files.
IOException	makeIOE(String key, Exception ex) Convert any exception to an IOE, if needed.
void	revertCommit(SinglePendingCommit commit, BulkOperationState operationState) Revert a pending commit by deleting the destination.
void	taskCompleted(boolean success) Note that a task has completed.
static List<com.amazonaws.services.s3.model.PartETag>	toPartEtags(List<String> tagIds) Convert an ordered list of strings to a list of index etag parts.
String	toString()
SinglePendingCommit	uploadFileToPendingCommit(File localFile, org.apache.hadoop.fs.Path destPath, String partition, long uploadPartSize, org.apache.hadoop.util.Progressable progress) Upload all the data in the local file, returning the information needed to commit the work.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

PENDINGSET_FILTER

public static final org.apache.hadoop.fs.PathFilter PENDINGSET_FILTER

Filter to find all {code .pendingset} files.

PENDING_FILTER

public static final org.apache.hadoop.fs.PathFilter PENDING_FILTER

Filter to find all {code .pending} files.

Constructor Detail

CommitOperations

public CommitOperations(S3AFileSystem fs)

Instantiate.

Parameters:

fs - FS to bind to

CommitOperations

public CommitOperations(S3AFileSystem fs,
                        CommitterStatistics committerStatistics)

Instantiate.

Parameters:

fs - FS to bind to

committerStatistics - committer statistics

Method Detail

toPartEtags

public static List<com.amazonaws.services.s3.model.PartETag> toPartEtags(List<String> tagIds)

Convert an ordered list of strings to a list of index etag parts.

Parameters:

tagIds - list of tags

Returns:

same list, now in numbered tuples

toString

public String toString()

Overrides:

toString in class Object

getStatistics

protected CommitterStatistics getStatistics()

Returns:

statistics.

getIOStatistics

public org.apache.hadoop.fs.statistics.IOStatistics getIOStatistics()

Specified by:

getIOStatistics in interface org.apache.hadoop.fs.statistics.IOStatisticsSource

locateAllSinglePendingCommits

public List<org.apache.hadoop.fs.LocatedFileStatus> locateAllSinglePendingCommits(org.apache.hadoop.fs.Path pendingDir,
                                                                                  boolean recursive)
                                                                           throws IOException

Locate all files with the pending suffix under a directory.

Parameters:

pendingDir - directory

recursive - recursive listing?

Returns:

the list of all located entries

Throws:

IOException - if there is a problem listing the path.

loadSinglePendingCommits

public org.apache.commons.lang3.tuple.Pair<PendingSet,List<org.apache.commons.lang3.tuple.Pair<org.apache.hadoop.fs.LocatedFileStatus,IOException>>> loadSinglePendingCommits(org.apache.hadoop.fs.Path pendingDir,
                                                                                                                                                                              boolean recursive)
                                                                                                                                                                       throws IOException

Load all single pending commits in the directory. All load failures are logged and then added to list of files which would not load.

Parameters:

pendingDir - directory containing commits

recursive - do a recursive scan?

Returns:

tuple of loaded entries and those pending files which would not load/validate.

Throws:

IOException - on a failure to list the files.

makeIOE

public IOException makeIOE(String key,
                           Exception ex)

Convert any exception to an IOE, if needed.

Parameters:

key - key to use in a path exception

ex - exception

Returns:

an IOE, either the passed in value or a new one wrapping the other exception.

abortAllSinglePendingCommits

public CommitOperations.MaybeIOE abortAllSinglePendingCommits(org.apache.hadoop.fs.Path pendingDir,
                                                              boolean recursive)
                                                       throws IOException

Enumerate all pending files in a dir/tree, abort.

Parameters:

pendingDir - directory of pending operations

recursive - recurse?

Returns:

the outcome of all the abort operations

Throws:

IOException - if there is a problem listing the path.

ls

protected org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus> ls(org.apache.hadoop.fs.Path path,
                                                                                         boolean recursive)
                                                                                  throws IOException

List files.

Parameters:

path - path

recursive - recursive listing?

Returns:

iterator

Throws:

IOException - failure

listPendingUploadsUnderPath

public List<com.amazonaws.services.s3.model.MultipartUpload> listPendingUploadsUnderPath(org.apache.hadoop.fs.Path dest)
                                                                                  throws IOException

List all pending uploads to the destination FS under a path.

Parameters:

dest - destination path

Returns:

A list of the pending uploads to any directory under that path.

Throws:

IOException - IO failure

abortPendingUploadsUnderPath

public int abortPendingUploadsUnderPath(org.apache.hadoop.fs.Path dest)
                                 throws IOException

Abort all pending uploads to the destination FS under a path.

Parameters:

dest - destination path

Returns:

a count of the number of uploads aborted.

Throws:

IOException - IO failure

deleteSuccessMarker

public void deleteSuccessMarker(org.apache.hadoop.fs.Path outputPath)
                         throws IOException

Delete any existing _SUCCESS file.

Parameters:

outputPath - output directory

Throws:

IOException - IO problem

createSuccessMarker

public void createSuccessMarker(org.apache.hadoop.fs.Path outputPath,
                                SuccessData successData,
                                boolean addMetrics)
                         throws IOException

Save the success data to the _SUCCESS file.

Parameters:

outputPath - output directory

successData - success data to save.

addMetrics - should the FS metrics be added?

Throws:

IOException - IO problem

revertCommit

public void revertCommit(SinglePendingCommit commit,
                         BulkOperationState operationState)
                  throws IOException

Revert a pending commit by deleting the destination.

Parameters:

commit - pending commit

operationState - nullable operational state for a bulk update

Throws:

IOException - failure

uploadFileToPendingCommit

public SinglePendingCommit uploadFileToPendingCommit(File localFile,
                                                     org.apache.hadoop.fs.Path destPath,
                                                     String partition,
                                                     long uploadPartSize,
                                                     org.apache.hadoop.util.Progressable progress)
                                              throws IOException

Upload all the data in the local file, returning the information needed to commit the work.

Parameters:

localFile - local file (be a file)

destPath - destination path

partition - partition/subdir. Not used

uploadPartSize - size of upload

progress - progress callback

Returns:

a pending upload entry

Throws:

IOException - failure

addFileSystemStatistics

public void addFileSystemStatistics(Map<String,Long> dest)

Add the filesystem statistics to the map; overwriting anything with the same name.

Parameters:

dest - destination map

taskCompleted

public void taskCompleted(boolean success)

Note that a task has completed.

Parameters:

success - success flag

jobCompleted

public void jobCompleted(boolean success)

Note that a job has completed.

Parameters:

success - success flag

initiateCommitOperation

public CommitOperations.CommitContext initiateCommitOperation(org.apache.hadoop.fs.Path path)
                                                       throws IOException

Begin the final commit.

Parameters:

path - path for all work.

Returns:

the commit context to pass in.

Throws:

IOException - failure.

extractMagicFileLength

public static Optional<Long> extractMagicFileLength(org.apache.hadoop.fs.FileSystem fs,
                                                    org.apache.hadoop.fs.Path path)
                                             throws IOException

Get the magic file length of a file. If the FS doesn't support the API, the attribute is missing or the parse to long fails, then Optional.empty() is returned. Static for some easier testability.

Parameters:

fs - filesystem

path - path

Returns:

either a length or None.

Throws:

IOException - on error