org.apache.hadoop.fs.s3a.commit

Class CommitUtils

java.lang.Object

org.apache.hadoop.fs.s3a.commit.CommitUtils

public final class CommitUtils
extends Object

Static utility methods related to S3A commitment processing, both staging and magic. Do not use in any codepath intended to be used from the S3AFS except in the committers themselves.

Method Summary

Modifier and Type	Method and Description
static String	extractJobID(org.apache.hadoop.conf.Configuration conf) Extract the job ID from a configuration.
static S3AFileSystem	getS3AFileSystem(org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration conf, boolean magicCommitRequired) Get the S3A FS of a path.
static void	validateCollectionClass(Iterable it, Class classname) Verify that all instances in a collection are of the given class.
static void	verifyIsMagicCommitFS(S3AFileSystem fs) Verify that an S3A FS instance is a magic commit FS.
static void	verifyIsMagicCommitPath(S3AFileSystem fs, org.apache.hadoop.fs.Path path) Verify that the path is a magic one.
static S3AFileSystem	verifyIsS3AFS(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path) Verify that an FS is an S3A FS.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

verifyIsMagicCommitPath

public static void verifyIsMagicCommitPath(S3AFileSystem fs,
                                           org.apache.hadoop.fs.Path path)
                                    throws PathCommitException

Verify that the path is a magic one.

Parameters:

fs - filesystem

path - path

Throws:

PathCommitException - if the path isn't a magic commit path

verifyIsMagicCommitFS

public static void verifyIsMagicCommitFS(S3AFileSystem fs)
                                  throws PathCommitException

Verify that an S3A FS instance is a magic commit FS.

Parameters:

fs - filesystem

Throws:

PathCommitException - if the FS isn't a magic commit FS.

verifyIsS3AFS

public static S3AFileSystem verifyIsS3AFS(org.apache.hadoop.fs.FileSystem fs,
                                          org.apache.hadoop.fs.Path path)
                                   throws PathCommitException

Verify that an FS is an S3A FS.

Parameters:

fs - filesystem

path - path to to use in exception

Returns:

the typecast FS.

Throws:

PathCommitException - if the FS is not an S3A FS.

getS3AFileSystem

public static S3AFileSystem getS3AFileSystem(org.apache.hadoop.fs.Path path,
                                             org.apache.hadoop.conf.Configuration conf,
                                             boolean magicCommitRequired)
                                      throws PathCommitException,
                                             IOException

Get the S3A FS of a path.

Parameters:

path - path to examine

conf - config

magicCommitRequired - is magic complete required in the FS?

Returns:

the filesystem

Throws:

PathCommitException - output path isn't to an S3A FS instance, or if magicCommitRequired is set, if doesn't support these commits.

IOException - failure to instantiate the FS

validateCollectionClass

public static void validateCollectionClass(Iterable it,
                                           Class classname)
                                    throws ValidationFailure

Verify that all instances in a collection are of the given class.

Parameters:

it - iterator

classname - classname to require

Throws:

ValidationFailure - on a failure

extractJobID

public static String extractJobID(org.apache.hadoop.conf.Configuration conf)

Extract the job ID from a configuration.

Parameters:

conf - configuration

Returns:

a job ID or null.