org.apache.hadoop.fs.s3a.commit

Class S3ACommitterFactory

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.hadoop.mapreduce.lib.output.PathOutputCommitterFactory

org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitterFactory

org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable

public class S3ACommitterFactory
extends AbstractS3ACommitterFactory

The S3A committer factory which chooses the committer based on the specific option chosen in a per-bucket basis from the property CommitConstants.FS_S3A_COMMITTER_NAME. This should be instantiated by using the property value CLASSNAME as the committer for the job, then set the filesystem property CommitConstants.FS_S3A_COMMITTER_NAME to one of

• CommitConstants.COMMITTER_NAME_FILE: File committer.
• CommitConstants.COMMITTER_NAME_DIRECTORY: Staging directory committer.
• CommitConstants.COMMITTER_NAME_PARTITIONED: Staging partitioned committer.
• CommitConstants.COMMITTER_NAME_MAGIC: the "Magic" committer
• InternalCommitterConstants.COMMITTER_NAME_STAGING: the "staging" committer, which isn't intended for use outside tests.

There are no checks to verify that the filesystem is compatible with the committer.

Field Summary

Fields

Modifier and Type	Field and Description
static String	CLASSNAME Name of this class: "org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory".

Fields inherited from class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitterFactory

LOG

Fields inherited from class org.apache.hadoop.mapreduce.lib.output.PathOutputCommitterFactory

COMMITTER_FACTORY_CLASS, COMMITTER_FACTORY_DEFAULT, COMMITTER_FACTORY_SCHEME, COMMITTER_FACTORY_SCHEME_PATTERN, FILE_COMMITTER_FACTORY, NAMED_COMMITTER_CLASS, NAMED_COMMITTER_FACTORY

Constructor Summary

Constructors

Constructor and Description
S3ACommitterFactory()

Method Summary

Modifier and Type	Method and Description
org.apache.hadoop.mapreduce.lib.output.PathOutputCommitter	createTaskCommitter(S3AFileSystem fileSystem, org.apache.hadoop.fs.Path outputPath, org.apache.hadoop.mapreduce.TaskAttemptContext context) Create a task committer.

Methods inherited from class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitterFactory

createOutputCommitter, getDestinationFileSystem

Methods inherited from class org.apache.hadoop.mapreduce.lib.output.PathOutputCommitterFactory

createCommitter, createFileOutputCommitter, getCommitterFactory

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

CLASSNAME

public static final String CLASSNAME

Name of this class: "org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory".

See Also:

Constant Field Values

Constructor Detail

S3ACommitterFactory

public S3ACommitterFactory()

Method Detail

createTaskCommitter

public org.apache.hadoop.mapreduce.lib.output.PathOutputCommitter createTaskCommitter(S3AFileSystem fileSystem,
                                                                                      org.apache.hadoop.fs.Path outputPath,
                                                                                      org.apache.hadoop.mapreduce.TaskAttemptContext context)
                                                                               throws IOException

Create a task committer.

Specified by:

createTaskCommitter in class AbstractS3ACommitterFactory

Parameters:

fileSystem - destination FS.

outputPath - final output path for work

context - job context

Returns:

a committer

Throws:

IOException - instantiation failure