org.apache.hadoop.fs.s3a.commit

Class AbstractS3ACommitterFactory

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.hadoop.mapreduce.lib.output.PathOutputCommitterFactory

org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitterFactory

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable

Direct Known Subclasses:

DirectoryStagingCommitterFactory, MagicS3GuardCommitterFactory, PartitionedStagingCommitterFactory, S3ACommitterFactory, StagingCommitterFactory

public abstract class AbstractS3ACommitterFactory
extends org.apache.hadoop.mapreduce.lib.output.PathOutputCommitterFactory

Dynamically create the output committer based on subclass type and settings.

Field Summary

Fields

Modifier and Type	Field and Description
static org.slf4j.Logger	LOG

Fields inherited from class org.apache.hadoop.mapreduce.lib.output.PathOutputCommitterFactory

COMMITTER_FACTORY_CLASS, COMMITTER_FACTORY_DEFAULT, COMMITTER_FACTORY_SCHEME, COMMITTER_FACTORY_SCHEME_PATTERN, FILE_COMMITTER_FACTORY, NAMED_COMMITTER_CLASS, NAMED_COMMITTER_FACTORY

Constructor Summary

Constructors

Constructor and Description
AbstractS3ACommitterFactory()

Method Summary

Modifier and Type	Method and Description
org.apache.hadoop.mapreduce.lib.output.PathOutputCommitter	createOutputCommitter(org.apache.hadoop.fs.Path outputPath, org.apache.hadoop.mapreduce.TaskAttemptContext context)
abstract org.apache.hadoop.mapreduce.lib.output.PathOutputCommitter	createTaskCommitter(S3AFileSystem fileSystem, org.apache.hadoop.fs.Path outputPath, org.apache.hadoop.mapreduce.TaskAttemptContext context) Implementation point: create a task committer for a specific filesystem.
protected org.apache.hadoop.fs.FileSystem	getDestinationFileSystem(org.apache.hadoop.fs.Path outputPath, org.apache.hadoop.mapreduce.JobContext context) Get the destination filesystem, returning null if there is none.

Methods inherited from class org.apache.hadoop.mapreduce.lib.output.PathOutputCommitterFactory

createCommitter, createFileOutputCommitter, getCommitterFactory

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final org.slf4j.Logger LOG

Constructor Detail

AbstractS3ACommitterFactory

public AbstractS3ACommitterFactory()

Method Detail

createOutputCommitter

public org.apache.hadoop.mapreduce.lib.output.PathOutputCommitter createOutputCommitter(org.apache.hadoop.fs.Path outputPath,
                                                                                        org.apache.hadoop.mapreduce.TaskAttemptContext context)
                                                                                 throws IOException

Overrides:

createOutputCommitter in class org.apache.hadoop.mapreduce.lib.output.PathOutputCommitterFactory

Throws:

IOException

getDestinationFileSystem

protected org.apache.hadoop.fs.FileSystem getDestinationFileSystem(org.apache.hadoop.fs.Path outputPath,
                                                                   org.apache.hadoop.mapreduce.JobContext context)
                                                            throws IOException

Get the destination filesystem, returning null if there is none. Code using this must explicitly or implicitly look for a null value in the response.

Parameters:

outputPath - output path

context - job/task context

Returns:

the destination filesystem, if it can be determined

Throws:

IOException - if the FS cannot be instantiated

createTaskCommitter

public abstract org.apache.hadoop.mapreduce.lib.output.PathOutputCommitter createTaskCommitter(S3AFileSystem fileSystem,
                                                                                               org.apache.hadoop.fs.Path outputPath,
                                                                                               org.apache.hadoop.mapreduce.TaskAttemptContext context)
                                                                                        throws IOException

Implementation point: create a task committer for a specific filesystem.

Parameters:

fileSystem - destination FS.

outputPath - final output path for work

context - task context

Returns:

a committer

Throws:

IOException - any problem, including the FS not supporting the desired committer