| Package | Description |
|---|---|
| org.apache.hadoop.fs.s3a |
S3A Filesystem.
|
| org.apache.hadoop.fs.s3a.commit |
Support for committing the output of analytics jobs directly to S3.
|
| org.apache.hadoop.fs.s3a.commit.magic |
This is the "Magic" committer and support.
|
| org.apache.hadoop.fs.s3a.commit.staging |
The staging committers.
|
| org.apache.hadoop.fs.s3a.s3guard |
This package contains classes related to S3Guard: a feature of S3A to mask
the eventual consistency behavior of S3 and optimize access patterns by
coordinating with a strongly consistent external store for file system
metadata.
|
| Modifier and Type | Method and Description |
|---|---|
static void |
InconsistentAmazonS3Client.clearInconsistency(S3AFileSystem fs)
Clear any accumulated inconsistency state.
|
static void |
InconsistentAmazonS3Client.setFailureInjectionPolicy(S3AFileSystem fs,
FailureInjectionPolicy policy)
A way for tests to patch in a different fault injection policy at runtime.
|
| Constructor and Description |
|---|
Listing(S3AFileSystem owner) |
ProgressableProgressListener(S3AFileSystem fs,
String key,
com.amazonaws.services.s3.transfer.Upload upload,
org.apache.hadoop.util.Progressable progress)
Instantiate.
|
WriteOperationHelper(S3AFileSystem owner,
org.apache.hadoop.conf.Configuration conf)
Constructor.
|
| Modifier and Type | Method and Description |
|---|---|
S3AFileSystem |
AbstractS3ACommitter.getDestS3AFS()
Get the destination as an S3A Filesystem; casting it.
|
static S3AFileSystem |
CommitUtils.getS3AFileSystem(org.apache.hadoop.fs.Path path,
org.apache.hadoop.conf.Configuration conf,
boolean magicCommitRequired)
Get the S3A FS of a path.
|
static S3AFileSystem |
CommitUtils.verifyIsS3AFS(org.apache.hadoop.fs.FileSystem fs,
org.apache.hadoop.fs.Path path)
Verify that an FS is an S3A FS.
|
| Modifier and Type | Method and Description |
|---|---|
abstract org.apache.hadoop.mapreduce.lib.output.PathOutputCommitter |
AbstractS3ACommitterFactory.createTaskCommitter(S3AFileSystem fileSystem,
org.apache.hadoop.fs.Path outputPath,
org.apache.hadoop.mapreduce.TaskAttemptContext context)
Implementation point: create a task committer for a specific filesystem.
|
org.apache.hadoop.mapreduce.lib.output.PathOutputCommitter |
S3ACommitterFactory.createTaskCommitter(S3AFileSystem fileSystem,
org.apache.hadoop.fs.Path outputPath,
org.apache.hadoop.mapreduce.TaskAttemptContext context)
Create a task committer.
|
static void |
CommitUtils.verifyIsMagicCommitFS(S3AFileSystem fs)
Verify that an S3A FS instance is a magic commit FS.
|
static void |
CommitUtils.verifyIsMagicCommitPath(S3AFileSystem fs,
org.apache.hadoop.fs.Path path)
Verify that the path is a magic one.
|
| Constructor and Description |
|---|
CommitOperations(S3AFileSystem fs)
Instantiate.
|
MagicCommitIntegration(S3AFileSystem owner,
boolean magicCommitEnabled)
Instantiate.
|
| Modifier and Type | Method and Description |
|---|---|
org.apache.hadoop.mapreduce.lib.output.PathOutputCommitter |
MagicS3GuardCommitterFactory.createTaskCommitter(S3AFileSystem fileSystem,
org.apache.hadoop.fs.Path outputPath,
org.apache.hadoop.mapreduce.TaskAttemptContext context) |
| Modifier and Type | Method and Description |
|---|---|
org.apache.hadoop.mapreduce.lib.output.PathOutputCommitter |
StagingCommitterFactory.createTaskCommitter(S3AFileSystem fileSystem,
org.apache.hadoop.fs.Path outputPath,
org.apache.hadoop.mapreduce.TaskAttemptContext context) |
org.apache.hadoop.mapreduce.lib.output.PathOutputCommitter |
DirectoryStagingCommitterFactory.createTaskCommitter(S3AFileSystem fileSystem,
org.apache.hadoop.fs.Path outputPath,
org.apache.hadoop.mapreduce.TaskAttemptContext context) |
org.apache.hadoop.mapreduce.lib.output.PathOutputCommitter |
PartitionedStagingCommitterFactory.createTaskCommitter(S3AFileSystem fileSystem,
org.apache.hadoop.fs.Path outputPath,
org.apache.hadoop.mapreduce.TaskAttemptContext context) |
| Modifier and Type | Method and Description |
|---|---|
protected S3AFileSystem |
S3GuardTool.getFilesystem() |
| Modifier and Type | Method and Description |
|---|---|
protected void |
S3GuardTool.setFilesystem(S3AFileSystem filesystem) |
Copyright © 2008–2021 Apache Software Foundation. All rights reserved.