Package | Description |
---|---|
org.apache.hadoop.fs.s3a.commit |
Support for committing the output of analytics jobs directly to S3.
|
org.apache.hadoop.fs.s3a.commit.magic |
This is the "Magic" committer and support.
|
org.apache.hadoop.fs.s3a.commit.staging |
The staging committers.
|
Modifier and Type | Method and Description |
---|---|
static AbstractS3ACommitter.ActiveCommit |
AbstractS3ACommitter.ActiveCommit.empty()
Get the empty entry.
|
static AbstractS3ACommitter.ActiveCommit |
AbstractS3ACommitter.ActiveCommit.fromStatusIterator(org.apache.hadoop.fs.FileSystem pendingFS,
org.apache.hadoop.fs.RemoteIterator<? extends org.apache.hadoop.fs.FileStatus> statuses)
Create an active commit of the given pending files.
|
protected abstract AbstractS3ACommitter.ActiveCommit |
AbstractS3ACommitter.listPendingUploadsToCommit(CommitContext commitContext)
Get the list of pending uploads for this job attempt.
|
Modifier and Type | Method and Description |
---|---|
protected void |
AbstractS3ACommitter.abortPendingUploads(CommitContext commitContext,
AbstractS3ACommitter.ActiveCommit pending,
boolean suppressExceptions,
boolean deleteRemoteFiles)
Abort all pending uploads in the list.
|
protected void |
AbstractS3ACommitter.commitJobInternal(CommitContext commitContext,
AbstractS3ACommitter.ActiveCommit pending)
Internal Job commit operation: where the S3 requests are made
(potentially in parallel).
|
protected void |
AbstractS3ACommitter.commitPendingUploads(CommitContext commitContext,
AbstractS3ACommitter.ActiveCommit pending)
Commit all the pending uploads.
|
protected SuccessData |
AbstractS3ACommitter.maybeCreateSuccessMarkerFromCommits(CommitContext commitContext,
AbstractS3ACommitter.ActiveCommit pending)
if the job requires a success marker on a successful job,
create the file
CommitConstants._SUCCESS . |
protected void |
AbstractS3ACommitter.precommitCheckPendingFiles(CommitContext commitContext,
AbstractS3ACommitter.ActiveCommit pending)
Run a precommit check that all files are loadable.
|
void |
AbstractS3ACommitter.preCommitJob(CommitContext commitContext,
AbstractS3ACommitter.ActiveCommit pending)
Subclass-specific pre-Job-commit actions.
|
Modifier and Type | Method and Description |
---|---|
protected AbstractS3ACommitter.ActiveCommit |
MagicS3GuardCommitter.listPendingUploadsToCommit(CommitContext commitContext)
Get the list of pending uploads for this job attempt, by listing
all .pendingset files in the job attempt directory.
|
Modifier and Type | Method and Description |
---|---|
protected AbstractS3ACommitter.ActiveCommit |
StagingCommitter.listPendingUploads(CommitContext commitContext,
boolean suppressExceptions)
Get the list of pending uploads for this job attempt.
|
protected AbstractS3ACommitter.ActiveCommit |
StagingCommitter.listPendingUploadsToAbort(CommitContext commitContext)
Get the list of pending uploads for this job attempt, swallowing
exceptions.
|
protected AbstractS3ACommitter.ActiveCommit |
StagingCommitter.listPendingUploadsToCommit(CommitContext commitContext)
Get the list of pending uploads for this job attempt.
|
Modifier and Type | Method and Description |
---|---|
void |
StagingCommitter.preCommitJob(CommitContext commitContext,
AbstractS3ACommitter.ActiveCommit pending)
Pre-commit actions for a job.
|
void |
PartitionedStagingCommitter.preCommitJob(CommitContext commitContext,
AbstractS3ACommitter.ActiveCommit pending)
All
Job-side conflict resolution.
|
void |
DirectoryStagingCommitter.preCommitJob(CommitContext commitContext,
AbstractS3ACommitter.ActiveCommit pending)
Pre-commit actions for a job.
|
Copyright © 2008–2024 Apache Software Foundation. All rights reserved.