org.apache.hadoop.fs.s3a.commit

Class AbstractS3ACommitter.ActiveCommit

java.lang.Object

org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter.ActiveCommit

Enclosing class:

AbstractS3ACommitter

public static final class AbstractS3ACommitter.ActiveCommit
extends Object

State of the active commit operation. It contains a list of all pendingset files to load as the source of outstanding commits to complete/abort, and tracks the files uploaded. To avoid running out of heap by loading all the source files simultaneously:

1. The list of files to load is passed round but the contents are only loaded on demand.
2. The number of written files tracked for logging in the _SUCCESS file are limited to a small amount -enough for testing only.

Constructor Summary

Constructors

Constructor and Description
ActiveCommit(org.apache.hadoop.fs.FileSystem sourceFS, List<? extends org.apache.hadoop.fs.FileStatus> sourceFiles) Construct from a source FS and list of files.

Method Summary

Modifier and Type	Method and Description
void	add(org.apache.hadoop.fs.FileStatus status)
static AbstractS3ACommitter.ActiveCommit	empty() Get the empty entry.
static AbstractS3ACommitter.ActiveCommit	fromStatusIterator(org.apache.hadoop.fs.FileSystem pendingFS, org.apache.hadoop.fs.RemoteIterator<? extends org.apache.hadoop.fs.FileStatus> statuses) Create an active commit of the given pending files.
long	getCommittedBytes()
int	getCommittedFileCount()
List<String>	getCommittedObjects()
org.apache.hadoop.fs.statistics.IOStatisticsSnapshot	getIOStatistics()
List<org.apache.hadoop.fs.FileStatus>	getSourceFiles()
org.apache.hadoop.fs.FileSystem	getSourceFS()
boolean	isEmpty()
void	pendingsetCommitted(org.apache.hadoop.fs.statistics.IOStatistics sourceStatistics) Callback when a pendingset has been committed, including any source statistics.
int	size()
void	uploadCommitted(String key, long size) Note that a file was committed.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ActiveCommit

public ActiveCommit(org.apache.hadoop.fs.FileSystem sourceFS,
                    List<? extends org.apache.hadoop.fs.FileStatus> sourceFiles)

Construct from a source FS and list of files.

Parameters:

sourceFS - filesystem containing the list of pending files

sourceFiles - .pendingset files to load and commit.

Method Detail

fromStatusIterator

public static AbstractS3ACommitter.ActiveCommit fromStatusIterator(org.apache.hadoop.fs.FileSystem pendingFS,
                                                                   org.apache.hadoop.fs.RemoteIterator<? extends org.apache.hadoop.fs.FileStatus> statuses)
                                                            throws IOException

Create an active commit of the given pending files.

Parameters:

pendingFS - source filesystem.

statuses - iterator of file status or subclass to use.

Returns:

the commit

Throws:

IOException - if the iterator raises one.

empty

public static AbstractS3ACommitter.ActiveCommit empty()

Get the empty entry.

Returns:

an active commit with no pending files.

getSourceFiles

public List<org.apache.hadoop.fs.FileStatus> getSourceFiles()

getSourceFS

public org.apache.hadoop.fs.FileSystem getSourceFS()

uploadCommitted

public void uploadCommitted(String key,
                            long size)

Note that a file was committed. Increase the counter of files and total size. If there is room in the committedFiles list, the file will be added to the list and so end up in the _SUCCESS file.

Parameters:

key - key of the committed object.

size - size in bytes.

pendingsetCommitted

public void pendingsetCommitted(org.apache.hadoop.fs.statistics.IOStatistics sourceStatistics)

Callback when a pendingset has been committed, including any source statistics.

Parameters:

sourceStatistics - any source statistics

getIOStatistics

public org.apache.hadoop.fs.statistics.IOStatisticsSnapshot getIOStatistics()

getCommittedObjects

public List<String> getCommittedObjects()

getCommittedFileCount

public int getCommittedFileCount()

getCommittedBytes

public long getCommittedBytes()

size

public int size()

isEmpty

public boolean isEmpty()

add

public void add(org.apache.hadoop.fs.FileStatus status)