org.apache.hadoop.fs.s3a.commit

Class MagicCommitIntegration

java.lang.Object

org.apache.hadoop.fs.s3a.commit.MagicCommitIntegration

public class MagicCommitIntegration
extends Object

Adds the code needed for S3A to support magic committers. It's pulled out to keep S3A FS class slightly less complex. This class can be instantiated even when magic commit is disabled; in this case:

1. isMagicCommitPath(Path) will always return false.
2. createTracker(Path, String) will always return an instance of PutTracker.

Important

: must not directly or indirectly import a class which uses any datatype in hadoop-mapreduce.

Constructor Summary

Constructors

Constructor and Description
MagicCommitIntegration(S3AFileSystem owner, boolean magicCommitEnabled) Instantiate.

Method Summary

Modifier and Type	Method and Description
PutTracker	createTracker(org.apache.hadoop.fs.Path path, String key) Given a path and a key to that same path, create a tracker for it.
boolean	isMagicCommitEnabled() Is magic commit enabled?
boolean	isMagicCommitPath(org.apache.hadoop.fs.Path path) Predicate: is a path a magic commit path?
String	keyOfFinalDestination(List<String> elements, String key) Given an (elements, key) pair, return the key of the final destination of the PUT, that is: where the final path is expected to go?

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

MagicCommitIntegration

public MagicCommitIntegration(S3AFileSystem owner,
                              boolean magicCommitEnabled)

Instantiate.

Parameters:

owner - owner class

magicCommitEnabled - is magic commit enabled.

Method Detail

keyOfFinalDestination

public String keyOfFinalDestination(List<String> elements,
                                    String key)

Given an (elements, key) pair, return the key of the final destination of the PUT, that is: where the final path is expected to go?

Parameters:

elements - path split to elements

key - key

Returns:

key for final put. If this is not a magic commit, the same as the key in.

createTracker

public PutTracker createTracker(org.apache.hadoop.fs.Path path,
                                String key)

Given a path and a key to that same path, create a tracker for it. This specific tracker will be chosen based on whether or not the path is a magic one.

Parameters:

path - path of nominal write

key - key of path of nominal write

Returns:

the tracker for this operation.

isMagicCommitEnabled

public boolean isMagicCommitEnabled()

Is magic commit enabled?

Returns:

true if magic commit is turned on.

isMagicCommitPath

public boolean isMagicCommitPath(org.apache.hadoop.fs.Path path)

Predicate: is a path a magic commit path?

Parameters:

path - path to examine

Returns:

true if the path is or is under a magic directory