org.apache.hadoop.fs.s3a.commit.magic

Class MagicCommitTracker

java.lang.Object

org.apache.hadoop.fs.s3a.commit.PutTracker

org.apache.hadoop.fs.s3a.commit.magic.MagicCommitTracker

Direct Known Subclasses:

InMemoryMagicCommitTracker, S3MagicCommitTracker

@InterfaceAudience.Private
public abstract class MagicCommitTracker
extends PutTracker

Put tracker for Magic commits.

Important

: must not directly or indirectly import a class which uses any datatype in hadoop-mapreduce.

Field Summary

Fields

Modifier and Type	Field and Description
protected static byte[]	EMPTY
static org.slf4j.Logger	LOG

Constructor Summary

Constructors

Constructor and Description
MagicCommitTracker(org.apache.hadoop.fs.Path path, String bucket, String originalDestKey, String destKey, String pendingsetKey, WriteOperationHelper writer, PutTrackerStatistics trackerStatistics) Magic commit tracker.

Method Summary

Modifier and Type	Method and Description
abstract boolean	aboutToComplete(String uploadId, List<software.amazon.awssdk.services.s3.model.CompletedPart> parts, long bytesWritten, org.apache.hadoop.fs.statistics.IOStatistics iostatistics) Complete operation: generate the final commit data, put it.
String	getBucket()
String	getOriginalDestKey()
org.apache.hadoop.fs.Path	getPath()
String	getPendingPartKey()
PutTrackerStatistics	getTrackerStatistics()
WriteOperationHelper	getWriter()
boolean	initialize() Initialize the tracker.
boolean	outputImmediatelyVisible() Flag to indicate that output is not visible after the stream is closed.
String	toString()

Methods inherited from class org.apache.hadoop.fs.s3a.commit.PutTracker

getDestKey

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

LOG

public static final org.slf4j.Logger LOG

EMPTY

protected static final byte[] EMPTY

Constructor Detail

MagicCommitTracker

public MagicCommitTracker(org.apache.hadoop.fs.Path path,
                          String bucket,
                          String originalDestKey,
                          String destKey,
                          String pendingsetKey,
                          WriteOperationHelper writer,
                          PutTrackerStatistics trackerStatistics)

Magic commit tracker.

Parameters:

path - path nominally being written to

bucket - dest bucket

originalDestKey - the original key, in the magic directory.

destKey - key for the destination

pendingsetKey - key of the pendingset file

writer - writer instance to use for operations; includes audit span

trackerStatistics - tracker statistics

Method Detail

initialize

public boolean initialize()
                   throws IOException

Initialize the tracker.

Overrides:

initialize in class PutTracker

Returns:

true, indicating that the multipart commit must start.

Throws:

IOException - any IO problem.

outputImmediatelyVisible

public boolean outputImmediatelyVisible()

Flag to indicate that output is not visible after the stream is closed.

Overrides:

outputImmediatelyVisible in class PutTracker

Returns:

true

aboutToComplete

public abstract boolean aboutToComplete(String uploadId,
                                        List<software.amazon.awssdk.services.s3.model.CompletedPart> parts,
                                        long bytesWritten,
                                        org.apache.hadoop.fs.statistics.IOStatistics iostatistics)
                                 throws IOException

Complete operation: generate the final commit data, put it.

Overrides:

aboutToComplete in class PutTracker

Parameters:

uploadId - Upload ID

parts - list of parts

bytesWritten - bytes written

iostatistics - nullable IO statistics

Returns:

false, indicating that the commit must fail.

Throws:

IOException - any IO problem.

IllegalArgumentException - bad argument

toString

public String toString()

Overrides:

toString in class PutTracker

getOriginalDestKey

public String getOriginalDestKey()

getPendingPartKey

public String getPendingPartKey()

getPath

public org.apache.hadoop.fs.Path getPath()

getBucket

public String getBucket()

getWriter

public WriteOperationHelper getWriter()

getTrackerStatistics

public PutTrackerStatistics getTrackerStatistics()