org.apache.hadoop.fs.s3a.s3guard

Class RenameTracker

java.lang.Object

org.apache.hadoop.fs.s3a.impl.AbstractStoreOperation

org.apache.hadoop.fs.s3a.s3guard.RenameTracker

Direct Known Subclasses:

DelayedUpdateRenameTracker, ProgressiveRenameTracker

public abstract class RenameTracker
extends AbstractStoreOperation

A class which manages updating the metastore with the rename process as initiated in the S3AFilesystem rename.

Subclasses must provide an implementation and return it in MetadataStore.initiateRenameOperation().

The operationState field/constructor argument is an opaque state to be passed down to the metastore in its move operations; this allows the stores to manage ongoing state -while still being able to share rename tracker implementations.

This is to avoid performance problems wherein the progressive rename tracker causes the store to repeatedly create and write duplicate ancestor entries for every file added.

Field Summary

Fields

Modifier and Type	Field and Description
static org.slf4j.Logger	LOG

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	RenameTracker(String name, StoreContext storeContext, MetadataStore metadataStore, org.apache.hadoop.fs.Path sourceRoot, org.apache.hadoop.fs.Path dest, BulkOperationState operationState) Constructor.

Method Summary

Modifier and Type	Method and Description
void	completeRename() Complete the operation.
protected IOException	convertToIOException(Exception ex) Convert a passed in exception (expected to be an IOE or AWS exception) into an IOException.
IOException	deleteFailed(Exception e, List<org.apache.hadoop.fs.Path> pathsToDelete, List<org.apache.hadoop.fs.Path> undeletedObjects) The delete failed.
void	directoryMarkerCopied(org.apache.hadoop.fs.Path sourcePath, org.apache.hadoop.fs.Path destPath, boolean addAncestors) A directory marker has been copied.
abstract void	fileCopied(org.apache.hadoop.fs.Path childSource, S3ObjectAttributes sourceAttributes, S3ObjectAttributes destAttributes, org.apache.hadoop.fs.Path destPath, long blockSize, boolean addAncestors) A file has been copied.
org.apache.hadoop.fs.Path	getDest()
protected MetadataStore	getMetadataStore() Get the metadata store.
BulkOperationState	getOperationState()
String	getOwner()
org.apache.hadoop.fs.Path	getSourceRoot()
void	moveSourceDirectory() Top level directory move.
protected void	noteRenameFinished() Note that the rename has finished by closing the duration info; this will log the duration of the operation at debug.
IOException	renameFailed(Exception ex) Rename has failed.
void	sourceObjectsDeleted(Collection<org.apache.hadoop.fs.Path> paths) Note that source objects have been deleted.
String	toString()

Methods inherited from class org.apache.hadoop.fs.s3a.impl.AbstractStoreOperation

getStoreContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

LOG

public static final org.slf4j.Logger LOG

Constructor Detail

RenameTracker

protected RenameTracker(String name,
                        StoreContext storeContext,
                        MetadataStore metadataStore,
                        org.apache.hadoop.fs.Path sourceRoot,
                        org.apache.hadoop.fs.Path dest,
                        BulkOperationState operationState)

Constructor.

Parameters:

name - tracker name for logs.

storeContext - store context.

metadataStore - the store

sourceRoot - source path.

dest - destination path.

operationState - ongoing move state.

Method Detail

toString

public String toString()

Overrides:

toString in class Object

getSourceRoot

public org.apache.hadoop.fs.Path getSourceRoot()

getDest

public org.apache.hadoop.fs.Path getDest()

getOwner

public String getOwner()

getOperationState

public BulkOperationState getOperationState()

getMetadataStore

protected MetadataStore getMetadataStore()

Get the metadata store.

Returns:

a non-null store.

fileCopied

public abstract void fileCopied(org.apache.hadoop.fs.Path childSource,
                                S3ObjectAttributes sourceAttributes,
                                S3ObjectAttributes destAttributes,
                                org.apache.hadoop.fs.Path destPath,
                                long blockSize,
                                boolean addAncestors)
                         throws IOException

A file has been copied.

Parameters:

childSource - source of the file. This may actually be different from the path of the sourceAttributes. (HOW?)

sourceAttributes - status of source.

destAttributes - destination attributes

destPath - destination path.

blockSize - block size.

addAncestors - should ancestors be added?

Throws:

IOException - failure.

directoryMarkerCopied

public void directoryMarkerCopied(org.apache.hadoop.fs.Path sourcePath,
                                  org.apache.hadoop.fs.Path destPath,
                                  boolean addAncestors)
                           throws IOException

A directory marker has been copied.

Parameters:

sourcePath - source path.

destPath - destination path.

addAncestors - should ancestors be added?

Throws:

IOException - failure.

deleteFailed

public IOException deleteFailed(Exception e,
                                List<org.apache.hadoop.fs.Path> pathsToDelete,
                                List<org.apache.hadoop.fs.Path> undeletedObjects)

The delete failed.

By the time this is called, the metastore will already have been updated with the results of any partial delete failure, such that all files known to have been deleted will have been removed.

Parameters:

e - exception

pathsToDelete - paths which were to be deleted.

undeletedObjects - list of objects which were not deleted.

moveSourceDirectory

public void moveSourceDirectory()
                         throws IOException

Top level directory move. This is invoked after all child entries have been copied

Throws:

IOException - on failure

sourceObjectsDeleted

public void sourceObjectsDeleted(Collection<org.apache.hadoop.fs.Path> paths)
                          throws IOException

Note that source objects have been deleted. The metastore will already have been updated.

Parameters:

paths - path of objects deleted.

Throws:

IOException

completeRename

public void completeRename()
                    throws IOException

Complete the operation.

Throws:

IOException - failure.

noteRenameFinished

protected void noteRenameFinished()

Note that the rename has finished by closing the duration info; this will log the duration of the operation at debug.

renameFailed

public IOException renameFailed(Exception ex)

Rename has failed.

The metastore now needs to be updated with its current state even though the operation is incomplete. Implementations MUST NOT throw exceptions here, as this is going to be invoked in an exception handler. catch and log or catch and return/wrap.

The base implementation returns the IOE passed in and translates any AWS exception into an IOE.

Parameters:

ex - the exception which caused the failure. This is either an IOException or and AWS exception

Returns:

an IOException to throw in an exception.

convertToIOException

protected IOException convertToIOException(Exception ex)

Convert a passed in exception (expected to be an IOE or AWS exception) into an IOException.

Parameters:

ex - exception caught

Returns:

the exception to throw in the failure handler.