org.apache.hadoop.fs.s3a.impl

Class RenameOperation

java.lang.Object

org.apache.hadoop.fs.s3a.impl.AbstractStoreOperation

org.apache.hadoop.fs.s3a.impl.ExecutingStoreOperation<Long>

org.apache.hadoop.fs.s3a.impl.RenameOperation

All Implemented Interfaces:

org.apache.hadoop.util.functional.CallableRaisingIOE<Long>

public class RenameOperation
extends ExecutingStoreOperation<Long>

A parallelized rename operation.

The parallel execution is in groups of size InternalConstants.RENAME_PARALLEL_LIMIT; it is only after one group completes that the next group is initiated.

Once enough files have been copied that they meet the InternalConstants.MAX_ENTRIES_TO_DELETE threshold, a delete is initiated. If it succeeds, the rename continues with the next group of files.

Directory Markers which have child entries are never copied; only those which represent empty directories are copied in the rename. The DirMarkerTracker tracks which markers must be copied, and which can simply be deleted from the source. As a result: rename always purges all non-leaf directory markers from the copied tree. This is to ensure that even if a directory tree is copied from an authoritative path to a non-authoritative one there is never any contamination of the non-auth path with markers.

The rename operation implements the classic HDFS rename policy of rename(file, dir) renames the file under the directory.

There is no validation of input and output paths. Callers are required to themselves verify that destination is not under the source, above the source, the source itself, etc, etc.

Constructor Summary

Constructors

Constructor and Description
RenameOperation(StoreContext storeContext, org.apache.hadoop.fs.Path sourcePath, String sourceKey, S3AFileStatus sourceStatus, org.apache.hadoop.fs.Path destPath, String destKey, S3AFileStatus destStatus, OperationCallbacks callbacks, int pageSize, boolean dirOperationsPurgeUploads) Initiate the rename.

Method Summary

Modifier and Type	Method and Description
protected IOException	convertToIOException(Exception ex) Convert a passed in exception (expected to be an IOE or AWS exception) into an IOException.
Long	execute() Execute the operation.
Optional<Long>	getUploadsAborted() Get the count of uploads aborted.
protected CompletableFuture<org.apache.hadoop.fs.Path>	initiateCopy(S3ALocatedFileStatus source, String key, String newDestKey, org.apache.hadoop.fs.Path childDestPath) Initiate a copy operation in the executor.
protected void	recursiveDirectoryRename() Execute a full recursive rename.
protected org.apache.hadoop.fs.Path	renameFileToDest() The source is a file: rename it to the destination, which will be under the current destination path if that is a directory.

Methods inherited from class org.apache.hadoop.fs.s3a.impl.ExecutingStoreOperation

apply, executeOnlyOnce

Methods inherited from class org.apache.hadoop.fs.s3a.impl.AbstractStoreOperation

activateAuditSpan, getAuditSpan, getStoreContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

RenameOperation

public RenameOperation(StoreContext storeContext,
                       org.apache.hadoop.fs.Path sourcePath,
                       String sourceKey,
                       S3AFileStatus sourceStatus,
                       org.apache.hadoop.fs.Path destPath,
                       String destKey,
                       S3AFileStatus destStatus,
                       OperationCallbacks callbacks,
                       int pageSize,
                       boolean dirOperationsPurgeUploads)

Initiate the rename.

Parameters:

storeContext - store context

sourcePath - source path

sourceKey - key of source

sourceStatus - pre-fetched source status

destPath - destination path.

destKey - destination key

destStatus - destination status.

callbacks - callback provider

pageSize - size of delete requests

dirOperationsPurgeUploads - Do directory operations purge pending uploads?

Method Detail

getUploadsAborted

public Optional<Long> getUploadsAborted()

Get the count of uploads aborted. Non-empty iff enabled, and the operations completed without errors.

Returns:

count of aborted uploads.

execute

@Retries.RetryMixed
public Long execute()
                                 throws IOException

Description copied from class: ExecutingStoreOperation

Execute the operation. Subclasses MUST call ExecutingStoreOperation.executeOnlyOnce() so as to force the (atomic) re-entrancy check.

Specified by:

execute in class ExecutingStoreOperation<Long>

Returns:

the result.

Throws:

IOException - IO problem

renameFileToDest

protected org.apache.hadoop.fs.Path renameFileToDest()
                                              throws IOException

The source is a file: rename it to the destination, which will be under the current destination path if that is a directory.

Returns:

the path of the object created.

Throws:

IOException - failure

recursiveDirectoryRename

protected void recursiveDirectoryRename()
                                 throws IOException

Execute a full recursive rename. There is a special handling of directly markers here -only leaf markers are copied. This reduces incompatibility "regions" across versions.

Throws:

IOException - failure

initiateCopy

protected CompletableFuture<org.apache.hadoop.fs.Path> initiateCopy(S3ALocatedFileStatus source,
                                                                    String key,
                                                                    String newDestKey,
                                                                    org.apache.hadoop.fs.Path childDestPath)

Initiate a copy operation in the executor.

Parameters:

source - status of the source object.

key - source key

newDestKey - destination key

childDestPath - destination path.

Returns:

the future.

convertToIOException

protected IOException convertToIOException(Exception ex)

Convert a passed in exception (expected to be an IOE or AWS exception) into an IOException.

Parameters:

ex - exception caught

Returns:

the exception to throw in the failure handler.