org.apache.hadoop.fs.s3a.impl

Class RenameOperation

java.lang.Object

org.apache.hadoop.fs.s3a.impl.AbstractStoreOperation

org.apache.hadoop.fs.s3a.impl.ExecutingStoreOperation<Long>

org.apache.hadoop.fs.s3a.impl.RenameOperation

public class RenameOperation
extends ExecutingStoreOperation<Long>

A parallelized rename operation which updates the metastore in the process, through whichever RenameTracker the store provides. The parallel execution is in groups of size InternalConstants.RENAME_PARALLEL_LIMIT; it is only after one group completes that the next group is initiated. Once enough files have been copied that they meet the InternalConstants.MAX_ENTRIES_TO_DELETE threshold, a delete is initiated. If it succeeds, the rename continues with the next group of files. The RenameTracker has the task of keeping the metastore up to date as the rename proceeds. The rename operation implements the classic HDFS rename policy of rename(file, dir) renames the file under the directory. There is no validation of input and output paths. Callers are required to themselves verify that destination is not under the source, above the source, the source itself, etc, etc.

Constructor Summary

Constructors

Constructor and Description
RenameOperation(StoreContext storeContext, org.apache.hadoop.fs.Path sourcePath, String sourceKey, S3AFileStatus sourceStatus, org.apache.hadoop.fs.Path destPath, String destKey, S3AFileStatus destStatus, OperationCallbacks callbacks, int pageSize) Initiate the rename.

Method Summary

Modifier and Type	Method and Description
Long	execute() Execute the operation.
protected CompletableFuture<org.apache.hadoop.fs.Path>	initiateCopy(S3ALocatedFileStatus source, String key, org.apache.hadoop.fs.Path childSourcePath, String newDestKey, org.apache.hadoop.fs.Path childDestPath) Initiate a copy operation in the executor.
protected void	recursiveDirectoryRename() Execute a full recursive rename.
protected void	renameFileToDest() The source is a file: rename it to the destination.

Methods inherited from class org.apache.hadoop.fs.s3a.impl.ExecutingStoreOperation

executeOnlyOnce

Methods inherited from class org.apache.hadoop.fs.s3a.impl.AbstractStoreOperation

getStoreContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

RenameOperation

public RenameOperation(StoreContext storeContext,
                       org.apache.hadoop.fs.Path sourcePath,
                       String sourceKey,
                       S3AFileStatus sourceStatus,
                       org.apache.hadoop.fs.Path destPath,
                       String destKey,
                       S3AFileStatus destStatus,
                       OperationCallbacks callbacks,
                       int pageSize)

Initiate the rename.

Parameters:

storeContext - store context

sourcePath - source path

sourceKey - key of source

sourceStatus - pre-fetched source status

destPath - destination path.

destKey - destination key

destStatus - destination status.

callbacks - callback provider

pageSize - size of delete requests

Method Detail

execute

@Retries.RetryMixed
public Long execute()
                                 throws IOException

Description copied from class: ExecutingStoreOperation

Execute the operation. Subclasses MUST call ExecutingStoreOperation.executeOnlyOnce() so as to force the (atomic) re-entrancy check.

Specified by:

execute in class ExecutingStoreOperation<Long>

Returns:

the result.

Throws:

IOException - IO problem

renameFileToDest

protected void renameFileToDest()
                         throws IOException

The source is a file: rename it to the destination.

Throws:

IOException - failure

recursiveDirectoryRename

protected void recursiveDirectoryRename()
                                 throws IOException

Execute a full recursive rename. The source is a file: rename it to the destination.

Throws:

IOException - failure

initiateCopy

protected CompletableFuture<org.apache.hadoop.fs.Path> initiateCopy(S3ALocatedFileStatus source,
                                                                    String key,
                                                                    org.apache.hadoop.fs.Path childSourcePath,
                                                                    String newDestKey,
                                                                    org.apache.hadoop.fs.Path childDestPath)

Initiate a copy operation in the executor.

Parameters:

source - status of the source object.

key - source key

childSourcePath - source as a path.

newDestKey - destination key

childDestPath - destination path.

Returns:

the future.