org.apache.hadoop.fs.s3a.impl

Class DeleteOperation

java.lang.Object

org.apache.hadoop.fs.s3a.impl.AbstractStoreOperation

org.apache.hadoop.fs.s3a.impl.ExecutingStoreOperation<Boolean>

org.apache.hadoop.fs.s3a.impl.DeleteOperation

All Implemented Interfaces:

org.apache.hadoop.util.functional.CallableRaisingIOE<Boolean>

public class DeleteOperation
extends ExecutingStoreOperation<Boolean>

Implementation of the delete() operation. This issues only one bulk delete at a time, intending to update S3Guard after every request succeeded. Now that S3Guard has been removed, it would be possible to issue multiple delete calls in parallel. If this is done, then it may be good to experiment with different page sizes. The default value is InternalConstants.MAX_ENTRIES_TO_DELETE, the maximum a single POST permits.

Smaller pages executed in parallel may have different performance characteristics when deleting very large directories. Any exploration of options here MUST be done with performance measurements taken from test runs in EC2 against local S3 stores, so as to ensure network latencies do not skew the results.

Constructor Summary

Constructors

Constructor and Description
DeleteOperation(StoreContext context, S3AFileStatus status, boolean recursive, OperationCallbacks callbacks, int pageSize, boolean dirOperationsPurgeUploads) Constructor.

Method Summary

Modifier and Type	Method and Description
protected void	deleteDirectoryTree(org.apache.hadoop.fs.Path path, String dirKey) Delete a directory tree.
Boolean	execute() Delete a file or directory tree.
long	getFilesDeleted()
Optional<Long>	getUploadsAborted() Get the count of uploads aborted.

Methods inherited from class org.apache.hadoop.fs.s3a.impl.ExecutingStoreOperation

apply, executeOnlyOnce

Methods inherited from class org.apache.hadoop.fs.s3a.impl.AbstractStoreOperation

activateAuditSpan, getAuditSpan, getStoreContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DeleteOperation

public DeleteOperation(StoreContext context,
                       S3AFileStatus status,
                       boolean recursive,
                       OperationCallbacks callbacks,
                       int pageSize,
                       boolean dirOperationsPurgeUploads)

Constructor.

Parameters:

context - store context

status - pre-fetched source status

recursive - recursive delete?

callbacks - callback provider

pageSize - size of delete pages

dirOperationsPurgeUploads - Do directory operations purge pending uploads?

Method Detail

getFilesDeleted

public long getFilesDeleted()

getUploadsAborted

public Optional<Long> getUploadsAborted()

Get the count of uploads aborted. Non-empty iff enabled, and the operations completed without errors.

Returns:

count of aborted uploads.

execute

@Retries.RetryTranslated
public Boolean execute()
                                         throws IOException

Delete a file or directory tree.

This call does not create any fake parent directory; that is left to the caller. The actual delete call is done in a separate thread. Only one delete at a time is submitted, however, to reduce the complexity of recovering from failures.

With S3Guard removed, the problem of updating any DynamoDB store has gone away -delete calls could now be issued in parallel. However, rate limiting may be required to keep write load below the throttling point. Every entry in a single bulk delete call counts as a single write request -overloading an S3 partition with delete calls has been a problem in the past.

Specified by:

execute in class ExecutingStoreOperation<Boolean>

Returns:

true, except in the corner cases of root directory deletion

Throws:

org.apache.hadoop.fs.PathIsNotEmptyDirectoryException - if the path is a dir and this is not a recursive delete.

IOException - list failures or an inability to delete a file.

deleteDirectoryTree

@Retries.RetryTranslated
protected void deleteDirectoryTree(org.apache.hadoop.fs.Path path,
                                                            String dirKey)
                                                     throws IOException

Delete a directory tree.

This is done by asking the filesystem for a list of all objects under the directory path.

Once the first pageSize worth of objects has been listed, a batch delete is queued for execution in a separate thread; subsequent batches block waiting for the first call to complete or fail before again, being deleted in the separate thread.

After all listed objects are queued for deletion,

Parameters:

path - directory path

dirKey - directory key

Throws:

IOException - failure