org.apache.hadoop.fs.s3a.impl

Class MultiObjectDeleteSupport

java.lang.Object

org.apache.hadoop.fs.s3a.impl.AbstractStoreOperation

org.apache.hadoop.fs.s3a.impl.MultiObjectDeleteSupport

public final class MultiObjectDeleteSupport
extends AbstractStoreOperation

Support for Multi Object Deletion.

Field Summary

Fields

Modifier and Type	Field and Description
static String	ACCESS_DENIED This is the exception exit code if access was denied on a delete.

Constructor Summary

Constructors

Constructor and Description
MultiObjectDeleteSupport(StoreContext context, BulkOperationState operationState) Initiate with a store context.

Method Summary

Modifier and Type	Method and Description
static List<org.apache.hadoop.fs.Path>	convertToPaths(Collection<com.amazonaws.services.s3.model.DeleteObjectsRequest.KeyVersion> keysToDelete, Function<String,org.apache.hadoop.fs.Path> qualifier) Given a list of delete requests, convert them all to paths.
static List<org.apache.hadoop.fs.Path>	extractUndeletedPaths(com.amazonaws.services.s3.model.MultiObjectDeleteException deleteException, Function<String,org.apache.hadoop.fs.Path> qualifierFn) Build a list of undeleted paths from a MultiObjectDeleteException.
List<org.apache.hadoop.fs.Path>	keysToPaths(Collection<com.amazonaws.services.s3.model.DeleteObjectsRequest.KeyVersion> keysToDelete) Given a list of delete requests, convert them all to paths.
org.apache.commons.lang3.tuple.Triple<List<org.apache.hadoop.fs.Path>,List<org.apache.hadoop.fs.Path>,List<org.apache.commons.lang3.tuple.Pair<org.apache.hadoop.fs.Path,IOException>>>	processDeleteFailure(com.amazonaws.services.s3.model.MultiObjectDeleteException deleteException, List<com.amazonaws.services.s3.model.DeleteObjectsRequest.KeyVersion> keysToDelete) Process a delete failure by removing from the metastore all entries which where deleted, as inferred from the delete failures exception and the original list of files to delete declares to have been deleted.
List<org.apache.hadoop.fs.Path>	processDeleteFailureGenericException(Exception ex, List<com.amazonaws.services.s3.model.DeleteObjectsRequest.KeyVersion> keysToDelete) A delete operation failed.
org.apache.commons.lang3.tuple.Pair<List<org.apache.hadoop.fs.Path>,List<org.apache.hadoop.fs.Path>>	splitUndeletedKeys(com.amazonaws.services.s3.model.MultiObjectDeleteException deleteException, Collection<com.amazonaws.services.s3.model.DeleteObjectsRequest.KeyVersion> keysToDelete) Process a multi object delete exception by building two paths from the delete request: one of all deleted files, one of all undeleted values.
static IOException	translateDeleteException(String message, com.amazonaws.services.s3.model.MultiObjectDeleteException deleteException) A MultiObjectDeleteException is raised if one or more paths listed in a bulk DELETE operation failed.

Methods inherited from class org.apache.hadoop.fs.s3a.impl.AbstractStoreOperation

getStoreContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

ACCESS_DENIED

public static final String ACCESS_DENIED

This is the exception exit code if access was denied on a delete. "AccessDenied".

See Also:

Constant Field Values

Constructor Detail

MultiObjectDeleteSupport

public MultiObjectDeleteSupport(StoreContext context,
                                BulkOperationState operationState)

Initiate with a store context.

Parameters:

context - store context.

operationState - any ongoing bulk operation.

Method Detail

translateDeleteException

public static IOException translateDeleteException(String message,
                                                   com.amazonaws.services.s3.model.MultiObjectDeleteException deleteException)

A MultiObjectDeleteException is raised if one or more paths listed in a bulk DELETE operation failed. The top-level exception is therefore just "something wasn't deleted", but doesn't include the what or the why. This translation will extract an AccessDeniedException if that's one of the causes, otherwise grabs the status code and uses it in the returned exception.

Parameters:

message - text for the exception

deleteException - the delete exception. to translate

Returns:

an IOE with more detail.

splitUndeletedKeys

public org.apache.commons.lang3.tuple.Pair<List<org.apache.hadoop.fs.Path>,List<org.apache.hadoop.fs.Path>> splitUndeletedKeys(com.amazonaws.services.s3.model.MultiObjectDeleteException deleteException,
                                                                                                                               Collection<com.amazonaws.services.s3.model.DeleteObjectsRequest.KeyVersion> keysToDelete)

Process a multi object delete exception by building two paths from the delete request: one of all deleted files, one of all undeleted values. The latter are those rejected in the delete call.

Parameters:

deleteException - the delete exception.

keysToDelete - the keys in the delete request

Returns:

tuple of (undeleted, deleted) paths.

keysToPaths

public List<org.apache.hadoop.fs.Path> keysToPaths(Collection<com.amazonaws.services.s3.model.DeleteObjectsRequest.KeyVersion> keysToDelete)

Given a list of delete requests, convert them all to paths.

Parameters:

keysToDelete - list of keys for the delete operation.

Returns:

the paths.

convertToPaths

public static List<org.apache.hadoop.fs.Path> convertToPaths(Collection<com.amazonaws.services.s3.model.DeleteObjectsRequest.KeyVersion> keysToDelete,
                                                             Function<String,org.apache.hadoop.fs.Path> qualifier)

Given a list of delete requests, convert them all to paths.

Parameters:

keysToDelete - list of keys for the delete operation.

qualifier - path qualifier

Returns:

the paths.

processDeleteFailure

public org.apache.commons.lang3.tuple.Triple<List<org.apache.hadoop.fs.Path>,List<org.apache.hadoop.fs.Path>,List<org.apache.commons.lang3.tuple.Pair<org.apache.hadoop.fs.Path,IOException>>> processDeleteFailure(com.amazonaws.services.s3.model.MultiObjectDeleteException deleteException,
                                                                                                                                                                                                                    List<com.amazonaws.services.s3.model.DeleteObjectsRequest.KeyVersion> keysToDelete)

Process a delete failure by removing from the metastore all entries which where deleted, as inferred from the delete failures exception and the original list of files to delete declares to have been deleted.

Parameters:

deleteException - the delete exception.

keysToDelete - collection of keys which had been requested.

Returns:

a tuple of (undeleted, deleted, failures)

extractUndeletedPaths

public static List<org.apache.hadoop.fs.Path> extractUndeletedPaths(com.amazonaws.services.s3.model.MultiObjectDeleteException deleteException,
                                                                    Function<String,org.apache.hadoop.fs.Path> qualifierFn)

Build a list of undeleted paths from a MultiObjectDeleteException. Outside of unit tests, the qualifier function should be S3AFileSystem.keyToQualifiedPath(String).

Parameters:

deleteException - the delete exception.

qualifierFn - function to qualify paths

Returns:

the possibly empty list of paths.

processDeleteFailureGenericException

public List<org.apache.hadoop.fs.Path> processDeleteFailureGenericException(Exception ex,
                                                                            List<com.amazonaws.services.s3.model.DeleteObjectsRequest.KeyVersion> keysToDelete)

A delete operation failed. Currently just returns the list of all paths.

Parameters:

ex - exception.

keysToDelete - the keys which were being deleted.

Returns:

all paths which were not deleted.