org.apache.hadoop.fs.s3a

Class InconsistentAmazonS3Client

java.lang.Object

com.amazonaws.AmazonWebServiceClient

com.amazonaws.services.s3.AmazonS3Client

org.apache.hadoop.fs.s3a.InconsistentAmazonS3Client

All Implemented Interfaces:

com.amazonaws.services.s3.AmazonS3, com.amazonaws.services.s3.internal.S3DirectSpi

@InterfaceAudience.Private
 @InterfaceStability.Unstable
public class InconsistentAmazonS3Client
extends com.amazonaws.services.s3.AmazonS3Client

A wrapper around AmazonS3 that injects inconsistency and/or errors. Used for testing S3Guard. Currently only delays listing visibility, not affecting GET.

Field Summary

Fields inherited from class com.amazonaws.services.s3.AmazonS3Client

awsCredentialsProvider, configFactory, errorResponseHandler, S3_SERVICE_NAME

Fields inherited from class com.amazonaws.AmazonWebServiceClient

client, clientConfiguration, endpoint, LOGGING_AWS_REQUEST_METRIC, requestHandler2s, timeOffset

Fields inherited from interface com.amazonaws.services.s3.AmazonS3

ENDPOINT_PREFIX

Constructor Summary

Constructors

Constructor and Description
InconsistentAmazonS3Client(com.amazonaws.auth.AWSCredentialsProvider credentials, com.amazonaws.ClientConfiguration clientConfiguration, org.apache.hadoop.conf.Configuration conf) Instantiate.

Method Summary

Modifier and Type	Method and Description
static InconsistentAmazonS3Client	castFrom(com.amazonaws.services.s3.AmazonS3 c) Convenience function for test code to cast from supertype.
void	clearInconsistency() Clear all oustanding inconsistent keys.
static void	clearInconsistency(S3AFileSystem fs) Clear any accumulated inconsistency state.
com.amazonaws.services.s3.model.CompleteMultipartUploadResult	completeMultipartUpload(com.amazonaws.services.s3.model.CompleteMultipartUploadRequest completeMultipartUploadRequest)
void	deleteObject(com.amazonaws.services.s3.model.DeleteObjectRequest deleteObjectRequest)
com.amazonaws.services.s3.model.DeleteObjectsResult	deleteObjects(com.amazonaws.services.s3.model.DeleteObjectsRequest deleteObjectsRequest)
long	getDelayKeyMsec()
com.amazonaws.services.s3.model.S3Object	getObject(com.amazonaws.services.s3.model.GetObjectRequest var1)
com.amazonaws.services.s3.model.S3Object	getObject(String bucketName, String key)
com.amazonaws.services.s3.model.InitiateMultipartUploadResult	initiateMultipartUpload(com.amazonaws.services.s3.model.InitiateMultipartUploadRequest initiateMultipartUploadRequest)
com.amazonaws.services.s3.model.MultipartUploadListing	listMultipartUploads(com.amazonaws.services.s3.model.ListMultipartUploadsRequest listMultipartUploadsRequest)
com.amazonaws.services.s3.model.ObjectListing	listObjects(com.amazonaws.services.s3.model.ListObjectsRequest listObjectsRequest)
com.amazonaws.services.s3.model.ListObjectsV2Result	listObjectsV2(com.amazonaws.services.s3.model.ListObjectsV2Request request)
com.amazonaws.services.s3.model.PutObjectResult	putObject(com.amazonaws.services.s3.model.PutObjectRequest putObjectRequest)
static void	setFailureInjectionPolicy(S3AFileSystem fs, FailureInjectionPolicy policy) A way for tests to patch in a different fault injection policy at runtime.
void	setFailureLimit(int limit) Set the limit on failures before all operations pass through.
void	setThrottleProbability(float throttleProbability) Set the probability of throttling a request.
String	toString()
com.amazonaws.services.s3.model.UploadPartResult	uploadPart(com.amazonaws.services.s3.model.UploadPartRequest uploadPartRequest)

Methods inherited from class com.amazonaws.services.s3.AmazonS3Client

abortMultipartUpload, builder, changeObjectStorageClass, copyObject, copyObject, copyPart, createBucket, createBucket, createBucket, createBucket, createRequest, createRequest, createSigner, createSigner, createSignerProvider, deleteBucket, deleteBucket, deleteBucketAnalyticsConfiguration, deleteBucketAnalyticsConfiguration, deleteBucketCrossOriginConfiguration, deleteBucketCrossOriginConfiguration, deleteBucketEncryption, deleteBucketEncryption, deleteBucketInventoryConfiguration, deleteBucketInventoryConfiguration, deleteBucketLifecycleConfiguration, deleteBucketLifecycleConfiguration, deleteBucketMetricsConfiguration, deleteBucketMetricsConfiguration, deleteBucketPolicy, deleteBucketPolicy, deleteBucketReplicationConfiguration, deleteBucketReplicationConfiguration, deleteBucketTaggingConfiguration, deleteBucketTaggingConfiguration, deleteBucketWebsiteConfiguration, deleteBucketWebsiteConfiguration, deleteObject, deleteObjectTagging, deletePublicAccessBlock, deleteVersion, deleteVersion, disableRequesterPays, doesBucketExist, doesBucketExistV2, doesObjectExist, download, download, enableRequesterPays, generatePresignedUrl, generatePresignedUrl, generatePresignedUrl, getBucketAccelerateConfiguration, getBucketAccelerateConfiguration, getBucketAcl, getBucketAcl, getBucketAnalyticsConfiguration, getBucketAnalyticsConfiguration, getBucketCrossOriginConfiguration, getBucketCrossOriginConfiguration, getBucketEncryption, getBucketEncryption, getBucketInventoryConfiguration, getBucketInventoryConfiguration, getBucketLifecycleConfiguration, getBucketLifecycleConfiguration, getBucketLocation, getBucketLocation, getBucketLoggingConfiguration, getBucketLoggingConfiguration, getBucketMetricsConfiguration, getBucketMetricsConfiguration, getBucketNotificationConfiguration, getBucketNotificationConfiguration, getBucketPolicy, getBucketPolicy, getBucketPolicyStatus, getBucketReplicationConfiguration, getBucketReplicationConfiguration, getBucketTaggingConfiguration, getBucketTaggingConfiguration, getBucketVersioningConfiguration, getBucketVersioningConfiguration, getBucketWebsiteConfiguration, getBucketWebsiteConfiguration, getCachedResponseMetadata, getObject, getObjectAcl, getObjectAcl, getObjectAcl, getObjectAsString, getObjectLegalHold, getObjectLockConfiguration, getObjectMetadata, getObjectMetadata, getObjectRetention, getObjectTagging, getPublicAccessBlock, getRegion, getRegionName, getResourceUrl, getS3AccountOwner, getS3AccountOwner, getUrl, headBucket, isRequesterPaysEnabled, listBucketAnalyticsConfigurations, listBucketInventoryConfigurations, listBucketMetricsConfigurations, listBuckets, listBuckets, listNextBatchOfObjects, listNextBatchOfObjects, listNextBatchOfVersions, listNextBatchOfVersions, listObjects, listObjects, listObjectsV2, listObjectsV2, listParts, listVersions, listVersions, listVersions, newInitiateMultipartUploadRequest, populateRequesterPaysHeader, populateRequestMetadata, presignRequest, putObject, putObject, putObject, restoreObject, restoreObject, restoreObjectV2, selectObjectContent, setBucketAccelerateConfiguration, setBucketAccelerateConfiguration, setBucketAcl, setBucketAcl, setBucketAcl, setBucketAcl, setBucketAcl, setBucketAnalyticsConfiguration, setBucketAnalyticsConfiguration, setBucketCrossOriginConfiguration, setBucketCrossOriginConfiguration, setBucketEncryption, setBucketInventoryConfiguration, setBucketInventoryConfiguration, setBucketLifecycleConfiguration, setBucketLifecycleConfiguration, setBucketLoggingConfiguration, setBucketMetricsConfiguration, setBucketMetricsConfiguration, setBucketNotificationConfiguration, setBucketNotificationConfiguration, setBucketPolicy, setBucketPolicy, setBucketReplicationConfiguration, setBucketReplicationConfiguration, setBucketTaggingConfiguration, setBucketTaggingConfiguration, setBucketVersioningConfiguration, setBucketWebsiteConfiguration, setBucketWebsiteConfiguration, setEndpoint, setObjectAcl, setObjectAcl, setObjectAcl, setObjectAcl, setObjectAcl, setObjectAcl, setObjectAcl, setObjectLegalHold, setObjectLockConfiguration, setObjectRedirectLocation, setObjectRetention, setObjectTagging, setPublicAccessBlock, setRegion, setS3ClientOptions, upload, useStrictHostNameVerification, waiters

Methods inherited from class com.amazonaws.AmazonWebServiceClient

addRequestHandler, addRequestHandler, beforeClientExecution, beforeMarshalling, calculateCRC32FromCompressedData, checkMutability, configureRegion, createExecutionContext, createExecutionContext, createExecutionContext, endClientExecution, endClientExecution, getClientConfiguration, getClientId, getEndpointPrefix, getMonitoringListeners, getRequestMetricsCollector, getServiceAbbreviation, getServiceName, getServiceNameIntern, getSigner, getSignerByURI, getSignerOverride, getSignerProvider, getSignerRegionOverride, getSigningRegion, getTimeOffset, isCsmEnabled, isProfilingEnabled, isRequestMetricsEnabled, makeImmutable, removeRequestHandler, removeRequestHandler, requestMetricCollector, setEndpoint, setEndpointPrefix, setServiceNameIntern, setSignerRegionOverride, setTimeOffset, shouldGenerateClientSideMonitoringEvents, shutdown, withEndpoint, withRegion, withRegion, withTimeOffset

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface com.amazonaws.services.s3.AmazonS3

shutdown

Constructor Detail

InconsistentAmazonS3Client

public InconsistentAmazonS3Client(com.amazonaws.auth.AWSCredentialsProvider credentials,
                                  com.amazonaws.ClientConfiguration clientConfiguration,
                                  org.apache.hadoop.conf.Configuration conf)

Instantiate. This subclasses a deprecated constructor of the parent AmazonS3Client class; we can't use the builder API because, that only creates the consistent client.

Parameters:

credentials - credentials to auth.

clientConfiguration - connection settings

conf - hadoop configuration.

Method Detail

clearInconsistency

public static void clearInconsistency(S3AFileSystem fs)
                               throws Exception

Clear any accumulated inconsistency state. Used by tests to make paths visible again.

Parameters:

fs - S3AFileSystem under test

Throws:

Exception - on failure

setFailureInjectionPolicy

public static void setFailureInjectionPolicy(S3AFileSystem fs,
                                             FailureInjectionPolicy policy)
                                      throws Exception

A way for tests to patch in a different fault injection policy at runtime.

Parameters:

fs - filesystem under test

policy - failure injection settings to set

Throws:

Exception - on failure

toString

public String toString()

Overrides:

toString in class Object

clearInconsistency

public void clearInconsistency()

Clear all oustanding inconsistent keys. After calling this function, listings should behave normally (no failure injection), until additional keys are matched for delay, e.g. via putObject(), deleteObject().

castFrom

public static InconsistentAmazonS3Client castFrom(com.amazonaws.services.s3.AmazonS3 c)
                                           throws Exception

Convenience function for test code to cast from supertype.

Parameters:

c - supertype to cast from

Returns:

subtype, not null

Throws:

Exception - on error

deleteObjects

public com.amazonaws.services.s3.model.DeleteObjectsResult deleteObjects(com.amazonaws.services.s3.model.DeleteObjectsRequest deleteObjectsRequest)
                                                                  throws com.amazonaws.AmazonClientException,
                                                                         com.amazonaws.AmazonServiceException

Specified by:

deleteObjects in interface com.amazonaws.services.s3.AmazonS3

Overrides:

deleteObjects in class com.amazonaws.services.s3.AmazonS3Client

Throws:

com.amazonaws.AmazonClientException

com.amazonaws.AmazonServiceException

deleteObject

public void deleteObject(com.amazonaws.services.s3.model.DeleteObjectRequest deleteObjectRequest)
                  throws com.amazonaws.AmazonClientException,
                         com.amazonaws.AmazonServiceException

Specified by:

deleteObject in interface com.amazonaws.services.s3.AmazonS3

Overrides:

deleteObject in class com.amazonaws.services.s3.AmazonS3Client

Throws:

com.amazonaws.AmazonClientException

com.amazonaws.AmazonServiceException

putObject

public com.amazonaws.services.s3.model.PutObjectResult putObject(com.amazonaws.services.s3.model.PutObjectRequest putObjectRequest)
                                                          throws com.amazonaws.AmazonClientException,
                                                                 com.amazonaws.AmazonServiceException

Specified by:

putObject in interface com.amazonaws.services.s3.AmazonS3

Specified by:

putObject in interface com.amazonaws.services.s3.internal.S3DirectSpi

Overrides:

putObject in class com.amazonaws.services.s3.AmazonS3Client

Throws:

com.amazonaws.AmazonClientException

com.amazonaws.AmazonServiceException

listObjects

public com.amazonaws.services.s3.model.ObjectListing listObjects(com.amazonaws.services.s3.model.ListObjectsRequest listObjectsRequest)
                                                          throws com.amazonaws.AmazonClientException,
                                                                 com.amazonaws.AmazonServiceException

Specified by:

listObjects in interface com.amazonaws.services.s3.AmazonS3

Overrides:

listObjects in class com.amazonaws.services.s3.AmazonS3Client

Throws:

com.amazonaws.AmazonClientException

com.amazonaws.AmazonServiceException

listObjectsV2

public com.amazonaws.services.s3.model.ListObjectsV2Result listObjectsV2(com.amazonaws.services.s3.model.ListObjectsV2Request request)
                                                                  throws com.amazonaws.AmazonClientException,
                                                                         com.amazonaws.AmazonServiceException

Specified by:

listObjectsV2 in interface com.amazonaws.services.s3.AmazonS3

Overrides:

listObjectsV2 in class com.amazonaws.services.s3.AmazonS3Client

Throws:

com.amazonaws.AmazonClientException

com.amazonaws.AmazonServiceException

completeMultipartUpload

public com.amazonaws.services.s3.model.CompleteMultipartUploadResult completeMultipartUpload(com.amazonaws.services.s3.model.CompleteMultipartUploadRequest completeMultipartUploadRequest)
                                                                                      throws com.amazonaws.SdkClientException,
                                                                                             com.amazonaws.AmazonServiceException

Specified by:

completeMultipartUpload in interface com.amazonaws.services.s3.AmazonS3

Specified by:

completeMultipartUpload in interface com.amazonaws.services.s3.internal.S3DirectSpi

Overrides:

completeMultipartUpload in class com.amazonaws.services.s3.AmazonS3Client

Throws:

com.amazonaws.SdkClientException

com.amazonaws.AmazonServiceException

uploadPart

public com.amazonaws.services.s3.model.UploadPartResult uploadPart(com.amazonaws.services.s3.model.UploadPartRequest uploadPartRequest)
                                                            throws com.amazonaws.SdkClientException,
                                                                   com.amazonaws.AmazonServiceException

Specified by:

uploadPart in interface com.amazonaws.services.s3.AmazonS3

Specified by:

uploadPart in interface com.amazonaws.services.s3.internal.S3DirectSpi

Overrides:

uploadPart in class com.amazonaws.services.s3.AmazonS3Client

Throws:

com.amazonaws.SdkClientException

com.amazonaws.AmazonServiceException

initiateMultipartUpload

public com.amazonaws.services.s3.model.InitiateMultipartUploadResult initiateMultipartUpload(com.amazonaws.services.s3.model.InitiateMultipartUploadRequest initiateMultipartUploadRequest)
                                                                                      throws com.amazonaws.SdkClientException,
                                                                                             com.amazonaws.AmazonServiceException

Specified by:

initiateMultipartUpload in interface com.amazonaws.services.s3.AmazonS3

Specified by:

initiateMultipartUpload in interface com.amazonaws.services.s3.internal.S3DirectSpi

Overrides:

initiateMultipartUpload in class com.amazonaws.services.s3.AmazonS3Client

Throws:

com.amazonaws.SdkClientException

com.amazonaws.AmazonServiceException

listMultipartUploads

public com.amazonaws.services.s3.model.MultipartUploadListing listMultipartUploads(com.amazonaws.services.s3.model.ListMultipartUploadsRequest listMultipartUploadsRequest)
                                                                            throws com.amazonaws.SdkClientException,
                                                                                   com.amazonaws.AmazonServiceException

Specified by:

listMultipartUploads in interface com.amazonaws.services.s3.AmazonS3

Overrides:

listMultipartUploads in class com.amazonaws.services.s3.AmazonS3Client

Throws:

com.amazonaws.SdkClientException

com.amazonaws.AmazonServiceException

getDelayKeyMsec

public long getDelayKeyMsec()

setThrottleProbability

public void setThrottleProbability(float throttleProbability)

Set the probability of throttling a request.

Parameters:

throttleProbability - the probability of a request being throttled.

setFailureLimit

public void setFailureLimit(int limit)

Set the limit on failures before all operations pass through. This resets the failure count.

Parameters:

limit - limit; "0" means "no limit"

getObject

public com.amazonaws.services.s3.model.S3Object getObject(com.amazonaws.services.s3.model.GetObjectRequest var1)
                                                   throws com.amazonaws.SdkClientException,
                                                          com.amazonaws.AmazonServiceException

Specified by:

getObject in interface com.amazonaws.services.s3.AmazonS3

Specified by:

getObject in interface com.amazonaws.services.s3.internal.S3DirectSpi

Overrides:

getObject in class com.amazonaws.services.s3.AmazonS3Client

Throws:

com.amazonaws.SdkClientException

com.amazonaws.AmazonServiceException

getObject

public com.amazonaws.services.s3.model.S3Object getObject(String bucketName,
                                                          String key)
                                                   throws com.amazonaws.SdkClientException,
                                                          com.amazonaws.AmazonServiceException

Specified by:

getObject in interface com.amazonaws.services.s3.AmazonS3

Overrides:

getObject in class com.amazonaws.services.s3.AmazonS3Client

Throws:

com.amazonaws.SdkClientException

com.amazonaws.AmazonServiceException