org.apache.hadoop.fs.s3a

Class S3AUtils

java.lang.Object

org.apache.hadoop.fs.s3a.S3AUtils

@InterfaceAudience.Private
 @InterfaceStability.Evolving
public final class S3AUtils
extends Object

Utility methods for S3A code.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	S3AUtils.CallOnLocatedFileStatus An interface for use in lambda-expressions working with directory tree listings.
static interface	S3AUtils.LocatedFileStatusMap<T> An interface for use in lambda-expressions working with directory tree listings.

Field Summary

Fields

Modifier and Type	Field and Description
static org.apache.hadoop.fs.PathFilter	ACCEPT_ALL A Path filter which accepts all filenames.
static org.apache.hadoop.fs.PathFilter	HIDDEN_FILE_FILTER Path filter which ignores any file which starts with .
static String	SSE_C_NO_KEY_ERROR Encryption SSE-C used but the config lacks an encryption key.
static String	SSE_S3_WITH_KEY_ERROR Encryption SSE-S3 is used but the caller also set an encryption key.

Method Summary

Modifier and Type	Method and Description
static long	applyLocatedFiles(org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus> iterator, S3AUtils.CallOnLocatedFileStatus eval) Apply an operation to every LocatedFileStatus in a remote iterator.
static void	clearBucketOption(org.apache.hadoop.conf.Configuration conf, String bucket, String genericKey) Clear a bucket-specific property.
static void	closeAll(org.slf4j.Logger log, Closeable... closeables) Close the Closeable objects and ignore any Exception or null pointers.
static void	closeAutocloseables(org.slf4j.Logger log, AutoCloseable... closeables) Close the Closeable objects and ignore any Exception or null pointers.
static com.amazonaws.ClientConfiguration	createAwsConf(org.apache.hadoop.conf.Configuration conf, String bucket) Create a new AWS ClientConfiguration.
static com.amazonaws.auth.AWSCredentialsProvider	createAWSCredentialProvider(org.apache.hadoop.conf.Configuration conf, Class<?> credClass, URI uri) Create an AWS credential provider from its class by using reflection.
static AWSCredentialProviderList	createAWSCredentialProviderSet(URI binding, org.apache.hadoop.conf.Configuration conf) Create the AWS credentials from the providers, the URI and the key Constants.AWS_CREDENTIALS_PROVIDER in the configuration.
static S3AFileStatus	createFileStatus(org.apache.hadoop.fs.Path keyPath, com.amazonaws.services.s3.model.S3ObjectSummary summary, long blockSize, String owner) Create a files status instance from a listing.
static S3AFileStatus	createUploadFileStatus(org.apache.hadoop.fs.Path keyPath, boolean isDir, long size, long blockSize, String owner) Create a file status for object we just uploaded.
static long	dateToLong(Date date) Date to long conversion.
static void	deleteQuietly(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, boolean recursive) Delete a path quietly: failures are logged at DEBUG.
static void	deleteWithWarning(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, boolean recursive) Delete a path: failures are logged at WARN.
static int	ensureOutputParameterInRange(String name, long size) Ensure that the long value is in the range of an integer.
static IOException	extractException(String operation, String path, ExecutionException ee) Extract an exception from a failed future, and convert to an IOE.
static <T> List<T>	flatmapLocatedFiles(org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus> iterator, S3AUtils.LocatedFileStatusMap<Optional<T>> eval) Map an operation to every LocatedFileStatus in a remote iterator, returning a list of the all results which were not empty.
static S3xLoginHelper.Login	getAWSAccessKeys(URI name, org.apache.hadoop.conf.Configuration conf) Return the access key and secret for S3 API use.
static String	getBucketOption(org.apache.hadoop.conf.Configuration conf, String bucket, String genericKey) Get a bucket-specific property.
static long	getMultipartSizeProperty(org.apache.hadoop.conf.Configuration conf, String property, long defVal) Get a size property from the configuration: this property must be at least equal to Constants.MULTIPART_MIN_SIZE.
static void	initConnectionSettings(org.apache.hadoop.conf.Configuration conf, com.amazonaws.ClientConfiguration awsConf) Initializes all AWS SDK settings related to connection management.
static void	initProxySupport(org.apache.hadoop.conf.Configuration conf, String bucket, com.amazonaws.ClientConfiguration awsConf) Initializes AWS SDK proxy support in the AWS client configuration if the S3A settings enable it.
static boolean	isThrottleException(Exception ex) Is the exception an instance of a throttling exception.
static List<org.apache.hadoop.fs.LocatedFileStatus>	listAndFilter(org.apache.hadoop.fs.FileSystem fileSystem, org.apache.hadoop.fs.Path path, boolean recursive, org.apache.hadoop.fs.PathFilter filter) List located files and filter them as a classic listFiles(path, filter) would do.
static Class<?>[]	loadAWSProviderClasses(org.apache.hadoop.conf.Configuration conf, String key, Class<?>... defaultValue) Load list of AWS credential provider/credential provider factory classes.
static long	longBytesOption(org.apache.hadoop.conf.Configuration conf, String key, long defVal, long min) Get a long option >= the minimum allowed value, supporting memory prefixes K,M,G,T,P.
static String	lookupPassword(String bucket, org.apache.hadoop.conf.Configuration conf, String baseKey) Get a password from a configuration, including JCEKS files, handling both the absolute key and bucket override.
static String	lookupPassword(String bucket, org.apache.hadoop.conf.Configuration conf, String baseKey, String overrideVal) Deprecated.
static String	lookupPassword(String bucket, org.apache.hadoop.conf.Configuration conf, String baseKey, String overrideVal, String defVal) Get a password from a configuration, including JCEKS files, handling both the absolute key and bucket override.
static <T> List<T>	mapLocatedFiles(org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus> iterator, S3AUtils.LocatedFileStatusMap<T> eval) Map an operation to every LocatedFileStatus in a remote iterator, returning a list of the results.
static <T> Optional<T>	maybe(boolean include, T value) Convert a value into a non-empty Optional instance if the value of include is true.
static boolean	objectRepresentsDirectory(String name, long size) Predicate: does the object represent a directory?.
static org.apache.hadoop.conf.Configuration	propagateBucketOptions(org.apache.hadoop.conf.Configuration source, String bucket) Propagates bucket-specific settings into generic S3A configuration keys.
static void	setBucketOption(org.apache.hadoop.conf.Configuration conf, String bucket, String genericKey, String value) Set a bucket-specific property to a particular value.
static boolean	signifiesConnectionBroken(com.amazonaws.SdkBaseException ex) Cue that an AWS exception is likely to be an EOF Exception based on the message coming back from an XML/JSON parser.
static String	stringify(com.amazonaws.services.s3.model.AmazonS3Exception e) Get low level details of an amazon exception for logging; multi-line.
static String	stringify(com.amazonaws.AmazonServiceException e) Get low level details of an amazon exception for logging; multi-line.
static String	stringify(com.amazonaws.services.s3.model.S3ObjectSummary summary) String information about a summary entry for debug messages.
static IOException	translateDynamoDBException(String path, String message, com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException ddbException) Translate a DynamoDB exception into an IOException.
static IOException	translateException(String operation, org.apache.hadoop.fs.Path path, com.amazonaws.AmazonClientException exception) Translate an exception raised in an operation into an IOException.
static IOException	translateException(String operation, String path, com.amazonaws.SdkBaseException exception) Translate an exception raised in an operation into an IOException.
static IOException	translateMultiObjectDeleteException(String message, com.amazonaws.services.s3.model.MultiObjectDeleteException ex) A MultiObjectDeleteException is raised if one or more delete objects listed in a bulk DELETE operation failed.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

SSE_C_NO_KEY_ERROR

public static final String SSE_C_NO_KEY_ERROR

Encryption SSE-C used but the config lacks an encryption key.

SSE_S3_WITH_KEY_ERROR

public static final String SSE_S3_WITH_KEY_ERROR

Encryption SSE-S3 is used but the caller also set an encryption key.

HIDDEN_FILE_FILTER

public static final org.apache.hadoop.fs.PathFilter HIDDEN_FILE_FILTER

Path filter which ignores any file which starts with . or _.

ACCEPT_ALL

public static final org.apache.hadoop.fs.PathFilter ACCEPT_ALL

A Path filter which accepts all filenames.

Method Detail

translateException

public static IOException translateException(String operation,
                                             org.apache.hadoop.fs.Path path,
                                             com.amazonaws.AmazonClientException exception)

Translate an exception raised in an operation into an IOException. The specific type of IOException depends on the class of AmazonClientException passed in, and any status codes included in the operation. That is: HTTP error codes are examined and can be used to build a more specific response.

Parameters:

operation - operation

path - path operated on (must not be null)

exception - amazon exception raised

Returns:

an IOE which wraps the caught exception.

See Also:

S3 Error responses, Amazon S3 Error Best Practices, Dynamo DB Commmon errors

translateException

public static IOException translateException(@Nullable
                                             String operation,
                                             String path,
                                             com.amazonaws.SdkBaseException exception)

Translate an exception raised in an operation into an IOException. The specific type of IOException depends on the class of AmazonClientException passed in, and any status codes included in the operation. That is: HTTP error codes are examined and can be used to build a more specific response.

Parameters:

operation - operation

path - path operated on (may be null)

exception - amazon exception raised

Returns:

an IOE which wraps the caught exception.

extractException

public static IOException extractException(String operation,
                                           String path,
                                           ExecutionException ee)

Extract an exception from a failed future, and convert to an IOE.

Parameters:

operation - operation which failed

path - path operated on (may be null)

ee - execution exception

Returns:

an IOE which can be thrown

isThrottleException

public static boolean isThrottleException(Exception ex)

Is the exception an instance of a throttling exception. That is an AmazonServiceException with a 503 response, any exception from DynamoDB for limits exceeded, an AWSServiceThrottledException, or anything which the AWS SDK's RetryUtils considers to be a throttling exception.

Parameters:

ex - exception to examine

Returns:

true if it is considered a throttling exception

signifiesConnectionBroken

public static boolean signifiesConnectionBroken(com.amazonaws.SdkBaseException ex)

Cue that an AWS exception is likely to be an EOF Exception based on the message coming back from an XML/JSON parser. This is likely to be brittle, so only a hint.

Parameters:

ex - exception

Returns:

true if this is believed to be a sign the connection was broken.

translateDynamoDBException

public static IOException translateDynamoDBException(String path,
                                                     String message,
                                                     com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException ddbException)

Translate a DynamoDB exception into an IOException.

Parameters:

path - path in the DDB

message - preformatted message for the exception

ddbException - exception

Returns:

an exception to throw.

translateMultiObjectDeleteException

public static IOException translateMultiObjectDeleteException(String message,
                                                              com.amazonaws.services.s3.model.MultiObjectDeleteException ex)

A MultiObjectDeleteException is raised if one or more delete objects listed in a bulk DELETE operation failed. The top-level exception is therefore just "something wasn't deleted", but doesn't include the what or the why. This translation will extract an AccessDeniedException if that's one of the causes, otherwise grabs the status code and uses it in the returned exception.

Parameters:

message - text for the exception

ex - exception to translate

Returns:

an IOE with more detail.

stringify

public static String stringify(com.amazonaws.AmazonServiceException e)

Get low level details of an amazon exception for logging; multi-line.

Parameters:

e - exception

Returns:

string details

stringify

public static String stringify(com.amazonaws.services.s3.model.AmazonS3Exception e)

Get low level details of an amazon exception for logging; multi-line.

Parameters:

e - exception

Returns:

string details

createFileStatus

public static S3AFileStatus createFileStatus(org.apache.hadoop.fs.Path keyPath,
                                             com.amazonaws.services.s3.model.S3ObjectSummary summary,
                                             long blockSize,
                                             String owner)

Create a files status instance from a listing.

Parameters:

keyPath - path to entry

summary - summary from AWS

blockSize - block size to declare.

owner - owner of the file

Returns:

a status entry

createUploadFileStatus

public static S3AFileStatus createUploadFileStatus(org.apache.hadoop.fs.Path keyPath,
                                                   boolean isDir,
                                                   long size,
                                                   long blockSize,
                                                   String owner)

Create a file status for object we just uploaded. For files, we use current time as modification time, since s3a uses S3's service-based modification time, which will not be available until we do a getFileStatus() later on.

Parameters:

keyPath - path for created object

isDir - true iff directory

size - file length

blockSize - block size for file status

owner - Hadoop username

Returns:

a status entry

objectRepresentsDirectory

public static boolean objectRepresentsDirectory(String name,
                                                long size)

Predicate: does the object represent a directory?.

Parameters:

name - object name

size - object size

Returns:

true if it meets the criteria for being an object

dateToLong

public static long dateToLong(Date date)

Date to long conversion. Handles null Dates that can be returned by AWS by returning 0

Parameters:

date - date from AWS query

Returns:

timestamp of the object

createAWSCredentialProviderSet

public static AWSCredentialProviderList createAWSCredentialProviderSet(URI binding,
                                                                       org.apache.hadoop.conf.Configuration conf)
                                                                throws IOException

Create the AWS credentials from the providers, the URI and the key Constants.AWS_CREDENTIALS_PROVIDER in the configuration.

Parameters:

binding - Binding URI, may contain user:pass login details; may be null

conf - filesystem configuration

Returns:

a credentials provider list

Throws:

IOException - Problems loading the providers (including reading secrets from credential files).

loadAWSProviderClasses

public static Class<?>[] loadAWSProviderClasses(org.apache.hadoop.conf.Configuration conf,
                                                String key,
                                                Class<?>... defaultValue)
                                         throws IOException

Load list of AWS credential provider/credential provider factory classes.

Parameters:

conf - configuration

key - key

defaultValue - list of default values

Returns:

the list of classes, possibly empty

Throws:

IOException - on a failure to load the list.

createAWSCredentialProvider

public static com.amazonaws.auth.AWSCredentialsProvider createAWSCredentialProvider(org.apache.hadoop.conf.Configuration conf,
                                                                                    Class<?> credClass,
                                                                                    URI uri)
                                                                             throws IOException

Create an AWS credential provider from its class by using reflection. The class must implement one of the following means of construction, which are attempted in order:

1. a public constructor accepting java.net.URI and org.apache.hadoop.conf.Configuration
2. a public static method named getInstance that accepts no arguments and returns an instance of com.amazonaws.auth.AWSCredentialsProvider, or
3. a public default constructor.

Parameters:

conf - configuration

credClass - credential class

uri - URI of the FS

Returns:

the instantiated class

Throws:

IOException - on any instantiation failure.

getAWSAccessKeys

public static S3xLoginHelper.Login getAWSAccessKeys(URI name,
                                                    org.apache.hadoop.conf.Configuration conf)
                                             throws IOException

Return the access key and secret for S3 API use. or indicated in the UserInfo of the name URI param.

Parameters:

name - the URI for which we need the access keys; may be null

conf - the Configuration object to interrogate for keys.

Returns:

AWSAccessKeys

Throws:

IOException - problems retrieving passwords from KMS.

lookupPassword

@Deprecated
public static String lookupPassword(String bucket,
                                                org.apache.hadoop.conf.Configuration conf,
                                                String baseKey,
                                                String overrideVal)
                                         throws IOException

Deprecated.

Get a password from a configuration, including JCEKS files, handling both the absolute key and bucket override.

Parameters:

bucket - bucket or "" if none known

conf - configuration

baseKey - base key to look up, e.g "fs.s3a.secret.key"

overrideVal - override value: if non empty this is used instead of querying the configuration.

Returns:

a password or "".

Throws:

IOException - on any IO problem

IllegalArgumentException - bad arguments

lookupPassword

public static String lookupPassword(String bucket,
                                    org.apache.hadoop.conf.Configuration conf,
                                    String baseKey)
                             throws IOException

Get a password from a configuration, including JCEKS files, handling both the absolute key and bucket override.

Parameters:

bucket - bucket or "" if none known

conf - configuration

baseKey - base key to look up, e.g "fs.s3a.secret.key"

Returns:

a password or "".

Throws:

IOException - on any IO problem

IllegalArgumentException - bad arguments

lookupPassword

public static String lookupPassword(String bucket,
                                    org.apache.hadoop.conf.Configuration conf,
                                    String baseKey,
                                    String overrideVal,
                                    String defVal)
                             throws IOException

Get a password from a configuration, including JCEKS files, handling both the absolute key and bucket override.

Parameters:

bucket - bucket or "" if none known

conf - configuration

baseKey - base key to look up, e.g "fs.s3a.secret.key"

overrideVal - override value: if non empty this is used instead of querying the configuration.

defVal - value to return if there is no password

Returns:

a password or the value of defVal.

Throws:

IOException - on any IO problem

IllegalArgumentException - bad arguments

stringify

public static String stringify(com.amazonaws.services.s3.model.S3ObjectSummary summary)

String information about a summary entry for debug messages.

Parameters:

summary - summary object

Returns:

string value

longBytesOption

public static long longBytesOption(org.apache.hadoop.conf.Configuration conf,
                                   String key,
                                   long defVal,
                                   long min)

Get a long option >= the minimum allowed value, supporting memory prefixes K,M,G,T,P.

Parameters:

conf - configuration

key - key to look up

defVal - default value

min - minimum value

Returns:

the value

Throws:

IllegalArgumentException - if the value is below the minimum

getMultipartSizeProperty

public static long getMultipartSizeProperty(org.apache.hadoop.conf.Configuration conf,
                                            String property,
                                            long defVal)

Get a size property from the configuration: this property must be at least equal to Constants.MULTIPART_MIN_SIZE. If it is too small, it is rounded up to that minimum, and a warning printed.

Parameters:

conf - configuration

property - property name

defVal - default value

Returns:

the value, guaranteed to be above the minimum size

ensureOutputParameterInRange

public static int ensureOutputParameterInRange(String name,
                                               long size)

Ensure that the long value is in the range of an integer.

Parameters:

name - property name for error messages

size - original size

Returns:

the size, guaranteed to be less than or equal to the max value of an integer.

propagateBucketOptions

public static org.apache.hadoop.conf.Configuration propagateBucketOptions(org.apache.hadoop.conf.Configuration source,
                                                                          String bucket)

Propagates bucket-specific settings into generic S3A configuration keys. This is done by propagating the values of the form fs.s3a.bucket.${bucket}.key to fs.s3a.key, for all values of "key" other than a small set of unmodifiable values. The source of the updated property is set to the key name of the bucket property, to aid in diagnostics of where things came from. Returns a new configuration. Why the clone? You can use the same conf for different filesystems, and the original values are not updated. The fs.s3a.impl property cannot be set, nor can any with the prefix fs.s3a.bucket. This method does not propagate security provider path information from the S3A property into the Hadoop common provider: callers must call patchSecurityCredentialProviders(Configuration) explicitly.

Parameters:

source - Source Configuration object.

bucket - bucket name. Must not be empty.

Returns:

a (potentially) patched clone of the original.

deleteQuietly

public static void deleteQuietly(org.apache.hadoop.fs.FileSystem fs,
                                 org.apache.hadoop.fs.Path path,
                                 boolean recursive)

Delete a path quietly: failures are logged at DEBUG.

Parameters:

fs - filesystem

path - path

recursive - recursive?

deleteWithWarning

public static void deleteWithWarning(org.apache.hadoop.fs.FileSystem fs,
                                     org.apache.hadoop.fs.Path path,
                                     boolean recursive)

Delete a path: failures are logged at WARN.

Parameters:

fs - filesystem

path - path

recursive - recursive?

createAwsConf

public static com.amazonaws.ClientConfiguration createAwsConf(org.apache.hadoop.conf.Configuration conf,
                                                              String bucket)
                                                       throws IOException

Create a new AWS ClientConfiguration. All clients to AWS services MUST use this for consistent setup of connectivity, UA, proxy settings.

Parameters:

conf - The Hadoop configuration

bucket - Optional bucket to use to look up per-bucket proxy secrets

Returns:

new AWS client configuration

Throws:

IOException

initConnectionSettings

public static void initConnectionSettings(org.apache.hadoop.conf.Configuration conf,
                                          com.amazonaws.ClientConfiguration awsConf)

Initializes all AWS SDK settings related to connection management.

Parameters:

conf - Hadoop configuration

awsConf - AWS SDK configuration

initProxySupport

public static void initProxySupport(org.apache.hadoop.conf.Configuration conf,
                                    String bucket,
                                    com.amazonaws.ClientConfiguration awsConf)
                             throws IllegalArgumentException,
                                    IOException

Initializes AWS SDK proxy support in the AWS client configuration if the S3A settings enable it.

Parameters:

conf - Hadoop configuration

bucket - Optional bucket to use to look up per-bucket proxy secrets

awsConf - AWS SDK configuration to update

Throws:

IllegalArgumentException - if misconfigured

IOException - problem getting username/secret from password source.

applyLocatedFiles

public static long applyLocatedFiles(org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus> iterator,
                                     S3AUtils.CallOnLocatedFileStatus eval)
                              throws IOException

Apply an operation to every LocatedFileStatus in a remote iterator.

Parameters:

iterator - iterator from a list

eval - closure to evaluate

Returns:

the number of files processed

Throws:

IOException - anything in the closure, or iteration logic.

mapLocatedFiles

public static <T> List<T> mapLocatedFiles(org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus> iterator,
                                          S3AUtils.LocatedFileStatusMap<T> eval)
                                   throws IOException

Map an operation to every LocatedFileStatus in a remote iterator, returning a list of the results.

Type Parameters:

T - return type of map

Parameters:

iterator - iterator from a list

eval - closure to evaluate

Returns:

the list of mapped results.

Throws:

IOException - anything in the closure, or iteration logic.

flatmapLocatedFiles

public static <T> List<T> flatmapLocatedFiles(org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus> iterator,
                                              S3AUtils.LocatedFileStatusMap<Optional<T>> eval)
                                       throws IOException

Map an operation to every LocatedFileStatus in a remote iterator, returning a list of the all results which were not empty.

Type Parameters:

T - return type of map

Parameters:

iterator - iterator from a list

eval - closure to evaluate

Returns:

the flattened list of mapped results.

Throws:

IOException - anything in the closure, or iteration logic.

listAndFilter

public static List<org.apache.hadoop.fs.LocatedFileStatus> listAndFilter(org.apache.hadoop.fs.FileSystem fileSystem,
                                                                         org.apache.hadoop.fs.Path path,
                                                                         boolean recursive,
                                                                         org.apache.hadoop.fs.PathFilter filter)
                                                                  throws IOException

List located files and filter them as a classic listFiles(path, filter) would do.

Parameters:

fileSystem - filesystem

path - path to list

recursive - recursive listing?

filter - filter for the filename

Returns:

the filtered list of entries

Throws:

IOException - IO failure.

maybe

public static <T> Optional<T> maybe(boolean include,
                                    T value)

Convert a value into a non-empty Optional instance if the value of include is true.

Type Parameters:

T - type of option.

Parameters:

include - flag to indicate the value is to be included.

value - value to return

Returns:

if include is false, Optional.empty. Otherwise, the value.

closeAll

public static void closeAll(org.slf4j.Logger log,
                            Closeable... closeables)

Close the Closeable objects and ignore any Exception or null pointers. (This is the SLF4J equivalent of that in IOUtils).

Parameters:

log - the log to log at debug level. Can be null.

closeables - the objects to close

closeAutocloseables

public static void closeAutocloseables(org.slf4j.Logger log,
                                       AutoCloseable... closeables)

Close the Closeable objects and ignore any Exception or null pointers. (This is the SLF4J equivalent of that in IOUtils).

Parameters:

log - the log to log at debug level. Can be null.

closeables - the objects to close

setBucketOption

public static void setBucketOption(org.apache.hadoop.conf.Configuration conf,
                                   String bucket,
                                   String genericKey,
                                   String value)

Set a bucket-specific property to a particular value. If the generic key passed in has an fs.s3a. prefix, that's stripped off, so that when the the bucket properties are propagated down to the generic values, that value gets copied down.

Parameters:

conf - configuration to set

bucket - bucket name

genericKey - key; can start with "fs.s3a."

value - value to set

clearBucketOption

public static void clearBucketOption(org.apache.hadoop.conf.Configuration conf,
                                     String bucket,
                                     String genericKey)

Clear a bucket-specific property. If the generic key passed in has an fs.s3a. prefix, that's stripped off, so that when the the bucket properties are propagated down to the generic values, that value gets copied down.

Parameters:

conf - configuration to set

bucket - bucket name

genericKey - key; can start with "fs.s3a."

getBucketOption

public static String getBucketOption(org.apache.hadoop.conf.Configuration conf,
                                     String bucket,
                                     String genericKey)

Get a bucket-specific property. If the generic key passed in has an fs.s3a. prefix, that's stripped off.

Parameters:

conf - configuration to set

bucket - bucket name

genericKey - key; can start with "fs.s3a."

Returns:

the bucket option, null if there is none