org.apache.hadoop.fs.s3a

Class S3AUtils

java.lang.Object

org.apache.hadoop.fs.s3a.S3AUtils

@InterfaceAudience.Private
 @InterfaceStability.Evolving
public final class S3AUtils
extends Object

Utility methods for S3A code. Some methods are marked LimitedPrivate since they are being used in an external project.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	S3AUtils.CallOnLocatedFileStatus An interface for use in lambda-expressions working with directory tree listings.
static interface	S3AUtils.LocatedFileStatusMap<T> An interface for use in lambda-expressions working with directory tree listings.

Field Summary

Fields

Modifier and Type	Field and Description
static org.apache.hadoop.fs.PathFilter	ACCEPT_ALL A Path filter which accepts all filenames.
static String	EOF_MESSAGE_IN_XML_PARSER
static String	EOF_READ_DIFFERENT_LENGTH
static org.apache.hadoop.fs.PathFilter	HIDDEN_FILE_FILTER Path filter which ignores any file which starts with .
static String	SSE_C_NO_KEY_ERROR Encryption SSE-C used but the config lacks an encryption key.
static String	SSE_S3_WITH_KEY_ERROR Encryption SSE-S3 is used but the caller also set an encryption key.

Method Summary

Modifier and Type	Method and Description
static long	applyLocatedFiles(org.apache.hadoop.fs.RemoteIterator<? extends org.apache.hadoop.fs.LocatedFileStatus> iterator, S3AUtils.CallOnLocatedFileStatus eval) Apply an operation to every LocatedFileStatus in a remote iterator.
static EncryptionSecrets	buildEncryptionSecrets(String bucket, org.apache.hadoop.conf.Configuration conf) Get the server-side encryption or client side encryption algorithm.
static boolean	checkDiskBuffer(org.apache.hadoop.conf.Configuration conf) Check whether the configuration for S3ABlockOutputStream is consistent or not.
static void	clearBucketOption(org.apache.hadoop.conf.Configuration conf, String bucket, String genericKey) Clear a bucket-specific property.
static void	closeAll(org.slf4j.Logger log, Closeable... closeables) Deprecated.
static void	closeAutocloseables(org.slf4j.Logger log, AutoCloseable... closeables) Close the Closeable objects and ignore any Exception or null pointers.
static S3AFileStatus	createFileStatus(org.apache.hadoop.fs.Path keyPath, software.amazon.awssdk.services.s3.model.S3Object s3Object, long blockSize, String owner, String eTag, String versionId, boolean isCSEEnabled) Create a files status instance from a listing.
static S3AFileStatus	createUploadFileStatus(org.apache.hadoop.fs.Path keyPath, boolean isDir, long size, long blockSize, String owner, String eTag, String versionId) Create a file status for object we just uploaded.
static long	dateToLong(Date date) Date to long conversion.
static void	deleteQuietly(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, boolean recursive) Delete a path quietly: failures are logged at DEBUG.
static void	deleteWithWarning(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, boolean recursive) Delete a path: failures are logged at WARN.
static int	ensureOutputParameterInRange(String name, long size) Ensure that the long value is in the range of an integer.
static IOException	extractException(String operation, String path, CompletionException ce) Extract an exception from a failed future, and convert to an IOE.
static IOException	extractException(String operation, String path, ExecutionException ee) Extract an exception from a failed future, and convert to an IOE.
static <T> List<T>	flatmapLocatedFiles(org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus> iterator, S3AUtils.LocatedFileStatusMap<Optional<T>> eval) Map an operation to every LocatedFileStatus in a remote iterator, returning a list of the all results which were not empty.
static String	formatRange(long rangeStart, long rangeEnd) Format a byte range for a request header.
static S3xLoginHelper.Login	getAWSAccessKeys(URI name, org.apache.hadoop.conf.Configuration conf) Return the access key and secret for S3 API use.
static String	getBucketOption(org.apache.hadoop.conf.Configuration conf, String bucket, String genericKey) Get a bucket-specific property.
static S3AEncryptionMethods	getEncryptionAlgorithm(String bucket, org.apache.hadoop.conf.Configuration conf) Get the server-side encryption or client side encryption algorithm.
static <InstanceT> InstanceT	getInstanceFromReflection(String className, org.apache.hadoop.conf.Configuration conf, URI uri, Class<? extends InstanceT> interfaceImplemented, String methodName, String configKey) Creates an instance of a class using reflection.
static long	getMultipartSizeProperty(org.apache.hadoop.conf.Configuration conf, String property, long defVal) Get a size property from the configuration: this property must be at least equal to Constants.MULTIPART_MIN_SIZE.
static String	getS3EncryptionKey(String bucket, org.apache.hadoop.conf.Configuration conf) Get any S3 encryption key, without propagating exceptions from JCEKs files.
static String	getS3EncryptionKey(String bucket, org.apache.hadoop.conf.Configuration conf, boolean propagateExceptions) Get any SSE/CSE key from a configuration/credential provider.
static Map<String,String>	getTrimmedStringCollectionSplitByEquals(org.apache.hadoop.conf.Configuration configuration, String name) Get the equal op (=) delimited key-value pairs of the name property as a collection of pair of Strings, trimmed of the leading and trailing whitespace after delimiting the name by comma and new line separator.
static int	intOption(org.apache.hadoop.conf.Configuration conf, String key, int defVal, int min) Get a integer option >= the minimum allowed value.
static boolean	isMessageTranslatableToEOF(software.amazon.awssdk.core.exception.SdkException ex) Cue that an AWS exception is likely to be an EOF Exception based on the message coming back from the client.
static boolean	isThrottleException(Exception ex) Is the exception an instance of a throttling exception.
static S3AFileStatus[]	iteratorToStatuses(org.apache.hadoop.fs.RemoteIterator<S3AFileStatus> iterator) Convert the data of an iterator of S3AFileStatus to an array.
static org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus>	listAndFilter(org.apache.hadoop.fs.FileSystem fileSystem, org.apache.hadoop.fs.Path path, boolean recursive, org.apache.hadoop.fs.PathFilter filter) List located files and filter them as a classic listFiles(path, filter) would do.
static long	longBytesOption(org.apache.hadoop.conf.Configuration conf, String key, long defVal, long min) Get a long option >= the minimum allowed value, supporting memory prefixes K,M,G,T,P.
static long	longOption(org.apache.hadoop.conf.Configuration conf, String key, long defVal, long min) Get a long option >= the minimum allowed value.
static String	lookupPassword(String bucket, org.apache.hadoop.conf.Configuration conf, String baseKey) Get a password from a configuration, including JCEKS files, handling both the absolute key and bucket override.
static String	lookupPassword(String bucket, org.apache.hadoop.conf.Configuration conf, String baseKey, String overrideVal) Deprecated.
static String	lookupPassword(String bucket, org.apache.hadoop.conf.Configuration conf, String baseKey, String overrideVal, String defVal) Get a password from a configuration, including JCEKS files, handling both the absolute key and bucket override.
static <T> List<T>	mapLocatedFiles(org.apache.hadoop.fs.RemoteIterator<? extends org.apache.hadoop.fs.LocatedFileStatus> iterator, S3AUtils.LocatedFileStatusMap<T> eval) Map an operation to every LocatedFileStatus in a remote iterator, returning a list of the results.
static <T> Optional<T>	maybe(boolean include, T value) Convert a value into a non-empty Optional instance if the value of include is true.
static String	maybeAddTrailingSlash(String key) Turns a path (relative or otherwise) into an S3 key, adding a trailing "/" if the path is not the root and does not already have a "/" at the end.
static boolean	objectRepresentsDirectory(String name) Predicate: does the object represent a directory?.
static org.apache.hadoop.conf.Configuration	propagateBucketOptions(org.apache.hadoop.conf.Configuration source, String bucket) Propagates bucket-specific settings into generic S3A configuration keys.
static void	setBucketOption(org.apache.hadoop.conf.Configuration conf, String bucket, String genericKey, String value) Set a bucket-specific property to a particular value.
static boolean	setIfDefined(org.apache.hadoop.conf.Configuration config, String key, String val, String origin) Set a key if the value is non-empty.
static String	stringify(software.amazon.awssdk.awscore.exception.AwsServiceException e) Get low level details of an amazon exception for logging; multi-line.
static String	stringify(software.amazon.awssdk.services.s3.model.S3Object s3Object) String information about a summary entry for debug messages.
static IOException	translateException(String operation, org.apache.hadoop.fs.Path path, software.amazon.awssdk.core.exception.SdkException exception) Translate an exception raised in an operation into an IOException.
static IOException	translateException(String operation, String path, software.amazon.awssdk.core.exception.SdkException exception) Translate an exception raised in an operation into an IOException.
static void	validateOutputStreamConfiguration(org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration conf) Validates the output stream configuration.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

SSE_C_NO_KEY_ERROR

public static final String SSE_C_NO_KEY_ERROR

Encryption SSE-C used but the config lacks an encryption key.

SSE_S3_WITH_KEY_ERROR

public static final String SSE_S3_WITH_KEY_ERROR

Encryption SSE-S3 is used but the caller also set an encryption key.

EOF_MESSAGE_IN_XML_PARSER

public static final String EOF_MESSAGE_IN_XML_PARSER

See Also:

Constant Field Values

EOF_READ_DIFFERENT_LENGTH

public static final String EOF_READ_DIFFERENT_LENGTH

See Also:

Constant Field Values

HIDDEN_FILE_FILTER

public static final org.apache.hadoop.fs.PathFilter HIDDEN_FILE_FILTER

Path filter which ignores any file which starts with . or _.

ACCEPT_ALL

public static final org.apache.hadoop.fs.PathFilter ACCEPT_ALL

A Path filter which accepts all filenames.

Method Detail

translateException

public static IOException translateException(String operation,
                                             org.apache.hadoop.fs.Path path,
                                             software.amazon.awssdk.core.exception.SdkException exception)

Translate an exception raised in an operation into an IOException. The specific type of IOException depends on the class of SdkException passed in, and any status codes included in the operation. That is: HTTP error codes are examined and can be used to build a more specific response.

Parameters:

operation - operation

path - path operated on (must not be null)

exception - amazon exception raised

Returns:

an IOE which wraps the caught exception.

See Also:

S3 Error responses, Amazon S3 Error Best Practices

translateException

public static IOException translateException(@Nullable
                                             String operation,
                                             @Nullable
                                             String path,
                                             software.amazon.awssdk.core.exception.SdkException exception)

Translate an exception raised in an operation into an IOException. The specific type of IOException depends on the class of SdkException passed in, and any status codes included in the operation. That is: HTTP error codes are examined and can be used to build a more specific response.

Parameters:

operation - operation

path - path operated on (may be null)

exception - amazon exception raised

Returns:

an IOE which wraps the caught exception.

extractException

public static IOException extractException(String operation,
                                           String path,
                                           ExecutionException ee)

Extract an exception from a failed future, and convert to an IOE.

Parameters:

operation - operation which failed

path - path operated on (may be null)

ee - execution exception

Returns:

an IOE which can be thrown

extractException

public static IOException extractException(String operation,
                                           String path,
                                           CompletionException ce)

Extract an exception from a failed future, and convert to an IOE.

Parameters:

operation - operation which failed

path - path operated on (may be null)

ce - completion exception

Returns:

an IOE which can be thrown

isThrottleException

public static boolean isThrottleException(Exception ex)

Is the exception an instance of a throttling exception. That is an AmazonServiceException with a 503 response, an AWSServiceThrottledException, or anything which the AWS SDK's RetryUtils considers to be a throttling exception.

Parameters:

ex - exception to examine

Returns:

true if it is considered a throttling exception

isMessageTranslatableToEOF

public static boolean isMessageTranslatableToEOF(software.amazon.awssdk.core.exception.SdkException ex)

Cue that an AWS exception is likely to be an EOF Exception based on the message coming back from the client. This is likely to be brittle, so only a hint.

Parameters:

ex - exception

Returns:

true if this is believed to be a sign the connection was broken.

stringify

public static String stringify(software.amazon.awssdk.awscore.exception.AwsServiceException e)

Get low level details of an amazon exception for logging; multi-line.

Parameters:

e - exception

Returns:

string details

createFileStatus

public static S3AFileStatus createFileStatus(org.apache.hadoop.fs.Path keyPath,
                                             software.amazon.awssdk.services.s3.model.S3Object s3Object,
                                             long blockSize,
                                             String owner,
                                             String eTag,
                                             String versionId,
                                             boolean isCSEEnabled)

Create a files status instance from a listing.

Parameters:

keyPath - path to entry

s3Object - s3Object entry

blockSize - block size to declare.

owner - owner of the file

eTag - S3 object eTag or null if unavailable

versionId - S3 object versionId or null if unavailable

isCSEEnabled - is client side encryption enabled?

Returns:

a status entry

createUploadFileStatus

public static S3AFileStatus createUploadFileStatus(org.apache.hadoop.fs.Path keyPath,
                                                   boolean isDir,
                                                   long size,
                                                   long blockSize,
                                                   String owner,
                                                   String eTag,
                                                   String versionId)

Create a file status for object we just uploaded. For files, we use current time as modification time, since s3a uses S3's service-based modification time, which will not be available until we do a getFileStatus() later on.

Parameters:

keyPath - path for created object

isDir - true iff directory

size - file length

blockSize - block size for file status

owner - Hadoop username

eTag - S3 object eTag or null if unavailable

versionId - S3 object versionId or null if unavailable

Returns:

a status entry

objectRepresentsDirectory

public static boolean objectRepresentsDirectory(String name)

Predicate: does the object represent a directory?.

Parameters:

name - object name

Returns:

true if it meets the criteria for being an object

dateToLong

public static long dateToLong(Date date)

Date to long conversion. Handles null Dates that can be returned by AWS by returning 0

Parameters:

date - date from AWS query

Returns:

timestamp of the object

getInstanceFromReflection

public static <InstanceT> InstanceT getInstanceFromReflection(String className,
                                                              org.apache.hadoop.conf.Configuration conf,
                                                              @Nullable
                                                              URI uri,
                                                              Class<? extends InstanceT> interfaceImplemented,
                                                              String methodName,
                                                              String configKey)
                                                       throws IOException

Creates an instance of a class using reflection. The class must implement one of the following means of construction, which are attempted in order:

1. a public constructor accepting java.net.URI and org.apache.hadoop.conf.Configuration
2. a public constructor accepting org.apache.hadoop.conf.Configuration
3. a public static method named as per methodName, that accepts no arguments and returns an instance of specified type, or
4. a public default constructor.

Type Parameters:

InstanceT - Instance of class

Parameters:

className - name of class for which instance is to be created

conf - configuration

uri - URI of the FS

interfaceImplemented - interface that this class implements

methodName - name of factory method to be invoked

configKey - config key under which this class is specified

Returns:

instance of the specified class

Throws:

IOException - on any problem

setIfDefined

public static boolean setIfDefined(org.apache.hadoop.conf.Configuration config,
                                   String key,
                                   String val,
                                   String origin)

Set a key if the value is non-empty.

Parameters:

config - config to patch

key - key to set

val - value to probe and set

origin - origin

Returns:

true if the property was set

getAWSAccessKeys

public static S3xLoginHelper.Login getAWSAccessKeys(URI name,
                                                    org.apache.hadoop.conf.Configuration conf)
                                             throws IOException

Return the access key and secret for S3 API use. or indicated in the UserInfo of the name URI param.

Parameters:

name - the URI for which we need the access keys; may be null

conf - the Configuration object to interrogate for keys.

Returns:

AWSAccessKeys

Throws:

IOException - problems retrieving passwords from KMS.

lookupPassword

@Deprecated
public static String lookupPassword(String bucket,
                                                org.apache.hadoop.conf.Configuration conf,
                                                String baseKey,
                                                String overrideVal)
                                         throws IOException

Deprecated.

Get a password from a configuration, including JCEKS files, handling both the absolute key and bucket override.

Parameters:

bucket - bucket or "" if none known

conf - configuration

baseKey - base key to look up, e.g "fs.s3a.secret.key"

overrideVal - override value: if non empty this is used instead of querying the configuration.

Returns:

a password or "".

Throws:

IOException - on any IO problem

IllegalArgumentException - bad arguments

lookupPassword

public static String lookupPassword(String bucket,
                                    org.apache.hadoop.conf.Configuration conf,
                                    String baseKey)
                             throws IOException

Get a password from a configuration, including JCEKS files, handling both the absolute key and bucket override.

Parameters:

bucket - bucket or "" if none known

conf - configuration

baseKey - base key to look up, e.g "fs.s3a.secret.key"

Returns:

a password or "".

Throws:

IOException - on any IO problem

IllegalArgumentException - bad arguments

lookupPassword

@InterfaceAudience.LimitedPrivate(value="Ranger")
public static String lookupPassword(String bucket,
                                                                                      org.apache.hadoop.conf.Configuration conf,
                                                                                      String baseKey,
                                                                                      String overrideVal,
                                                                                      String defVal)
                                                                               throws IOException

Get a password from a configuration, including JCEKS files, handling both the absolute key and bucket override.
Note: LimitedPrivate for ranger repository to get secrets.

Parameters:

bucket - bucket or "" if none known

conf - configuration

baseKey - base key to look up, e.g "fs.s3a.secret.key"

overrideVal - override value: if non empty this is used instead of querying the configuration.

defVal - value to return if there is no password

Returns:

a password or the value of defVal.

Throws:

IOException - on any IO problem

IllegalArgumentException - bad arguments

stringify

public static String stringify(software.amazon.awssdk.services.s3.model.S3Object s3Object)

String information about a summary entry for debug messages.

Parameters:

s3Object - s3Object entry

Returns:

string value

intOption

public static int intOption(org.apache.hadoop.conf.Configuration conf,
                            String key,
                            int defVal,
                            int min)

Get a integer option >= the minimum allowed value.

Parameters:

conf - configuration

key - key to look up

defVal - default value

min - minimum value

Returns:

the value

Throws:

IllegalArgumentException - if the value is below the minimum

longOption

public static long longOption(org.apache.hadoop.conf.Configuration conf,
                              String key,
                              long defVal,
                              long min)

Get a long option >= the minimum allowed value.

Parameters:

conf - configuration

key - key to look up

defVal - default value

min - minimum value

Returns:

the value

Throws:

IllegalArgumentException - if the value is below the minimum

longBytesOption

public static long longBytesOption(org.apache.hadoop.conf.Configuration conf,
                                   String key,
                                   long defVal,
                                   long min)

Get a long option >= the minimum allowed value, supporting memory prefixes K,M,G,T,P.

Parameters:

conf - configuration

key - key to look up

defVal - default value

min - minimum value

Returns:

the value

Throws:

IllegalArgumentException - if the value is below the minimum

getMultipartSizeProperty

public static long getMultipartSizeProperty(org.apache.hadoop.conf.Configuration conf,
                                            String property,
                                            long defVal)

Get a size property from the configuration: this property must be at least equal to Constants.MULTIPART_MIN_SIZE. If it is too small, it is rounded up to that minimum, and a warning printed.

Parameters:

conf - configuration

property - property name

defVal - default value

Returns:

the value, guaranteed to be above the minimum size

validateOutputStreamConfiguration

public static void validateOutputStreamConfiguration(org.apache.hadoop.fs.Path path,
                                                     org.apache.hadoop.conf.Configuration conf)
                                              throws org.apache.hadoop.fs.PathIOException

Validates the output stream configuration.

Parameters:

path - path: for error messages

conf - : configuration object for the given context

Throws:

org.apache.hadoop.fs.PathIOException - Unsupported configuration.

checkDiskBuffer

public static boolean checkDiskBuffer(org.apache.hadoop.conf.Configuration conf)

Check whether the configuration for S3ABlockOutputStream is consistent or not. Multipart uploads allow all kinds of fast buffers to be supported. When the option is disabled only disk buffers are allowed to be used as the file size might be bigger than the buffer size that can be allocated.

Parameters:

conf - : configuration object for the given context

Returns:

true if the disk buffer and the multipart settings are supported

ensureOutputParameterInRange

public static int ensureOutputParameterInRange(String name,
                                               long size)

Ensure that the long value is in the range of an integer.

Parameters:

name - property name for error messages

size - original size

Returns:

the size, guaranteed to be less than or equal to the max value of an integer.

propagateBucketOptions

@InterfaceAudience.LimitedPrivate(value="Ranger")
public static org.apache.hadoop.conf.Configuration propagateBucketOptions(org.apache.hadoop.conf.Configuration source,
                                                                                                                            String bucket)

Propagates bucket-specific settings into generic S3A configuration keys. This is done by propagating the values of the form fs.s3a.bucket.${bucket}.key to fs.s3a.key, for all values of "key" other than a small set of unmodifiable values. The source of the updated property is set to the key name of the bucket property, to aid in diagnostics of where things came from. Returns a new configuration. Why the clone? You can use the same conf for different filesystems, and the original values are not updated. The fs.s3a.impl property cannot be set, nor can any with the prefix fs.s3a.bucket. This method does not propagate security provider path information from the S3A property into the Hadoop common provider: callers must call patchSecurityCredentialProviders(Configuration) explicitly.
Note: LimitedPrivate for ranger repository to set up per-bucket configurations.

Parameters:

source - Source Configuration object.

bucket - bucket name. Must not be empty.

Returns:

a (potentially) patched clone of the original.

deleteQuietly

public static void deleteQuietly(org.apache.hadoop.fs.FileSystem fs,
                                 org.apache.hadoop.fs.Path path,
                                 boolean recursive)

Delete a path quietly: failures are logged at DEBUG.

Parameters:

fs - filesystem

path - path

recursive - recursive?

deleteWithWarning

public static void deleteWithWarning(org.apache.hadoop.fs.FileSystem fs,
                                     org.apache.hadoop.fs.Path path,
                                     boolean recursive)

Delete a path: failures are logged at WARN.

Parameters:

fs - filesystem

path - path

recursive - recursive?

iteratorToStatuses

public static S3AFileStatus[] iteratorToStatuses(org.apache.hadoop.fs.RemoteIterator<S3AFileStatus> iterator)
                                          throws IOException

Convert the data of an iterator of S3AFileStatus to an array.

Parameters:

iterator - a non-null iterator

Returns:

a possibly-empty array of file status entries

Throws:

IOException - failure

applyLocatedFiles

public static long applyLocatedFiles(org.apache.hadoop.fs.RemoteIterator<? extends org.apache.hadoop.fs.LocatedFileStatus> iterator,
                                     S3AUtils.CallOnLocatedFileStatus eval)
                              throws IOException

Apply an operation to every LocatedFileStatus in a remote iterator.

Parameters:

iterator - iterator from a list

eval - closure to evaluate

Returns:

the number of files processed

Throws:

IOException - anything in the closure, or iteration logic.

mapLocatedFiles

public static <T> List<T> mapLocatedFiles(org.apache.hadoop.fs.RemoteIterator<? extends org.apache.hadoop.fs.LocatedFileStatus> iterator,
                                          S3AUtils.LocatedFileStatusMap<T> eval)
                                   throws IOException

Map an operation to every LocatedFileStatus in a remote iterator, returning a list of the results.

Type Parameters:

T - return type of map

Parameters:

iterator - iterator from a list

eval - closure to evaluate

Returns:

the list of mapped results.

Throws:

IOException - anything in the closure, or iteration logic.

flatmapLocatedFiles

public static <T> List<T> flatmapLocatedFiles(org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus> iterator,
                                              S3AUtils.LocatedFileStatusMap<Optional<T>> eval)
                                       throws IOException

Map an operation to every LocatedFileStatus in a remote iterator, returning a list of the all results which were not empty.

Type Parameters:

T - return type of map

Parameters:

iterator - iterator from a list

eval - closure to evaluate

Returns:

the flattened list of mapped results.

Throws:

IOException - anything in the closure, or iteration logic.

listAndFilter

public static org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus> listAndFilter(org.apache.hadoop.fs.FileSystem fileSystem,
                                                                                                        org.apache.hadoop.fs.Path path,
                                                                                                        boolean recursive,
                                                                                                        org.apache.hadoop.fs.PathFilter filter)
                                                                                                 throws IOException

List located files and filter them as a classic listFiles(path, filter) would do. This will be incremental, fetching pages async. While it is rare for job to have many thousands of files, jobs against versioned buckets may return earlier if there are many non-visible objects.

Parameters:

fileSystem - filesystem

path - path to list

recursive - recursive listing?

filter - filter for the filename

Returns:

interator over the entries.

Throws:

IOException - IO failure.

maybe

public static <T> Optional<T> maybe(boolean include,
                                    T value)

Convert a value into a non-empty Optional instance if the value of include is true.

Type Parameters:

T - type of option.

Parameters:

include - flag to indicate the value is to be included.

value - value to return

Returns:

if include is false, Optional.empty. Otherwise, the value.

getS3EncryptionKey

public static String getS3EncryptionKey(String bucket,
                                        org.apache.hadoop.conf.Configuration conf)

Get any S3 encryption key, without propagating exceptions from JCEKs files.

Parameters:

bucket - bucket to query for

conf - configuration to examine

Returns:

the encryption key or ""

Throws:

IllegalArgumentException - bad arguments.

getS3EncryptionKey

public static String getS3EncryptionKey(String bucket,
                                        org.apache.hadoop.conf.Configuration conf,
                                        boolean propagateExceptions)
                                 throws IOException

Get any SSE/CSE key from a configuration/credential provider. This operation handles the case where the option has been set in the provider or configuration to the option SERVER_SIDE_ENCRYPTION_KEY. IOExceptions raised during retrieval are swallowed.

Parameters:

bucket - bucket to query for

conf - configuration to examine

propagateExceptions - should IO exceptions be rethrown?

Returns:

the encryption key or ""

Throws:

IllegalArgumentException - bad arguments.

IOException - if propagateExceptions==true and reading a JCEKS file raised an IOE

getEncryptionAlgorithm

public static S3AEncryptionMethods getEncryptionAlgorithm(String bucket,
                                                          org.apache.hadoop.conf.Configuration conf)
                                                   throws IOException

Get the server-side encryption or client side encryption algorithm. This includes validation of the configuration, checking the state of the encryption key given the chosen algorithm.

Parameters:

bucket - bucket to query for

conf - configuration to scan

Returns:

the encryption mechanism (which will be NONE unless one is set.

Throws:

IOException - on JCKES lookup or invalid method/key configuration.

buildEncryptionSecrets

public static EncryptionSecrets buildEncryptionSecrets(String bucket,
                                                       org.apache.hadoop.conf.Configuration conf)
                                                throws IOException

Get the server-side encryption or client side encryption algorithm. This includes validation of the configuration, checking the state of the encryption key given the chosen algorithm.

Parameters:

bucket - bucket to query for

conf - configuration to scan

Returns:

the encryption mechanism (which will be NONE unless one is set and secrets.

Throws:

IOException - on JCKES lookup or invalid method/key configuration.

closeAll

@Deprecated
public static void closeAll(org.slf4j.Logger log,
                                        Closeable... closeables)

Deprecated.

Close the Closeable objects and ignore any Exception or null pointers. This is obsolete: use IOUtils.cleanupWithLogger(Logger, Closeable...)

Parameters:

log - the log to log at debug level. Can be null.

closeables - the objects to close

closeAutocloseables

public static void closeAutocloseables(org.slf4j.Logger log,
                                       AutoCloseable... closeables)

Close the Closeable objects and ignore any Exception or null pointers. (This is the SLF4J equivalent of that in IOUtils).

Parameters:

log - the log to log at debug level. Can be null.

closeables - the objects to close

setBucketOption

public static void setBucketOption(org.apache.hadoop.conf.Configuration conf,
                                   String bucket,
                                   String genericKey,
                                   String value)

Set a bucket-specific property to a particular value. If the generic key passed in has an fs.s3a. prefix, that's stripped off, so that when the the bucket properties are propagated down to the generic values, that value gets copied down.

Parameters:

conf - configuration to set

bucket - bucket name

genericKey - key; can start with "fs.s3a."

value - value to set

clearBucketOption

public static void clearBucketOption(org.apache.hadoop.conf.Configuration conf,
                                     String bucket,
                                     String genericKey)

Clear a bucket-specific property. If the generic key passed in has an fs.s3a. prefix, that's stripped off, so that when the the bucket properties are propagated down to the generic values, that value gets copied down.

Parameters:

conf - configuration to set

bucket - bucket name

genericKey - key; can start with "fs.s3a."

getBucketOption

public static String getBucketOption(org.apache.hadoop.conf.Configuration conf,
                                     String bucket,
                                     String genericKey)

Get a bucket-specific property. If the generic key passed in has an fs.s3a. prefix, that's stripped off.

Parameters:

conf - configuration to set

bucket - bucket name

genericKey - key; can start with "fs.s3a."

Returns:

the bucket option, null if there is none

maybeAddTrailingSlash

public static String maybeAddTrailingSlash(String key)

Turns a path (relative or otherwise) into an S3 key, adding a trailing "/" if the path is not the root and does not already have a "/" at the end.

Parameters:

key - s3 key or ""

Returns:

the with a trailing "/", or, if it is the root key, "",

formatRange

public static String formatRange(long rangeStart,
                                 long rangeEnd)

Format a byte range for a request header. See https://www.rfc-editor.org/rfc/rfc9110.html#section-14.1.2

Parameters:

rangeStart - the start byte offset

rangeEnd - the end byte offset (inclusive)

Returns:

a formatted byte range

getTrimmedStringCollectionSplitByEquals

public static Map<String,String> getTrimmedStringCollectionSplitByEquals(org.apache.hadoop.conf.Configuration configuration,
                                                                         String name)

Get the equal op (=) delimited key-value pairs of the name property as a collection of pair of Strings, trimmed of the leading and trailing whitespace after delimiting the name by comma and new line separator. If no such property is specified then empty Map is returned.

Parameters:

configuration - the configuration object.

name - property name.

Returns:

property value as a Map of Strings, or empty Map.