org.apache.hadoop.fs.s3a.impl

Class InternalConstants

java.lang.Object

org.apache.hadoop.fs.s3a.impl.InternalConstants

public final class InternalConstants
extends Object

Internal constants private only to the S3A codebase. Please don't refer to these outside of this module & its tests. If you find you need to then either the code is doing something it should not, or these constants need to be uprated to being public and stable entries.

Field Summary

Fields

Modifier and Type	Field and Description
static String	AP_INACCESSIBLE Error message to indicate Access Points are not accessible or don't exist.
static String	AP_REQUIRED_EXCEPTION Error message to indicate Access Points are required to be used for S3 access.
static String	ARN_BUCKET_OPTION AccessPoint ARN for the bucket.
static String	AUTH_SCHEME_AWS_SIGV_4 AWS V4 Auth Scheme to use when creating signers: "aws.auth#sigv4".
static String	AWS_REGION_SYSPROP The system property used by the AWS SDK to identify the region.
static Set<String>	CREATE_FILE_KEYS The known keys used in a createFile call.
static int	CSE_PADDING_LENGTH S3 client side encryption adds padding to the content length of constant length of 16 bytes (at the moment, since we have only 1 content encryption algorithm).
static int	DEFAULT_BLOCKSIZE Default blocksize as used in blocksize and FS status queries: 33554432.
static int	DEFAULT_UPLOAD_PART_COUNT_LIMIT Maximum entries you can upload in a single file write/copy/upload.
static boolean	DELETE_CONSIDERED_IDEMPOTENT This declared delete as idempotent.
static int	DRAIN_BUFFER_SIZE size of a buffer to create when draining the stream.
static int	MAX_ENTRIES_TO_DELETE The maximum number of entries that can be deleted in any bulk delete call to S3: 1000.
static int	RENAME_PARALLEL_LIMIT This is an arbitrary value: 10.
static List<String>	S3A_DYNAMIC_CAPABILITIES Dynamic Path capabilities to be evaluated in the BucketInfo tool.
static Set<String>	S3A_OPENFILE_KEYS The known keys used in a standard openFile call.
static int	SC_200_OK 200 status code: OK.
static int	SC_301_MOVED_PERMANENTLY 301 status code: Moved Permanently.
static int	SC_307_TEMPORARY_REDIRECT 307 status code: Temporary Redirect.
static int	SC_400_BAD_REQUEST 400 status code: Bad Request.
static int	SC_401_UNAUTHORIZED 401 status code: Unauthorized.
static int	SC_403 Deprecated.
static int	SC_403_FORBIDDEN 403 status code: Forbidden.
static int	SC_404 Deprecated.
static int	SC_404_NOT_FOUND 404 status code: Not Found.
static int	SC_405_METHOD_NOT_ALLOWED 405 status code: Method Not Allowed.
static int	SC_409_CONFLICT 409 status code: Conflict.
static int	SC_410_GONE 410 status code: Gone.
static int	SC_412_PRECONDITION_FAILED 412 status code: Precondition Failed.
static int	SC_415_UNSUPPORTED_MEDIA_TYPE 415 status code: Content type unsupported by this store.
static int	SC_416_RANGE_NOT_SATISFIABLE 416 status code: Range Not Satisfiable.
static int	SC_429_TOO_MANY_REQUESTS_GCS 429 status code: This is the google GCS throttle message.
static int	SC_443_NO_RESPONSE 443 status code: No Response (unofficial).
static int	SC_444_NO_RESPONSE 444 status code: No Response (unofficial).
static int	SC_500_INTERNAL_SERVER_ERROR 500 status code: Internal Server Error.
static int	SC_501_NOT_IMPLEMENTED 501 status code: method not implemented.
static int	SC_503_SERVICE_UNAVAILABLE 503 status code: Service Unavailable.
static int	SC_504_GATEWAY_TIMEOUT 504 Gateway Timeout.
static String	SDK_V2_UPGRADE_LOG_NAME Name of the log for events related to the SDK V2 upgrade.
static String	THROTTLE_LOG_NAME Name of the log for throttling events.
static String	UPLOAD_PART_COUNT_LIMIT A configuration option for test use only: maximum part count on block writes/uploads.
static String	X_DIRECTORY Directory marker attribute: see HADOOP-16613.

Method Summary

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DELETE_CONSIDERED_IDEMPOTENT

public static final boolean DELETE_CONSIDERED_IDEMPOTENT

This declared delete as idempotent. This is an "interesting" topic in past Hadoop FS work. Essentially: with a single caller, DELETE is idempotent but in a shared filesystem, it is is very much not so. Here, on the basis that isn't a filesystem with consistency guarantees, retryable results in files being deleted.

See Also:

Constant Field Values

DRAIN_BUFFER_SIZE

public static final int DRAIN_BUFFER_SIZE

size of a buffer to create when draining the stream.

See Also:

Constant Field Values

RENAME_PARALLEL_LIMIT

public static final int RENAME_PARALLEL_LIMIT

This is an arbitrary value: 10. It declares how many parallel copy operations in a single rename can be queued before the operation pauses and awaits completion. A very large value wouldn't just starve other threads from performing work, there's a risk that the S3 store itself would throttle operations (which all go to the same shard). It is not currently configurable just to avoid people choosing values which work on a microbenchmark (single rename, no other work, ...) but don't scale well to execution in a large process against a common store, all while separate processes are working with the same shard of storage. It should be a factor of MAX_ENTRIES_TO_DELETE so that all copies will have finished before deletion is contemplated. (There's always a block for that, it just makes more sense to perform the bulk delete after another block of copies have completed).

See Also:

Constant Field Values

MAX_ENTRIES_TO_DELETE

public static final int MAX_ENTRIES_TO_DELETE

The maximum number of entries that can be deleted in any bulk delete call to S3: 1000.

See Also:

Constant Field Values

DEFAULT_BLOCKSIZE

public static final int DEFAULT_BLOCKSIZE

Default blocksize as used in blocksize and FS status queries: 33554432.

See Also:

Constant Field Values

S3A_OPENFILE_KEYS

@InterfaceStability.Unstable
public static final Set<String> S3A_OPENFILE_KEYS

The known keys used in a standard openFile call.

SC_200_OK

public static final int SC_200_OK

200 status code: OK.

See Also:

Constant Field Values

SC_301_MOVED_PERMANENTLY

public static final int SC_301_MOVED_PERMANENTLY

301 status code: Moved Permanently.

See Also:

Constant Field Values

SC_307_TEMPORARY_REDIRECT

public static final int SC_307_TEMPORARY_REDIRECT

307 status code: Temporary Redirect.

See Also:

Constant Field Values

SC_400_BAD_REQUEST

public static final int SC_400_BAD_REQUEST

400 status code: Bad Request.

See Also:

Constant Field Values

SC_401_UNAUTHORIZED

public static final int SC_401_UNAUTHORIZED

401 status code: Unauthorized.

See Also:

Constant Field Values

SC_403_FORBIDDEN

public static final int SC_403_FORBIDDEN

403 status code: Forbidden.

See Also:

Constant Field Values

SC_403

@Deprecated
public static final int SC_403

Deprecated.

403 error code.

See Also:

Constant Field Values

SC_404_NOT_FOUND

public static final int SC_404_NOT_FOUND

404 status code: Not Found.

See Also:

Constant Field Values

SC_404

@Deprecated
public static final int SC_404

Deprecated.

404 error code.

See Also:

Constant Field Values

SC_405_METHOD_NOT_ALLOWED

public static final int SC_405_METHOD_NOT_ALLOWED

405 status code: Method Not Allowed.

See Also:

Constant Field Values

SC_409_CONFLICT

public static final int SC_409_CONFLICT

409 status code: Conflict. Example: creating a bucket twice.

See Also:

Constant Field Values

SC_410_GONE

public static final int SC_410_GONE

410 status code: Gone.

See Also:

Constant Field Values

SC_412_PRECONDITION_FAILED

public static final int SC_412_PRECONDITION_FAILED

412 status code: Precondition Failed.

See Also:

Constant Field Values

SC_415_UNSUPPORTED_MEDIA_TYPE

public static final int SC_415_UNSUPPORTED_MEDIA_TYPE

415 status code: Content type unsupported by this store.

See Also:

Constant Field Values

SC_416_RANGE_NOT_SATISFIABLE

public static final int SC_416_RANGE_NOT_SATISFIABLE

416 status code: Range Not Satisfiable.

See Also:

Constant Field Values

SC_429_TOO_MANY_REQUESTS_GCS

public static final int SC_429_TOO_MANY_REQUESTS_GCS

429 status code: This is the google GCS throttle message.

See Also:

Constant Field Values

SC_443_NO_RESPONSE

public static final int SC_443_NO_RESPONSE

443 status code: No Response (unofficial).

See Also:

Constant Field Values

SC_444_NO_RESPONSE

public static final int SC_444_NO_RESPONSE

444 status code: No Response (unofficial).

See Also:

Constant Field Values

SC_500_INTERNAL_SERVER_ERROR

public static final int SC_500_INTERNAL_SERVER_ERROR

500 status code: Internal Server Error.

See Also:

Constant Field Values

SC_501_NOT_IMPLEMENTED

public static final int SC_501_NOT_IMPLEMENTED

501 status code: method not implemented.

See Also:

Constant Field Values

SC_503_SERVICE_UNAVAILABLE

public static final int SC_503_SERVICE_UNAVAILABLE

503 status code: Service Unavailable. on AWS S3: throttle response.

See Also:

Constant Field Values

SC_504_GATEWAY_TIMEOUT

public static final int SC_504_GATEWAY_TIMEOUT

504 Gateway Timeout. AWS SDK considers retryable.

See Also:

Constant Field Values

THROTTLE_LOG_NAME

public static final String THROTTLE_LOG_NAME

Name of the log for throttling events. Value: "org.apache.hadoop.fs.s3a.throttled".

See Also:

Constant Field Values

SDK_V2_UPGRADE_LOG_NAME

public static final String SDK_V2_UPGRADE_LOG_NAME

Name of the log for events related to the SDK V2 upgrade.

See Also:

Constant Field Values

X_DIRECTORY

public static final String X_DIRECTORY

Directory marker attribute: see HADOOP-16613. Value: "application/x-directory".

See Also:

Constant Field Values

UPLOAD_PART_COUNT_LIMIT

@VisibleForTesting
public static final String UPLOAD_PART_COUNT_LIMIT

A configuration option for test use only: maximum part count on block writes/uploads. Value: "fs.s3a.internal.upload.part.count.limit".

See Also:

Constant Field Values

DEFAULT_UPLOAD_PART_COUNT_LIMIT

public static final int DEFAULT_UPLOAD_PART_COUNT_LIMIT

Maximum entries you can upload in a single file write/copy/upload. Value: 10000.

See Also:

Constant Field Values

AWS_REGION_SYSPROP

public static final String AWS_REGION_SYSPROP

The system property used by the AWS SDK to identify the region.

See Also:

Constant Field Values

CSE_PADDING_LENGTH

public static final int CSE_PADDING_LENGTH

S3 client side encryption adds padding to the content length of constant length of 16 bytes (at the moment, since we have only 1 content encryption algorithm). Use this to subtract while listing the content length when certain conditions are met.

See Also:

Constant Field Values

AP_REQUIRED_EXCEPTION

public static final String AP_REQUIRED_EXCEPTION

Error message to indicate Access Points are required to be used for S3 access.

See Also:

Constant Field Values

AP_INACCESSIBLE

public static final String AP_INACCESSIBLE

Error message to indicate Access Points are not accessible or don't exist.

See Also:

Constant Field Values

ARN_BUCKET_OPTION

public static final String ARN_BUCKET_OPTION

AccessPoint ARN for the bucket. When set as a bucket override the requests for that bucket will go through the AccessPoint.

See Also:

Constant Field Values

CREATE_FILE_KEYS

public static final Set<String> CREATE_FILE_KEYS

The known keys used in a createFile call.

S3A_DYNAMIC_CAPABILITIES

public static final List<String> S3A_DYNAMIC_CAPABILITIES

Dynamic Path capabilities to be evaluated in the BucketInfo tool.

AUTH_SCHEME_AWS_SIGV_4

public static final String AUTH_SCHEME_AWS_SIGV_4

AWS V4 Auth Scheme to use when creating signers: "aws.auth#sigv4".

See Also:

Constant Field Values