org.apache.hadoop.fs.s3a.impl

Class HeaderProcessing

java.lang.Object

org.apache.hadoop.fs.s3a.impl.AbstractStoreOperation

org.apache.hadoop.fs.s3a.impl.HeaderProcessing

public class HeaderProcessing
extends AbstractStoreOperation

Part of the S3A FS where object headers are processed. Implements all the various XAttr read operations. Those APIs all expect byte arrays back. Metadata cloning is also implemented here, so as to stay in sync with custom header logic. The standard header names are extracted from the AWS SDK. The S3A connector does not (currently) support setting them, though it would be possible to do so through the createFile() builder API.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	HeaderProcessing.HeaderProcessingCallbacks

Field Summary

Fields

Modifier and Type	Field and Description
static String	CONTENT_TYPE_APPLICATION_XML XML content type : "application/xml".
static String	CONTENT_TYPE_OCTET_STREAM Content type of generic binary objects.
static String	CONTENT_TYPE_X_DIRECTORY Directory content type : "application/x-directory".
static String	HEADER_REFERRER HTTP Referrer for logs: "Referer".
static String	XA_ARCHIVE_STATUS object archive status; empty if not on S3 Glacier (i.e all normal files should be non-archived as S3A and applications don't handle archived data) Value "header.x-amz-archive-status".
static String	XA_CACHE_CONTROL Standard HTTP header found on some S3 objects: "header.Cache-Control".
static String	XA_CONTENT_DISPOSITION Standard HTTP header found on some S3 objects: "header.Content-Disposition".
static String	XA_CONTENT_ENCODING Content encoding; can be configured: "header.Content-Encoding".
static String	XA_CONTENT_LANGUAGE Standard HTTP header found on some S3 objects: "header.Content-Language".
static String	XA_CONTENT_LENGTH Length XAttr: "header.Content-Length".
static String	XA_CONTENT_MD5 Standard HTTP header found on some S3 objects: "header.Content-MD5".
static String	XA_CONTENT_RANGE Content range: "header.Content-Range".
static String	XA_CONTENT_TYPE Content type: may be set when uploading.
static String	XA_ENCRYPTION_KEY_ID
static String	XA_ETAG Etag Header "header.ETag".
static String	XA_LAST_MODIFIED last modified XAttr: "header.Last-Modified".
static String	XA_OBJECT_LOCK_LEGAL_HOLD_STATUS Object legal hold status.
static String	XA_OBJECT_LOCK_MODE Object lock mode.
static String	XA_OBJECT_LOCK_RETAIN_UNTIL_DATE ISO8601 expiry date of object lock hold.
static String	XA_OBJECT_REPLICATION_STATUS Replication status for cross-region replicated objects.
static String	XA_S3_VERSION_ID Version ID; empty for non-versioned buckets/data.
static String	XA_SERVER_SIDE_ENCRYPTION The server-side encryption algorithm to use with AWS-managed keys: "header.x-amz-server-side-encryption".
static String[]	XA_STANDARD_HEADERS Standard headers which are retrieved from HEAD Requests and set as XAttrs if the response included the relevant header.
static String	XA_STORAGE_CLASS Storage Class XAttr: "header.x-amz-storage-class".

Constructor Summary

Constructors

Constructor and Description
HeaderProcessing(StoreContext storeContext, HeaderProcessing.HeaderProcessingCallbacks callbacks) Construct.

Method Summary

Modifier and Type	Method and Description
static void	cloneObjectMetadata(software.amazon.awssdk.services.s3.model.HeadObjectResponse source, Map<String,String> dest, software.amazon.awssdk.services.s3.model.CopyObjectRequest.Builder copyObjectRequestBuilder) Creates a copy of the passed metadata.
static String	decodeBytes(byte[] bytes) Get the string value from the bytes.
static byte[]	encodeBytes(Object s) Stringify an object and return its bytes in UTF-8 encoding.
static Optional<Long>	extractXAttrLongValue(byte[] data) Convert an XAttr byte array to a long.
byte[]	getXAttr(org.apache.hadoop.fs.Path path, String name) Get an XAttr name and value for a file or directory.
Map<String,byte[]>	getXAttrs(org.apache.hadoop.fs.Path path) See FileSystem.getXAttrs(path.
Map<String,byte[]>	getXAttrs(org.apache.hadoop.fs.Path path, List<String> names) See FileSystem.getXAttrs(path, names.
List<String>	listXAttrs(org.apache.hadoop.fs.Path path) See FileSystem.listXAttrs(path).

Methods inherited from class org.apache.hadoop.fs.s3a.impl.AbstractStoreOperation

activateAuditSpan, getAuditSpan, getStoreContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

XA_CACHE_CONTROL

public static final String XA_CACHE_CONTROL

Standard HTTP header found on some S3 objects: "header.Cache-Control".

See Also:

Constant Field Values

XA_CONTENT_DISPOSITION

public static final String XA_CONTENT_DISPOSITION

Standard HTTP header found on some S3 objects: "header.Content-Disposition".

See Also:

Constant Field Values

XA_CONTENT_ENCODING

public static final String XA_CONTENT_ENCODING

Content encoding; can be configured: "header.Content-Encoding".

See Also:

Constant Field Values

XA_CONTENT_LANGUAGE

public static final String XA_CONTENT_LANGUAGE

Standard HTTP header found on some S3 objects: "header.Content-Language".

See Also:

Constant Field Values

XA_CONTENT_LENGTH

public static final String XA_CONTENT_LENGTH

Length XAttr: "header.Content-Length".

See Also:

Constant Field Values

XA_CONTENT_MD5

public static final String XA_CONTENT_MD5

Standard HTTP header found on some S3 objects: "header.Content-MD5".

See Also:

Constant Field Values

XA_CONTENT_RANGE

public static final String XA_CONTENT_RANGE

Content range: "header.Content-Range". This is returned on GET requests with ranges.

See Also:

Constant Field Values

XA_CONTENT_TYPE

public static final String XA_CONTENT_TYPE

Content type: may be set when uploading. "header.Content-Type".

See Also:

Constant Field Values

XA_ETAG

public static final String XA_ETAG

Etag Header "header.ETag". Also accessible via ObjectMetadata.getEtag(), where it can be retrieved via getFileChecksum(path) if the S3A connector is enabled.

See Also:

Constant Field Values

XA_LAST_MODIFIED

public static final String XA_LAST_MODIFIED

last modified XAttr: "header.Last-Modified".

See Also:

Constant Field Values

XA_ARCHIVE_STATUS

public static final String XA_ARCHIVE_STATUS

object archive status; empty if not on S3 Glacier (i.e all normal files should be non-archived as S3A and applications don't handle archived data) Value "header.x-amz-archive-status".

See Also:

Constant Field Values

XA_OBJECT_LOCK_LEGAL_HOLD_STATUS

public static final String XA_OBJECT_LOCK_LEGAL_HOLD_STATUS

Object legal hold status. "header.x-amz-object-lock-legal-hold".

See Also:

Constant Field Values

XA_OBJECT_LOCK_MODE

public static final String XA_OBJECT_LOCK_MODE

Object lock mode. "header.x-amz-object-lock-mode".

See Also:

Constant Field Values

XA_OBJECT_LOCK_RETAIN_UNTIL_DATE

public static final String XA_OBJECT_LOCK_RETAIN_UNTIL_DATE

ISO8601 expiry date of object lock hold. "header.x-amz-object-lock-retain-until-date".

See Also:

Constant Field Values

XA_OBJECT_REPLICATION_STATUS

public static final String XA_OBJECT_REPLICATION_STATUS

Replication status for cross-region replicated objects. "header.x-amz-replication-status".

See Also:

Constant Field Values

XA_S3_VERSION_ID

public static final String XA_S3_VERSION_ID

Version ID; empty for non-versioned buckets/data. "header.x-amz-version-id".

See Also:

Constant Field Values

XA_SERVER_SIDE_ENCRYPTION

public static final String XA_SERVER_SIDE_ENCRYPTION

The server-side encryption algorithm to use with AWS-managed keys: "header.x-amz-server-side-encryption".

See Also:

Constant Field Values

XA_ENCRYPTION_KEY_ID

public static final String XA_ENCRYPTION_KEY_ID

See Also:

Constant Field Values

XA_STORAGE_CLASS

public static final String XA_STORAGE_CLASS

Storage Class XAttr: "header.x-amz-storage-class".

See Also:

Constant Field Values

HEADER_REFERRER

public static final String HEADER_REFERRER

HTTP Referrer for logs: "Referer". This can be found in S3 logs, but is not set as an attribute in objects. important: the header value is deliberately a mis-spelling, as that is defined in RFC-1945.

See Also:

Constant Field Values

XA_STANDARD_HEADERS

public static final String[] XA_STANDARD_HEADERS

Standard headers which are retrieved from HEAD Requests and set as XAttrs if the response included the relevant header.

CONTENT_TYPE_OCTET_STREAM

public static final String CONTENT_TYPE_OCTET_STREAM

Content type of generic binary objects. This is the default for uploaded objects.

See Also:

Constant Field Values

CONTENT_TYPE_APPLICATION_XML

public static final String CONTENT_TYPE_APPLICATION_XML

XML content type : "application/xml". This is application/xml, not text/xml, and is what a HEAD of / returns as the type of a root path.

See Also:

Constant Field Values

CONTENT_TYPE_X_DIRECTORY

public static final String CONTENT_TYPE_X_DIRECTORY

Directory content type : "application/x-directory". Matches use/expectations of AWS S3 console.

See Also:

Constant Field Values

Constructor Detail

HeaderProcessing

public HeaderProcessing(StoreContext storeContext,
                        HeaderProcessing.HeaderProcessingCallbacks callbacks)

Construct.

Parameters:

storeContext - store context.

callbacks - callbacks to the store

Method Detail

encodeBytes

public static byte[] encodeBytes(@Nullable
                                 Object s)

Stringify an object and return its bytes in UTF-8 encoding.

Parameters:

s - source

Returns:

encoded object or an empty buffer

decodeBytes

public static String decodeBytes(byte[] bytes)

Get the string value from the bytes. if null : return null, otherwise the UTF-8 decoded bytes.

Parameters:

bytes - source bytes

Returns:

decoded value

getXAttr

public byte[] getXAttr(org.apache.hadoop.fs.Path path,
                       String name)
                throws IOException

Get an XAttr name and value for a file or directory.

Parameters:

path - Path to get extended attribute

name - XAttr name.

Returns:

byte[] XAttr value or null

Throws:

IOException - IO failure

getXAttrs

public Map<String,byte[]> getXAttrs(org.apache.hadoop.fs.Path path)
                             throws IOException

See FileSystem.getXAttrs(path.

Parameters:

path - Path to get extended attributes

Returns:

Map describing the XAttrs of the file or directory

Throws:

IOException - IO failure

listXAttrs

public List<String> listXAttrs(org.apache.hadoop.fs.Path path)
                        throws IOException

See FileSystem.listXAttrs(path).

Parameters:

path - Path to get extended attributes

Returns:

List of supported XAttrs

Throws:

IOException - IO failure

getXAttrs

public Map<String,byte[]> getXAttrs(org.apache.hadoop.fs.Path path,
                                    List<String> names)
                             throws IOException

See FileSystem.getXAttrs(path, names.

Parameters:

path - Path to get extended attributes

names - XAttr names.

Returns:

Map describing the XAttrs of the file or directory

Throws:

IOException - IO failure

extractXAttrLongValue

public static Optional<Long> extractXAttrLongValue(byte[] data)

Convert an XAttr byte array to a long. testability.

Parameters:

data - data to parse

Returns:

either a length or none

cloneObjectMetadata

public static void cloneObjectMetadata(software.amazon.awssdk.services.s3.model.HeadObjectResponse source,
                                       Map<String,String> dest,
                                       software.amazon.awssdk.services.s3.model.CopyObjectRequest.Builder copyObjectRequestBuilder)

Creates a copy of the passed metadata. This operation does not copy the X_HEADER_MAGIC_MARKER header to avoid confusion. If a marker file is renamed, it loses information about any remapped file. If new fields are added to ObjectMetadata which are not present in the user metadata headers, they will not be picked up or cloned unless this operation is updated.

Parameters:

source - the source metadata to copy

dest - the metadata to update; this is the return value.

copyObjectRequestBuilder - CopyObjectRequest builder