org.apache.hadoop.fs.s3a

Class S3AFileSystem

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.hadoop.fs.FileSystem

org.apache.hadoop.fs.s3a.S3AFileSystem

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.conf.Configurable, org.apache.hadoop.fs.PathCapabilities, org.apache.hadoop.fs.StreamCapabilities, org.apache.hadoop.security.token.DelegationTokenIssuer

@InterfaceAudience.Private
 @InterfaceStability.Evolving
public class S3AFileSystem
extends org.apache.hadoop.fs.FileSystem
implements org.apache.hadoop.fs.StreamCapabilities

The core S3A Filesystem implementation. This subclass is marked as private as code should not be creating it directly; use FileSystem.get(Configuration) and variants to create one. If cast to S3AFileSystem, extra methods and features may be accessed. Consider those private and unstable. Because it prints some of the state of the instrumentation, the output of toString() must also be considered unstable.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.fs.FileSystem

org.apache.hadoop.fs.FileSystem.DirectoryEntries, org.apache.hadoop.fs.FileSystem.DirListingIterator<T extends org.apache.hadoop.fs.FileStatus>, org.apache.hadoop.fs.FileSystem.Statistics

Nested classes/interfaces inherited from interface org.apache.hadoop.fs.StreamCapabilities

org.apache.hadoop.fs.StreamCapabilities.StreamCapability

Field Summary

Fields

Modifier and Type	Field and Description
static int	DEFAULT_BLOCKSIZE Default blocksize as used in blocksize and FS status queries.
static boolean	DELETE_CONSIDERED_IDEMPOTENT This declared delete as idempotent.
static org.slf4j.Logger	LOG

Fields inherited from class org.apache.hadoop.fs.FileSystem

DEFAULT_FS, FS_DEFAULT_NAME_KEY, SHUTDOWN_HOOK_PRIORITY, statistics, TRASH_PREFIX, USER_HOME_PREFIX

Fields inherited from interface org.apache.hadoop.fs.StreamCapabilities

DROPBEHIND, HFLUSH, HSYNC, READAHEAD, UNBUFFER

Constructor Summary

Constructors

Constructor and Description
S3AFileSystem()

Method Summary

Modifier and Type	Method and Description
void	abortOutstandingMultipartUploads(long seconds) Abort all outstanding MPUs older than a given age.
org.apache.hadoop.fs.FSDataOutputStream	append(org.apache.hadoop.fs.Path f, int bufferSize, org.apache.hadoop.util.Progressable progress) Append to an existing file (optional operation).
protected URI	canonicalizeUri(URI rawUri)
void	checkPath(org.apache.hadoop.fs.Path path) Check that a Path belongs to this FileSystem.
void	close() Close the filesystem.
protected S3ListResult	continueListObjects(S3ListRequest request, S3ListResult prevResult) List the next set of objects.
void	copyFromLocalFile(boolean delSrc, boolean overwrite, org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst) The src file is on the local disk.
org.apache.hadoop.fs.FSDataOutputStream	create(org.apache.hadoop.fs.Path f, org.apache.hadoop.fs.permission.FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, org.apache.hadoop.util.Progressable progress) Create an FSDataOutputStream at the indicated Path with write-progress reporting.
org.apache.hadoop.fs.FSDataOutputStream	createNonRecursive(org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.permission.FsPermission permission, EnumSet<org.apache.hadoop.fs.CreateFlag> flags, int bufferSize, short replication, long blockSize, org.apache.hadoop.util.Progressable progress)
protected static S3AStorageStatistics	createStorageStatistics() Create the storage statistics or bind to an existing one.
protected void	decrementGauge(Statistic statistic, long count) Decrement a gauge by a specific value.
boolean	delete(org.apache.hadoop.fs.Path f, boolean recursive) Delete a Path.
protected void	deleteObject(String key) Delete an object.
protected void	entryPoint(Statistic operation) Entry point to an operation.
boolean	exists(org.apache.hadoop.fs.Path f) Override superclass so as to add statistic collection.
com.amazonaws.services.s3.AmazonS3	getAmazonS3ClientForTesting(String reason) Returns the S3 client used by this filesystem.
String	getBucket() Get the bucket of this filesystem.
String	getBucketLocation() Get the region of a bucket.
String	getBucketLocation(String bucketName) Get the region of a bucket.
String	getCanonicalServiceName() Override getCanonicalServiceName because we don't support token in S3A.
long	getDefaultBlockSize() Deprecated. use FileSystem.getDefaultBlockSize(Path) instead
int	getDefaultPort()
DirectoryPolicy	getDirectoryMarkerPolicy() Get the directory marker policy of this filesystem.
org.apache.hadoop.fs.store.EtagChecksum	getFileChecksum(org.apache.hadoop.fs.Path f, long length) When enabled, get the etag of a object at the path via HEAD request and return it as a checksum object.
org.apache.hadoop.fs.FileStatus	getFileStatus(org.apache.hadoop.fs.Path f) Return a file status object that represents the path.
S3AInputPolicy	getInputPolicy() Get the input policy for this FS instance.
S3AInstrumentation	getInstrumentation() Get S3A Instrumentation.
MetadataStore	getMetadataStore() Get the metadata store.
long	getMultiPartThreshold() Get the threshold for multipart files.
com.amazonaws.services.s3.model.ObjectMetadata	getObjectMetadata(org.apache.hadoop.fs.Path path) Low-level call to get at the object metadata.
protected com.amazonaws.services.s3.model.ObjectMetadata	getObjectMetadata(String key) Request object metadata; increments counters in the process.
long	getPartitionSize() Get the partition size for multipart operations.
String	getScheme() Return the protocol scheme for the FileSystem.
S3AEncryptionMethods	getServerSideEncryptionAlgorithm() Get the encryption algorithm of this endpoint.
S3AStorageStatistics	getStorageStatistics() Get the storage statistics of this filesystem.
URI	getUri() Returns a URI whose scheme and authority identify this FileSystem.
String	getUsername() Get the username of the FS.
org.apache.hadoop.fs.Path	getWorkingDirectory() Get the current working directory for the given file system.
WriteOperationHelper	getWriteOperationHelper() Get a WriteOperationHelper instance.
org.apache.hadoop.fs.FileStatus[]	globStatus(org.apache.hadoop.fs.Path pathPattern) Increments the statistic Statistic.INVOCATION_GLOB_STATUS.
org.apache.hadoop.fs.FileStatus[]	globStatus(org.apache.hadoop.fs.Path pathPattern, org.apache.hadoop.fs.PathFilter filter) Override superclass so as to add statistic collection.
boolean	hasCapability(String capability) Deprecated.
boolean	hasMetadataStore() Does this Filesystem have a metadata store?
boolean	hasPathCapability(org.apache.hadoop.fs.Path path, String capability)
protected void	incrementGauge(Statistic statistic, long count) Increment a gauge by a specific value.
void	incrementPutCompletedStatistics(boolean success, long bytes) At the end of a put/multipart upload operation, update the relevant counters and gauges.
void	incrementPutProgressStatistics(String key, long bytes) Callback for use in progress callbacks from put/multipart upload events.
void	incrementPutStartStatistics(long bytes) At the start of a put/multipart upload operation, update the relevant counters.
void	incrementReadOperations() Increment read operations.
protected void	incrementStatistic(Statistic statistic) Increment a statistic by 1.
protected void	incrementStatistic(Statistic statistic, long count) Increment a statistic by a specific value.
void	incrementWriteOperations() Increment the write operation counter.
void	initialize(URI name, org.apache.hadoop.conf.Configuration originalConf) Called after a new FileSystem instance is constructed.
org.apache.hadoop.fs.FileStatus[]	innerListStatus(org.apache.hadoop.fs.Path f) List the statuses of the files/directories in the given path if the path is a directory.
boolean	isDirectory(org.apache.hadoop.fs.Path f) Override superclass so as to add statistic collection.
boolean	isFile(org.apache.hadoop.fs.Path f) Override superclass so as to add statistic collection.
boolean	isMagicCommitEnabled() Is magic commit enabled?
boolean	isMagicCommitPath(org.apache.hadoop.fs.Path path) Predicate: is a path a magic commit path? True if magic commit is enabled and the path qualifies as special.
org.apache.hadoop.fs.Path	keyToQualifiedPath(String key) Convert a key to a fully qualified path.
org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus>	listFiles(org.apache.hadoop.fs.Path f, boolean recursive) .
org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus>	listFilesAndEmptyDirectories(org.apache.hadoop.fs.Path f, boolean recursive)
org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus>	listLocatedStatus(org.apache.hadoop.fs.Path f) Override superclass so as to add statistic collection.
org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus>	listLocatedStatus(org.apache.hadoop.fs.Path f, org.apache.hadoop.fs.PathFilter filter) .
List<com.amazonaws.services.s3.model.MultipartUpload>	listMultipartUploads(String prefix) Listing all multipart uploads; limited to the first few hundred.
protected S3ListResult	listObjects(S3ListRequest request) Initiate a listObjects operation, incrementing metrics in the process.
org.apache.hadoop.fs.FileStatus[]	listStatus(org.apache.hadoop.fs.Path f) List the statuses of the files/directories in the given path if the path is a directory.
MultipartUtils.UploadIterator	listUploads(String prefix) List any pending multipart uploads whose keys begin with prefix, using an iterator that can handle an unlimited number of entries.
void	metastoreOperationRetried(Exception ex, int retries, boolean idempotent) Callback from Invoker when an operation against a metastore is retried.
boolean	mkdirs(org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.permission.FsPermission permission) Make the given path and all non-existent parents into directories.
S3AInstrumentation.CommitterStatistics	newCommitterStatistics() Create a new instance of the committer statistics.
com.amazonaws.services.s3.model.ObjectMetadata	newObjectMetadata() Create a new object metadata instance.
com.amazonaws.services.s3.model.ObjectMetadata	newObjectMetadata(long length) Create a new object metadata instance.
com.amazonaws.services.s3.model.PutObjectRequest	newPutObjectRequest(String key, com.amazonaws.services.s3.model.ObjectMetadata metadata, File srcfile) Create a putObject request.
org.apache.hadoop.fs.FSDataInputStream	open(org.apache.hadoop.fs.Path f, int bufferSize) Opens an FSDataInputStream at the indicated Path.
void	operationRetried(Exception ex) Callback when an operation was retried.
void	operationRetried(String text, Exception ex, int retries, boolean idempotent) Callback from Invoker when an operation is retried.
String	pathToKey(org.apache.hadoop.fs.Path path) Turns a path (relative or otherwise) into an S3 key.
UploadInfo	putObject(com.amazonaws.services.s3.model.PutObjectRequest putObjectRequest) Start a transfer-manager managed async PUT of an object, incrementing the put requests and put bytes counters.
org.apache.hadoop.fs.Path	qualify(org.apache.hadoop.fs.Path path) Qualify a path.
boolean	rename(org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst) Renames Path src to Path dst.
protected void	setAmazonS3Client(com.amazonaws.services.s3.AmazonS3 client) Set the client -used in mocking tests to force in a different client.
protected void	setBucket(String bucket) Set the bucket.
void	setInputPolicy(S3AInputPolicy inputPolicy) Change the input policy for this FS.
protected void	setOptionalCopyObjectRequestParameters(com.amazonaws.services.s3.model.CopyObjectRequest copyObjectRequest)
protected void	setOptionalMultipartUploadRequestParameters(com.amazonaws.services.s3.model.InitiateMultipartUploadRequest req)
protected void	setOptionalUploadPartRequestParameters(com.amazonaws.services.s3.model.UploadPartRequest request) Sets server side encryption parameters to the part upload request when encryption is enabled.
protected void	setUri(URI uri) Set the URI field through S3xLoginHelper.
void	setWorkingDirectory(org.apache.hadoop.fs.Path newDir) Set the current working directory for the given file system.
AWSCredentialProviderList	shareCredentials(String purpose) Get a shared copy of the AWS credentials, with its reference counter updated.
String	toString()
protected void	verifyBucketExists() Verify that the bucket exists.

Methods inherited from class org.apache.hadoop.fs.FileSystem

access, append, append, appendFile, areSymlinksEnabled, cancelDeleteOnExit, clearStatistics, closeAll, closeAllForUGI, completeLocalOutput, concat, copyFromLocalFile, copyFromLocalFile, copyFromLocalFile, copyToLocalFile, copyToLocalFile, copyToLocalFile, create, create, create, create, create, create, create, create, create, create, create, create, createFile, createNewFile, createNonRecursive, createNonRecursive, createPathHandle, createSnapshot, createSnapshot, createSymlink, delete, deleteOnExit, deleteSnapshot, enableSymlinks, fixRelativePart, get, get, get, getAclStatus, getAdditionalTokenIssuers, getAllStatistics, getAllStoragePolicies, getBlockSize, getCanonicalUri, getChildFileSystems, getContentSummary, getDefaultBlockSize, getDefaultReplication, getDefaultReplication, getDefaultUri, getDelegationToken, getFileBlockLocations, getFileBlockLocations, getFileChecksum, getFileLinkStatus, getFileSystemClass, getFSofPath, getGlobalStorageStatistics, getHomeDirectory, getInitialWorkingDirectory, getLength, getLinkTarget, getLocal, getName, getNamed, getPathHandle, getQuotaUsage, getReplication, getServerDefaults, getServerDefaults, getStatistics, getStatistics, getStatus, getStatus, getStoragePolicy, getTrashRoot, getTrashRoots, getUsed, getUsed, getXAttr, getXAttrs, getXAttrs, listCorruptFileBlocks, listStatus, listStatus, listStatus, listStatusBatch, listStatusIterator, listXAttrs, makeQualified, mkdirs, mkdirs, modifyAclEntries, moveFromLocalFile, moveFromLocalFile, moveToLocalFile, msync, newInstance, newInstance, newInstance, newInstanceLocal, open, open, open, primitiveCreate, primitiveMkdir, primitiveMkdir, printStatistics, processDeleteOnExit, removeAcl, removeAclEntries, removeDefaultAcl, removeXAttr, rename, renameSnapshot, resolveLink, resolvePath, satisfyStoragePolicy, setAcl, setDefaultUri, setDefaultUri, setOwner, setPermission, setQuota, setQuotaByStorageType, setReplication, setStoragePolicy, setTimes, setVerifyChecksum, setWriteChecksum, setXAttr, setXAttr, startLocalOutput, supportsSymlinks, truncate, unsetStoragePolicy

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.hadoop.security.token.DelegationTokenIssuer

addDelegationTokens, collectDelegationTokens

Field Detail

DEFAULT_BLOCKSIZE

public static final int DEFAULT_BLOCKSIZE

Default blocksize as used in blocksize and FS status queries.

See Also:

Constant Field Values

DELETE_CONSIDERED_IDEMPOTENT

public static final boolean DELETE_CONSIDERED_IDEMPOTENT

This declared delete as idempotent. This is an "interesting" topic in past Hadoop FS work. Essentially: with a single caller, DELETE is idempotent but in a shared filesystem, it is is very much not so. Here, on the basis that isn't a filesystem with consistency guarantees, retryable results in files being deleted.

See Also:

Constant Field Values

LOG

public static final org.slf4j.Logger LOG

Constructor Detail

S3AFileSystem

public S3AFileSystem()

Method Detail

initialize

public void initialize(URI name,
                       org.apache.hadoop.conf.Configuration originalConf)
                throws IOException

Called after a new FileSystem instance is constructed.

Overrides:

initialize in class org.apache.hadoop.fs.FileSystem

Parameters:

name - a uri whose authority section names the host, port, etc. for this FileSystem

originalConf - the configuration to use for the FS. The bucket-specific options are patched over the base ones before any use is made of the config.

Throws:

IOException

createStorageStatistics

protected static S3AStorageStatistics createStorageStatistics()

Create the storage statistics or bind to an existing one.

Returns:

a storage statistics instance.

verifyBucketExists

@Retries.RetryTranslated
protected void verifyBucketExists()
                                                    throws FileNotFoundException,
                                                           IOException

Verify that the bucket exists. This does not check permissions, not even read access. Retry policy: retrying, translated.

Throws:

FileNotFoundException - the bucket is absent

IOException - any other problem talking to S3

getInstrumentation

public S3AInstrumentation getInstrumentation()

Get S3A Instrumentation. For test purposes.

Returns:

this instance's instrumentation.

abortOutstandingMultipartUploads

@Retries.RetryTranslated
public void abortOutstandingMultipartUploads(long seconds)
                                                               throws IOException

Abort all outstanding MPUs older than a given age.

Parameters:

seconds - time in seconds

Throws:

IOException - on any failure, other than 403 "permission denied"

getScheme

public String getScheme()

Return the protocol scheme for the FileSystem.

Overrides:

getScheme in class org.apache.hadoop.fs.FileSystem

Returns:

"s3a"

getUri

public URI getUri()

Returns a URI whose scheme and authority identify this FileSystem.

Specified by:

getUri in class org.apache.hadoop.fs.FileSystem

setUri

protected void setUri(URI uri)

Set the URI field through S3xLoginHelper. Exported for testing.

Parameters:

uri - filesystem URI.

getDefaultPort

public int getDefaultPort()

Overrides:

getDefaultPort in class org.apache.hadoop.fs.FileSystem

getAmazonS3ClientForTesting

public com.amazonaws.services.s3.AmazonS3 getAmazonS3ClientForTesting(String reason)

Returns the S3 client used by this filesystem. Warning: this must only be used for testing, as it bypasses core S3A operations.

Parameters:

reason - a justification for requesting access.

Returns:

AmazonS3Client

setAmazonS3Client

protected void setAmazonS3Client(com.amazonaws.services.s3.AmazonS3 client)

Set the client -used in mocking tests to force in a different client.

Parameters:

client - client.

getBucketLocation

@Retries.RetryTranslated
public String getBucketLocation()
                                                  throws IOException

Get the region of a bucket.

Returns:

the region in which a bucket is located

Throws:

IOException - on any failure.

getBucketLocation

@Retries.RetryTranslated
public String getBucketLocation(String bucketName)
                                                  throws IOException

Get the region of a bucket. Retry policy: retrying, translated.

Parameters:

bucketName - the name of the bucket

Returns:

the region in which a bucket is located

Throws:

IOException - on any failure.

getInputPolicy

@InterfaceStability.Unstable
public S3AInputPolicy getInputPolicy()

Get the input policy for this FS instance.

Returns:

the input policy

getServerSideEncryptionAlgorithm

public S3AEncryptionMethods getServerSideEncryptionAlgorithm()

Get the encryption algorithm of this endpoint.

Returns:

the encryption algorithm.

getBucket

public String getBucket()

Get the bucket of this filesystem.

Returns:

the bucket

setBucket

protected void setBucket(String bucket)

Set the bucket.

Parameters:

bucket - the bucket

setInputPolicy

@InterfaceStability.Unstable
public void setInputPolicy(S3AInputPolicy inputPolicy)

Change the input policy for this FS.

Parameters:

inputPolicy - new policy

pathToKey

public String pathToKey(org.apache.hadoop.fs.Path path)

Turns a path (relative or otherwise) into an S3 key.

Parameters:

path - input path, may be relative to the working dir

Returns:

a key excluding the leading "/", or, if it is the root path, ""

keyToQualifiedPath

public org.apache.hadoop.fs.Path keyToQualifiedPath(String key)

Convert a key to a fully qualified path.

Parameters:

key - input key

Returns:

the fully qualified path including URI scheme and bucket name.

qualify

public org.apache.hadoop.fs.Path qualify(org.apache.hadoop.fs.Path path)

Qualify a path.

Parameters:

path - path to qualify

Returns:

a qualified path.

checkPath

public void checkPath(org.apache.hadoop.fs.Path path)

Check that a Path belongs to this FileSystem. Unlike the superclass, this version does not look at authority, only hostnames.

Overrides:

checkPath in class org.apache.hadoop.fs.FileSystem

Parameters:

path - to check

Throws:

IllegalArgumentException - if there is an FS mismatch

canonicalizeUri

protected URI canonicalizeUri(URI rawUri)

Overrides:

canonicalizeUri in class org.apache.hadoop.fs.FileSystem

open

public org.apache.hadoop.fs.FSDataInputStream open(org.apache.hadoop.fs.Path f,
                                                   int bufferSize)
                                            throws IOException

Opens an FSDataInputStream at the indicated Path.

Specified by:

open in class org.apache.hadoop.fs.FileSystem

Parameters:

f - the file name to open

bufferSize - the size of the buffer to be used.

Throws:

IOException

create

public org.apache.hadoop.fs.FSDataOutputStream create(org.apache.hadoop.fs.Path f,
                                                      org.apache.hadoop.fs.permission.FsPermission permission,
                                                      boolean overwrite,
                                                      int bufferSize,
                                                      short replication,
                                                      long blockSize,
                                                      org.apache.hadoop.util.Progressable progress)
                                               throws IOException

Create an FSDataOutputStream at the indicated Path with write-progress reporting. Retry policy: retrying, translated on the getFileStatus() probe. No data is uploaded to S3 in this call, so retry issues related to that.

Specified by:

create in class org.apache.hadoop.fs.FileSystem

Parameters:

f - the file name to open

permission - the permission to set.

overwrite - if a file with this name already exists, then if true, the file will be overwritten, and if false an error will be thrown.

bufferSize - the size of the buffer to be used.

replication - required block replication for the file.

blockSize - the requested block size.

progress - the progress reporter.

Throws:

IOException - in the event of IO related errors.

See Also:

FileSystem.setPermission(Path, FsPermission)

getWriteOperationHelper

@InterfaceAudience.Private
public WriteOperationHelper getWriteOperationHelper()

Get a WriteOperationHelper instance. This class permits other low-level operations against the store. It is unstable and only intended for code with intimate knowledge of the object store. If using this, be prepared for changes even on minor point releases.

Returns:

a new helper.

createNonRecursive

public org.apache.hadoop.fs.FSDataOutputStream createNonRecursive(org.apache.hadoop.fs.Path path,
                                                                  org.apache.hadoop.fs.permission.FsPermission permission,
                                                                  EnumSet<org.apache.hadoop.fs.CreateFlag> flags,
                                                                  int bufferSize,
                                                                  short replication,
                                                                  long blockSize,
                                                                  org.apache.hadoop.util.Progressable progress)
                                                           throws IOException

Overrides:

createNonRecursive in class org.apache.hadoop.fs.FileSystem

Throws:

FileNotFoundException - if the parent directory is not present -or is not a directory.

IOException

append

public org.apache.hadoop.fs.FSDataOutputStream append(org.apache.hadoop.fs.Path f,
                                                      int bufferSize,
                                                      org.apache.hadoop.util.Progressable progress)
                                               throws IOException

Append to an existing file (optional operation).

Specified by:

append in class org.apache.hadoop.fs.FileSystem

Parameters:

f - the existing file to be appended.

bufferSize - the size of the buffer to be used.

progress - for reporting progress if it is not null.

Throws:

IOException - indicating that append is not supported.

rename

public boolean rename(org.apache.hadoop.fs.Path src,
                      org.apache.hadoop.fs.Path dst)
               throws IOException

Renames Path src to Path dst. Can take place on local fs or remote DFS. Warning: S3 does not support renames. This method does a copy which can take S3 some time to execute with large files and directories. Since there is no Progressable passed in, this can time out jobs. Note: This implementation differs with other S3 drivers. Specifically:

       Fails if src is a file and dst is a directory.
       Fails if src is a directory and dst is a file.
       Fails if the parent of dst does not exist or is a file.
       Fails if dst is a directory that is not empty.
 

Specified by:

rename in class org.apache.hadoop.fs.FileSystem

Parameters:

src - path to be renamed

dst - new path after rename

Returns:

true if rename is successful

Throws:

IOException - on IO failure

getObjectMetadata

@Retries.RetryRaw
public com.amazonaws.services.s3.model.ObjectMetadata getObjectMetadata(org.apache.hadoop.fs.Path path)
                                                                                   throws IOException

Low-level call to get at the object metadata.

Parameters:

path - path to the object

Returns:

metadata

Throws:

IOException - IO and object access problems.

hasMetadataStore

public boolean hasMetadataStore()

Does this Filesystem have a metadata store?

Returns:

true iff the FS has been instantiated with a metadata store

getMetadataStore

public MetadataStore getMetadataStore()

Get the metadata store. This will always be non-null, but may be bound to the NullMetadataStore.

Returns:

the metadata store of this FS instance

entryPoint

protected void entryPoint(Statistic operation)
                   throws IOException

Entry point to an operation. Increments the statistic; verifies the FS is active.

Parameters:

operation - The operation to increment

Throws:

IOException - if the

incrementStatistic

protected void incrementStatistic(Statistic statistic)

Increment a statistic by 1. This increments both the instrumentation and storage statistics.

Parameters:

statistic - The operation to increment

incrementStatistic

protected void incrementStatistic(Statistic statistic,
                                  long count)

Increment a statistic by a specific value. This increments both the instrumentation and storage statistics.

Parameters:

statistic - The operation to increment

count - the count to increment

decrementGauge

protected void decrementGauge(Statistic statistic,
                              long count)

Decrement a gauge by a specific value.

Parameters:

statistic - The operation to decrement

count - the count to decrement

incrementGauge

protected void incrementGauge(Statistic statistic,
                              long count)

Increment a gauge by a specific value.

Parameters:

statistic - The operation to increment

count - the count to increment

operationRetried

public void operationRetried(Exception ex)

Callback when an operation was retried. Increments the statistics of ignored errors or throttled requests, depending up on the exception class.

Parameters:

ex - exception.

operationRetried

public void operationRetried(String text,
                             Exception ex,
                             int retries,
                             boolean idempotent)

Callback from Invoker when an operation is retried.

Parameters:

text - text of the operation

ex - exception

retries - number of retries

idempotent - is the method idempotent

metastoreOperationRetried

public void metastoreOperationRetried(Exception ex,
                                      int retries,
                                      boolean idempotent)

Callback from Invoker when an operation against a metastore is retried. Always increments the Statistic.S3GUARD_METADATASTORE_RETRY statistic/counter; if it is a throttling exception will update the associated throttled metrics/statistics.

Parameters:

ex - exception

retries - number of retries

idempotent - is the method idempotent

getStorageStatistics

public S3AStorageStatistics getStorageStatistics()

Get the storage statistics of this filesystem.

Overrides:

getStorageStatistics in class org.apache.hadoop.fs.FileSystem

Returns:

the storage statistics

getObjectMetadata

@Retries.RetryRaw
protected com.amazonaws.services.s3.model.ObjectMetadata getObjectMetadata(String key)
                                                                                      throws IOException

Request object metadata; increments counters in the process. Retry policy: retry untranslated.

Parameters:

key - key

Returns:

the metadata

Throws:

IOException - if the retry invocation raises one (it shouldn't).

listObjects

@Retries.RetryRaw
protected S3ListResult listObjects(S3ListRequest request)
                                              throws IOException

Initiate a listObjects operation, incrementing metrics in the process. Retry policy: retry untranslated.

Parameters:

request - request to initiate

Returns:

the results

Throws:

IOException - if the retry invocation raises one (it shouldn't).

continueListObjects

@Retries.RetryRaw
protected S3ListResult continueListObjects(S3ListRequest request,
                                                             S3ListResult prevResult)
                                                      throws IOException

List the next set of objects. Retry policy: retry untranslated.

Parameters:

request - last list objects request to continue

prevResult - last paged result to continue from

Returns:

the next result object

Throws:

IOException - none, just there for retryUntranslated.

incrementReadOperations

public void incrementReadOperations()

Increment read operations.

incrementWriteOperations

public void incrementWriteOperations()

Increment the write operation counter. This is somewhat inaccurate, as it appears to be invoked more often than needed in progress callbacks.

deleteObject

@Retries.RetryRaw
protected void deleteObject(String key)
                                       throws com.amazonaws.AmazonClientException,
                                              IOException

Delete an object. This is the low-level internal call which does not update the metastore. Increments the OBJECT_DELETE_REQUESTS and write operation statistics. This call does not create any mock parent entries. Retry policy: retry untranslated; delete considered idempotent.

Parameters:

key - key to blob to delete.

Throws:

com.amazonaws.AmazonClientException - problems working with S3

org.apache.hadoop.fs.InvalidRequestException - if the request was rejected due to a mistaken attempt to delete the root directory.

IOException

newPutObjectRequest

public com.amazonaws.services.s3.model.PutObjectRequest newPutObjectRequest(String key,
                                                                            com.amazonaws.services.s3.model.ObjectMetadata metadata,
                                                                            File srcfile)

Create a putObject request. Adds the ACL and metadata

Parameters:

key - key of object

metadata - metadata header

srcfile - source file

Returns:

the request

newObjectMetadata

public com.amazonaws.services.s3.model.ObjectMetadata newObjectMetadata()

Create a new object metadata instance. Any standard metadata headers are added here, for example: encryption.

Returns:

a new metadata instance

newObjectMetadata

public com.amazonaws.services.s3.model.ObjectMetadata newObjectMetadata(long length)

Create a new object metadata instance. Any standard metadata headers are added here, for example: encryption.

Parameters:

length - length of data to set in header.

Returns:

a new metadata instance

putObject

@Retries.OnceRaw
public UploadInfo putObject(com.amazonaws.services.s3.model.PutObjectRequest putObjectRequest)

Start a transfer-manager managed async PUT of an object, incrementing the put requests and put bytes counters. It does not update the other counters, as existing code does that as progress callbacks come in. Byte length is calculated from the file length, or, if there is no file, from the content length of the header. Because the operation is async, any stream supplied in the request must reference data (files, buffers) which stay valid until the upload completes. Retry policy: N/A: the transfer manager is performing the upload.

Parameters:

putObjectRequest - the request

Returns:

the upload initiated

incrementPutStartStatistics

public void incrementPutStartStatistics(long bytes)

At the start of a put/multipart upload operation, update the relevant counters.

Parameters:

bytes - bytes in the request.

incrementPutCompletedStatistics

public void incrementPutCompletedStatistics(boolean success,
                                            long bytes)

At the end of a put/multipart upload operation, update the relevant counters and gauges.

Parameters:

success - did the operation succeed?

bytes - bytes in the request.

incrementPutProgressStatistics

public void incrementPutProgressStatistics(String key,
                                           long bytes)

Callback for use in progress callbacks from put/multipart upload events. Increments those statistics which are expected to be updated during the ongoing upload operation.

Parameters:

key - key to file that is being written (for logging)

bytes - bytes successfully uploaded.

delete

@Retries.RetryTranslated
public boolean delete(org.apache.hadoop.fs.Path f,
                                               boolean recursive)
                                        throws IOException

Delete a Path. This operation is at least O(files), with added overheads to enumerate the path. It is also not atomic.

Specified by:

delete in class org.apache.hadoop.fs.FileSystem

Parameters:

f - the path to delete.

recursive - if path is a directory and set to true, the directory is deleted else throws an exception. In case of a file the recursive can be set to either true or false.

Returns:

true if the path existed and then was deleted; false if there was no path in the first place, or the corner cases of root path deletion have surfaced.

Throws:

IOException - due to inability to delete a directory or file.

listStatus

public org.apache.hadoop.fs.FileStatus[] listStatus(org.apache.hadoop.fs.Path f)
                                             throws FileNotFoundException,
                                                    IOException

List the statuses of the files/directories in the given path if the path is a directory.

Specified by:

listStatus in class org.apache.hadoop.fs.FileSystem

Parameters:

f - given path

Returns:

the statuses of the files/directories in the given patch

Throws:

FileNotFoundException - when the path does not exist; IOException see specific implementation

IOException

innerListStatus

public org.apache.hadoop.fs.FileStatus[] innerListStatus(org.apache.hadoop.fs.Path f)
                                                  throws FileNotFoundException,
                                                         IOException,
                                                         com.amazonaws.AmazonClientException

List the statuses of the files/directories in the given path if the path is a directory.

Parameters:

f - given path

Returns:

the statuses of the files/directories in the given patch

Throws:

FileNotFoundException - when the path does not exist;

IOException - due to an IO problem.

com.amazonaws.AmazonClientException - on failures inside the AWS SDK

setWorkingDirectory

public void setWorkingDirectory(org.apache.hadoop.fs.Path newDir)

Set the current working directory for the given file system. All relative paths will be resolved relative to it.

Specified by:

setWorkingDirectory in class org.apache.hadoop.fs.FileSystem

Parameters:

newDir - the current working directory.

getWorkingDirectory

public org.apache.hadoop.fs.Path getWorkingDirectory()

Get the current working directory for the given file system.

Specified by:

getWorkingDirectory in class org.apache.hadoop.fs.FileSystem

Returns:

the directory pathname

getUsername

public String getUsername()

Get the username of the FS.

Returns:

the short name of the user who instantiated the FS

mkdirs

public boolean mkdirs(org.apache.hadoop.fs.Path path,
                      org.apache.hadoop.fs.permission.FsPermission permission)
               throws IOException,
                      org.apache.hadoop.fs.FileAlreadyExistsException

Make the given path and all non-existent parents into directories. Has the semantics of Unix 'mkdir -p'. Existence of the directory hierarchy is not an error.

Specified by:

mkdirs in class org.apache.hadoop.fs.FileSystem

Parameters:

path - path to create

permission - to apply to f

Returns:

true if a directory was created or already existed

Throws:

org.apache.hadoop.fs.FileAlreadyExistsException - there is a file at the path specified

IOException - other IO problems

getFileStatus

@Retries.RetryTranslated
public org.apache.hadoop.fs.FileStatus getFileStatus(org.apache.hadoop.fs.Path f)
                                                                       throws IOException

Return a file status object that represents the path.

Specified by:

getFileStatus in class org.apache.hadoop.fs.FileSystem

Parameters:

f - The path we want information from

Returns:

a FileStatus object

Throws:

FileNotFoundException - when the path does not exist

IOException - on other problems.

copyFromLocalFile

public void copyFromLocalFile(boolean delSrc,
                              boolean overwrite,
                              org.apache.hadoop.fs.Path src,
                              org.apache.hadoop.fs.Path dst)
                       throws IOException

The src file is on the local disk. Add it to FS at the given dst name. This version doesn't need to create a temporary file to calculate the md5. Sadly this doesn't seem to be used by the shell cp :( delSrc indicates if the source should be removed

Overrides:

copyFromLocalFile in class org.apache.hadoop.fs.FileSystem

Parameters:

delSrc - whether to delete the src

overwrite - whether to overwrite an existing file

src - path

dst - path

Throws:

IOException - IO problem

org.apache.hadoop.fs.FileAlreadyExistsException - the destination file exists and overwrite==false

com.amazonaws.AmazonClientException - failure in the AWS SDK

close

public void close()
           throws IOException

Close the filesystem. This shuts down all transfers.

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Overrides:

close in class org.apache.hadoop.fs.FileSystem

Throws:

IOException - IO problem

getCanonicalServiceName

public String getCanonicalServiceName()

Override getCanonicalServiceName because we don't support token in S3A.

Specified by:

getCanonicalServiceName in interface org.apache.hadoop.security.token.DelegationTokenIssuer

Overrides:

getCanonicalServiceName in class org.apache.hadoop.fs.FileSystem

setOptionalMultipartUploadRequestParameters

protected void setOptionalMultipartUploadRequestParameters(com.amazonaws.services.s3.model.InitiateMultipartUploadRequest req)

setOptionalUploadPartRequestParameters

protected void setOptionalUploadPartRequestParameters(com.amazonaws.services.s3.model.UploadPartRequest request)

Sets server side encryption parameters to the part upload request when encryption is enabled.

Parameters:

request - upload part request

setOptionalCopyObjectRequestParameters

protected void setOptionalCopyObjectRequestParameters(com.amazonaws.services.s3.model.CopyObjectRequest copyObjectRequest)
                                               throws IOException

Throws:

IOException

getDefaultBlockSize

@Deprecated
public long getDefaultBlockSize()

Deprecated. use FileSystem.getDefaultBlockSize(Path) instead

Return the number of bytes that large input files should be optimally be split into to minimize I/O time.

Overrides:

getDefaultBlockSize in class org.apache.hadoop.fs.FileSystem

getDirectoryMarkerPolicy

public DirectoryPolicy getDirectoryMarkerPolicy()

Get the directory marker policy of this filesystem.

Returns:

the marker policy.

toString

public String toString()

Overrides:

toString in class Object

getPartitionSize

public long getPartitionSize()

Get the partition size for multipart operations.

Returns:

the value as set during initialization

getMultiPartThreshold

public long getMultiPartThreshold()

Get the threshold for multipart files.

Returns:

the value as set during initialization

isMagicCommitEnabled

public boolean isMagicCommitEnabled()

Is magic commit enabled?

Returns:

true if magic commit support is turned on.

isMagicCommitPath

public boolean isMagicCommitPath(org.apache.hadoop.fs.Path path)

Predicate: is a path a magic commit path? True if magic commit is enabled and the path qualifies as special.

Parameters:

path - path to examine

Returns:

true if the path is or is under a magic directory

globStatus

public org.apache.hadoop.fs.FileStatus[] globStatus(org.apache.hadoop.fs.Path pathPattern)
                                             throws IOException

Increments the statistic Statistic.INVOCATION_GLOB_STATUS.

Overrides:

globStatus in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

globStatus

public org.apache.hadoop.fs.FileStatus[] globStatus(org.apache.hadoop.fs.Path pathPattern,
                                                    org.apache.hadoop.fs.PathFilter filter)
                                             throws IOException

Override superclass so as to add statistic collection.

Overrides:

globStatus in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

exists

public boolean exists(org.apache.hadoop.fs.Path f)
               throws IOException

Override superclass so as to add statistic collection.

Overrides:

exists in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

isDirectory

public boolean isDirectory(org.apache.hadoop.fs.Path f)
                    throws IOException

Override superclass so as to add statistic collection.

Overrides:

isDirectory in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

isFile

public boolean isFile(org.apache.hadoop.fs.Path f)
               throws IOException

Override superclass so as to add statistic collection.

Overrides:

isFile in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

getFileChecksum

@Retries.RetryTranslated
public org.apache.hadoop.fs.store.EtagChecksum getFileChecksum(org.apache.hadoop.fs.Path f,
                                                                                        long length)
                                                                                 throws IOException

When enabled, get the etag of a object at the path via HEAD request and return it as a checksum object.

1. If a tag has not changed, consider the object unchanged.
2. Two tags being different does not imply the data is different.

Different S3 implementations may offer different guarantees. This check is (currently) only made if Constants.ETAG_CHECKSUM_ENABLED is set; turning it on has caused problems with Distcp (HADOOP-15273).

Overrides:

getFileChecksum in class org.apache.hadoop.fs.FileSystem

Parameters:

f - The file path

length - The length of the file range for checksum calculation

Returns:

The EtagChecksum or null if checksums are not enabled or supported.

Throws:

IOException - IO failure

See Also:

Common Response Headers

listFiles

@Retries.OnceTranslated
public org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus> listFiles(org.apache.hadoop.fs.Path f,
                                                                                                                     boolean recursive)
                                                                                                              throws FileNotFoundException,
                                                                                                                     IOException

. This implementation is optimized for S3, which can do a bulk listing off all entries under a path in one single operation. Thus there is no need to recursively walk the directory tree. Instead a ListObjectsRequest is created requesting a (windowed) listing of all entries under the given path. This is used to construct an ObjectListingIterator instance, iteratively returning the sequence of lists of elements under the path. This is then iterated over in a FileStatusListingIterator, which generates S3AFileStatus instances, one per listing entry. These are then translated into LocatedFileStatus instances. This is essentially a nested and wrapped set of iterators, with some generator classes; an architecture which may become less convoluted using lambda-expressions.

Overrides:

listFiles in class org.apache.hadoop.fs.FileSystem

Parameters:

f - a path

recursive - if the subdirectories need to be traversed recursively

Returns:

an iterator that traverses statuses of the files/directories in the given path

Throws:

FileNotFoundException - if path does not exist

IOException - if any I/O error occurred

listFilesAndEmptyDirectories

@Retries.OnceTranslated
public org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus> listFilesAndEmptyDirectories(org.apache.hadoop.fs.Path f,
                                                                                                                                        boolean recursive)
                                                                                                                                 throws IOException

Throws:

IOException

listLocatedStatus

public org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus> listLocatedStatus(org.apache.hadoop.fs.Path f)
                                                                                              throws FileNotFoundException,
                                                                                                     IOException

Override superclass so as to add statistic collection.

Overrides:

listLocatedStatus in class org.apache.hadoop.fs.FileSystem

Throws:

FileNotFoundException

IOException

listLocatedStatus

@Retries.OnceTranslated(value="s3guard not retrying")
public org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus> listLocatedStatus(org.apache.hadoop.fs.Path f,
                                                                                                                                                           org.apache.hadoop.fs.PathFilter filter)
                                                                                                                                                    throws FileNotFoundException,
                                                                                                                                                           IOException

. S3 Optimized directory listing. The initial operation performs the first bulk listing; extra listings will take place when all the current set of results are used up.

Overrides:

listLocatedStatus in class org.apache.hadoop.fs.FileSystem

Parameters:

f - a path

filter - a path filter

Returns:

an iterator that traverses statuses of the files/directories in the given path

Throws:

FileNotFoundException - if path does not exist

IOException - if any I/O error occurred

listUploads

public MultipartUtils.UploadIterator listUploads(@Nullable
                                                 String prefix)
                                          throws IOException

List any pending multipart uploads whose keys begin with prefix, using an iterator that can handle an unlimited number of entries. See listMultipartUploads(String) for a non-iterator version of this.

Parameters:

prefix - optional key prefix to search

Returns:

Iterator over multipart uploads.

Throws:

IOException - on failure

listMultipartUploads

@InterfaceAudience.Private
 @Retries.RetryTranslated
public List<com.amazonaws.services.s3.model.MultipartUpload> listMultipartUploads(String prefix)
                                                                                                                                throws IOException

Listing all multipart uploads; limited to the first few hundred. See listUploads(String) for an iterator-based version that does not limit the number of entries returned. Retry policy: retry, translated.

Parameters:

prefix - prefix to scan for, "" for none

Returns:

a listing of multipart uploads.

Throws:

IOException - IO failure, including any uprated AmazonClientException

newCommitterStatistics

public S3AInstrumentation.CommitterStatistics newCommitterStatistics()

Create a new instance of the committer statistics.

Returns:

a new committer statistics instance

hasPathCapability

public boolean hasPathCapability(org.apache.hadoop.fs.Path path,
                                 String capability)
                          throws IOException

Specified by:

hasPathCapability in interface org.apache.hadoop.fs.PathCapabilities

Overrides:

hasPathCapability in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

hasCapability

@Deprecated
public boolean hasCapability(String capability)

Deprecated.

Return the capabilities of this filesystem instance. This has been supplanted by hasPathCapability(Path, String).

Specified by:

hasCapability in interface org.apache.hadoop.fs.StreamCapabilities

Parameters:

capability - string to query the stream support for.

Returns:

whether the FS instance has the capability.

shareCredentials

public AWSCredentialProviderList shareCredentials(String purpose)

Get a shared copy of the AWS credentials, with its reference counter updated. Caller is required to call close() on this after they have finished using it.

Parameters:

purpose - what is this for? This is initially for logging

Returns:

a reference to shared credentials.