S3AFileSystem (Apache Hadoop Amazon Web Services support 3.4.0 API)

java.lang.Object
- org.apache.hadoop.conf.Configured
- - org.apache.hadoop.fs.FileSystem
  - - org.apache.hadoop.fs.s3a.S3AFileSystem

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.conf.Configurable, org.apache.hadoop.fs.PathCapabilities, AWSPolicyProvider, DelegationTokenProvider, org.apache.hadoop.fs.statistics.IOStatisticsSource, org.apache.hadoop.fs.store.audit.ActiveThreadSpanSource<AuditSpanS3A>, org.apache.hadoop.fs.store.audit.AuditSpanSource<AuditSpanS3A>, org.apache.hadoop.fs.StreamCapabilities, org.apache.hadoop.security.token.DelegationTokenIssuer
```
@InterfaceAudience.Private
 @InterfaceStability.Evolving
public class S3AFileSystem
extends org.apache.hadoop.fs.FileSystem
implements org.apache.hadoop.fs.StreamCapabilities, AWSPolicyProvider, DelegationTokenProvider, org.apache.hadoop.fs.statistics.IOStatisticsSource, org.apache.hadoop.fs.store.audit.AuditSpanSource<AuditSpanS3A>, org.apache.hadoop.fs.store.audit.ActiveThreadSpanSource<AuditSpanS3A>
```
The core S3A Filesystem implementation. This subclass is marked as private as code should not be creating it directly; use FileSystem.get(Configuration) and variants to create one. If cast to S3AFileSystem, extra methods and features may be accessed. Consider those private and unstable. Because it prints some of the state of the instrumentation, the output of toString() must also be considered unstable.

Nested Class Summary

Nested Classes
Modifier and Type	Class and Description
`protected class`	`S3AFileSystem.CopyFromLocalCallbacksImpl`
`protected class`	`S3AFileSystem.GetContentSummaryCallbacksImpl` Callbacks from the `GetContentSummaryOperation`.
`protected class`	`S3AFileSystem.HeaderProcessingCallbacksImpl` Operations needed for Header Processing.
`protected class`	`S3AFileSystem.ListingOperationCallbacksImpl` Callbacks from `Listing`.
`protected class`	`S3AFileSystem.MkdirOperationCallbacksImpl` Callbacks from the `MkdirOperation`.

Nested classes/interfaces inherited from class org.apache.hadoop.fs.FileSystem
org.apache.hadoop.fs.FileSystem.DirectoryEntries, org.apache.hadoop.fs.FileSystem.DirListingIterator<T extends org.apache.hadoop.fs.FileStatus>, org.apache.hadoop.fs.FileSystem.Statistics

Nested classes/interfaces inherited from interface org.apache.hadoop.fs.StreamCapabilities
org.apache.hadoop.fs.StreamCapabilities.StreamCapability

Nested classes/interfaces inherited from interface org.apache.hadoop.fs.s3a.auth.delegation.AWSPolicyProvider
AWSPolicyProvider.AccessLevel

Field Summary

Fields
Modifier and Type Field and Description

static int DEFAULT_BLOCKSIZE
Default blocksize as used in blocksize and FS status queries.

static org.slf4j.Logger LOG
- Fields inherited from class org.apache.hadoop.fs.FileSystem
  DEFAULT_FS, FS_DEFAULT_NAME_KEY, SHUTDOWN_HOOK_PRIORITY, statistics, TRASH_PREFIX, USER_HOME_PREFIX
- Fields inherited from interface org.apache.hadoop.fs.StreamCapabilities
  ABORTABLE_STREAM, DROPBEHIND, HFLUSH, HSYNC, IOSTATISTICS, IOSTATISTICS_CONTEXT, PREADBYTEBUFFER, READAHEAD, READBYTEBUFFER, UNBUFFER, VECTOREDIO
- Fields inherited from interface org.apache.hadoop.security.token.DelegationTokenIssuer
  TOKEN_LOG

Fields
Modifier and Type	Field and Description
`static int`	`DEFAULT_BLOCKSIZE` Default blocksize as used in blocksize and FS status queries.
`static org.slf4j.Logger`	`LOG`

Constructor Summary

Constructors
Constructor and Description

S3AFileSystem()

Constructors
Constructor and Description
`S3AFileSystem()`

Method Summary

All Methods Static Methods Instance Methods Concrete Methods Deprecated Methods
Modifier and Type	Method and Description
`void`	`abortMultipartUpload(software.amazon.awssdk.services.s3.model.MultipartUpload upload)` Abort a multipart upload.
`void`	`abortMultipartUpload(String destKey, String uploadId)` Abort a multipart upload.
`void`	`abortOutstandingMultipartUploads(long seconds)` Abort all outstanding MPUs older than a given age.
`void`	`access(org.apache.hadoop.fs.Path f, org.apache.hadoop.fs.permission.FsAction mode)` Soft check of access by forwarding to the audit manager and so on to the auditor.
`boolean`	`allowAuthoritative(org.apache.hadoop.fs.Path path)` Is a path to be considered as authoritative? is a store with the supplied path under one of the paths declared as authoritative.
`org.apache.hadoop.fs.FSDataOutputStream`	`append(org.apache.hadoop.fs.Path f, int bufferSize, org.apache.hadoop.util.Progressable progress)` Append to an existing file (optional operation).
`boolean`	`cancelDeleteOnExit(org.apache.hadoop.fs.Path f)` Cancel the scheduled deletion of the path when the FileSystem is closed.
`protected URI`	`canonicalizeUri(URI rawUri)` Override the base canonicalization logic and relay to `S3xLoginHelper.canonicalizeUri(URI, int)`.
`void`	`checkPath(org.apache.hadoop.fs.Path path)` Check that a Path belongs to this FileSystem.
`void`	`close()` Close the filesystem.
`protected S3ListResult`	`continueListObjects(S3ListRequest request, S3ListResult prevResult, org.apache.hadoop.fs.statistics.DurationTrackerFactory trackerFactory)` List the next set of objects.
`void`	`copyFromLocalFile(boolean delSrc, boolean overwrite, org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst)` The src file is on the local disk.
`org.apache.hadoop.fs.FSDataOutputStream`	`create(org.apache.hadoop.fs.Path f, org.apache.hadoop.fs.permission.FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, org.apache.hadoop.util.Progressable progress)` Create an FSDataOutputStream at the indicated Path with write-progress reporting.
`protected CopyFromLocalOperation.CopyFromLocalOperationCallbacks`	`createCopyFromLocalCallbacks(AuditSpanS3A span)` Create the CopyFromLocalCallbacks; protected to assist in mocking.
`DelegationOperations`	`createDelegationOperations()` Create an instance of the delegation operations.
`org.apache.hadoop.fs.FSDataOutputStreamBuilder`	`createFile(org.apache.hadoop.fs.Path path)` Create instance of an FSDataOutputStreamBuilder for creating a file at the given path.
`protected GetContentSummaryOperation.GetContentSummaryCallbacks`	`createGetContentSummaryCallbacks()` Override point: create the callbacks for getContentSummary.
`protected HeaderProcessing.HeaderProcessingCallbacks`	`createHeaderProcessingCallbacks()` Create the callbacks.
`S3ListRequest`	`createListObjectsRequest(String key, String delimiter)` Create a `ListObjectsRequest` request against this bucket, with the maximum keys returned in a query set by `maxKeys`.
`MarkerToolOperations`	`createMarkerToolOperations(String target)` Create a marker tools operations binding for this store.
`MkdirOperation.MkdirCallbacks`	`createMkdirOperationCallbacks()` Override point: create the callbacks for Mkdir.
`S3AMultipartUploaderBuilder`	`createMultipartUploader(org.apache.hadoop.fs.Path basePath)`
`org.apache.hadoop.fs.FSDataOutputStream`	`createNonRecursive(org.apache.hadoop.fs.Path p, org.apache.hadoop.fs.permission.FsPermission permission, EnumSet<org.apache.hadoop.fs.CreateFlag> flags, int bufferSize, short replication, long blockSize, org.apache.hadoop.util.Progressable progress)` The S3A implementations downgrades to the recursive creation, to avoid any race conditions with parent entries "disappearing".
`protected S3AReadOpContext`	`createReadContext(org.apache.hadoop.fs.FileStatus fileStatus, org.apache.hadoop.fs.store.audit.AuditSpan auditSpan)` Create the read context for reading from the referenced file, using FS state as well as the status.
`protected RequestFactory`	`createRequestFactory()` Build the request factory.
`protected S3AInternals`	`createS3AInternals()` Create the S3AInternals; left as something mocking subclasses may want to override.
`AuditSpanS3A`	`createSpan(String operation, String path1, String path2)` Start an operation; this informs the audit service of the event and then sets it as the active span.
`protected static S3AStorageStatistics`	`createStorageStatistics(org.apache.hadoop.fs.statistics.IOStatistics ioStatistics)` Create the storage statistics or bind to an existing one.
`StoreContext`	`createStoreContext()` Build an immutable store context.
`WriteOperationHelper`	`createWriteOperationHelper(org.apache.hadoop.fs.store.audit.AuditSpan auditSpan)` Create a Write Operation Helper with the given span.
`protected void`	`decrementGauge(Statistic statistic, long count)` Decrement a gauge by a specific value.
`boolean`	`delete(org.apache.hadoop.fs.Path f, boolean recursive)` Delete a Path.
`protected void`	`deleteObject(String key)` Delete an object.
`boolean`	`deleteOnExit(org.apache.hadoop.fs.Path f)` This override bypasses checking for existence.
`protected boolean`	`deleteWithoutCloseCheck(org.apache.hadoop.fs.Path f, boolean recursive)` Same as delete(), except that it does not check if fs is closed.
`protected org.apache.hadoop.fs.store.audit.AuditSpan`	`entryPoint(Statistic operation, org.apache.hadoop.fs.Path path)` Entry point to an operation.
`protected org.apache.hadoop.fs.store.audit.AuditSpan`	`entryPoint(Statistic operation, String path1, String path2)` Entry point to an operation.
`boolean`	`exists(org.apache.hadoop.fs.Path f)` Override superclass so as to add statistic collection.
`AuditSpanS3A`	`getActiveAuditSpan()` Get the active audit span.
`org.apache.hadoop.security.token.DelegationTokenIssuer[]`	`getAdditionalTokenIssuers()` Ask any DT plugin for any extra token issuers.
`AuditManagerS3A`	`getAuditManager()` The audit manager.
`OperationAuditor`	`getAuditor()` Get the auditor; valid once initialized.
`org.apache.hadoop.fs.store.audit.AuditSpanSource`	`getAuditSpanSource()` Get the audit span source; allows for components like the committers to have a source of spans without being hard coded to the FS only.
`String`	`getBucket()` Get the bucket of this filesystem.
`String`	`getBucketLocation()` S3AInternals method.
`protected software.amazon.awssdk.services.s3.model.HeadBucketResponse`	`getBucketMetadata()` Request bucket metadata.
`String`	`getCanonicalServiceName()` Return a service name iff delegation tokens are enabled and the token binding is issuing delegation tokens.
`URI`	`getCanonicalUri()` Get the canonical URI.
`ChangeDetectionPolicy`	`getChangeDetectionPolicy()` Get the change detection policy for this FS instance.
`org.apache.hadoop.fs.ContentSummary`	`getContentSummary(org.apache.hadoop.fs.Path f)` This is a very slow operation against object storage.
`long`	`getDefaultBlockSize()` Return the number of bytes that large input files should be optimally be split into to minimize I/O time.
`int`	`getDefaultPort()`
`org.apache.hadoop.security.token.Token<AbstractS3ATokenIdentifier>`	`getDelegationToken(String renewer)` Get a delegation token if the FS is set up for them.
`Optional<S3ADelegationTokens>`	`getDelegationTokens()` Get the delegation token support for this filesystem; not null iff delegation support is enabled.
`DirectoryPolicy`	`getDirectoryMarkerPolicy()` Get the directory marker policy of this filesystem.
`protected org.apache.hadoop.fs.statistics.DurationTrackerFactory`	`getDurationTrackerFactory()` Get the factory for duration tracking.
`EncryptionSecrets`	`getEncryptionSecrets()` Get the encryption secrets.
`org.apache.hadoop.fs.store.EtagChecksum`	`getFileChecksum(org.apache.hadoop.fs.Path f, long length)` When enabled, get the etag of a object at the path via HEAD request and return it as a checksum object.
`org.apache.hadoop.fs.FileStatus`	`getFileStatus(org.apache.hadoop.fs.Path f)` Return a file status object that represents the path.
`org.apache.hadoop.security.token.Token<? extends org.apache.hadoop.security.token.TokenIdentifier>`	`getFsDelegationToken()`
`org.apache.hadoop.fs.FileSystem.Statistics`	`getFsStatistics()` Get FS Statistic for this S3AFS instance.
`S3AInputPolicy`	`getInputPolicy()` Get the input policy for this FS instance.
`S3AInstrumentation`	`getInstrumentation()` Get S3A Instrumentation.
`org.apache.hadoop.fs.statistics.IOStatistics`	`getIOStatistics()` Get the instrumentation's IOStatistics.
`Listing`	`getListing()` Get current listing instance.
`long`	`getMultiPartThreshold()` Get the threshold for multipart files.
`software.amazon.awssdk.services.s3.model.HeadObjectResponse`	`getObjectMetadata(org.apache.hadoop.fs.Path path)` Deprecated. use S3AInternals API.
`protected software.amazon.awssdk.services.s3.model.HeadObjectResponse`	`getObjectMetadata(String key, ChangeTracker changeTracker, Invoker changeInvoker, String operation)` Request object metadata; increments counters in the process.
`org.apache.hadoop.security.UserGroupInformation`	`getOwner()` Get the owner of this FS: who created it?
`long`	`getPartitionSize()` Get the partition size for multipart operations.
`RequestFactory`	`getRequestFactory()` Get the request factory which uses this store's audit span.
`S3AInternals`	`getS3AInternals()` Get the S3AInternals.
`S3AEncryptionMethods`	`getS3EncryptionAlgorithm()` Get the encryption algorithm of this connector.
`String`	`getScheme()` Return the protocol scheme for the FileSystem.
`S3AStorageStatistics`	`getStorageStatistics()` Get the storage statistics of this filesystem.
`URI`	`getUri()` Returns a URI whose scheme and authority identify this FileSystem.
`String`	`getUsername()` Get the username of the FS.
`org.apache.hadoop.fs.Path`	`getWorkingDirectory()` Get the current working directory for the given file system.
`WriteOperationHelper`	`getWriteOperationHelper()` Create a Write Operation Helper with the current active span.
`byte[]`	`getXAttr(org.apache.hadoop.fs.Path path, String name)`
`Map<String,byte[]>`	`getXAttrs(org.apache.hadoop.fs.Path path)`
`Map<String,byte[]>`	`getXAttrs(org.apache.hadoop.fs.Path path, List<String> names)`
`org.apache.hadoop.fs.FileStatus[]`	`globStatus(org.apache.hadoop.fs.Path pathPattern)` Increments the statistic `Statistic.INVOCATION_GLOB_STATUS`.
`org.apache.hadoop.fs.FileStatus[]`	`globStatus(org.apache.hadoop.fs.Path pathPattern, org.apache.hadoop.fs.PathFilter filter)` Increments the statistic `Statistic.INVOCATION_GLOB_STATUS`.
`boolean`	`hasCapability(String capability)` Deprecated.
`boolean`	`hasPathCapability(org.apache.hadoop.fs.Path path, String capability)`
`protected void`	`incrementGauge(Statistic statistic, long count)` Increment a gauge by a specific value.
`void`	`incrementPutCompletedStatistics(boolean success, long bytes)` At the end of a put/multipart upload operation, update the relevant counters and gauges.
`void`	`incrementPutProgressStatistics(String key, long bytes)` Callback for use in progress callbacks from put/multipart upload events.
`void`	`incrementPutStartStatistics(long bytes)` At the start of a put/multipart upload operation, update the relevant counters.
`void`	`incrementReadOperations()` Increment read operations.
`protected void`	`incrementStatistic(Statistic statistic)` Increment a statistic by 1.
`protected void`	`incrementStatistic(Statistic statistic, long count)` Increment a statistic by a specific value.
`void`	`incrementWriteOperations()` Increment the write operation counter.
`void`	`initialize(URI name, org.apache.hadoop.conf.Configuration originalConf)` Called after a new FileSystem instance is constructed.
`protected void`	`initializeAuditService()` Initialize and launch the audit manager and service.
`static void`	`initializeClass()` This is purely for testing, as it force initializes all static initializers.
`protected void`	`initializeStatisticsBinding()` Initialize the statistics binding.
`boolean`	`isCSEEnabled()` a method to know if Client side encryption is enabled or not.
`boolean`	`isDirectory(org.apache.hadoop.fs.Path f)` Optimized probe for a path referencing a dir.
`boolean`	`isFile(org.apache.hadoop.fs.Path f)` Optimized probe for a path referencing a file.
`boolean`	`isMagicCommitEnabled()` Is magic commit enabled?
`boolean`	`isMagicCommitPath(org.apache.hadoop.fs.Path path)` Predicate: is a path a magic commit path? True if magic commit is enabled and the path qualifies as special, and is not a a .pending or .pendingset file,
`boolean`	`isMultipartUploadEnabled()`
`org.apache.hadoop.fs.Path`	`keyToQualifiedPath(String key)` Convert a key to a fully qualified path.
`List<RoleModel.Statement>`	`listAWSPolicyRules(Set<AWSPolicyProvider.AccessLevel> access)` Build the AWS policy for restricted access to the resources needed by this bucket.
`org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus>`	`listFiles(org.apache.hadoop.fs.Path f, boolean recursive)` .
`org.apache.hadoop.fs.RemoteIterator<S3ALocatedFileStatus>`	`listFilesAndEmptyDirectories(org.apache.hadoop.fs.Path f, boolean recursive)` Recursive List of files and empty directories.
`org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus>`	`listLocatedStatus(org.apache.hadoop.fs.Path f)` Override superclass so as to add statistic collection.
`org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus>`	`listLocatedStatus(org.apache.hadoop.fs.Path f, org.apache.hadoop.fs.PathFilter filter)` .
`List<software.amazon.awssdk.services.s3.model.MultipartUpload>`	`listMultipartUploads(String prefix)` Listing all multipart uploads; limited to the first few hundred.
`protected S3ListResult`	`listObjects(S3ListRequest request, org.apache.hadoop.fs.statistics.DurationTrackerFactory trackerFactory)` Initiate a `listObjects` operation, incrementing metrics in the process.
`org.apache.hadoop.fs.FileStatus[]`	`listStatus(org.apache.hadoop.fs.Path f)` List the statuses of the files/directories in the given path if the path is a directory.
`org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.FileStatus>`	`listStatusIterator(org.apache.hadoop.fs.Path p)` Override subclass such that we benefit for async listing done in `S3AFileSystem`.
`org.apache.hadoop.fs.RemoteIterator<software.amazon.awssdk.services.s3.model.MultipartUpload>`	`listUploads(String prefix)` List any pending multipart uploads whose keys begin with prefix, using an iterator that can handle an unlimited number of entries.
`org.apache.hadoop.fs.RemoteIterator<software.amazon.awssdk.services.s3.model.MultipartUpload>`	`listUploadsUnderPrefix(StoreContext storeContext, String prefix)` List any pending multipart uploads whose keys begin with prefix, using an iterator that can handle an unlimited number of entries.
`List<String>`	`listXAttrs(org.apache.hadoop.fs.Path path)`
`org.apache.hadoop.fs.Path`	`makeQualified(org.apache.hadoop.fs.Path path)`
`String`	`maybeAddTrailingSlash(String key)` Turns a path (relative or otherwise) into an S3 key, adding a trailing "/" if the path is not the root and does not already have a "/" at the end.
`protected void`	`maybeCreateFakeParentDirectory(org.apache.hadoop.fs.Path path)` Create a fake parent directory if required.
`boolean`	`mkdirs(org.apache.hadoop.fs.Path p, org.apache.hadoop.fs.permission.FsPermission permission)` Make the given path and all non-existent parents into directories.
`CommitterStatistics`	`newCommitterStatistics()` Create a new instance of the committer statistics.
`software.amazon.awssdk.services.s3.model.PutObjectRequest.Builder`	`newPutObjectRequestBuilder(String key, long length, boolean isDirectoryMarker)` Create a putObject request builder.
`protected org.apache.hadoop.fs.statistics.DurationTrackerFactory`	`nonNullDurationTrackerFactory(org.apache.hadoop.fs.statistics.DurationTrackerFactory factory)` Given a possibly null duration tracker factory, return a non-null one for use in tracking durations -either that or the FS tracker itself.
`org.apache.hadoop.fs.FSDataInputStream`	`open(org.apache.hadoop.fs.Path f, int bufferSize)` Opens an FSDataInputStream at the indicated Path.
`CompletableFuture<org.apache.hadoop.fs.FSDataInputStream>`	`openFileWithOptions(org.apache.hadoop.fs.Path rawPath, org.apache.hadoop.fs.impl.OpenFileParameters parameters)` Initiate the open() operation.
`void`	`operationRetried(Exception ex)` Callback when an operation was retried.
`void`	`operationRetried(String text, Exception ex, int retries, boolean idempotent)` Callback from `Invoker` when an operation is retried.
`String`	`pathToKey(org.apache.hadoop.fs.Path path)` Turns a path (relative or otherwise) into an S3 key.
`protected void`	`processDeleteOnExit()` Delete all paths that were marked as delete-on-exit.
`UploadInfo`	`putObject(software.amazon.awssdk.services.s3.model.PutObjectRequest putObjectRequest, File file, ProgressableProgressListener listener)` Start a transfer-manager managed async PUT of an object, incrementing the put requests and put bytes counters.
`org.apache.hadoop.fs.Path`	`qualify(org.apache.hadoop.fs.Path path)` Qualify a path.
`void`	`removeKeys(List<software.amazon.awssdk.services.s3.model.ObjectIdentifier> keysToDelete, boolean deleteFakeDir)` Invoke `removeKeysS3(List, boolean)`.
`boolean`	`rename(org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst)` Renames Path src to Path dst.
`protected void`	`setAmazonS3Client(software.amazon.awssdk.services.s3.S3Client client)` Set the client -used in mocking tests to force in a different client.
`protected void`	`setBucket(String bucket)` Set the bucket.
`protected void`	`setEncryptionSecrets(EncryptionSecrets secrets)` Set the encryption secrets for requests.
`void`	`setInputPolicy(S3AInputPolicy inputPolicy)` Deprecated. use openFile() options
`protected void`	`setUri(URI fsUri, boolean canonicalize)` Set the URI field through `S3xLoginHelper` and optionally `canonicalizeUri(URI)` Exported for testing.
`void`	`setWorkingDirectory(org.apache.hadoop.fs.Path newDir)` Set the current working directory for the given file system.
`AWSCredentialProviderList`	`shareCredentials(String purpose)` Get a shared copy of the AWS credentials, with its reference counter updated.
`protected void`	`stopAllServices()` Stop all services.
`String`	`toString()`
`protected void`	`verifyBucketExists()` Verify that the bucket exists.

Methods inherited from class org.apache.hadoop.fs.FileSystem
append, append, append, append, appendFile, areSymlinksEnabled, clearStatistics, closeAll, closeAllForUGI, completeLocalOutput, concat, copyFromLocalFile, copyFromLocalFile, copyFromLocalFile, copyToLocalFile, copyToLocalFile, copyToLocalFile, create, create, create, create, create, create, create, create, create, create, create, create, createDataInputStreamBuilder, createDataInputStreamBuilder, createDataOutputStreamBuilder, createNewFile, createNonRecursive, createNonRecursive, createPathHandle, createSnapshot, createSnapshot, createSymlink, delete, deleteSnapshot, enableSymlinks, fixRelativePart, get, get, get, getAclStatus, getAllStatistics, getAllStoragePolicies, getBlockSize, getChildFileSystems, getDefaultBlockSize, getDefaultReplication, getDefaultReplication, getDefaultUri, getEnclosingRoot, getFileBlockLocations, getFileBlockLocations, getFileChecksum, getFileLinkStatus, getFileSystemClass, getFSofPath, getGlobalStorageStatistics, getHomeDirectory, getInitialWorkingDirectory, getLength, getLinkTarget, getLocal, getName, getNamed, getPathHandle, getQuotaUsage, getReplication, getServerDefaults, getServerDefaults, getStatistics, getStatistics, getStatus, getStatus, getStoragePolicy, getTrashRoot, getTrashRoots, getUsed, getUsed, listCorruptFileBlocks, listStatus, listStatus, listStatus, listStatusBatch, mkdirs, mkdirs, modifyAclEntries, moveFromLocalFile, moveFromLocalFile, moveToLocalFile, msync, newInstance, newInstance, newInstance, newInstanceLocal, open, open, open, openFile, openFile, openFileWithOptions, primitiveCreate, primitiveMkdir, primitiveMkdir, printStatistics, removeAcl, removeAclEntries, removeDefaultAcl, removeXAttr, rename, renameSnapshot, resolveLink, resolvePath, satisfyStoragePolicy, setAcl, setDefaultUri, setDefaultUri, setOwner, setPermission, setQuota, setQuotaByStorageType, setReplication, setStoragePolicy, setTimes, setVerifyChecksum, setWriteChecksum, setXAttr, setXAttr, startLocalOutput, supportsSymlinks, truncate, unsetStoragePolicy

Methods inherited from class org.apache.hadoop.conf.Configured
getConf, setConf

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.hadoop.security.token.DelegationTokenIssuer
addDelegationTokens, collectDelegationTokens

- Field Detail
  - DEFAULT_BLOCKSIZE
```
public static final int DEFAULT_BLOCKSIZE
```
    Default blocksize as used in blocksize and FS status queries.
    
    See Also:
    
    Constant Field Values
  - LOG
```
public static final org.slf4j.Logger LOG
```
- Constructor Detail
  - S3AFileSystem
```
public S3AFileSystem()
```
- Method Detail
  - initialize
```
public void initialize(URI name,
                       org.apache.hadoop.conf.Configuration originalConf)
                throws IOException
```
    Called after a new FileSystem instance is constructed.
    
    Overrides:
    
    initialize in class org.apache.hadoop.fs.FileSystem
    
    Parameters:
    
    name - a uri whose authority section names the host, port, etc. for this FileSystem
    
    originalConf - the configuration to use for the FS. The bucket-specific options are patched over the base ones before any use is made of the config.
    
    Throws:
    
    IOException
  - initializeStatisticsBinding
```
protected void initializeStatisticsBinding()
```
    Initialize the statistics binding. This is done by creating an IntegratedS3AStatisticsContext with callbacks to get the FS's instrumentation and FileSystem.statistics field; the latter may change after initialize(URI, Configuration), so needs to be dynamically adapted. Protected so that (mock) subclasses can replace it with a different statistics binding, if desired.
  - createStorageStatistics
```
protected static S3AStorageStatistics createStorageStatistics(org.apache.hadoop.fs.statistics.IOStatistics ioStatistics)
```
    Create the storage statistics or bind to an existing one.
    
    Parameters:
    
    ioStatistics - IOStatistics to build the storage statistics from.
    
    Returns:
    
    a storage statistics instance; expected to be that of the FS.
  - verifyBucketExists
```
@Retries.RetryTranslated
protected void verifyBucketExists()
                                                    throws UnknownStoreException,
                                                           IOException
```
    Verify that the bucket exists. Retry policy: retrying, translated.
    
    Throws:
    
    UnknownStoreException - the bucket is absent
    
    IOException - any other problem talking to S3
  - getInstrumentation
```
@VisibleForTesting
public S3AInstrumentation getInstrumentation()
```
    Get S3A Instrumentation. For test purposes.
    
    Returns:
    
    this instance's instrumentation.
  - getFsStatistics
```
@VisibleForTesting
public org.apache.hadoop.fs.FileSystem.Statistics getFsStatistics()
```
    Get FS Statistic for this S3AFS instance.
    
    Returns:
    
    FS statistic instance.
  - getListing
```
public Listing getListing()
```
    Get current listing instance.
    
    Returns:
    
    this instance's listing.
  - initializeAuditService
```
protected void initializeAuditService()
                               throws IOException
```
    Initialize and launch the audit manager and service. As this takes the FS IOStatistics store, it must be invoked after instrumentation is initialized.
    
    Throws:
    
    IOException - failure to instantiate/initialize.
  - getAuditManager
```
@InterfaceAudience.Private
public AuditManagerS3A getAuditManager()
```
    The audit manager.
    
    Returns:
    
    the audit manager
  - getAuditor
```
@InterfaceAudience.Private
public OperationAuditor getAuditor()
```
    Get the auditor; valid once initialized.
    
    Returns:
    
    the auditor.
  - getActiveAuditSpan
```
@InterfaceAudience.Private
public AuditSpanS3A getActiveAuditSpan()
```
    Get the active audit span.
    
    Specified by:
    
    getActiveAuditSpan in interface org.apache.hadoop.fs.store.audit.ActiveThreadSpanSource<AuditSpanS3A>
    
    Returns:
    
    the span.
  - getAuditSpanSource
```
@InterfaceAudience.Private
public org.apache.hadoop.fs.store.audit.AuditSpanSource getAuditSpanSource()
```
    Get the audit span source; allows for components like the committers to have a source of spans without being hard coded to the FS only.
    
    Returns:
    
    the source of spans -base implementation is this instance.
  - createSpan
```
public AuditSpanS3A createSpan(String operation,
                               @Nullable
                               String path1,
                               @Nullable
                               String path2)
                        throws IOException
```
    Start an operation; this informs the audit service of the event and then sets it as the active span.
    
    Specified by:
    
    createSpan in interface org.apache.hadoop.fs.store.audit.AuditSpanSource<AuditSpanS3A>
    
    Parameters:
    
    operation - operation name.
    
    path1 - first path of operation
    
    path2 - second path of operation
    
    Returns:
    
    a span for the audit
    
    Throws:
    
    IOException - failure
  - createRequestFactory
```
protected RequestFactory createRequestFactory()
```
    Build the request factory. MUST be called after reading encryption secrets from settings/ delegation token. Protected, in case test/mock classes want to implement their own variants.
    
    Returns:
    
    request factory.
  - getRequestFactory
```
@VisibleForTesting
public RequestFactory getRequestFactory()
```
    Get the request factory which uses this store's audit span.
    
    Returns:
    
    the request factory.
  - createDelegationOperations
```
@VisibleForTesting
public DelegationOperations createDelegationOperations()
```
    Create an instance of the delegation operations.
    
    Returns:
    
    callbacks for DT support.
  - setEncryptionSecrets
```
protected void setEncryptionSecrets(EncryptionSecrets secrets)
```
    Set the encryption secrets for requests.
    
    Parameters:
    
    secrets - secrets
  - getEncryptionSecrets
```
public EncryptionSecrets getEncryptionSecrets()
```
    Get the encryption secrets. This potentially sensitive information and must be treated with care.
    
    Returns:
    
    the current encryption secrets.
  - abortOutstandingMultipartUploads
```
@Retries.RetryTranslated
public void abortOutstandingMultipartUploads(long seconds)
                                                               throws IOException
```
    Abort all outstanding MPUs older than a given age.
    
    Parameters:
    
    seconds - time in seconds
    
    Throws:
    
    IOException - on any failure, other than 403 "permission denied"
  - getScheme
```
public String getScheme()
```
    Return the protocol scheme for the FileSystem.
    
    Overrides:
    
    getScheme in class org.apache.hadoop.fs.FileSystem
    
    Returns:
    
    "s3a"
  - getUri
```
public URI getUri()
```
    Returns a URI whose scheme and authority identify this FileSystem.
    
    Specified by:
    
    getUri in class org.apache.hadoop.fs.FileSystem
  - setUri
```
@VisibleForTesting
protected void setUri(URI fsUri,
                                         boolean canonicalize)
```
    Set the URI field through S3xLoginHelper and optionally canonicalizeUri(URI) Exported for testing.
    
    Parameters:
    
    fsUri - filesystem URI.
    
    canonicalize - true if the URI should be canonicalized.
  - getCanonicalUri
```
public URI getCanonicalUri()
```
    Get the canonical URI.
    
    Overrides:
    
    getCanonicalUri in class org.apache.hadoop.fs.FileSystem
    
    Returns:
    
    the canonical URI of this FS.
  - getDefaultPort
```
@VisibleForTesting
public int getDefaultPort()
```
    Overrides:
    
    getDefaultPort in class org.apache.hadoop.fs.FileSystem
  - setAmazonS3Client
```
protected void setAmazonS3Client(software.amazon.awssdk.services.s3.S3Client client)
```
    Set the client -used in mocking tests to force in a different client.
    
    Parameters:
    
    client - client.
  - getBucketLocation
```
@AuditEntryPoint
 @Retries.RetryTranslated
public String getBucketLocation()
                                                                    throws IOException
```
    S3AInternals method. .
    
    Throws:
    
    IOException
  - createS3AInternals
```
protected S3AInternals createS3AInternals()
```
    Create the S3AInternals; left as something mocking subclasses may want to override.
    
    Returns:
    
    the internal implementation
  - getS3AInternals
```
public S3AInternals getS3AInternals()
```
    Get the S3AInternals.
    
    Returns:
    
    the internal implementation
  - getInputPolicy
```
@InterfaceStability.Unstable
public S3AInputPolicy getInputPolicy()
```
    Get the input policy for this FS instance.
    
    Returns:
    
    the input policy
  - getChangeDetectionPolicy
```
@VisibleForTesting
public ChangeDetectionPolicy getChangeDetectionPolicy()
```
    Get the change detection policy for this FS instance. Only public to allow access in tests in other packages.
    
    Returns:
    
    the change detection policy
  - getS3EncryptionAlgorithm
```
public S3AEncryptionMethods getS3EncryptionAlgorithm()
```
    Get the encryption algorithm of this connector.
    
    Returns:
    
    the encryption algorithm.
  - getBucket
```
@InterfaceAudience.Public
 @InterfaceStability.Stable
public String getBucket()
```
    Get the bucket of this filesystem.
    
    Returns:
    
    the bucket
  - setBucket
```
@VisibleForTesting
protected void setBucket(String bucket)
```
    Set the bucket.
    
    Parameters:
    
    bucket - the bucket
  - setInputPolicy
```
@InterfaceStability.Unstable
 @Deprecated
public void setInputPolicy(S3AInputPolicy inputPolicy)
```
    Deprecated. use openFile() options
    
    Change the input policy for this FS. This is now a no-op, retained in case some application or external test invokes it.
    
    Parameters:
    
    inputPolicy - new policy
  - pathToKey
```
@VisibleForTesting
public String pathToKey(org.apache.hadoop.fs.Path path)
```
    Turns a path (relative or otherwise) into an S3 key.
    
    Parameters:
    
    path - input path, may be relative to the working dir
    
    Returns:
    
    a key excluding the leading "/", or, if it is the root path, ""
  - maybeAddTrailingSlash
```
@InterfaceAudience.Private
public String maybeAddTrailingSlash(String key)
```
    Turns a path (relative or otherwise) into an S3 key, adding a trailing "/" if the path is not the root and does not already have a "/" at the end.
    
    Parameters:
    
    key - s3 key or ""
    
    Returns:
    
    the with a trailing "/", or, if it is the root key, "",
  - keyToQualifiedPath
```
public org.apache.hadoop.fs.Path keyToQualifiedPath(String key)
```
    Convert a key to a fully qualified path. This includes fixing up the URI so that if it ends with a trailing slash, that is corrected, similar to Path.normalizePath().
    
    Parameters:
    
    key - input key
    
    Returns:
    
    the fully qualified path including URI scheme and bucket name.
  - makeQualified
```
public org.apache.hadoop.fs.Path makeQualified(org.apache.hadoop.fs.Path path)
```
    Overrides:
    
    makeQualified in class org.apache.hadoop.fs.FileSystem
  - qualify
```
public org.apache.hadoop.fs.Path qualify(org.apache.hadoop.fs.Path path)
```
    Qualify a path. This includes fixing up the URI so that if it ends with a trailing slash, that is corrected, similar to Path.normalizePath().
    
    Parameters:
    
    path - path to qualify
    
    Returns:
    
    a qualified path.
  - checkPath
```
public void checkPath(org.apache.hadoop.fs.Path path)
```
    Check that a Path belongs to this FileSystem. Unlike the superclass, this version does not look at authority, only hostnames.
    
    Overrides:
    
    checkPath in class org.apache.hadoop.fs.FileSystem
    
    Parameters:
    
    path - to check
    
    Throws:
    
    IllegalArgumentException - if there is an FS mismatch
  - canonicalizeUri
```
protected URI canonicalizeUri(URI rawUri)
```
    Override the base canonicalization logic and relay to S3xLoginHelper.canonicalizeUri(URI, int). This allows for the option of changing this logic for better DT handling.
    
    Overrides:
    
    canonicalizeUri in class org.apache.hadoop.fs.FileSystem
    
    Parameters:
    
    rawUri - raw URI.
    
    Returns:
    
    the canonical URI to use in delegation tokens and file context.
  - open
```
@Retries.RetryTranslated
public org.apache.hadoop.fs.FSDataInputStream open(org.apache.hadoop.fs.Path f,
                                                                            int bufferSize)
                                                                     throws IOException
```
    Opens an FSDataInputStream at the indicated Path.
    
    Specified by:
    
    open in class org.apache.hadoop.fs.FileSystem
    
    Parameters:
    
    f - the file name to open
    
    bufferSize - the size of the buffer to be used.
    
    Throws:
    
    IOException
  - createReadContext
```
@VisibleForTesting
protected S3AReadOpContext createReadContext(org.apache.hadoop.fs.FileStatus fileStatus,
                                                                org.apache.hadoop.fs.store.audit.AuditSpan auditSpan)
```
    Create the read context for reading from the referenced file, using FS state as well as the status.
    
    Parameters:
    
    fileStatus - file status.
    
    auditSpan - audit span.
    
    Returns:
    
    a context for read operations.
  - create
```
@AuditEntryPoint
public org.apache.hadoop.fs.FSDataOutputStream create(org.apache.hadoop.fs.Path f,
                                                                       org.apache.hadoop.fs.permission.FsPermission permission,
                                                                       boolean overwrite,
                                                                       int bufferSize,
                                                                       short replication,
                                                                       long blockSize,
                                                                       org.apache.hadoop.util.Progressable progress)
                                                                throws IOException
```
    Create an FSDataOutputStream at the indicated Path with write-progress reporting. Retry policy: retrying, translated on the getFileStatus() probe. No data is uploaded to S3 in this call, so retry issues related to that.
    
    Specified by:
    
    create in class org.apache.hadoop.fs.FileSystem
    
    Parameters:
    
    f - the file name to open
    
    permission - the permission to set.
    
    overwrite - if a file with this name already exists, then if true, the file will be overwritten, and if false an error will be thrown.
    
    bufferSize - the size of the buffer to be used.
    
    replication - required block replication for the file.
    
    blockSize - the requested block size.
    
    progress - the progress reporter.
    
    Throws:
    
    IOException - in the event of IO related errors.
    
    See Also:
    
    FileSystem.setPermission(Path, FsPermission)
  - getWriteOperationHelper
```
@InterfaceAudience.Private
public WriteOperationHelper getWriteOperationHelper()
```
    Create a Write Operation Helper with the current active span. All operations made through this helper will activate the span before execution. This class permits other low-level operations against the store. It is unstable and only intended for code with intimate knowledge of the object store. If using this, be prepared for changes even on minor point releases.
    
    Returns:
    
    a new helper.
  - createWriteOperationHelper
```
@InterfaceAudience.Private
public WriteOperationHelper createWriteOperationHelper(org.apache.hadoop.fs.store.audit.AuditSpan auditSpan)
```
    Create a Write Operation Helper with the given span. All operations made through this helper will activate the span before execution.
    
    Parameters:
    
    auditSpan - audit span
    
    Returns:
    
    a new helper.
  - createFile
```
@AuditEntryPoint
public org.apache.hadoop.fs.FSDataOutputStreamBuilder createFile(org.apache.hadoop.fs.Path path)
```
    Create instance of an FSDataOutputStreamBuilder for creating a file at the given path.
    
    Overrides:
    
    createFile in class org.apache.hadoop.fs.FileSystem
    
    Parameters:
    
    path - path to create
    
    Returns:
    
    a builder.
    
    Throws:
    
    UncheckedIOException - for problems creating the audit span
  - createNonRecursive
```
@AuditEntryPoint
public org.apache.hadoop.fs.FSDataOutputStream createNonRecursive(org.apache.hadoop.fs.Path p,
                                                                                   org.apache.hadoop.fs.permission.FsPermission permission,
                                                                                   EnumSet<org.apache.hadoop.fs.CreateFlag> flags,
                                                                                   int bufferSize,
                                                                                   short replication,
                                                                                   long blockSize,
                                                                                   org.apache.hadoop.util.Progressable progress)
                                                                            throws IOException
```
    The S3A implementations downgrades to the recursive creation, to avoid any race conditions with parent entries "disappearing".
    
    Overrides:
    
    createNonRecursive in class org.apache.hadoop.fs.FileSystem
    
    Throws:
    
    IOException
  - append
```
public org.apache.hadoop.fs.FSDataOutputStream append(org.apache.hadoop.fs.Path f,
                                                      int bufferSize,
                                                      org.apache.hadoop.util.Progressable progress)
                                               throws IOException
```
    Append to an existing file (optional operation).
    
    Specified by:
    
    append in class org.apache.hadoop.fs.FileSystem
    
    Parameters:
    
    f - the existing file to be appended.
    
    bufferSize - the size of the buffer to be used.
    
    progress - for reporting progress if it is not null.
    
    Throws:
    
    IOException - indicating that append is not supported.
  - rename
```
@AuditEntryPoint
 @Retries.RetryTranslated
public boolean rename(org.apache.hadoop.fs.Path src,
                                                                 org.apache.hadoop.fs.Path dst)
                                                          throws IOException
```
    Renames Path src to Path dst. Can take place on local fs or remote DFS. Warning: S3 does not support renames. This method does a copy which can take S3 some time to execute with large files and directories. Since there is no Progressable passed in, this can time out jobs. Note: This implementation differs with other S3 drivers. Specifically:
```
       Fails if src is a file and dst is a directory.
       Fails if src is a directory and dst is a file.
       Fails if the parent of dst does not exist or is a file.
       Fails if dst is a directory that is not empty.
 
```
    Specified by:
    
    rename in class org.apache.hadoop.fs.FileSystem
    
    Parameters:
    
    src - path to be renamed
    
    dst - new path after rename
    
    Returns:
    
    true if rename is successful
    
    Throws:
    
    IOException - on IO failure
  - getFsDelegationToken
```
public org.apache.hadoop.security.token.Token<? extends org.apache.hadoop.security.token.TokenIdentifier> getFsDelegationToken()
                                                                                                                        throws IOException
```
    Specified by:
    
    getFsDelegationToken in interface DelegationTokenProvider
    
    Throws:
    
    IOException
  - getObjectMetadata
```
@AuditEntryPoint
 @InterfaceAudience.LimitedPrivate(value="utilities")
 @Retries.RetryTranslated
 @Deprecated
public software.amazon.awssdk.services.s3.model.HeadObjectResponse getObjectMetadata(org.apache.hadoop.fs.Path path)
                                                                                                                                                                                            throws IOException
```
    Deprecated. use S3AInternals API.
    
    Low-level call to get at the object metadata. This method is used in some external applications and so must be viewed as a public entry point.
    
    Parameters:
    
    path - path to the object. This will be qualified.
    
    Returns:
    
    metadata
    
    Throws:
    
    IOException - IO and object access problems.
  - entryPoint
```
protected org.apache.hadoop.fs.store.audit.AuditSpan entryPoint(Statistic operation,
                                                                org.apache.hadoop.fs.Path path)
                                                         throws IOException
```
    Entry point to an operation. Increments the statistic; verifies the FS is active.
    
    Parameters:
    
    operation - The operation being invoked
    
    path - first path of operation
    
    Returns:
    
    a span for the audit
    
    Throws:
    
    IOException - failure of audit service
  - entryPoint
```
protected org.apache.hadoop.fs.store.audit.AuditSpan entryPoint(Statistic operation,
                                                                @Nullable
                                                                String path1,
                                                                @Nullable
                                                                String path2)
                                                         throws IOException
```
    Entry point to an operation. Increments the statistic; verifies the FS is active.
    
    Parameters:
    
    operation - The operation being invoked
    
    path1 - first path of operation
    
    path2 - second path of operation
    
    Returns:
    
    a span for the audit
    
    Throws:
    
    IOException - failure of audit service
  - incrementStatistic
```
protected void incrementStatistic(Statistic statistic)
```
    Increment a statistic by 1. This increments both the instrumentation and storage statistics.
    
    Parameters:
    
    statistic - The operation to increment
  - incrementStatistic
```
protected void incrementStatistic(Statistic statistic,
                                  long count)
```
    Increment a statistic by a specific value. This increments both the instrumentation and storage statistics.
    
    Parameters:
    
    statistic - The operation to increment
    
    count - the count to increment
  - decrementGauge
```
protected void decrementGauge(Statistic statistic,
                              long count)
```
    Decrement a gauge by a specific value.
    
    Parameters:
    
    statistic - The operation to decrement
    
    count - the count to decrement
  - incrementGauge
```
protected void incrementGauge(Statistic statistic,
                              long count)
```
    Increment a gauge by a specific value.
    
    Parameters:
    
    statistic - The operation to increment
    
    count - the count to increment
  - operationRetried
```
public void operationRetried(Exception ex)
```
    Callback when an operation was retried. Increments the statistics of ignored errors or throttled requests, depending up on the exception class.
    
    Parameters:
    
    ex - exception.
  - operationRetried
```
public void operationRetried(String text,
                             Exception ex,
                             int retries,
                             boolean idempotent)
```
    Callback from Invoker when an operation is retried.
    
    Parameters:
    
    text - text of the operation
    
    ex - exception
    
    retries - number of retries
    
    idempotent - is the method idempotent
  - getStorageStatistics
```
public S3AStorageStatistics getStorageStatistics()
```
    Get the storage statistics of this filesystem.
    
    Overrides:
    
    getStorageStatistics in class org.apache.hadoop.fs.FileSystem
    
    Returns:
    
    the storage statistics
  - getIOStatistics
```
public org.apache.hadoop.fs.statistics.IOStatistics getIOStatistics()
```
    Get the instrumentation's IOStatistics.
    
    Specified by:
    
    getIOStatistics in interface org.apache.hadoop.fs.statistics.IOStatisticsSource
    
    Returns:
    
    statistics
  - getDurationTrackerFactory
```
protected org.apache.hadoop.fs.statistics.DurationTrackerFactory getDurationTrackerFactory()
```
    Get the factory for duration tracking.
    
    Returns:
    
    a factory from the instrumentation.
  - nonNullDurationTrackerFactory
```
protected org.apache.hadoop.fs.statistics.DurationTrackerFactory nonNullDurationTrackerFactory(org.apache.hadoop.fs.statistics.DurationTrackerFactory factory)
```
    Given a possibly null duration tracker factory, return a non-null one for use in tracking durations -either that or the FS tracker itself.
    
    Parameters:
    
    factory - factory.
    
    Returns:
    
    a non-null factory.
  - getObjectMetadata
```
@Retries.RetryRaw
protected software.amazon.awssdk.services.s3.model.HeadObjectResponse getObjectMetadata(String key,
                                                                                                          ChangeTracker changeTracker,
                                                                                                          Invoker changeInvoker,
                                                                                                          String operation)
                                                                                                   throws IOException
```
    Request object metadata; increments counters in the process. Retry policy: retry untranslated. Uses changeTracker to detect an unexpected file version (eTag or versionId)
    
    Parameters:
    
    key - key
    
    changeTracker - the change tracker to detect unexpected object version
    
    changeInvoker - the invoker providing the retry policy
    
    operation - the operation (e.g. "read" or "copy") triggering this call
    
    Returns:
    
    the metadata
    
    Throws:
    
    IOException - if the retry invocation raises one (it shouldn't).
    
    RemoteFileChangedException - if an unexpected version is detected
  - getBucketMetadata
```
@AuditEntryPoint
 @Retries.RetryTranslated
protected software.amazon.awssdk.services.s3.model.HeadBucketResponse getBucketMetadata()
                                                                                                                            throws IOException
```
    Request bucket metadata.
    
    Returns:
    
    the metadata
    
    Throws:
    
    UnknownStoreException - the bucket is absent
    
    IOException - any other problem talking to S3
  - listObjects
```
@Retries.RetryRaw
protected S3ListResult listObjects(S3ListRequest request,
                                                     @Nullable
                                                     org.apache.hadoop.fs.statistics.DurationTrackerFactory trackerFactory)
                                              throws IOException
```
    Initiate a listObjects operation, incrementing metrics in the process. Retry policy: retry untranslated.
    
    Parameters:
    
    request - request to initiate
    
    trackerFactory - duration tracking
    
    Returns:
    
    the results
    
    Throws:
    
    IOException - if the retry invocation raises one (it shouldn't).
  - continueListObjects
```
@Retries.RetryRaw
protected S3ListResult continueListObjects(S3ListRequest request,
                                                             S3ListResult prevResult,
                                                             org.apache.hadoop.fs.statistics.DurationTrackerFactory trackerFactory)
                                                      throws IOException
```
    List the next set of objects. Retry policy: retry untranslated.
    
    Parameters:
    
    request - last list objects request to continue
    
    prevResult - last paged result to continue from
    
    trackerFactory - duration tracking
    
    Returns:
    
    the next result object
    
    Throws:
    
    IOException - none, just there for retryUntranslated.
  - incrementReadOperations
```
public void incrementReadOperations()
```
    Increment read operations.
  - incrementWriteOperations
```
public void incrementWriteOperations()
```
    Increment the write operation counter. This is somewhat inaccurate, as it appears to be invoked more often than needed in progress callbacks.
  - deleteObject
```
@VisibleForTesting
 @Retries.RetryRaw
protected void deleteObject(String key)
                                                           throws software.amazon.awssdk.core.exception.SdkException,
                                                                  IOException
```
    Delete an object. Increments the OBJECT_DELETE_REQUESTS and write operation statistics. This call does not create any mock parent entries. Retry policy: retry untranslated; delete considered idempotent.
    
    Parameters:
    
    key - key to blob to delete.
    
    Throws:
    
    software.amazon.awssdk.core.exception.SdkException - problems working with S3
    
    org.apache.hadoop.fs.InvalidRequestException - if the request was rejected due to a mistaken attempt to delete the root directory.
    
    IOException
  - newPutObjectRequestBuilder
```
public software.amazon.awssdk.services.s3.model.PutObjectRequest.Builder newPutObjectRequestBuilder(String key,
                                                                                                    long length,
                                                                                                    boolean isDirectoryMarker)
```
    Create a putObject request builder. Adds the ACL and metadata
    
    Parameters:
    
    key - key of object
    
    length - length of object to be uploaded
    
    isDirectoryMarker - true if object to be uploaded is a directory marker
    
    Returns:
    
    the request
  - putObject
```
@Retries.OnceRaw
public UploadInfo putObject(software.amazon.awssdk.services.s3.model.PutObjectRequest putObjectRequest,
                                             File file,
                                             ProgressableProgressListener listener)
```
    Start a transfer-manager managed async PUT of an object, incrementing the put requests and put bytes counters. It does not update the other counters, as existing code does that as progress callbacks come in. Byte length is calculated from the file length, or, if there is no file, from the content length of the header. Because the operation is async, any stream supplied in the request must reference data (files, buffers) which stay valid until the upload completes. Retry policy: N/A: the transfer manager is performing the upload. Auditing: must be inside an audit span.
    
    Parameters:
    
    putObjectRequest - the request
    
    file - the file to be uploaded
    
    listener - the progress listener for the request
    
    Returns:
    
    the upload initiated
  - incrementPutStartStatistics
```
public void incrementPutStartStatistics(long bytes)
```
    At the start of a put/multipart upload operation, update the relevant counters.
    
    Parameters:
    
    bytes - bytes in the request.
  - incrementPutCompletedStatistics
```
public void incrementPutCompletedStatistics(boolean success,
                                            long bytes)
```
    At the end of a put/multipart upload operation, update the relevant counters and gauges.
    
    Parameters:
    
    success - did the operation succeed?
    
    bytes - bytes in the request.
  - incrementPutProgressStatistics
```
public void incrementPutProgressStatistics(String key,
                                           long bytes)
```
    Callback for use in progress callbacks from put/multipart upload events. Increments those statistics which are expected to be updated during the ongoing upload operation.
    
    Parameters:
    
    key - key to file that is being written (for logging)
    
    bytes - bytes successfully uploaded.
  - removeKeys
```
@VisibleForTesting
 @Retries.RetryRaw
public void removeKeys(List<software.amazon.awssdk.services.s3.model.ObjectIdentifier> keysToDelete,
                                                             boolean deleteFakeDir)
                                                      throws MultiObjectDeleteException,
                                                             software.amazon.awssdk.awscore.exception.AwsServiceException,
                                                             IOException
```
    Invoke removeKeysS3(List, boolean). If a MultiObjectDeleteException is raised, the relevant statistics are updated.
    
    Parameters:
    
    keysToDelete - collection of keys to delete on the s3-backend. if empty, no request is made of the object store.
    
    deleteFakeDir - indicates whether this is for deleting fake dirs
    
    Throws:
    
    org.apache.hadoop.fs.InvalidRequestException - if the request was rejected due to a mistaken attempt to delete the root directory.
    
    MultiObjectDeleteException - one or more of the keys could not be deleted in a multiple object delete operation.
    
    software.amazon.awssdk.awscore.exception.AwsServiceException - amazon-layer failure.
    
    IOException - other IO Exception.
  - delete
```
@Retries.RetryTranslated
 @AuditEntryPoint
public boolean delete(org.apache.hadoop.fs.Path f,
                                                                 boolean recursive)
                                                          throws IOException
```
    Delete a Path. This operation is at least O(files), with added overheads to enumerate the path. It is also not atomic.
    
    Specified by:
    
    delete in class org.apache.hadoop.fs.FileSystem
    
    Parameters:
    
    f - the path to delete.
    
    recursive - if path is a directory and set to true, the directory is deleted else throws an exception. In case of a file the recursive can be set to either true or false.
    
    Returns:
    
    true if the path existed and then was deleted; false if there was no path in the first place, or the corner cases of root path deletion have surfaced.
    
    Throws:
    
    IOException - due to inability to delete a directory or file.
  - deleteWithoutCloseCheck
```
@VisibleForTesting
protected boolean deleteWithoutCloseCheck(org.apache.hadoop.fs.Path f,
                                                             boolean recursive)
                                                      throws IOException
```
    Same as delete(), except that it does not check if fs is closed.
    
    Parameters:
    
    f - the path to delete.
    
    recursive - if path is a directory and set to true, the directory is deleted else throws an exception. In case of a file the recursive can be set to either true or false.
    
    Returns:
    
    true if the path existed and then was deleted; false if there was no path in the first place, or the corner cases of root path deletion have surfaced.
    
    Throws:
    
    IOException - due to inability to delete a directory or file.
  - maybeCreateFakeParentDirectory
```
@Retries.RetryTranslated
 @VisibleForTesting
protected void maybeCreateFakeParentDirectory(org.apache.hadoop.fs.Path path)
                                                                                    throws IOException,
                                                                                           software.amazon.awssdk.core.exception.SdkException
```
    Create a fake parent directory if required. That is: it parent is not the root path and does not yet exist.
    
    Parameters:
    
    path - whose parent is created if needed.
    
    Throws:
    
    IOException - IO problem
    
    software.amazon.awssdk.core.exception.SdkException
  - listStatusIterator
```
@AuditEntryPoint
public org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.FileStatus> listStatusIterator(org.apache.hadoop.fs.Path p)
                                                                                                         throws FileNotFoundException,
                                                                                                                IOException
```
    Override subclass such that we benefit for async listing done in S3AFileSystem. See Listing#ObjectListingIterator.
    
    Overrides:
    
    listStatusIterator in class org.apache.hadoop.fs.FileSystem
    
    Throws:
    
    FileNotFoundException
    
    IOException
  - listStatus
```
@AuditEntryPoint
public org.apache.hadoop.fs.FileStatus[] listStatus(org.apache.hadoop.fs.Path f)
                                                              throws FileNotFoundException,
                                                                     IOException
```
    List the statuses of the files/directories in the given path if the path is a directory.
    
    Specified by:
    
    listStatus in class org.apache.hadoop.fs.FileSystem
    
    Parameters:
    
    f - given path
    
    Returns:
    
    the statuses of the files/directories in the given patch
    
    Throws:
    
    FileNotFoundException - when the path does not exist; IOException see specific implementation
    
    IOException
  - allowAuthoritative
```
public boolean allowAuthoritative(org.apache.hadoop.fs.Path path)
```
    Is a path to be considered as authoritative? is a store with the supplied path under one of the paths declared as authoritative.
    
    Parameters:
    
    path - path
    
    Returns:
    
    true if the path is auth
  - createListObjectsRequest
```
@VisibleForTesting
public S3ListRequest createListObjectsRequest(String key,
                                                                 String delimiter)
```
    Create a ListObjectsRequest request against this bucket, with the maximum keys returned in a query set by maxKeys.
    
    Parameters:
    
    key - key for request
    
    delimiter - any delimiter
    
    Returns:
    
    the request
  - setWorkingDirectory
```
public void setWorkingDirectory(org.apache.hadoop.fs.Path newDir)
```
    Set the current working directory for the given file system. All relative paths will be resolved relative to it.
    
    Specified by:
    
    setWorkingDirectory in class org.apache.hadoop.fs.FileSystem
    
    Parameters:
    
    newDir - the current working directory.
  - getWorkingDirectory
```
public org.apache.hadoop.fs.Path getWorkingDirectory()
```
    Get the current working directory for the given file system.
    
    Specified by:
    
    getWorkingDirectory in class org.apache.hadoop.fs.FileSystem
    
    Returns:
    
    the directory pathname
  - getUsername
```
public String getUsername()
```
    Get the username of the FS.
    
    Returns:
    
    the short name of the user who instantiated the FS
  - getOwner
```
public org.apache.hadoop.security.UserGroupInformation getOwner()
```
    Get the owner of this FS: who created it?
    
    Returns:
    
    the owner of the FS.
  - mkdirs
```
@AuditEntryPoint
public boolean mkdirs(org.apache.hadoop.fs.Path p,
                                       org.apache.hadoop.fs.permission.FsPermission permission)
                                throws IOException,
                                       org.apache.hadoop.fs.FileAlreadyExistsException
```
    Make the given path and all non-existent parents into directories. Has the semantics of Unix 'mkdir -p'. Existence of the directory hierarchy is not an error. Parent elements are scanned to see if any are a file, except under "MAGIC PATH" paths. There the FS assumes that the destination directory creation did that scan and that paths in job/task attempts are all "well formed"
    
    Specified by:
    
    mkdirs in class org.apache.hadoop.fs.FileSystem
    
    Parameters:
    
    p - path to create
    
    permission - to apply to path
    
    Returns:
    
    true if a directory was created or already existed
    
    Throws:
    
    org.apache.hadoop.fs.FileAlreadyExistsException - there is a file at the path specified or is discovered on one of its ancestors.
    
    IOException - other IO problems
  - createMkdirOperationCallbacks
```
@VisibleForTesting
public MkdirOperation.MkdirCallbacks createMkdirOperationCallbacks()
```
    Override point: create the callbacks for Mkdir. This does not create a new span; caller must be in one.
    
    Returns:
    
    an implementation of the MkdirCallbacks,
  - getContentSummary
```
@Retries.RetryTranslated
 @AuditEntryPoint
public org.apache.hadoop.fs.ContentSummary getContentSummary(org.apache.hadoop.fs.Path f)
                                                                                                 throws IOException
```
    This is a very slow operation against object storage. Execute it as a single span with whatever optimizations have been implemented.
    
    Overrides:
    
    getContentSummary in class org.apache.hadoop.fs.FileSystem
    
    Throws:
    
    IOException
  - createGetContentSummaryCallbacks
```
protected GetContentSummaryOperation.GetContentSummaryCallbacks createGetContentSummaryCallbacks()
```
    Override point: create the callbacks for getContentSummary. This does not create a new span; caller must be in one.
    
    Returns:
    
    an implementation of the GetContentSummaryCallbacksImpl
  - access
```
@AuditEntryPoint
public void access(org.apache.hadoop.fs.Path f,
                                    org.apache.hadoop.fs.permission.FsAction mode)
                             throws org.apache.hadoop.security.AccessControlException,
                                    FileNotFoundException,
                                    IOException
```
    Soft check of access by forwarding to the audit manager and so on to the auditor.
    
    Overrides:
    
    access in class org.apache.hadoop.fs.FileSystem
    
    Throws:
    
    org.apache.hadoop.security.AccessControlException
    
    FileNotFoundException
    
    IOException
  - getFileStatus
```
@AuditEntryPoint
 @Retries.RetryTranslated
public org.apache.hadoop.fs.FileStatus getFileStatus(org.apache.hadoop.fs.Path f)
                                                                                         throws IOException
```
    Return a file status object that represents the path.
    
    Specified by:
    
    getFileStatus in class org.apache.hadoop.fs.FileSystem
    
    Parameters:
    
    f - The path we want information from
    
    Returns:
    
    a FileStatus object
    
    Throws:
    
    FileNotFoundException - when the path does not exist
    
    IOException - on other problems.
  - copyFromLocalFile
```
@AuditEntryPoint
public void copyFromLocalFile(boolean delSrc,
                                               boolean overwrite,
                                               org.apache.hadoop.fs.Path src,
                                               org.apache.hadoop.fs.Path dst)
                                        throws IOException
```
    The src file is on the local disk. Add it to FS at the given dst name. This version doesn't need to create a temporary file to calculate the md5. If Constants.OPTIMIZED_COPY_FROM_LOCAL is set to false, the superclass implementation is used.
    
    Overrides:
    
    copyFromLocalFile in class org.apache.hadoop.fs.FileSystem
    
    Parameters:
    
    delSrc - whether to delete the src
    
    overwrite - whether to overwrite an existing file
    
    src - path
    
    dst - path
    
    Throws:
    
    IOException - IO problem
    
    org.apache.hadoop.fs.FileAlreadyExistsException - the destination file exists and overwrite==false
  - createCopyFromLocalCallbacks
```
protected CopyFromLocalOperation.CopyFromLocalOperationCallbacks createCopyFromLocalCallbacks(AuditSpanS3A span)
                                                                                       throws IOException
```
    Create the CopyFromLocalCallbacks; protected to assist in mocking.
    
    Parameters:
    
    span - audit span.
    
    Returns:
    
    the callbacks
    
    Throws:
    
    IOException - failure to get the local fs.
  - deleteOnExit
```
public boolean deleteOnExit(org.apache.hadoop.fs.Path f)
                     throws IOException
```
    This override bypasses checking for existence.
    
    Overrides:
    
    deleteOnExit in class org.apache.hadoop.fs.FileSystem
    
    Parameters:
    
    f - the path to delete; this may be unqualified.
    
    Returns:
    
    true, always. * @param f the path to delete.
    
    Throws:
    
    IOException - IO failure
  - cancelDeleteOnExit
```
public boolean cancelDeleteOnExit(org.apache.hadoop.fs.Path f)
```
    Cancel the scheduled deletion of the path when the FileSystem is closed.
    
    Overrides:
    
    cancelDeleteOnExit in class org.apache.hadoop.fs.FileSystem
    
    Parameters:
    
    f - the path to cancel deletion
    
    Returns:
    
    true if the path was found in the delete-on-exit list.
  - processDeleteOnExit
```
protected void processDeleteOnExit()
```
    Delete all paths that were marked as delete-on-exit. This recursively deletes all files and directories in the specified paths. It does not check if file exists and filesystem is closed. The time to process this operation is O(paths), with the actual time dependent on the time for existence and deletion operations to complete, successfully or not.
    
    Overrides:
    
    processDeleteOnExit in class org.apache.hadoop.fs.FileSystem
  - close
```
public void close()
           throws IOException
```
    Close the filesystem. This shuts down all transfers.
    
    Specified by:
    
    close in interface Closeable
    
    Specified by:
    
    close in interface AutoCloseable
    
    Overrides:
    
    close in class org.apache.hadoop.fs.FileSystem
    
    Throws:
    
    IOException - IO problem
  - stopAllServices
```
protected void stopAllServices()
```
    Stop all services. This is invoked in close() and during failures of initialize() -make sure that all operations here are robust to failures in both the expected state of this FS and of failures while being stopped.
  - getDelegationTokens
```
@VisibleForTesting
public Optional<S3ADelegationTokens> getDelegationTokens()
```
    Get the delegation token support for this filesystem; not null iff delegation support is enabled.
    
    Returns:
    
    the token support, or an empty option.
  - getCanonicalServiceName
```
public String getCanonicalServiceName()
```
    Return a service name iff delegation tokens are enabled and the token binding is issuing delegation tokens.
    
    Specified by:
    
    getCanonicalServiceName in interface org.apache.hadoop.security.token.DelegationTokenIssuer
    
    Overrides:
    
    getCanonicalServiceName in class org.apache.hadoop.fs.FileSystem
    
    Returns:
    
    the canonical service name or null
  - getDelegationToken
```
@AuditEntryPoint
public org.apache.hadoop.security.token.Token<AbstractS3ATokenIdentifier> getDelegationToken(String renewer)
                                                                                                       throws IOException
```
    Get a delegation token if the FS is set up for them. If the user already has a token, it is returned, even if it has expired.
    
    Specified by:
    
    getDelegationToken in interface org.apache.hadoop.security.token.DelegationTokenIssuer
    
    Overrides:
    
    getDelegationToken in class org.apache.hadoop.fs.FileSystem
    
    Parameters:
    
    renewer - the account name that is allowed to renew the token.
    
    Returns:
    
    the delegation token or null
    
    Throws:
    
    IOException - IO failure
  - getAdditionalTokenIssuers
```
public org.apache.hadoop.security.token.DelegationTokenIssuer[] getAdditionalTokenIssuers()
                                                                                   throws IOException
```
    Ask any DT plugin for any extra token issuers. These do not get told of the encryption secrets and can return any type of token. This allows DT plugins to issue extra tokens for ancillary services.
    
    Specified by:
    
    getAdditionalTokenIssuers in interface org.apache.hadoop.security.token.DelegationTokenIssuer
    
    Overrides:
    
    getAdditionalTokenIssuers in class org.apache.hadoop.fs.FileSystem
    
    Throws:
    
    IOException
  - listAWSPolicyRules
```
@InterfaceAudience.Private
public List<RoleModel.Statement> listAWSPolicyRules(Set<AWSPolicyProvider.AccessLevel> access)
```
    Build the AWS policy for restricted access to the resources needed by this bucket. if needed, and KMS operations.
    
    Specified by:
    
    listAWSPolicyRules in interface AWSPolicyProvider
    
    Parameters:
    
    access - access level desired.
    
    Returns:
    
    a policy for use in roles
  - getDefaultBlockSize
```
public long getDefaultBlockSize()
```
    Return the number of bytes that large input files should be optimally be split into to minimize I/O time.
    
    Overrides:
    
    getDefaultBlockSize in class org.apache.hadoop.fs.FileSystem
  - getDirectoryMarkerPolicy
```
public DirectoryPolicy getDirectoryMarkerPolicy()
```
    Get the directory marker policy of this filesystem.
    
    Returns:
    
    the marker policy.
  - toString
```
public String toString()
```
    Overrides:
    
    toString in class Object
  - getPartitionSize
```
public long getPartitionSize()
```
    Get the partition size for multipart operations.
    
    Returns:
    
    the value as set during initialization
  - getMultiPartThreshold
```
public long getMultiPartThreshold()
```
    Get the threshold for multipart files.
    
    Returns:
    
    the value as set during initialization
  - isMagicCommitEnabled
```
public boolean isMagicCommitEnabled()
```
    Is magic commit enabled?
    
    Returns:
    
    true if magic commit support is turned on.
  - isMagicCommitPath
```
public boolean isMagicCommitPath(org.apache.hadoop.fs.Path path)
```
    Predicate: is a path a magic commit path? True if magic commit is enabled and the path qualifies as special, and is not a a .pending or .pendingset file,
    
    Parameters:
    
    path - path to examine
    
    Returns:
    
    true if writing a file to the path triggers a "magic" write.
  - globStatus
```
public org.apache.hadoop.fs.FileStatus[] globStatus(org.apache.hadoop.fs.Path pathPattern)
                                             throws IOException
```
    Increments the statistic Statistic.INVOCATION_GLOB_STATUS. Override superclass so as to disable symlink resolution as symlinks are not supported by S3A.
    
    Overrides:
    
    globStatus in class org.apache.hadoop.fs.FileSystem
    
    Throws:
    
    IOException
  - globStatus
```
@AuditEntryPoint
public org.apache.hadoop.fs.FileStatus[] globStatus(org.apache.hadoop.fs.Path pathPattern,
                                                                     org.apache.hadoop.fs.PathFilter filter)
                                                              throws IOException
```
    Increments the statistic Statistic.INVOCATION_GLOB_STATUS. Override superclass so as to disable symlink resolution as symlinks are not supported by S3A. Although an AuditEntryPoint, the globber itself will talk do the filesystem through the filesystem API, so its operations will all appear part of separate operations.
    
    Overrides:
    
    globStatus in class org.apache.hadoop.fs.FileSystem
    
    Throws:
    
    IOException
  - exists
```
@AuditEntryPoint
public boolean exists(org.apache.hadoop.fs.Path f)
                                throws IOException
```
    Override superclass so as to add statistic collection.
    
    Overrides:
    
    exists in class org.apache.hadoop.fs.FileSystem
    
    Throws:
    
    IOException
  - isDirectory
```
@AuditEntryPoint
public boolean isDirectory(org.apache.hadoop.fs.Path f)
                                     throws IOException
```
    Optimized probe for a path referencing a dir. Even though it is optimized to a single HEAD, applications should not over-use this method...it is all too common.
    
    Overrides:
    
    isDirectory in class org.apache.hadoop.fs.FileSystem
    
    Throws:
    
    IOException
  - isFile
```
@AuditEntryPoint
public boolean isFile(org.apache.hadoop.fs.Path f)
                                throws IOException
```
    Optimized probe for a path referencing a file. Even though it is optimized to a single HEAD, applications should not over-use this method...it is all too common.
    
    Overrides:
    
    isFile in class org.apache.hadoop.fs.FileSystem
    
    Throws:
    
    IOException
  - getFileChecksum
```
@Retries.RetryTranslated
 @AuditEntryPoint
public org.apache.hadoop.fs.store.EtagChecksum getFileChecksum(org.apache.hadoop.fs.Path f,
                                                                                                          long length)
                                                                                                   throws IOException
```
    When enabled, get the etag of a object at the path via HEAD request and return it as a checksum object.
    1. If a tag has not changed, consider the object unchanged.
    2. Two tags being different does not imply the data is different.
    Different S3 implementations may offer different guarantees. This check is (currently) only made if Constants.ETAG_CHECKSUM_ENABLED is set; turning it on has caused problems with Distcp (HADOOP-15273).
    Overrides:
    
    getFileChecksum in class org.apache.hadoop.fs.FileSystem
    
    Parameters:
    
    f - The file path
    
    length - The length of the file range for checksum calculation
    
    Returns:
    
    The EtagChecksum or null if checksums are not enabled or supported.
    
    Throws:
    
    IOException - IO failure
    
    See Also:
    
    Common Response Headers
  - getXAttr
```
@AuditEntryPoint
public byte[] getXAttr(org.apache.hadoop.fs.Path path,
                                        String name)
                                 throws IOException
```
    Overrides:
    
    getXAttr in class org.apache.hadoop.fs.FileSystem
    
    Throws:
    
    IOException
  - getXAttrs
```
@AuditEntryPoint
public Map<String,byte[]> getXAttrs(org.apache.hadoop.fs.Path path)
                                              throws IOException
```
    Overrides:
    
    getXAttrs in class org.apache.hadoop.fs.FileSystem
    
    Throws:
    
    IOException
  - getXAttrs
```
@AuditEntryPoint
public Map<String,byte[]> getXAttrs(org.apache.hadoop.fs.Path path,
                                                     List<String> names)
                                              throws IOException
```
    Overrides:
    
    getXAttrs in class org.apache.hadoop.fs.FileSystem
    
    Throws:
    
    IOException
  - listXAttrs
```
@AuditEntryPoint
public List<String> listXAttrs(org.apache.hadoop.fs.Path path)
                                         throws IOException
```
    Overrides:
    
    listXAttrs in class org.apache.hadoop.fs.FileSystem
    
    Throws:
    
    IOException
  - createHeaderProcessingCallbacks
```
protected HeaderProcessing.HeaderProcessingCallbacks createHeaderProcessingCallbacks()
```
    Create the callbacks.
    
    Returns:
    
    An implementation of the header processing callbacks.
  - listFiles
```
@Retries.RetryTranslated
 @AuditEntryPoint
public org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus> listFiles(org.apache.hadoop.fs.Path f,
                                                                                                                                        boolean recursive)
                                                                                                                                 throws FileNotFoundException,
                                                                                                                                        IOException
```
    . This implementation is optimized for S3, which can do a bulk listing off all entries under a path in one single operation. Thus there is no need to recursively walk the directory tree. Instead a ListObjectsRequest is created requesting a (windowed) listing of all entries under the given path. This is used to construct an ObjectListingIterator instance, iteratively returning the sequence of lists of elements under the path. This is then iterated over in a FileStatusListingIterator, which generates S3AFileStatus instances, one per listing entry. These are then translated into LocatedFileStatus instances. This is essentially a nested and wrapped set of iterators, with some generator classes.
    
    Overrides:
    
    listFiles in class org.apache.hadoop.fs.FileSystem
    
    Parameters:
    
    f - a path
    
    recursive - if the subdirectories need to be traversed recursively
    
    Returns:
    
    an iterator that traverses statuses of the files/directories in the given path
    
    Throws:
    
    FileNotFoundException - if path does not exist
    
    IOException - if any I/O error occurred
  - listFilesAndEmptyDirectories
```
@InterfaceAudience.Private
 @Retries.RetryTranslated
 @AuditEntryPoint
public org.apache.hadoop.fs.RemoteIterator<S3ALocatedFileStatus> listFilesAndEmptyDirectories(org.apache.hadoop.fs.Path f,
                                                                                                                                                                     boolean recursive)
                                                                                                                                                              throws IOException
```
    Recursive List of files and empty directories.
    
    Parameters:
    
    f - path to list from
    
    recursive - recursive?
    
    Returns:
    
    an iterator.
    
    Throws:
    
    IOException - failure
  - listLocatedStatus
```
public org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus> listLocatedStatus(org.apache.hadoop.fs.Path f)
                                                                                              throws FileNotFoundException,
                                                                                                     IOException
```
    Override superclass so as to add statistic collection.
    
    Overrides:
    
    listLocatedStatus in class org.apache.hadoop.fs.FileSystem
    
    Throws:
    
    FileNotFoundException
    
    IOException
  - listLocatedStatus
```
@Retries.OnceTranslated(value="s3guard not retrying")
 @AuditEntryPoint
public org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus> listLocatedStatus(org.apache.hadoop.fs.Path f,
                                                                                                                                                                             org.apache.hadoop.fs.PathFilter filter)
                                                                                                                                                                      throws FileNotFoundException,
                                                                                                                                                                             IOException
```
    . S3 Optimized directory listing. The initial operation performs the first bulk listing; extra listings will take place when all the current set of results are used up.
    
    Overrides:
    
    listLocatedStatus in class org.apache.hadoop.fs.FileSystem
    
    Parameters:
    
    f - a path
    
    filter - a path filter
    
    Returns:
    
    an iterator that traverses statuses of the files/directories in the given path
    
    Throws:
    
    FileNotFoundException - if path does not exist
    
    IOException - if any I/O error occurred
  - listUploads
```
@InterfaceAudience.Private
 @Retries.RetryTranslated
 @AuditEntryPoint
public org.apache.hadoop.fs.RemoteIterator<software.amazon.awssdk.services.s3.model.MultipartUpload> listUploads(@Nullable
                                                                                                                                                                                        String prefix)
                                                                                                                                                                                 throws IOException
```
    List any pending multipart uploads whose keys begin with prefix, using an iterator that can handle an unlimited number of entries. See listMultipartUploads(String) for a non-iterator version of this.
    
    Parameters:
    
    prefix - optional key prefix to search
    
    Returns:
    
    Iterator over multipart uploads.
    
    Throws:
    
    IOException - on failure
  - listUploadsUnderPrefix
```
@InterfaceAudience.Private
 @Retries.RetryTranslated
public org.apache.hadoop.fs.RemoteIterator<software.amazon.awssdk.services.s3.model.MultipartUpload> listUploadsUnderPrefix(StoreContext storeContext,
                                                                                                                                                                                 @Nullable
                                                                                                                                                                                 String prefix)
                                                                                                                                                                          throws IOException
```
    List any pending multipart uploads whose keys begin with prefix, using an iterator that can handle an unlimited number of entries. See listMultipartUploads(String) for a non-iterator version of this.
    
    Parameters:
    
    storeContext - store conext.
    
    prefix - optional key prefix to search
    
    Returns:
    
    Iterator over multipart uploads.
    
    Throws:
    
    IOException - on failure
  - listMultipartUploads
```
@InterfaceAudience.Private
 @Retries.RetryTranslated
public List<software.amazon.awssdk.services.s3.model.MultipartUpload> listMultipartUploads(String prefix)
                                                                                                                                         throws IOException
```
    Listing all multipart uploads; limited to the first few hundred. See listUploads(String) for an iterator-based version that does not limit the number of entries returned. Retry policy: retry, translated.
    
    Parameters:
    
    prefix - prefix to scan for, "" for none
    
    Returns:
    
    a listing of multipart uploads.
    
    Throws:
    
    IOException - IO failure, including any uprated SdkException
  - abortMultipartUpload
```
@Retries.OnceTranslated
public void abortMultipartUpload(String destKey,
                                                         String uploadId)
                                                  throws IOException
```
    Abort a multipart upload. Retry policy: none.
    
    Parameters:
    
    destKey - destination key
    
    uploadId - Upload ID
    
    Throws:
    
    IOException - IO failure, including any uprated SdkException
  - abortMultipartUpload
```
@Retries.OnceTranslated
public void abortMultipartUpload(software.amazon.awssdk.services.s3.model.MultipartUpload upload)
                                                  throws IOException
```
    Abort a multipart upload. Retry policy: none.
    
    Parameters:
    
    upload - the listed upload to abort.
    
    Throws:
    
    IOException - IO failure, including any uprated SdkException
  - newCommitterStatistics
```
public CommitterStatistics newCommitterStatistics()
```
    Create a new instance of the committer statistics.
    
    Returns:
    
    a new committer statistics instance
  - hasPathCapability
```
public boolean hasPathCapability(org.apache.hadoop.fs.Path path,
                                 String capability)
                          throws IOException
```
    Specified by:
    
    hasPathCapability in interface org.apache.hadoop.fs.PathCapabilities
    
    Overrides:
    
    hasPathCapability in class org.apache.hadoop.fs.FileSystem
    
    Throws:
    
    IOException
  - hasCapability
```
@Deprecated
public boolean hasCapability(String capability)
```
    Deprecated.
    
    Return the capabilities of this filesystem instance. This has been supplanted by hasPathCapability(Path, String).
    
    Specified by:
    
    hasCapability in interface org.apache.hadoop.fs.StreamCapabilities
    
    Parameters:
    
    capability - string to query the stream support for.
    
    Returns:
    
    whether the FS instance has the capability.
  - shareCredentials
```
public AWSCredentialProviderList shareCredentials(String purpose)
```
    Get a shared copy of the AWS credentials, with its reference counter updated. Caller is required to call close() on this after they have finished using it.
    
    Parameters:
    
    purpose - what is this for? This is initially for logging
    
    Returns:
    
    a reference to shared credentials.
  - openFileWithOptions
```
@Retries.RetryTranslated
 @AuditEntryPoint
public CompletableFuture<org.apache.hadoop.fs.FSDataInputStream> openFileWithOptions(org.apache.hadoop.fs.Path rawPath,
                                                                                                                                org.apache.hadoop.fs.impl.OpenFileParameters parameters)
                                                                                                                         throws IOException
```
    Initiate the open() operation. This is invoked from both the FileSystem and FileContext APIs. It's declared as an audit entry point but the span creation is pushed down into the open operation s it ultimately calls.
    
    Overrides:
    
    openFileWithOptions in class org.apache.hadoop.fs.FileSystem
    
    Parameters:
    
    rawPath - path to the file
    
    parameters - open file parameters from the builder.
    
    Returns:
    
    a future which will evaluate to the opened file.
    
    Throws:
    
    IOException - failure to resolve the link.
    
    IllegalArgumentException - unknown mandatory key
  - createMultipartUploader
```
@AuditEntryPoint
public S3AMultipartUploaderBuilder createMultipartUploader(org.apache.hadoop.fs.Path basePath)
                                                                     throws IOException
```
    Overrides:
    
    createMultipartUploader in class org.apache.hadoop.fs.FileSystem
    
    Throws:
    
    IOException
  - createStoreContext
```
@InterfaceAudience.Private
public StoreContext createStoreContext()
```
    Build an immutable store context. If called while the FS is being initialized, some of the context will be incomplete. new store context instances should be created as appropriate.
    
    Returns:
    
    the store context of this FS.
  - createMarkerToolOperations
```
@AuditEntryPoint
 @InterfaceAudience.Private
public MarkerToolOperations createMarkerToolOperations(String target)
                                                                                             throws IOException
```
    Create a marker tools operations binding for this store. Auditing:
    
    Parameters:
    
    target - target path
    
    Returns:
    
    callbacks for operations.
    
    Throws:
    
    IOException - if raised during span creation
  - initializeClass
```
@InterfaceAudience.Private
public static void initializeClass()
```
    This is purely for testing, as it force initializes all static initializers. See HADOOP-17385 for details.
  - isCSEEnabled
```
public boolean isCSEEnabled()
```
    a method to know if Client side encryption is enabled or not.
    
    Returns:
    
    a boolean stating if CSE is enabled.
  - isMultipartUploadEnabled
```
public boolean isMultipartUploadEnabled()
```

Class S3AFileSystem

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.fs.FileSystem

Nested classes/interfaces inherited from interface org.apache.hadoop.fs.StreamCapabilities

Nested classes/interfaces inherited from interface org.apache.hadoop.fs.s3a.auth.delegation.AWSPolicyProvider

Field Summary

Fields inherited from class org.apache.hadoop.fs.FileSystem

Fields inherited from interface org.apache.hadoop.fs.StreamCapabilities

Fields inherited from interface org.apache.hadoop.security.token.DelegationTokenIssuer

Constructor Summary

Method Summary

Methods inherited from class org.apache.hadoop.fs.FileSystem

Methods inherited from class org.apache.hadoop.conf.Configured

Methods inherited from class java.lang.Object

Methods inherited from interface org.apache.hadoop.security.token.DelegationTokenIssuer

Field Detail

DEFAULT_BLOCKSIZE

LOG

Constructor Detail

S3AFileSystem

Method Detail

initialize

initializeStatisticsBinding

createStorageStatistics

verifyBucketExists

getInstrumentation

getFsStatistics

getListing

initializeAuditService

getAuditManager

getAuditor

getActiveAuditSpan

getAuditSpanSource

createSpan

createRequestFactory

getRequestFactory

createDelegationOperations

setEncryptionSecrets

getEncryptionSecrets

abortOutstandingMultipartUploads

getScheme

getUri

setUri

getCanonicalUri

getDefaultPort

setAmazonS3Client

getBucketLocation

createS3AInternals

getS3AInternals

getInputPolicy

getChangeDetectionPolicy

getS3EncryptionAlgorithm

getBucket

setBucket

setInputPolicy

pathToKey

maybeAddTrailingSlash

keyToQualifiedPath

makeQualified

qualify

checkPath

canonicalizeUri

open

createReadContext

create

getWriteOperationHelper

createWriteOperationHelper

createFile

createNonRecursive

append

rename

getFsDelegationToken

getObjectMetadata

entryPoint

entryPoint

incrementStatistic

incrementStatistic

decrementGauge

incrementGauge

operationRetried