Index (Apache Hadoop Amazon Web Services support 3.2.1 API)

A B C D E F G H I J K L M N O P Q R S T U V W _

A

abort() - Method in class org.apache.hadoop.fs.s3a.InconsistentS3Object.InconsistentS3InputStream
abort(Path, UploadHandle) - Method in class org.apache.hadoop.fs.s3a.S3AMultipartUploader
abortAllSinglePendingCommits(Path, boolean) - Method in class org.apache.hadoop.fs.s3a.commit.CommitOperations: Enumerate all pending files in a dir/tree, abort.
aborted - Variable in class org.apache.hadoop.fs.s3a.S3AInstrumentation.InputStreamStatistics
abortJob(JobContext, JobStatus.State) - Method in class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter
abortJobInternal(JobContext, boolean) - Method in class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter: The internal job abort operation; can be overridden in tests.
abortJobInternal(JobContext, boolean) - Method in class org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter
abortMultipartCommit(String, String) - Method in class org.apache.hadoop.fs.s3a.commit.CommitOperations: Create an AbortMultipartUpload request and POST it to S3, incrementing statistics afterwards.
abortMultipartCommit(String, String) - Method in class org.apache.hadoop.fs.s3a.WriteOperationHelper: Abort a multipart commit operation.
abortMultipartUpload(String, String, Invoker.Retried) - Method in class org.apache.hadoop.fs.s3a.WriteOperationHelper: Abort a multipart upload operation.
abortMultipartUpload(MultipartUpload) - Method in class org.apache.hadoop.fs.s3a.WriteOperationHelper: Abort a multipart commit operation.
abortMultipartUploadsUnderPath(String) - Method in class org.apache.hadoop.fs.s3a.WriteOperationHelper: Abort multipart uploads under a path: limited to the first few hundred.
abortOutstandingMultipartUploads(long) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Abort all outstanding MPUs older than a given age.
abortPendingUploads(JobContext, List<SinglePendingCommit>, boolean) - Method in class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter: Abort all pending uploads in the list.
abortPendingUploadsInCleanup(boolean) - Method in class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter: Abort all pending uploads to the destination directory during job cleanup operations.
abortPendingUploadsInCleanup(boolean) - Method in class org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter
abortPendingUploadsUnderPath(Path) - Method in class org.apache.hadoop.fs.s3a.commit.CommitOperations: Abort all pending uploads to the destination FS under a path.
abortSingleCommit(SinglePendingCommit) - Method in class org.apache.hadoop.fs.s3a.commit.CommitOperations: Abort the multipart commit supplied.
abortTask(TaskAttemptContext) - Method in class org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter: Abort a task.
abortTask(TaskAttemptContext) - Method in class org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter: Abort the task.
abortWith(Tasks.Task<I, ?>) - Method in class org.apache.hadoop.fs.s3a.commit.Tasks.Builder
aboutToComplete(String, List<PartETag>, long) - Method in class org.apache.hadoop.fs.s3a.commit.magic.MagicCommitTracker: Complete operation: generate the final commit data, put it.
aboutToComplete(String, List<PartETag>, long) - Method in class org.apache.hadoop.fs.s3a.commit.PutTracker: Callback when the upload is is about to complete.
AbstractS3ACommitter - Class in org.apache.hadoop.fs.s3a.commit: Abstract base class for S3A committers; allows for any commonality between different architectures.
AbstractS3ACommitter(Path, TaskAttemptContext) - Constructor for class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter: Create a committer.
AbstractS3ACommitterFactory - Class in org.apache.hadoop.fs.s3a.commit: Dynamically create the output committer based on subclass type and settings.
AbstractS3ACommitterFactory() - Constructor for class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitterFactory
ACCEPT_ALL - Static variable in class org.apache.hadoop.fs.s3a.S3AUtils: A Path filter which accepts all filenames.
ACCESS_KEY - Static variable in class org.apache.hadoop.fs.s3a.Constants
action - Variable in class org.apache.hadoop.fs.s3a.auth.RoleModel.Statement
add(AWSCredentialsProvider) - Method in class org.apache.hadoop.fs.s3a.AWSCredentialProviderList: Add a new provider.
add(SinglePendingCommit) - Method in class org.apache.hadoop.fs.s3a.commit.files.PendingSet: Add a commit.
addActions(String...) - Method in class org.apache.hadoop.fs.s3a.auth.RoleModel.Statement
addAgeOptions() - Method in class org.apache.hadoop.fs.s3a.s3guard.S3GuardTool
addAncestors(MetadataStore, Path, String) - Static method in class org.apache.hadoop.fs.s3a.s3guard.S3Guard
addDiagnostic(String, String) - Method in class org.apache.hadoop.fs.s3a.commit.files.SuccessData: Add a diagnostics entry.
addFileSystemStatistics(Map<String, Long>) - Method in class org.apache.hadoop.fs.s3a.commit.CommitOperations: Add the filesystem statistics to the map; overwriting anything with the same name.
addMoveAncestors(MetadataStore, Collection<Path>, Collection<PathMetadata>, Path, Path, Path, String) - Static method in class org.apache.hadoop.fs.s3a.s3guard.S3Guard: Helper method that records the move of all ancestors of a path.
addMoveDir(MetadataStore, Collection<Path>, Collection<PathMetadata>, Path, Path, String) - Static method in class org.apache.hadoop.fs.s3a.s3guard.S3Guard: Helper function that records the move of directory paths, adding resulting metadata to the supplied lists.
addMoveFile(MetadataStore, Collection<Path>, Collection<PathMetadata>, Path, Path, long, long, String) - Static method in class org.apache.hadoop.fs.s3a.s3guard.S3Guard: Like S3Guard.addMoveDir(MetadataStore, Collection, Collection, Path, Path, String) (), but for files.
addResources(String...) - Method in class org.apache.hadoop.fs.s3a.auth.RoleModel.Statement
addUUID(String, String) - Static method in class org.apache.hadoop.fs.s3a.commit.staging.Paths: Insert the UUID to a path if it is not there already.
addValueToQuantiles(Statistic, long) - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation: Add a value to a quantiles statistic.
AGE_OPTIONS_USAGE - Static variable in class org.apache.hadoop.fs.s3a.s3guard.S3GuardTool
ALL_DDB_TABLES - Static variable in class org.apache.hadoop.fs.s3a.auth.RolePolicies: All DynamoDB tables: "arn:aws:dynamodb:*".
ALLOW_S3_AND_SGUARD - Static variable in class org.apache.hadoop.fs.s3a.auth.RolePolicies: Policy for all S3 and S3Guard operations, and SSE-KMS.
AnonymousAWSCredentialsProvider - Class in org.apache.hadoop.fs.s3a: AnonymousAWSCredentialsProvider supports anonymous access to AWS services through the AWS SDK.
AnonymousAWSCredentialsProvider() - Constructor for class org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider
append(Path, int, Progressable) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Append to an existing file (optional operation).
append(Path, int, Progressable) - Method in class org.apache.hadoop.fs.s3native.NativeS3FileSystem: Deprecated.
applyLocatedFiles(RemoteIterator<LocatedFileStatus>, S3AUtils.CallOnLocatedFileStatus) - Static method in class org.apache.hadoop.fs.s3a.S3AUtils: Apply an operation to every LocatedFileStatus in a remote iterator.
applyRevisionConstraint(GetObjectRequest, String) - Method in class org.apache.hadoop.fs.s3a.impl.ChangeDetectionPolicy: Applies the given revisionId as a server-side qualification on the GetObjectRequest.
assertQualified(Path) - Static method in class org.apache.hadoop.fs.s3a.s3guard.S3Guard: Assert that the path is qualified with a host and scheme.
assertQualified(Path...) - Static method in class org.apache.hadoop.fs.s3a.s3guard.S3Guard: Assert that all paths are valid.
ASSUMED_ROLE_ARN - Static variable in class org.apache.hadoop.fs.s3a.Constants: AWS Role to request.
ASSUMED_ROLE_CREDENTIALS_DEFAULT - Static variable in class org.apache.hadoop.fs.s3a.Constants
ASSUMED_ROLE_CREDENTIALS_PROVIDER - Static variable in class org.apache.hadoop.fs.s3a.Constants: list of providers to authenticate for the assumed role.
ASSUMED_ROLE_POLICY - Static variable in class org.apache.hadoop.fs.s3a.Constants: JSON policy containing the policy to apply to the role.
ASSUMED_ROLE_SESSION_DURATION - Static variable in class org.apache.hadoop.fs.s3a.Constants: Duration of assumed roles before a refresh is attempted.
ASSUMED_ROLE_SESSION_DURATION_DEFAULT - Static variable in class org.apache.hadoop.fs.s3a.Constants: Default duration of an assumed role.
ASSUMED_ROLE_SESSION_NAME - Static variable in class org.apache.hadoop.fs.s3a.Constants: Session name for the assumed role, must be valid characters according to the AWS APIs.
ASSUMED_ROLE_STS_ENDPOINT - Static variable in class org.apache.hadoop.fs.s3a.Constants: Security Token Service Endpoint.
ASSUMED_ROLE_STS_ENDPOINT_REGION - Static variable in class org.apache.hadoop.fs.s3a.Constants: Region for the STS endpoint; only relevant if the endpoint is set.
ASSUMED_ROLE_STS_ENDPOINT_REGION_DEFAULT - Static variable in class org.apache.hadoop.fs.s3a.Constants: Default value for the STS endpoint region; needed for v4 signing.
AssumedRoleCredentialProvider - Class in org.apache.hadoop.fs.s3a.auth: Support IAM Assumed roles by instantiating an instance of STSAssumeRoleSessionCredentialsProvider from configuration properties, including wiring up the inner authenticator, and, unless overridden, creating a session name from the current user.
AssumedRoleCredentialProvider(URI, Configuration) - Constructor for class org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider: Instantiate.
available() - Method in class org.apache.hadoop.fs.s3a.InconsistentS3Object.InconsistentS3InputStream
available() - Method in class org.apache.hadoop.fs.s3a.S3AInputStream
AWS_CREDENTIALS_PROVIDER - Static variable in class org.apache.hadoop.fs.s3a.Constants
AWSBadRequestException - Exception in org.apache.hadoop.fs.s3a: A 400 "Bad Request" exception was received.
AWSBadRequestException(String, AmazonServiceException) - Constructor for exception org.apache.hadoop.fs.s3a.AWSBadRequestException: Instantiate.
AWSClientIOException - Exception in org.apache.hadoop.fs.s3a: IOException equivalent of an AmazonClientException.
AWSClientIOException(String, SdkBaseException) - Constructor for exception org.apache.hadoop.fs.s3a.AWSClientIOException
AWSCredentialProviderList - Class in org.apache.hadoop.fs.s3a: A list of providers.
AWSCredentialProviderList() - Constructor for class org.apache.hadoop.fs.s3a.AWSCredentialProviderList: Empty instance.
AWSCredentialProviderList(Collection<AWSCredentialsProvider>) - Constructor for class org.apache.hadoop.fs.s3a.AWSCredentialProviderList: Create with an initial list of providers.
AWSNoResponseException - Exception in org.apache.hadoop.fs.s3a: Status code 443, no response from server.
AWSNoResponseException(String, AmazonServiceException) - Constructor for exception org.apache.hadoop.fs.s3a.AWSNoResponseException
AWSRedirectException - Exception in org.apache.hadoop.fs.s3a: Request is redirected.
AWSRedirectException(String, AmazonServiceException) - Constructor for exception org.apache.hadoop.fs.s3a.AWSRedirectException: Instantiate.
AWSS3IOException - Exception in org.apache.hadoop.fs.s3a: Wrap a AmazonS3Exception as an IOE, relaying all getters.
AWSS3IOException(String, AmazonS3Exception) - Constructor for exception org.apache.hadoop.fs.s3a.AWSS3IOException: Instantiate.
AWSServiceIOException - Exception in org.apache.hadoop.fs.s3a: A specific exception from AWS operations.
AWSServiceIOException(String, AmazonServiceException) - Constructor for exception org.apache.hadoop.fs.s3a.AWSServiceIOException: Instantiate.
AWSServiceThrottledException - Exception in org.apache.hadoop.fs.s3a: Exception raised when a service was throttled.
AWSServiceThrottledException(String, AmazonServiceException) - Constructor for exception org.apache.hadoop.fs.s3a.AWSServiceThrottledException: Instantiate.
AWSStatus500Exception - Exception in org.apache.hadoop.fs.s3a: A 500 response came back from a service.
AWSStatus500Exception(String, AmazonServiceException) - Constructor for exception org.apache.hadoop.fs.s3a.AWSStatus500Exception

B

backwardSeekOperations - Variable in class org.apache.hadoop.fs.s3a.S3AInstrumentation.InputStreamStatistics
badState(String, Object...) - Static method in class org.apache.hadoop.fs.s3a.s3guard.S3GuardTool: Build the exception to raise on a bad store/bucket state.
BASE - Static variable in class org.apache.hadoop.fs.s3a.commit.CommitConstants: Marker of the start of a directory tree for calculating the final path names: "__base".
basePathChildren(List<String>) - Static method in class org.apache.hadoop.fs.s3a.commit.MagicCommitPaths: Get any child path elements under any __base path, or an empty list if there is either: no __base path element, or no child entries under it.
bindCommitData(List<PartETag>) - Method in class org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit: Set the commit data.
blocksActivelyAllocated() - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation.OutputStreamStatistics: Get counters of blocks actively allocated; my be inaccurate if the numbers change during the (non-synchronized) calculation.
blocksAllocated() - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation.OutputStreamStatistics
blocksReleased() - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation.OutputStreamStatistics
BUCKET_RESOURCE_F - Static variable in class org.apache.hadoop.fs.s3a.auth.RoleModel
BUFFER_DIR - Static variable in class org.apache.hadoop.fs.s3a.Constants
builder(Configuration, String, AWSCredentialsProvider, String, String) - Static method in class org.apache.hadoop.fs.s3a.auth.STSClientFactory: Create the builder ready for any final configuration options.
buildFSURI(URI) - Static method in class org.apache.hadoop.fs.s3native.S3xLoginHelper: Build the filesystem URI.
buildThreadPool(JobContext) - Method in class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter: Returns an ExecutorService for parallel tasks.
bytesBackwardsOnSeek - Variable in class org.apache.hadoop.fs.s3a.S3AInstrumentation.InputStreamStatistics
bytesDiscardedInAbort - Variable in class org.apache.hadoop.fs.s3a.S3AInstrumentation.InputStreamStatistics
bytesRead - Variable in class org.apache.hadoop.fs.s3a.S3AInstrumentation.InputStreamStatistics
bytesRead(long) - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation.InputStreamStatistics: Increment the bytes read counter by the number of bytes; no-op if the argument is negative.
bytesReadInClose - Variable in class org.apache.hadoop.fs.s3a.S3AInstrumentation.InputStreamStatistics
bytesSkippedOnSeek - Variable in class org.apache.hadoop.fs.s3a.S3AInstrumentation.InputStreamStatistics

C

call(LocatedFileStatus) - Method in interface org.apache.hadoop.fs.s3a.S3AUtils.CallOnLocatedFileStatus
call(LocatedFileStatus) - Method in interface org.apache.hadoop.fs.s3a.S3AUtils.LocatedFileStatusMap
CANNED_ACL - Static variable in class org.apache.hadoop.fs.s3a.Constants
canonicalizeUri(URI) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem
canonicalizeUri(URI, int) - Static method in class org.apache.hadoop.fs.s3native.S3xLoginHelper: Canonicalize the given URI.
castFrom(AmazonS3) - Static method in class org.apache.hadoop.fs.s3a.InconsistentAmazonS3Client: Convenience function for test code to cast from supertype.
CHANGE_DETECT_MODE - Static variable in class org.apache.hadoop.fs.s3a.Constants: Mode to run change detection in.
CHANGE_DETECT_MODE_CLIENT - Static variable in class org.apache.hadoop.fs.s3a.Constants: Change is detected on the client side by comparing the returned id with the expected id.
CHANGE_DETECT_MODE_DEFAULT - Static variable in class org.apache.hadoop.fs.s3a.Constants: Default change detection mode: server.
CHANGE_DETECT_MODE_NONE - Static variable in class org.apache.hadoop.fs.s3a.Constants: Change detection is turned off.
CHANGE_DETECT_MODE_SERVER - Static variable in class org.apache.hadoop.fs.s3a.Constants: Change is detected by passing the expected value in the GetObject request.
CHANGE_DETECT_MODE_WARN - Static variable in class org.apache.hadoop.fs.s3a.Constants: Change is detected on the client side by comparing the returned id with the expected id.
CHANGE_DETECT_REQUIRE_VERSION - Static variable in class org.apache.hadoop.fs.s3a.Constants: If true, raises a RemoteFileChangedException exception when S3 doesn't provide the attribute defined by fs.s3a.change.detection.source.
CHANGE_DETECT_REQUIRE_VERSION_DEFAULT - Static variable in class org.apache.hadoop.fs.s3a.Constants: Default change detection require version: true.
CHANGE_DETECT_SOURCE - Static variable in class org.apache.hadoop.fs.s3a.Constants: Where to get the value to use in change detection.
CHANGE_DETECT_SOURCE_DEFAULT - Static variable in class org.apache.hadoop.fs.s3a.Constants: Default change detection mechanism: eTag.
CHANGE_DETECT_SOURCE_ETAG - Static variable in class org.apache.hadoop.fs.s3a.Constants: eTag as the change detection mechanism.
CHANGE_DETECT_SOURCE_VERSION_ID - Static variable in class org.apache.hadoop.fs.s3a.Constants: Object versionId as the change detection mechanism.
CHANGE_DETECTED - Static variable in class org.apache.hadoop.fs.s3a.impl.ChangeDetectionPolicy
CHANGE_REPORTED_BY_S3 - Static variable in class org.apache.hadoop.fs.s3a.impl.ChangeTracker
ChangeDetectionPolicy - Class in org.apache.hadoop.fs.s3a.impl: Object change detection policy.
ChangeDetectionPolicy(ChangeDetectionPolicy.Mode, boolean) - Constructor for class org.apache.hadoop.fs.s3a.impl.ChangeDetectionPolicy
ChangeDetectionPolicy.Mode - Enum in org.apache.hadoop.fs.s3a.impl: What to do when change is detected.
ChangeDetectionPolicy.Source - Enum in org.apache.hadoop.fs.s3a.impl: The S3 object attribute used to detect change.
ChangeTracker - Class in org.apache.hadoop.fs.s3a.impl: Change tracking for input streams: the revision ID/etag the previous request is recorded and when the next request comes in, it is compared.
ChangeTracker(String, ChangeDetectionPolicy, AtomicLong) - Constructor for class org.apache.hadoop.fs.s3a.impl.ChangeTracker: Create a change tracker.
checkNotEmpty() - Method in class org.apache.hadoop.fs.s3a.AWSCredentialProviderList: Verify that the provider list is not empty.
checkPath(Path) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Check that a Path belongs to this FileSystem.
checkPath(Configuration, URI, Path, int) - Static method in class org.apache.hadoop.fs.s3native.S3xLoginHelper: Check the path, ignoring authentication details.
CLASSNAME - Static variable in class org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitterFactory: Name of this class: "org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitterFactory".
CLASSNAME - Static variable in class org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory: Name of this class: "org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory".
CLASSNAME - Static variable in class org.apache.hadoop.fs.s3a.commit.staging.DirectoryStagingCommitterFactory: Name of this class: "org.apache.hadoop.fs.s3a.commit.staging.DirectoryStagingCommitterFactory".
CLASSNAME - Static variable in class org.apache.hadoop.fs.s3a.commit.staging.PartitionedStagingCommitterFactory: Name of this class: "org.apache.hadoop.fs.s3a.commit.staging.PartitionedStagingCommitterFactory".
CLASSNAME - Static variable in class org.apache.hadoop.fs.s3a.commit.staging.StagingCommitterFactory: Name of this class: "org.apache.hadoop.fs.s3a.commit.staging.StagingCommitterFactory".
cleanup(JobContext, boolean) - Method in class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter: Cleanup the job context, including aborting anything pending.
cleanup(JobContext, boolean) - Method in class org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter
cleanupJob(JobContext) - Method in class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter
cleanupStagingDirs() - Method in class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter: Clean up any staging directories.
cleanupStagingDirs() - Method in class org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter: Delete the magic directory.
cleanupStagingDirs() - Method in class org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter
clearBucketOption(Configuration, String, String) - Static method in class org.apache.hadoop.fs.s3a.S3AUtils: Clear a bucket-specific property.
clearInconsistency(S3AFileSystem) - Static method in class org.apache.hadoop.fs.s3a.InconsistentAmazonS3Client: Clear any accumulated inconsistency state.
clearInconsistency() - Method in class org.apache.hadoop.fs.s3a.InconsistentAmazonS3Client: Clear all oustanding inconsistent keys.
clearTempFolderInfo(TaskAttemptID) - Static method in class org.apache.hadoop.fs.s3a.commit.staging.Paths: Remove all information held about task attempts.
close() - Method in class org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider: Propagate the close() call to the inner stsProvider.
close() - Method in class org.apache.hadoop.fs.s3a.AWSCredentialProviderList: Close routine will close all providers in the list which implement Closeable.
close() - Method in class org.apache.hadoop.fs.s3a.commit.DurationInfo
close() - Method in class org.apache.hadoop.fs.s3a.InconsistentS3Object
close() - Method in class org.apache.hadoop.fs.s3a.InconsistentS3Object.InconsistentS3InputStream
close() - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Close the filesystem.
close() - Method in class org.apache.hadoop.fs.s3a.S3AInputStream: Close the stream.
close() - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation
close() - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation.InputStreamStatistics: Close triggers the merge of statistics into the filesystem's instrumentation instance.
close() - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation.OutputStreamStatistics: Output stream has closed.
close() - Method in class org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
close() - Method in class org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore
close() - Method in class org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore
closeAll(Logger, Closeable...) - Static method in class org.apache.hadoop.fs.s3a.S3AUtils: Close the Closeable objects and ignore any Exception or null pointers.
closeAutocloseables(Logger, AutoCloseable...) - Static method in class org.apache.hadoop.fs.s3a.S3AUtils: Close the Closeable objects and ignore any Exception or null pointers.
closed - Variable in class org.apache.hadoop.fs.s3a.S3AInstrumentation.InputStreamStatistics
closeOperations - Variable in class org.apache.hadoop.fs.s3a.S3AInstrumentation.InputStreamStatistics
commit(SinglePendingCommit, String) - Method in class org.apache.hadoop.fs.s3a.commit.CommitOperations: Commit a single pending commit; exceptions are caught and converted to an outcome.
commitAborted() - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation.CommitterStatistics: A commit has been aborted.
commitCompleted(long) - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation.CommitterStatistics: A commit has been completed.
CommitConstants - Class in org.apache.hadoop.fs.s3a.commit: Constants for working with committers.
commitCreated() - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation.CommitterStatistics: A commit has been created.
commitFailed() - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation.CommitterStatistics
commitJob(JobContext) - Method in class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter: Commit work.
commitJobInternal(JobContext, List<SinglePendingCommit>) - Method in class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter: Internal Job commit operation: where the S3 requests are made (potentially in parallel).
CommitOperations - Class in org.apache.hadoop.fs.s3a.commit: The implementation of the various actions a committer needs.
CommitOperations(S3AFileSystem) - Constructor for class org.apache.hadoop.fs.s3a.commit.CommitOperations: Instantiate.
CommitOperations.MaybeIOE - Class in org.apache.hadoop.fs.s3a.commit: A holder for a possible IOException; the call CommitOperations.MaybeIOE.maybeRethrow() will throw any exception passed into the constructor, and be a no-op if none was.
commitOrFail(SinglePendingCommit) - Method in class org.apache.hadoop.fs.s3a.commit.CommitOperations: Commit the operation, throwing an exception on any failure.
commitPendingUploads(JobContext, List<SinglePendingCommit>) - Method in class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter: Commit a list of pending uploads.
commitReverted() - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation.CommitterStatistics
commitTask(TaskAttemptContext) - Method in class org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter
commitTask(TaskAttemptContext) - Method in class org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter
commitTaskInternal(TaskAttemptContext, List<? extends FileStatus>) - Method in class org.apache.hadoop.fs.s3a.commit.staging.PartitionedStagingCommitter
commitTaskInternal(TaskAttemptContext, List<? extends FileStatus>) - Method in class org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter: Commit the task by uploading all created files and then writing a pending entry for them.
COMMITTER_NAME_DIRECTORY - Static variable in class org.apache.hadoop.fs.s3a.commit.CommitConstants: Option for CommitConstants.FS_S3A_COMMITTER_NAME: directory output committer: "directory".
COMMITTER_NAME_FILE - Static variable in class org.apache.hadoop.fs.s3a.commit.CommitConstants: Option for CommitConstants.FS_S3A_COMMITTER_NAME: classic/file output committer: "file".
COMMITTER_NAME_MAGIC - Static variable in class org.apache.hadoop.fs.s3a.commit.CommitConstants: Option for CommitConstants.FS_S3A_COMMITTER_NAME: magic output committer: "magic".
COMMITTER_NAME_PARTITIONED - Static variable in class org.apache.hadoop.fs.s3a.commit.CommitConstants: Option for CommitConstants.FS_S3A_COMMITTER_NAME: partition output committer: "partitioned".
COMMITTER_NAME_STAGING - Static variable in class org.apache.hadoop.fs.s3a.commit.InternalCommitterConstants: This is the staging committer base class; only used for testing.
CommitterStatistics() - Constructor for class org.apache.hadoop.fs.s3a.S3AInstrumentation.CommitterStatistics
commitUploaded(long) - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation.CommitterStatistics: Data has been uploaded to be committed in a subsequent operation.
commitUploaded(long) - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation.OutputStreamStatistics: Data has been uploaded to be committed in a subsequent operation; to be called at the end of the write.
CommitUtils - Class in org.apache.hadoop.fs.s3a.commit: Static utility methods related to S3A commitment processing, both staging and magic.
CommitUtilsWithMR - Class in org.apache.hadoop.fs.s3a.commit: These are commit utility methods which import classes from hadoop-mapreduce, and so only work when that module is on the classpath.
complete(Path, Map<Integer, PartHandle>, UploadHandle) - Method in class org.apache.hadoop.fs.s3a.S3AMultipartUploader
completeMPUwithRetries(String, String, List<PartETag>, long, AtomicInteger) - Method in class org.apache.hadoop.fs.s3a.WriteOperationHelper: This completes a multipart upload to the destination key via finalizeMultipartUpload().
completeMultipartUpload(CompleteMultipartUploadRequest) - Method in class org.apache.hadoop.fs.s3a.InconsistentAmazonS3Client
CONFLICT_MODE_APPEND - Static variable in class org.apache.hadoop.fs.s3a.commit.CommitConstants: Conflict mode: "append".
CONFLICT_MODE_FAIL - Static variable in class org.apache.hadoop.fs.s3a.commit.CommitConstants: Conflict mode: "fail".
CONFLICT_MODE_REPLACE - Static variable in class org.apache.hadoop.fs.s3a.commit.CommitConstants: Conflict mode: "replace".
ConflictResolution - Enum in org.apache.hadoop.fs.s3a.commit.staging: Enum of conflict resolution algorithms.
connectivityFailure - Variable in class org.apache.hadoop.fs.s3a.S3ARetryPolicy: Client connectivity: fixed retries without care for idempotency.
Constants - Class in org.apache.hadoop.fs.s3a: All the constants used with the S3AFileSystem.
containsBasePath(List<String>) - Static method in class org.apache.hadoop.fs.s3a.commit.MagicCommitPaths: Does the list of magic elements contain a base path marker?
CONTEXT - Static variable in class org.apache.hadoop.fs.s3a.S3AInstrumentation: "s3aFileSystem" Currently all s3a metrics are placed in a single "context".
continueListObjects(S3ListRequest, S3ListResult) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: List the next set of objects.
copyFromLocalFile(boolean, boolean, Path, Path) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: The src file is on the local disk.
counter(String, String) - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation: Create a counter in the registry.
counter(Statistic) - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation: Create a counter in the registry.
create(Path, FsPermission, boolean, int, short, long, Progressable) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Create an FSDataOutputStream at the indicated Path with write-progress reporting.
create(Path, FsPermission, boolean, int, short, long, Progressable) - Method in class org.apache.hadoop.fs.s3native.NativeS3FileSystem: Deprecated.
CREATE_SUCCESSFUL_JOB_OUTPUT_DIR_MARKER - Static variable in class org.apache.hadoop.fs.s3a.commit.CommitConstants: Flag to trigger creation of a marker file on job completion.
createAwsConf(Configuration, String) - Static method in class org.apache.hadoop.fs.s3a.S3AUtils: Create a new AWS ClientConfiguration.
createAWSCredentialProvider(Configuration, Class<?>, URI) - Static method in class org.apache.hadoop.fs.s3a.S3AUtils: Create an AWS credential provider from its class by using reflection.
createAWSCredentialProviderSet(URI, Configuration) - Static method in class org.apache.hadoop.fs.s3a.S3AUtils: Create the AWS credentials from the providers, the URI and the key Constants.AWS_CREDENTIALS_PROVIDER in the configuration.
createDynamoDBClient(String, String, AWSCredentialsProvider) - Method in interface org.apache.hadoop.fs.s3a.s3guard.DynamoDBClientFactory: Create a DynamoDB client object from configuration.
createDynamoDBClient(String, String, AWSCredentialsProvider) - Method in class org.apache.hadoop.fs.s3a.s3guard.DynamoDBClientFactory.DefaultDynamoDBClientFactory
createExceptionMap() - Method in class org.apache.hadoop.fs.s3a.S3ARetryPolicy: Subclasses can override this like a constructor to change behavior: call superclass method, then modify it as needed, and return it.
createExceptionMap() - Method in class org.apache.hadoop.fs.s3a.S3GuardExistsRetryPolicy
createFileStatus(Path, S3ObjectSummary, long, String) - Static method in class org.apache.hadoop.fs.s3a.S3AUtils: Create a files status instance from a listing.
createMultipartUploader(FileSystem, Configuration) - Method in class org.apache.hadoop.fs.s3a.S3AMultipartUploader.Factory
createNonRecursive(Path, FsPermission, EnumSet<CreateFlag>, int, short, long, Progressable) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem
createOutputCommitter(Path, TaskAttemptContext) - Method in class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitterFactory
createPolicy(ChangeDetectionPolicy.Mode, ChangeDetectionPolicy.Source, boolean) - Static method in class org.apache.hadoop.fs.s3a.impl.ChangeDetectionPolicy: Create a policy.
createPutObjectRequest(String, InputStream, long) - Method in class org.apache.hadoop.fs.s3a.WriteOperationHelper: Create a PutObjectRequest request against the specific key.
createPutObjectRequest(String, File) - Method in class org.apache.hadoop.fs.s3a.WriteOperationHelper: Create a PutObjectRequest request to upload a file.
createS3Client(URI, String, AWSCredentialsProvider) - Method in class org.apache.hadoop.fs.s3a.DefaultS3ClientFactory
createS3Client(URI, String, AWSCredentialsProvider) - Method in interface org.apache.hadoop.fs.s3a.S3ClientFactory: Creates a new AmazonS3 client.
createStorageStatistics() - Static method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Create the storage statistics or bind to an existing one.
createSuccessMarker(Path, SuccessData, boolean) - Method in class org.apache.hadoop.fs.s3a.commit.CommitOperations: Save the success data to the _SUCCESS file.
createTaskCommitter(S3AFileSystem, Path, TaskAttemptContext) - Method in class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitterFactory: Implementation point: create a task committer for a specific filesystem.
createTaskCommitter(S3AFileSystem, Path, TaskAttemptContext) - Method in class org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitterFactory
createTaskCommitter(S3AFileSystem, Path, TaskAttemptContext) - Method in class org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory: Create a task committer.
createTaskCommitter(S3AFileSystem, Path, TaskAttemptContext) - Method in class org.apache.hadoop.fs.s3a.commit.staging.DirectoryStagingCommitterFactory
createTaskCommitter(S3AFileSystem, Path, TaskAttemptContext) - Method in class org.apache.hadoop.fs.s3a.commit.staging.PartitionedStagingCommitterFactory
createTaskCommitter(S3AFileSystem, Path, TaskAttemptContext) - Method in class org.apache.hadoop.fs.s3a.commit.staging.StagingCommitterFactory
createThrottleRetryPolicy(Configuration) - Method in class org.apache.hadoop.fs.s3a.S3ARetryPolicy: Create the throttling policy.
createThrottleRetryPolicy(Configuration) - Method in class org.apache.hadoop.fs.s3a.s3guard.S3GuardDataAccessRetryPolicy
createTracker(Path, String) - Method in class org.apache.hadoop.fs.s3a.commit.MagicCommitIntegration: Given a path and a key to that same path, create a tracker for it.
createUploadFileStatus(Path, boolean, long, long, String) - Static method in class org.apache.hadoop.fs.s3a.S3AUtils: Create a file status for object we just uploaded.
createWrappedCommitter(JobContext, Configuration) - Method in class org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter: Create the wrapped committer.
CredentialInitializationException - Exception in org.apache.hadoop.fs.s3a: Exception which Hadoop's AWSCredentialsProvider implementations should throw when there is a problem with the credential setup.
CredentialInitializationException(String, Throwable) - Constructor for exception org.apache.hadoop.fs.s3a.CredentialInitializationException
CredentialInitializationException(String) - Constructor for exception org.apache.hadoop.fs.s3a.CredentialInitializationException

D

dateToLong(Date) - Static method in class org.apache.hadoop.fs.s3a.S3AUtils: Date to long conversion.
DAYS_FLAG - Static variable in class org.apache.hadoop.fs.s3a.s3guard.S3GuardTool
DDB_ADMIN - Static variable in class org.apache.hadoop.fs.s3a.auth.RolePolicies: Operations needed for DDB/S3Guard Admin.
DDB_ALL_OPERATIONS - Static variable in class org.apache.hadoop.fs.s3a.auth.RolePolicies: All DynamoDB operations: "dynamodb:*".
DDB_BATCH_GET_ITEM - Static variable in class org.apache.hadoop.fs.s3a.auth.RolePolicies: Permission for DDB operation: "dynamodb:BatchGetItem".
DDB_BATCH_WRITE_ITEM - Static variable in class org.apache.hadoop.fs.s3a.auth.RolePolicies: Batch write permission for DDB: "dynamodb:BatchWriteItem".
DDB_DELETE_ITEM - Static variable in class org.apache.hadoop.fs.s3a.auth.RolePolicies: Permission for DDB delete operation: "dynamodb:DeleteItem".
DDB_DESCRIBE_TABLE - Static variable in class org.apache.hadoop.fs.s3a.auth.RolePolicies: Permission for DDB describeTable() operation: "dynamodb:DescribeTable".
DDB_GET_ITEM - Static variable in class org.apache.hadoop.fs.s3a.auth.RolePolicies: Permission for DDB operation to get a record: "dynamodb:GetItem".
DDB_PUT_ITEM - Static variable in class org.apache.hadoop.fs.s3a.auth.RolePolicies: Permission for DDB write record operation: "dynamodb:PutItem".
DDB_QUERY - Static variable in class org.apache.hadoop.fs.s3a.auth.RolePolicies: Permission to query the DDB table: "dynamodb:Query".
DDB_UPDATE_ITEM - Static variable in class org.apache.hadoop.fs.s3a.auth.RolePolicies: Permission for DDB update single item operation: "dynamodb:UpdateItem".
DDBPathMetadata - Class in org.apache.hadoop.fs.s3a.s3guard: DDBPathMetadata wraps PathMetadata and adds the isAuthoritativeDir flag to provide support for authoritative directory listings in DynamoDBMetadataStore.
DDBPathMetadata(PathMetadata, boolean) - Constructor for class org.apache.hadoop.fs.s3a.s3guard.DDBPathMetadata
DDBPathMetadata(PathMetadata) - Constructor for class org.apache.hadoop.fs.s3a.s3guard.DDBPathMetadata
DDBPathMetadata(FileStatus) - Constructor for class org.apache.hadoop.fs.s3a.s3guard.DDBPathMetadata
DDBPathMetadata(FileStatus, Tristate, boolean) - Constructor for class org.apache.hadoop.fs.s3a.s3guard.DDBPathMetadata
DDBPathMetadata(FileStatus, Tristate, boolean, boolean) - Constructor for class org.apache.hadoop.fs.s3a.s3guard.DDBPathMetadata
decrementGauge(Statistic, long) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Decrement a gauge by a specific value.
decrementGauge(Statistic, long) - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation: Decrement a specific gauge.
DEFAULT_BLOCKSIZE - Static variable in class org.apache.hadoop.fs.s3a.S3AFileSystem: Default blocksize as used in blocksize and FS status queries.
DEFAULT_CANNED_ACL - Static variable in class org.apache.hadoop.fs.s3a.Constants
DEFAULT_COMMITTER_THREADS - Static variable in class org.apache.hadoop.fs.s3a.commit.CommitConstants: Default value for CommitConstants.FS_S3A_COMMITTER_THREADS: 8.
DEFAULT_CONFLICT_MODE - Static variable in class org.apache.hadoop.fs.s3a.commit.CommitConstants: Default conflict mode: "fail".
DEFAULT_CREATE_SUCCESSFUL_JOB_DIR_MARKER - Static variable in class org.apache.hadoop.fs.s3a.commit.CommitConstants: Default job marker option: true.
DEFAULT_DELAY_KEY_MSEC - Static variable in class org.apache.hadoop.fs.s3a.FailureInjectionPolicy: How many seconds affected keys will have delayed visibility.
DEFAULT_DELAY_KEY_PROBABILITY - Static variable in class org.apache.hadoop.fs.s3a.FailureInjectionPolicy
DEFAULT_DELAY_KEY_SUBSTRING - Static variable in class org.apache.hadoop.fs.s3a.FailureInjectionPolicy: Keys containing this substring will be subject to delayed visibility.
DEFAULT_ESTABLISH_TIMEOUT - Static variable in class org.apache.hadoop.fs.s3a.Constants
DEFAULT_FAST_BUFFER_SIZE - Static variable in class org.apache.hadoop.fs.s3a.Constants
DEFAULT_FAST_UPLOAD - Static variable in class org.apache.hadoop.fs.s3a.Constants: Deprecated.
DEFAULT_FAST_UPLOAD_ACTIVE_BLOCKS - Static variable in class org.apache.hadoop.fs.s3a.Constants: Limit of queued block upload operations before writes block.
DEFAULT_FAST_UPLOAD_BUFFER - Static variable in class org.apache.hadoop.fs.s3a.Constants: Default buffer option: "disk".
DEFAULT_KEEPALIVE_TIME - Static variable in class org.apache.hadoop.fs.s3a.Constants
DEFAULT_LIST_VERSION - Static variable in class org.apache.hadoop.fs.s3a.Constants
DEFAULT_MAGIC_COMMITTER_ENABLED - Static variable in class org.apache.hadoop.fs.s3a.commit.CommitConstants: Is the committer enabled by default? No.
DEFAULT_MAX_ERROR_RETRIES - Static variable in class org.apache.hadoop.fs.s3a.Constants
DEFAULT_MAX_PAGING_KEYS - Static variable in class org.apache.hadoop.fs.s3a.Constants
DEFAULT_MAX_THREADS - Static variable in class org.apache.hadoop.fs.s3a.Constants
DEFAULT_MAX_TOTAL_TASKS - Static variable in class org.apache.hadoop.fs.s3a.Constants
DEFAULT_MAXIMUM_CONNECTIONS - Static variable in class org.apache.hadoop.fs.s3a.Constants
DEFAULT_METADATASTORE_AUTHORITATIVE - Static variable in class org.apache.hadoop.fs.s3a.Constants
DEFAULT_MIN_MULTIPART_THRESHOLD - Static variable in class org.apache.hadoop.fs.s3a.Constants
DEFAULT_MULTIPART_SIZE - Static variable in class org.apache.hadoop.fs.s3a.Constants
DEFAULT_PURGE_EXISTING_MULTIPART - Static variable in class org.apache.hadoop.fs.s3a.Constants
DEFAULT_PURGE_EXISTING_MULTIPART_AGE - Static variable in class org.apache.hadoop.fs.s3a.Constants
DEFAULT_READAHEAD_RANGE - Static variable in class org.apache.hadoop.fs.s3a.Constants
DEFAULT_S3_CLIENT_FACTORY_IMPL - Static variable in class org.apache.hadoop.fs.s3a.Constants
DEFAULT_S3GUARD_METASTORE_LOCAL_ENTRY_TTL - Static variable in class org.apache.hadoop.fs.s3a.Constants
DEFAULT_S3GUARD_METASTORE_LOCAL_MAX_RECORDS - Static variable in class org.apache.hadoop.fs.s3a.Constants
DEFAULT_SECURE_CONNECTIONS - Static variable in class org.apache.hadoop.fs.s3a.Constants
DEFAULT_SOCKET_RECV_BUFFER - Static variable in class org.apache.hadoop.fs.s3a.Constants
DEFAULT_SOCKET_SEND_BUFFER - Static variable in class org.apache.hadoop.fs.s3a.Constants
DEFAULT_SOCKET_TIMEOUT - Static variable in class org.apache.hadoop.fs.s3a.Constants
DEFAULT_STAGING_COMMITTER_UNIQUE_FILENAMES - Static variable in class org.apache.hadoop.fs.s3a.commit.CommitConstants: Default value for CommitConstants.FS_S3A_COMMITTER_STAGING_UNIQUE_FILENAMES: true.
DefaultDynamoDBClientFactory() - Constructor for class org.apache.hadoop.fs.s3a.s3guard.DynamoDBClientFactory.DefaultDynamoDBClientFactory
DefaultS3ClientFactory - Class in org.apache.hadoop.fs.s3a: The default S3ClientFactory implementation.
DefaultS3ClientFactory() - Constructor for class org.apache.hadoop.fs.s3a.DefaultS3ClientFactory
delete(Path, boolean) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Delete a Path.
delete(Path) - Method in class org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
delete(Path) - Method in class org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore
delete(Path) - Method in interface org.apache.hadoop.fs.s3a.s3guard.MetadataStore: Deletes exactly one path, leaving a tombstone to prevent lingering, inconsistent copies of it from being listed.
delete(Path) - Method in class org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore
delete(Path, boolean) - Method in class org.apache.hadoop.fs.s3native.NativeS3FileSystem: Deprecated.
DELETE_CONSIDERED_IDEMPOTENT - Static variable in class org.apache.hadoop.fs.s3a.S3AFileSystem: This declared delete as idempotent.
deleteDestinationPaths(JobContext) - Method in class org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter: Delete the working paths of a job.
deleteObject(DeleteObjectRequest) - Method in class org.apache.hadoop.fs.s3a.InconsistentAmazonS3Client
deleteObject(String) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Delete an object.
deleteObjects(DeleteObjectsRequest) - Method in class org.apache.hadoop.fs.s3a.InconsistentAmazonS3Client
deleteQuietly(FileSystem, Path, boolean) - Static method in class org.apache.hadoop.fs.s3a.S3AUtils: Delete a path quietly: failures are logged at DEBUG.
deleteSubtree(Path) - Method in class org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
deleteSubtree(Path) - Method in class org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore
deleteSubtree(Path) - Method in interface org.apache.hadoop.fs.s3a.s3guard.MetadataStore: Deletes the entire sub-tree rooted at the given path, leaving tombstones to prevent lingering, inconsistent copies of it from being listed.
deleteSubtree(Path) - Method in class org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore
deleteSuccessMarker(Path) - Method in class org.apache.hadoop.fs.s3a.commit.CommitOperations: Delete any existing _SUCCESS file.
deleteTaskAttemptPathQuietly(TaskAttemptContext) - Method in class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter: Delete the task attempt path without raising any errors.
deleteTaskWorkingPathQuietly(JobContext) - Method in class org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter: Delete the working path of a task; no-op if there is none, that is: this is a job.
deleteWithWarning(FileSystem, Path, boolean) - Static method in class org.apache.hadoop.fs.s3a.S3AUtils: Delete a path: failures are logged at WARN.
DescendantsIterator - Class in org.apache.hadoop.fs.s3a.s3guard: DescendantsIterator is a RemoteIterator that implements pre-ordering breadth-first traversal (BFS) of a path and all of its descendants recursively.
DescendantsIterator(MetadataStore, PathMetadata) - Constructor for class org.apache.hadoop.fs.s3a.s3guard.DescendantsIterator: Creates a new DescendantsIterator.
destinationPath() - Method in class org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit: Build the destination path of the object.
destroy() - Method in class org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
destroy() - Method in class org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore
destroy() - Method in interface org.apache.hadoop.fs.s3a.s3guard.MetadataStore: Destroy all resources associated with the metadata store.
destroy() - Method in class org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore
directory(Path) - Static method in class org.apache.hadoop.fs.s3a.auth.RoleModel: Given a directory path, return the S3 resource to it.
DIRECTORY_COMMITTER_FACTORY - Static variable in class org.apache.hadoop.fs.s3a.commit.InternalCommitterConstants: Directory committer factory: "org.apache.hadoop.fs.s3a.commit.staging.DirectoryStagingCommitterFactory".
directoryCreated() - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation: Indicate that S3A created a directory.
directoryDeleted() - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation: Indicate that S3A just deleted a directory.
DirectoryStagingCommitter - Class in org.apache.hadoop.fs.s3a.commit.staging: This commits to a directory.
DirectoryStagingCommitter(Path, TaskAttemptContext) - Constructor for class org.apache.hadoop.fs.s3a.commit.staging.DirectoryStagingCommitter
DirectoryStagingCommitterFactory - Class in org.apache.hadoop.fs.s3a.commit.staging: Factory for the Directory committer.
DirectoryStagingCommitterFactory() - Constructor for class org.apache.hadoop.fs.s3a.commit.staging.DirectoryStagingCommitterFactory
DirListingMetadata - Class in org.apache.hadoop.fs.s3a.s3guard: DirListingMetadata models a directory listing stored in a MetadataStore.
DirListingMetadata(Path, Collection<PathMetadata>, boolean) - Constructor for class org.apache.hadoop.fs.s3a.s3guard.DirListingMetadata: Create a directory listing metadata container.
DirListingMetadata(DirListingMetadata) - Constructor for class org.apache.hadoop.fs.s3a.s3guard.DirListingMetadata: Copy constructor.
dirListingUnion(MetadataStore, Path, List<FileStatus>, DirListingMetadata, boolean) - Static method in class org.apache.hadoop.fs.s3a.s3guard.S3Guard: Given directory listing metadata from both the backing store and the MetadataStore, merge the two sources of truth to create a consistent view of the current directory contents, which can be returned to clients.
dirMetaToStatuses(DirListingMetadata) - Static method in class org.apache.hadoop.fs.s3a.s3guard.S3Guard: Convert the data of a directory listing to an array of FileStatus entries.
dstFileStatus - Variable in class org.apache.hadoop.fs.s3a.S3AOpContext: FileStatus for "destination" path being operated on.
dump(String, String, String, boolean) - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation: Dump all the metrics to a string.
dumpDiagnostics(String, String, String) - Method in class org.apache.hadoop.fs.s3a.commit.files.SuccessData: Dump the diagnostics (if any) to a string.
dumpMetrics(String, String, String) - Method in class org.apache.hadoop.fs.s3a.commit.files.SuccessData: Dump the metrics (if any) to a string.
Duration - Class in org.apache.hadoop.fs.s3a.commit: Little duration counter.
Duration() - Constructor for class org.apache.hadoop.fs.s3a.commit.Duration
DurationInfo - Class in org.apache.hadoop.fs.s3a.commit: A duration with logging of final state at info in the close() call.
DurationInfo(Logger, String, Object...) - Constructor for class org.apache.hadoop.fs.s3a.commit.DurationInfo: Create the duration text from a String.format() code call.
DynamoDBClientFactory - Interface in org.apache.hadoop.fs.s3a.s3guard: Interface to create a DynamoDB client.
DynamoDBClientFactory.DefaultDynamoDBClientFactory - Class in org.apache.hadoop.fs.s3a.s3guard: The default implementation for creating an AmazonDynamoDB.
DynamoDBMetadataStore - Class in org.apache.hadoop.fs.s3a.s3guard: DynamoDBMetadataStore is a MetadataStore that persists file system metadata to DynamoDB.
DynamoDBMetadataStore() - Constructor for class org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore

E

E_BAD_PATH - Static variable in class org.apache.hadoop.fs.s3a.commit.InternalCommitterConstants: Error message for bad path: "Path does not represent a magic-commit path".
E_DEST_EXISTS - Static variable in class org.apache.hadoop.fs.s3a.commit.InternalCommitterConstants: Error text when the destination path exists and the committer must abort the job/task "Destination path exists and committer conflict resolution mode is \"fail\"".
E_INCOMPATIBLE_VERSION - Static variable in class org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore: Error: version mismatch.
E_NO_MAGIC_PATH_ELEMENT - Static variable in class org.apache.hadoop.fs.s3a.commit.InternalCommitterConstants: Error message for a path without a magic element in the list: "No __magic element in path".
E_NO_ROLE - Static variable in class org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider
E_NO_VERSION_MARKER - Static variable in class org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore: Error: version marker not found in table.
E_NORMAL_FS - Static variable in class org.apache.hadoop.fs.s3a.commit.InternalCommitterConstants: Error message if filesystem isn't magic: "Filesystem does not have support for \'magic\' committer enabled in configuration option fs.s3a.committer.magic.enabled".
E_WRONG_FS - Static variable in class org.apache.hadoop.fs.s3a.commit.InternalCommitterConstants: Error message if the dest FS isn't S3A: "Output path is not on an S3A Filesystem".
effect(boolean) - Static method in class org.apache.hadoop.fs.s3a.auth.RoleModel: Map a bool to an effect.
effect - Variable in class org.apache.hadoop.fs.s3a.auth.RoleModel.Statement: Default effect is Deny; forces callers to switch on Allow.
elementsToKey(List<String>) - Static method in class org.apache.hadoop.fs.s3a.commit.MagicCommitPaths: Take a list of elements and create an S3 key by joining them with "/" between each one.
EMPTY - Static variable in class org.apache.hadoop.fs.s3native.S3xLoginHelper.Login
EMPTY_DIR - Static variable in class org.apache.hadoop.fs.s3a.s3guard.DirListingMetadata: Convenience parameter for passing into constructor.
ENABLE_MULTI_DELETE - Static variable in class org.apache.hadoop.fs.s3a.Constants
ENDPOINT - Static variable in class org.apache.hadoop.fs.s3a.Constants
ensureOutputParameterInRange(String, long) - Static method in class org.apache.hadoop.fs.s3a.S3AUtils: Ensure that the long value is in the range of an integer.
entryPoint(Statistic) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Entry point to an operation.
equals(Object) - Method in class org.apache.hadoop.fs.s3a.S3AFileStatus: Compare if this object is equal to another object.
equals(Object) - Method in class org.apache.hadoop.fs.s3a.s3guard.DDBPathMetadata
equals(Object) - Method in class org.apache.hadoop.fs.s3a.s3guard.PathMetadata
equals(Object) - Method in class org.apache.hadoop.fs.s3native.S3xLoginHelper.Login: Equality test matches user and password.
errorIgnored() - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation: Note that an error was ignored.
ESTABLISH_TIMEOUT - Static variable in class org.apache.hadoop.fs.s3a.Constants
ETAG_CHECKSUM_ENABLED - Static variable in class org.apache.hadoop.fs.s3a.Constants: Should etags be exposed as checksums?
ETAG_CHECKSUM_ENABLED_DEFAULT - Static variable in class org.apache.hadoop.fs.s3a.Constants: Default value: false.
execute() - Method in interface org.apache.hadoop.fs.s3a.Invoker.Operation
execute() - Method in interface org.apache.hadoop.fs.s3a.Invoker.VoidOperation
executeWith(ExecutorService) - Method in class org.apache.hadoop.fs.s3a.commit.Tasks.Builder: Declare executor service: if null, the tasks are executed in a single thread.
exists(Path) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Override superclass so as to add statistic collection.
exit(int, String) - Static method in class org.apache.hadoop.fs.s3a.s3guard.S3GuardTool
extractException(String, String, ExecutionException) - Static method in class org.apache.hadoop.fs.s3a.S3AUtils: Extract an exception from a failed future, and convert to an IOE.
extractInnerException(IllegalArgumentException) - Static method in class org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore: Take an IllegalArgumentException raised by a DDB operation and if it contains an inner SDK exception, unwrap it.

F

Factory() - Constructor for class org.apache.hadoop.fs.s3a.S3AMultipartUploader.Factory
fail - Variable in class org.apache.hadoop.fs.s3a.S3ARetryPolicy: No retry on network and tangible API issues.
FAIL_INJECT_CLIENT_FACTORY - Static variable in class org.apache.hadoop.fs.s3a.Constants
FAIL_INJECT_INCONSISTENCY_KEY - Static variable in class org.apache.hadoop.fs.s3a.Constants: Inconsistency (visibility delay) injection settings.
FAIL_INJECT_INCONSISTENCY_MSEC - Static variable in class org.apache.hadoop.fs.s3a.Constants
FAIL_INJECT_INCONSISTENCY_PROBABILITY - Static variable in class org.apache.hadoop.fs.s3a.Constants
FAIL_INJECT_THROTTLE_PROBABILITY - Static variable in class org.apache.hadoop.fs.s3a.Constants
failDestinationExists(Path, String) - Method in class org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter: Generate a PathExistsException because the destination exists.
FailureInjectionPolicy - Class in org.apache.hadoop.fs.s3a: Simple object which stores current failure injection settings.
FailureInjectionPolicy(Configuration) - Constructor for class org.apache.hadoop.fs.s3a.FailureInjectionPolicy
fakeDirsDeleted(int) - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation: Indicate that fake directory request was made.
FAST_BUFFER_SIZE - Static variable in class org.apache.hadoop.fs.s3a.Constants: Deprecated.
FAST_UPLOAD - Static variable in class org.apache.hadoop.fs.s3a.Constants: Deprecated.
FAST_UPLOAD_ACTIVE_BLOCKS - Static variable in class org.apache.hadoop.fs.s3a.Constants: Maximum Number of blocks a single output stream can have active (uploading, or queued to the central FileSystem instance's pool of queued operations.
FAST_UPLOAD_BUFFER - Static variable in class org.apache.hadoop.fs.s3a.Constants: What buffer to use.
FAST_UPLOAD_BUFFER_ARRAY - Static variable in class org.apache.hadoop.fs.s3a.Constants: Use an in-memory array.
FAST_UPLOAD_BUFFER_DISK - Static variable in class org.apache.hadoop.fs.s3a.Constants: Buffer blocks to disk: "disk".
FAST_UPLOAD_BYTEBUFFER - Static variable in class org.apache.hadoop.fs.s3a.Constants: Use a byte buffer.
fileCreated() - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation: Indicate that S3A created a file.
fileDeleted(int) - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation: Indicate that S3A deleted one or more files.
filename(List<String>) - Static method in class org.apache.hadoop.fs.s3a.commit.MagicCommitPaths: Get the filename of a path: the last element.
filesCopied(int, long) - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation: Indicate that S3A copied some files within the store.
FILESYSTEM_TEMP_PATH - Static variable in class org.apache.hadoop.fs.s3a.commit.staging.StagingCommitterConstants: The temporary path for staging data, if not explicitly set.
finalDestination(List<String>) - Static method in class org.apache.hadoop.fs.s3a.commit.MagicCommitPaths: Calculates the final destination of a file.
finished() - Method in class org.apache.hadoop.fs.s3a.commit.Duration
fixedRetries - Variable in class org.apache.hadoop.fs.s3a.S3ARetryPolicy: Base policy from configuration.
flatmapLocatedFiles(RemoteIterator<LocatedFileStatus>, S3AUtils.LocatedFileStatusMap<Optional<T>>) - Static method in class org.apache.hadoop.fs.s3a.S3AUtils: Map an operation to every LocatedFileStatus in a remote iterator, returning a list of the all results which were not empty.
foreach(Iterable<I>) - Static method in class org.apache.hadoop.fs.s3a.commit.Tasks
foreach(I[]) - Static method in class org.apache.hadoop.fs.s3a.commit.Tasks
forgetMetadata(Path) - Method in class org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
forgetMetadata(Path) - Method in class org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore
forgetMetadata(Path) - Method in interface org.apache.hadoop.fs.s3a.s3guard.MetadataStore: Removes the record of exactly one path.
forgetMetadata(Path) - Method in class org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore
formatAppAttemptDir(int) - Static method in class org.apache.hadoop.fs.s3a.commit.CommitUtilsWithMR: Format the application attempt directory.
forwardSeekOperations - Variable in class org.apache.hadoop.fs.s3a.S3AInstrumentation.InputStreamStatistics
fromBool(boolean) - Static method in enum org.apache.hadoop.fs.s3a.Tristate
fromFileStatus(FileStatus, Tristate) - Static method in class org.apache.hadoop.fs.s3a.S3AFileStatus: Convenience constructor for creating from a vanilla FileStatus plus an isEmptyDirectory flag.
fromSymbol(String) - Static method in enum org.apache.hadoop.fs.s3a.Statistic: Get a statistic from a symbol.
FS_S3A - Static variable in class org.apache.hadoop.fs.s3a.Constants
FS_S3A_BLOCK_SIZE - Static variable in class org.apache.hadoop.fs.s3a.Constants
FS_S3A_BUCKET_PREFIX - Static variable in class org.apache.hadoop.fs.s3a.Constants: Prefix for S3A bucket-specific properties: "fs.s3a.bucket.".
FS_S3A_COMMITTER_NAME - Static variable in class org.apache.hadoop.fs.s3a.commit.CommitConstants: Option to identify the S3A committer: "fs.s3a.committer.name".
FS_S3A_COMMITTER_STAGING_ABORT_PENDING_UPLOADS - Static variable in class org.apache.hadoop.fs.s3a.commit.CommitConstants: Should the staging committers abort all pending uploads to the destination directory? Default: true.
FS_S3A_COMMITTER_STAGING_CONFLICT_MODE - Static variable in class org.apache.hadoop.fs.s3a.commit.CommitConstants: Staging committer conflict resolution policy: "fs.s3a.committer.staging.conflict-mode".
FS_S3A_COMMITTER_STAGING_TMP_PATH - Static variable in class org.apache.hadoop.fs.s3a.commit.CommitConstants: Path in the cluster filesystem for temporary data: "fs.s3a.committer.staging.tmp.path".
FS_S3A_COMMITTER_STAGING_UNIQUE_FILENAMES - Static variable in class org.apache.hadoop.fs.s3a.commit.CommitConstants: Option for final files to have a uniqueness name through job attempt info, falling back to a new UUID if there is no job attempt information to use.
FS_S3A_COMMITTER_STAGING_UUID - Static variable in class org.apache.hadoop.fs.s3a.commit.InternalCommitterConstants: A unique identifier to use for this work: "fs.s3a.committer.staging.uuid".
FS_S3A_COMMITTER_THREADS - Static variable in class org.apache.hadoop.fs.s3a.commit.CommitConstants: Number of threads in committers for parallel operations on files (upload, commit, abort, delete...): "fs.s3a.committer.threads".
FS_S3A_PREFIX - Static variable in class org.apache.hadoop.fs.s3a.Constants: Prefix for all S3A properties: "fs.s3a.".

G

gauge(String, String) - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation: Create a gauge in the registry.
get(Path) - Method in class org.apache.hadoop.fs.s3a.s3guard.DirListingMetadata: Lookup entry within this directory listing.
get(Path) - Method in class org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
get(Path, boolean) - Method in class org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
get(Path) - Method in class org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore
get(Path, boolean) - Method in class org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore
get(Path) - Method in interface org.apache.hadoop.fs.s3a.s3guard.MetadataStore: Gets metadata for a path.
get(Path, boolean) - Method in interface org.apache.hadoop.fs.s3a.s3guard.MetadataStore: Gets metadata for a path.
get(Path) - Method in class org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore
get(Path, boolean) - Method in class org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore
getAdditionalDetails() - Method in exception org.apache.hadoop.fs.s3a.AWSS3IOException
getAmazonDynamoDB() - Method in class org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
getAmazonS3ClientForTesting(String) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Returns the S3 client used by this filesystem.
getAppAttemptId(JobContext) - Static method in class org.apache.hadoop.fs.s3a.commit.CommitUtilsWithMR: Get the Application Attempt ID for this job.
getAWSAccessKeys(URI, Configuration) - Static method in class org.apache.hadoop.fs.s3a.S3AUtils: Return the access key and secret for S3 API use.
getBaseMagicTaskAttemptPath(TaskAttemptContext, Path) - Static method in class org.apache.hadoop.fs.s3a.commit.CommitUtilsWithMR: Get the base Magic attempt path, without any annotations to mark relative references.
getBaseTaskAttemptPath(TaskAttemptContext) - Method in class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter: Compute the base path where the output of a task attempt is written.
getBaseTaskAttemptPath(TaskAttemptContext) - Method in class org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter
getBaseTaskAttemptPath(TaskAttemptContext) - Method in class org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter: Return the local work path as the destination for writing work.
getBatchWriteCapacityExceededCount() - Method in class org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
getBucket() - Method in class org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit
getBucket() - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Get the bucket of this filesystem.
getBucketLocation() - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Get the region of a bucket.
getBucketLocation(String) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Get the region of a bucket.
getBucketName() - Method in class org.apache.hadoop.fs.s3a.InconsistentS3Object
getBucketOption(Configuration, String, String) - Static method in class org.apache.hadoop.fs.s3a.S3AUtils: Get a bucket-specific property.
getBytesPendingUpload() - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation.OutputStreamStatistics: Get the number of bytes pending upload.
getCanonicalServiceName() - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Override getCanonicalServiceName because we don't support token in S3A.
getCause() - Method in exception org.apache.hadoop.fs.s3a.AWSClientIOException
getCause() - Method in exception org.apache.hadoop.fs.s3a.AWSS3IOException
getCause() - Method in exception org.apache.hadoop.fs.s3a.AWSServiceIOException
getChangeDetectionPolicy() - Method in class org.apache.hadoop.fs.s3a.S3AReadOpContext
getCommandFormat() - Method in class org.apache.hadoop.fs.s3a.s3guard.S3GuardTool
getCommitOperations() - Method in class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter: Get the commit actions instance.
getCommits() - Method in class org.apache.hadoop.fs.s3a.commit.files.PendingSet
getCommittedTaskPath(TaskAttemptContext) - Method in class org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter: Compute the path where the output of a committed task is stored until the entire job is committed.
getCommittedTaskPath(int, TaskAttemptContext) - Method in class org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter: Compute the path where the output of a committed task is stored until the entire job is committed for a specific application attempt.
getCommitter() - Method in class org.apache.hadoop.fs.s3a.commit.files.SuccessData
getCommonPrefixes() - Method in class org.apache.hadoop.fs.s3a.S3ListResult
getConf() - Method in class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter
getConfictModeOption(JobContext, Configuration) - Static method in class org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter: Get the conflict mode option string.
getConfigurationOption(JobContext, Configuration, String, String) - Static method in class org.apache.hadoop.fs.s3a.commit.CommitUtilsWithMR: Get a configuration option, with any value in the job configuration taking priority over that in the filesystem.
getConflictResolutionMode(JobContext, Configuration) - Method in class org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter: Returns the ConflictResolution mode for this commit.
getContentRangeFinish() - Method in class org.apache.hadoop.fs.s3a.S3AInputStream
getContentRangeStart() - Method in class org.apache.hadoop.fs.s3a.S3AInputStream
getCounterValue(Statistic) - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation: Get the value of a counter.
getCounterValue(String) - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation: Get the value of a counter.
getCreated() - Method in class org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit: When was the upload created?
getCredentials() - Method in class org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider
getCredentials() - Method in class org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider: Get credentials.
getCredentials() - Method in class org.apache.hadoop.fs.s3a.AWSCredentialProviderList: Iterate through the list of providers, to find one with credentials.
getCredentials() - Method in class org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider
getCredentials() - Method in class org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider
getDate() - Method in class org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit: Timestamp as date; no expectation of parseability.
getDate() - Method in class org.apache.hadoop.fs.s3a.commit.files.SuccessData
getDefaultBlockSize() - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Deprecated.
use FileSystem.getDefaultBlockSize(Path) instead
getDefaultPort() - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem
getDelayKeyMsec() - Method in class org.apache.hadoop.fs.s3a.FailureInjectionPolicy
getDelayKeyMsec() - Method in class org.apache.hadoop.fs.s3a.InconsistentAmazonS3Client
getDelayKeyProbability() - Method in class org.apache.hadoop.fs.s3a.FailureInjectionPolicy
getDelayKeySubstring() - Method in class org.apache.hadoop.fs.s3a.FailureInjectionPolicy
getDescription() - Method in class org.apache.hadoop.fs.s3a.commit.files.SuccessData
getDescription() - Method in enum org.apache.hadoop.fs.s3a.Statistic
getDestFS() - Method in class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter: Get the destination FS, creating it on demand if needed.
getDestinationFileSystem(Path, JobContext) - Method in class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitterFactory: Get the destination filesystem, returning null if there is none.
getDestinationFS(Path, Configuration) - Method in class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter: Get the destination filesystem from the output path and the configuration.
getDestinationKey() - Method in class org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit
getDestKey() - Method in class org.apache.hadoop.fs.s3a.commit.PutTracker: get the destination key.
getDestS3AFS() - Method in class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter: Get the destination as an S3A Filesystem; casting it.
getDiagnostics() - Method in class org.apache.hadoop.fs.s3a.commit.files.SuccessData
getDiagnostics() - Method in class org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
getDiagnostics() - Method in class org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore
getDiagnostics() - Method in interface org.apache.hadoop.fs.s3a.s3guard.MetadataStore: Get any diagnostics information from a store, as a list of (key, value) tuples for display.
getDiagnostics() - Method in class org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore
getDirListingMeta() - Method in class org.apache.hadoop.fs.s3a.s3guard.LocalMetadataEntry
getDurationString() - Method in class org.apache.hadoop.fs.s3a.commit.Duration
getErrorCode() - Method in exception org.apache.hadoop.fs.s3a.AWSServiceIOException
getErrorResponseXml() - Method in exception org.apache.hadoop.fs.s3a.AWSS3IOException
getEtags() - Method in class org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit
getException() - Method in class org.apache.hadoop.fs.s3a.commit.CommitOperations.MaybeIOE: Get any exception.
getExitCode() - Method in exception org.apache.hadoop.fs.s3a.RenameFailedException
getExtendedRequestId() - Method in exception org.apache.hadoop.fs.s3a.AWSS3IOException
getExtraData() - Method in class org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit: Any custom extra data committer subclasses may choose to add.
getFailureLimit() - Method in class org.apache.hadoop.fs.s3a.FailureInjectionPolicy
getFileChecksum(Path, long) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: When enabled, get the etag of a object at the path via HEAD request and return it as a checksum object.
getFileMeta() - Method in class org.apache.hadoop.fs.s3a.s3guard.LocalMetadataEntry
getFilename() - Method in class org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit: This is the filename of the pending file itself.
getFilenames() - Method in class org.apache.hadoop.fs.s3a.commit.files.SuccessData
getFileStatus(Path) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Return a file status object that represents the path.
getFileStatus() - Method in class org.apache.hadoop.fs.s3a.s3guard.PathMetadata
getFileStatus(Path) - Method in class org.apache.hadoop.fs.s3native.NativeS3FileSystem: Deprecated.
getFilesystem() - Method in class org.apache.hadoop.fs.s3a.s3guard.S3GuardTool
getFinalKey(String, JobContext) - Method in class org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter: Returns the final S3 key for a relative path.
getFinalPath(String, JobContext) - Method in class org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter: Returns the final S3 location for a relative path as a Hadoop Path.
getHostname() - Method in class org.apache.hadoop.fs.s3a.commit.files.SuccessData
getInputPolicy() - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Get the input policy for this FS instance.
getInputPolicy() - Method in class org.apache.hadoop.fs.s3a.S3AReadOpContext: Get the IO policy.
getInstrumentation() - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Get S3A Instrumentation.
getInvoker() - Method in class org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
getJobAttemptFileSystem(JobContext) - Method in class org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter: Get the filesystem for the job attempt.
getJobAttemptPath(JobContext) - Method in class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter: Compute the path where the output of a given job attempt will be placed.
getJobAttemptPath(int) - Method in class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter: Compute the path where the output of a given job attempt will be placed.
getJobAttemptPath(int) - Method in class org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter: Compute the path where the output of a given job attempt will be placed.
getJobAttemptPath(JobContext, Path) - Static method in class org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter: Compute the path where the output of a given job attempt will be placed.
getJobAttemptPath(int) - Method in class org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter
getJobAttemptPath(JobContext) - Method in class org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter: For a job attempt path, the staging committer returns that of the wrapped committer.
getJobContext() - Method in class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter: Get the job/task context this committer was instantiated with.
getJobId() - Method in class org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit
getKey() - Method in class org.apache.hadoop.fs.s3a.InconsistentS3Object
getLength() - Method in class org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit: Destination file size.
getLength() - Method in class org.apache.hadoop.fs.s3a.UploadInfo
getListing() - Method in class org.apache.hadoop.fs.s3a.s3guard.DirListingMetadata
getLocalTaskAttemptTempDir(Configuration, String, TaskAttemptID) - Static method in class org.apache.hadoop.fs.s3a.commit.staging.Paths: Get the task attempt temporary directory in the local filesystem.
getLogNoVersionSupport() - Method in class org.apache.hadoop.fs.s3a.impl.ChangeDetectionPolicy
getLong(String) - Method in class org.apache.hadoop.fs.s3a.S3AStorageStatistics
getLongStatistics() - Method in class org.apache.hadoop.fs.s3a.S3AStorageStatistics
getMagicJobAttemptPath(int, Path) - Static method in class org.apache.hadoop.fs.s3a.commit.CommitUtilsWithMR: Compute the "magic" path for a job attempt.
getMagicJobAttemptsPath(Path) - Static method in class org.apache.hadoop.fs.s3a.commit.CommitUtilsWithMR: Get the location of magic job attempts.
getMagicTaskAttemptPath(TaskAttemptContext, Path) - Static method in class org.apache.hadoop.fs.s3a.commit.CommitUtilsWithMR: Compute the path where the output of a task attempt is stored until that task is committed.
getMagicTaskAttemptsPath(JobContext, Path) - Static method in class org.apache.hadoop.fs.s3a.commit.CommitUtilsWithMR: Compute the path where the output of magic task attempts are stored.
getMessage() - Method in exception org.apache.hadoop.fs.s3a.AWSClientIOException
getMetadataStore() - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Get the metadata store.
getMetadataStore(FileSystem) - Static method in class org.apache.hadoop.fs.s3a.s3guard.S3Guard: Create a new instance of the configured MetadataStore.
getMethod() - Method in enum org.apache.hadoop.fs.s3a.S3AEncryptionMethods
getMethod(String) - Static method in enum org.apache.hadoop.fs.s3a.S3AEncryptionMethods: Get the encryption mechanism from the value provided.
getMetrics() - Method in class org.apache.hadoop.fs.s3a.commit.files.SuccessData
getMetrics(MetricsCollector, boolean) - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation
getMetricsSystem() - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation
getMode() - Method in class org.apache.hadoop.fs.s3a.impl.ChangeDetectionPolicy
getModificationTime() - Method in class org.apache.hadoop.fs.s3a.S3AFileStatus: Get the modification time of the file/directory.
getMultipartSizeProperty(Configuration, String, long) - Static method in class org.apache.hadoop.fs.s3a.S3AUtils: Get a size property from the configuration: this property must be at least equal to Constants.MULTIPART_MIN_SIZE.
getMultiPartThreshold() - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Get the threshold for multipart files.
getMultipartUploadCommitsDirectory(Configuration, String) - Static method in class org.apache.hadoop.fs.s3a.commit.staging.Paths: Build a qualified temporary path for the multipart upload commit information in the cluster filesystem.
getName() - Method in class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter: Get the name of this committer.
getName() - Method in class org.apache.hadoop.fs.s3a.commit.files.SuccessData
getName() - Method in class org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter
getName() - Method in class org.apache.hadoop.fs.s3a.commit.staging.DirectoryStagingCommitter
getName() - Method in class org.apache.hadoop.fs.s3a.commit.staging.PartitionedStagingCommitter
getName() - Method in class org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter
getObject(GetObjectRequest) - Method in class org.apache.hadoop.fs.s3a.InconsistentAmazonS3Client
getObject(String, String) - Method in class org.apache.hadoop.fs.s3a.InconsistentAmazonS3Client
getObjectContent() - Method in class org.apache.hadoop.fs.s3a.InconsistentS3Object
getObjectMetadata() - Method in class org.apache.hadoop.fs.s3a.InconsistentS3Object
getObjectMetadata(Path) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Low-level call to get at the object metadata.
getObjectMetadata(String) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Request object metadata; increments counters in the process.
getObjectSummaries() - Method in class org.apache.hadoop.fs.s3a.S3ListResult
getOutputPath() - Method in class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter: Final path of output, in the destination FS.
getParent(String) - Static method in class org.apache.hadoop.fs.s3a.commit.staging.Paths: Get the parent path of a string path: everything up to but excluding the last "/" in the path.
getPartCount() - Method in class org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit: Get the number of etags.
getPartition(String) - Static method in class org.apache.hadoop.fs.s3a.commit.staging.Paths: Returns the partition of a relative file path, or null if the path is a file name with no relative directory.
getPartitions(Path, List<? extends FileStatus>) - Static method in class org.apache.hadoop.fs.s3a.commit.staging.Paths: Get the set of partitions from the list of files being staged.
getPartitionSize() - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Get the partition size for multipart operations.
getPassword() - Method in class org.apache.hadoop.fs.s3native.S3xLoginHelper.Login
getPath() - Method in class org.apache.hadoop.fs.s3a.S3AReadOpContext: Get the path of this read.
getPath() - Method in class org.apache.hadoop.fs.s3a.s3guard.DirListingMetadata
getPolicy(Configuration) - Static method in class org.apache.hadoop.fs.s3a.impl.ChangeDetectionPolicy: Reads the change detection policy from Configuration.
getPolicy(String) - Static method in enum org.apache.hadoop.fs.s3a.S3AInputPolicy: Choose an FS access policy.
getPos() - Method in class org.apache.hadoop.fs.s3a.S3AInputStream
getRawResponseContent() - Method in exception org.apache.hadoop.fs.s3a.AWSServiceIOException
getReadahead() - Method in class org.apache.hadoop.fs.s3a.S3AInputStream: Get the current readahead value.
getReadahead() - Method in class org.apache.hadoop.fs.s3a.S3AReadOpContext: Get the readahead for this operation.
getReadInvoker() - Method in class org.apache.hadoop.fs.s3a.S3AReadOpContext: Get invoker to use for read operations.
getReadThrottleEventCount() - Method in class org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore: Get the count of read throttle events.
getRedirectLocation() - Method in class org.apache.hadoop.fs.s3a.InconsistentS3Object
getRefCount() - Method in class org.apache.hadoop.fs.s3a.AWSCredentialProviderList: Get the current reference count.
getRegistry() - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation: Get the metrics registry.
getRelativePath(Path, Path) - Static method in class org.apache.hadoop.fs.s3a.commit.staging.Paths: Using URI#relativize(), build the relative path from the base path to the full path.
getRequestId() - Method in exception org.apache.hadoop.fs.s3a.AWSServiceIOException
getRetryCallback() - Method in class org.apache.hadoop.fs.s3a.Invoker
getRetryPolicy() - Method in class org.apache.hadoop.fs.s3a.Invoker
getRevisionId(ObjectMetadata, String) - Method in class org.apache.hadoop.fs.s3a.impl.ChangeDetectionPolicy: Pulls the attribute this policy uses to detect change out of the S3 object metadata.
getRevisionId() - Method in class org.apache.hadoop.fs.s3a.impl.ChangeTracker
getRole() - Method in class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter: Used in logging and reporting to help disentangle messages.
getS3AFileSystem(Path, Configuration, boolean) - Static method in class org.apache.hadoop.fs.s3a.commit.CommitUtils: Get the S3A FS of a path.
getS3AStreamStatistics() - Method in class org.apache.hadoop.fs.s3a.S3AInputStream: Access the input stream statistics.
getS3GuardInstrumentation() - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation: Create a S3Guard instrumentation instance.
getSaved() - Method in class org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit: When was the upload saved?
getScheme() - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Return the protocol scheme for the FileSystem.
getScheme() - Method in class org.apache.hadoop.fs.s3a.S3AStorageStatistics
getScheme() - Method in class org.apache.hadoop.fs.s3native.NativeS3FileSystem: Deprecated.

Return the protocol scheme for the FileSystem.
getServerSideEncryptionAlgorithm() - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Get the encryption algorithm of this endpoint.
getServiceName() - Method in exception org.apache.hadoop.fs.s3a.AWSServiceIOException
getSource() - Method in class org.apache.hadoop.fs.s3a.impl.ChangeDetectionPolicy
getSource() - Method in class org.apache.hadoop.fs.s3a.impl.ChangeTracker
getStatistics() - Method in class org.apache.hadoop.fs.s3a.commit.CommitOperations
getStatusCode() - Method in exception org.apache.hadoop.fs.s3a.AWSServiceIOException
getStorageStatistics() - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Get the storage statistics of this filesystem.
getStore() - Method in class org.apache.hadoop.fs.s3a.s3guard.S3GuardTool
getSymbol() - Method in enum org.apache.hadoop.fs.s3a.Statistic
getTableName() - Method in class org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
getTaggingCount() - Method in class org.apache.hadoop.fs.s3a.InconsistentS3Object
getTaskAttemptFilesystem(TaskAttemptContext) - Method in class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter: Get the task attempt path filesystem.
getTaskAttemptPath(TaskAttemptContext) - Method in class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter: Compute the path where the output of a task attempt is stored until that task is committed.
getTaskAttemptPath(TaskAttemptContext) - Method in class org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter: Compute the path where the output of a task attempt is stored until that task is committed.
getTaskAttemptPath(TaskAttemptContext, Path) - Static method in class org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter: Compute the path where the output of a task attempt is stored until that task is committed.
getTaskId() - Method in class org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit
getTaskOutput(TaskAttemptContext) - Method in class org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter: Lists the output of a task under the task attempt path.
getTempJobAttemptPath(int, Path) - Static method in class org.apache.hadoop.fs.s3a.commit.CommitUtilsWithMR: Compute a path for temporary data associated with a job.
getTempTaskAttemptPath(TaskAttemptContext) - Method in class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter: Get a temporary directory for data.
getTempTaskAttemptPath(TaskAttemptContext, Path) - Static method in class org.apache.hadoop.fs.s3a.commit.CommitUtilsWithMR: Compute the path where the output of a given job attempt will be placed.
getTempTaskAttemptPath(TaskAttemptContext) - Method in class org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter: Get a temporary directory for data.
getTempTaskAttemptPath(TaskAttemptContext) - Method in class org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter
getText() - Method in class org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit: Arbitrary notes.
getThrottleProbability() - Method in class org.apache.hadoop.fs.s3a.FailureInjectionPolicy
getTimestamp() - Method in class org.apache.hadoop.fs.s3a.commit.files.SuccessData
getUpload() - Method in class org.apache.hadoop.fs.s3a.UploadInfo
getUploadId() - Method in class org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit
getUploadUUID(Configuration, String) - Static method in class org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter: Get the UUID of an upload; may be the job ID.
getUploadUUID(Configuration, JobID) - Static method in class org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter: Get the UUID of a Job.
getUri() - Method in class org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit
getUri() - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Returns a URI whose scheme and authority identify this FileSystem.
getUri() - Method in class org.apache.hadoop.fs.s3native.NativeS3FileSystem: Deprecated.
getUriDefaultPort() - Method in class org.apache.hadoop.fs.s3a.S3A
getUsage() - Method in class org.apache.hadoop.fs.s3a.s3guard.S3GuardTool
getUser() - Method in class org.apache.hadoop.fs.s3native.S3xLoginHelper.Login
getUsername() - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Get the username of the FS.
getUUID() - Method in class org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter: A UUID for this upload, as calculated with.
getV1() - Method in class org.apache.hadoop.fs.s3a.S3ListRequest
getV1() - Method in class org.apache.hadoop.fs.s3a.S3ListResult
getV2() - Method in class org.apache.hadoop.fs.s3a.S3ListRequest
getV2() - Method in class org.apache.hadoop.fs.s3a.S3ListResult
getVersion() - Method in class org.apache.hadoop.fs.s3a.commit.files.PendingSet
getVersion() - Method in class org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit
getVersionMismatchCounter() - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation.InputStreamStatistics: Get a reference to the version mismatch counter.
getVersionMismatches() - Method in class org.apache.hadoop.fs.s3a.impl.ChangeTracker
getWorkingDirectory() - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Get the current working directory for the given file system.
getWorkingDirectory() - Method in class org.apache.hadoop.fs.s3native.NativeS3FileSystem: Deprecated.
getWorkPath() - Method in class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter: This is the critical method for FileOutputFormat; it declares the path for work.
getWriteOperationHelper() - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Get a WriteOperationHelper instance.
getWriteThrottleEventCount() - Method in class org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore: Get the count of write throttle events.
globStatus(Path) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Increments the statistic Statistic.INVOCATION_GLOB_STATUS.
globStatus(Path, PathFilter) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Override superclass so as to add statistic collection.

H

HADOOP_TMP_DIR - Static variable in class org.apache.hadoop.fs.s3a.Constants: default hadoop temp dir on local system: "hadoop.tmp.dir".
hasCapability(String) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Return the capabilities of this filesystem instance.
hasDirMeta() - Method in class org.apache.hadoop.fs.s3a.s3guard.LocalMetadataEntry
hasException() - Method in class org.apache.hadoop.fs.s3a.commit.CommitOperations.MaybeIOE: Is there an exception in this class?
hashCode() - Method in class org.apache.hadoop.fs.s3a.S3AFileStatus: Returns a hash code value for the object, which is defined as the hash code of the path name.
hashCode() - Method in class org.apache.hadoop.fs.s3a.s3guard.DDBPathMetadata
hashCode() - Method in class org.apache.hadoop.fs.s3a.s3guard.PathMetadata
hashCode() - Method in class org.apache.hadoop.fs.s3native.S3xLoginHelper.Login
hasLogin() - Method in class org.apache.hadoop.fs.s3native.S3xLoginHelper.Login: Predicate to verify login details are defined.
hasMetadataStore() - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Does this Filesystem have a metadata store?
hasNext() - Method in class org.apache.hadoop.fs.s3a.MultipartUtils.UploadIterator
hasNext() - Method in class org.apache.hadoop.fs.s3a.s3guard.DescendantsIterator
hasNext() - Method in class org.apache.hadoop.fs.s3a.s3guard.MetadataStoreListFilesIterator
hasPathMeta() - Method in class org.apache.hadoop.fs.s3a.s3guard.LocalMetadataEntry
HEADER - Static variable in class org.apache.hadoop.fs.s3a.S3AMultipartUploader: Header for Parts: "S3A-part01".
HIDDEN_FILE_FILTER - Static variable in class org.apache.hadoop.fs.s3a.S3AUtils: Path filter which ignores any file which starts with .
HOURS_FLAG - Static variable in class org.apache.hadoop.fs.s3a.s3guard.S3GuardTool
humanTime(long) - Static method in class org.apache.hadoop.fs.s3a.commit.Duration

I

ignoreIOExceptions(Logger, String, String, Invoker.Operation<T>) - Static method in class org.apache.hadoop.fs.s3a.Invoker: Execute an operation and ignore all raised IOExceptions; log at INFO; full stack only at DEBUG.
ignoreIOExceptions(Logger, String, String, Invoker.VoidOperation) - Static method in class org.apache.hadoop.fs.s3a.Invoker: Execute an operation and ignore all raised IOExceptions; log at INFO; full stack only at DEBUG.
InconsistentAmazonS3Client - Class in org.apache.hadoop.fs.s3a: A wrapper around AmazonS3 that injects inconsistency and/or errors.
InconsistentAmazonS3Client(AWSCredentialsProvider, ClientConfiguration, Configuration) - Constructor for class org.apache.hadoop.fs.s3a.InconsistentAmazonS3Client: Instantiate.
InconsistentS3ClientFactory - Class in org.apache.hadoop.fs.s3a: S3 Client factory used for testing with eventual consistency fault injection.
InconsistentS3ClientFactory() - Constructor for class org.apache.hadoop.fs.s3a.InconsistentS3ClientFactory
InconsistentS3InputStream(S3ObjectInputStream) - Constructor for class org.apache.hadoop.fs.s3a.InconsistentS3Object.InconsistentS3InputStream
InconsistentS3Object - Class in org.apache.hadoop.fs.s3a: Wrapper around S3Object so we can do failure injection on getObjectContent() and S3ObjectInputStream.
InconsistentS3Object(S3Object, FailureInjectionPolicy) - Constructor for class org.apache.hadoop.fs.s3a.InconsistentS3Object
InconsistentS3Object.InconsistentS3InputStream - Class in org.apache.hadoop.fs.s3a: Wraps S3ObjectInputStream and implements failure injection.
incrementCounter(Statistic, long) - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation: Increment a specific counter.
incrementCounter(Statistic, AtomicLong) - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation: Increment a specific counter.
incrementCounter(Statistic, long) - Method in class org.apache.hadoop.fs.s3a.S3AStorageStatistics: Increment a specific counter.
incrementGauge(Statistic, long) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Increment a gauge by a specific value.
incrementGauge(Statistic, long) - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation: Increment a specific gauge.
incrementPutCompletedStatistics(boolean, long) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: At the end of a put/multipart upload operation, update the relevant counters and gauges.
incrementPutProgressStatistics(String, long) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Callback for use in progress callbacks from put/multipart upload events.
incrementPutStartStatistics(long) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: At the start of a put/multipart upload operation, update the relevant counters.
incrementReadOperations() - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Increment read operations.
incrementStatistic(Statistic) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Increment a statistic by 1.
incrementStatistic(Statistic, long) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Increment a statistic by a specific value.
incrementWriteOperations() - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Increment the write operation counter.
initConnectionSettings(Configuration, ClientConfiguration) - Static method in class org.apache.hadoop.fs.s3a.S3AUtils: Initializes all AWS SDK settings related to connection management.
initFileOutputCommitterOptions(JobContext) - Method in class org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter: Init the context config with everything needed for the file output committer.
initialize() - Method in class org.apache.hadoop.fs.s3a.commit.magic.MagicCommitTracker: Initialize the tracker.
initialize() - Method in class org.apache.hadoop.fs.s3a.commit.PutTracker: Startup event.
initialize(URI, Configuration) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Called after a new FileSystem instance is constructed.
initialize(Path) - Method in class org.apache.hadoop.fs.s3a.S3AMultipartUploader
initialize(FileSystem) - Method in class org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore: Performs one-time initialization of the metadata store.
initialize(Configuration) - Method in class org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore: Performs one-time initialization of the metadata store via configuration.
initialize(FileSystem) - Method in class org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore
initialize(Configuration) - Method in class org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore
initialize(FileSystem) - Method in interface org.apache.hadoop.fs.s3a.s3guard.MetadataStore: Performs one-time initialization of the metadata store.
initialize(Configuration) - Method in interface org.apache.hadoop.fs.s3a.s3guard.MetadataStore: Performs one-time initialization of the metadata store via configuration.
initialize(FileSystem) - Method in class org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore
initialize(Configuration) - Method in class org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore
initialize(URI, Configuration) - Method in class org.apache.hadoop.fs.s3native.NativeS3FileSystem: Deprecated.

Always fail to initialize.
initialized() - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation.S3GuardInstrumentation: Initialized event.
initiateMultipartUpload(InitiateMultipartUploadRequest) - Method in class org.apache.hadoop.fs.s3a.InconsistentAmazonS3Client
initiateMultiPartUpload(String) - Method in class org.apache.hadoop.fs.s3a.WriteOperationHelper: Start the multipart upload process.
initOutput(Path) - Method in class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter: Init the output filesystem and path.
initProxySupport(Configuration, String, ClientConfiguration) - Static method in class org.apache.hadoop.fs.s3a.S3AUtils: Initializes AWS SDK proxy support in the AWS client configuration if the S3A settings enable it.
innerListStatus(Path) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: List the statuses of the files/directories in the given path if the path is a directory.
INPUT_FADV_NORMAL - Static variable in class org.apache.hadoop.fs.s3a.Constants: General input.
INPUT_FADV_RANDOM - Static variable in class org.apache.hadoop.fs.s3a.Constants: Optimized purely for random seek+read/positionedRead operations; The performance of sequential IO may be reduced in exchange for more efficient seek() operations.
INPUT_FADV_SEQUENTIAL - Static variable in class org.apache.hadoop.fs.s3a.Constants: Optimized for sequential access.
INPUT_FADVISE - Static variable in class org.apache.hadoop.fs.s3a.Constants: Which input strategy to use for buffering, seeking and similar when reading data.
inputPolicy - Variable in class org.apache.hadoop.fs.s3a.S3AInstrumentation.InputStreamStatistics
inputPolicySet(int) - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation.InputStreamStatistics: The input policy has been switched.
InternalCommitterConstants - Class in org.apache.hadoop.fs.s3a.commit: These are internal constants not intended for public use.
invalidArgs(String, Object...) - Static method in class org.apache.hadoop.fs.s3a.s3guard.S3GuardTool: Build the exception to raise on invalid arguments.
Invoker - Class in org.apache.hadoop.fs.s3a: Class to provide lambda expression invocation of AWS operations.
Invoker(RetryPolicy, Invoker.Retried) - Constructor for class org.apache.hadoop.fs.s3a.Invoker: Instantiate.
Invoker.Operation<T> - Interface in org.apache.hadoop.fs.s3a: Arbitrary operation throwing an IOException.
Invoker.Retried - Interface in org.apache.hadoop.fs.s3a: Callback for retry and notification operations.
Invoker.VoidOperation - Interface in org.apache.hadoop.fs.s3a: Void operation which may raise an IOException.
isAuthoritative() - Method in class org.apache.hadoop.fs.s3a.s3guard.DirListingMetadata
isAuthoritativeDir() - Method in class org.apache.hadoop.fs.s3a.s3guard.DDBPathMetadata
isClosed() - Method in class org.apache.hadoop.fs.s3a.AWSCredentialProviderList: Get the closed flag.
isDeleted() - Method in class org.apache.hadoop.fs.s3a.s3guard.PathMetadata
isDirectory(Path) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Override superclass so as to add statistic collection.
isEmpty() - Method in class org.apache.hadoop.fs.s3a.s3guard.DirListingMetadata: Is the underlying directory known to be empty?
isEmptyDirectory() - Method in class org.apache.hadoop.fs.s3a.S3AFileStatus
isEmptyDirectory() - Method in class org.apache.hadoop.fs.s3a.s3guard.PathMetadata: Query if a directory is empty.
isFile(Path) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Override superclass so as to add statistic collection.
isMagicCommitEnabled() - Method in class org.apache.hadoop.fs.s3a.commit.MagicCommitIntegration: Is magic commit enabled?
isMagicCommitEnabled() - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Is magic commit enabled?
isMagicCommitPath(Path) - Method in class org.apache.hadoop.fs.s3a.commit.MagicCommitIntegration: Predicate: is a path a magic commit path?
isMagicCommitPath(Path) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Predicate: is a path a magic commit path? True if magic commit is enabled and the path qualifies as special.
isMagicPath(List<String>) - Static method in class org.apache.hadoop.fs.s3a.commit.MagicCommitPaths: Is a path in the magic tree?
isNullMetadataStore(MetadataStore) - Static method in class org.apache.hadoop.fs.s3a.s3guard.S3Guard: Although NullMetadataStore does nothing, callers may wish to avoid work (fast path) when the NullMetadataStore is in use.
isRequesterCharged() - Method in class org.apache.hadoop.fs.s3a.InconsistentS3Object
isRequireVersion() - Method in class org.apache.hadoop.fs.s3a.impl.ChangeDetectionPolicy
isRetryable() - Method in exception org.apache.hadoop.fs.s3a.AWSServiceIOException
isRetryable() - Method in exception org.apache.hadoop.fs.s3a.CredentialInitializationException: This exception is not going to go away if you try calling it again.
isThrottleException(Exception) - Static method in class org.apache.hadoop.fs.s3a.S3AUtils: Is the exception an instance of a throttling exception.
isTracked(String) - Method in class org.apache.hadoop.fs.s3a.S3AStorageStatistics
isTruncated() - Method in class org.apache.hadoop.fs.s3a.S3ListResult
isV1() - Method in class org.apache.hadoop.fs.s3a.S3ListRequest: Is this a v1 API request or v2?
isV1() - Method in class org.apache.hadoop.fs.s3a.S3ListResult: Is this a v1 API result or v2?
iterator() - Method in class org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit: Iterate over the etags.
iterator() - Method in class org.apache.hadoop.fs.s3a.S3AStorageStatistics

J

JAVA_IO_TMPDIR - Static variable in class org.apache.hadoop.fs.s3a.commit.staging.StagingCommitterConstants
jobCompleted(boolean) - Method in class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter: Job completion outcome; this may be subclassed in tests.
jobCompleted(boolean) - Method in class org.apache.hadoop.fs.s3a.commit.CommitOperations: Note that a job has completed.
jobCompleted(boolean) - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation.CommitterStatistics
jobIdString(JobContext) - Static method in class org.apache.hadoop.fs.s3a.commit.CommitUtilsWithMR: Get a string value of a job ID; returns meaningful text if there is no ID.
jobName(JobContext) - Static method in class org.apache.hadoop.fs.s3a.commit.CommitUtilsWithMR: Get a job name; returns meaningful text if there is no name.
joinMap(Map<String, ?>, String, String, String) - Static method in class org.apache.hadoop.fs.s3a.commit.files.SuccessData: Join any map of string to value into a string, sorting the keys first.

K

KEEPALIVE_TIME - Static variable in class org.apache.hadoop.fs.s3a.Constants
keyOfFinalDestination(List<String>, String) - Method in class org.apache.hadoop.fs.s3a.commit.MagicCommitIntegration: Given an (elements, key) pair, return the key of the final destination of the PUT, that is: where the final path is expected to go?
keyToQualifiedPath(String) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Convert a key to a fully qualified path.
KMS_ALL_KEYS - Static variable in class org.apache.hadoop.fs.s3a.auth.RolePolicies: Arn for all KMS keys: "arn:aws:kms:*".
KMS_ALL_OPERATIONS - Static variable in class org.apache.hadoop.fs.s3a.auth.RolePolicies: All KMS operations: "kms:*".
KMS_DECRYPT - Static variable in class org.apache.hadoop.fs.s3a.auth.RolePolicies: Decrypt data encrypted with SSE-KMS: "kms:Decrypt".
KMS_ENCRYPT - Static variable in class org.apache.hadoop.fs.s3a.auth.RolePolicies: KMS encryption.
KMS_GENERATE_DATA_KEY - Static variable in class org.apache.hadoop.fs.s3a.auth.RolePolicies: This is used by S3 to generate a per-object encryption key and the encrypted value of this, the latter being what it tags the object with for later decryption: "kms:GenerateDataKey".

L

lastElement(List<String>) - Static method in class org.apache.hadoop.fs.s3a.commit.MagicCommitPaths: Last element of a (non-empty) list.
LIST_VERSION - Static variable in class org.apache.hadoop.fs.s3a.Constants: S3 API level parameters.
listAndFilter(FileSystem, Path, boolean, PathFilter) - Static method in class org.apache.hadoop.fs.s3a.S3AUtils: List located files and filter them as a classic listFiles(path, filter) would do.
listChildren(Path) - Method in class org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
listChildren(Path) - Method in class org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore
listChildren(Path) - Method in interface org.apache.hadoop.fs.s3a.s3guard.MetadataStore: Lists metadata for all direct children of a path.
listChildren(Path) - Method in class org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore
listFiles(Path, boolean) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: .
listFilesAndEmptyDirectories(Path, boolean) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem
Listing - Class in org.apache.hadoop.fs.s3a: Place for the S3A listing classes; keeps all the small classes under control.
Listing(S3AFileSystem) - Constructor for class org.apache.hadoop.fs.s3a.Listing
listLocatedStatus(Path) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Override superclass so as to add statistic collection.
listLocatedStatus(Path, PathFilter) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: .
listMultipartUploads(ListMultipartUploadsRequest) - Method in class org.apache.hadoop.fs.s3a.InconsistentAmazonS3Client
listMultipartUploads(String) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Listing all multipart uploads; limited to the first few hundred.
listObjects(ListObjectsRequest) - Method in class org.apache.hadoop.fs.s3a.InconsistentAmazonS3Client
listObjects(S3ListRequest) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Initiate a listObjects operation, incrementing metrics in the process.
listObjectsV2(ListObjectsV2Request) - Method in class org.apache.hadoop.fs.s3a.InconsistentAmazonS3Client
listPendingUploads(JobContext, boolean) - Method in class org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter: Get the list of pending uploads for this job attempt.
listPendingUploadsToAbort(JobContext) - Method in class org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter: Get the list of pending uploads for this job attempt, swallowing exceptions.
listPendingUploadsToCommit(JobContext) - Method in class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter: Get the list of pending uploads for this job attempt.
listPendingUploadsToCommit(JobContext) - Method in class org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter: Get the list of pending uploads for this job attempt, by listing all .pendingset files in the job attempt directory.
listPendingUploadsToCommit(JobContext) - Method in class org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter: Get the list of pending uploads for this job attempt.
listPendingUploadsUnderPath(Path) - Method in class org.apache.hadoop.fs.s3a.commit.CommitOperations: List all pending uploads to the destination FS under a path.
listProviderNames() - Method in class org.apache.hadoop.fs.s3a.AWSCredentialProviderList: List all the providers' names.
listStatus(Path) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: List the statuses of the files/directories in the given path if the path is a directory.
listStatus(Path) - Method in class org.apache.hadoop.fs.s3native.NativeS3FileSystem: Deprecated.
listTombstones() - Method in class org.apache.hadoop.fs.s3a.s3guard.DirListingMetadata
listTombstones() - Method in class org.apache.hadoop.fs.s3a.s3guard.MetadataStoreListFilesIterator
listUploads(String) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: List any pending multipart uploads whose keys begin with prefix, using an iterator that can handle an unlimited number of entries.
load(FileSystem, Path) - Static method in class org.apache.hadoop.fs.s3a.commit.files.PendingSet: Load an instance from a file, then validate it.
load(FileSystem, Path) - Static method in class org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit: Load an instance from a file, then validate it.
load(FileSystem, Path) - Static method in class org.apache.hadoop.fs.s3a.commit.files.SuccessData: Load an instance from a file, then validate it.
loadAWSProviderClasses(Configuration, String, Class<?>...) - Static method in class org.apache.hadoop.fs.s3a.S3AUtils: Load list of AWS credential provider/credential provider factory classes.
loadPendingsetFiles(JobContext, boolean, FileSystem, Iterable<? extends FileStatus>) - Method in class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter: Try to read every pendingset file and build a list of them/ In the case of a failure to read the file, exceptions are held until all reads have been attempted.
loadSinglePendingCommits(Path, boolean) - Method in class org.apache.hadoop.fs.s3a.commit.CommitOperations: Load all single pending commits in the directory.
LocalMetadataEntry - Class in org.apache.hadoop.fs.s3a.s3guard: LocalMetadataEntry is used to store entries in the cache of LocalMetadataStore.
LocalMetadataStore - Class in org.apache.hadoop.fs.s3a.s3guard: This is a local, in-memory implementation of MetadataStore.
LocalMetadataStore() - Constructor for class org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore
locateAllSinglePendingCommits(Path, boolean) - Method in class org.apache.hadoop.fs.s3a.commit.CommitOperations: Locate all files with the pending suffix under a directory.
LOG - Static variable in class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitterFactory
LOG - Static variable in class org.apache.hadoop.fs.s3a.commit.magic.MagicCommitTracker
LOG - Static variable in class org.apache.hadoop.fs.s3a.DefaultS3ClientFactory
LOG - Static variable in class org.apache.hadoop.fs.s3a.S3AFileSystem
LOG - Static variable in interface org.apache.hadoop.fs.s3a.s3guard.DynamoDBClientFactory
LOG - Static variable in class org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
LOG - Static variable in class org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore
LOG - Static variable in class org.apache.hadoop.fs.s3a.s3guard.MetadataStoreListFilesIterator
LOG - Static variable in class org.apache.hadoop.fs.s3native.NativeS3FileSystem: Deprecated.
LOG_EVENT - Static variable in class org.apache.hadoop.fs.s3a.Invoker: Log summary at info, full stack at debug.
LogExactlyOnce - Class in org.apache.hadoop.fs.s3a.impl: Log exactly once, even across threads.
LogExactlyOnce(Logger) - Constructor for class org.apache.hadoop.fs.s3a.impl.LogExactlyOnce
Login() - Constructor for class org.apache.hadoop.fs.s3native.S3xLoginHelper.Login: Create an instance with no login details.
Login(String, String) - Constructor for class org.apache.hadoop.fs.s3native.S3xLoginHelper.Login
LOGIN_WARNING - Static variable in class org.apache.hadoop.fs.s3native.S3xLoginHelper
longBytesOption(Configuration, String, long, long) - Static method in class org.apache.hadoop.fs.s3a.S3AUtils: Get a long option >= the minimum allowed value, supporting memory prefixes K,M,G,T,P.
lookupGauge(String) - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation: Look up a gauge.
lookupMetric(String) - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation: Look up a metric from both the registered set and the lighter weight stream entries.
lookupPassword(String, Configuration, String, String) - Static method in class org.apache.hadoop.fs.s3a.S3AUtils: Deprecated.
lookupPassword(String, Configuration, String) - Static method in class org.apache.hadoop.fs.s3a.S3AUtils: Get a password from a configuration, including JCEKS files, handling both the absolute key and bucket override.
lookupPassword(String, Configuration, String, String, String) - Static method in class org.apache.hadoop.fs.s3a.S3AUtils: Get a password from a configuration, including JCEKS files, handling both the absolute key and bucket override.
lookupQuantiles(String) - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation: Look up a quantiles.
ls(Path, boolean) - Method in class org.apache.hadoop.fs.s3a.commit.CommitOperations: List files.

M

MAGIC - Static variable in class org.apache.hadoop.fs.s3a.commit.CommitConstants: Path for "magic" writes: path and CommitConstants.PENDING_SUFFIX files: "__magic".
MAGIC_COMMITTER_ENABLED - Static variable in class org.apache.hadoop.fs.s3a.commit.CommitConstants: Flag to indicate whether support for the Magic committer is enabled in the filesystem.
MAGIC_COMMITTER_FACTORY - Static variable in class org.apache.hadoop.fs.s3a.commit.InternalCommitterConstants: Magic committer factory: "org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitterFactory".
MAGIC_COMMITTER_PREFIX - Static variable in class org.apache.hadoop.fs.s3a.commit.CommitConstants: Flag to indicate whether support for the Magic committer is enabled in the filesystem.
MagicCommitIntegration - Class in org.apache.hadoop.fs.s3a.commit: Adds the code needed for S3A to support magic committers.
MagicCommitIntegration(S3AFileSystem, boolean) - Constructor for class org.apache.hadoop.fs.s3a.commit.MagicCommitIntegration: Instantiate.
MagicCommitPaths - Class in org.apache.hadoop.fs.s3a.commit: Operations on (magic) paths.
MagicCommitTracker - Class in org.apache.hadoop.fs.s3a.commit.magic: Put tracker for Magic commits.
MagicCommitTracker(Path, String, String, String, String, WriteOperationHelper) - Constructor for class org.apache.hadoop.fs.s3a.commit.magic.MagicCommitTracker: Magic commit tracker.
magicElementIndex(List<String>) - Static method in class org.apache.hadoop.fs.s3a.commit.MagicCommitPaths: Get the index of the magic path element.
magicPathChildren(List<String>) - Static method in class org.apache.hadoop.fs.s3a.commit.MagicCommitPaths: Get the child path elements under the magic path.
magicPathParents(List<String>) - Static method in class org.apache.hadoop.fs.s3a.commit.MagicCommitPaths: Get the parent path elements of the magic path.
MagicS3GuardCommitter - Class in org.apache.hadoop.fs.s3a.commit.magic: This is a dedicated committer which requires the "magic" directory feature of the S3A Filesystem to be enabled; it then uses paths for task and job attempts in magic paths, so as to ensure that the final output goes direct to the destination directory.
MagicS3GuardCommitter(Path, TaskAttemptContext) - Constructor for class org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter: Create a task committer.
MagicS3GuardCommitterFactory - Class in org.apache.hadoop.fs.s3a.commit.magic: Factory for the Magic committer.
MagicS3GuardCommitterFactory() - Constructor for class org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitterFactory
magicSubdir(Path) - Static method in class org.apache.hadoop.fs.s3a.commit.MagicCommitPaths: Get the magic subdirectory of a destination directory.
main(String[]) - Static method in class org.apache.hadoop.fs.s3a.s3guard.S3GuardTool: Main entry point.
makeDirsOrdered(MetadataStore, List<Path>, String, boolean) - Static method in class org.apache.hadoop.fs.s3a.s3guard.S3Guard: Deprecated.
this is no longer called by S3AFilesystem.innerMkDirs. See: HADOOP-15079 (January 2018). It is currently retained because of its discussion in the method on atomicity and in case we need to reinstate it or adapt the current process of directory marker creation. But it is not being tested and so may age with time...consider deleting it in future if it's clear there's no need for it.
makeIOE(String, Exception) - Method in class org.apache.hadoop.fs.s3a.commit.CommitOperations: Convert any exception to an IOE, if needed.
mapLocatedFiles(RemoteIterator<LocatedFileStatus>, S3AUtils.LocatedFileStatusMap<T>) - Static method in class org.apache.hadoop.fs.s3a.S3AUtils: Map an operation to every LocatedFileStatus in a remote iterator, returning a list of the results.
markDeleted(Path) - Method in class org.apache.hadoop.fs.s3a.s3guard.DirListingMetadata: Replace an entry with a tombstone.
markSupported() - Method in class org.apache.hadoop.fs.s3a.S3AInputStream
MATCH_ALL_KEYS - Static variable in class org.apache.hadoop.fs.s3a.FailureInjectionPolicy: Special config value since we can't store empty strings in XML.
MAX_ERROR_RETRIES - Static variable in class org.apache.hadoop.fs.s3a.Constants
MAX_MULTIPART_COUNT - Static variable in class org.apache.hadoop.fs.s3a.Constants: Maximum number of partitions in a multipart upload: 10000.
MAX_PAGING_KEYS - Static variable in class org.apache.hadoop.fs.s3a.Constants
MAX_READ_FAILURES - Static variable in class org.apache.hadoop.fs.s3a.InconsistentS3Object
MAX_THREADS - Static variable in class org.apache.hadoop.fs.s3a.Constants
MAX_TOTAL_TASKS - Static variable in class org.apache.hadoop.fs.s3a.Constants
MAXIMUM_CONNECTIONS - Static variable in class org.apache.hadoop.fs.s3a.Constants
maybe(boolean, T) - Static method in class org.apache.hadoop.fs.s3a.S3AUtils: Convert a value into a non-empty Optional instance if the value of include is true.
maybeApplyConstraint(GetObjectRequest) - Method in class org.apache.hadoop.fs.s3a.impl.ChangeTracker: Apply any revision control set by the policy if it is to be enforced on the server.
maybeCreateSuccessMarker(JobContext, List<String>) - Method in class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter: if the job requires a success marker on a successful job, create the file CommitConstants._SUCCESS.
maybeCreateSuccessMarkerFromCommits(JobContext, List<SinglePendingCommit>) - Method in class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter: if the job requires a success marker on a successful job, create the file CommitConstants._SUCCESS.
maybeIgnore(boolean, String, Invoker.VoidOperation) - Method in class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter: Execute an operation; maybe suppress any raised IOException.
maybeIgnore(boolean, String, IOException) - Method in class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter: Log or rethrow a caught IOException.
MaybeIOE(IOException) - Constructor for class org.apache.hadoop.fs.s3a.commit.CommitOperations.MaybeIOE: Construct with an exception.
maybeRethrow() - Method in class org.apache.hadoop.fs.s3a.commit.CommitOperations.MaybeIOE: Rethrow any exception.
META_FLAG - Static variable in class org.apache.hadoop.fs.s3a.s3guard.S3GuardTool
MetadataStore - Interface in org.apache.hadoop.fs.s3a.s3guard: MetadataStore defines the set of operations that any metadata store implementation must provide.
METADATASTORE_AUTHORITATIVE - Static variable in class org.apache.hadoop.fs.s3a.Constants: Whether or not to allow MetadataStore to be source of truth.
MetadataStoreCapabilities - Class in org.apache.hadoop.fs.s3a.s3guard: All the capability constants used for the MetadataStore implementations.
MetadataStoreListFilesIterator - Class in org.apache.hadoop.fs.s3a.s3guard: MetadataStoreListFilesIterator is a RemoteIterator that is similar to DescendantsIterator but does not return directories that have (or may have) children, and will also provide access to the set of tombstones to allow recently deleted S3 objects to be filtered out from a corresponding request.
MetadataStoreListFilesIterator(MetadataStore, PathMetadata, boolean) - Constructor for class org.apache.hadoop.fs.s3a.s3guard.MetadataStoreListFilesIterator
metastoreOperationRetried(Exception, int, boolean) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Callback from Invoker when an operation against a metastore is retried.
METRIC_TAG_BUCKET - Static variable in class org.apache.hadoop.fs.s3a.S3AInstrumentation: "bucket" The name of a field added to metrics records that indicates the hostname portion of the FS URL.
METRIC_TAG_FILESYSTEM_ID - Static variable in class org.apache.hadoop.fs.s3a.S3AInstrumentation: "s3aFileSystemId" The name of a field added to metrics records that uniquely identifies a specific FileSystem instance.
METRICS_SYSTEM_NAME - Static variable in class org.apache.hadoop.fs.s3a.S3AInstrumentation: "s3a-file-system" The name of the s3a-specific metrics system instance used for s3a metrics.
MIN_MULTIPART_THRESHOLD - Static variable in class org.apache.hadoop.fs.s3a.Constants
MINUTES_FLAG - Static variable in class org.apache.hadoop.fs.s3a.s3guard.S3GuardTool
mkdirs(Path, FsPermission) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Make the given path and all non-existent parents into directories.
mkdirs(Path, FsPermission) - Method in class org.apache.hadoop.fs.s3native.NativeS3FileSystem: Deprecated.
move(Collection<Path>, Collection<PathMetadata>) - Method in class org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
move(Collection<Path>, Collection<PathMetadata>) - Method in class org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore
move(Collection<Path>, Collection<PathMetadata>) - Method in interface org.apache.hadoop.fs.s3a.s3guard.MetadataStore: Record the effects of a FileSystem.rename(Path, Path) in the MetadataStore.
move(Collection<Path>, Collection<PathMetadata>) - Method in class org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore
MULTIPART_MIN_SIZE - Static variable in class org.apache.hadoop.fs.s3a.Constants: The minimum multipart size which S3 supports.
MULTIPART_SIZE - Static variable in class org.apache.hadoop.fs.s3a.Constants
MultipartUtils - Class in org.apache.hadoop.fs.s3a: MultipartUtils upload-specific functions for use by S3AFileSystem and Hadoop CLI.
MultipartUtils.UploadIterator - Class in org.apache.hadoop.fs.s3a: Iterator over multipart uploads.

N

NAME - Static variable in class org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider
NAME - Static variable in class org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider
NAME - Static variable in class org.apache.hadoop.fs.s3a.commit.files.SuccessData: Name to include in persisted data, so as to differentiate from any other manifests: "org.apache.hadoop.fs.s3a.commit.files.SuccessData/1".
NAME - Static variable in class org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter: Name: "magic".
NAME - Static variable in class org.apache.hadoop.fs.s3a.commit.staging.DirectoryStagingCommitter: Name: "directory".
NAME - Static variable in class org.apache.hadoop.fs.s3a.commit.staging.PartitionedStagingCommitter: Name: "partitioned".
NAME - Static variable in class org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter: Name: "staging".
NAME - Static variable in class org.apache.hadoop.fs.s3a.S3AStorageStatistics
NAME - Static variable in class org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider
NAME - Static variable in class org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider
NativeS3FileSystem - Class in org.apache.hadoop.fs.s3native: Deprecated.
Replaced by the S3A client.
NativeS3FileSystem() - Constructor for class org.apache.hadoop.fs.s3native.NativeS3FileSystem: Deprecated.
needsTaskCommit(TaskAttemptContext) - Method in class org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter: Did this task write any files in the work directory? Probes for a task existing by looking to see if the attempt dir exists.
needsTaskCommit(TaskAttemptContext) - Method in class org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter
newAmazonS3Client(AWSCredentialsProvider, ClientConfiguration) - Method in class org.apache.hadoop.fs.s3a.DefaultS3ClientFactory: Wrapper around constructor for AmazonS3 client.
newAmazonS3Client(AWSCredentialsProvider, ClientConfiguration) - Method in class org.apache.hadoop.fs.s3a.InconsistentS3ClientFactory: Create the inconsistent client.
newCommitterStatistics() - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Create a new instance of the committer statistics.
newObjectMetadata() - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Create a new object metadata instance.
newObjectMetadata(long) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Create a new object metadata instance.
newObjectMetadata(long) - Method in class org.apache.hadoop.fs.s3a.WriteOperationHelper: Create a new object metadata instance.
newPutObjectRequest(String, ObjectMetadata, File) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Create a putObject request.
newSid() - Static method in class org.apache.hadoop.fs.s3a.auth.RoleModel: Statement ID factory.
newUploadPartRequest(String, String, int, int, InputStream, File, Long) - Method in class org.apache.hadoop.fs.s3a.WriteOperationHelper: Create and initialize a part request of a multipart upload.
next() - Method in class org.apache.hadoop.fs.s3a.MultipartUtils.UploadIterator
next() - Method in class org.apache.hadoop.fs.s3a.s3guard.DescendantsIterator
next() - Method in class org.apache.hadoop.fs.s3a.s3guard.MetadataStoreListFilesIterator
NO_AWS_CREDENTIAL_PROVIDERS - Static variable in class org.apache.hadoop.fs.s3a.AWSCredentialProviderList
NO_OP - Static variable in class org.apache.hadoop.fs.s3a.Invoker: No op for a retrying callback.
NoAuthWithAWSException - Exception in org.apache.hadoop.fs.s3a.auth: A specific subclass of AmazonClientException which can be used in the retry logic to fail fast when there is any authentication problem.
NoAuthWithAWSException(String, Throwable) - Constructor for exception org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException
NoAuthWithAWSException(String) - Constructor for exception org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException
NONE - Static variable in class org.apache.hadoop.fs.s3a.commit.CommitOperations.MaybeIOE
NoVersionAttributeException - Exception in org.apache.hadoop.fs.s3a: Indicates the S3 object does not provide the versioning attribute required by the configured change detection policy.
NoVersionAttributeException(String, String) - Constructor for exception org.apache.hadoop.fs.s3a.NoVersionAttributeException: Constructs a NoVersionAttributeException.
NullMetadataStore - Class in org.apache.hadoop.fs.s3a.s3guard: A no-op implementation of MetadataStore.
NullMetadataStore() - Constructor for class org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore
numEntries() - Method in class org.apache.hadoop.fs.s3a.s3guard.DirListingMetadata

O

objectRepresentsDirectory(String, long) - Static method in class org.apache.hadoop.fs.s3a.S3AUtils: Predicate: does the object represent a directory?.
of(IOException) - Static method in class org.apache.hadoop.fs.s3a.commit.CommitOperations.MaybeIOE: Get an instance based on the exception: either a value or a reference to CommitOperations.MaybeIOE.NONE.
once(String, String, Invoker.Operation<T>) - Static method in class org.apache.hadoop.fs.s3a.Invoker: Execute a function, translating any exception into an IOException.
once(String, String, Invoker.VoidOperation) - Static method in class org.apache.hadoop.fs.s3a.Invoker: Execute an operation with no result.
onChangeDetected(String, String, String, long, String, long) - Method in class org.apache.hadoop.fs.s3a.impl.ChangeDetectionPolicy: Takes appropriate action based on mode when a change has been detected.
onFailure(Tasks.FailureTask<I, ?>) - Method in class org.apache.hadoop.fs.s3a.commit.Tasks.Builder
onFailure(String, IOException, int, boolean) - Method in interface org.apache.hadoop.fs.s3a.Invoker.Retried: Retry event in progress (before any sleep).
open(Path, int) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Opens an FSDataInputStream at the indicated Path.
open(Path, int) - Method in class org.apache.hadoop.fs.s3native.NativeS3FileSystem: Deprecated.
openOperations - Variable in class org.apache.hadoop.fs.s3a.S3AInstrumentation.InputStreamStatistics
OPERATION_OPEN - Static variable in class org.apache.hadoop.fs.s3a.S3AInputStream
OPERATION_REOPEN - Static variable in class org.apache.hadoop.fs.s3a.S3AInputStream
operationRetried(String, Exception, int, boolean) - Method in class org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider: Callback from Invoker when an operation is retried.
operationRetried(Exception) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Callback when an operation was retried.
operationRetried(String, Exception, int, boolean) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Callback from Invoker when an operation is retried.
org.apache.hadoop.fs.s3a - package org.apache.hadoop.fs.s3a: S3A Filesystem.
org.apache.hadoop.fs.s3a.auth - package org.apache.hadoop.fs.s3a.auth: Authentication and permissions support.
org.apache.hadoop.fs.s3a.commit - package org.apache.hadoop.fs.s3a.commit: Support for committing the output of analytics jobs directly to S3.
org.apache.hadoop.fs.s3a.commit.files - package org.apache.hadoop.fs.s3a.commit.files: Persistent data formats for the committers.
org.apache.hadoop.fs.s3a.commit.magic - package org.apache.hadoop.fs.s3a.commit.magic: This is the "Magic" committer and support.
org.apache.hadoop.fs.s3a.commit.staging - package org.apache.hadoop.fs.s3a.commit.staging: The staging committers.
org.apache.hadoop.fs.s3a.impl - package org.apache.hadoop.fs.s3a.impl: Implementation classes private to the S3A store.
org.apache.hadoop.fs.s3a.s3guard - package org.apache.hadoop.fs.s3a.s3guard: This package contains classes related to S3Guard: a feature of S3A to mask the eventual consistency behavior of S3 and optimize access patterns by coordinating with a strongly consistent external store for file system metadata.
org.apache.hadoop.fs.s3native - package org.apache.hadoop.fs.s3native: A distributed implementation of FileSystem for reading and writing files on Amazon S3.
outputImmediatelyVisible() - Method in class org.apache.hadoop.fs.s3a.commit.magic.MagicCommitTracker: Flag to indicate that output is not visible after the stream is closed.
outputImmediatelyVisible() - Method in class org.apache.hadoop.fs.s3a.commit.PutTracker: Flag to indicate that output is not immediately visible after the stream is closed.
OutputStreamStatistics(FileSystem.Statistics) - Constructor for class org.apache.hadoop.fs.s3a.S3AInstrumentation.OutputStreamStatistics

P

PARTITION_COMMITTER_FACTORY - Static variable in class org.apache.hadoop.fs.s3a.commit.InternalCommitterConstants: Partitioned committer factory: "org.apache.hadoop.fs.s3a.commit.staging.PartitionedStagingCommitterFactory".
PartitionedStagingCommitter - Class in org.apache.hadoop.fs.s3a.commit.staging: Partitioned committer.
PartitionedStagingCommitter(Path, TaskAttemptContext) - Constructor for class org.apache.hadoop.fs.s3a.commit.staging.PartitionedStagingCommitter
PartitionedStagingCommitterFactory - Class in org.apache.hadoop.fs.s3a.commit.staging: Factory for the PartitionedStagingCommitter.
PartitionedStagingCommitterFactory() - Constructor for class org.apache.hadoop.fs.s3a.commit.staging.PartitionedStagingCommitterFactory
path(Path, String...) - Static method in class org.apache.hadoop.fs.s3a.commit.staging.Paths: Varags constructor of paths.
PATH_STYLE_ACCESS - Static variable in class org.apache.hadoop.fs.s3a.Constants
PathCommitException - Exception in org.apache.hadoop.fs.s3a.commit: Path exception to use for various commit issues.
PathCommitException(String, Throwable) - Constructor for exception org.apache.hadoop.fs.s3a.commit.PathCommitException
PathCommitException(String, String) - Constructor for exception org.apache.hadoop.fs.s3a.commit.PathCommitException
PathCommitException(Path, String) - Constructor for exception org.apache.hadoop.fs.s3a.commit.PathCommitException
PathCommitException(String, String, Throwable) - Constructor for exception org.apache.hadoop.fs.s3a.commit.PathCommitException
PathMetadata - Class in org.apache.hadoop.fs.s3a.s3guard: PathMetadata models path metadata stored in the MetadataStore.
PathMetadata(FileStatus) - Constructor for class org.apache.hadoop.fs.s3a.s3guard.PathMetadata: Creates a new PathMetadata containing given FileStatus.
PathMetadata(FileStatus, Tristate) - Constructor for class org.apache.hadoop.fs.s3a.s3guard.PathMetadata
PathMetadata(FileStatus, Tristate, boolean) - Constructor for class org.apache.hadoop.fs.s3a.s3guard.PathMetadata
Paths - Class in org.apache.hadoop.fs.s3a.commit.staging: Path operations for the staging committers.
pathToKey(Path) - Static method in class org.apache.hadoop.fs.s3a.auth.RoleModel: Variant of S3AFileSystem.pathToKey(Path) which doesn't care about working directories, so can be static and stateless.
pathToKey(Path) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Turns a path (relative or otherwise) into an S3 key.
PENDING_FILTER - Static variable in class org.apache.hadoop.fs.s3a.commit.CommitOperations: Filter to find all {code .pending} files.
PENDING_SUFFIX - Static variable in class org.apache.hadoop.fs.s3a.commit.CommitConstants: Suffix applied to pending commit metadata: ".pending".
PendingSet - Class in org.apache.hadoop.fs.s3a.commit.files: Persistent format for multiple pending commits.
PendingSet() - Constructor for class org.apache.hadoop.fs.s3a.commit.files.PendingSet
PendingSet(int) - Constructor for class org.apache.hadoop.fs.s3a.commit.files.PendingSet
PENDINGSET_FILTER - Static variable in class org.apache.hadoop.fs.s3a.commit.CommitOperations: Filter to find all {code .pendingset} files.
PENDINGSET_SUFFIX - Static variable in class org.apache.hadoop.fs.s3a.commit.CommitConstants: Suffix applied to multiple pending commit metadata: ".pendingset".
PersistentCommitData - Class in org.apache.hadoop.fs.s3a.commit.files: Class for single/multiple commit data structures.
PersistentCommitData() - Constructor for class org.apache.hadoop.fs.s3a.commit.files.PersistentCommitData
PERSISTS_AUTHORITATIVE_BIT - Static variable in class org.apache.hadoop.fs.s3a.s3guard.MetadataStoreCapabilities: This capability tells if the metadata store supports authoritative directories.
policy(RoleModel.Statement...) - Static method in class org.apache.hadoop.fs.s3a.auth.RoleModel: From a set of statements, create a policy.
policy(List<RoleModel.Statement>) - Static method in class org.apache.hadoop.fs.s3a.auth.RoleModel: From a set of statements, create a policy.
Policy(List<RoleModel.Statement>) - Constructor for class org.apache.hadoop.fs.s3a.auth.RoleModel.Policy
Policy(RoleModel.Statement...) - Constructor for class org.apache.hadoop.fs.s3a.auth.RoleModel.Policy
policySetCount - Variable in class org.apache.hadoop.fs.s3a.S3AInstrumentation.InputStreamStatistics
preCommitJob(JobContext, List<SinglePendingCommit>) - Method in class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter: Subclass-specific pre commit actions.
preCommitJob(JobContext, List<SinglePendingCommit>) - Method in class org.apache.hadoop.fs.s3a.commit.staging.DirectoryStagingCommitter: Pre-commit actions for a job.
preCommitJob(JobContext, List<SinglePendingCommit>) - Method in class org.apache.hadoop.fs.s3a.commit.staging.PartitionedStagingCommitter: Job-side conflict resolution.
prettyPrint(StringBuilder) - Method in class org.apache.hadoop.fs.s3a.s3guard.DirListingMetadata: Log contents to supplied StringBuilder in a pretty fashion.
prettyPrint() - Method in class org.apache.hadoop.fs.s3a.s3guard.DirListingMetadata
prettyPrint(StringBuilder) - Method in class org.apache.hadoop.fs.s3a.s3guard.PathMetadata: Log contents to supplied StringBuilder in a pretty fashion.
prettyPrint() - Method in class org.apache.hadoop.fs.s3a.s3guard.PathMetadata
printStoreDiagnostics(PrintStream, MetadataStore) - Static method in class org.apache.hadoop.fs.s3a.s3guard.S3GuardTool: Retrieve and Print store diagnostics.
processResponse(S3Object, String, long) - Method in class org.apache.hadoop.fs.s3a.impl.ChangeTracker: Process the response from the server for validation against the change policy.
ProgressableProgressListener - Class in org.apache.hadoop.fs.s3a: Listener to progress from AWS regarding transfers.
ProgressableProgressListener(S3AFileSystem, String, Upload, Progressable) - Constructor for class org.apache.hadoop.fs.s3a.ProgressableProgressListener: Instantiate.
progressChanged(ProgressEvent) - Method in class org.apache.hadoop.fs.s3a.ProgressableProgressListener
propagateBucketOptions(Configuration, String) - Static method in class org.apache.hadoop.fs.s3a.S3AUtils: Propagates bucket-specific settings into generic S3A configuration keys.
PROXY_DOMAIN - Static variable in class org.apache.hadoop.fs.s3a.Constants
PROXY_HOST - Static variable in class org.apache.hadoop.fs.s3a.Constants
PROXY_PASSWORD - Static variable in class org.apache.hadoop.fs.s3a.Constants
PROXY_PORT - Static variable in class org.apache.hadoop.fs.s3a.Constants
PROXY_USERNAME - Static variable in class org.apache.hadoop.fs.s3a.Constants
PROXY_WORKSTATION - Static variable in class org.apache.hadoop.fs.s3a.Constants
prune(long) - Method in class org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
prune(long, String) - Method in class org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore: Prune files, in batches.
prune(long) - Method in class org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore
prune(long, String) - Method in class org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore
prune(long) - Method in interface org.apache.hadoop.fs.s3a.s3guard.MetadataStore: Clear any metadata older than a specified time from the repository.
prune(long, String) - Method in interface org.apache.hadoop.fs.s3a.s3guard.MetadataStore: Same as MetadataStore.prune(long), but with an additional keyPrefix parameter to filter the pruned keys with a prefix.
prune(long) - Method in class org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore
prune(long, String) - Method in class org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore
PURGE_EXISTING_MULTIPART - Static variable in class org.apache.hadoop.fs.s3a.Constants
PURGE_EXISTING_MULTIPART_AGE - Static variable in class org.apache.hadoop.fs.s3a.Constants
put(FileStatus) - Method in class org.apache.hadoop.fs.s3a.s3guard.DirListingMetadata: Add an entry to the directory listing.
put(PathMetadata) - Method in class org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
put(Collection<PathMetadata>) - Method in class org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
put(DirListingMetadata) - Method in class org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore: Save directory listing metadata.
put(PathMetadata) - Method in class org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore
put(DirListingMetadata) - Method in class org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore
put(Collection<PathMetadata>) - Method in class org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore
put(PathMetadata) - Method in interface org.apache.hadoop.fs.s3a.s3guard.MetadataStore: Saves metadata for exactly one path.
put(Collection<PathMetadata>) - Method in interface org.apache.hadoop.fs.s3a.s3guard.MetadataStore: Saves metadata for any number of paths.
put(DirListingMetadata) - Method in interface org.apache.hadoop.fs.s3a.s3guard.MetadataStore: Save directory listing metadata.
put(PathMetadata) - Method in class org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore
put(Collection<PathMetadata>) - Method in class org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore
put(DirListingMetadata) - Method in class org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore
putAndReturn(MetadataStore, S3AFileStatus, S3AInstrumentation) - Static method in class org.apache.hadoop.fs.s3a.s3guard.S3Guard: Helper function which puts a given S3AFileStatus into the MetadataStore and returns the same S3AFileStatus.
putObject(PutObjectRequest) - Method in class org.apache.hadoop.fs.s3a.InconsistentAmazonS3Client
putObject(PutObjectRequest) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Start a transfer-manager managed async PUT of an object, incrementing the put requests and put bytes counters.
putObject(PutObjectRequest) - Method in class org.apache.hadoop.fs.s3a.WriteOperationHelper: PUT an object directly (i.e.
putPart(Path, InputStream, int, UploadHandle, long) - Method in class org.apache.hadoop.fs.s3a.S3AMultipartUploader
PutTracker - Class in org.apache.hadoop.fs.s3a.commit: Multipart put tracker.
PutTracker(String) - Constructor for class org.apache.hadoop.fs.s3a.commit.PutTracker: Instantiate.

Q

qualify(Path) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Qualify a path.
quantiles(Statistic, String, String, int) - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation: Create a quantiles in the registry.
quietly(String, String, Invoker.VoidOperation) - Static method in class org.apache.hadoop.fs.s3a.Invoker: Execute an operation; any exception raised is simply caught and logged at debug.
quietlyEval(String, String, Invoker.Operation<T>) - Static method in class org.apache.hadoop.fs.s3a.Invoker: Execute an operation; any exception raised is caught and logged at debug.

R

read() - Method in class org.apache.hadoop.fs.s3a.InconsistentS3Object.InconsistentS3InputStream
read(byte[], int, int) - Method in class org.apache.hadoop.fs.s3a.InconsistentS3Object.InconsistentS3InputStream
read() - Method in class org.apache.hadoop.fs.s3a.S3AInputStream
read(byte[], int, int) - Method in class org.apache.hadoop.fs.s3a.S3AInputStream: This updates the statistics on read operations started and whether or not the read operation "completed", that is: returned the exact number of bytes requested.
READ_FLAG - Static variable in class org.apache.hadoop.fs.s3a.s3guard.S3GuardTool
READAHEAD_RANGE - Static variable in class org.apache.hadoop.fs.s3a.Constants: read ahead buffer size to prevent connection re-establishments.
readException() - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation.InputStreamStatistics: An ignored stream read exception was received.
readExceptions - Variable in class org.apache.hadoop.fs.s3a.S3AInstrumentation.InputStreamStatistics
readFully(long, byte[], int, int) - Method in class org.apache.hadoop.fs.s3a.S3AInputStream: Subclass readFully() operation which only seeks at the start of the series of operations; seeking back at the end.
readFullyOperations - Variable in class org.apache.hadoop.fs.s3a.S3AInstrumentation.InputStreamStatistics
readFullyOperationStarted(long, long) - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation.InputStreamStatistics: A PositionedRead.read(position, buffer, offset, length) operation has just started.
readOperationCompleted(int, int) - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation.InputStreamStatistics: A read operation has completed.
readOperations - Variable in class org.apache.hadoop.fs.s3a.S3AInstrumentation.InputStreamStatistics
readOperationStarted(long, long) - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation.InputStreamStatistics: A read(byte[] buf, int off, int len) operation has started.
readsIncomplete - Variable in class org.apache.hadoop.fs.s3a.S3AInstrumentation.InputStreamStatistics
recoverTask(TaskAttemptContext) - Method in class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter: Task recovery considered unsupported: Warn and fail.
refresh() - Method in class org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider
refresh() - Method in class org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider
refresh() - Method in class org.apache.hadoop.fs.s3a.AWSCredentialProviderList: Refresh all child entries.
refresh() - Method in class org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider
refresh() - Method in class org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider
REGION_FLAG - Static variable in class org.apache.hadoop.fs.s3a.s3guard.S3GuardTool
rejectSecretsInURIs(URI) - Static method in class org.apache.hadoop.fs.s3native.S3xLoginHelper: Extract the login details from a URI, raising an exception if the URI contains them.
remainingInCurrentRequest() - Method in class org.apache.hadoop.fs.s3a.S3AInputStream: Bytes left in the current request.
remainingInFile() - Method in class org.apache.hadoop.fs.s3a.S3AInputStream: Bytes left in stream.
RemoteFileChangedException - Exception in org.apache.hadoop.fs.s3a: Indicates the S3 object is out of sync with the expected version.
RemoteFileChangedException(String, String, String) - Constructor for exception org.apache.hadoop.fs.s3a.RemoteFileChangedException: Constructs a RemoteFileChangedException.
remove(Path) - Method in class org.apache.hadoop.fs.s3a.s3guard.DirListingMetadata: Remove entry from this directory.
rename(Path, Path) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Renames Path src to Path dst.
rename(Path, Path) - Method in class org.apache.hadoop.fs.s3native.NativeS3FileSystem: Deprecated.
RenameFailedException - Exception in org.apache.hadoop.fs.s3a: Error to indicate that a specific rename failed.
RenameFailedException(String, String, Throwable) - Constructor for exception org.apache.hadoop.fs.s3a.RenameFailedException
RenameFailedException(String, String, String) - Constructor for exception org.apache.hadoop.fs.s3a.RenameFailedException
RenameFailedException(Path, Path, String) - Constructor for exception org.apache.hadoop.fs.s3a.RenameFailedException
requiresDelayedCommitOutputInFileSystem() - Method in class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter: Flag to indicate whether or not the destination filesystem needs to be configured to support magic paths where the output isn't immediately visible.
requiresDelayedCommitOutputInFileSystem() - Method in class org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter: Require magic paths in the FS client.
reset() - Method in class org.apache.hadoop.fs.s3a.S3AStorageStatistics
resetConnection() - Method in class org.apache.hadoop.fs.s3a.S3AInputStream: Forcibly reset the stream, by aborting the connection.
resetTempFolderCache() - Static method in class org.apache.hadoop.fs.s3a.commit.staging.Paths: Reset the temp folder cache; useful in tests.
resource(String, String, boolean) - Static method in class org.apache.hadoop.fs.s3a.auth.RoleModel: Create a resource.
resource(Path, boolean, boolean) - Static method in class org.apache.hadoop.fs.s3a.auth.RoleModel: Given a path, return the S3 resource to it.
resource - Variable in class org.apache.hadoop.fs.s3a.auth.RoleModel.Statement
Retries - Class in org.apache.hadoop.fs.s3a: Declaration of retry policy for documentation only.
Retries() - Constructor for class org.apache.hadoop.fs.s3a.Retries
Retries.OnceExceptionsSwallowed - Annotation Type in org.apache.hadoop.fs.s3a: One attempt, Exceptions are swallowed.
Retries.OnceMixed - Annotation Type in org.apache.hadoop.fs.s3a: No retry, expect a bit of both.
Retries.OnceRaw - Annotation Type in org.apache.hadoop.fs.s3a: No retry, exceptions are not translated.
Retries.OnceTranslated - Annotation Type in org.apache.hadoop.fs.s3a: No retry, exceptions are translated.
Retries.RetryExceptionsSwallowed - Annotation Type in org.apache.hadoop.fs.s3a: Retried, Exceptions are swallowed.
Retries.RetryMixed - Annotation Type in org.apache.hadoop.fs.s3a: Retried, mixed translation.
Retries.RetryRaw - Annotation Type in org.apache.hadoop.fs.s3a: Retried, no translation.
Retries.RetryTranslated - Annotation Type in org.apache.hadoop.fs.s3a: Retried, exceptions are translated.
retry(String, String, boolean, Invoker.Retried, Invoker.VoidOperation) - Method in class org.apache.hadoop.fs.s3a.Invoker: Execute a void operation with retry processing.
retry(String, String, boolean, Invoker.VoidOperation) - Method in class org.apache.hadoop.fs.s3a.Invoker: Execute a void operation with the default retry callback invoked.
retry(String, String, boolean, Invoker.Operation<T>) - Method in class org.apache.hadoop.fs.s3a.Invoker: Execute a function with the default retry callback invoked.
retry(String, String, boolean, Invoker.Retried, Invoker.Operation<T>) - Method in class org.apache.hadoop.fs.s3a.Invoker: Execute a function with retry processing.
retry(String, String, boolean, Invoker.Operation<T>) - Method in class org.apache.hadoop.fs.s3a.WriteOperationHelper: Execute a function with retry processing.
RETRY_INTERVAL - Static variable in class org.apache.hadoop.fs.s3a.Constants: Interval between retry attempts.: "fs.s3a.retry.interval".
RETRY_INTERVAL_DEFAULT - Static variable in class org.apache.hadoop.fs.s3a.Constants: Default retry interval: "500ms".
RETRY_LIMIT - Static variable in class org.apache.hadoop.fs.s3a.Constants: Number of times to retry any repeatable S3 client request on failure, excluding throttling requests: "fs.s3a.retry.limit".
RETRY_LIMIT_DEFAULT - Static variable in class org.apache.hadoop.fs.s3a.Constants: Default retry limit: 20.
RETRY_THROTTLE_INTERVAL - Static variable in class org.apache.hadoop.fs.s3a.Constants: Interval between retry attempts on throttled requests: "fs.s3a.retry.throttle.interval".
RETRY_THROTTLE_INTERVAL_DEFAULT - Static variable in class org.apache.hadoop.fs.s3a.Constants: Default throttled retry interval: "500ms".
RETRY_THROTTLE_LIMIT - Static variable in class org.apache.hadoop.fs.s3a.Constants: Number of times to retry any throttled request: "fs.s3a.retry.throttle.limit".
RETRY_THROTTLE_LIMIT_DEFAULT - Static variable in class org.apache.hadoop.fs.s3a.Constants: Default throttled retry limit: 20.
retryIdempotentCalls - Variable in class org.apache.hadoop.fs.s3a.S3ARetryPolicy: Rejection of all non-idempotent calls except specific failures.
retrying() - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation.S3GuardInstrumentation: S3Guard is retrying after a (retryable) failure.
retryUntranslated(String, boolean, Invoker.Operation<T>) - Method in class org.apache.hadoop.fs.s3a.Invoker: Execute a function with retry processing and no translation.
retryUntranslated(String, boolean, Invoker.Retried, Invoker.Operation<T>) - Method in class org.apache.hadoop.fs.s3a.Invoker: Execute a function with retry processing: AWS SDK Exceptions are not translated.
revertCommit(SinglePendingCommit) - Method in class org.apache.hadoop.fs.s3a.commit.CommitOperations: Revert a pending commit by deleting the destination.
revertCommit(String) - Method in class org.apache.hadoop.fs.s3a.WriteOperationHelper: Revert a commit by deleting the file.
revertWith(Tasks.Task<I, ?>) - Method in class org.apache.hadoop.fs.s3a.commit.Tasks.Builder
RoleElt() - Constructor for class org.apache.hadoop.fs.s3a.auth.RoleModel.RoleElt
RoleModel - Class in org.apache.hadoop.fs.s3a.auth: Jackson Role Model for Role Properties, for API clients and tests.
RoleModel() - Constructor for class org.apache.hadoop.fs.s3a.auth.RoleModel
RoleModel.Effects - Enum in org.apache.hadoop.fs.s3a.auth: Effect options.
RoleModel.Policy - Class in org.apache.hadoop.fs.s3a.auth: A policy is one or more statements.
RoleModel.RoleElt - Class in org.apache.hadoop.fs.s3a.auth: Any element in a role.
RoleModel.Statement - Class in org.apache.hadoop.fs.s3a.auth: A single statement.
RolePolicies - Class in org.apache.hadoop.fs.s3a.auth: Operations, statements and policies covering the operations needed to work with S3 and S3Guard.
run(Tasks.Task<I, E>) - Method in class org.apache.hadoop.fs.s3a.commit.Tasks.Builder
run(I, Exception) - Method in interface org.apache.hadoop.fs.s3a.commit.Tasks.FailureTask: process a failure.
run(I) - Method in interface org.apache.hadoop.fs.s3a.commit.Tasks.Task
run(String[]) - Method in class org.apache.hadoop.fs.s3a.s3guard.S3GuardTool
run(String[], PrintStream) - Method in class org.apache.hadoop.fs.s3a.s3guard.S3GuardTool: Run the tool, capturing the output (if the tool supports that).
run(Configuration, String...) - Static method in class org.apache.hadoop.fs.s3a.s3guard.S3GuardTool: Execute the command with the given arguments.

S

S3_ABORT_MULTIPART_UPLOAD - Static variable in class org.apache.hadoop.fs.s3a.auth.RolePolicies: abort multipart upload is needed for the S3A Commit protocols.
S3_ALL_BUCKETS - Static variable in class org.apache.hadoop.fs.s3a.auth.RolePolicies: All S3 buckets: "arn:aws:s3:::*".
S3_ALL_DELETE - Static variable in class org.apache.hadoop.fs.s3a.auth.RolePolicies: All s3:Delete* operations.
S3_ALL_GET - Static variable in class org.apache.hadoop.fs.s3a.auth.RolePolicies: All s3:Get* operations.
S3_ALL_LIST_BUCKET - Static variable in class org.apache.hadoop.fs.s3a.auth.RolePolicies
S3_ALL_LIST_OPERATIONS - Static variable in class org.apache.hadoop.fs.s3a.auth.RolePolicies
S3_ALL_OPERATIONS - Static variable in class org.apache.hadoop.fs.s3a.auth.RolePolicies: All S3 operations: "s3:*".
S3_ALL_PUT - Static variable in class org.apache.hadoop.fs.s3a.auth.RolePolicies: S3 Put*.
S3_CLIENT_FACTORY_IMPL - Static variable in class org.apache.hadoop.fs.s3a.Constants
S3_DELETE_OBJECT - Static variable in class org.apache.hadoop.fs.s3a.auth.RolePolicies
S3_DELETE_OBJECT_TAGGING - Static variable in class org.apache.hadoop.fs.s3a.auth.RolePolicies
S3_DELETE_OBJECT_VERSION - Static variable in class org.apache.hadoop.fs.s3a.auth.RolePolicies
S3_DELETE_OBJECT_VERSION_TAGGING - Static variable in class org.apache.hadoop.fs.s3a.auth.RolePolicies
S3_GET_BUCKET_LOCATION - Static variable in class org.apache.hadoop.fs.s3a.auth.RolePolicies
S3_GET_OBJECT - Static variable in class org.apache.hadoop.fs.s3a.auth.RolePolicies
S3_GET_OBJECT_ACL - Static variable in class org.apache.hadoop.fs.s3a.auth.RolePolicies
S3_GET_OBJECT_TAGGING - Static variable in class org.apache.hadoop.fs.s3a.auth.RolePolicies
S3_GET_OBJECT_TORRENT - Static variable in class org.apache.hadoop.fs.s3a.auth.RolePolicies
S3_GET_OBJECT_VERSION - Static variable in class org.apache.hadoop.fs.s3a.auth.RolePolicies
S3_GET_OBJECT_VERSION_ACL - Static variable in class org.apache.hadoop.fs.s3a.auth.RolePolicies
S3_GET_OBJECT_VERSION_TAGGING - Static variable in class org.apache.hadoop.fs.s3a.auth.RolePolicies
S3_GET_OBJECT_VERSION_TORRENT - Static variable in class org.apache.hadoop.fs.s3a.auth.RolePolicies
S3_LIST_BUCKET - Static variable in class org.apache.hadoop.fs.s3a.auth.RolePolicies
S3_LIST_BUCKET_MULTPART_UPLOADS - Static variable in class org.apache.hadoop.fs.s3a.auth.RolePolicies: This is used by the abort operation in S3A commit work.
S3_LIST_MULTIPART_UPLOAD_PARTS - Static variable in class org.apache.hadoop.fs.s3a.auth.RolePolicies: List multipart upload is needed for the S3A Commit protocols.
S3_METADATA_STORE_IMPL - Static variable in class org.apache.hadoop.fs.s3a.Constants
S3_PATH_READ_OPERATIONS - Static variable in class org.apache.hadoop.fs.s3a.auth.RolePolicies: Actions needed to read a file in S3 through S3A, excluding S3Guard and SSE-KMS.
S3_PATH_RW_OPERATIONS - Static variable in class org.apache.hadoop.fs.s3a.auth.RolePolicies: Actions needed to write data to an S3A Path.
S3_PATH_WRITE_OPERATIONS - Static variable in class org.apache.hadoop.fs.s3a.auth.RolePolicies: Actions needed to write data to an S3A Path.
S3_PUT_OBJECT - Static variable in class org.apache.hadoop.fs.s3a.auth.RolePolicies
S3_PUT_OBJECT_ACL - Static variable in class org.apache.hadoop.fs.s3a.auth.RolePolicies
S3_PUT_OBJECT_TAGGING - Static variable in class org.apache.hadoop.fs.s3a.auth.RolePolicies
S3_PUT_OBJECT_VERSION_ACL - Static variable in class org.apache.hadoop.fs.s3a.auth.RolePolicies
S3_PUT_OBJECT_VERSION_TAGGING - Static variable in class org.apache.hadoop.fs.s3a.auth.RolePolicies
S3_RESTORE_OBJECT - Static variable in class org.apache.hadoop.fs.s3a.auth.RolePolicies
S3_ROOT_READ_OPERATIONS - Static variable in class org.apache.hadoop.fs.s3a.auth.RolePolicies: Base actions needed to read data from S3 through S3A, excluding SSE-KMS data and S3Guard-ed buckets.
S3_ROOT_RW_OPERATIONS - Static variable in class org.apache.hadoop.fs.s3a.auth.RolePolicies: Actions needed for R/W IO from the root of a bucket.
S3A - Class in org.apache.hadoop.fs.s3a: S3A implementation of AbstractFileSystem.
S3A(URI, Configuration) - Constructor for class org.apache.hadoop.fs.s3a.S3A
S3A_COMMITTER_FACTORY - Static variable in class org.apache.hadoop.fs.s3a.commit.CommitConstants: S3 Committer factory: "org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory".
S3A_COMMITTER_FACTORY_KEY - Static variable in class org.apache.hadoop.fs.s3a.commit.CommitConstants: Key to set for the S3A schema to use the specific committer.
S3A_DEFAULT_PORT - Static variable in class org.apache.hadoop.fs.s3a.Constants
S3A_SECURITY_CREDENTIAL_PROVIDER_PATH - Static variable in class org.apache.hadoop.fs.s3a.Constants: Extra set of security credentials which will be prepended to that set in "hadoop.security.credential.provider.path".
S3ACommitterFactory - Class in org.apache.hadoop.fs.s3a.commit: The S3A committer factory which chooses the committer based on the specific option chosen in a per-bucket basis from the property CommitConstants.FS_S3A_COMMITTER_NAME.
S3ACommitterFactory() - Constructor for class org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory
S3AEncryptionMethods - Enum in org.apache.hadoop.fs.s3a: This enum is to centralize the encryption methods and the value required in the configuration.
S3AFileStatus - Class in org.apache.hadoop.fs.s3a: File status for an S3A "file".
S3AFileStatus(boolean, Path, String) - Constructor for class org.apache.hadoop.fs.s3a.S3AFileStatus: Create a directory status.
S3AFileStatus(Tristate, Path, String) - Constructor for class org.apache.hadoop.fs.s3a.S3AFileStatus: Create a directory status.
S3AFileStatus(long, long, Path, long, String) - Constructor for class org.apache.hadoop.fs.s3a.S3AFileStatus: A simple file.
S3AFileSystem - Class in org.apache.hadoop.fs.s3a: The core S3A Filesystem implementation.
S3AFileSystem() - Constructor for class org.apache.hadoop.fs.s3a.S3AFileSystem
S3AInputPolicy - Enum in org.apache.hadoop.fs.s3a: Filesystem input policy.
S3AInputStream - Class in org.apache.hadoop.fs.s3a: The input stream for an S3A object.
S3AInputStream(S3AReadOpContext, S3ObjectAttributes, long, AmazonS3) - Constructor for class org.apache.hadoop.fs.s3a.S3AInputStream: Create the stream.
S3AInstrumentation - Class in org.apache.hadoop.fs.s3a: Instrumentation of S3a.
S3AInstrumentation(URI) - Constructor for class org.apache.hadoop.fs.s3a.S3AInstrumentation
S3AInstrumentation.CommitterStatistics - Class in org.apache.hadoop.fs.s3a: Instrumentation exported to S3Guard Committers.
S3AInstrumentation.InputStreamStatistics - Class in org.apache.hadoop.fs.s3a: Statistics updated by an input stream during its actual operation.
S3AInstrumentation.OutputStreamStatistics - Class in org.apache.hadoop.fs.s3a: Statistics updated by an output stream during its actual operation.
S3AInstrumentation.S3GuardInstrumentation - Class in org.apache.hadoop.fs.s3a: Instrumentation exported to S3Guard.
S3AMultipartUploader - Class in org.apache.hadoop.fs.s3a: MultipartUploader for S3AFileSystem.
S3AMultipartUploader(FileSystem, Configuration) - Constructor for class org.apache.hadoop.fs.s3a.S3AMultipartUploader
S3AMultipartUploader.Factory - Class in org.apache.hadoop.fs.s3a: Factory for creating MultipartUploader objects for s3a:// FileSystems.
S3AOpContext - Class in org.apache.hadoop.fs.s3a: Base class for operation context struct passed through codepaths for main S3AFileSystem operations.
S3AOpContext(boolean, Invoker, Invoker, FileSystem.Statistics, S3AInstrumentation, FileStatus) - Constructor for class org.apache.hadoop.fs.s3a.S3AOpContext: Alternate constructor that allows passing in two invokers, the common one, and another with the S3Guard Retry Policy.
S3AOpContext(boolean, Invoker, FileSystem.Statistics, S3AInstrumentation, FileStatus) - Constructor for class org.apache.hadoop.fs.s3a.S3AOpContext: Constructor using common invoker and retry policy.
S3AReadOpContext - Class in org.apache.hadoop.fs.s3a: Read-specific operation context struct.
S3AReadOpContext(Path, boolean, Invoker, Invoker, FileSystem.Statistics, S3AInstrumentation, FileStatus, S3AInputPolicy, ChangeDetectionPolicy, long) - Constructor for class org.apache.hadoop.fs.s3a.S3AReadOpContext: Instantiate.
S3ARetryPolicy - Class in org.apache.hadoop.fs.s3a: The S3A request retry policy.
S3ARetryPolicy(Configuration) - Constructor for class org.apache.hadoop.fs.s3a.S3ARetryPolicy: Instantiate.
S3AStorageStatistics - Class in org.apache.hadoop.fs.s3a: Storage statistics for S3A.
S3AStorageStatistics() - Constructor for class org.apache.hadoop.fs.s3a.S3AStorageStatistics
S3AUtils - Class in org.apache.hadoop.fs.s3a: Utility methods for S3A code.
S3AUtils.CallOnLocatedFileStatus - Interface in org.apache.hadoop.fs.s3a: An interface for use in lambda-expressions working with directory tree listings.
S3AUtils.LocatedFileStatusMap<T> - Interface in org.apache.hadoop.fs.s3a: An interface for use in lambda-expressions working with directory tree listings.
S3ClientFactory - Interface in org.apache.hadoop.fs.s3a: Factory for creation of AmazonS3 client instances.
S3Guard - Class in org.apache.hadoop.fs.s3a.s3guard: Logic for integrating MetadataStore with S3A.
S3GUARD_CLI_PRUNE_AGE - Static variable in class org.apache.hadoop.fs.s3a.Constants: Minimum period of time (in milliseconds) to keep metadata (may only be applied when a prune command is manually run).
S3GUARD_DDB_BACKGROUND_SLEEP_MSEC_DEFAULT - Static variable in class org.apache.hadoop.fs.s3a.Constants
S3GUARD_DDB_BACKGROUND_SLEEP_MSEC_KEY - Static variable in class org.apache.hadoop.fs.s3a.Constants: Period of time (in milliseconds) to sleep between batches of writes.
S3GUARD_DDB_BATCH_WRITE_REQUEST_LIMIT - Static variable in class org.apache.hadoop.fs.s3a.Constants: The maximum put or delete requests per BatchWriteItem request.
S3GUARD_DDB_CLIENT_FACTORY_IMPL - Static variable in class org.apache.hadoop.fs.s3a.s3guard.S3Guard
S3GUARD_DDB_MAX_RETRIES - Static variable in class org.apache.hadoop.fs.s3a.Constants
S3GUARD_DDB_MAX_RETRIES_DEFAULT - Static variable in class org.apache.hadoop.fs.s3a.Constants: Max retries on batched/throttled DynamoDB operations before giving up and throwing an IOException.
S3GUARD_DDB_REGION_KEY - Static variable in class org.apache.hadoop.fs.s3a.Constants: The region of the DynamoDB service.
S3GUARD_DDB_TABLE_CAPACITY_READ_DEFAULT - Static variable in class org.apache.hadoop.fs.s3a.Constants
S3GUARD_DDB_TABLE_CAPACITY_READ_KEY - Static variable in class org.apache.hadoop.fs.s3a.Constants
S3GUARD_DDB_TABLE_CAPACITY_WRITE_DEFAULT - Static variable in class org.apache.hadoop.fs.s3a.Constants
S3GUARD_DDB_TABLE_CAPACITY_WRITE_KEY - Static variable in class org.apache.hadoop.fs.s3a.Constants
S3GUARD_DDB_TABLE_CREATE_KEY - Static variable in class org.apache.hadoop.fs.s3a.Constants: Whether to create the DynamoDB table if the table does not exist.
S3GUARD_DDB_TABLE_NAME_KEY - Static variable in class org.apache.hadoop.fs.s3a.Constants: The DynamoDB table name to use.
S3GUARD_DDB_TABLE_TAG - Static variable in class org.apache.hadoop.fs.s3a.Constants: A prefix for adding tags to the DDB Table upon creation.
S3GUARD_DDB_TEST_TABLE_NAME_KEY - Static variable in class org.apache.hadoop.fs.s3a.Constants: Test table name to use during DynamoDB integration test.
S3GUARD_DDB_THROTTLE_RETRY_INTERVAL - Static variable in class org.apache.hadoop.fs.s3a.Constants
S3GUARD_DDB_THROTTLE_RETRY_INTERVAL_DEFAULT - Static variable in class org.apache.hadoop.fs.s3a.Constants
S3GUARD_METASTORE_DYNAMO - Static variable in class org.apache.hadoop.fs.s3a.Constants: Use DynamoDB for the metadata: "org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore".
S3GUARD_METASTORE_LOCAL - Static variable in class org.apache.hadoop.fs.s3a.Constants: Use Local memory for the metadata: "org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore".
S3GUARD_METASTORE_LOCAL_ENTRY_TTL - Static variable in class org.apache.hadoop.fs.s3a.Constants: Time to live in milliseconds in LocalMetadataStore.
S3GUARD_METASTORE_LOCAL_MAX_RECORDS - Static variable in class org.apache.hadoop.fs.s3a.Constants: Maximum number of records in LocalMetadataStore.
S3GUARD_METASTORE_NULL - Static variable in class org.apache.hadoop.fs.s3a.Constants: The default "Null" metadata store: "org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore".
S3GuardDataAccessRetryPolicy - Class in org.apache.hadoop.fs.s3a.s3guard: A Retry policy whose throttling comes from the S3Guard config options.
S3GuardDataAccessRetryPolicy(Configuration) - Constructor for class org.apache.hadoop.fs.s3a.s3guard.S3GuardDataAccessRetryPolicy
S3GuardExistsRetryPolicy - Class in org.apache.hadoop.fs.s3a: Slightly-modified retry policy for cases when the file is present in the MetadataStore, but may be still throwing FileNotFoundException from S3.
S3GuardExistsRetryPolicy(Configuration) - Constructor for class org.apache.hadoop.fs.s3a.S3GuardExistsRetryPolicy: Instantiate.
S3GuardInstrumentation() - Constructor for class org.apache.hadoop.fs.s3a.S3AInstrumentation.S3GuardInstrumentation
S3GuardTool - Class in org.apache.hadoop.fs.s3a.s3guard: CLI to manage S3Guard Metadata Store.
S3GuardTool(Configuration, String...) - Constructor for class org.apache.hadoop.fs.s3a.s3guard.S3GuardTool: Constructor a S3Guard tool with HDFS configuration.
S3ListRequest - Class in org.apache.hadoop.fs.s3a: API version-independent container for S3 List requests.
S3ListRequest(ListObjectsRequest, ListObjectsV2Request) - Constructor for class org.apache.hadoop.fs.s3a.S3ListRequest
S3ListResult - Class in org.apache.hadoop.fs.s3a: API version-independent container for S3 List responses.
S3ListResult(ObjectListing, ListObjectsV2Result) - Constructor for class org.apache.hadoop.fs.s3a.S3ListResult
S3N_FOLDER_SUFFIX - Static variable in class org.apache.hadoop.fs.s3a.Constants
S3xLoginHelper - Class in org.apache.hadoop.fs.s3native: Class to aid logging in to S3 endpoints.
S3xLoginHelper.Login - Class in org.apache.hadoop.fs.s3native: Simple tuple of login details.
save(FileSystem, Path, boolean) - Method in class org.apache.hadoop.fs.s3a.commit.files.PendingSet
save(FileSystem, Path, boolean) - Method in class org.apache.hadoop.fs.s3a.commit.files.PersistentCommitData: Save to a hadoop filesystem.
save(FileSystem, Path, boolean) - Method in class org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit
save(FileSystem, Path, boolean) - Method in class org.apache.hadoop.fs.s3a.commit.files.SuccessData
SECONDS_FLAG - Static variable in class org.apache.hadoop.fs.s3a.s3guard.S3GuardTool
SECRET_KEY - Static variable in class org.apache.hadoop.fs.s3a.Constants
SECURE_CONNECTIONS - Static variable in class org.apache.hadoop.fs.s3a.Constants
seek(long) - Method in class org.apache.hadoop.fs.s3a.S3AInputStream
seekBackwards(long) - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation.InputStreamStatistics: Seek backwards, incrementing the seek and backward seek counters.
seekForwards(long) - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation.InputStreamStatistics: Record a forward seek, adding a seek operation, a forward seek operation, and any bytes skipped.
seekOperations - Variable in class org.apache.hadoop.fs.s3a.S3AInstrumentation.InputStreamStatistics
seekToNewSource(long) - Method in class org.apache.hadoop.fs.s3a.S3AInputStream
serializer() - Static method in class org.apache.hadoop.fs.s3a.commit.files.PendingSet: Get a JSON serializer for this class.
serializer() - Static method in class org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit: Get a JSON serializer for this class.
SERVER_SIDE_ENCRYPTION_AES256 - Static variable in class org.apache.hadoop.fs.s3a.Constants: Deprecated.
SERVER_SIDE_ENCRYPTION_ALGORITHM - Static variable in class org.apache.hadoop.fs.s3a.Constants
SERVER_SIDE_ENCRYPTION_KEY - Static variable in class org.apache.hadoop.fs.s3a.Constants: Used to specify which AWS KMS key to use if Constants.SERVER_SIDE_ENCRYPTION_ALGORITHM is SSE-KMS (will default to aws/s3 master key if left blank).
SESSION_TOKEN - Static variable in class org.apache.hadoop.fs.s3a.Constants
setAllowed(boolean) - Method in class org.apache.hadoop.fs.s3a.auth.RoleModel.Statement
setAmazonS3Client(AmazonS3) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Set the client -used in mocking tests to force in a different client.
setAuthoritative(boolean) - Method in class org.apache.hadoop.fs.s3a.s3guard.DirListingMetadata: Marks this directory listing as full and authoritative.
setAuthoritativeDir(boolean) - Method in class org.apache.hadoop.fs.s3a.s3guard.DDBPathMetadata
setBucket(String) - Method in class org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit
setBucket(String) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Set the bucket.
setBucketName(String) - Method in class org.apache.hadoop.fs.s3a.InconsistentS3Object
setBucketOption(Configuration, String, String, String) - Static method in class org.apache.hadoop.fs.s3a.S3AUtils: Set a bucket-specific property to a particular value.
setCommits(List<SinglePendingCommit>) - Method in class org.apache.hadoop.fs.s3a.commit.files.PendingSet
setCommitter(String) - Method in class org.apache.hadoop.fs.s3a.commit.files.SuccessData
setConf(Configuration) - Method in class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter
setCreated(long) - Method in class org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit
setDate(String) - Method in class org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit
setDate(String) - Method in class org.apache.hadoop.fs.s3a.commit.files.SuccessData
setDescription(String) - Method in class org.apache.hadoop.fs.s3a.commit.files.SuccessData
setDestFS(FileSystem) - Method in class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter: Set the destination FS: the FS of the final output.
setDestinationKey(String) - Method in class org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit
setDiagnostics(Map<String, String>) - Method in class org.apache.hadoop.fs.s3a.commit.files.SuccessData
setDirListingMetadata(DirListingMetadata) - Method in class org.apache.hadoop.fs.s3a.s3guard.LocalMetadataEntry
setEtags(List<String>) - Method in class org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit
setExtraData(Map<String, String>) - Method in class org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit
setFailureInjectionPolicy(S3AFileSystem, FailureInjectionPolicy) - Static method in class org.apache.hadoop.fs.s3a.InconsistentAmazonS3Client: A way for tests to patch in a different fault injection policy at runtime.
setFailureLimit(int) - Method in class org.apache.hadoop.fs.s3a.FailureInjectionPolicy
setFailureLimit(int) - Method in class org.apache.hadoop.fs.s3a.InconsistentAmazonS3Client: Set the limit on failures before all operations pass through.
setFilename(String) - Method in class org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit
setFilenames(List<String>) - Method in class org.apache.hadoop.fs.s3a.commit.files.SuccessData
setFilesystem(S3AFileSystem) - Method in class org.apache.hadoop.fs.s3a.s3guard.S3GuardTool
setHostname(String) - Method in class org.apache.hadoop.fs.s3a.commit.files.SuccessData
setInputPolicy(S3AInputPolicy) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Change the input policy for this FS.
setJobId(String) - Method in class org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit
setKey(String) - Method in class org.apache.hadoop.fs.s3a.InconsistentS3Object
setLength(long) - Method in class org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit
setMetrics(Map<String, Long>) - Method in class org.apache.hadoop.fs.s3a.commit.files.SuccessData
setName(String) - Method in class org.apache.hadoop.fs.s3a.commit.files.SuccessData
setObjectContent(S3ObjectInputStream) - Method in class org.apache.hadoop.fs.s3a.InconsistentS3Object
setObjectContent(InputStream) - Method in class org.apache.hadoop.fs.s3a.InconsistentS3Object
setObjectMetadata(ObjectMetadata) - Method in class org.apache.hadoop.fs.s3a.InconsistentS3Object
setOptionalCopyObjectRequestParameters(CopyObjectRequest) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem
setOptionalMultipartUploadRequestParameters(InitiateMultipartUploadRequest) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem
setOptionalUploadPartRequestParameters(UploadPartRequest) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Sets server side encryption parameters to the part upload request when encryption is enabled.
setOutputPath(Path) - Method in class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter: Set the output path.
setPathMetadata(PathMetadata) - Method in class org.apache.hadoop.fs.s3a.s3guard.LocalMetadataEntry
setReadahead(Long) - Method in class org.apache.hadoop.fs.s3a.S3AInputStream
setRedirectLocation(String) - Method in class org.apache.hadoop.fs.s3a.InconsistentS3Object
setRequesterCharged(boolean) - Method in class org.apache.hadoop.fs.s3a.InconsistentS3Object
setSaved(long) - Method in class org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit
setStore(MetadataStore) - Method in class org.apache.hadoop.fs.s3a.s3guard.S3GuardTool
setTaggingCount(Integer) - Method in class org.apache.hadoop.fs.s3a.InconsistentS3Object
setTaskId(String) - Method in class org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit
setText(String) - Method in class org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit
setThrottleProbability(float) - Method in class org.apache.hadoop.fs.s3a.FailureInjectionPolicy: Set the probability of throttling a request.
setThrottleProbability(float) - Method in class org.apache.hadoop.fs.s3a.InconsistentAmazonS3Client: Set the probability of throttling a request.
setTimestamp(long) - Method in class org.apache.hadoop.fs.s3a.commit.files.SuccessData
setupJob(JobContext) - Method in class org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter
setupJob(JobContext) - Method in class org.apache.hadoop.fs.s3a.commit.staging.DirectoryStagingCommitter
setupJob(JobContext) - Method in class org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter: Set up the job, including calling the same method on the wrapped committer.
setUploadId(String) - Method in class org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit
setupTask(TaskAttemptContext) - Method in class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter: Base job setup deletes the success marker.
setupTask(TaskAttemptContext) - Method in class org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter
setupTask(TaskAttemptContext) - Method in class org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter
setUri(String) - Method in class org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit
setUri(URI) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Set the URI field through S3xLoginHelper.
setVersion(int) - Method in class org.apache.hadoop.fs.s3a.commit.files.PendingSet
setVersion(int) - Method in class org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit
setWorkingDirectory(Path) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Set the current working directory for the given file system.
setWorkingDirectory(Path) - Method in class org.apache.hadoop.fs.s3native.NativeS3FileSystem: Deprecated.
setWorkPath(Path) - Method in class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter: Set the work path for this committer.
share() - Method in class org.apache.hadoop.fs.s3a.AWSCredentialProviderList: Get a reference to this object with an updated reference count.
shareCredentials(String) - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Get a shared copy of the AWS credentials, with its reference counter updated.
shouldDelay(String) - Method in class org.apache.hadoop.fs.s3a.FailureInjectionPolicy: Should we delay listing visibility for this key?
shouldRetry(Exception, int, int, boolean) - Method in class org.apache.hadoop.fs.s3a.S3ARetryPolicy
sid - Variable in class org.apache.hadoop.fs.s3a.auth.RoleModel.Statement
signifiesConnectionBroken(SdkBaseException) - Static method in class org.apache.hadoop.fs.s3a.S3AUtils: Cue that an AWS exception is likely to be an EOF Exception based on the message coming back from an XML/JSON parser.
SIGNING_ALGORITHM - Static variable in class org.apache.hadoop.fs.s3a.Constants
SimpleAWSCredentialsProvider - Class in org.apache.hadoop.fs.s3a: Support simple credentials for authenticating with AWS.
SimpleAWSCredentialsProvider(URI, Configuration) - Constructor for class org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider
SinglePendingCommit - Class in org.apache.hadoop.fs.s3a.commit.files: This is the serialization format for uploads yet to be committerd.
SinglePendingCommit() - Constructor for class org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit
size() - Method in class org.apache.hadoop.fs.s3a.commit.files.PendingSet: Number of commits.
skip(long) - Method in class org.apache.hadoop.fs.s3a.InconsistentS3Object.InconsistentS3InputStream
SOCKET_RECV_BUFFER - Static variable in class org.apache.hadoop.fs.s3a.Constants
SOCKET_SEND_BUFFER - Static variable in class org.apache.hadoop.fs.s3a.Constants
SOCKET_TIMEOUT - Static variable in class org.apache.hadoop.fs.s3a.Constants
SPARK_APP_ID - Static variable in class org.apache.hadoop.fs.s3a.commit.staging.StagingCommitterConstants: The App ID for jobs.
SPARK_WRITE_UUID - Static variable in class org.apache.hadoop.fs.s3a.commit.staging.StagingCommitterConstants: The UUID for jobs: "spark.sql.sources.writeJobUUID".
splitPathToElements(Path) - Static method in class org.apache.hadoop.fs.s3a.commit.MagicCommitPaths: Take an absolute path, split it into a list of elements.
SSE_C_NO_KEY_ERROR - Static variable in class org.apache.hadoop.fs.s3a.S3AUtils: Encryption SSE-C used but the config lacks an encryption key.
SSE_S3_WITH_KEY_ERROR - Static variable in class org.apache.hadoop.fs.s3a.S3AUtils: Encryption SSE-S3 is used but the caller also set an encryption key.
STAGING_COMMITTER_FACTORY - Static variable in class org.apache.hadoop.fs.s3a.commit.InternalCommitterConstants: Directory committer factory: "org.apache.hadoop.fs.s3a.commit.staging.StagingCommitterFactory".
STAGING_UPLOADS - Static variable in class org.apache.hadoop.fs.s3a.commit.staging.StagingCommitterConstants: Filename used under ~/${UUID} for the staging files.
StagingCommitter - Class in org.apache.hadoop.fs.s3a.commit.staging: Committer based on the contributed work of the Netflix multipart committers. The working directory of each task is actually under a temporary path in the local filesystem; jobs write directly into it.
StagingCommitter(Path, TaskAttemptContext) - Constructor for class org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter: Committer for a single task attempt.
StagingCommitterConstants - Class in org.apache.hadoop.fs.s3a.commit.staging: Internal staging committer constants.
StagingCommitterFactory - Class in org.apache.hadoop.fs.s3a.commit.staging: Factory for the staging committer.
StagingCommitterFactory() - Constructor for class org.apache.hadoop.fs.s3a.commit.staging.StagingCommitterFactory
statement - Variable in class org.apache.hadoop.fs.s3a.auth.RoleModel.Policy
statement(boolean, String, String...) - Static method in class org.apache.hadoop.fs.s3a.auth.RoleModel: Create a statement.
statement(boolean, Path, boolean, boolean, String...) - Static method in class org.apache.hadoop.fs.s3a.auth.RoleModel: Create a statement.
Statement(RoleModel.Effects) - Constructor for class org.apache.hadoop.fs.s3a.auth.RoleModel.Statement
STATEMENT_ALL_DDB - Static variable in class org.apache.hadoop.fs.s3a.auth.RolePolicies: Statement to allow all DDB access.
STATEMENT_ALL_S3 - Static variable in class org.apache.hadoop.fs.s3a.auth.RolePolicies: Allow all S3 Operations.
STATEMENT_ALLOW_SSE_KMS_READ - Static variable in class org.apache.hadoop.fs.s3a.auth.RolePolicies: Statement to allow read access to KMS keys, so the ability to read SSE-KMS data,, but not decrypt it.
STATEMENT_ALLOW_SSE_KMS_RW - Static variable in class org.apache.hadoop.fs.s3a.auth.RolePolicies: Statement to allow KMS R/W access access, so full use of SSE-KMS.
STATEMENT_S3GUARD_CLIENT - Static variable in class org.apache.hadoop.fs.s3a.auth.RolePolicies: Statement to allow all client operations needed for S3Guard, but none of the admin operations.
Statistic - Enum in org.apache.hadoop.fs.s3a: Statistic which are collected in S3A.
STATUS_CODE - Static variable in exception org.apache.hadoop.fs.s3a.AWSBadRequestException: HTTP status code which signals this failure mode was triggered: 400.
STATUS_CODE - Static variable in exception org.apache.hadoop.fs.s3a.AWSServiceThrottledException: HTTP status code which signals this failure mode was triggered: 503.
stopAbortsOnFailure() - Method in class org.apache.hadoop.fs.s3a.commit.Tasks.Builder
stopOnFailure() - Method in class org.apache.hadoop.fs.s3a.commit.Tasks.Builder
stopRevertsOnFailure() - Method in class org.apache.hadoop.fs.s3a.commit.Tasks.Builder
STORE_CAPABILITY_MAGIC_COMMITTER - Static variable in class org.apache.hadoop.fs.s3a.commit.CommitConstants: Flag to indicate that a store supports magic committers.
storeClosed() - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation.S3GuardInstrumentation
storeNotFound(FileNotFoundException) - Static method in class org.apache.hadoop.fs.s3a.s3guard.S3GuardTool: Handle store not found by converting to an exit exception with specific error code.
STREAM_CAPABILITY_MAGIC_OUTPUT - Static variable in class org.apache.hadoop.fs.s3a.commit.CommitConstants: Flag to indicate whether a stream is a magic output stream; returned in StreamCapabilities Value: "s3a:magic.output.stream".
streamClose(boolean, long) - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation.InputStreamStatistics: The inner stream was closed.
streamOpened() - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation.InputStreamStatistics: The inner stream was opened.
stringify(AmazonServiceException) - Static method in class org.apache.hadoop.fs.s3a.S3AUtils: Get low level details of an amazon exception for logging; multi-line.
stringify(AmazonS3Exception) - Static method in class org.apache.hadoop.fs.s3a.S3AUtils: Get low level details of an amazon exception for logging; multi-line.
stringify(S3ObjectSummary) - Static method in class org.apache.hadoop.fs.s3a.S3AUtils: String information about a summary entry for debug messages.
STSClientFactory - Class in org.apache.hadoop.fs.s3a.auth: Factory for creating STS Clients.
STSClientFactory() - Constructor for class org.apache.hadoop.fs.s3a.auth.STSClientFactory
SuccessData - Class in org.apache.hadoop.fs.s3a.commit.files: Summary data saved into a _SUCCESS marker file.
SuccessData() - Constructor for class org.apache.hadoop.fs.s3a.commit.files.SuccessData
suppressExceptions() - Method in class org.apache.hadoop.fs.s3a.commit.Tasks.Builder
suppressExceptions(boolean) - Method in class org.apache.hadoop.fs.s3a.commit.Tasks.Builder

T

TABLE_ROOT - Static variable in class org.apache.hadoop.fs.s3a.commit.staging.StagingCommitterConstants: Name of the root partition :"table_root".
TAG_FLAG - Static variable in class org.apache.hadoop.fs.s3a.s3guard.S3GuardTool
tagTable() - Method in class org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore: Add tags from configuration to the existing DynamoDB table.
taskCompleted(boolean) - Method in class org.apache.hadoop.fs.s3a.commit.CommitOperations: Note that a task has completed.
taskCompleted(boolean) - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation.CommitterStatistics
Tasks - Class in org.apache.hadoop.fs.s3a.commit: Utility class for parallel execution, takes closures for the various actions.
Tasks.Builder<I> - Class in org.apache.hadoop.fs.s3a.commit: Builder for task execution.
Tasks.FailureTask<I,E extends Exception> - Interface in org.apache.hadoop.fs.s3a.commit: Callback invoked on a failure.
Tasks.Task<I,E extends Exception> - Interface in org.apache.hadoop.fs.s3a.commit: Callback invoked to process an item.
TEMP_DATA - Static variable in class org.apache.hadoop.fs.s3a.commit.CommitConstants: Temp data which is not auto-committed: "__temp-data".
tempDirForStaging(FileSystem, Configuration) - Static method in class org.apache.hadoop.fs.s3a.commit.staging.Paths: Try to come up with a good temp directory for different filesystems.
TEMPORARY - Static variable in class org.apache.hadoop.fs.s3a.commit.CommitConstants: This is the "Pending" directory of the FileOutputCommitter; data written here is, in that algorithm, renamed into place.
TemporaryAWSCredentialsProvider - Class in org.apache.hadoop.fs.s3a: Support session credentials for authenticating with AWS.
TemporaryAWSCredentialsProvider(Configuration) - Constructor for class org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider
TemporaryAWSCredentialsProvider(URI, Configuration) - Constructor for class org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider
throttled() - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation.S3GuardInstrumentation: Throttled request.
throttlePolicy - Variable in class org.apache.hadoop.fs.s3a.S3ARetryPolicy: Policy for throttle requests, which are considered repeatable, even for non-idempotent calls, as the service rejected the call entirely.
time() - Method in class org.apache.hadoop.fs.s3a.commit.Duration
toBytes() - Method in class org.apache.hadoop.fs.s3a.commit.files.PendingSet
toBytes() - Method in class org.apache.hadoop.fs.s3a.commit.files.PersistentCommitData: Serialize to JSON and then to a byte array, after performing a preflight validation of the data to be saved.
toBytes() - Method in class org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit
toBytes() - Method in class org.apache.hadoop.fs.s3a.commit.files.SuccessData
toJson(RoleModel.Policy) - Method in class org.apache.hadoop.fs.s3a.auth.RoleModel
toMap() - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation: Copy all the metrics to a map of (name, long-value).
tombstone(Path) - Static method in class org.apache.hadoop.fs.s3a.s3guard.PathMetadata: Create a tombstone from the current time.
toPartEtags(List<String>) - Static method in class org.apache.hadoop.fs.s3a.commit.CommitOperations: Convert an ordered list of strings to a list of index etag parts.
toString() - Method in class org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider
toString() - Method in class org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider
toString() - Method in class org.apache.hadoop.fs.s3a.AWSCredentialProviderList: The string value is this class name and the string values of nested providers.
toString() - Method in class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter
toString() - Method in class org.apache.hadoop.fs.s3a.commit.CommitOperations.MaybeIOE
toString() - Method in class org.apache.hadoop.fs.s3a.commit.CommitOperations
toString() - Method in class org.apache.hadoop.fs.s3a.commit.Duration
toString() - Method in class org.apache.hadoop.fs.s3a.commit.DurationInfo
toString() - Method in class org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit
toString() - Method in class org.apache.hadoop.fs.s3a.commit.files.SuccessData
toString() - Method in class org.apache.hadoop.fs.s3a.commit.magic.MagicCommitTracker
toString() - Method in class org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter
toString() - Method in class org.apache.hadoop.fs.s3a.commit.PutTracker
toString() - Method in class org.apache.hadoop.fs.s3a.commit.staging.PartitionedStagingCommitter
toString() - Method in class org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter
toString() - Method in class org.apache.hadoop.fs.s3a.FailureInjectionPolicy
toString() - Method in class org.apache.hadoop.fs.s3a.impl.ChangeTracker
toString() - Method in class org.apache.hadoop.fs.s3a.InconsistentAmazonS3Client
toString() - Method in class org.apache.hadoop.fs.s3a.InconsistentS3Object
toString() - Method in class org.apache.hadoop.fs.s3a.S3AFileStatus
toString() - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem
toString() - Method in enum org.apache.hadoop.fs.s3a.S3AInputPolicy
toString() - Method in class org.apache.hadoop.fs.s3a.S3AInputStream: String value includes statistics as well as stream state.
toString() - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation.InputStreamStatistics: String operator describes all the current statistics.
toString() - Method in class org.apache.hadoop.fs.s3a.S3AInstrumentation.OutputStreamStatistics
toString() - Method in class org.apache.hadoop.fs.s3a.S3AReadOpContext
toString() - Method in class org.apache.hadoop.fs.s3a.s3guard.DirListingMetadata
toString() - Method in class org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
toString() - Method in class org.apache.hadoop.fs.s3a.s3guard.LocalMetadataEntry
toString() - Method in class org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore
toString() - Method in class org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore
toString() - Method in class org.apache.hadoop.fs.s3a.s3guard.PathMetadata
toString() - Method in class org.apache.hadoop.fs.s3a.S3ListRequest
toString() - Method in class org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider
toString() - Method in enum org.apache.hadoop.fs.s3a.Statistic: The string value is simply the symbol.
toString() - Method in class org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider
toString() - Method in class org.apache.hadoop.fs.s3a.WriteOperationHelper: The toString method is intended to be used in logging/toString calls.
toString(URI) - Static method in class org.apache.hadoop.fs.s3native.S3xLoginHelper: Create a stripped down string value for error messages.
touch(long) - Method in class org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit: Set the various timestamp fields to the supplied value.
toUri(String) - Static method in class org.apache.hadoop.fs.s3a.s3guard.S3GuardTool: Convert a path to a URI, catching any URISyntaxException and converting to an invalid args exception.
translateDynamoDBException(String, String, AmazonDynamoDBException) - Static method in class org.apache.hadoop.fs.s3a.S3AUtils: Translate a DynamoDB exception into an IOException.
translateException(String, Path, AmazonClientException) - Static method in class org.apache.hadoop.fs.s3a.S3AUtils: Translate an exception raised in an operation into an IOException.
translateException(String, String, SdkBaseException) - Static method in class org.apache.hadoop.fs.s3a.S3AUtils: Translate an exception raised in an operation into an IOException.
translateMultiObjectDeleteException(String, MultiObjectDeleteException) - Static method in class org.apache.hadoop.fs.s3a.S3AUtils: A MultiObjectDeleteException is raised if one or more delete objects listed in a bulk DELETE operation failed.
Tristate - Enum in org.apache.hadoop.fs.s3a: Simple enum to express {true, false, don't know}.
trueWithProbability(float) - Static method in class org.apache.hadoop.fs.s3a.FailureInjectionPolicy

U

updateParameters(Map<String, String>) - Method in class org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore
updateParameters(Map<String, String>) - Method in class org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore
updateParameters(Map<String, String>) - Method in interface org.apache.hadoop.fs.s3a.s3guard.MetadataStore: Tune/update parameters for an existing table.
updateParameters(Map<String, String>) - Method in class org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore
uploadCompleted() - Method in class org.apache.hadoop.fs.s3a.ProgressableProgressListener: Method to invoke after upload has completed.
uploadFileToPendingCommit(File, Path, String, long) - Method in class org.apache.hadoop.fs.s3a.commit.CommitOperations: Upload all the data in the local file, returning the information needed to commit the work.
UploadInfo - Class in org.apache.hadoop.fs.s3a: Simple struct that contains information about a S3 upload.
UploadInfo(Upload, long) - Constructor for class org.apache.hadoop.fs.s3a.UploadInfo
UploadIterator(AmazonS3, Invoker, String, int, String) - Constructor for class org.apache.hadoop.fs.s3a.MultipartUtils.UploadIterator
uploadObject(PutObjectRequest) - Method in class org.apache.hadoop.fs.s3a.WriteOperationHelper: PUT an object via the transfer manager.
uploadPart(UploadPartRequest) - Method in class org.apache.hadoop.fs.s3a.InconsistentAmazonS3Client
uploadPart(UploadPartRequest) - Method in class org.apache.hadoop.fs.s3a.WriteOperationHelper: Upload part of a multi-partition file.
USER_AGENT_PREFIX - Static variable in class org.apache.hadoop.fs.s3a.Constants
userAborted(String, Object...) - Static method in class org.apache.hadoop.fs.s3a.s3guard.S3GuardTool: Build the exception to raise on user-aborted action.
useUniqueFilenames() - Method in class org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter: Is this committer using unique filenames?

V

v1(ListObjectsRequest) - Static method in class org.apache.hadoop.fs.s3a.S3ListRequest: Restricted constructors to ensure v1 or v2, not both.
v1(ObjectListing) - Static method in class org.apache.hadoop.fs.s3a.S3ListResult: Restricted constructors to ensure v1 or v2, not both.
v2(ListObjectsV2Request) - Static method in class org.apache.hadoop.fs.s3a.S3ListRequest: Restricted constructors to ensure v1 or v2, not both.
v2(ListObjectsV2Result) - Static method in class org.apache.hadoop.fs.s3a.S3ListResult: Restricted constructors to ensure v1 or v2, not both.
validate() - Method in class org.apache.hadoop.fs.s3a.auth.RoleModel.Policy: Validation includes validating all statements.
validate() - Method in class org.apache.hadoop.fs.s3a.auth.RoleModel.RoleElt: validation operation.
validate() - Method in class org.apache.hadoop.fs.s3a.auth.RoleModel.Statement
validate() - Method in class org.apache.hadoop.fs.s3a.commit.files.PendingSet: Validate the data: those fields which must be non empty, must be set.
validate() - Method in class org.apache.hadoop.fs.s3a.commit.files.PersistentCommitData: Validate the data: those fields which must be non empty, must be set.
validate() - Method in class org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit
validate() - Method in class org.apache.hadoop.fs.s3a.commit.files.SuccessData
validateCollectionClass(Iterable, Class) - Static method in class org.apache.hadoop.fs.s3a.commit.CommitUtils: Verify that all instances in a collection are of the given class.
ValidationFailure - Exception in org.apache.hadoop.fs.s3a.commit: Exception raised on validation failures; kept as an IOException for consistency with other failures.
ValidationFailure(String, Object...) - Constructor for exception org.apache.hadoop.fs.s3a.commit.ValidationFailure: Create an instance with string formatting applied to the message and arguments.
value() - Method in class org.apache.hadoop.fs.s3a.commit.Duration
valueOf(String) - Static method in enum org.apache.hadoop.fs.s3a.auth.RoleModel.Effects: Returns the enum constant of this type with the specified name.
valueOf(String) - Static method in enum org.apache.hadoop.fs.s3a.commit.staging.ConflictResolution: Returns the enum constant of this type with the specified name.
valueOf(String) - Static method in enum org.apache.hadoop.fs.s3a.impl.ChangeDetectionPolicy.Mode: Returns the enum constant of this type with the specified name.
valueOf(String) - Static method in enum org.apache.hadoop.fs.s3a.impl.ChangeDetectionPolicy.Source: Returns the enum constant of this type with the specified name.
valueOf(String) - Static method in enum org.apache.hadoop.fs.s3a.S3AEncryptionMethods: Returns the enum constant of this type with the specified name.
valueOf(String) - Static method in enum org.apache.hadoop.fs.s3a.S3AInputPolicy: Returns the enum constant of this type with the specified name.
valueOf(String) - Static method in enum org.apache.hadoop.fs.s3a.Statistic: Returns the enum constant of this type with the specified name.
valueOf(String) - Static method in enum org.apache.hadoop.fs.s3a.Tristate: Returns the enum constant of this type with the specified name.
values() - Static method in enum org.apache.hadoop.fs.s3a.auth.RoleModel.Effects: Returns an array containing the constants of this enum type, in the order they are declared.
values() - Static method in enum org.apache.hadoop.fs.s3a.commit.staging.ConflictResolution: Returns an array containing the constants of this enum type, in the order they are declared.
values() - Static method in enum org.apache.hadoop.fs.s3a.impl.ChangeDetectionPolicy.Mode: Returns an array containing the constants of this enum type, in the order they are declared.
values() - Static method in enum org.apache.hadoop.fs.s3a.impl.ChangeDetectionPolicy.Source: Returns an array containing the constants of this enum type, in the order they are declared.
values() - Static method in enum org.apache.hadoop.fs.s3a.S3AEncryptionMethods: Returns an array containing the constants of this enum type, in the order they are declared.
values() - Static method in enum org.apache.hadoop.fs.s3a.S3AInputPolicy: Returns an array containing the constants of this enum type, in the order they are declared.
values() - Static method in enum org.apache.hadoop.fs.s3a.Statistic: Returns an array containing the constants of this enum type, in the order they are declared.
values() - Static method in enum org.apache.hadoop.fs.s3a.Tristate: Returns an array containing the constants of this enum type, in the order they are declared.
verify(boolean, String, Object...) - Static method in exception org.apache.hadoop.fs.s3a.commit.ValidationFailure: Verify that a condition holds.
verifyBucketExists() - Method in class org.apache.hadoop.fs.s3a.S3AFileSystem: Verify that the bucket exists.
verifyIsMagicCommitFS(S3AFileSystem) - Static method in class org.apache.hadoop.fs.s3a.commit.CommitUtils: Verify that an S3A FS instance is a magic commit FS.
verifyIsMagicCommitPath(S3AFileSystem, Path) - Static method in class org.apache.hadoop.fs.s3a.commit.CommitUtils: Verify that the path is a magic one.
verifyIsS3AFS(FileSystem, Path) - Static method in class org.apache.hadoop.fs.s3a.commit.CommitUtils: Verify that an FS is an S3A FS.
version - Variable in class org.apache.hadoop.fs.s3a.auth.RoleModel.Policy
VERSION - Static variable in class org.apache.hadoop.fs.s3a.auth.RoleModel
VERSION - Static variable in class org.apache.hadoop.fs.s3a.commit.files.PendingSet: Supported version value: 1.
VERSION - Static variable in class org.apache.hadoop.fs.s3a.commit.files.PersistentCommitData: Supported version value: 1.
VERSION - Static variable in class org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore: Current version number.
VERSION_MARKER - Static variable in class org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore: parent/child name to use in the version marker.

W

warn(String, Object...) - Method in class org.apache.hadoop.fs.s3a.impl.LogExactlyOnce
withExitCode(boolean) - Method in exception org.apache.hadoop.fs.s3a.RenameFailedException: Set the exit code.
withoutTombstones() - Method in class org.apache.hadoop.fs.s3a.s3guard.DirListingMetadata
WRITE_FLAG - Static variable in class org.apache.hadoop.fs.s3a.s3guard.S3GuardTool
writeFailed(Exception) - Method in class org.apache.hadoop.fs.s3a.WriteOperationHelper: Callback on a write failure.
WriteOperationHelper - Class in org.apache.hadoop.fs.s3a: Helper for low-level operations against an S3 Bucket for writing data and creating and committing pending writes.
WriteOperationHelper(S3AFileSystem, Configuration) - Constructor for class org.apache.hadoop.fs.s3a.WriteOperationHelper: Constructor.
writeSuccessful(long) - Method in class org.apache.hadoop.fs.s3a.WriteOperationHelper: Callback on a successful write.

_

_SUCCESS - Static variable in class org.apache.hadoop.fs.s3a.commit.CommitConstants: Marker file to create on success: "_SUCCESS".

A B C D E F G H I J K L M N O P Q R S T U V W _