org.apache.hadoop.fs.s3a.impl

Class StoreContext

java.lang.Object

org.apache.hadoop.fs.s3a.impl.StoreContext

All Implemented Interfaces:

org.apache.hadoop.fs.store.audit.ActiveThreadSpanSource<org.apache.hadoop.fs.store.audit.AuditSpan>

@InterfaceAudience.LimitedPrivate(value="S3A Filesystem and extensions")
 @InterfaceStability.Unstable
public class StoreContext
extends Object
implements org.apache.hadoop.fs.store.audit.ActiveThreadSpanSource<org.apache.hadoop.fs.store.audit.AuditSpan>

This class provides the core context of the S3A filesystem to subsidiary components, without exposing the entire parent class. This is eliminate explicit recursive coupling. Where methods on the FS are to be invoked, they are referenced via the ContextAccessors interface, so tests can implement their own. Warning: this really is private and unstable. Do not use outside the org.apache.hadoop.fs.s3a package, or in extension points such as DelegationTokens.

Method Summary

Modifier and Type	Method and Description
File	createTempFile(String prefix, long size) Create a temporary file somewhere.
ExecutorService	createThrottledExecutor() Create a new executor with the capacity defined in executorCapacity.
ExecutorService	createThrottledExecutor(int capacity) Create a new executor service with a given capacity.
void	decrementGauge(Statistic statistic, long count) Decrement a gauge by a specific value.
String	fullKey(S3AFileStatus stat) Build the full S3 key for a request from the status entry, possibly adding a "/" if it represents directory and it does not have a trailing slash already.
org.apache.hadoop.fs.store.audit.AuditSpan	getActiveAuditSpan() Return the active audit span.
org.apache.hadoop.fs.store.audit.AuditSpanSource<AuditSpanS3A>	getAuditor() Get the auditor.
String	getBucket()
String	getBucketLocation() Get the location of the bucket.
ChangeDetectionPolicy	getChangeDetectionPolicy()
org.apache.hadoop.conf.Configuration	getConfiguration()
ContextAccessors	getContextAccessors()
ExecutorService	getExecutor()
URI	getFsURI()
S3AInputPolicy	getInputPolicy()
S3AStatisticsContext	getInstrumentation() Get the statistics context for this StoreContext.
Invoker	getInvoker()
org.apache.hadoop.security.UserGroupInformation	getOwner() Get the owner of the filesystem.
RequestFactory	getRequestFactory() Get the request factory.
S3AStorageStatistics	getStorageStatistics() Get the storage statistics of this filesystem.
String	getUsername()
void	incrementGauge(Statistic statistic, long count) Increment a gauge by a specific value.
void	incrementStatistic(Statistic statistic) Increment a statistic by 1.
void	incrementStatistic(Statistic statistic, long count) Increment a statistic by a specific value.
boolean	isCSEEnabled() return if the store context have client side encryption enabled.
boolean	isMultiObjectDeleteEnabled()
boolean	isUseListV1()
org.apache.hadoop.fs.Path	keyToPath(String key) Convert a key to a fully qualified path.
org.apache.hadoop.fs.Path	makeQualified(org.apache.hadoop.fs.Path path) Qualify a path.
String	pathToKey(org.apache.hadoop.fs.Path path) Turns a path (relative or otherwise) into an S3 key.
<T> CompletableFuture<T>	submit(CompletableFuture<T> future, Callable<T> call) Submit a closure for execution in the executor returned by getExecutor().

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

getFsURI

public URI getFsURI()

getBucket

public String getBucket()

getConfiguration

public org.apache.hadoop.conf.Configuration getConfiguration()

getUsername

public String getUsername()

getExecutor

public ExecutorService getExecutor()

getInvoker

public Invoker getInvoker()

getInstrumentation

public S3AStatisticsContext getInstrumentation()

Get the statistics context for this StoreContext.

Returns:

the statistics context this store context was created with.

getInputPolicy

public S3AInputPolicy getInputPolicy()

getChangeDetectionPolicy

public ChangeDetectionPolicy getChangeDetectionPolicy()

isMultiObjectDeleteEnabled

public boolean isMultiObjectDeleteEnabled()

isUseListV1

public boolean isUseListV1()

getContextAccessors

public ContextAccessors getContextAccessors()

keyToPath

public org.apache.hadoop.fs.Path keyToPath(String key)

Convert a key to a fully qualified path.

Parameters:

key - input key

Returns:

the fully qualified path including URI scheme and bucket name.

pathToKey

public String pathToKey(org.apache.hadoop.fs.Path path)

Turns a path (relative or otherwise) into an S3 key.

Parameters:

path - input path, may be relative to the working dir

Returns:

a key excluding the leading "/", or, if it is the root path, ""

makeQualified

public org.apache.hadoop.fs.Path makeQualified(org.apache.hadoop.fs.Path path)

Qualify a path.

Parameters:

path - path to qualify/normalize

Returns:

possibly new path.

getStorageStatistics

public S3AStorageStatistics getStorageStatistics()

Get the storage statistics of this filesystem.

Returns:

the storage statistics

incrementStatistic

public void incrementStatistic(Statistic statistic)

Increment a statistic by 1. This increments both the instrumentation and storage statistics.

Parameters:

statistic - The operation to increment

incrementStatistic

public void incrementStatistic(Statistic statistic,
                               long count)

Increment a statistic by a specific value. This increments both the instrumentation and storage statistics.

Parameters:

statistic - The operation to increment

count - the count to increment

decrementGauge

public void decrementGauge(Statistic statistic,
                           long count)

Decrement a gauge by a specific value.

Parameters:

statistic - The operation to decrement

count - the count to decrement

incrementGauge

public void incrementGauge(Statistic statistic,
                           long count)

Increment a gauge by a specific value.

Parameters:

statistic - The operation to increment

count - the count to increment

createThrottledExecutor

public ExecutorService createThrottledExecutor(int capacity)

Create a new executor service with a given capacity. This executor submits works to the executor, using a SemaphoredDelegatingExecutor to limit the number of requests coming in from a specific client. Because this delegates to an existing thread pool, the cost of creating a new instance here is low. As the throttling is per instance, separate instances should be created for each operation which wishes to execute work in parallel without saturating the base executor. This is important if either the duration of each operation is long or the submission rate of work is high.

Parameters:

capacity - maximum capacity of this executor.

Returns:

an executor for submitting work.

createThrottledExecutor

public ExecutorService createThrottledExecutor()

Create a new executor with the capacity defined in executorCapacity.

Returns:

a new executor for exclusive use by the caller.

getOwner

public org.apache.hadoop.security.UserGroupInformation getOwner()

Get the owner of the filesystem.

Returns:

the user who created this filesystem.

createTempFile

public File createTempFile(String prefix,
                           long size)
                    throws IOException

Create a temporary file somewhere.

Parameters:

prefix - prefix for the temporary file

size - expected size.

Returns:

a file reference.

Throws:

IOException - failure.

getBucketLocation

public String getBucketLocation()
                         throws IOException

Get the location of the bucket.

Returns:

the bucket location.

Throws:

IOException - failure.

fullKey

public String fullKey(S3AFileStatus stat)

Build the full S3 key for a request from the status entry, possibly adding a "/" if it represents directory and it does not have a trailing slash already.

Parameters:

stat - status to build the key from

Returns:

a key for a delete request

submit

public <T> CompletableFuture<T> submit(CompletableFuture<T> future,
                                       Callable<T> call)

Submit a closure for execution in the executor returned by getExecutor().

Type Parameters:

T - type of future

Parameters:

future - future for the result.

call - callable to invoke.

Returns:

the future passed in

getAuditor

public org.apache.hadoop.fs.store.audit.AuditSpanSource<AuditSpanS3A> getAuditor()

Get the auditor.

Returns:

auditor.

getActiveAuditSpan

public org.apache.hadoop.fs.store.audit.AuditSpan getActiveAuditSpan()

Return the active audit span. This is thread local -it MUST be passed into workers. To ensure the correct span is used, it SHOULD be collected as early as possible, ideally during construction/ or service init/start.

Specified by:

getActiveAuditSpan in interface org.apache.hadoop.fs.store.audit.ActiveThreadSpanSource<org.apache.hadoop.fs.store.audit.AuditSpan>

Returns:

active audit span.

getRequestFactory

public RequestFactory getRequestFactory()

Get the request factory.

Returns:

the factory for requests.

isCSEEnabled

public boolean isCSEEnabled()

return if the store context have client side encryption enabled.

Returns:

boolean indicating if CSE is enabled or not.