org.apache.hadoop.fs.s3a.impl

Class StoreContext

java.lang.Object

org.apache.hadoop.fs.s3a.impl.StoreContext

@InterfaceAudience.Private
 @InterfaceStability.Unstable
public class StoreContext
extends Object

This class provides the core context of the S3A filesystem to subsidiary components, without exposing the entire parent class. This is eliminate explicit recursive coupling. Where methods on the FS are to be invoked, they are referenced via the ContextAccessors interface, so tests can implement their own. Warning: this really is private and unstable. Do not use outside the org.apache.hadoop.fs.s3a package.

Constructor Summary

Constructors

Constructor and Description

StoreContext(URI fsURI, String bucket, org.apache.hadoop.conf.Configuration configuration, String username, org.apache.hadoop.security.UserGroupInformation owner, com.google.common.util.concurrent.ListeningExecutorService executor, int executorCapacity, Invoker invoker, S3AInstrumentation instrumentation, S3AStorageStatistics storageStatistics, S3AInputPolicy inputPolicy, ChangeDetectionPolicy changeDetectionPolicy, boolean multiObjectDeleteEnabled, MetadataStore metadataStore, boolean useListV1, ContextAccessors contextAccessors, ITtlTimeProvider timeProvider) Instantiate.

Method Summary

Modifier and Type	Method and Description
protected Object	clone()
File	createTempFile(String prefix, long size) Create a temporary file somewhere.
com.google.common.util.concurrent.ListeningExecutorService	createThrottledExecutor() Create a new executor with the capacity defined in executorCapacity.
com.google.common.util.concurrent.ListeningExecutorService	createThrottledExecutor(int capacity) Create a new executor service with a given capacity.
void	decrementGauge(Statistic statistic, long count) Decrement a gauge by a specific value.
String	fullKey(S3AFileStatus stat) Build the full S3 key for a request from the status entry, possibly adding a "/" if it represents directory and it does not have a trailing slash already.
String	getBucket()
String	getBucketLocation() Get the location of the bucket.
ChangeDetectionPolicy	getChangeDetectionPolicy()
org.apache.hadoop.conf.Configuration	getConfiguration()
com.google.common.util.concurrent.ListeningExecutorService	getExecutor()
URI	getFsURI()
S3AInputPolicy	getInputPolicy()
S3AInstrumentation	getInstrumentation()
Invoker	getInvoker()
MetadataStore	getMetadataStore()
org.apache.hadoop.security.UserGroupInformation	getOwner() Get the owner of the filesystem.
S3AStorageStatistics	getStorageStatistics() Get the storage statistics of this filesystem.
ITtlTimeProvider	getTimeProvider() Get the time provider.
String	getUsername()
void	incrementGauge(Statistic statistic, long count) Increment a gauge by a specific value.
void	incrementStatistic(Statistic statistic) Increment a statistic by 1.
void	incrementStatistic(Statistic statistic, long count) Increment a statistic by a specific value.
boolean	isMultiObjectDeleteEnabled()
boolean	isUseListV1()
org.apache.hadoop.fs.Path	keyToPath(String key) Convert a key to a fully qualified path.
String	pathToKey(org.apache.hadoop.fs.Path path) Turns a path (relative or otherwise) into an S3 key.

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

StoreContext

public StoreContext(URI fsURI,
                    String bucket,
                    org.apache.hadoop.conf.Configuration configuration,
                    String username,
                    org.apache.hadoop.security.UserGroupInformation owner,
                    com.google.common.util.concurrent.ListeningExecutorService executor,
                    int executorCapacity,
                    Invoker invoker,
                    S3AInstrumentation instrumentation,
                    S3AStorageStatistics storageStatistics,
                    S3AInputPolicy inputPolicy,
                    ChangeDetectionPolicy changeDetectionPolicy,
                    boolean multiObjectDeleteEnabled,
                    MetadataStore metadataStore,
                    boolean useListV1,
                    ContextAccessors contextAccessors,
                    ITtlTimeProvider timeProvider)

Instantiate. No attempt to use a builder here as outside tests this should only be created in the S3AFileSystem.

Method Detail

clone

protected Object clone()
                throws CloneNotSupportedException

Overrides:

clone in class Object

Throws:

CloneNotSupportedException

getFsURI

public URI getFsURI()

getBucket

public String getBucket()

getConfiguration

public org.apache.hadoop.conf.Configuration getConfiguration()

getUsername

public String getUsername()

getExecutor

public com.google.common.util.concurrent.ListeningExecutorService getExecutor()

getInvoker

public Invoker getInvoker()

getInstrumentation

public S3AInstrumentation getInstrumentation()

getInputPolicy

public S3AInputPolicy getInputPolicy()

getChangeDetectionPolicy

public ChangeDetectionPolicy getChangeDetectionPolicy()

isMultiObjectDeleteEnabled

public boolean isMultiObjectDeleteEnabled()

getMetadataStore

public MetadataStore getMetadataStore()

isUseListV1

public boolean isUseListV1()

keyToPath

public org.apache.hadoop.fs.Path keyToPath(String key)

Convert a key to a fully qualified path.

Parameters:

key - input key

Returns:

the fully qualified path including URI scheme and bucket name.

pathToKey

public String pathToKey(org.apache.hadoop.fs.Path path)

Turns a path (relative or otherwise) into an S3 key.

Parameters:

path - input path, may be relative to the working dir

Returns:

a key excluding the leading "/", or, if it is the root path, ""

getStorageStatistics

public S3AStorageStatistics getStorageStatistics()

Get the storage statistics of this filesystem.

Returns:

the storage statistics

incrementStatistic

public void incrementStatistic(Statistic statistic)

Increment a statistic by 1. This increments both the instrumentation and storage statistics.

Parameters:

statistic - The operation to increment

incrementStatistic

public void incrementStatistic(Statistic statistic,
                               long count)

Increment a statistic by a specific value. This increments both the instrumentation and storage statistics.

Parameters:

statistic - The operation to increment

count - the count to increment

decrementGauge

public void decrementGauge(Statistic statistic,
                           long count)

Decrement a gauge by a specific value.

Parameters:

statistic - The operation to decrement

count - the count to decrement

incrementGauge

public void incrementGauge(Statistic statistic,
                           long count)

Increment a gauge by a specific value.

Parameters:

statistic - The operation to increment

count - the count to increment

createThrottledExecutor

public com.google.common.util.concurrent.ListeningExecutorService createThrottledExecutor(int capacity)

Create a new executor service with a given capacity. This executor submits works to the executor, using a SemaphoredDelegatingExecutor to limit the number of requests coming in from a specific client. Because this delegates to an existing thread pool, the cost of creating a new instance here is low. As the throttling is per instance, separate instances should be created for each operation which wishes to execute work in parallel without saturating the base executor. This is important if either the duration of each operation is long or the submission rate of work is high.

Parameters:

capacity - maximum capacity of this executor.

Returns:

an executor for submitting work.

createThrottledExecutor

public com.google.common.util.concurrent.ListeningExecutorService createThrottledExecutor()

Create a new executor with the capacity defined in executorCapacity.

Returns:

a new executor for exclusive use by the caller.

getOwner

public org.apache.hadoop.security.UserGroupInformation getOwner()

Get the owner of the filesystem.

Returns:

the user who created this filesystem.

createTempFile

public File createTempFile(String prefix,
                           long size)
                    throws IOException

Create a temporary file somewhere.

Parameters:

prefix - prefix for the temporary file

size - expected size.

Returns:

a file reference.

Throws:

IOException - failure.

getBucketLocation

public String getBucketLocation()
                         throws IOException

Get the location of the bucket.

Returns:

the bucket location.

Throws:

IOException - failure.

getTimeProvider

public ITtlTimeProvider getTimeProvider()

Get the time provider.

Returns:

the time source.

fullKey

public String fullKey(S3AFileStatus stat)

Build the full S3 key for a request from the status entry, possibly adding a "/" if it represents directory and it does not have a trailing slash already.

Parameters:

stat - status to build the key from

Returns:

a key for a delete request