org.apache.hadoop.fs.s3a.impl

Class AWSClientConfig

java.lang.Object

org.apache.hadoop.fs.s3a.impl.AWSClientConfig

public final class AWSClientConfig
extends Object

Methods for configuring the S3 client. These methods are used when creating and configuring the HTTP clients which communicate with the S3 service.

See software.amazon.awssdk.http.SdkHttpConfigurationOption for the default values.

Method Summary

Modifier and Type	Method and Description
static software.amazon.awssdk.http.nio.netty.NettyNioAsyncHttpClient.Builder	createAsyncHttpClientBuilder(org.apache.hadoop.conf.Configuration conf) Create and configure the async http client with timeouts for: connection acquisition, max idle, timeout, TTL, socket and keepalive.
static software.amazon.awssdk.http.nio.netty.ProxyConfiguration	createAsyncProxyConfiguration(org.apache.hadoop.conf.Configuration conf, String bucket) Configures the proxy for the async http client.
static software.amazon.awssdk.core.client.config.ClientOverrideConfiguration.Builder	createClientConfigBuilder(org.apache.hadoop.conf.Configuration conf, String awsServiceIdentifier) Create the config for a given service...the service identifier is used to determine signature implementation.
static software.amazon.awssdk.http.apache.ApacheHttpClient.Builder	createHttpClientBuilder(org.apache.hadoop.conf.Configuration conf) Create and configure the http client-based connector with timeouts for: connection acquisition, max idle, timeout, TTL, socket and keepalive.
static software.amazon.awssdk.http.apache.ProxyConfiguration	createProxyConfiguration(org.apache.hadoop.conf.Configuration conf, String bucket) Configures the proxy.
static software.amazon.awssdk.core.retry.RetryPolicy.Builder	createRetryPolicyBuilder(org.apache.hadoop.conf.Configuration conf) Configures the retry policy.
static Duration	getMinimumOperationDuration() Get the current minimum operation duration.
static void	resetMinimumOperationDuration() Reset the minimum operation duration to the default.
static void	setMinimumOperationDuration(Duration duration) Set the minimum operation duration.
static void	setRequestTimeout(software.amazon.awssdk.awscore.AwsRequest.Builder builder, Duration timeout) Set a custom ApiCallTimeout for a single request.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

createClientConfigBuilder

public static software.amazon.awssdk.core.client.config.ClientOverrideConfiguration.Builder createClientConfigBuilder(org.apache.hadoop.conf.Configuration conf,
                                                                                                                      String awsServiceIdentifier)
                                                                                                               throws IOException

Create the config for a given service...the service identifier is used to determine signature implementation.

Parameters:

conf - configuration

awsServiceIdentifier - service

Returns:

the builder inited with signer, timeouts and UA.

Throws:

IOException - failure.

RuntimeException - some failures creating an http signer

createHttpClientBuilder

public static software.amazon.awssdk.http.apache.ApacheHttpClient.Builder createHttpClientBuilder(org.apache.hadoop.conf.Configuration conf)
                                                                                           throws IOException

Create and configure the http client-based connector with timeouts for: connection acquisition, max idle, timeout, TTL, socket and keepalive. SSL channel mode is set up via NetworkBinding.bindSSLChannelMode(Configuration, ApacheHttpClient.Builder).

Parameters:

conf - The Hadoop configuration

Returns:

Http client builder

Throws:

IOException - on any problem

createAsyncHttpClientBuilder

public static software.amazon.awssdk.http.nio.netty.NettyNioAsyncHttpClient.Builder createAsyncHttpClientBuilder(org.apache.hadoop.conf.Configuration conf)

Create and configure the async http client with timeouts for: connection acquisition, max idle, timeout, TTL, socket and keepalive. This is netty based and does not allow for the SSL channel mode to be set.

Parameters:

conf - The Hadoop configuration

Returns:

Async Http client builder

createRetryPolicyBuilder

public static software.amazon.awssdk.core.retry.RetryPolicy.Builder createRetryPolicyBuilder(org.apache.hadoop.conf.Configuration conf)

Configures the retry policy. Retry policy is RetryMode.ADAPTIVE, which "dynamically limits the rate of AWS requests to maximize success rate", possibly at the expense of latency. Based on the ABFS experience, it is better to limit the rate requests are made rather than have to resort to exponential backoff after failures come in -especially as that backoff is per http connection.

Parameters:

conf - The Hadoop configuration

Returns:

Retry policy builder

createProxyConfiguration

public static software.amazon.awssdk.http.apache.ProxyConfiguration createProxyConfiguration(org.apache.hadoop.conf.Configuration conf,
                                                                                             String bucket)
                                                                                      throws IOException

Configures the proxy.

Parameters:

conf - The Hadoop configuration

bucket - Optional bucket to use to look up per-bucket proxy secrets

Returns:

Proxy configuration

Throws:

IOException - on any IO problem

createAsyncProxyConfiguration

public static software.amazon.awssdk.http.nio.netty.ProxyConfiguration createAsyncProxyConfiguration(org.apache.hadoop.conf.Configuration conf,
                                                                                                     String bucket)
                                                                                              throws IOException

Configures the proxy for the async http client.

Although this is netty specific, it is part of the AWS SDK public API and not any shaded netty internal class.

Parameters:

conf - The Hadoop configuration

bucket - Optional bucket to use to look up per-bucket proxy secrets

Returns:

Proxy configuration

Throws:

IOException - on any IO problem

resetMinimumOperationDuration

@VisibleForTesting
public static void resetMinimumOperationDuration()

Reset the minimum operation duration to the default. For test use only; Logs at INFO.

This MUST be called in test teardown in any test suite which called setMinimumOperationDuration(Duration).

setMinimumOperationDuration

@VisibleForTesting
public static void setMinimumOperationDuration(Duration duration)

Set the minimum operation duration. This is for testing and will log at info; does require a non-negative duration.

Test suites must call resetMinimumOperationDuration() in their teardown to avoid breaking subsequent tests in the same process.

Parameters:

duration - non-negative duration

Throws:

IllegalArgumentException - if the duration is negative.

getMinimumOperationDuration

public static Duration getMinimumOperationDuration()

Get the current minimum operation duration.

Returns:

current duration.

setRequestTimeout

public static void setRequestTimeout(software.amazon.awssdk.awscore.AwsRequest.Builder builder,
                                     Duration timeout)

Set a custom ApiCallTimeout for a single request. This allows for a longer timeout to be used in data upload requests than that for all other S3 interactions; This does not happen by default in the V2 SDK (see HADOOP-19295).

If the timeout is zero, the request is not patched.

Parameters:

builder - builder to patch.

timeout - timeout