org.apache.hadoop.fs.s3a

Class Invoker

java.lang.Object

org.apache.hadoop.fs.s3a.Invoker

@InterfaceAudience.Private
public class Invoker
extends Object

Class to provide lambda expression invocation of AWS operations. The core retry logic is in retryUntranslated(String, boolean, Retried, CallableRaisingIOE); the other retry() and retryUntranslated() calls are wrappers. The static once(String, String, CallableRaisingIOE) and once(String, String, InvocationRaisingIOE) calls take an operation and return it with AWS exceptions translated to IOEs of some form. The retry logic on a failure is defined by the retry policy passed in the constructor; the standard retry policy is S3ARetryPolicy, though others may be used. The constructor also takes two Invoker.Retried callbacks. The caughtCallback is called whenever an exception (IOE or AWS) is caught, before the retry processing looks at it. The retryCallback is invoked after a retry is scheduled but before the sleep. These callbacks can be used for reporting and incrementing statistics. The static quietly(String, String, InvocationRaisingIOE) and quietlyEval(String, String, CallableRaisingIOE) calls exist to take any operation and quietly catch and log at debug. The return value of quietlyEval(String, String, CallableRaisingIOE) is a java 8 optional, which can then be used in java8-expressions.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	Invoker.Retried Callback for retry and notification operations.

Field Summary

Fields

Modifier and Type	Field and Description
static Invoker.Retried	LOG_EVENT Log retries at debug.
static Invoker.Retried	NO_OP No op for a retrying callback.

Constructor Summary

Constructors

Constructor and Description
Invoker(org.apache.hadoop.io.retry.RetryPolicy retryPolicy, Invoker.Retried retryCallback) Instantiate.

Method Summary

Modifier and Type	Method and Description
Invoker.Retried	getRetryCallback()
org.apache.hadoop.io.retry.RetryPolicy	getRetryPolicy()
static <T> void	ignoreIOExceptions(org.slf4j.Logger log, String action, String path, org.apache.hadoop.util.functional.CallableRaisingIOE<T> operation) Execute an operation and ignore all raised IOExceptions; log at INFO; full stack only at DEBUG.
static void	ignoreIOExceptions(org.slf4j.Logger log, String action, String path, org.apache.hadoop.util.functional.InvocationRaisingIOE operation) Execute an operation and ignore all raised IOExceptions; log at INFO; full stack only at DEBUG.
void	maybeRetry(boolean doRetry, String action, String path, boolean idempotent, org.apache.hadoop.util.functional.InvocationRaisingIOE operation) Execute a void operation with the default retry callback invoked when doRetry=true, else just once.
<T> T	maybeRetry(boolean doRetry, String action, String path, boolean idempotent, Invoker.Retried retrying, org.apache.hadoop.util.functional.CallableRaisingIOE<T> operation) Execute a function with retry processing when doRetry=true, else just once.
void	maybeRetry(boolean doRetry, String action, String path, boolean idempotent, Invoker.Retried retrying, org.apache.hadoop.util.functional.InvocationRaisingIOE operation) Execute a void operation with retry processing when doRetry=true, else just once.
static <T> T	once(String action, String path, org.apache.hadoop.util.functional.CallableRaisingIOE<T> operation) Execute a function, translating any exception into an IOException.
static void	once(String action, String path, org.apache.hadoop.util.functional.InvocationRaisingIOE operation) Execute an operation with no result.
static <T> T	onceInTheFuture(String action, String path, Future<T> future) Wait for a future, translating SdkException into an IOException.
static <T> T	onceTrackingDuration(String action, String path, org.apache.hadoop.fs.statistics.DurationTracker tracker, org.apache.hadoop.util.functional.CallableRaisingIOE<T> operation) Execute a function, translating any exception into an IOException.
static void	quietly(String action, String path, org.apache.hadoop.util.functional.InvocationRaisingIOE operation) Execute an operation; any exception raised is simply caught and logged at debug.
static <T> Optional<T>	quietlyEval(String action, String path, org.apache.hadoop.util.functional.CallableRaisingIOE<T> operation) Execute an operation; any exception raised is caught and logged at debug.
<T> T	retry(String action, String path, boolean idempotent, org.apache.hadoop.util.functional.CallableRaisingIOE<T> operation) Execute a function with the default retry callback invoked.
void	retry(String action, String path, boolean idempotent, org.apache.hadoop.util.functional.InvocationRaisingIOE operation) Execute a void operation with the default retry callback invoked.
<T> T	retry(String action, String path, boolean idempotent, Invoker.Retried retrying, org.apache.hadoop.util.functional.CallableRaisingIOE<T> operation) Execute a function with retry processing.
void	retry(String action, String path, boolean idempotent, Invoker.Retried retrying, org.apache.hadoop.util.functional.InvocationRaisingIOE operation) Execute a void operation with retry processing.
<T> T	retryUntranslated(String text, boolean idempotent, org.apache.hadoop.util.functional.CallableRaisingIOE<T> operation) Execute a function with retry processing and no translation.
<T> T	retryUntranslated(String text, boolean idempotent, Invoker.Retried retrying, org.apache.hadoop.util.functional.CallableRaisingIOE<T> operation) Execute a function with retry processing: AWS SDK Exceptions are not translated.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

NO_OP

public static final Invoker.Retried NO_OP

No op for a retrying callback.

LOG_EVENT

public static final Invoker.Retried LOG_EVENT

Log retries at debug.

Constructor Detail

Invoker

public Invoker(org.apache.hadoop.io.retry.RetryPolicy retryPolicy,
               Invoker.Retried retryCallback)

Instantiate.

Parameters:

retryPolicy - retry policy for all operations.

retryCallback - standard retry policy

Method Detail

getRetryPolicy

public org.apache.hadoop.io.retry.RetryPolicy getRetryPolicy()

getRetryCallback

public Invoker.Retried getRetryCallback()

once

@Retries.OnceTranslated
public static <T> T once(String action,
                                                 String path,
                                                 org.apache.hadoop.util.functional.CallableRaisingIOE<T> operation)
                                          throws IOException

Execute a function, translating any exception into an IOException.

Type Parameters:

T - type of return value

Parameters:

action - action to execute (used in error messages)

path - path of work (used in error messages)

operation - operation to execute

Returns:

the result of the function call

Throws:

IOException - any IOE raised, or translated exception

onceTrackingDuration

@Retries.OnceTranslated
public static <T> T onceTrackingDuration(String action,
                                                                 String path,
                                                                 org.apache.hadoop.fs.statistics.DurationTracker tracker,
                                                                 org.apache.hadoop.util.functional.CallableRaisingIOE<T> operation)
                                                          throws IOException

Execute a function, translating any exception into an IOException. The supplied duration tracker instance is updated with success/failure.

Type Parameters:

T - type of return value

Parameters:

action - action to execute (used in error messages)

path - path of work (used in error messages)

tracker - tracker to update

operation - operation to execute

Returns:

the result of the function call

Throws:

IOException - any IOE raised, or translated exception

once

@Retries.OnceTranslated
public static void once(String action,
                                                String path,
                                                org.apache.hadoop.util.functional.InvocationRaisingIOE operation)
                                         throws IOException

Execute an operation with no result.

Parameters:

action - action to execute (used in error messages)

path - path of work (used in error messages)

operation - operation to execute

Throws:

IOException - any IOE raised, or translated exception

onceInTheFuture

@Retries.OnceTranslated
public static <T> T onceInTheFuture(String action,
                                                            String path,
                                                            Future<T> future)
                                                     throws IOException

Wait for a future, translating SdkException into an IOException.

Type Parameters:

T - type of return value

Parameters:

action - action to execute (used in error messages)

path - path of work (used in error messages)

future - future to await for

Returns:

the result of the function call

Throws:

IOException - any IOE raised, or translated exception

RuntimeException - any other runtime exception

ignoreIOExceptions

public static <T> void ignoreIOExceptions(org.slf4j.Logger log,
                                          String action,
                                          String path,
                                          org.apache.hadoop.util.functional.CallableRaisingIOE<T> operation)

Execute an operation and ignore all raised IOExceptions; log at INFO; full stack only at DEBUG.

Type Parameters:

T - type of operation

Parameters:

log - log to use.

action - action to include in log

path - optional path to include in log

operation - operation to execute

ignoreIOExceptions

public static void ignoreIOExceptions(org.slf4j.Logger log,
                                      String action,
                                      String path,
                                      org.apache.hadoop.util.functional.InvocationRaisingIOE operation)

Execute an operation and ignore all raised IOExceptions; log at INFO; full stack only at DEBUG.

Parameters:

log - log to use.

action - action to include in log

path - optional path to include in log

operation - operation to execute

retry

@Retries.RetryTranslated
public void retry(String action,
                                           String path,
                                           boolean idempotent,
                                           Invoker.Retried retrying,
                                           org.apache.hadoop.util.functional.InvocationRaisingIOE operation)
                                    throws IOException

Execute a void operation with retry processing.

Parameters:

action - action to execute (used in error messages)

path - path of work (used in error messages)

idempotent - does the operation have semantics which mean that it can be retried even if was already executed?

retrying - callback on retries

operation - operation to execute

Throws:

IOException - any IOE raised, or translated exception

maybeRetry

@Retries.RetryTranslated
public void maybeRetry(boolean doRetry,
                                                String action,
                                                String path,
                                                boolean idempotent,
                                                Invoker.Retried retrying,
                                                org.apache.hadoop.util.functional.InvocationRaisingIOE operation)
                                         throws IOException

Execute a void operation with retry processing when doRetry=true, else just once.

Parameters:

doRetry - true if retries should be performed

action - action to execute (used in error messages)

path - path of work (used in error messages)

idempotent - does the operation have semantics which mean that it can be retried even if was already executed?

retrying - callback on retries

operation - operation to execute

Throws:

IOException - any IOE raised, or translated exception

retry

@Retries.RetryTranslated
public void retry(String action,
                                           String path,
                                           boolean idempotent,
                                           org.apache.hadoop.util.functional.InvocationRaisingIOE operation)
                                    throws IOException

Execute a void operation with the default retry callback invoked.

Parameters:

action - action to execute (used in error messages)

path - path of work (used in error messages)

idempotent - does the operation have semantics which mean that it can be retried even if was already executed?

operation - operation to execute

Throws:

IOException - any IOE raised, or translated exception

maybeRetry

@Retries.RetryTranslated
public void maybeRetry(boolean doRetry,
                                                String action,
                                                String path,
                                                boolean idempotent,
                                                org.apache.hadoop.util.functional.InvocationRaisingIOE operation)
                                         throws IOException

Execute a void operation with the default retry callback invoked when doRetry=true, else just once.

Parameters:

doRetry - true if retries should be performed

action - action to execute (used in error messages)

path - path of work (used in error messages)

idempotent - does the operation have semantics which mean that it can be retried even if was already executed?

operation - operation to execute

Throws:

IOException - any IOE raised, or translated exception

retry

@Retries.RetryTranslated
public <T> T retry(String action,
                                            @Nullable
                                            String path,
                                            boolean idempotent,
                                            org.apache.hadoop.util.functional.CallableRaisingIOE<T> operation)
                                     throws IOException

Execute a function with the default retry callback invoked.

Type Parameters:

T - type of return value

Parameters:

action - action to execute (used in error messages)

path - path of work (used in error messages)

idempotent - does the operation have semantics which mean that it can be retried even if was already executed?

operation - operation to execute

Returns:

the result of the call

Throws:

IOException - any IOE raised, or translated exception

retry

@Retries.RetryTranslated
public <T> T retry(String action,
                                            @Nullable
                                            String path,
                                            boolean idempotent,
                                            Invoker.Retried retrying,
                                            org.apache.hadoop.util.functional.CallableRaisingIOE<T> operation)
                                     throws IOException

Execute a function with retry processing. Uses once(String, String, CallableRaisingIOE) as the inner invocation mechanism before retry logic is performed.

Type Parameters:

T - type of return value

Parameters:

action - action to execute (used in error messages)

path - path of work (used in error messages)

idempotent - does the operation have semantics which mean that it can be retried even if was already executed?

retrying - callback on retries

operation - operation to execute

Returns:

the result of the call

Throws:

IOException - any IOE raised, or translated exception

maybeRetry

@Retries.RetryTranslated
public <T> T maybeRetry(boolean doRetry,
                                                 String action,
                                                 @Nullable
                                                 String path,
                                                 boolean idempotent,
                                                 Invoker.Retried retrying,
                                                 org.apache.hadoop.util.functional.CallableRaisingIOE<T> operation)
                                          throws IOException

Execute a function with retry processing when doRetry=true, else just once. Uses once(String, String, CallableRaisingIOE) as the inner invocation mechanism before retry logic is performed.

Type Parameters:

T - type of return value

Parameters:

doRetry - true if retries should be performed

action - action to execute (used in error messages)

path - path of work (used in error messages)

idempotent - does the operation have semantics which mean that it can be retried even if was already executed?

retrying - callback on retries

operation - operation to execute

Returns:

the result of the call

Throws:

IOException - any IOE raised, or translated exception

retryUntranslated

@Retries.RetryRaw
public <T> T retryUntranslated(String text,
                                                 boolean idempotent,
                                                 org.apache.hadoop.util.functional.CallableRaisingIOE<T> operation)
                                          throws IOException

Execute a function with retry processing and no translation. and the default retry callback.

Type Parameters:

T - type of return value

Parameters:

text - description for the catching callback

idempotent - does the operation have semantics which mean that it can be retried even if was already executed?

operation - operation to execute

Returns:

the result of the call

Throws:

IOException - any IOE raised

RuntimeException - any Runtime exception raised

retryUntranslated

@Retries.RetryRaw
public <T> T retryUntranslated(String text,
                                                 boolean idempotent,
                                                 Invoker.Retried retrying,
                                                 org.apache.hadoop.util.functional.CallableRaisingIOE<T> operation)
                                          throws IOException

Execute a function with retry processing: AWS SDK Exceptions are not translated. This is method which the others eventually invoke.

Type Parameters:

T - type of return value

Parameters:

text - text to include in messages

idempotent - does the operation have semantics which mean that it can be retried even if was already executed?

retrying - callback on retries

operation - operation to execute

Returns:

the result of the call

Throws:

IOException - any IOE raised

software.amazon.awssdk.core.exception.SdkException - any AWS exception raised

RuntimeException - : these are never caught and retries.

quietly

public static void quietly(String action,
                           String path,
                           org.apache.hadoop.util.functional.InvocationRaisingIOE operation)

Execute an operation; any exception raised is simply caught and logged at debug.

Parameters:

action - action to execute

path - path (for exception construction)

operation - operation

quietlyEval

public static <T> Optional<T> quietlyEval(String action,
                                          String path,
                                          org.apache.hadoop.util.functional.CallableRaisingIOE<T> operation)

Execute an operation; any exception raised is caught and logged at debug. The result is only non-empty if the operation succeeded

Type Parameters:

T - type to return

Parameters:

action - action to execute

path - path (for exception construction)

operation - operation

Returns:

the result of a successful operation