org.apache.hadoop.fs.s3a

Class S3AInstrumentation

java.lang.Object

org.apache.hadoop.fs.s3a.S3AInstrumentation

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.metrics2.MetricsSource

@InterfaceAudience.Private
 @InterfaceStability.Evolving
public class S3AInstrumentation
extends Object
implements Closeable, org.apache.hadoop.metrics2.MetricsSource

Instrumentation of S3a. Derived from the AzureFileSystemInstrumentation. Counters and metrics are generally addressed in code by their name or Statistic key. There may be some Statistics which do not have an entry here. To avoid attempts to access such counters failing, the operations to increment/query metric values are designed to handle lookup failures.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
class	S3AInstrumentation.CommitterStatistics Instrumentation exported to S3Guard Committers.
class	S3AInstrumentation.InputStreamStatistics Statistics updated by an input stream during its actual operation.
class	S3AInstrumentation.OutputStreamStatistics Statistics updated by an output stream during its actual operation.
class	S3AInstrumentation.S3GuardInstrumentation Instrumentation exported to S3Guard.

Field Summary

Fields

Modifier and Type	Field and Description
static String	CONTEXT "s3aFileSystem" Currently all s3a metrics are placed in a single "context".
static String	METRIC_TAG_BUCKET "bucket" The name of a field added to metrics records that indicates the hostname portion of the FS URL.
static String	METRIC_TAG_FILESYSTEM_ID "s3aFileSystemId" The name of a field added to metrics records that uniquely identifies a specific FileSystem instance.
static String	METRICS_SYSTEM_NAME "s3a-file-system" The name of the s3a-specific metrics system instance used for s3a metrics.

Constructor Summary

Constructors

Constructor and Description
S3AInstrumentation(URI name)

Method Summary

Modifier and Type	Method and Description
void	addValueToQuantiles(Statistic op, long value) Add a value to a quantiles statistic.
void	close()
protected org.apache.hadoop.metrics2.lib.MutableCounterLong	counter(Statistic op) Create a counter in the registry.
protected org.apache.hadoop.metrics2.lib.MutableCounterLong	counter(String name, String desc) Create a counter in the registry.
void	decrementGauge(Statistic op, long count) Decrement a specific gauge.
void	directoryCreated() Indicate that S3A created a directory.
void	directoryDeleted() Indicate that S3A just deleted a directory.
String	dump(String prefix, String separator, String suffix, boolean all) Dump all the metrics to a string.
void	errorIgnored() Note that an error was ignored.
void	fakeDirsDeleted(int count) Indicate that fake directory request was made.
void	fileCreated() Indicate that S3A created a file.
void	fileDeleted(int count) Indicate that S3A deleted one or more files.
void	filesCopied(int files, long size) Indicate that S3A copied some files within the store.
protected org.apache.hadoop.metrics2.lib.MutableGaugeLong	gauge(String name, String desc) Create a gauge in the registry.
long	getCounterValue(Statistic statistic) Get the value of a counter.
long	getCounterValue(String name) Get the value of a counter.
void	getMetrics(org.apache.hadoop.metrics2.MetricsCollector collector, boolean all)
org.apache.hadoop.metrics2.MetricsSystem	getMetricsSystem()
org.apache.hadoop.metrics2.lib.MetricsRegistry	getRegistry() Get the metrics registry.
S3AInstrumentation.S3GuardInstrumentation	getS3GuardInstrumentation() Create a S3Guard instrumentation instance.
void	incrementCounter(Statistic op, AtomicLong count) Increment a specific counter.
void	incrementCounter(Statistic op, long count) Increment a specific counter.
void	incrementGauge(Statistic op, long count) Increment a specific gauge.
org.apache.hadoop.metrics2.lib.MutableGaugeLong	lookupGauge(String name) Look up a gauge.
org.apache.hadoop.metrics2.lib.MutableMetric	lookupMetric(String name) Look up a metric from both the registered set and the lighter weight stream entries.
org.apache.hadoop.metrics2.lib.MutableQuantiles	lookupQuantiles(String name) Look up a quantiles.
protected org.apache.hadoop.metrics2.lib.MutableQuantiles	quantiles(Statistic op, String sampleName, String valueName, int interval) Create a quantiles in the registry.
Map<String,Long>	toMap() Copy all the metrics to a map of (name, long-value).

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

METRICS_SYSTEM_NAME

public static final String METRICS_SYSTEM_NAME

"s3a-file-system" The name of the s3a-specific metrics system instance used for s3a metrics.

See Also:

Constant Field Values

CONTEXT

public static final String CONTEXT

"s3aFileSystem" Currently all s3a metrics are placed in a single "context". Distinct contexts may be used in the future.

See Also:

Constant Field Values

METRIC_TAG_FILESYSTEM_ID

public static final String METRIC_TAG_FILESYSTEM_ID

"s3aFileSystemId" The name of a field added to metrics records that uniquely identifies a specific FileSystem instance.

See Also:

Constant Field Values

METRIC_TAG_BUCKET

public static final String METRIC_TAG_BUCKET

"bucket" The name of a field added to metrics records that indicates the hostname portion of the FS URL.

See Also:

Constant Field Values

Constructor Detail

S3AInstrumentation

public S3AInstrumentation(URI name)

Method Detail

getMetricsSystem

public org.apache.hadoop.metrics2.MetricsSystem getMetricsSystem()

counter

protected final org.apache.hadoop.metrics2.lib.MutableCounterLong counter(String name,
                                                                          String desc)

Create a counter in the registry.

Parameters:

name - counter name

desc - counter description

Returns:

a new counter

counter

protected final org.apache.hadoop.metrics2.lib.MutableCounterLong counter(Statistic op)

Create a counter in the registry.

Parameters:

op - statistic to count

Returns:

a new counter

gauge

protected final org.apache.hadoop.metrics2.lib.MutableGaugeLong gauge(String name,
                                                                      String desc)

Create a gauge in the registry.

Parameters:

name - name gauge name

desc - description

Returns:

the gauge

quantiles

protected final org.apache.hadoop.metrics2.lib.MutableQuantiles quantiles(Statistic op,
                                                                          String sampleName,
                                                                          String valueName,
                                                                          int interval)

Create a quantiles in the registry.

Parameters:

op - statistic to collect

sampleName - sample name of the quantiles

valueName - value name of the quantiles

interval - interval of the quantiles in seconds

Returns:

the created quantiles metric

getRegistry

public org.apache.hadoop.metrics2.lib.MetricsRegistry getRegistry()

Get the metrics registry.

Returns:

the registry

dump

public String dump(String prefix,
                   String separator,
                   String suffix,
                   boolean all)

Dump all the metrics to a string.

Parameters:

prefix - prefix before every entry

separator - separator between name and value

suffix - suffix

all - get all the metrics even if the values are not changed.

Returns:

a string dump of the metrics

getCounterValue

public long getCounterValue(Statistic statistic)

Get the value of a counter.

Parameters:

statistic - the operation

Returns:

its value, or 0 if not found.

getCounterValue

public long getCounterValue(String name)

Get the value of a counter. If the counter is null, return 0.

Parameters:

name - the name of the counter

Returns:

its value.

lookupGauge

public org.apache.hadoop.metrics2.lib.MutableGaugeLong lookupGauge(String name)

Look up a gauge.

Parameters:

name - gauge name

Returns:

the gauge or null

Throws:

ClassCastException - if the metric is not a Gauge.

lookupQuantiles

public org.apache.hadoop.metrics2.lib.MutableQuantiles lookupQuantiles(String name)

Look up a quantiles.

Parameters:

name - quantiles name

Returns:

the quantiles or null

Throws:

ClassCastException - if the metric is not a Quantiles.

lookupMetric

public org.apache.hadoop.metrics2.lib.MutableMetric lookupMetric(String name)

Look up a metric from both the registered set and the lighter weight stream entries.

Parameters:

name - metric name

Returns:

the metric or null

fileCreated

public void fileCreated()

Indicate that S3A created a file.

fileDeleted

public void fileDeleted(int count)

Indicate that S3A deleted one or more files.

Parameters:

count - number of files.

fakeDirsDeleted

public void fakeDirsDeleted(int count)

Indicate that fake directory request was made.

Parameters:

count - number of directory entries included in the delete request.

directoryCreated

public void directoryCreated()

Indicate that S3A created a directory.

directoryDeleted

public void directoryDeleted()

Indicate that S3A just deleted a directory.

filesCopied

public void filesCopied(int files,
                        long size)

Indicate that S3A copied some files within the store.

Parameters:

files - number of files

size - total size in bytes

errorIgnored

public void errorIgnored()

Note that an error was ignored.

incrementCounter

public void incrementCounter(Statistic op,
                             long count)

Increment a specific counter. No-op if not defined.

Parameters:

op - operation

count - increment value

addValueToQuantiles

public void addValueToQuantiles(Statistic op,
                                long value)

Add a value to a quantiles statistic. No-op if the quantile isn't found.

Parameters:

op - operation to look up.

value - value to add.

Throws:

ClassCastException - if the metric is not a Quantiles.

incrementCounter

public void incrementCounter(Statistic op,
                             AtomicLong count)

Increment a specific counter. No-op if not defined.

Parameters:

op - operation

count - atomic long containing value

incrementGauge

public void incrementGauge(Statistic op,
                           long count)

Increment a specific gauge. No-op if not defined.

Parameters:

op - operation

count - increment value

Throws:

ClassCastException - if the metric is of the wrong type

decrementGauge

public void decrementGauge(Statistic op,
                           long count)

Decrement a specific gauge. No-op if not defined.

Parameters:

op - operation

count - increment value

Throws:

ClassCastException - if the metric is of the wrong type

getS3GuardInstrumentation

public S3AInstrumentation.S3GuardInstrumentation getS3GuardInstrumentation()

Create a S3Guard instrumentation instance. There's likely to be at most one instance of this per FS instance.

Returns:

the S3Guard instrumentation point.

getMetrics

public void getMetrics(org.apache.hadoop.metrics2.MetricsCollector collector,
                       boolean all)

Specified by:

getMetrics in interface org.apache.hadoop.metrics2.MetricsSource

close

public void close()

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

toMap

public Map<String,Long> toMap()

Copy all the metrics to a map of (name, long-value).

Returns:

a map of the metrics