org.apache.hadoop.fs.s3a

Class S3AInstrumentation

java.lang.Object

org.apache.hadoop.fs.s3a.S3AInstrumentation

All Implemented Interfaces:

Closeable, AutoCloseable, CountersAndGauges, org.apache.hadoop.fs.statistics.DurationTrackerFactory, org.apache.hadoop.fs.statistics.IOStatisticsSource, org.apache.hadoop.metrics2.MetricsSource

@InterfaceAudience.Private
 @InterfaceStability.Evolving
public class S3AInstrumentation
extends Object
implements Closeable, org.apache.hadoop.metrics2.MetricsSource, CountersAndGauges, org.apache.hadoop.fs.statistics.IOStatisticsSource

Instrumentation of S3A.

History

1. HADOOP-13028. Initial implementation. Derived from the AzureFileSystemInstrumentation.
2. Broadly (and directly) used in S3A. The use of direct references causes "problems" in mocking tests.
3. HADOOP-16830. IOStatistics. Move to an interface and implementation design for the different inner classes.

Counters and metrics are generally addressed in code by their name or Statistic key. There may be some Statistics which do not have an entry here. To avoid attempts to access such counters failing, the operations to increment/query metric values are designed to handle lookup failures.

S3AFileSystem StorageStatistics are dynamically derived from the IOStatistics.

The toString() operation includes the entire IOStatistics when this class's log is set to DEBUG. This keeps the logs somewhat manageable on normal runs, but allows for more reporting.

Field Summary

Fields

Modifier and Type	Field and Description
static String	CONTEXT "s3aFileSystem" Currently all s3a metrics are placed in a single "context".
static String	METRIC_TAG_BUCKET "bucket" The name of a field added to metrics records that indicates the hostname portion of the FS URL.
static String	METRIC_TAG_FILESYSTEM_ID "s3aFileSystemId" The name of a field added to metrics records that uniquely identifies a specific FileSystem instance.
static String	METRICS_SYSTEM_NAME "s3a-file-system" The name of the s3a-specific metrics system instance used for s3a metrics.

Constructor Summary

Constructors

Constructor and Description
S3AInstrumentation(URI name) Construct the instrumentation for a filesystem.

Method Summary

Modifier and Type	Method and Description
void	addValueToQuantiles(Statistic op, long value) Add a value to a quantiles statistic.
void	close()
protected org.apache.hadoop.metrics2.lib.MutableCounterLong	counter(Statistic op) Create a counter in the registry.
protected org.apache.hadoop.metrics2.lib.MutableCounterLong	counter(String name, String desc) Create a counter in the registry.
org.apache.hadoop.fs.statistics.impl.IOStatisticsStore	createMetricsUpdatingStore() Create an IOStatistics store which updates FS metrics as well as IOStatistics.
void	decrementGauge(Statistic op, long count) Decrement a specific gauge.
void	directoryCreated() Indicate that S3A created a directory.
void	directoryDeleted() Indicate that S3A just deleted a directory.
String	dump(String prefix, String separator, String suffix, boolean all) Dump all the metrics to a string.
protected void	duration(Statistic op) Registering a duration adds the success and failure counters.
void	errorIgnored() Note that an error was ignored.
void	fakeDirsDeleted(int count) Indicate that fake directory request was made.
void	fileCreated() Indicate that S3A created a file.
void	fileDeleted(int count) Indicate that S3A deleted one or more files.
void	filesCopied(int files, long size) Indicate that S3A copied some files within the store.
protected org.apache.hadoop.metrics2.lib.MutableGaugeLong	gauge(Statistic op) Create a gauge in the registry.
protected org.apache.hadoop.metrics2.lib.MutableGaugeLong	gauge(String name, String desc) Create a gauge in the registry.
long	getCounterValue(Statistic statistic) Get the value of a counter.
long	getCounterValue(String name) Get the value of a counter.
org.apache.hadoop.fs.statistics.DurationTrackerFactory	getDurationTrackerFactory() Get the duration tracker factory.
org.apache.hadoop.fs.statistics.impl.IOStatisticsStore	getIOStatistics() Get the instance IO Statistics.
void	getMetrics(org.apache.hadoop.metrics2.MetricsCollector collector, boolean all)
String	getMetricSourceName() if registered with the metrics, return the name of the source.
org.apache.hadoop.metrics2.lib.MetricsRegistry	getRegistry() Get the metrics registry.
void	incrementCounter(Statistic op, AtomicLong count) Increments a mutable counter and the matching instance IOStatistics counter with the value of the atomic long.
void	incrementCounter(Statistic op, long count) Increments a mutable counter and the matching instance IOStatistics counter.
void	incrementGauge(Statistic op, long count) Increment a specific gauge.
org.apache.hadoop.metrics2.lib.MutableGaugeLong	lookupGauge(String name) Look up a gauge.
org.apache.hadoop.metrics2.lib.MutableMetric	lookupMetric(String name) Look up a metric from both the registered set and the lighter weight stream entries.
org.apache.hadoop.metrics2.lib.MutableQuantiles	lookupQuantiles(String name) Look up a quantiles.
CommitterStatistics	newCommitterStatistics() Create a new instance of the committer statistics.
DelegationTokenStatistics	newDelegationTokenStatistics() Create a delegation token statistics instance.
S3AInputStreamStatistics	newInputStreamStatistics(org.apache.hadoop.fs.FileSystem.Statistics filesystemStatistics) Create a stream input statistics instance.
BlockOutputStreamStatistics	newOutputStreamStatistics(org.apache.hadoop.fs.FileSystem.Statistics filesystemStatistics) Create a stream output statistics instance.
protected org.apache.hadoop.metrics2.lib.MutableQuantiles	quantiles(Statistic op, String sampleName, String valueName, int interval) Create a quantiles in the registry.
void	recordDuration(Statistic op, boolean success, Duration duration) Add the duration as a timed statistic, deriving statistic name from the operation symbol and the outcome.
Map<String,Long>	toMap() Copy all the metrics to a map of (name, long-value).
String	toString() String representation.
org.apache.hadoop.fs.statistics.DurationTracker	trackDuration(String key, long count) The duration tracker updates the metrics with the count and IOStatistics will full duration information.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.hadoop.fs.statistics.DurationTrackerFactory

trackDuration

Field Detail

METRICS_SYSTEM_NAME

public static final String METRICS_SYSTEM_NAME

"s3a-file-system" The name of the s3a-specific metrics system instance used for s3a metrics.

See Also:

Constant Field Values

CONTEXT

public static final String CONTEXT

"s3aFileSystem" Currently all s3a metrics are placed in a single "context". Distinct contexts may be used in the future.

See Also:

Constant Field Values

METRIC_TAG_FILESYSTEM_ID

public static final String METRIC_TAG_FILESYSTEM_ID

"s3aFileSystemId" The name of a field added to metrics records that uniquely identifies a specific FileSystem instance.

See Also:

Constant Field Values

METRIC_TAG_BUCKET

public static final String METRIC_TAG_BUCKET

"bucket" The name of a field added to metrics records that indicates the hostname portion of the FS URL.

See Also:

Constant Field Values

Constructor Detail

S3AInstrumentation

public S3AInstrumentation(URI name)

Construct the instrumentation for a filesystem.

Parameters:

name - URI of filesystem.

Method Detail

counter

protected final org.apache.hadoop.metrics2.lib.MutableCounterLong counter(String name,
                                                                          String desc)

Create a counter in the registry.

Parameters:

name - counter name

desc - counter description

Returns:

a new counter

counter

protected final org.apache.hadoop.metrics2.lib.MutableCounterLong counter(Statistic op)

Create a counter in the registry.

Parameters:

op - statistic to count

Returns:

a new counter

duration

protected final void duration(Statistic op)

Registering a duration adds the success and failure counters.

Parameters:

op - statistic to track

gauge

protected final org.apache.hadoop.metrics2.lib.MutableGaugeLong gauge(String name,
                                                                      String desc)

Create a gauge in the registry.

Parameters:

name - name gauge name

desc - description

Returns:

the gauge

gauge

protected final org.apache.hadoop.metrics2.lib.MutableGaugeLong gauge(Statistic op)

Create a gauge in the registry.

Parameters:

op - statistic to count

Returns:

the gauge

quantiles

protected final org.apache.hadoop.metrics2.lib.MutableQuantiles quantiles(Statistic op,
                                                                          String sampleName,
                                                                          String valueName,
                                                                          int interval)

Create a quantiles in the registry.

Parameters:

op - statistic to collect

sampleName - sample name of the quantiles

valueName - value name of the quantiles

interval - interval of the quantiles in seconds

Returns:

the created quantiles metric

getRegistry

public org.apache.hadoop.metrics2.lib.MetricsRegistry getRegistry()

Get the metrics registry.

Returns:

the registry

dump

public String dump(String prefix,
                   String separator,
                   String suffix,
                   boolean all)

Dump all the metrics to a string.

Parameters:

prefix - prefix before every entry

separator - separator between name and value

suffix - suffix

all - get all the metrics even if the values are not changed.

Returns:

a string dump of the metrics

getCounterValue

public long getCounterValue(Statistic statistic)

Get the value of a counter.

Parameters:

statistic - the operation

Returns:

its value, or 0 if not found.

getCounterValue

public long getCounterValue(String name)

Get the value of a counter. If the counter is null, return 0.

Parameters:

name - the name of the counter

Returns:

its value.

lookupGauge

public org.apache.hadoop.metrics2.lib.MutableGaugeLong lookupGauge(String name)

Look up a gauge.

Parameters:

name - gauge name

Returns:

the gauge or null

Throws:

ClassCastException - if the metric is not a Gauge.

lookupQuantiles

public org.apache.hadoop.metrics2.lib.MutableQuantiles lookupQuantiles(String name)

Look up a quantiles.

Parameters:

name - quantiles name

Returns:

the quantiles or null

Throws:

ClassCastException - if the metric is not a Quantiles.

lookupMetric

public org.apache.hadoop.metrics2.lib.MutableMetric lookupMetric(String name)

Look up a metric from both the registered set and the lighter weight stream entries.

Parameters:

name - metric name

Returns:

the metric or null

getIOStatistics

public org.apache.hadoop.fs.statistics.impl.IOStatisticsStore getIOStatistics()

Get the instance IO Statistics.

Specified by:

getIOStatistics in interface org.apache.hadoop.fs.statistics.IOStatisticsSource

Returns:

statistics.

getDurationTrackerFactory

public org.apache.hadoop.fs.statistics.DurationTrackerFactory getDurationTrackerFactory()

Get the duration tracker factory.

Returns:

duration tracking for the instrumentation.

trackDuration

public org.apache.hadoop.fs.statistics.DurationTracker trackDuration(String key,
                                                                     long count)

The duration tracker updates the metrics with the count and IOStatistics will full duration information.

Specified by:

trackDuration in interface org.apache.hadoop.fs.statistics.DurationTrackerFactory

Parameters:

key - statistic key prefix

count - #of times to increment the matching counter in this operation.

Returns:

a duration tracker.

createMetricsUpdatingStore

public org.apache.hadoop.fs.statistics.impl.IOStatisticsStore createMetricsUpdatingStore()

Create an IOStatistics store which updates FS metrics as well as IOStatistics.

Returns:

instance of the store.

toString

public String toString()

String representation. Includes the IOStatistics when logging is at DEBUG.

Overrides:

toString in class Object

Returns:

a string form.

fileCreated

public void fileCreated()

Indicate that S3A created a file.

fileDeleted

public void fileDeleted(int count)

Indicate that S3A deleted one or more files.

Parameters:

count - number of files.

fakeDirsDeleted

public void fakeDirsDeleted(int count)

Indicate that fake directory request was made.

Parameters:

count - number of directory entries included in the delete request.

directoryCreated

public void directoryCreated()

Indicate that S3A created a directory.

directoryDeleted

public void directoryDeleted()

Indicate that S3A just deleted a directory.

filesCopied

public void filesCopied(int files,
                        long size)

Indicate that S3A copied some files within the store.

Parameters:

files - number of files

size - total size in bytes

errorIgnored

public void errorIgnored()

Note that an error was ignored.

incrementCounter

public void incrementCounter(Statistic op,
                             long count)

Increments a mutable counter and the matching instance IOStatistics counter. No-op if the counter is not defined, or the count == 0.

Specified by:

incrementCounter in interface CountersAndGauges

Parameters:

op - operation

count - increment value

addValueToQuantiles

public void addValueToQuantiles(Statistic op,
                                long value)

Add a value to a quantiles statistic. No-op if the quantile isn't found.

Specified by:

addValueToQuantiles in interface CountersAndGauges

Parameters:

op - operation to look up.

value - value to add.

Throws:

ClassCastException - if the metric is not a Quantiles.

incrementCounter

public void incrementCounter(Statistic op,
                             AtomicLong count)

Increments a mutable counter and the matching instance IOStatistics counter with the value of the atomic long. No-op if the counter is not defined, or the count == 0.

Parameters:

op - operation

count - atomic long containing value

incrementGauge

public void incrementGauge(Statistic op,
                           long count)

Increment a specific gauge. No-op if not defined.

Specified by:

incrementGauge in interface CountersAndGauges

Parameters:

op - operation

count - increment value

Throws:

ClassCastException - if the metric is of the wrong type

decrementGauge

public void decrementGauge(Statistic op,
                           long count)

Decrement a specific gauge. No-op if not defined.

Specified by:

decrementGauge in interface CountersAndGauges

Parameters:

op - operation

count - increment value

Throws:

ClassCastException - if the metric is of the wrong type

recordDuration

public void recordDuration(Statistic op,
                           boolean success,
                           Duration duration)

Add the duration as a timed statistic, deriving statistic name from the operation symbol and the outcome.

Specified by:

recordDuration in interface CountersAndGauges

Parameters:

op - operation

success - was the operation a success?

duration - how long did it take

newInputStreamStatistics

public S3AInputStreamStatistics newInputStreamStatistics(@Nullable
                                                         org.apache.hadoop.fs.FileSystem.Statistics filesystemStatistics)

Create a stream input statistics instance.

Parameters:

filesystemStatistics - FS Statistics to update in close().

Returns:

the new instance

newCommitterStatistics

public CommitterStatistics newCommitterStatistics()

Create a new instance of the committer statistics.

Returns:

a new committer statistics instance

getMetrics

public void getMetrics(org.apache.hadoop.metrics2.MetricsCollector collector,
                       boolean all)

Specified by:

getMetrics in interface org.apache.hadoop.metrics2.MetricsSource

getMetricSourceName

public String getMetricSourceName()

if registered with the metrics, return the name of the source.

Returns:

the name of the metrics, or null if this instance is not bonded.

close

public void close()

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

newOutputStreamStatistics

public BlockOutputStreamStatistics newOutputStreamStatistics(org.apache.hadoop.fs.FileSystem.Statistics filesystemStatistics)

Create a stream output statistics instance.

Parameters:

filesystemStatistics - thread-local FS statistics.

Returns:

the new instance

newDelegationTokenStatistics

public DelegationTokenStatistics newDelegationTokenStatistics()

Create a delegation token statistics instance.

Returns:

an instance of delegation token statistics

toMap

public Map<String,Long> toMap()

Copy all the metrics to a map of (name, long-value).

Returns:

a map of the metrics