Interface QueryMetrics<QueryType extends Query<?>>
-
- Type Parameters:
QueryType
-
- All Known Subinterfaces:
GroupByQueryMetrics
,SearchQueryMetrics
,TimeseriesQueryMetrics
,TopNQueryMetrics
- All Known Implementing Classes:
DefaultGroupByQueryMetrics
,DefaultQueryMetrics
,DefaultSearchQueryMetrics
,DefaultTimeseriesQueryMetrics
,DefaultTopNQueryMetrics
public interface QueryMetrics<QueryType extends Query<?>>
Abstraction wrappingServiceMetricEvent.Builder
and allowing to control what metrics are actually emitted, what dimensions do they have, etc. Goals of QueryMetrics --------------------- 1. Skipping or partial filtering of particular dimensions or metrics entirely. Implementation could leave the body of the corresponding method empty, or implement random filtering like: public void reportCpuTime(long timeNs) { if (ThreadLocalRandom.current().nextDouble() < 0.1) { super.reportCpuTime(timeNs); } } 2. Ability to add new dimensions and metrics, possibly expensive to compute, or expensive to process (long string values, high cardinality, etc.) and not to affect existing Druid installations, by skipping (see 1.) those dimensions and metrics entirely in the default QueryMetrics implementations. Users who need those expensive dimensions and metrics, could explicitly emit them in their own QueryMetrics. 3. Control over the time unit, in which time metrics are emitted. By default (seeDefaultQueryMetrics
and it's subclasses) it's milliseconds, but if queries are fast, it could be not precise enough. 4. Control over the dimension and metric names. Here, "control" is provided to the operator of a Druid cluster, who would exercise that control through a site-specific extension adding XxxQueryMetricsFactory impl(s). Types of methods in this interface ---------------------------------- 1. Methods, pulling some dimensions from the query object. These methods are used to populate the metric before the query is run. These methods accept a single `QueryType query` parameter.query(Query)
calls all methods of this type, hence pulling all available information from the query object as dimensions. 2. Methods for setting dimensions, which become known in the process of the query execution or after the query is completed. 3. Methods to register metrics to be emitted later in bulk viaemit(ServiceEmitter)
. These methods return this QueryMetrics object back for chaining. Names of these methods start with "report" prefix. Implementors expectations ------------------------- QueryMetrics is expected to be changed often, in every Druid release (including "patch" releases). Users who create their custom implementations of QueryMetrics should be ready to fix the code of their QueryMetrics (implement new methods) when they update Druid. Broken builds of custom extensions, containing custom QueryMetrics is the way to notify users that Druid core "wants" to emit new dimension or metric, and the user handles them manually: if the new dimension or metric is useful and not very expensive to process and store then emit, skip (see above Goals, 1.) otherwise.Despite this interface is annotated as
ExtensionPoint
and some of it's methods asPublicApi
, it may be changed in breaking ways even in minor releases.If implementors of custom QueryMetrics don't want to fix builds on every Druid release (e. g. if they want to add a single dimension to emitted events and don't want to alter other dimensions and emitted metrics), they could inherit their custom QueryMetrics from
DefaultQueryMetrics
or query-specific default implementation class, such asDefaultTopNQueryMetrics
. Those classes are guaranteed to stay around and implement new methods, added to the QueryMetrics interface (or a query-specific subinterface). However, there is no 100% guarantee of compatibility, because methods could not only be added to QueryMetrics, existing methods could also be changed or removed.QueryMetrics is designed for use from a single thread, implementations shouldn't care about thread-safety. Adding new methods to QueryMetrics ---------------------------------- 1. When adding a new method for setting a dimension, which could be pulled from the query object, always make them accept a single `QueryType query` parameter, letting the implementations to do all the work of carving the dimension value out of the query object. 2. When adding a new method for setting a dimension, which becomes known in the process of the query execution or after the query is completed, design it so that as little work as possible is done for preparing arguments for this method, and as much work as possible is done in the implementations of this method, if they decide to actually emit this dimension. 3. When adding a new method for registering metrics, make it to accept the metric value in the smallest reasonable unit (i. e. nanoseconds for time metrics, bytes for metrics of data size, etc.), allowing the implementations of this method to round the value up to more coarse-grained units, if they don't need the maximum precision. Making subinterfaces of QueryMetrics for emitting custom dimensions and/or metrics for specific query types ----------------------------------------------------------------------------------------------------------- If a query type (e. g.
SegmentMetadataQuery
(it's runners) needs to emit custom dimensions and/or metrics which doesn't make sense for all other query types, the following steps should be executed: 1. Create `interface SegmentMetadataQueryMetrics extends QueryMetrics` (here and below "SegmentMetadata" is the query type) with additional methods (see "Adding new methods" section above). 2. Create `class DefaultSegmentMetadataQueryMetrics implements SegmentMetadataQueryMetrics`. This class should implement extra methods from SegmentMetadataQueryMetrics interfaces with empty bodies, AND DELEGATE ALL OTHER METHODS TO A QueryMetrics OBJECT, provided as a sole parameter in DefaultSegmentMetadataQueryMetrics constructor. NOTE: query(), dataSource(), queryType(), interval(), hasFilters(), duration(), queryId(), sqlQueryId(), and context() methods or any "pre-query-execution-time" methods should either have a empty body or throw exception. 3. Create `interface SegmentMetadataQueryMetricsFactory` with a single method `SegmentMetadataQueryMetrics makeMetrics(SegmentMetadataQuery query);`. 4. Create `class DefaultSegmentMetadataQueryMetricsFactory implements SegmentMetadataQueryMetricsFactory`, which acceptsGenericQueryMetricsFactory
as injected constructor parameter, and implements makeMetrics() as `return new DefaultSegmentMetadataQueryMetrics(genericQueryMetricsFactory.makeMetrics(query));` 5. Inject and use SegmentMetadataQueryMetricsFactory instead ofGenericQueryMetricsFactory
inSegmentMetadataQueryQueryToolChest
. 6. Establish injection of SegmentMetadataQueryMetricsFactory using config and provider method in QueryToolChestModule (see how it is done in QueryToolChestModule) for existing query types with custom metrics, e. g.SearchQueryMetricsFactory
), if the query type belongs to the core druid-processing, e. g. SegmentMetadataQuery. If the query type defined in an extension, you can specify `binder.bind(ScanQueryMetricsFactory.class).to(DefaultScanQueryMetricsFactory.class)` in the extension's Guice module, if the query type is defined in an extension, e. g. ScanQuery. Or establish similar configuration, as for the core query types. This complex procedure is needed to ensure customGenericQueryMetricsFactory
specified by users still works for the query type when query type decides to create their custom QueryMetrics subclass.TopNQueryMetrics
,GroupByQueryMetrics
, andTimeseriesQueryMetrics
are implemented differently, because they are introduced at the same time as the whole QueryMetrics abstraction and their default implementations have to actually emit more dimensions than the default generic QueryMetrics. So those subinterfaces shouldn't be taken as direct examples for following the plan specified above. ReferSearchQueryMetricsFactory
as an implementation example of this procedure.
-
-
Method Summary
All Methods Instance Methods Abstract Methods Modifier and Type Method Description void
context(QueryType query)
SetsQuery.getContext()
of the given query as dimension.void
dataSource(QueryType query)
SetsQuery.getDataSource()
of the given query as dimension.void
duration(QueryType query)
SetsQuery.getDuration()
of the given query as dimension.void
emit(ServiceEmitter emitter)
Emits all metrics, registered since the lastemit()
call on this QueryMetrics object.void
hasFilters(QueryType query)
SetsQuery.hasFilters()
of the given query as dimension.void
identity(String identity)
Sets identity of the requester for a query.void
interval(QueryType query)
SetsQuery.getIntervals()
of the given query as dimension.BitmapResultFactory<?>
makeBitmapResultFactory(BitmapFactory factory)
Creates aBitmapResultFactory
which may record some information along bitmap construction frompreFilters(List)
.void
parallelMergeParallelism(int parallelism)
Sets broker merge parallelism, if parallel merges are enabled.void
postFilters(List<Filter> postFilters)
void
preFilters(List<Filter> preFilters)
void
query(QueryType query)
Pulls all information from the query object into dimensions of future metrics.void
queryId(String queryId)
Sets id of the given query as dimension.void
queryId(QueryType query)
SetsQuery.getId()
of the given query as dimension.void
queryType(QueryType query)
SetsQuery.getType()
of the given query as dimension.void
remoteAddress(String remoteAddress)
QueryMetrics<QueryType>
reportBackPressureTime(long timeNs)
Registers "time that channel is unreadable (backpressure)" metric.QueryMetrics<QueryType>
reportBitmapConstructionTime(long timeNs)
Reports the time spent constructing bitmap frompreFilters(List)
of the query.QueryMetrics<QueryType>
reportCpuTime(long timeNs)
Registers "cpu time" metric.QueryMetrics<QueryType>
reportNodeBytes(long byteCount)
Registers "node bytes" metric.QueryMetrics<QueryType>
reportNodeTime(long timeNs)
Registers "node time" metric.QueryMetrics<QueryType>
reportNodeTimeToFirstByte(long timeNs)
Registers "time to first byte" metric.QueryMetrics<QueryType>
reportParallelMergeFastestPartitionTime(long timeNs)
Reports broker "wall" time in nanoseconds for the fastest parallel merge sequence partition to be 'initialized', where 'initialized' is time to the first result batch is populated from data servers and merging can begin.QueryMetrics<QueryType>
reportParallelMergeInputRows(long numRows)
Reports total number of input rows processed by the broker during parallel merge.QueryMetrics<QueryType>
reportParallelMergeInputSequences(long numSequences)
Reports total number of input sequences processed by the broker during parallel merge.QueryMetrics<QueryType>
reportParallelMergeOutputRows(long numRows)
Reports broker total number of output rows after merging and combining input sequences (should be less than or equal to the value supplied toreportParallelMergeInputRows(long)
.QueryMetrics<QueryType>
reportParallelMergeParallelism(int parallelism)
Reports number of parallel tasks the broker used to process the query during parallel merge.QueryMetrics<QueryType>
reportParallelMergeSlowestPartitionTime(long timeNs)
Reports broker "wall" time in nanoseconds for the slowest parallel merge sequence partition to be 'initialized', where 'initialized' is time to the first result batch is populated from data servers and merging can begin.QueryMetrics<QueryType>
reportParallelMergeTaskCount(long numTasks)
Reports broker total number of fork join pool tasks required to complete queryQueryMetrics<QueryType>
reportParallelMergeTotalCpuTime(long timeNs)
Reports broker total CPU time in nanoseconds where fork join merge combine tasks were doing workQueryMetrics<QueryType>
reportParallelMergeTotalTime(long timeNs)
Reports broker total "wall" time in nanoseconds from parallel merge start sequence creation to total consumption.QueryMetrics<QueryType>
reportPreFilteredRows(long numRows)
Reports the number of rows to scan in the segment after applyingpreFilters(List)
.QueryMetrics<QueryType>
reportQueriedSegmentCount(long segmentCount)
Registers "segments queried count" metric.QueryMetrics<QueryType>
reportQueryBytes(long byteCount)
Registers "query bytes" metric.QueryMetrics<QueryType>
reportQueryTime(long timeNs)
Registers "query time" metric.QueryMetrics<QueryType>
reportSegmentAndCacheTime(long timeNs)
Registers "segmentAndCache time" metric.QueryMetrics<QueryType>
reportSegmentRows(long numRows)
Reports the total number of rows in the processed segment.QueryMetrics<QueryType>
reportSegmentTime(long timeNs)
Registers "segment time" metric.QueryMetrics<QueryType>
reportWaitTime(long timeNs)
Registers "wait time" metric.void
segment(String segmentIdentifier)
void
server(String host)
void
sqlQueryId(String sqlQueryId)
Sets sqlQueryId as a dimensionvoid
sqlQueryId(QueryType query)
SetsQuery.getSqlQueryId()
of the given query as dimensionvoid
status(String status)
void
subQueryId(QueryType query)
SetsQuery.getSubQueryId()
of the given query as dimension.void
success(boolean success)
void
vectorized(boolean vectorized)
Sets whether or not a segment scan has been vectorized.
-
-
-
Method Detail
-
query
void query(QueryType query)
Pulls all information from the query object into dimensions of future metrics.
-
dataSource
void dataSource(QueryType query)
SetsQuery.getDataSource()
of the given query as dimension.
-
queryType
void queryType(QueryType query)
SetsQuery.getType()
of the given query as dimension.
-
interval
void interval(QueryType query)
SetsQuery.getIntervals()
of the given query as dimension.
-
hasFilters
void hasFilters(QueryType query)
SetsQuery.hasFilters()
of the given query as dimension.
-
duration
void duration(QueryType query)
SetsQuery.getDuration()
of the given query as dimension.
-
queryId
void queryId(QueryType query)
SetsQuery.getId()
of the given query as dimension.
-
queryId
void queryId(String queryId)
Sets id of the given query as dimension.
-
subQueryId
void subQueryId(QueryType query)
SetsQuery.getSubQueryId()
of the given query as dimension.
-
sqlQueryId
void sqlQueryId(QueryType query)
SetsQuery.getSqlQueryId()
of the given query as dimension
-
sqlQueryId
void sqlQueryId(String sqlQueryId)
Sets sqlQueryId as a dimension
-
context
void context(QueryType query)
SetsQuery.getContext()
of the given query as dimension.
-
server
void server(String host)
-
remoteAddress
void remoteAddress(String remoteAddress)
-
status
void status(String status)
-
success
void success(boolean success)
-
segment
void segment(String segmentIdentifier)
-
identity
void identity(String identity)
Sets identity of the requester for a query. SeeAuthenticationResult
.
-
vectorized
void vectorized(boolean vectorized)
Sets whether or not a segment scan has been vectorized. Generally expected to only be attached to segment-level metrics, since at whole-query level we might have a mix of vectorized and non-vectorized segment scans.
-
parallelMergeParallelism
void parallelMergeParallelism(int parallelism)
Sets broker merge parallelism, if parallel merges are enabled. This will only appear in broker level metrics. This value is identical to thereportParallelMergeParallelism(int)
metric value, but optionally also available as a dimension.
-
makeBitmapResultFactory
BitmapResultFactory<?> makeBitmapResultFactory(BitmapFactory factory)
Creates aBitmapResultFactory
which may record some information along bitmap construction frompreFilters(List)
. The returned BitmapResultFactory may add some dimensions to this QueryMetrics from it'sBitmapResultFactory.toImmutableBitmap(Object)
method. SeeBitmapResultFactory
Javadoc for more information.
-
reportQueryTime
QueryMetrics<QueryType> reportQueryTime(long timeNs)
Registers "query time" metric. Measures the time between a Jetty thread starting to handle a query, and the response being fully written to the response output stream. Does not include time spent waiting in a queue before the query runs.
-
reportQueryBytes
QueryMetrics<QueryType> reportQueryBytes(long byteCount)
Registers "query bytes" metric. Measures the total number of bytes written by the query server thread to the response output stream. Emitted once per query.
-
reportQueriedSegmentCount
QueryMetrics<QueryType> reportQueriedSegmentCount(long segmentCount)
Registers "segments queried count" metric.
-
reportWaitTime
QueryMetrics<QueryType> reportWaitTime(long timeNs)
Registers "wait time" metric. Measures the total time segment-processing runnables spent waiting for execution in the processing thread pool. Emitted once per segment.
-
reportSegmentTime
QueryMetrics<QueryType> reportSegmentTime(long timeNs)
Registers "segment time" metric. Measures the total wall-clock time spent operating on segments in processing threads. Emitted once per segment.
-
reportSegmentAndCacheTime
QueryMetrics<QueryType> reportSegmentAndCacheTime(long timeNs)
Registers "segmentAndCache time" metric. Measures the total wall-clock time spent in processing threads, either operating on segments or retrieving items from cache. Emitted once per segment.
-
reportCpuTime
QueryMetrics<QueryType> reportCpuTime(long timeNs)
Registers "cpu time" metric.
-
reportNodeTimeToFirstByte
QueryMetrics<QueryType> reportNodeTimeToFirstByte(long timeNs)
Registers "time to first byte" metric.
-
reportBackPressureTime
QueryMetrics<QueryType> reportBackPressureTime(long timeNs)
Registers "time that channel is unreadable (backpressure)" metric.
-
reportNodeTime
QueryMetrics<QueryType> reportNodeTime(long timeNs)
Registers "node time" metric.
-
reportNodeBytes
QueryMetrics<QueryType> reportNodeBytes(long byteCount)
Registers "node bytes" metric.
-
reportBitmapConstructionTime
QueryMetrics<QueryType> reportBitmapConstructionTime(long timeNs)
Reports the time spent constructing bitmap frompreFilters(List)
of the query. Not reported, if there are no preFilters.
-
reportSegmentRows
QueryMetrics<QueryType> reportSegmentRows(long numRows)
Reports the total number of rows in the processed segment.
-
reportPreFilteredRows
QueryMetrics<QueryType> reportPreFilteredRows(long numRows)
Reports the number of rows to scan in the segment after applyingpreFilters(List)
. If the are no preFilters, this metric is equal toreportSegmentRows(long)
.
-
reportParallelMergeParallelism
QueryMetrics<QueryType> reportParallelMergeParallelism(int parallelism)
Reports number of parallel tasks the broker used to process the query during parallel merge. This value is identical to theparallelMergeParallelism(int)
dimension value, but optionally also available as a metric.
-
reportParallelMergeInputSequences
QueryMetrics<QueryType> reportParallelMergeInputSequences(long numSequences)
Reports total number of input sequences processed by the broker during parallel merge.
-
reportParallelMergeInputRows
QueryMetrics<QueryType> reportParallelMergeInputRows(long numRows)
Reports total number of input rows processed by the broker during parallel merge.
-
reportParallelMergeOutputRows
QueryMetrics<QueryType> reportParallelMergeOutputRows(long numRows)
Reports broker total number of output rows after merging and combining input sequences (should be less than or equal to the value supplied toreportParallelMergeInputRows(long)
.
-
reportParallelMergeTaskCount
QueryMetrics<QueryType> reportParallelMergeTaskCount(long numTasks)
Reports broker total number of fork join pool tasks required to complete query
-
reportParallelMergeTotalCpuTime
QueryMetrics<QueryType> reportParallelMergeTotalCpuTime(long timeNs)
Reports broker total CPU time in nanoseconds where fork join merge combine tasks were doing work
-
reportParallelMergeTotalTime
QueryMetrics<QueryType> reportParallelMergeTotalTime(long timeNs)
Reports broker total "wall" time in nanoseconds from parallel merge start sequence creation to total consumption.
-
reportParallelMergeFastestPartitionTime
QueryMetrics<QueryType> reportParallelMergeFastestPartitionTime(long timeNs)
Reports broker "wall" time in nanoseconds for the fastest parallel merge sequence partition to be 'initialized', where 'initialized' is time to the first result batch is populated from data servers and merging can begin. Similar to query 'time to first byte' metrics, except is a composite of the whole group of data servers which are present in the merge partition, which all must supply an initial result batch before merging can actually begin.
-
reportParallelMergeSlowestPartitionTime
QueryMetrics<QueryType> reportParallelMergeSlowestPartitionTime(long timeNs)
Reports broker "wall" time in nanoseconds for the slowest parallel merge sequence partition to be 'initialized', where 'initialized' is time to the first result batch is populated from data servers and merging can begin. Similar to query 'time to first byte' metrics, except is a composite of the whole group of data servers which are present in the merge partition, which all must supply an initial result batch before merging can actually begin.
-
emit
void emit(ServiceEmitter emitter)
Emits all metrics, registered since the lastemit()
call on this QueryMetrics object.
-
-