org.apache.druid.query

Class DruidProcessingConfig

java.lang.Object

org.apache.druid.java.util.common.concurrent.ExecutorServiceConfig

org.apache.druid.query.DruidProcessingConfig

All Implemented Interfaces:

ColumnConfig

public abstract class DruidProcessingConfig
extends ExecutorServiceConfig
implements ColumnConfig

Field Summary

Fields

Modifier and Type	Field and Description
static int	DEFAULT_INITIAL_BUFFERS_FOR_INTERMEDIATE_POOL
static int	DEFAULT_MERGE_POOL_AWAIT_SHUTDOWN_MILLIS
static int	DEFAULT_NUM_MERGE_BUFFERS
static HumanReadableBytes	DEFAULT_PROCESSING_BUFFER_SIZE_BYTES
static int	MAX_DEFAULT_PROCESSING_BUFFER_SIZE_BYTES

Fields inherited from class org.apache.druid.java.util.common.concurrent.ExecutorServiceConfig

DEFAULT_NUM_THREADS

Fields inherited from interface org.apache.druid.segment.column.ColumnConfig

ALWAYS_USE_INDEXES, DEFAULT

Constructor Summary

Constructors

Constructor and Description
DruidProcessingConfig()

Method Summary

Modifier and Type	Method and Description
static long	computeMaxMemoryFromMaxHeapSize()
long	getMergePoolAwaitShutdownMillis()
int	getMergePoolDefaultMaxQueryParallelism()
int	getMergePoolParallelism()
int	getMergePoolParallelismConfigured()
int	getMergePoolSmallBatchRows()
int	getMergePoolTargetTaskRunTimeMillis()
int	getMergePoolTaskInitialYieldRows()
int	getNumInitalBuffersForIntermediatePool()
int	getNumMergeBuffers()
int	getNumMergeBuffersConfigured() Returns the number of merge buffers _explicitly_ configured, or -1 if it is not explicitly configured, that is not a valid number of buffers.
int	getNumThreadsConfigured() Returns the number of threads _explicitly_ configured, or -1 if it is not explicitly configured, that is not a valid number of threads.
String	getTmpDir()
int	intermediateComputeSizeBytes()
HumanReadableBytes	intermediateComputeSizeBytesConfigured()
boolean	isFifo()
int	poolCacheMaxCount()
double	skipValuePredicateIndexScale() If the total number of rows in a column multiplied by this value is smaller than the total number of bitmap index operations required to perform to use DruidPredicateIndexes then for any ColumnIndexSupplier which chooses to participate in this config it will skip computing the index, in favor of doing a full scan and using a ValueMatcher instead.
double	skipValueRangeIndexScale() If the total number of rows in a column multiplied by this value is smaller than the total number of bitmap index operations required to perform to use LexicographicalRangeIndexes or NumericRangeIndexes, then for any ColumnIndexSupplier which chooses to participate in this config it will skip computing the index, indicated by a return value of null from the 'forRange' methods, to force the filter to be processed with a scan using a ValueMatcher instead.
boolean	useParallelMergePool()
boolean	useParallelMergePoolConfigured()

Methods inherited from class org.apache.druid.java.util.common.concurrent.ExecutorServiceConfig

getFormatString, getNumThreads

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_NUM_MERGE_BUFFERS

public static final int DEFAULT_NUM_MERGE_BUFFERS

See Also:

Constant Field Values

DEFAULT_PROCESSING_BUFFER_SIZE_BYTES

public static final HumanReadableBytes DEFAULT_PROCESSING_BUFFER_SIZE_BYTES

MAX_DEFAULT_PROCESSING_BUFFER_SIZE_BYTES

public static final int MAX_DEFAULT_PROCESSING_BUFFER_SIZE_BYTES

See Also:

Constant Field Values

DEFAULT_MERGE_POOL_AWAIT_SHUTDOWN_MILLIS

public static final int DEFAULT_MERGE_POOL_AWAIT_SHUTDOWN_MILLIS

See Also:

Constant Field Values

DEFAULT_INITIAL_BUFFERS_FOR_INTERMEDIATE_POOL

public static final int DEFAULT_INITIAL_BUFFERS_FOR_INTERMEDIATE_POOL

See Also:

Constant Field Values

Constructor Detail

DruidProcessingConfig

public DruidProcessingConfig()

Method Detail

intermediateComputeSizeBytesConfigured

public HumanReadableBytes intermediateComputeSizeBytesConfigured()

intermediateComputeSizeBytes

public int intermediateComputeSizeBytes()

computeMaxMemoryFromMaxHeapSize

public static long computeMaxMemoryFromMaxHeapSize()

poolCacheMaxCount

public int poolCacheMaxCount()

getNumInitalBuffersForIntermediatePool

public int getNumInitalBuffersForIntermediatePool()

getNumThreadsConfigured

public int getNumThreadsConfigured()

Description copied from class: ExecutorServiceConfig

Returns the number of threads _explicitly_ configured, or -1 if it is not explicitly configured, that is not a valid number of threads. To get the configured value or the default (valid) number, use ExecutorServiceConfig.getNumThreads(). This method exists for ability to distinguish between the default value set when there is no explicit config, and an explicitly configured value.

Overrides:

getNumThreadsConfigured in class ExecutorServiceConfig

getNumMergeBuffers

public int getNumMergeBuffers()

getNumMergeBuffersConfigured

public int getNumMergeBuffersConfigured()

Returns the number of merge buffers _explicitly_ configured, or -1 if it is not explicitly configured, that is not a valid number of buffers. To get the configured value or the default (valid) number, use getNumMergeBuffers(). This method exists for ability to distinguish between the default value set when there is no explicit config, and an explicitly configured value.

skipValueRangeIndexScale

public double skipValueRangeIndexScale()

Description copied from interface: ColumnConfig

If the total number of rows in a column multiplied by this value is smaller than the total number of bitmap index operations required to perform to use LexicographicalRangeIndexes or NumericRangeIndexes, then for any ColumnIndexSupplier which chooses to participate in this config it will skip computing the index, indicated by a return value of null from the 'forRange' methods, to force the filter to be processed with a scan using a ValueMatcher instead.

For range indexes on columns where every value has an index, the number of bitmap operations is determined by how many individual values fall in the range, a subset of the columns total cardinality.

Currently only the NestedCommonFormatColumn implementations of ColumnIndexSupplier support this behavior.

This can make some standalone filters faster in cases where the overhead of walking the value dictionary and combining bitmaps to construct a BitmapOffset or BitmapVectorOffset can exceed the cost of just using doing a full scan and using a ValueMatcher.

Where this is especially useful is in cases where the range index is used as part of some AndFilter, which segment processing partitions into groups of 'pre' filters, composed of those which should use indexes, and 'post' filters, which should use a matcher on the offset created by the indexes to filter the remaining results. This value pushes what would have been expensive index computations to go into the 'pre' group into using a value matcher as part of the 'post' group instead, sometimes providing an order of magnitude or higher performance increase.

Specified by:

skipValueRangeIndexScale in interface ColumnConfig

skipValuePredicateIndexScale

public double skipValuePredicateIndexScale()

Description copied from interface: ColumnConfig

If the total number of rows in a column multiplied by this value is smaller than the total number of bitmap index operations required to perform to use DruidPredicateIndexes then for any ColumnIndexSupplier which chooses to participate in this config it will skip computing the index, in favor of doing a full scan and using a ValueMatcher instead. This is indicated returning null from ColumnIndexSupplier.as(Class) even though it would have otherwise been able to create a BitmapColumnIndex. For predicate indexes, this is determined by the total value cardinality of the column for columns with an index for every value.

Currently only the NestedCommonFormatColumn implementations of ColumnIndexSupplier support this behavior.

This can make some standalone filters faster in cases where the overhead of walking the value dictionary and combining bitmaps to construct a BitmapOffset or BitmapVectorOffset can exceed the cost of just using doing a full scan and using a ValueMatcher.

Where this is especially useful is in cases where the predicate index is used as part of some AndFilter, which segment processing partitions into groups of 'pre' filters, composed of those which should use indexes, and 'post' filters, which should use a matcher on the offset created by the indexes to filter the remaining results. This value pushes what would have been expensive index computations to go into the 'pre' group into using a value matcher as part of the 'post' group instead, sometimes providing an order of magnitude or higher performance increase.

This value is separate from ColumnConfig.skipValueRangeIndexScale() since the dynamics of computing predicate indexes is potentially different than the much cheaper range calculations (especially for numeric values), so having a separate control knob allows for corrections to be done to tune things separately from ranges.

Specified by:

skipValuePredicateIndexScale in interface ColumnConfig

isFifo

public boolean isFifo()

getTmpDir

public String getTmpDir()

useParallelMergePoolConfigured

public boolean useParallelMergePoolConfigured()

useParallelMergePool

public boolean useParallelMergePool()

getMergePoolParallelismConfigured

public int getMergePoolParallelismConfigured()

getMergePoolParallelism

public int getMergePoolParallelism()

getMergePoolAwaitShutdownMillis

public long getMergePoolAwaitShutdownMillis()

getMergePoolDefaultMaxQueryParallelism

public int getMergePoolDefaultMaxQueryParallelism()

getMergePoolTargetTaskRunTimeMillis

public int getMergePoolTargetTaskRunTimeMillis()

getMergePoolTaskInitialYieldRows

public int getMergePoolTaskInitialYieldRows()

getMergePoolSmallBatchRows

public int getMergePoolSmallBatchRows()