Class IncrementalIndex
- java.lang.Object
-
- org.apache.druid.segment.incremental.IncrementalIndex
-
- All Implemented Interfaces:
Closeable
,AutoCloseable
,Iterable<Row>
,Expr.InputBindingInspector
,ColumnInspector
- Direct Known Subclasses:
OnheapIncrementalIndex
public abstract class IncrementalIndex extends Object implements Iterable<Row>, Closeable, ColumnInspector
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description static class
IncrementalIndex.DimensionDesc
static class
IncrementalIndex.MetricDesc
-
Field Summary
Fields Modifier and Type Field Description protected boolean
preserveExistingMetrics
-
Constructor Summary
Constructors Modifier Constructor Description protected
IncrementalIndex(IncrementalIndexSchema incrementalIndexSchema, boolean deserializeComplexMetrics, boolean concurrentEventAdd, boolean preserveExistingMetrics, boolean useMaxMemoryEstimates)
Setting deserializeComplexMetrics to false is necessary for intermediate aggregation such as groupBy that should not deserialize input columns using ComplexMetricSerde for aggregators that return complex metrics.
-
Method Summary
All Methods Static Methods Instance Methods Abstract Methods Concrete Methods Modifier and Type Method Description IncrementalIndexAddResult
add(InputRow row)
Adds a new row.IncrementalIndexAddResult
add(InputRow row, boolean skipMaxRowsInMemoryCheck)
Adds a new row.protected abstract org.apache.druid.segment.incremental.IncrementalIndex.AddToFactsResult
addToFacts(InputRow row, IncrementalIndexRow key, ThreadLocal<InputRow> rowContainer, com.google.common.base.Supplier<InputRow> rowSupplier, boolean skipMaxRowsInMemoryCheck)
abstract boolean
canAppendRow()
void
close()
protected Comparator<IncrementalIndexRow>
dimsComparator()
InputRow
formatRow(InputRow row)
AtomicLong
getBytesInMemory()
ColumnCapabilities
getColumnCapabilities(String columnName)
Returns capabilities of a particular column.ColumnFormat
getColumnFormat(String columnName)
Map<String,ColumnFormat>
getColumnFormats()
List<String>
getColumnNames()
static ParseException
getCombinedParseException(InputRow row, List<String> dimParseExceptionMessages, List<String> aggParseExceptionMessages)
IncrementalIndex.DimensionDesc
getDimension(String dimension)
Integer
getDimensionIndex(String dimension)
List<String>
getDimensionNames()
List<String>
getDimensionOrder()
List<IncrementalIndex.DimensionDesc>
getDimensions()
DimensionsSpec
getDimensionsSpec()
Returns dimensionsSpec from the ingestionSpec.abstract org.apache.druid.segment.incremental.IncrementalIndex.FactsHolder
getFacts()
org.joda.time.Interval
getInterval()
abstract int
getLastRowIndex()
org.joda.time.DateTime
getMaxIngestedEventTime()
org.joda.time.DateTime
getMaxTime()
Metadata
getMetadata()
AggregatorFactory[]
getMetricAggs()
protected abstract double
getMetricDoubleValue(int rowOffset, int aggOffset)
protected abstract float
getMetricFloatValue(int rowOffset, int aggOffset)
protected abstract long
getMetricLongValue(int rowOffset, int aggOffset)
List<String>
getMetricNames()
protected abstract Object
getMetricObjectValue(int rowOffset, int aggOffset)
org.joda.time.DateTime
getMinTime()
abstract String
getOutOfRowsReason()
protected abstract void
initAggs(AggregatorFactory[] metrics, com.google.common.base.Supplier<InputRow> rowSupplier, boolean deserializeComplexMetrics, boolean concurrentEventAdd)
boolean
isEmpty()
protected abstract boolean
isNull(int rowOffset, int aggOffset)
boolean
isRollup()
abstract Iterable<Row>
iterableWithPostAggregations(List<PostAggregator> postAggs, boolean descending)
Iterator<Row>
iterator()
void
loadDimensionIterable(Iterable<String> oldDimensionOrder, Map<String,ColumnFormat> oldColumnCapabilities)
Currently called to initialize IncrementalIndex dimension order during index creation Index dimension ordering could be changed to initialize from DimensionsSpec after resolution of https://github.com/apache/druid/issues/2011protected ColumnSelectorFactory
makeColumnSelectorFactory(AggregatorFactory agg, com.google.common.base.Supplier<InputRow> in, boolean deserializeComplexMetrics)
static ColumnSelectorFactory
makeColumnSelectorFactory(VirtualColumns virtualColumns, AggregatorFactory agg, com.google.common.base.Supplier<InputRow> in, boolean deserializeComplexMetrics)
Column selector used at ingestion time for inputs to aggregators.static ColumnCapabilitiesImpl
makeDefaultCapabilitiesFromValueType(ColumnType type)
ColumnValueSelector<?>
makeMetricColumnValueSelector(String metric, IncrementalIndexRowHolder currEntry)
int
size()
-
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
-
Methods inherited from interface org.apache.druid.segment.ColumnInspector
getColumnCapabilitiesWithDefault, getType
-
Methods inherited from interface org.apache.druid.math.expr.Expr.InputBindingInspector
areNumeric, areNumeric, areSameTypes, areSameTypes, areScalar, areScalar, canVectorize, canVectorize
-
Methods inherited from interface java.lang.Iterable
forEach, spliterator
-
-
-
-
Constructor Detail
-
IncrementalIndex
protected IncrementalIndex(IncrementalIndexSchema incrementalIndexSchema, boolean deserializeComplexMetrics, boolean concurrentEventAdd, boolean preserveExistingMetrics, boolean useMaxMemoryEstimates)
Setting deserializeComplexMetrics to false is necessary for intermediate aggregation such as groupBy that should not deserialize input columns using ComplexMetricSerde for aggregators that return complex metrics.Set concurrentEventAdd to true to indicate that adding of input row should be thread-safe (for example, groupBy where the multiple threads can add concurrently to the IncrementalIndex).
- Parameters:
incrementalIndexSchema
- the schema to use for incremental indexdeserializeComplexMetrics
- flag whether or not to call ComplexMetricExtractor.extractValue() on the input value for aggregators that return metrics other than float.concurrentEventAdd
- flag whether ot not adding of input rows should be thread-safepreserveExistingMetrics
- When set to true, for any row that already has metric (with the same name defined in metricSpec), the metric aggregator in metricSpec is skipped and the existing metric is unchanged. If the row does not already have the metric, then the metric aggregator is applied on the source column as usual. This should only be set for DruidInputSource since that is the only case where we can have existing metrics. This is currently only use by auto compaction and should not be use for anything else.useMaxMemoryEstimates
- true if max values should be used to estimate memory
-
-
Method Detail
-
makeColumnSelectorFactory
public static ColumnSelectorFactory makeColumnSelectorFactory(VirtualColumns virtualColumns, AggregatorFactory agg, com.google.common.base.Supplier<InputRow> in, boolean deserializeComplexMetrics)
Column selector used at ingestion time for inputs to aggregators.- Parameters:
agg
- the aggregatorin
- ingestion-time input row supplierdeserializeComplexMetrics
- whether complex objects should be deserialized by aComplexMetricExtractor
- Returns:
- column selector factory
-
getFacts
public abstract org.apache.druid.segment.incremental.IncrementalIndex.FactsHolder getFacts()
-
canAppendRow
public abstract boolean canAppendRow()
-
getOutOfRowsReason
public abstract String getOutOfRowsReason()
-
initAggs
protected abstract void initAggs(AggregatorFactory[] metrics, com.google.common.base.Supplier<InputRow> rowSupplier, boolean deserializeComplexMetrics, boolean concurrentEventAdd)
-
addToFacts
protected abstract org.apache.druid.segment.incremental.IncrementalIndex.AddToFactsResult addToFacts(InputRow row, IncrementalIndexRow key, ThreadLocal<InputRow> rowContainer, com.google.common.base.Supplier<InputRow> rowSupplier, boolean skipMaxRowsInMemoryCheck) throws IndexSizeExceededException
- Throws:
IndexSizeExceededException
-
getLastRowIndex
public abstract int getLastRowIndex()
-
getMetricFloatValue
protected abstract float getMetricFloatValue(int rowOffset, int aggOffset)
-
getMetricLongValue
protected abstract long getMetricLongValue(int rowOffset, int aggOffset)
-
getMetricObjectValue
protected abstract Object getMetricObjectValue(int rowOffset, int aggOffset)
-
getMetricDoubleValue
protected abstract double getMetricDoubleValue(int rowOffset, int aggOffset)
-
isNull
protected abstract boolean isNull(int rowOffset, int aggOffset)
-
isRollup
public boolean isRollup()
-
close
public void close()
- Specified by:
close
in interfaceAutoCloseable
- Specified by:
close
in interfaceCloseable
-
getColumnFormats
public Map<String,ColumnFormat> getColumnFormats()
-
getColumnCapabilities
@Nullable public ColumnCapabilities getColumnCapabilities(String columnName)
Description copied from interface:ColumnInspector
Returns capabilities of a particular column.- Specified by:
getColumnCapabilities
in interfaceColumnInspector
- Parameters:
columnName
- column name- Returns:
- capabilities, or null
-
getColumnFormat
@Nullable public ColumnFormat getColumnFormat(String columnName)
-
add
public IncrementalIndexAddResult add(InputRow row) throws IndexSizeExceededException
Adds a new row. The row might correspond with another row that already exists, in which case this will update that row instead of inserting a new one.Calls to add() are thread safe.
- Parameters:
row
- the row of data to add- Returns:
- the number of rows in the data set after adding the InputRow. If any parse failure occurs, a
ParseException
is returned inIncrementalIndexAddResult
. - Throws:
IndexSizeExceededException
-
add
public IncrementalIndexAddResult add(InputRow row, boolean skipMaxRowsInMemoryCheck) throws IndexSizeExceededException
Adds a new row. The row might correspond with another row that already exists, in which case this will update that row instead of inserting a new one.Calls to add() are thread safe.
- Parameters:
row
- the row of data to addskipMaxRowsInMemoryCheck
- whether or not to skip the check of rows exceeding the max rows limit- Returns:
- the number of rows in the data set after adding the InputRow. If any parse failure occurs, a
ParseException
is returned inIncrementalIndexAddResult
. - Throws:
IndexSizeExceededException
- this exception is thrown once it reaches max rows limit and skipMaxRowsInMemoryCheck is set to false.
-
getCombinedParseException
@Nullable public static ParseException getCombinedParseException(InputRow row, @Nullable List<String> dimParseExceptionMessages, @Nullable List<String> aggParseExceptionMessages)
-
isEmpty
public boolean isEmpty()
-
size
public int size()
-
getBytesInMemory
public AtomicLong getBytesInMemory()
-
getMetricAggs
public AggregatorFactory[] getMetricAggs()
-
getDimensionsSpec
public DimensionsSpec getDimensionsSpec()
Returns dimensionsSpec from the ingestionSpec.
-
getDimensions
public List<IncrementalIndex.DimensionDesc> getDimensions()
-
getDimension
@Nullable public IncrementalIndex.DimensionDesc getDimension(String dimension)
-
makeMetricColumnValueSelector
public ColumnValueSelector<?> makeMetricColumnValueSelector(String metric, IncrementalIndexRowHolder currEntry)
-
getInterval
public org.joda.time.Interval getInterval()
-
getMinTime
@Nullable public org.joda.time.DateTime getMinTime()
-
getMaxTime
@Nullable public org.joda.time.DateTime getMaxTime()
-
makeDefaultCapabilitiesFromValueType
public static ColumnCapabilitiesImpl makeDefaultCapabilitiesFromValueType(ColumnType type)
-
loadDimensionIterable
public void loadDimensionIterable(Iterable<String> oldDimensionOrder, Map<String,ColumnFormat> oldColumnCapabilities)
Currently called to initialize IncrementalIndex dimension order during index creation Index dimension ordering could be changed to initialize from DimensionsSpec after resolution of https://github.com/apache/druid/issues/2011
-
getMetadata
public Metadata getMetadata()
-
iterableWithPostAggregations
public abstract Iterable<Row> iterableWithPostAggregations(@Nullable List<PostAggregator> postAggs, boolean descending)
-
getMaxIngestedEventTime
public org.joda.time.DateTime getMaxIngestedEventTime()
-
makeColumnSelectorFactory
protected ColumnSelectorFactory makeColumnSelectorFactory(AggregatorFactory agg, com.google.common.base.Supplier<InputRow> in, boolean deserializeComplexMetrics)
-
dimsComparator
protected final Comparator<IncrementalIndexRow> dimsComparator()
-
-