Package org.apache.druid.segment.incremental

Class IncrementalIndex

    • Field Detail

      • preserveExistingMetrics

        protected final boolean preserveExistingMetrics

    • Constructor Detail

      • IncrementalIndex

        protected IncrementalIndex​(IncrementalIndexSchema incrementalIndexSchema,
                           boolean deserializeComplexMetrics,
                           boolean concurrentEventAdd,
                           boolean preserveExistingMetrics,
                           boolean useMaxMemoryEstimates)
        Setting deserializeComplexMetrics to false is necessary for intermediate aggregation such as groupBy that should not deserialize input columns using ComplexMetricSerde for aggregators that return complex metrics.

        Set concurrentEventAdd to true to indicate that adding of input row should be thread-safe (for example, groupBy where the multiple threads can add concurrently to the IncrementalIndex).

        Parameters:
        incrementalIndexSchema - the schema to use for incremental index
        deserializeComplexMetrics - flag whether or not to call ComplexMetricExtractor.extractValue() on the input value for aggregators that return metrics other than float.
        concurrentEventAdd - flag whether ot not adding of input rows should be thread-safe
        preserveExistingMetrics - When set to true, for any row that already has metric (with the same name defined in metricSpec), the metric aggregator in metricSpec is skipped and the existing metric is unchanged. If the row does not already have the metric, then the metric aggregator is applied on the source column as usual. This should only be set for DruidInputSource since that is the only case where we can have existing metrics. This is currently only use by auto compaction and should not be use for anything else.
        useMaxMemoryEstimates - true if max values should be used to estimate memory

    • Method Detail

      • makeColumnSelectorFactory

        public static ColumnSelectorFactory makeColumnSelectorFactory​(VirtualColumns virtualColumns,
                                                              AggregatorFactory agg,
                                                              com.google.common.base.Supplier<InputRow> in,
                                                              boolean deserializeComplexMetrics)
        Column selector used at ingestion time for inputs to aggregators.
        Parameters:
        agg - the aggregator
        in - ingestion-time input row supplier
        deserializeComplexMetrics - whether complex objects should be deserialized by a ComplexMetricExtractor
        Returns:
        column selector factory

      • getFacts

        public abstract org.apache.druid.segment.incremental.IncrementalIndex.FactsHolder getFacts()

      • canAppendRow

        public abstract boolean canAppendRow()

      • getOutOfRowsReason

        public abstract String getOutOfRowsReason()

      • initAggs

        protected abstract void initAggs​(AggregatorFactory[] metrics,
                                 com.google.common.base.Supplier<InputRow> rowSupplier,
                                 boolean deserializeComplexMetrics,
                                 boolean concurrentEventAdd)

      • getLastRowIndex

        public abstract int getLastRowIndex()

      • getMetricFloatValue

        protected abstract float getMetricFloatValue​(int rowOffset,
                                             int aggOffset)

      • getMetricLongValue

        protected abstract long getMetricLongValue​(int rowOffset,
                                           int aggOffset)

      • getMetricObjectValue

        protected abstract Object getMetricObjectValue​(int rowOffset,
                                               int aggOffset)

      • getMetricDoubleValue

        protected abstract double getMetricDoubleValue​(int rowOffset,
                                               int aggOffset)

      • isNull

        protected abstract boolean isNull​(int rowOffset,
                                  int aggOffset)

      • isRollup

        public boolean isRollup()

      • add

        public IncrementalIndexAddResult add​(InputRow row,
                                     boolean skipMaxRowsInMemoryCheck)
                              throws IndexSizeExceededException
        Adds a new row. The row might correspond with another row that already exists, in which case this will update that row instead of inserting a new one.

        Calls to add() are thread safe.

        Parameters:
        row - the row of data to add
        skipMaxRowsInMemoryCheck - whether or not to skip the check of rows exceeding the max rows limit
        Returns:
        the number of rows in the data set after adding the InputRow. If any parse failure occurs, a ParseException is returned in IncrementalIndexAddResult.
        Throws:
        IndexSizeExceededException - this exception is thrown once it reaches max rows limit and skipMaxRowsInMemoryCheck is set to false.

      • isEmpty

        public boolean isEmpty()

      • size

        public int size()

      • getBytesInMemory

        public AtomicLong getBytesInMemory()

      • getDimensionsSpec

        public DimensionsSpec getDimensionsSpec()
        Returns dimensionsSpec from the ingestionSpec.

      • getDimensionNames

        public List<String> getDimensionNames()

      • getInterval

        public org.joda.time.Interval getInterval()

      • getMinTime

        @Nullable
public org.joda.time.DateTime getMinTime()

      • getMaxTime

        @Nullable
public org.joda.time.DateTime getMaxTime()

      • getDimensionOrder

        public List<String> getDimensionOrder()

      • loadDimensionIterable

        public void loadDimensionIterable​(Iterable<String> oldDimensionOrder,
                                  Map<String,​ColumnFormat> oldColumnCapabilities)
        Currently called to initialize IncrementalIndex dimension order during index creation Index dimension ordering could be changed to initialize from DimensionsSpec after resolution of https://github.com/apache/druid/issues/2011

      • getMetricNames

        public List<String> getMetricNames()

      • getColumnNames

        public List<String> getColumnNames()

      • getMetadata

        public Metadata getMetadata()

      • getMaxIngestedEventTime

        public org.joda.time.DateTime getMaxIngestedEventTime()