java.lang.Object
- org.apache.druid.segment.AutoTypeColumnIndexer

All Implemented Interfaces:: DimensionIndexer<StructuredData,StructuredData,StructuredData>

public class AutoTypeColumnIndexer
extends Object
implements DimensionIndexer<StructuredData,StructuredData,StructuredData>

Field Summary

Fields
Modifier and Type	Field	Description
`protected Object`	`constantValue`
`protected SortedMap<String,org.apache.druid.segment.AutoTypeColumnIndexer.FieldIndexer>`	`fieldIndexers`
`protected ValueDictionary`	`globalDictionary`
`protected boolean`	`hasNestedData`	Have we seen any objects? Empty objects in particular are sneaky, they don't have any nested paths, so we also broadly track if we have processed any objects `StructuredDataProcessor.ProcessResults.hasObjects()`
`protected boolean`	`hasNulls`	have we seen any null values?
`protected StructuredDataProcessor`	`indexerProcessor`
`protected boolean`	`isConstant`

Constructor Summary

Constructors
Constructor Description

AutoTypeColumnIndexer()

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type	Method	Description
`boolean`	`checkUnsortedEncodedKeyComponentsEqual(StructuredData lhs, StructuredData rhs)`	Check if two row value arrays from Row keys are equal.
`int`	`compareUnsortedEncodedKeyComponents(StructuredData lhs, StructuredData rhs)`	Compares the row values for this DimensionIndexer's dimension from a Row key.
`Object`	`convertUnsortedEncodedKeyComponentToActualList(StructuredData key)`	Given a row value array from a Row key, as described in the documentation for `DimensionIndexer.compareUnsortedEncodedKeyComponents(EncodedKeyComponentType, EncodedKeyComponentType)`, convert the unsorted encoded values to a list of actual values.
`ColumnValueSelector`	`convertUnsortedValuesToSorted(ColumnValueSelector selectorWithUnsortedValues)`	Converts dictionary-encoded row values from unspecified (random) encoding order, to sorted encoding.
`void`	`fillBitmapsFromUnsortedEncodedKeyComponent(StructuredData key, int rowNum, MutableBitmap[] bitmapIndexes, BitmapFactory factory)`	Helper function for building bitmap indexes for integer-encoded dimensions.
`int`	`getCardinality()`	Get the cardinality of this dimension's values.
`ColumnCapabilities`	`getColumnCapabilities()`
`Object`	`getConstantValue()`
`SortedMap<String,FieldTypeInfo.MutableTypeSet>`	`getFieldTypeInfo()`
`ColumnFormat`	`getFormat()`
`ColumnType`	`getLogicalType()`
`StructuredData`	`getMaxValue()`	Get the maximum dimension value seen by this indexer.
`StructuredData`	`getMinValue()`	Get the minimum dimension value seen by this indexer.
`CloseableIndexed<StructuredData>`	`getSortedIndexedValues()`	Returns an indexed structure of this dimension's sorted actual values.
`SortedValueDictionary`	`getSortedValueLookups()`
`int`	`getUnsortedEncodedKeyComponentHashCode(StructuredData key)`	Given a row value array from a Row key, generate a hashcode.
`StructuredData`	`getUnsortedEncodedValueFromSorted(StructuredData sortedIntermediateValue)`	Given an encoded value that was ordered by associated actual value, return the equivalent encoded value ordered by time of ingestion.
`boolean`	`isConstant()`
`ColumnValueSelector<?>`	`makeColumnValueSelector(IncrementalIndexRowHolder currEntry, IncrementalIndex.DimensionDesc desc)`	Return an object used to read values from this indexer's column.
`DimensionSelector`	`makeDimensionSelector(DimensionSpec spec, IncrementalIndexRowHolder currEntry, IncrementalIndex.DimensionDesc desc)`	Return an object used to read values from this indexer's column as Strings.
`EncodedKeyComponent<StructuredData>`	`processRowValsToUnsortedEncodedKeyComponent(Object dimValues, boolean reportParseExceptions)`	Encodes the given row value(s) of the dimension to be used within a row key.
`void`	`setSparseIndexed()`	This method will be called while building an `IncrementalIndex` whenever a known dimension column (either through an explicit schema on the ingestion spec, or auto-discovered while processing rows) is absent in any row that is processed, to allow an indexer to account for any missing rows if necessary.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - hasNulls
```
protected volatile boolean hasNulls
```
    have we seen any null values?
  - hasNestedData
```
protected volatile boolean hasNestedData
```
    Have we seen any objects? Empty objects in particular are sneaky, they don't have any nested paths, so we also broadly track if we have processed any objects StructuredDataProcessor.ProcessResults.hasObjects()
  - isConstant
```
protected volatile boolean isConstant
```
  - constantValue
```
@Nullable
protected volatile Object constantValue
```
  - fieldIndexers
```
protected SortedMap<String,org.apache.druid.segment.AutoTypeColumnIndexer.FieldIndexer> fieldIndexers
```
  - globalDictionary
```
protected final ValueDictionary globalDictionary
```
  - indexerProcessor
```
protected final StructuredDataProcessor indexerProcessor
```
- Constructor Detail
  - AutoTypeColumnIndexer
```
public AutoTypeColumnIndexer()
```
- Method Detail
  - processRowValsToUnsortedEncodedKeyComponent
```
public EncodedKeyComponent<StructuredData> processRowValsToUnsortedEncodedKeyComponent(@Nullable
                                                                                       Object dimValues,
                                                                                       boolean reportParseExceptions)
```
    Description copied from interface: DimensionIndexer
    
    Encodes the given row value(s) of the dimension to be used within a row key. It also updates the internal state of the DimensionIndexer, e.g. the dimLookup.
    For example, the dictionary-encoded String-type column will return an int[] containing dictionary IDs.
    
    Specified by:
    
    processRowValsToUnsortedEncodedKeyComponent in interface DimensionIndexer<StructuredData,StructuredData,StructuredData>
    
    Parameters:
    
    dimValues - Value(s) of the dimension in a row. This can either be a single value or a list of values (for multi-valued dimensions)
    
    reportParseExceptions - true if parse exceptions should be reported, false otherwise
    
    Returns:
    
    Encoded dimension value(s) to be used as a component for the row key. Contains an object of the DimensionIndexer and the effective size of the key component in bytes.
  - setSparseIndexed
```
public void setSparseIndexed()
```
    Description copied from interface: DimensionIndexer
    
    This method will be called while building an IncrementalIndex whenever a known dimension column (either through an explicit schema on the ingestion spec, or auto-discovered while processing rows) is absent in any row that is processed, to allow an indexer to account for any missing rows if necessary. Useful so that a string DimensionSelector built on top of an IncrementalIndex may accurately report DimensionDictionarySelector.nameLookupPossibleInAdvance() by allowing it to track if it has any implicit null valued rows. At index persist/merge time all missing columns for a row will be explicitly replaced with the value appropriate null or default value.
    
    Specified by:
    
    setSparseIndexed in interface DimensionIndexer<StructuredData,StructuredData,StructuredData>
  - getUnsortedEncodedValueFromSorted
```
public StructuredData getUnsortedEncodedValueFromSorted(StructuredData sortedIntermediateValue)
```
    Description copied from interface: DimensionIndexer
    
    Given an encoded value that was ordered by associated actual value, return the equivalent encoded value ordered by time of ingestion. Using the example in the class description: getUnsortedEncodedValueFromSorted(2) would return 0
    
    Specified by:
    
    getUnsortedEncodedValueFromSorted in interface DimensionIndexer<StructuredData,StructuredData,StructuredData>
    
    Parameters:
    
    sortedIntermediateValue - value to convert
    
    Returns:
    
    converted value
  - getSortedIndexedValues
```
public CloseableIndexed<StructuredData> getSortedIndexedValues()
```
    Description copied from interface: DimensionIndexer
    
    Returns an indexed structure of this dimension's sorted actual values. The integer IDs represent the ordering of the sorted values. Using the example in the class description: "Apple"=0, "Hello"=1, "World"=2
    
    Specified by:
    
    getSortedIndexedValues in interface DimensionIndexer<StructuredData,StructuredData,StructuredData>
    
    Returns:
    
    Sorted index of actual values
  - getSortedValueLookups
```
public SortedValueDictionary getSortedValueLookups()
```
  - getFieldTypeInfo
```
public SortedMap<String,FieldTypeInfo.MutableTypeSet> getFieldTypeInfo()
```
  - getMinValue
```
public StructuredData getMinValue()
```
    Description copied from interface: DimensionIndexer
    
    Get the minimum dimension value seen by this indexer. NOTE: On an in-memory segment (IncrementalIndex), we can determine min/max values by looking at the stream of row values seen in calls to processSingleRowValToIndexKey(). However, on a disk-backed segment (QueryableIndex), the numeric dimensions do not currently have any supporting index structures that can be used to efficiently determine min/max values. When numeric dimension support is added, the segment format should be changed to store min/max values, to avoid performing a full-column scan to determine these values for numeric dims.
    
    Specified by:
    
    getMinValue in interface DimensionIndexer<StructuredData,StructuredData,StructuredData>
    
    Returns:
    
    min value
  - getMaxValue
```
public StructuredData getMaxValue()
```
    Description copied from interface: DimensionIndexer
    
    Get the maximum dimension value seen by this indexer.
    
    Specified by:
    
    getMaxValue in interface DimensionIndexer<StructuredData,StructuredData,StructuredData>
    
    Returns:
    
    max value
  - getCardinality
```
public int getCardinality()
```
    Description copied from interface: DimensionIndexer
    
    Get the cardinality of this dimension's values.
    
    Specified by:
    
    getCardinality in interface DimensionIndexer<StructuredData,StructuredData,StructuredData>
    
    Returns:
    
    value cardinality
  - makeDimensionSelector
```
public DimensionSelector makeDimensionSelector(DimensionSpec spec,
                                               IncrementalIndexRowHolder currEntry,
                                               IncrementalIndex.DimensionDesc desc)
```
    Description copied from interface: DimensionIndexer
    
    Return an object used to read values from this indexer's column as Strings.
    
    Specified by:
    
    makeDimensionSelector in interface DimensionIndexer<StructuredData,StructuredData,StructuredData>
    
    Parameters:
    
    spec - Specifies the output name of a dimension and any extraction functions to be applied.
    
    currEntry - Provides access to the current Row object in the Cursor
    
    desc - Descriptor object for this dimension within an IncrementalIndex
    
    Returns:
    
    A new object that reads rows from currEntry
  - makeColumnValueSelector
```
public ColumnValueSelector<?> makeColumnValueSelector(IncrementalIndexRowHolder currEntry,
                                                      IncrementalIndex.DimensionDesc desc)
```
    Description copied from interface: DimensionIndexer
    
    Return an object used to read values from this indexer's column.
    
    Specified by:
    
    makeColumnValueSelector in interface DimensionIndexer<StructuredData,StructuredData,StructuredData>
    
    Parameters:
    
    currEntry - Provides access to the current Row object in the Cursor
    
    desc - Descriptor object for this dimension within an IncrementalIndex
    
    Returns:
    
    A new object that reads rows from currEntry
  - getColumnCapabilities
```
public ColumnCapabilities getColumnCapabilities()
```
    Specified by:
    
    getColumnCapabilities in interface DimensionIndexer<StructuredData,StructuredData,StructuredData>
  - getLogicalType
```
public ColumnType getLogicalType()
```
  - isConstant
```
public boolean isConstant()
```
  - getConstantValue
```
@Nullable
public Object getConstantValue()
```
  - getFormat
```
public ColumnFormat getFormat()
```
    Specified by:
    
    getFormat in interface DimensionIndexer<StructuredData,StructuredData,StructuredData>
  - compareUnsortedEncodedKeyComponents
```
public int compareUnsortedEncodedKeyComponents(@Nullable
                                               StructuredData lhs,
                                               @Nullable
                                               StructuredData rhs)
```
    Description copied from interface: DimensionIndexer
    
    Compares the row values for this DimensionIndexer's dimension from a Row key. The dimension value arrays within a Row key always use the "unsorted" ordering for encoded values. The row values are passed to this function as an Object, the implementer should cast them to the type appropriate for this dimension. For example, a dictionary encoded String implementation would cast the Objects as int[] arrays. When comparing, if the two arrays have different lengths, the shorter array should be ordered first. Otherwise, the implementer of this function should iterate through the unsorted encoded values, converting them to their actual type (e.g., performing a dictionary lookup for a dict-encoded String dimension), and comparing the actual values until a difference is found. Refer to StringDimensionIndexer.compareUnsortedEncodedKeyComponents() for a reference implementation. The comparison rules used by this method should match the rules used by DimensionHandler.getEncodedValueSelectorComparator(), otherwise incorrect ordering/merging of rows can occur during ingestion, causing issues such as imperfect rollup.
    
    Specified by:
    
    compareUnsortedEncodedKeyComponents in interface DimensionIndexer<StructuredData,StructuredData,StructuredData>
    
    Parameters:
    
    lhs - dimension value array from a Row key
    
    rhs - dimension value array from a Row key
    
    Returns:
    
    comparison of the two arrays
  - checkUnsortedEncodedKeyComponentsEqual
```
public boolean checkUnsortedEncodedKeyComponentsEqual(@Nullable
                                                      StructuredData lhs,
                                                      @Nullable
                                                      StructuredData rhs)
```
    Description copied from interface: DimensionIndexer
    
    Check if two row value arrays from Row keys are equal.
    
    Specified by:
    
    checkUnsortedEncodedKeyComponentsEqual in interface DimensionIndexer<StructuredData,StructuredData,StructuredData>
    
    Parameters:
    
    lhs - dimension value array from a Row key
    
    rhs - dimension value array from a Row key
    
    Returns:
    
    true if the two arrays are equal
  - getUnsortedEncodedKeyComponentHashCode
```
public int getUnsortedEncodedKeyComponentHashCode(@Nullable
                                                  StructuredData key)
```
    Description copied from interface: DimensionIndexer
    
    Given a row value array from a Row key, generate a hashcode.
    
    Specified by:
    
    getUnsortedEncodedKeyComponentHashCode in interface DimensionIndexer<StructuredData,StructuredData,StructuredData>
    
    Parameters:
    
    key - dimension value array from a Row key
    
    Returns:
    
    hashcode of the array
  - convertUnsortedEncodedKeyComponentToActualList
```
public Object convertUnsortedEncodedKeyComponentToActualList(StructuredData key)
```
    Description copied from interface: DimensionIndexer
    
    Given a row value array from a Row key, as described in the documentation for DimensionIndexer.compareUnsortedEncodedKeyComponents(EncodedKeyComponentType, EncodedKeyComponentType), convert the unsorted encoded values to a list of actual values. If the key has one element, this method should return a single Object instead of a list.
    
    Specified by:
    
    convertUnsortedEncodedKeyComponentToActualList in interface DimensionIndexer<StructuredData,StructuredData,StructuredData>
    
    Parameters:
    
    key - dimension value array from a Row key
    
    Returns:
    
    single value or list containing the actual values corresponding to the encoded values in the input array
  - convertUnsortedValuesToSorted
```
public ColumnValueSelector convertUnsortedValuesToSorted(ColumnValueSelector selectorWithUnsortedValues)
```
    Description copied from interface: DimensionIndexer
    
    Converts dictionary-encoded row values from unspecified (random) encoding order, to sorted encoding. This step is needed to be able to correctly map per-segment encoded values to global values on the next conversion step, DimensionMerger.convertSortedSegmentRowValuesToMergedRowValues(int, org.apache.druid.segment.ColumnValueSelector). The latter method requires sorted encoding values on the input, because DimensionMerger.writeMergedValueDictionary(java.util.List<org.apache.druid.segment.IndexableAdapter>) takes sorted lookups as it's input. For columns which do not use the DimensionMerger to merge dictionary encoded values, this method should provide a selector which is compatible with the expectations of DimensionMerger.processMergedRow(ColumnValueSelector), which might simply be to pass-through the 'unsorted' selector.
    
    Specified by:
    
    convertUnsortedValuesToSorted in interface DimensionIndexer<StructuredData,StructuredData,StructuredData>
  - fillBitmapsFromUnsortedEncodedKeyComponent
```
public void fillBitmapsFromUnsortedEncodedKeyComponent(StructuredData key,
                                                       int rowNum,
                                                       MutableBitmap[] bitmapIndexes,
                                                       BitmapFactory factory)
```
    Description copied from interface: DimensionIndexer
    
    Helper function for building bitmap indexes for integer-encoded dimensions. Called by IncrementalIndexAdapter as it iterates through its sequence of rows. Given a row value array from a Row key, with the current row number indicated by "rowNum", set the index for "rowNum" in the bitmap index for each value that appears in the row value array. For example, if key is an int[] array with values [1,3,4] for a dictionary-encoded String dimension, and rowNum is 27, this function would set bit 27 in bitmapIndexes[1], bitmapIndexes[3], and bitmapIndexes[4] See StringDimensionIndexer.fillBitmapsFromUnsortedEncodedKeyComponent() for a reference implementation. If a dimension type does not support bitmap indexes, this function will not be called and can be left unimplemented.
    
    Specified by:
    
    fillBitmapsFromUnsortedEncodedKeyComponent in interface DimensionIndexer<StructuredData,StructuredData,StructuredData>
    
    Parameters:
    
    key - dimension value array from a Row key
    
    rowNum - current row number
    
    bitmapIndexes - array of bitmaps, indexed by integer dimension value
    
    factory - bitmap factory

Class AutoTypeColumnIndexer

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

hasNulls

hasNestedData

isConstant

constantValue

fieldIndexers

globalDictionary

indexerProcessor

Constructor Detail

AutoTypeColumnIndexer

Method Detail

processRowValsToUnsortedEncodedKeyComponent

setSparseIndexed

getUnsortedEncodedValueFromSorted

getSortedIndexedValues

getSortedValueLookups

getFieldTypeInfo

getMinValue

getMaxValue

getCardinality

makeDimensionSelector

makeColumnValueSelector

getColumnCapabilities

getLogicalType

isConstant

getConstantValue

getFormat

compareUnsortedEncodedKeyComponents

checkUnsortedEncodedKeyComponentsEqual

getUnsortedEncodedKeyComponentHashCode

convertUnsortedEncodedKeyComponentToActualList

convertUnsortedValuesToSorted

fillBitmapsFromUnsortedEncodedKeyComponent