Package org.apache.druid.segment

Class DictionaryEncodedColumnMerger<T extends Comparable<T>>

java.lang.Object

org.apache.druid.segment.DictionaryEncodedColumnMerger<T>

All Implemented Interfaces:

DimensionMerger, DimensionMergerV9

Direct Known Subclasses:

StringDimensionMergerV9

public abstract class DictionaryEncodedColumnMerger<T extends Comparable<T>>
extends Object
implements DimensionMergerV9

Base structure for merging dictionary encoded columns

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	DictionaryEncodedColumnMerger.ConvertingBitmapValues
protected static interface	DictionaryEncodedColumnMerger.ExtendedIndexesMerger	Specifies any additional per value indexes which should be constructed when writeIndexes(List) is called, on top of the standard bitmap index created with mergeBitmaps(java.util.List<java.nio.IntBuffer>, org.apache.druid.collections.bitmap.BitmapFactory, org.apache.druid.segment.DictionaryEncodedColumnMerger.IndexSeeker[], int)
protected static class	DictionaryEncodedColumnMerger.IdConversionSerializer
protected static interface	DictionaryEncodedColumnMerger.IndexSeeker
protected static class	DictionaryEncodedColumnMerger.IndexSeekerWithConversion	Get old dictId from new dictId, and only support access in order
protected static class	DictionaryEncodedColumnMerger.IndexSeekerWithoutConversion
protected static class	DictionaryEncodedColumnMerger.PersistedIdConversion	Peristent dictionary id conversion mappings, artifacts created during segment merge which map old dictionary ids to new dictionary ids.
protected static class	DictionaryEncodedColumnMerger.PersistedIdConversions	Closer of DictionaryEncodedColumnMerger.PersistedIdConversion and a parent path which they are stored in for easy cleanup when the segment is closed.

Field Summary

Fields

Modifier and Type	Field	Description
protected List<IndexableAdapter>	adapters
protected GenericIndexedWriter<ImmutableBitmap>	bitmapWriter
protected ColumnCapabilities	capabilities
protected int	cardinality
protected Closer	closer
protected DictionaryMergingIterator<T>	dictionaryMergeIterator
protected int	dictionarySize
protected DictionaryWriter<T>	dictionaryWriter
protected ArrayList<IntBuffer>	dimConversions
protected String	dimensionName
protected ColumnarIntsSerializer	encodedValueSerializer
protected T	firstDictionaryValue
protected boolean	hasNull
protected IndexSpec	indexSpec
protected MutableBitmap	nullRowsBitmap
protected String	outputName
protected @MonotonicNonNull DictionaryEncodedColumnMerger.PersistedIdConversions	persistedIdConversions	This becomes non-null if markAsParent() is called indicating that this column is a base table 'parent' to some projection column, which requires persisting id conversion buffers to a temporary files.
protected ProgressIndicator	progress
protected int	rowCount
protected File	segmentBaseDir
protected SegmentWriteOutMedium	segmentWriteOutMedium
protected boolean	writeDictionary

Constructor Summary

Constructors

Constructor	Description
DictionaryEncodedColumnMerger(String dimensionName, String outputName, IndexSpec indexSpec, SegmentWriteOutMedium segmentWriteOutMedium, ColumnCapabilities capabilities, ProgressIndicator progress, File segmentBaseDir, Closer closer)

Method Summary

Modifier and Type	Method	Description
protected abstract T	coerceValue(T value)
ColumnValueSelector	convertSortedSegmentRowValuesToMergedRowValues(int segmentIndex, ColumnValueSelector source)	Creates a value selector, which converts values with per-segment, _sorted order_ (see DimensionIndexer.convertUnsortedValuesToSorted(org.apache.druid.segment.ColumnValueSelector)) encoding from the given selector to their equivalent representation in the merged set of rows.
protected abstract Comparator<Pair<Integer,com.google.common.collect.PeekingIterator<T>>>	getDictionaryMergingComparator()
protected DictionaryEncodedColumnMerger.ExtendedIndexesMerger	getExtendedIndexesMerger()
protected abstract Indexed<T>	getNullDimValue()
protected abstract ObjectStrategy<T>	getObjectStrategy()
boolean	hasOnlyNulls()	Returns true if this dimension has no data besides nulls.
protected DictionaryWriter<T>	makeDictionaryWriter(String fileName)
void	markAsParent()	Sets this merger as the "parent" of another merger for a "projection", allowing for this merger to preserve any state which might be required for the projection mergers to do their thing.
protected MutableBitmap	mergeBitmaps(List<IntBuffer> segmentRowNumConversions, BitmapFactory bmpFactory, DictionaryEncodedColumnMerger.IndexSeeker[] dictIdSeeker, int dictId)
void	processMergedRow(ColumnValueSelector selector)	Process a column value(s) (potentially multi-value) of a row from the given selector and update the DimensionMerger's internal state.
protected void	setupEncodedValueWriter()
protected DictionaryEncodedColumnMerger.IndexSeeker[]	toIndexSeekers(List<IndexableAdapter> adapters, ArrayList<IntBuffer> dimConversions, String dimension)
protected void	writeDictionary(Iterable<T> dictionaryValues)
void	writeIndexes(List<IntBuffer> segmentRowNumConversions)	Internally construct any index structures relevant to this DimensionMerger.
void	writeMergedValueDictionary(List<IndexableAdapter> adapters)	Given a list of segment adapters: - Read _sorted order_ (e.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.druid.segment.DimensionMergerV9

attachParent, makeColumnDescriptor

Field Detail

dimensionName

protected final String dimensionName

outputName

protected final String outputName

progress

protected final ProgressIndicator progress

closer

protected final Closer closer

indexSpec

protected final IndexSpec indexSpec

segmentWriteOutMedium

protected final SegmentWriteOutMedium segmentWriteOutMedium

nullRowsBitmap

protected final MutableBitmap nullRowsBitmap

capabilities

protected final ColumnCapabilities capabilities

dictionarySize

protected int dictionarySize

rowCount

protected int rowCount

cardinality

protected int cardinality

hasNull

protected boolean hasNull

writeDictionary

protected boolean writeDictionary

bitmapWriter

@Nullable
protected GenericIndexedWriter<ImmutableBitmap> bitmapWriter

dimConversions

@Nullable
protected ArrayList<IntBuffer> dimConversions

adapters

@Nullable
protected List<IndexableAdapter> adapters

dictionaryMergeIterator

@Nullable
protected DictionaryMergingIterator<T extends Comparable<T>> dictionaryMergeIterator

encodedValueSerializer

@Nullable
protected ColumnarIntsSerializer encodedValueSerializer

dictionaryWriter

@Nullable
protected DictionaryWriter<T extends Comparable<T>> dictionaryWriter

firstDictionaryValue

@Nullable
protected T extends Comparable<T> firstDictionaryValue

segmentBaseDir

protected File segmentBaseDir

persistedIdConversions

protected @MonotonicNonNull DictionaryEncodedColumnMerger.PersistedIdConversions persistedIdConversions

This becomes non-null if markAsParent() is called indicating that this column is a base table 'parent' to some projection column, which requires persisting id conversion buffers to a temporary files. If there are no projections defined (or projections which reference this column) then id conversion buffers will be freed after calling writeIndexes(List)

Constructor Detail

DictionaryEncodedColumnMerger

public DictionaryEncodedColumnMerger(String dimensionName,
                                     String outputName,
                                     IndexSpec indexSpec,
                                     SegmentWriteOutMedium segmentWriteOutMedium,
                                     ColumnCapabilities capabilities,
                                     ProgressIndicator progress,
                                     File segmentBaseDir,
                                     Closer closer)

Method Detail

getDictionaryMergingComparator

protected abstract Comparator<Pair<Integer,com.google.common.collect.PeekingIterator<T>>> getDictionaryMergingComparator()

getNullDimValue

protected abstract Indexed<T> getNullDimValue()

getObjectStrategy

protected abstract ObjectStrategy<T> getObjectStrategy()

coerceValue

@Nullable
protected abstract T coerceValue(T value)

markAsParent

public void markAsParent()

Description copied from interface: DimensionMergerV9

Sets this merger as the "parent" of another merger for a "projection", allowing for this merger to preserve any state which might be required for the projection mergers to do their thing. This method MUST be called prior to performing any merge work. Typically, this method is only implemented if DimensionMergerV9.attachParent(DimensionMergerV9, List) requires it.

Specified by:

markAsParent in interface DimensionMergerV9

writeMergedValueDictionary

public void writeMergedValueDictionary(List<IndexableAdapter> adapters)
                                throws IOException

Description copied from interface: DimensionMerger

Given a list of segment adapters: - Read _sorted order_ (e. g. see IncrementalIndexAdapter.getDimValueLookup(String)) dictionary encoding information from the adapters - Merge those sorted order dictionary into a one big sorted order dictionary and write this merged dictionary. The implementer should maintain knowledge of the "index number" of the adapters in the input list, i.e., the position of each adapter in the input list. This "index number" will be used to refer to specific segments later in DimensionMerger.convertSortedSegmentRowValuesToMergedRowValues(int, org.apache.druid.segment.ColumnValueSelector).

Specified by:

writeMergedValueDictionary in interface DimensionMerger

Parameters:

adapters - List of adapters to be merged.

Throws:

IOException

See Also:

DimensionIndexer.convertUnsortedValuesToSorted(org.apache.druid.segment.ColumnValueSelector)

convertSortedSegmentRowValuesToMergedRowValues

public ColumnValueSelector convertSortedSegmentRowValuesToMergedRowValues(int segmentIndex,
                                                                          ColumnValueSelector source)

Description copied from interface: DimensionMerger

Creates a value selector, which converts values with per-segment, _sorted order_ (see DimensionIndexer.convertUnsortedValuesToSorted(org.apache.druid.segment.ColumnValueSelector)) encoding from the given selector to their equivalent representation in the merged set of rows. This method is used by the index merging process to build the merged sequence of rows. The implementing class is expected to use the merged value metadata constructed during DimensionMerger.writeMergedValueDictionary(List), if applicable. For example, an implementation of this function for a dictionary-encoded String column would convert the segment-specific, sorted order dictionary values within the row to the common merged dictionary values determined during DimensionMerger.writeMergedValueDictionary(List).

Specified by:

convertSortedSegmentRowValuesToMergedRowValues in interface DimensionMerger

Parameters:

segmentIndex - indicates which segment the row originated from, in the order established in DimensionMerger.writeMergedValueDictionary(List)

source - the selector from which to take values to convert

Returns:

a selector with converted values

processMergedRow

public void processMergedRow(ColumnValueSelector selector)
                      throws IOException

Description copied from interface: DimensionMerger

Process a column value(s) (potentially multi-value) of a row from the given selector and update the DimensionMerger's internal state. After constructing a merged sequence of rows across segments, the index merging process will iterate through these rows and on each iteration, for each column, pass the column value selector to the corresponding DimensionMerger. This allows each DimensionMerger to build its internal view of the sequence of merged rows, to be written out to a segment later.

Specified by:

processMergedRow in interface DimensionMerger

Throws:

IOException

writeIndexes

public void writeIndexes(@Nullable
                         List<IntBuffer> segmentRowNumConversions)
                  throws IOException

Description copied from interface: DimensionMerger

Internally construct any index structures relevant to this DimensionMerger. After receiving the sequence of merged rows via iterated DimensionMerger.processMergedRow(org.apache.druid.segment.ColumnValueSelector) calls, the DimensionMerger can now build any index structures it needs. For example, a dictionary encoded String implementation would create its bitmap indexes for the merged segment during this step. The index merger will provide a list of row number conversion IntBuffer objects. Each IntBuffer is associated with one of the segments being merged; the position of the IntBuffer in the list corresponds to the position of segment adapters within the input list of DimensionMerger.writeMergedValueDictionary(List). For example, suppose there are two segments A and B. Row 24 from segment A maps to row 99 in the merged sequence of rows, The IntBuffer for segment A would have a mapping of 24 -> 99.

Specified by:

writeIndexes in interface DimensionMerger

Parameters:

segmentRowNumConversions - A list of row number conversion IntBuffer objects.

Throws:

IOException

makeDictionaryWriter

protected DictionaryWriter<T> makeDictionaryWriter(String fileName)

getExtendedIndexesMerger

@Nullable
protected DictionaryEncodedColumnMerger.ExtendedIndexesMerger getExtendedIndexesMerger()

setupEncodedValueWriter

protected void setupEncodedValueWriter()
                                throws IOException

Throws:

IOException

writeDictionary

protected void writeDictionary(Iterable<T> dictionaryValues)
                        throws IOException

Throws:

IOException

mergeBitmaps

protected MutableBitmap mergeBitmaps(@Nullable
                                     List<IntBuffer> segmentRowNumConversions,
                                     BitmapFactory bmpFactory,
                                     DictionaryEncodedColumnMerger.IndexSeeker[] dictIdSeeker,
                                     int dictId)
                              throws IOException

Throws:

IOException

hasOnlyNulls

public boolean hasOnlyNulls()

Description copied from interface: DimensionMerger

Returns true if this dimension has no data besides nulls. See NullColumnPartSerde for how null-only columns are stored in the segment.

Specified by:

hasOnlyNulls in interface DimensionMerger

toIndexSeekers

protected DictionaryEncodedColumnMerger.IndexSeeker[] toIndexSeekers(List<IndexableAdapter> adapters,
                                                                     ArrayList<IntBuffer> dimConversions,
                                                                     String dimension)