Package org.apache.druid.segment.data

Class CompressionFactory

java.lang.Object

org.apache.druid.segment.data.CompressionFactory

public class CompressionFactory
extends Object

Compression of metrics is done by using a combination of CompressionStrategy and Encoding(such as CompressionFactory.LongEncodingStrategy for type Long). CompressionStrategy is unaware of the data type and is based on byte operations. It must compress and decompress in block of bytes. Encoding refers to compression method relies on data format, so a different set of Encodings exist for each data type.

Storage Format : Byte 1 : version (currently 0x02) Byte 2 - 5 : number of values Byte 6 - 9 : size per block (even if block format isn't used, this is needed for backward compatibility) Byte 10 : compression strategy (contains a flag if there's an encoding byte, see below for how the flag is defined) Byte 11(optional) : encoding type

Encoding specific header (described below)

Block related header (if block compression is used, described in GenericIndexed)

Values

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	CompressionFactory.LongEncodingFormat
static interface	CompressionFactory.LongEncodingReader
static class	CompressionFactory.LongEncodingStrategy	The compression of decompression of encodings are separated into different enums.
static interface	CompressionFactory.LongEncodingWriter	This writer output encoded values to the given ByteBuffer or OutputStream.

Field Summary

Fields

Modifier and Type	Field	Description
static CompressionFactory.LongEncodingStrategy	DEFAULT_LONG_ENCODING_STRATEGY
static byte	DELTA_ENCODING_VERSION	Delta Encoding Header v1: Byte 1 : version Byte 2 - 9 : base value Byte 10 - 13 : number of bits per value
static CompressionFactory.LongEncodingFormat	LEGACY_LONG_ENCODING_FORMAT
static int	MAX_TABLE_SIZE
static byte	TABLE_ENCODING_VERSION	Table Encoding Header v1 : Byte 1 : version Byte 2 - 5 : table size Byte 6 - (6 + 8 * table size - 1) : table of encoding, where the ith 8-byte value is encoded as i

Method Summary

Modifier and Type	Method	Description
static byte	clearEncodingFlag(byte strategyId)
static ColumnarDoublesSerializer	getDoubleSerializer(String columnName, SegmentWriteOutMedium segmentWriteOutMedium, String filenameBase, ByteOrder byteOrder, CompressionStrategy compression)
static com.google.common.base.Supplier<ColumnarDoubles>	getDoubleSupplier(int totalSize, int sizePer, ByteBuffer fromBuffer, ByteOrder byteOrder, CompressionStrategy strategy)
static ColumnarFloatsSerializer	getFloatSerializer(String columnName, SegmentWriteOutMedium segmentWriteOutMedium, String filenameBase, ByteOrder order, CompressionStrategy compressionStrategy)
static com.google.common.base.Supplier<ColumnarFloats>	getFloatSupplier(int totalSize, int sizePer, ByteBuffer fromBuffer, ByteOrder order, CompressionStrategy strategy)
static ColumnarLongsSerializer	getLongSerializer(String columnName, SegmentWriteOutMedium segmentWriteOutMedium, String filenameBase, ByteOrder order, CompressionFactory.LongEncodingStrategy encodingStrategy, CompressionStrategy compressionStrategy)
static com.google.common.base.Supplier<ColumnarLongs>	getLongSupplier(int totalSize, int sizePer, ByteBuffer fromBuffer, ByteOrder order, CompressionFactory.LongEncodingFormat encodingFormat, CompressionStrategy strategy)
static boolean	hasEncodingFlag(byte strategyId)
static byte	setEncodingFlag(byte strategyId)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_LONG_ENCODING_STRATEGY

public static final CompressionFactory.LongEncodingStrategy DEFAULT_LONG_ENCODING_STRATEGY

LEGACY_LONG_ENCODING_FORMAT

public static final CompressionFactory.LongEncodingFormat LEGACY_LONG_ENCODING_FORMAT

DELTA_ENCODING_VERSION

public static final byte DELTA_ENCODING_VERSION

Delta Encoding Header v1: Byte 1 : version Byte 2 - 9 : base value Byte 10 - 13 : number of bits per value

See Also:

Constant Field Values

TABLE_ENCODING_VERSION

public static final byte TABLE_ENCODING_VERSION

Table Encoding Header v1 : Byte 1 : version Byte 2 - 5 : table size Byte 6 - (6 + 8 * table size - 1) : table of encoding, where the ith 8-byte value is encoded as i

See Also:

Constant Field Values

MAX_TABLE_SIZE

public static final int MAX_TABLE_SIZE

See Also:

Constant Field Values

Method Detail

hasEncodingFlag

public static boolean hasEncodingFlag(byte strategyId)

setEncodingFlag

public static byte setEncodingFlag(byte strategyId)

clearEncodingFlag

public static byte clearEncodingFlag(byte strategyId)

getLongSupplier

public static com.google.common.base.Supplier<ColumnarLongs> getLongSupplier(int totalSize,
                                                                             int sizePer,
                                                                             ByteBuffer fromBuffer,
                                                                             ByteOrder order,
                                                                             CompressionFactory.LongEncodingFormat encodingFormat,
                                                                             CompressionStrategy strategy)

getLongSerializer

public static ColumnarLongsSerializer getLongSerializer(String columnName,
                                                        SegmentWriteOutMedium segmentWriteOutMedium,
                                                        String filenameBase,
                                                        ByteOrder order,
                                                        CompressionFactory.LongEncodingStrategy encodingStrategy,
                                                        CompressionStrategy compressionStrategy)

getFloatSupplier

public static com.google.common.base.Supplier<ColumnarFloats> getFloatSupplier(int totalSize,
                                                                               int sizePer,
                                                                               ByteBuffer fromBuffer,
                                                                               ByteOrder order,
                                                                               CompressionStrategy strategy)

getFloatSerializer

public static ColumnarFloatsSerializer getFloatSerializer(String columnName,
                                                          SegmentWriteOutMedium segmentWriteOutMedium,
                                                          String filenameBase,
                                                          ByteOrder order,
                                                          CompressionStrategy compressionStrategy)

getDoubleSupplier

public static com.google.common.base.Supplier<ColumnarDoubles> getDoubleSupplier(int totalSize,
                                                                                 int sizePer,
                                                                                 ByteBuffer fromBuffer,
                                                                                 ByteOrder byteOrder,
                                                                                 CompressionStrategy strategy)

getDoubleSerializer

public static ColumnarDoublesSerializer getDoubleSerializer(String columnName,
                                                            SegmentWriteOutMedium segmentWriteOutMedium,
                                                            String filenameBase,
                                                            ByteOrder byteOrder,
                                                            CompressionStrategy compression)