com.kamikaze.pfordelta
Class LCPForDelta

java.lang.Object
  com.kamikaze.pfordelta.LCPForDelta

public class LCPForDelta
extends Object

Implementation of the optimized PForDelta algorithm for sorted integer arrays. The basic ideas are based on 1. Original algorithm from http://homepages.cwi.nl/~heman/downloads/msthesis.pdf 2. Optimization and variation from http://www2008.org/papers/pdf/p387-zhangA.pdf 3. Further optimization http://www2009.org/proceedings/pdf/p401.pdf As a part of the PForDelta implementation, Simple16 is used to compress exceptions. The original Simple16 algorithm can also be found in the above literature. This implementation overcomes the problem that Simple16 cannot deal with >2^28 numbers. This implementation is almost same as PForDelta in the same package, except that it is tuned especially for Lucene-4.0 Codec to achieve the best performance in Lucene-4.0.

Author:

hao yan, [email protected]

Constructor Summary

LCPForDelta()

Method Summary

static boolean	checkBigNumbers(int[] inputBlock, int blockSize, int bits) Check if the block contains big numbers that is greater than ((1<< bits)-1)
int	compress(int[] inBlock, int blockSize) Compress one block of blockSize integers using PForDelta with the optimal parameter b
int	compressOneBlockCore(int[] inputBlock, int blockSize, int bits) The core implementation of compressing a block with blockSize integers using PForDelta with the given parameter b
int	compressOneBlockCore2(int[] inputBlock, int blockSize, int bits)
static int	decompressBBitSlots(int[] outDecompSlots, int[] inCompBlock, int blockSize, int bits) Decompress b-bit slots
static int	decompressBBitSlotsWithHardCodes(int[] outDecompSlots, int[] inCompBlock, int blockSize, int bits)
static int	decompressBBitSlotsWithHardCodesWithIntBuffer(int[] outDecompSlots, IntBuffer inCompBlock, int blockSize, int bits)
static int	decompressBlockByS16(int[] outDecompBlock, int[] inCompBlock, int inStartOffsetInBits, int blockSize) Decompress a block of blockSize integers using Simple16 algorithm
static void	decompressBlockByS16WithIntBuffer(int[] outDecompBlock, IntBuffer inCompBlock, int blockSize)
static void	decompressBlockByS16WithIntBufferIntegrated(int[] outDecompBlock, IntBuffer inCompBlock, int blockSize, int[] expPosBuffer, int oribits)
static void	decompressOneBlock(int[] decompBlock, int[] inBlock, int blockSize) Decompress one block using PForDelta
static void	decompressOneBlockWithSize(int[] decompBlock, int[] inBlock, int blockSize, int[] expPosBuffer, int[] expHighBitsBuffer, int inBlockLen)
static void	decompressOneBlockWithSizeWithIntBuffer(int[] decompBlock, IntBuffer inBlock, int blockSize, int[] expPosBuffer, int[] expHighBitsBuffer, int inBlockLen)
static int	estimateCompressedSize(int[] inputBlock, int blockSize, int bits) Estimate the compressed size in ints of a block
int[]	getCompBuffer()
static int	readBits(int[] in, int inOffset, int bits) Read a certain number of bits of an integer into an integer array starting from the given start offset
static int	readBitsWithBuffer(int[] in, int inOffset, int bits)
void	setCompBuffer(int[] buffer)
static void	writeBits(int[] out, int val, int outOffset, int bits) Write a certain number of bits of an integer into an integer array starting from the given start offset

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

LCPForDelta

public LCPForDelta()

Method Detail

getCompBuffer

public int[] getCompBuffer()

setCompBuffer

public void setCompBuffer(int[] buffer)

compress

public int compress(int[] inBlock,
                    int blockSize)

Compress one block of blockSize integers using PForDelta with the optimal parameter b

Parameters:

inBlock - the block to be compressed

blockSize - the block size

Returns:

the compressed block

compressOneBlockCore

public int compressOneBlockCore(int[] inputBlock,
                                int blockSize,
                                int bits)
                         throws IllegalArgumentException

The core implementation of compressing a block with blockSize integers using PForDelta with the given parameter b

Parameters:

inputBlock - the block to be compressed

blockSize - the block size

bits - the the value of the parameter b

Returns:

the size of the compressed block (the number of ints)

Throws:

IllegalArgumentException

compressOneBlockCore2

public int compressOneBlockCore2(int[] inputBlock,
                                 int blockSize,
                                 int bits)
                          throws IllegalArgumentException

Throws:

IllegalArgumentException

decompressOneBlock

public static void decompressOneBlock(int[] decompBlock,
                                      int[] inBlock,
                                      int blockSize)

Decompress one block using PForDelta

Parameters:

decompBlock - the block that was decompressed

inBlock - the block to be decompressed

blockSize - the number of elements in the decompressed block

decompressOneBlockWithSize

public static void decompressOneBlockWithSize(int[] decompBlock,
                                              int[] inBlock,
                                              int blockSize,
                                              int[] expPosBuffer,
                                              int[] expHighBitsBuffer,
                                              int inBlockLen)

decompressOneBlockWithSizeWithIntBuffer

public static void decompressOneBlockWithSizeWithIntBuffer(int[] decompBlock,
                                                           IntBuffer inBlock,
                                                           int blockSize,
                                                           int[] expPosBuffer,
                                                           int[] expHighBitsBuffer,
                                                           int inBlockLen)

estimateCompressedSize

public static int estimateCompressedSize(int[] inputBlock,
                                         int blockSize,
                                         int bits)
                                  throws IllegalArgumentException

Estimate the compressed size in ints of a block

Parameters:

inputBlock - the block to be compressed

bits - the value of the parameter b

blockSize - the block size

Returns:

the compressed size in ints

Throws:

IllegalArgumentException

checkBigNumbers

public static boolean checkBigNumbers(int[] inputBlock,
                                      int blockSize,
                                      int bits)
                               throws IllegalArgumentException

Check if the block contains big numbers that is greater than ((1<< bits)-1)

Parameters:

inputBlock - the block to be compressed

bits - the numbers of bits to decide whether a number is a big number

blockSize - the block size

Returns:

true if there is any big numbers in the block

Throws:

IllegalArgumentException

decompressBBitSlots

public static int decompressBBitSlots(int[] outDecompSlots,
                                      int[] inCompBlock,
                                      int blockSize,
                                      int bits)

Decompress b-bit slots

Parameters:

outDecompSlots - decompressed block which is the output

inCompBlock - the compressed block which is the input

blockSize - the block size

bits - the value of the parameter b

Returns:

the compressed size in bits of the data that has been decompressed

decompressBBitSlotsWithHardCodes

public static int decompressBBitSlotsWithHardCodes(int[] outDecompSlots,
                                                   int[] inCompBlock,
                                                   int blockSize,
                                                   int bits)

decompressBBitSlotsWithHardCodesWithIntBuffer

public static int decompressBBitSlotsWithHardCodesWithIntBuffer(int[] outDecompSlots,
                                                                IntBuffer inCompBlock,
                                                                int blockSize,
                                                                int bits)

decompressBlockByS16

public static int decompressBlockByS16(int[] outDecompBlock,
                                       int[] inCompBlock,
                                       int inStartOffsetInBits,
                                       int blockSize)

Decompress a block of blockSize integers using Simple16 algorithm

Parameters:

outDecompBlock - the decompressed block which is the output

inCompBlock - the compressed block which is the input

blockSize - the block size

inStartOffsetInBits - the start offset in bits of the compressed block

Returns:

the compressed size in bits of the data that has been decompressed

decompressBlockByS16WithIntBuffer

public static void decompressBlockByS16WithIntBuffer(int[] outDecompBlock,
                                                     IntBuffer inCompBlock,
                                                     int blockSize)

decompressBlockByS16WithIntBufferIntegrated

public static void decompressBlockByS16WithIntBufferIntegrated(int[] outDecompBlock,
                                                               IntBuffer inCompBlock,
                                                               int blockSize,
                                                               int[] expPosBuffer,
                                                               int oribits)

writeBits

public static final void writeBits(int[] out,
                                   int val,
                                   int outOffset,
                                   int bits)

Write a certain number of bits of an integer into an integer array starting from the given start offset

Parameters:

out - the output array

val - the integer to be written

outOffset - the start offset in bits in the output array

bits - the number of bits to be written (bits>=0)

readBits

public static final int readBits(int[] in,
                                 int inOffset,
                                 int bits)

Read a certain number of bits of an integer into an integer array starting from the given start offset

Parameters:

in - the input array

inOffset - the start offset in bits in the input array

bits - the number of bits to be read, unlike writeBits(), readBits() does not deal with bits==0 and thus bits must > 0. When bits ==0, the calling functions will just skip the entire bits-bit slots without decoding them

Returns:

the bits bits of the input

readBitsWithBuffer

public static final int readBitsWithBuffer(int[] in,
                                           int inOffset,
                                           int bits)