Package io.github.jbellis.jvector.pq

Class LocallyAdaptiveVectorQuantization

java.lang.Object

io.github.jbellis.jvector.pq.LocallyAdaptiveVectorQuantization

All Implemented Interfaces:

VectorCompressor<LocallyAdaptiveVectorQuantization.QuantizedVector>

public class LocallyAdaptiveVectorQuantization
extends Object
implements VectorCompressor<LocallyAdaptiveVectorQuantization.QuantizedVector>

Implements Locally-Adaptive Vector Quantization (LVQ) as described in "Similarity search in the blink of an eye with compressed indices" (https://arxiv.org/abs/2304.04759). In particular, single-level LVQ-8 is used. To encode, a vector is first de-meaned by subtracting the global mean. Then, each component is quantized using a byte, where 0 = min and 255 = max. The bias and scale are stored for each component to allow for dequantization. Vectors are packed using Turbo LVQ as described in "Locally-Adaptive Quantization for Streaming Vector Search" (https://arxiv.org/pdf/2402.02044.pdf).

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	LocallyAdaptiveVectorQuantization.PackedVector	A Turbo LVQ vector that has been packed into a byte sequence.
static class	LocallyAdaptiveVectorQuantization.QuantizedVector	A LVQ-quantized vector.

Field Summary

Fields

Modifier and Type	Field	Description
final VectorFloat<?>	globalMean

Constructor Summary

Constructors

Constructor	Description
LocallyAdaptiveVectorQuantization(VectorFloat<?> globalMean)

Method Summary

Modifier and Type	Method	Description
int	compressedVectorSize()	the size of a compressed vector
int	compressorSize()	the size of the serialized compressor itself (NOT the size of compressed vectors)
static LocallyAdaptiveVectorQuantization	compute(RandomAccessVectorValues ravv)
CompressedVectors	createCompressedVectors(Object[] compressedVectors)
LocallyAdaptiveVectorQuantization.QuantizedVector	encode(VectorFloat<?> v)
LocallyAdaptiveVectorQuantization.QuantizedVector[]	encodeAll(RandomAccessVectorValues ravv, ForkJoinPool simdExecutor)
boolean	equals(Object o)
static LocallyAdaptiveVectorQuantization	load(RandomAccessReader in)
ScoreFunction.Reranker	scoreFunctionFrom(VectorFloat<?> query, VectorSimilarityFunction similarityFunction, LVQPackedVectors packedVectors)
void	write(DataOutput out, int version)

Methods inherited from class java.lang.Object

clone, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface io.github.jbellis.jvector.pq.VectorCompressor

encodeAll, encodeAll, write

Field Details

globalMean

public final VectorFloat<?> globalMean

Constructor Details

LocallyAdaptiveVectorQuantization

public LocallyAdaptiveVectorQuantization(VectorFloat<?> globalMean)

Method Details

compute

public static LocallyAdaptiveVectorQuantization compute(RandomAccessVectorValues ravv)

encodeAll

public LocallyAdaptiveVectorQuantization.QuantizedVector[] encodeAll(RandomAccessVectorValues ravv,
 ForkJoinPool simdExecutor)

Specified by:

encodeAll in interface VectorCompressor<LocallyAdaptiveVectorQuantization.QuantizedVector>

encode

public LocallyAdaptiveVectorQuantization.QuantizedVector encode(VectorFloat<?> v)

Specified by:

encode in interface VectorCompressor<LocallyAdaptiveVectorQuantization.QuantizedVector>

write

public void write(DataOutput out,
 int version)
           throws IOException

Specified by:

write in interface VectorCompressor<LocallyAdaptiveVectorQuantization.QuantizedVector>

Parameters:

out - DataOutput to write to

version - serialization version. Versions 2 and 3 are supported

Throws:

IOException

compressedVectorSize

public int compressedVectorSize()

Description copied from interface: VectorCompressor

the size of a compressed vector

Specified by:

compressedVectorSize in interface VectorCompressor<LocallyAdaptiveVectorQuantization.QuantizedVector>

compressorSize

public int compressorSize()

Description copied from interface: VectorCompressor

the size of the serialized compressor itself (NOT the size of compressed vectors)

Specified by:

compressorSize in interface VectorCompressor<LocallyAdaptiveVectorQuantization.QuantizedVector>

scoreFunctionFrom

public ScoreFunction.Reranker scoreFunctionFrom(VectorFloat<?> query,
 VectorSimilarityFunction similarityFunction,
 LVQPackedVectors packedVectors)

createCompressedVectors

public CompressedVectors createCompressedVectors(Object[] compressedVectors)

Specified by:

createCompressedVectors in interface VectorCompressor<LocallyAdaptiveVectorQuantization.QuantizedVector>

Parameters:

compressedVectors - must match the type T for this VectorCompressor, but it is declared as Object because we want callers to be able to use this without committing to a specific type T.

load

public static LocallyAdaptiveVectorQuantization load(RandomAccessReader in)
                                              throws IOException

Throws:

IOException

equals

public boolean equals(Object o)

Overrides:

equals in class Object