Package com.arcadedb.index.vector

Class HnswVectorIndexRAM<TId,TVector,TItem extends com.github.jelmerk.knn.Item<TId,TVector>,TDistance>

java.lang.Object

com.arcadedb.index.vector.HnswVectorIndexRAM<TId,TVector,TItem,TDistance>

Type Parameters:

TId - Type of the external identifier of an item

TVector - Type of the vector to perform distance calculation on

TItem - Type of items stored in the index

TDistance - Type of distance between items (expect any numeric type: float, double, int, ..)

All Implemented Interfaces:

com.github.jelmerk.knn.Index<TId,TVector,TItem,TDistance>, Serializable

public class HnswVectorIndexRAM<TId,TVector,TItem extends com.github.jelmerk.knn.Item<TId,TVector>,TDistance>
extends Object
implements com.github.jelmerk.knn.Index<TId,TVector,TItem,TDistance>

This work is derived from the excellent work made by Jelmer Kuperus on https://github.com/jelmerk/hnswlib. We forked the entire class only because it was not extensible (private members).

Implementation of Index that implements the hnsw algorithm.

See Also:

Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs, Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	HnswVectorIndexRAM.Builder<TId,TVector,TItem extends com.github.jelmerk.knn.Item<TId,TVector>,TDistance>	Builder for initializing an HnswVectorIndexRAM instance.
class	HnswVectorIndexRAM.ItemIterator
static class	HnswVectorIndexRAM.Node<TItem extends com.github.jelmerk.knn.Item>

Field Summary

Fields

Modifier and Type	Field	Description
protected HnswVectorIndexRAM.Node<TItem>	entryPoint

Fields inherited from interface com.github.jelmerk.knn.Index

DEFAULT_PROGRESS_UPDATE_INTERVAL

Method Summary

Modifier and Type	Method	Description
boolean	add(TItem item)
VectorIndexBuilder	createPersistentIndex(Database database)
List<com.github.jelmerk.knn.SearchResult<TItem,TDistance>>	findNearest(TVector destination, int k)
Optional<TItem>	get(TId id)
int	getDimensions()	Returns the dimensionality of the items stored in this index.
Comparator<TDistance>	getDistanceComparator()	Returns the comparator used to compare distances.
com.github.jelmerk.knn.DistanceFunction<TVector,TDistance>	getDistanceFunction()	Returns the distance function.
int	getEf()	The size of the dynamic list for the nearest neighbors (used during the search)
int	getEfConstruction()	Returns the parameter has the same meaning as ef, but controls the index time / index precision.
Integer	getEntryPoint()
int	getM()	Returns the number of bi-directional links created for every new element during construction.
int	getMaxItemCount()	Returns the maximum number of items the index can hold.
Collection<TItem>	items()
HnswVectorIndexRAM.ItemIterator	iterateNodes()
static <TId,TVector,TItem extends com.github.jelmerk.knn.Item<TId,TVector>,TDistance> HnswVectorIndexRAM<TId,TVector,TItem,TDistance>	load(File file)	Restores a HnswVectorIndexRAM from a File.
static <TId,TVector,TItem extends com.github.jelmerk.knn.Item<TId,TVector>,TDistance> HnswVectorIndexRAM<TId,TVector,TItem,TDistance>	load(File file, ClassLoader classLoader)	Restores a HnswVectorIndexRAM from a File.
static <TId,TVector,TItem extends com.github.jelmerk.knn.Item<TId,TVector>,TDistance> HnswVectorIndexRAM<TId,TVector,TItem,TDistance>	load(InputStream inputStream)	Restores a HnswVectorIndexRAM from an InputStream.
static <TId,TVector,TItem extends com.github.jelmerk.knn.Item<TId,TVector>,TDistance> HnswVectorIndexRAM<TId,TVector,TItem,TDistance>	load(InputStream inputStream, ClassLoader classLoader)	Restores a HnswVectorIndexRAM from an InputStream.
static <TId,TVector,TItem extends com.github.jelmerk.knn.Item<TId,TVector>,TDistance> HnswVectorIndexRAM<TId,TVector,TItem,TDistance>	load(Path path)	Restores a HnswVectorIndexRAM from a Path.
static <TId,TVector,TItem extends com.github.jelmerk.knn.Item<TId,TVector>,TDistance> HnswVectorIndexRAM<TId,TVector,TItem,TDistance>	load(Path path, ClassLoader classLoader)	Restores a HnswVectorIndexRAM from a Path.
static <TId,TVector,TItem extends com.github.jelmerk.knn.Item<TId,TVector>,TDistance extends Comparable<TDistance>> HnswVectorIndexRAM.Builder<TId,TVector,TItem,TDistance>	newBuilder(int dimensions, com.github.jelmerk.knn.DistanceFunction<TVector,TDistance> distanceFunction, int maxItemCount)	Start the process of building a new HNSW index.
static <TId,TVector,TItem extends com.github.jelmerk.knn.Item<TId,TVector>,TDistance> HnswVectorIndexRAM.Builder<TId,TVector,TItem,TDistance>	newBuilder(int dimensions, com.github.jelmerk.knn.DistanceFunction<TVector,TDistance> distanceFunction, Comparator<TDistance> distanceComparator, int maxItemCount)	Start the process of building a new HNSW index.
boolean	remove(TId id, long version)
void	resize(int newSize)	Changes the maximum capacity of the index.
void	save(OutputStream out)
void	setEf(int ef)	Set the size of the dynamic list for the nearest neighbors (used during the search)
int	size()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.github.jelmerk.knn.Index

addAll, addAll, addAll, contains, findNeighbors, save, save

Field Detail

entryPoint

protected volatile HnswVectorIndexRAM.Node<TItem extends com.github.jelmerk.knn.Item<TId,TVector>> entryPoint

Method Detail

size

public int size()

Specified by:

size in interface com.github.jelmerk.knn.Index<TId,TVector,TItem extends com.github.jelmerk.knn.Item<TId,TVector>,TDistance>

get

public Optional<TItem> get(TId id)

Specified by:

get in interface com.github.jelmerk.knn.Index<TId,TVector,TItem extends com.github.jelmerk.knn.Item<TId,TVector>,TDistance>

items

public Collection<TItem> items()

Specified by:

items in interface com.github.jelmerk.knn.Index<TId,TVector,TItem extends com.github.jelmerk.knn.Item<TId,TVector>,TDistance>

iterateNodes

public HnswVectorIndexRAM.ItemIterator iterateNodes()

remove

public boolean remove(TId id,
                      long version)

Specified by:

remove in interface com.github.jelmerk.knn.Index<TId,TVector,TItem extends com.github.jelmerk.knn.Item<TId,TVector>,TDistance>

add

public boolean add(TItem item)

Specified by:

add in interface com.github.jelmerk.knn.Index<TId,TVector,TItem extends com.github.jelmerk.knn.Item<TId,TVector>,TDistance>

findNearest

public List<com.github.jelmerk.knn.SearchResult<TItem,TDistance>> findNearest(TVector destination,
                                                                                    int k)

Specified by:

findNearest in interface com.github.jelmerk.knn.Index<TId,TVector,TItem extends com.github.jelmerk.knn.Item<TId,TVector>,TDistance>

resize

public void resize(int newSize)

Changes the maximum capacity of the index.

Parameters:

newSize - new size of the index

getDimensions

public int getDimensions()

Returns the dimensionality of the items stored in this index.

Returns:

the dimensionality of the items stored in this index

getM

public int getM()

Returns the number of bi-directional links created for every new element during construction.

Returns:

the number of bi-directional links created for every new element during construction

getEf

public int getEf()

The size of the dynamic list for the nearest neighbors (used during the search)

Returns:

The size of the dynamic list for the nearest neighbors

setEf

public void setEf(int ef)

Set the size of the dynamic list for the nearest neighbors (used during the search)

Parameters:

ef - The size of the dynamic list for the nearest neighbors

getEfConstruction

public int getEfConstruction()

Returns the parameter has the same meaning as ef, but controls the index time / index precision.

Returns:

the parameter has the same meaning as ef, but controls the index time / index precision

getDistanceFunction

public com.github.jelmerk.knn.DistanceFunction<TVector,TDistance> getDistanceFunction()

Returns the distance function.

Returns:

the distance function

getDistanceComparator

public Comparator<TDistance> getDistanceComparator()

Returns the comparator used to compare distances.

Returns:

the comparator used to compare distance

getMaxItemCount

public int getMaxItemCount()

Returns the maximum number of items the index can hold.

Returns:

the maximum number of items the index can hold

save

public void save(OutputStream out)
          throws IOException

Specified by:

save in interface com.github.jelmerk.knn.Index<TId,TVector,TItem extends com.github.jelmerk.knn.Item<TId,TVector>,TDistance>

Throws:

IOException

load

public static <TId,TVector,TItem extends com.github.jelmerk.knn.Item<TId,TVector>,TDistance> HnswVectorIndexRAM<TId,TVector,TItem,TDistance> load(File file)
                                                                                                                                                                                     throws IOException

Restores a HnswVectorIndexRAM from a File.

Type Parameters:

TId - Type of the external identifier of an item

TVector - Type of the vector to perform distance calculation on

TItem - Type of items stored in the index

TDistance - Type of distance between items (expect any numeric type: float, double, int, ..)

Parameters:

file - File to restore the index from

Returns:

The restored index

Throws:

IOException - in case of an I/O exception

load

public static <TId,TVector,TItem extends com.github.jelmerk.knn.Item<TId,TVector>,TDistance> HnswVectorIndexRAM<TId,TVector,TItem,TDistance> load(File file,
                                                                                                                                                                                            ClassLoader classLoader)
                                                                                                                                                                                     throws IOException

Restores a HnswVectorIndexRAM from a File.

Type Parameters:

TId - Type of the external identifier of an item

TVector - Type of the vector to perform distance calculation on

TItem - Type of items stored in the index

TDistance - Type of distance between items (expect any numeric type: float, double, int, ..)

Parameters:

file - File to restore the index from

classLoader - the classloader to use

Returns:

The restored index

Throws:

IOException - in case of an I/O exception

load

public static <TId,TVector,TItem extends com.github.jelmerk.knn.Item<TId,TVector>,TDistance> HnswVectorIndexRAM<TId,TVector,TItem,TDistance> load(Path path)
                                                                                                                                                                                     throws IOException

Restores a HnswVectorIndexRAM from a Path.

Type Parameters:

TId - Type of the external identifier of an item

TVector - Type of the vector to perform distance calculation on

TItem - Type of items stored in the index

TDistance - Type of distance between items (expect any numeric type: float, double, int, ..)

Parameters:

path - Path to restore the index from

Returns:

The restored index

Throws:

IOException - in case of an I/O exception

load

public static <TId,TVector,TItem extends com.github.jelmerk.knn.Item<TId,TVector>,TDistance> HnswVectorIndexRAM<TId,TVector,TItem,TDistance> load(Path path,
                                                                                                                                                                                            ClassLoader classLoader)
                                                                                                                                                                                     throws IOException

Restores a HnswVectorIndexRAM from a Path.

Type Parameters:

TId - Type of the external identifier of an item

TVector - Type of the vector to perform distance calculation on

TItem - Type of items stored in the index

TDistance - Type of distance between items (expect any numeric type: float, double, int, ..)

Parameters:

path - Path to restore the index from

classLoader - the classloader to use

Returns:

The restored index

Throws:

IOException - in case of an I/O exception

load

public static <TId,TVector,TItem extends com.github.jelmerk.knn.Item<TId,TVector>,TDistance> HnswVectorIndexRAM<TId,TVector,TItem,TDistance> load(InputStream inputStream)
                                                                                                                                                                                     throws IOException

Restores a HnswVectorIndexRAM from an InputStream.

Type Parameters:

TId - Type of the external identifier of an item

TVector - Type of the vector to perform distance calculation on

TItem - Type of items stored in the index

TDistance - Type of distance between items (expect any numeric type: float, double, int, ...).

Parameters:

inputStream - InputStream to restore the index from

Returns:

The restored index

Throws:

IOException - in case of an I/O exception

IllegalArgumentException - in case the file cannot be read

load

public static <TId,TVector,TItem extends com.github.jelmerk.knn.Item<TId,TVector>,TDistance> HnswVectorIndexRAM<TId,TVector,TItem,TDistance> load(InputStream inputStream,
                                                                                                                                                                                            ClassLoader classLoader)
                                                                                                                                                                                     throws IOException

Restores a HnswVectorIndexRAM from an InputStream.

Type Parameters:

TId - Type of the external identifier of an item

TVector - Type of the vector to perform distance calculation on

TItem - Type of items stored in the index

TDistance - Type of distance between items (expect any numeric type: float, double, int, ...).

Parameters:

inputStream - InputStream to restore the index from

classLoader - the classloader to use

Returns:

The restored index

Throws:

IOException - in case of an I/O exception

IllegalArgumentException - in case the file cannot be read

newBuilder

public static <TId,TVector,TItem extends com.github.jelmerk.knn.Item<TId,TVector>,TDistance extends Comparable<TDistance>> HnswVectorIndexRAM.Builder<TId,TVector,TItem,TDistance> newBuilder(int dimensions,
                                                                                                                                                                                                                                        com.github.jelmerk.knn.DistanceFunction<TVector,TDistance> distanceFunction,
                                                                                                                                                                                                                                        int maxItemCount)

Start the process of building a new HNSW index.

Type Parameters:

TVector - Type of the vector to perform distance calculation on

TDistance - Type of distance between items (expect any numeric type: float, double, int, ..)

Parameters:

dimensions - the dimensionality of the vectors stored in the index

distanceFunction - the distance function

maxItemCount - maximum number of items the index can hold

Returns:

a builder

newBuilder

public static <TId,TVector,TItem extends com.github.jelmerk.knn.Item<TId,TVector>,TDistance> HnswVectorIndexRAM.Builder<TId,TVector,TItem,TDistance> newBuilder(int dimensions,
                                                                                                                                                                                                          com.github.jelmerk.knn.DistanceFunction<TVector,TDistance> distanceFunction,
                                                                                                                                                                                                          Comparator<TDistance> distanceComparator,
                                                                                                                                                                                                          int maxItemCount)

Start the process of building a new HNSW index.

Type Parameters:

TVector - Type of the vector to perform distance calculation on

TDistance - Type of distance between items (expect any numeric type: float, double, int, ..)

Parameters:

dimensions - the dimensionality of the vectors stored in the index

distanceFunction - the distance function

distanceComparator - used to compare distances

maxItemCount - maximum number of items the index can hold

Returns:

a builder

createPersistentIndex

public VectorIndexBuilder createPersistentIndex(Database database)

getEntryPoint

public Integer getEntryPoint()