lsh

Type Members

trait Joiner extends AnyRef
class Lsh extends Joiner with Serializable

Lsh implementation as described in 'Randomized Algorithms and NLP: Using Locality Sensitive Hash Function for High Speed Noun Clustering' by Ravichandran et al.
class NearestNeighbours extends Joiner with Serializable

Brute force O(n^{2) method to compute exact nearest neighbours.
As this is a very expensive computation O(n}2) an additional sample parameter may be passed such that neighbours are just computed for a random fraction.
class QueryHamming extends QueryJoiner with Serializable

Implementation based on approximated cosine distances.
trait QueryJoiner extends AnyRef
class QueryLsh extends QueryJoiner with Serializable

Standard Lsh implementation.
class QueryNearestNeighbours extends QueryJoiner with Serializable

Brute force O(size(query) * size(catalog)) method to compute exact nearest neighbours for rows in the query matrix.
final case class Signature(index: Long, vector: Vector, bitSet: BitSet) extends Ordered[Signature] with Product with Serializable

An id with it's hash encoding and original vector.
class SlidingRDD[T] extends RDD[Seq[T]]

Represents a RDD from grouping items of its parent RDD in fixed size blocks by passing a sliding window over them.
class SlidingRDDPartition[T] extends Partition with Serializable

NOTE: both classes are copied from mllib and slightly modified since these classes are mllib private!
final case class SparseSignature(index: Long, bitSet: BitSet) extends Ordered[SparseSignature] with Product with Serializable

An id with it's hash encoding.
trait VectorDistance extends Serializable

interface defining similarity measurement between 2 vectors

Value Members

object Cosine extends VectorDistance

implementation of VectorDistance that computes cosine similarity between two vectors
object Main
def bitSetComparator(a: BitSet, b: BitSet): Int

Compares two bit sets according to the first different bit
def bitSetIsEqual(vec1: BitSet, vec2: BitSet): Boolean

Compares two bit sets for their equality
def bitSetToString(bs: BitSet): String

Returns a string representation of a BitSet
def distinct(matrix: RDD[MatrixEntry]): RDD[MatrixEntry]

Take distinct matrix entry values based on the indices only.
Take distinct matrix entry values based on the indices only. The actual values are discarded.
def hamming(vec1: BitSet, vec2: BitSet): Int

Returns the hamming distance between two bit vectors
def hammingToCosine(hammingDistance: Int, d: Double): Double

Approximates the cosine distance of two bit sets using their hamming distance
def localRandomMatrix(d: Int, numFeatures: Int): Matrix

Returns a local k by d matrix with random gaussian entries mean=0.
Returns a local k by d matrix with random gaussian entries mean=0.0 and std=1.0
This is a k by d matrix as it is multiplied by the input matrix
def matrixToBitSet(inputMatrix: IndexedRowMatrix, localRandomMatrix: Matrix): RDD[Signature]

Converts a given input matrix to a bit set representation using random hyperplanes
def matrixToBitSetSparse(inputMatrix: IndexedRowMatrix, localRandomMatrix: Matrix): RDD[SparseSignature]

Converts a given input matrix to a bit set representation using random hyperplanes
def vectorToBitSet(vector: Vector): BitSet

Converts a vector to a bit set by replacing all values of x with sign(x)

package lsh

Type Members

trait Joiner extends AnyRef

class Lsh extends Joiner with Serializable

class NearestNeighbours extends Joiner with Serializable

class QueryHamming extends QueryJoiner with Serializable

trait QueryJoiner extends AnyRef

class QueryLsh extends QueryJoiner with Serializable

class QueryNearestNeighbours extends QueryJoiner with Serializable

final case class Signature(index: Long, vector: Vector, bitSet: BitSet) extends Ordered[Signature] with Product with Serializable

class SlidingRDD[T] extends RDD[Seq[T]]

class SlidingRDDPartition[T] extends Partition with Serializable

final case class SparseSignature(index: Long, bitSet: BitSet) extends Ordered[SparseSignature] with Product with Serializable

trait VectorDistance extends Serializable

Value Members

object Cosine extends VectorDistance

object Main

def bitSetComparator(a: BitSet, b: BitSet): Int

def bitSetIsEqual(vec1: BitSet, vec2: BitSet): Boolean

def bitSetToString(bs: BitSet): String

def distinct(matrix: RDD[MatrixEntry]): RDD[MatrixEntry]

def hamming(vec1: BitSet, vec2: BitSet): Int

def hammingToCosine(hammingDistance: Int, d: Double): Double

def localRandomMatrix(d: Int, numFeatures: Int): Matrix

def matrixToBitSet(inputMatrix: IndexedRowMatrix, localRandomMatrix: Matrix): RDD[Signature]

def matrixToBitSetSparse(inputMatrix: IndexedRowMatrix, localRandomMatrix: Matrix): RDD[SparseSignature]

def vectorToBitSet(vector: Vector): BitSet

Inherited from AnyRef

Inherited from Any

Ungrouped