MinHasher
Related Docs:
object MinHasher
| package algebird
Decode two signatures into hash values, combine them somehow, and produce a new array
Decode two signatures into hash values, combine them somehow, and produce a new array
Initialize a byte array by generating hash values
Initialize a byte array by generating hash values
The number of bytes used for each hash in the signature
Maximum value the hash can take on (not 2*hashSize because of signed types)
These are from algebra.Monoid
Bucket keys to use for quickly finding other similar items via locality sensitive hashing
Useful for understanding the effects of numBands and numRows
Create a signature for an arbitrary value
Create a signature for a single String value
Create a signature for a single Long value
For explanation of the "bands" and "rows" see Ullman and Rajaraman
Set union
Set union
Useful for understanding the effects of numBands and numRows
Esimate Jaccard similarity (size of union / size of intersection)
Returns an instance of T calculated by summing all instances in
iter in one pass.
Returns an instance of T calculated by summing all instances in
iter in one pass. Returns None if iter is empty, else
Some[T].
instances of T to be combined
None if iter is empty, else an option value containing the summed T
Signature for empty set, needed to be a proper Monoid
Signature for empty set, needed to be a proper Monoid
Instances of MinHasher can create, combine, and compare fixed-sized signatures of arbitrarily sized sets.
A signature is represented by a byte array of approx maxBytes size. You can initialize a signature with a single element, usually a Long or String. You can combine any two set's signatures to produce the signature of their union. You can compare any two set's signatures to estimate their Jaccard similarity. You can use a set's signature to estimate the number of distinct values in the set. You can also use a combination of the above to estimate the size of the intersection of two sets from their signatures. The more bytes in the signature, the more accurate all of the above will be.
You can also use these signatures to quickly find similar sets without doing n^2 comparisons. Each signature is assigned to several buckets; sets whose signatures end up in the same bucket are likely to be similar. The targetThreshold controls the desired level of similarity - the higher the threshold, the more efficiently you can find all the similar sets.
This abstract superclass is generic with regards to the size of the hash used. Depending on the number of unique values in the domain of the sets, you may want a MinHasher16, a MinHasher32, or a new custom subclass.
This implementation is modeled after Chapter 3 of Ullman and Rajaraman's Mining of Massive Datasets: http://infolab.stanford.edu/~ullman/mmds/ch3a.pdf