com.twitter.algebird

CMS

sealed abstract class CMS[K] extends Serializable with CMSCounting[K, CMS]

A Count-Min sketch data structure that allows for counting and frequency estimation of elements in a data stream.

Tip: If you also need to track heavy hitters ("Top N" problems), take a look at TopCMS.

Usage

This example demonstrates how to count Long elements with CMS, i.e. K=Long.

Note that the actual counting is always performed with a Long, regardless of your choice of K. That is, the counting table behind the scenes is backed by Long values (at least in the current implementation), and thus the returned frequency estimates are always instances of Approximate[Long].

K

The type used to identify the elements to be counted.

Example:
  1. // Creates a monoid for a CMS that can count `Long` elements.
    val cmsMonoid: CMSMonoid[Long] = {
      val eps = 0.001
      val delta = 1E-10
      val seed = 1
      CMS.monoid[Long](eps, delta, seed)
    }
    
    // Creates a CMS instance that has counted the element `1L`.
    val cms: CMS[Long] = cmsMonoid.create(1L)
    
    // Estimates the frequency of `1L`
    val estimate: Approximate[Long] = cms.frequency(1L)
Linear Supertypes
CMSCounting[K, CMS], Serializable, AnyRef, Any
Known Subclasses
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. CMS
  2. CMSCounting
  3. Serializable
  4. AnyRef
  5. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Instance Constructors

  1. new CMS(params: CMSParams[K])

Abstract Value Members

  1. abstract def +(item: K, count: Long): CMS[K]

    Counts the item count times and returns the result as a new sketch.

    Counts the item count times and returns the result as a new sketch.

    Definition Classes
    CMSCounting
  2. abstract def ++(other: CMS[K]): CMS[K]

    Returns a new sketch that is the combination of this sketch and the other sketch.

    Returns a new sketch that is the combination of this sketch and the other sketch.

    Definition Classes
    CMSCounting
  3. abstract def frequency(item: K): Approximate[Long]

    Returns an estimate of the total number of times this item has been seen in the stream so far.

    Returns an estimate of the total number of times this item has been seen in the stream so far. This estimate is an upper bound.

    It is always true that estimatedFrequency >= trueFrequency. With probability p >= 1 - delta, it also holds that estimatedFrequency <= trueFrequency + eps * totalCount.

    Definition Classes
    CMSCounting
  4. abstract def innerProduct(other: CMS[K]): Approximate[Long]

    Returns an estimate of the inner product against another data stream.

    Returns an estimate of the inner product against another data stream.

    In other words, let a_i denote the number of times element i has been seen in the data stream summarized by this CMS, and let b_i denote the same for the other CMS. Then this returns an estimate of <a, b> = \sum a_i b_i.

    Note: This can also be viewed as the join size between two relations.

    It is always true that actualInnerProduct <= estimatedInnerProduct. With probability p >= 1 - delta, it also holds that estimatedInnerProduct <= actualInnerProduct + eps * thisTotalCount * otherTotalCount.

    Definition Classes
    CMSCounting
  5. abstract def totalCount: Long

    Total number of elements counted (i.

    Total number of elements counted (i.e. seen in the data stream) so far.

    Definition Classes
    CMSCounting

Concrete Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any
  3. final def ##(): Int

    Definition Classes
    AnyRef → Any
  4. def +(item: K): CMS[K]

    Counts the item and returns the result as a new sketch.

    Counts the item and returns the result as a new sketch.

    Definition Classes
    CMSCounting
  5. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  6. final def ==(arg0: Any): Boolean

    Definition Classes
    Any
  7. final def asInstanceOf[T0]: T0

    Definition Classes
    Any
  8. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  9. val delta: Double

    Returns the bound on the probability that a query estimate does NOT lie within some small interval (an interval that depends on eps) around the truth.

    Returns the bound on the probability that a query estimate does NOT lie within some small interval (an interval that depends on eps) around the truth.

    Definition Classes
    CMSCMSCounting
  10. def depth: Int

    Number of hash functions (also: number of rows in the counting table).

    Number of hash functions (also: number of rows in the counting table). This number is derived from delta.

    Definition Classes
    CMSCounting
  11. val eps: Double

    Returns the one-sided error bound on the error of each point query, i.

    Returns the one-sided error bound on the error of each point query, i.e. frequency estimate.

    Definition Classes
    CMSCMSCounting
  12. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  13. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  14. def f1: Long

    The first frequency moment is the total number of elements in the stream.

    The first frequency moment is the total number of elements in the stream.

    Definition Classes
    CMSCounting
  15. def f2: Approximate[Long]

    The second frequency moment is \sum a_i^2, where a_i is the count of the i-th element.

    The second frequency moment is \sum a_i^2, where a_i is the count of the i-th element.

    Definition Classes
    CMSCMSCounting
  16. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  17. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any
  18. def hashCode(): Int

    Definition Classes
    AnyRef → Any
  19. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any
  20. def maxExactCount: Int

    Number of exact counts a sparse CMS wants to keep.

    Number of exact counts a sparse CMS wants to keep. This number is derived from maxExactCountOpt.

    Definition Classes
    CMSCounting
  21. val maxExactCountOpt: Option[Int]

    An Option parameter about how many exact counts a sparse CMS wants to keep

    An Option parameter about how many exact counts a sparse CMS wants to keep

    Definition Classes
    CMSCMSCounting
  22. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  23. final def notify(): Unit

    Definition Classes
    AnyRef
  24. final def notifyAll(): Unit

    Definition Classes
    AnyRef
  25. val params: CMSParams[K]

  26. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef
  27. def toString(): String

    Definition Classes
    AnyRef → Any
  28. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  29. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  30. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  31. def width: Int

    Number of counters per hash function (also: number of columns in the counting table).

    Number of counters per hash function (also: number of columns in the counting table). This number is derived from eps.

    Definition Classes
    CMSCounting

Inherited from CMSCounting[K, CMS]

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped