com.twitter.algebird

Aggregator

object Aggregator extends Serializable

Aggregators compose well.

To create a parallel aggregator that operates on a single input in parallel, use: GeneratedTupleAggregator.from2((agg1, agg2))

Linear Supertypes
Serializable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. Aggregator
  2. Serializable
  3. AnyRef
  4. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any

  3. final def ##(): Int

    Definition Classes
    AnyRef → Any

  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any

  6. def appendMonoid[F, T, P](appnd: (T, F) ⇒ T, pres: (T) ⇒ P)(implicit m: Monoid[T]): MonoidAggregator[F, T, P]

    Obtain a MonoidAggregator that uses an efficient append operation for faster aggregation

    Obtain a MonoidAggregator that uses an efficient append operation for faster aggregation

    F

    Data input type

    T

    Aggregating Monoid type

    P

    Presentation (output) type

    appnd

    Function that appends the Monoid. Defines the MonoidAggregator.append method for this aggregator. Analogous to the 'seqop' function in Scala's sequence 'aggregate' method

    pres

    The presentation function

    m

    The Monoid type class

    Note

    The function 'appnd' is expected to obey the law: 

    appnd(t, f) == m.plus(t, appnd(m.zero, f))

  7. def appendMonoid[F, T](appnd: (T, F) ⇒ T)(implicit m: Monoid[T]): MonoidAggregator[F, T, T]

    Obtain a MonoidAggregator that uses an efficient append operation for faster aggregation.

    Obtain a MonoidAggregator that uses an efficient append operation for faster aggregation. Equivalent to 

    appendMonoid(appnd, identity[T]_)(m)

  8. def appendSemigroup[F, T, P](prep: (F) ⇒ T, appnd: (T, F) ⇒ T, pres: (T) ⇒ P)(implicit sg: Semigroup[T]): Aggregator[F, T, P]

    Obtain an Aggregator that uses an efficient append operation for faster aggregation

    Obtain an Aggregator that uses an efficient append operation for faster aggregation

    F

    Data input type

    T

    Aggregating Semigroup type

    P

    Presentation (output) type

    prep

    The preparation function. Expected to construct an instance of type T from a single data element.

    appnd

    Function that appends the Semigroup. Defines the Aggregator.append method for this aggregator. Analogous to the 'seqop' function in Scala's sequence 'aggregate' method

    pres

    The presentation function

    sg

    The Semigroup type class

    Note

    The functions 'appnd' and 'prep' are expected to obey the law: 

    appnd(t, f) == sg.plus(t, prep(f))

  9. def appendSemigroup[F, T](prep: (F) ⇒ T, appnd: (T, F) ⇒ T)(implicit sg: Semigroup[T]): Aggregator[F, T, T]

    Obtain an Aggregator that uses an efficient append operation for faster aggregation.

    Obtain an Aggregator that uses an efficient append operation for faster aggregation. Equivalent to 

    appendSemigroup(prep, appnd, identity[T]_)(sg)

  10. implicit def applicative[I]: Applicative[[O]Aggregator[I, _, O]]

  11. def approximatePercentile[T](percentile: Double, k: Int = QTreeAggregator.DefaultK)(implicit num: Numeric[T]): QTreeAggregatorLowerBound[T]

    Returns the lower bound of a given percentile where the percentile is between (0,1] The items that are iterated over cannot be negative.

  12. def approximatePercentileBounds[T](percentile: Double, k: Int = QTreeAggregator.DefaultK)(implicit num: Numeric[T]): QTreeAggregator[T]

    Returns the intersection of a bounded percentile where the percentile is between (0,1] The items that are iterated over cannot be negative.

  13. def approximateUniqueCount[T](implicit arg0: Hash128[T]): MonoidAggregator[T, Either[HLL, Set[T]], Long]

    Using a constant amount of memory, give an approximate unique count (~ 1% error).

    Using a constant amount of memory, give an approximate unique count (~ 1% error). This uses an exact set for up to 100 items, then HyperLogLog (HLL) with an 1.2% standard error which uses at most 8192 bytes for each HLL. For more control, see HyperLogLogAggregator.

  14. final def asInstanceOf[T0]: T0

    Definition Classes
    Any

  15. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  16. def const[T](t: T): MonoidAggregator[Any, Unit, T]

    This is a trivial aggregator that always returns a single value

  17. def count[T](pred: (T) ⇒ Boolean): MonoidAggregator[T, Long, Long]

    How many items satisfy a predicate

  18. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  19. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any

  20. def exists[T](pred: (T) ⇒ Boolean): MonoidAggregator[T, Boolean, Boolean]

    Do any items satisfy some predicate

  21. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  22. def forall[T](pred: (T) ⇒ Boolean): MonoidAggregator[T, Boolean, Boolean]

    Do all items satisfy a predicate

  23. def fromMonoid[F, T](implicit mon: Monoid[T], prep: (F) ⇒ T): MonoidAggregator[F, T, T]

  24. def fromMonoid[T](implicit mon: Monoid[T]): MonoidAggregator[T, T, T]

  25. def fromReduce[T](red: (T, T) ⇒ T): Aggregator[T, T, T]

    Using Aggregator.

    Using Aggregator.prepare,present you can add to this aggregator

  26. def fromRing[F, T](implicit rng: Ring[T], prep: (F) ⇒ T): RingAggregator[F, T, T]

  27. def fromRing[T](implicit rng: Ring[T]): RingAggregator[T, T, T]

  28. def fromSemigroup[T](implicit sg: Semigroup[T]): Aggregator[T, T, T]

  29. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any

  30. def hashCode(): Int

    Definition Classes
    AnyRef → Any

  31. def head[T]: Aggregator[T, T, T]

    Take the first (left most in reduce order) item found

  32. def immutableSortedReverseTake[T](count: Int)(implicit arg0: Ordering[T]): MonoidAggregator[T, TopK[T], Seq[T]]

    Immutable version of sortedReverseTake, for frameworks that check immutability of reduce functions.

  33. def immutableSortedTake[T](count: Int)(implicit arg0: Ordering[T]): MonoidAggregator[T, TopK[T], Seq[T]]

    Immutable version of sortedTake, for frameworks that check immutability of reduce functions.

  34. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any

  35. def last[T]: Aggregator[T, T, T]

    Take the last (right most in reduce order) item found

  36. def max[T](implicit arg0: Ordering[T]): Aggregator[T, T, T]

    Get the maximum item

  37. def maxBy[U, T](fn: (U) ⇒ T)(implicit arg0: Ordering[T]): Aggregator[U, U, U]

  38. def min[T](implicit arg0: Ordering[T]): Aggregator[T, T, T]

    Get the minimum item

  39. def minBy[U, T](fn: (U) ⇒ T)(implicit arg0: Ordering[T]): Aggregator[U, U, U]

  40. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  41. final def notify(): Unit

    Definition Classes
    AnyRef

  42. final def notifyAll(): Unit

    Definition Classes
    AnyRef

  43. def numericSum[T](implicit num: Numeric[T]): MonoidAggregator[T, Double, Double]

    An aggregator that sums Numeric values into Doubles.

    An aggregator that sums Numeric values into Doubles.

    This is really no more than converting to Double and then summing. The conversion to double means we don't have the overflow semantics of integer types on the jvm (e.g. Int.MaxValue + 1 == Int.MinValue).

    Note that if you instead wanted to aggregate Numeric values of a type T into the same type T (e.g. if you want MonoidAggregator[T, T, T] for some Numeric type T), you can directly use Aggregator.fromMonoid[T] after importing the numericRing implicit:

    > import com.twitter.algebird.Ring.numericRing > def numericAggregator[T: Numeric]: MonoidAggregator[T, T, T] = Aggregator.fromMonoid[T]

  44. def prepareMonoid[F, T](prep: (F) ⇒ T)(implicit m: Monoid[T]): MonoidAggregator[F, T, T]

  45. def prepareSemigroup[F, T](prep: (F) ⇒ T)(implicit sg: Semigroup[T]): Aggregator[F, T, T]

  46. def randomSample[T](prob: Double, seed: Int = DefaultSeed): MonoidAggregator[T, Option[Batched[T]], List[T]]

    Randomly selects input items where each item has an independent probability 'prob' of being selected.

    Randomly selects input items where each item has an independent probability 'prob' of being selected. This assumes that all sampled records can fit in memory, so use this only when the expected number of sampled values is small.

  47. def reservoirSample[T](count: Int, seed: Int = DefaultSeed): MonoidAggregator[T, PriorityQueue[(Double, T)], Seq[T]]

    Selects exactly 'count' of the input records randomly (or all of the records if there are less then 'count' total records).

    Selects exactly 'count' of the input records randomly (or all of the records if there are less then 'count' total records). This assumes that all 'count' of the records can fit in memory, so use this only for small values of 'count'.

  48. def size: MonoidAggregator[Any, Long, Long]

    This returns the number of items we find

  49. def sortByReverseTake[T, U](count: Int)(fn: (T) ⇒ U)(implicit arg0: Ordering[U]): MonoidAggregator[T, PriorityQueue[T], Seq[T]]

    Same as sortedReverseTake, but using a function that returns a value that has an Ordering.

    Same as sortedReverseTake, but using a function that returns a value that has an Ordering.

    This function is like writing list.sortBy(fn).reverse.take(count).

  50. def sortByTake[T, U](count: Int)(fn: (T) ⇒ U)(implicit arg0: Ordering[U]): MonoidAggregator[T, PriorityQueue[T], Seq[T]]

    Same as sortedTake, but using a function that returns a value that has an Ordering.

    Same as sortedTake, but using a function that returns a value that has an Ordering.

    This function is like writing list.sortBy(fn).take(count).

  51. def sortedReverseTake[T](count: Int)(implicit arg0: Ordering[T]): MonoidAggregator[T, PriorityQueue[T], Seq[T]]

    Take the largest count items using a heap

  52. def sortedTake[T](count: Int)(implicit arg0: Ordering[T]): MonoidAggregator[T, PriorityQueue[T], Seq[T]]

    Take the smallest count items using a heap

  53. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef

  54. def toList[T]: MonoidAggregator[T, Option[Batched[T]], List[T]]

    Put everything in a List.

    Put everything in a List. Note, this could fill the memory if the List is very large.

  55. def toSet[T]: MonoidAggregator[T, Set[T], Set[T]]

    Put everything in a Set.

    Put everything in a Set. Note, this could fill the memory if the Set is very large.

  56. def toString(): String

    Definition Classes
    AnyRef → Any

  57. def uniqueCount[T]: MonoidAggregator[T, Set[T], Int]

    This builds an in-memory Set, and then finally gets the size of that set.

    This builds an in-memory Set, and then finally gets the size of that set. This may not be scalable if the Uniques are very large. You might check the approximateUniqueCount or HyperLogLog Aggregator to get an approximate version of this that is scalable.

  58. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  59. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  60. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped