Reservoir

A key-value store using deterministic reservoir sampling.

Items are added with an associated key. Items may be retrieved by the corresponding key, and a list of keys can also be retrieved. If maxSize is not zero, then it dictates the maximum number of items that will be stored for each key. Once there are more items for a given key, they are replaced via reservoir sampling, such that each item has an equal probability of being included in the sample.

Deterministic means that for any given seed and bucket size, the sequence of values that are kept for any given key will always be the same, and that this is independent of any insertions for other keys. That is:

val reservoirA = ReservoirKVStore(10)
val reservoirB = ReservoirKVStore(10)
(0 until 100).foreach(i => reservoirA.add("key1", i))
(0 until 100).foreach(i => reservoirA.add("key2", i))
(0 until 100).foreach(i => {
  reservoirB.add("key1", i)
  reservoirB.add("key2", i)
})

After executing this code, reservoirA and reservoirB will be in identical states.

For more information on reservoir sampling, refer to [this page](https://en.wikipedia.org/wiki/Reservoir_sampling).

Note that, adding items has amortized O(1) runtime cost.

maxSize: Maximum size of each bucket in this reservoir key-value store.
seed: Seed to use for the random number generator used while sampling.
alwaysKeepLast: Boolean flag indicating whether to always store the last seen item. If set to true and the last seen item was not sampled to be stored, then it replaces the last item in the corresponding bucket.

Linear Supertypes

Serializable, Serializable, Product, Equals, AnyRef, Any

Instance Constructors

new Reservoir(maxSize: Int, seed: Long = 0L, alwaysKeepLast: Boolean = true)

maxSize
Maximum size of each bucket in this reservoir key-value store.
seed
Seed to use for the random number generator used while sampling.
alwaysKeepLast
Boolean flag indicating whether to always store the last seen item. If set to true and the last seen item was not sampled to be stored, then it replaces the last item in the corresponding bucket.

Value Members

final def !=(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def ##(): Int

Definition Classes
AnyRef → Any
final def ==(arg0: Any): Boolean

Definition Classes
AnyRef → Any
def add(key: K, item: V, transformFn: (V) ⇒ V = identity[V]): Unit

Adds a new item to the reservoir with the provided key.
Adds a new item to the reservoir with the provided key.
If the corresponding reservoir has not yet reached full size, then the new item is guaranteed to be added. If the reservoir is full, then the behavior of this method depends on the value of alwaysKeepLast.
If alwaysKeepLast is set to true, then the new item is guaranteed to be added to the reservoir, and either the previous last item will be replaced, or (with low probability) an older item will be replaced.
If alwaysKeepLast is set to false, then the new item may replace an old item with low probability.
If transformFn is provided, then it will be applied to transform the provided item (lazily, if and only if the item is going to be included in the reservoir).
key
Key for the item to add.
item
Item to add.
transformFn
Transform function for the item to add.
val alwaysKeepLast: Boolean

Boolean flag indicating whether to always store the last seen item.
Boolean flag indicating whether to always store the last seen item. If set to true and the last seen item was not sampled to be stored, then it replaces the last item in the corresponding bucket.
final def asInstanceOf[T0]: T0

Definition Classes
Any
def clone(): AnyRef

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( ... )
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def filter(filterFn: (V) ⇒ Boolean, key: Option[K] = None): Int

Filters the items in this reservoir using the provided filtering function.
Filters the items in this reservoir using the provided filtering function.
When filtering items from each reservoir bucket, we must update the internal state variable numItemsSeen, which is used for determining the rate of replacement in reservoir sampling. Ideally, numItemsSeen would contain the exact number of items that have ever been seen by the add function of this reservoir, and that satisfy the provided filtering function. However, the reservoir bucket does not have access to all of the items it has seen -- it only has access to the subset of items that have survived sampling (i.e., _items). Therefore, we estimate numItemsSeen by scaling its original value by the same ratio as the ratio of items that were not filtered out and that are currently stored in this reservoir bucket.
filterFn
Filtering function that returns true for the items to be kept in the reservoir.
key
Optional key for which to filter the values. If None (the default), then the values for all keys in the reservoir are filtered.
returns
Number of items filtered from this reservoir.
def finalize(): Unit

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
def items(key: K): List[V]

Returns all the items stored for the provided key and throws an exception if the key does not exist.
def keys: Iterable[K]

Returns all the keys in the reservoir.
val maxSize: Int

Maximum size of each bucket in this reservoir key-value store.
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def notify(): Unit

Definition Classes
AnyRef
final def notifyAll(): Unit

Definition Classes
AnyRef
val seed: Long

Seed to use for the random number generator used while sampling.
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )

Related Doc: package utilities

case class Reservoir[K, V](maxSize: Int, seed: Long = 0L, alwaysKeepLast: Boolean = true) extends Product with Serializable

Instance Constructors

new Reservoir(maxSize: Int, seed: Long = 0L, alwaysKeepLast: Boolean = true)

Value Members

final def !=(arg0: Any): Boolean

final def ##(): Int

final def ==(arg0: Any): Boolean

def add(key: K, item: V, transformFn: (V) ⇒ V = identity[V]): Unit

val alwaysKeepLast: Boolean

final def asInstanceOf[T0]: T0

def clone(): AnyRef

final def eq(arg0: AnyRef): Boolean

def filter(filterFn: (V) ⇒ Boolean, key: Option[K] = None): Int

def finalize(): Unit

final def getClass(): Class[_]

final def isInstanceOf[T0]: Boolean

def items(key: K): List[V]

def keys: Iterable[K]

val maxSize: Int

final def ne(arg0: AnyRef): Boolean

final def notify(): Unit

final def notifyAll(): Unit

val seed: Long

final def synchronized[T0](arg0: ⇒ T0): T0

final def wait(): Unit

final def wait(arg0: Long, arg1: Int): Unit

final def wait(arg0: Long): Unit

Inherited from Serializable

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from AnyRef

Inherited from Any

Ungrouped