com.twitter.summingbird.scalding.store
ProxyBatchedStore
Related Doc:
package store
The batcher for this store
The batcher for this store
instances of this trait MAY NOT change the logic here.
instances of this trait MAY NOT change the logic here. This always follows the rule that we look for existing data (avoiding reading deltas in that case), then we fall back to the last checkpointed output by calling readLast. In that case, we compute the results by rolling forward
For each batch, collect up values with the same key on mapside before the keys are expanded.
For each batch, collect up values with the same key on mapside before the keys are expanded.
This is the monadic version of readLast, returns the BatchID actually on disk
This is the monadic version of readLast, returns the BatchID actually on disk
Override this to set up store pruning, by default, no (key,value) pairs are pruned.
Override this to set up store pruning, by default, no (key,value) pairs are pruned. This is a house keeping function to permanently remove entries matching a criteria.
Reads the input data after the last batch written.
Reads the input data after the last batch written.
Returns: - the BatchID of the last batch written - the snapshot of the store just before this state - the data from this input covering all the time SINCE the last snapshot
This combines the current inputs along with the last checkpoint on disk to get a log of all deltas with a timestamp This is useful to leftJoin against a store.
This combines the current inputs along with the last checkpoint on disk to get a log of all deltas with a timestamp This is useful to leftJoin against a store. TODO: This should not limit to batch boundaries, the batch store should handle only writing the data for full batches, but we can materialize more data if it is needed downstream. Note: the returned time interval NOT include the time of the snapshot data point (which is exactly 1 millisecond before the start of the interval).
Get the most recent last batch and the ID (strictly less than the input ID) The "Last" is the stream with only the newest value for each key, within the batch combining the last from batchID and the deltas from batchID.next you get the stream for batchID.next
Get the most recent last batch and the ID (strictly less than the input ID) The "Last" is the stream with only the newest value for each key, within the batch combining the last from batchID and the deltas from batchID.next you get the stream for batchID.next
Override select if you don't want to materialize every batch.
Override select if you don't want to materialize every batch. Note that select MUST return a list containing the final batch in the supplied list; otherwise data would be lost.
This gives the batches needed to cover the requested input This will always be non-empty
This gives the batches needed to cover the requested input This will always be non-empty
For (firstNonZero - 1) we read empty.
For (firstNonZero - 1) we read empty. For all before we error on read. For all later, we proxy On write, we throw if batchID is less than firstNonZero
Record a computed batch of code
Record a computed batch of code
Use this class to easily change, for instance, the pruning for an existing store.