Package org.yamcs.yarch
Class PartitionManager
- java.lang.Object
-
- org.yamcs.yarch.PartitionManager
-
- Direct Known Subclasses:
RdbPartitionManager
public abstract class PartitionManager extends Object
Keeps track of partitions and histograms for one table.The partitioning is organised in a set of time based partitioning intervals, each interval being one day/month/year long, depending on the partitioning schema.
Each interval has associated a set of value based partitions.
In addition each interval has a set of histograms - one for each table column for which histograms have been created.
In the RocksDB implementation (both old and new) each interval corresponds to one rocksdb database directory.
In case there is no time based partitioning, there is only one interval.
In case there is no value based partitioning, there is only one partition in each interval.
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description static class
PartitionManager.Interval
For tables partitioned by time this holds all the partitions for a given time interval.
-
Field Summary
Fields Modifier and Type Field Description protected PartitionedTimeInterval<PartitionManager.Interval>
intervals
protected PartitioningSpec
partitioningSpec
protected PartitionManager.Interval
pcache
protected TableDefinition
tableDefinition
-
Constructor Summary
Constructors Constructor Description PartitionManager(TableDefinition tableDefinition)
-
Method Summary
All Methods Instance Methods Abstract Methods Concrete Methods Modifier and Type Method Description HistogramInfo
createAndGetHistogram(long instant, String columnName)
Partition
createAndGetPartition(long instant, Object value)
Creates (if not already existing) and returns the partition in which the instant,value should be written.protected abstract HistogramInfo
createHistogram(String columnName)
protected abstract HistogramInfo
createHistogramByTime(TimePartitionInfo pinfo, String columnName)
protected abstract Partition
createPartition(Object value)
Create a partition for value based partitioningprotected abstract Partition
createPartitionByTime(TimePartitionInfo pinfo, Object value)
Create a partition for time (and possible value) based partitioningPartition
getPartitionForTuple(Tuple t)
Gets partition where tuple has to be written.PartitioningSpec
getPartitioningSpec()
List<Partition>
getPartitions()
Retrieves the existing partitionsTableDefinition
getTableDefinition()
String
getTableName()
Get the name of the table whose partitions are managed by this object.Iterator<PartitionManager.Interval>
intervalIterator(TimeInterval timeInterval)
Iterates over all intervals overlapping with the timeInterval.Iterator<PartitionManager.Interval>
iterator(long start, Set<Object> partitionValueFilter)
Seeiterator(Set)
Iterator<PartitionManager.Interval>
iterator(Set<Object> partitionValueFilter)
Returns an iterator which at each step gives the list of partition corresponding to a time interval (so when we do a replay those partitions have to be played in parallel).Iterator<PartitionManager.Interval>
reverseIterator(long start, Set<Object> partitionValueFilter)
Iterator<PartitionManager.Interval>
reverseIterator(Set<Object> partitionValueFilter)
same as above, only in reverse direction
-
-
-
Field Detail
-
tableDefinition
protected final TableDefinition tableDefinition
-
partitioningSpec
protected final PartitioningSpec partitioningSpec
-
intervals
protected final PartitionedTimeInterval<PartitionManager.Interval> intervals
-
pcache
protected PartitionManager.Interval pcache
-
-
Constructor Detail
-
PartitionManager
public PartitionManager(TableDefinition tableDefinition)
-
-
Method Detail
-
iterator
public Iterator<PartitionManager.Interval> iterator(Set<Object> partitionValueFilter)
Returns an iterator which at each step gives the list of partition corresponding to a time interval (so when we do a replay those partitions have to be played in parallel). The iterator returns intervals sorted on time.- Parameters:
partitionValueFilter
- - return only partitions whose value are in the filter. If null, return all partitions;- Returns:
- iterator going over partitions
-
reverseIterator
public Iterator<PartitionManager.Interval> reverseIterator(Set<Object> partitionValueFilter)
same as above, only in reverse direction- Parameters:
partitionValueFilter
-- Returns:
-
iterator
public Iterator<PartitionManager.Interval> iterator(long start, Set<Object> partitionValueFilter)
Seeiterator(Set)
- Parameters:
start
-partitionValueFilter
- values - return only partitions whose value are in the filter. If null, return all partitions;- Returns:
- an iterator over the partitions starting at the specified start time
-
intervalIterator
public Iterator<PartitionManager.Interval> intervalIterator(TimeInterval timeInterval)
Iterates over all intervals overlapping with the timeInterval. Note that the timeInterval is considered closed at both ends (if set): [timeInterval.start, timeInterval.stop] whereas the partition intervals are considered closed at start and open at stop: [Interval.start, Interval.stop)
-
reverseIterator
public Iterator<PartitionManager.Interval> reverseIterator(long start, Set<Object> partitionValueFilter)
-
createAndGetPartition
public Partition createAndGetPartition(long instant, Object value) throws IOException
Creates (if not already existing) and returns the partition in which the instant,value should be written. value can be null (in case of no value partitioning)- Parameters:
instant
- - time for which the partition has to be created - can be TimeEncoding.INVALID in case value only or no partitioningvalue
- - value for which the partition has to be created - can be null in case of time only or no partitioning. For the enum partitions, the value is the index (type Short) rather than the string.- Returns:
- a Partition
- Throws:
IOException
-
createAndGetHistogram
public HistogramInfo createAndGetHistogram(long instant, String columnName)
-
getPartitionForTuple
public Partition getPartitionForTuple(Tuple t) throws IOException
Gets partition where tuple has to be written. Creates the partition if necessary.- Parameters:
t
-- Returns:
- the partition where the tuple has to be written
- Throws:
IOException
-
getTableName
public String getTableName()
Get the name of the table whose partitions are managed by this object.- Returns:
- the name of the table.
-
getTableDefinition
public TableDefinition getTableDefinition()
-
getPartitioningSpec
public PartitioningSpec getPartitioningSpec()
-
createPartitionByTime
protected abstract Partition createPartitionByTime(TimePartitionInfo pinfo, Object value) throws IOException
Create a partition for time (and possible value) based partitioning- Parameters:
pinfo
-value
-- Returns:
- Throws:
IOException
-
createPartition
protected abstract Partition createPartition(Object value)
Create a partition for value based partitioning- Parameters:
value
-- Returns:
-
createHistogramByTime
protected abstract HistogramInfo createHistogramByTime(TimePartitionInfo pinfo, String columnName)
-
createHistogram
protected abstract HistogramInfo createHistogram(String columnName)
-
-