Package org.apache.druid.frame.processor

Class SuperSorterProgressTracker

  • public class SuperSorterProgressTracker
extends Object
    A thread-safe class that keeps track of the progress of an n-way, multilevel merge sort. The logic for this class is coupled to the implementation of SuperSorter, which it is designed to track. It keeps a track of the following things: 1. Total levels to merge (Cannnot be modified once set) 2. Total mergers to merge in each level 3. Total merged batches in each level There are a few states in which this progress tracker can be in (depending on the SuperSorter's progres): 1. Unknown everything | totalMergingLevels = -1, mergersForUltimateLevel = UNKNOWN_TOTAL 2. Known total levels, but unknown output partitions | totalMergingLevels = set, mergersForUltimateLevelSet = UNKNOWN_TOTAL 3. Unknown total levels, but known output partitions | totalMergingLevels = -1, mergersForUltimateLevelSet != UNKNOWN_TOTAL 4. Known everything | totalMergingLevels = set, mergersForUltimateLevelSet != UNKNOWN_TOTAL Progress can only be quantified if the total levels are known snapshot function does the housekeeping to hide this operational complexity away from classes other than SuperSorter and SuperSorterProgressTracker

    • Constructor Detail

      • SuperSorterProgressTracker

        public SuperSorterProgressTracker()

    • Method Detail

      • setTotalMergingLevels

        public void setTotalMergingLevels​(int totalMergingLevels)
        Set total merging levels for the SuperSorter it is tracking. Can be set only once

      • setTotalMergersForLevel

        public void setTotalMergersForLevel​(int level,
                                    long totalMergers)
        Sets the total mergers for a level. Can be set only once, except for the ultimate level (if total levels are known) because they get overridden by totalMergersForUltimateLevel

      • setTotalMergersForUltimateLevel

        public void setTotalMergersForUltimateLevel​(long totalMergersForUltimateLevel)
        Sets the number of mergers in the ultimate level (number of mergers = number of output partitions). Can only be set once

      • addMergedBatchesForLevel

        public void addMergedBatchesForLevel​(int level,
                                     long additionalMergedBatches)
        This method is designed to be called during the course of the sorting. The batches once merged for a particular level can be marked as such through this.

      • markTriviallyComplete

        public void markTriviallyComplete()
        If the SuperSorter is trivially done without doing any work (for eg - empty input), the tracker can be marked as trivially complete. Once a tracker is marked as complete, the snapshots will always report back the progress digest as 1. Any modification to the state of the tracker (eg: calling setTotalMergersForLevel()) would proceed as regular, but