Package org.apache.druid.frame.processor
Class SuperSorter
- java.lang.Object
-
- org.apache.druid.frame.processor.SuperSorter
-
public class SuperSorter extends Object
Sorts and partitions a dataset using parallel external merge sort. Input is provided as a set ofReadableFrameChannel
and output is provided asOutputChannels
. Work is performed on a providedFrameProcessorExecutor
. The most central point for SuperSorter logic is therunWorkersIfPossible()
method, which determines what needs to be done next based on the current state of the SuperSorter. The logic is: 1) Read input channels intoinputBuffer
usingFrameChannelBatcher
, launched viarunNextBatcher()
, up to a limit ofmaxChannelsPerProcessor
per batcher. 2) Merge and write frames frominputBuffer
intoFrameFile
scratch files usingFrameChannelMerger
launched viarunNextLevelZeroMerger()
. 3a) Merge level 0 scratch files into level 1 scratch files usingFrameChannelMerger
launched fromrunNextMiddleMerger()
, processing up tomaxChannelsPerProcessor
files per merger. Continue this process through increasing level numbers, with the size of scratch files increasing by a factor ofmaxChannelsPerProcessor
each level. 3b) For the penultimate level, theFrameChannelMerger
launched byrunNextMiddleMerger()
writes partitionedFrameFile
scratch files. The penultimate level cannot be written untiloutputPartitionsFuture
resolves, so if it has not resolved yet by this point, the SuperSorter pauses. The SuperSorter resumes and writes the penultimate level's files when the future resolves. 4) Write the final level usingFrameChannelMerger
launched fromrunNextUltimateMerger()
. Outputs for this level are written to channels provided byoutputChannelFactory
, rather than scratch files. At all points, higher level processing is preferred over lower-level processing. Writing to final output files is preferred over intermediate, and writing to intermediate files is preferred over reading inputs. These preferences ensure that the amount of data buffered up in memory does not grow too large. Potential future work (things we could optimize if necessary): - Collapse merging to a single level if level zero has one merger, and we want to write one output partition. - Skip batching, and inject directly into level 0, if input channels are already individually fully-sorted. - Combine (for example: aggregate) while merging.
-
-
Field Summary
Fields Modifier and Type Field Description static int
UNKNOWN_LEVEL
static long
UNKNOWN_TOTAL
-
Constructor Summary
Constructors Constructor Description SuperSorter(List<ReadableFrameChannel> inputChannels, FrameReader frameReader, List<KeyColumn> sortKey, com.google.common.util.concurrent.ListenableFuture<ClusterByPartitions> outputPartitionsFuture, FrameProcessorExecutor exec, OutputChannelFactory outputChannelFactory, OutputChannelFactory intermediateOutputChannelFactory, int maxActiveProcessors, int maxChannelsPerProcessor, long rowLimit, String cancellationId, SuperSorterProgressTracker superSorterProgressTracker)
Initializes a SuperSorter.
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description com.google.common.util.concurrent.ListenableFuture<OutputChannels>
run()
Starts sorting.String
stateString()
Returns a string encapsulating the current state of this object.
-
-
-
Field Detail
-
UNKNOWN_LEVEL
public static final int UNKNOWN_LEVEL
- See Also:
- Constant Field Values
-
UNKNOWN_TOTAL
public static final long UNKNOWN_TOTAL
- See Also:
- Constant Field Values
-
-
Constructor Detail
-
SuperSorter
public SuperSorter(List<ReadableFrameChannel> inputChannels, FrameReader frameReader, List<KeyColumn> sortKey, com.google.common.util.concurrent.ListenableFuture<ClusterByPartitions> outputPartitionsFuture, FrameProcessorExecutor exec, OutputChannelFactory outputChannelFactory, OutputChannelFactory intermediateOutputChannelFactory, int maxActiveProcessors, int maxChannelsPerProcessor, long rowLimit, @Nullable String cancellationId, SuperSorterProgressTracker superSorterProgressTracker)
Initializes a SuperSorter.- Parameters:
inputChannels
- input channels. All frames in these channels must be sorted according to theClusterBy.getColumns()
, or else sorting will not produce correct output.frameReader
- frame reader for the input channelssortKey
- desired sorting orderoutputPartitionsFuture
- a future that resolves to the desired output partitions. Sorting will block prior to writing out final outputs until this future resolves. However, the sorter will be able to read all inputs even if this future is unresolved. If output need not be partitioned, useClusterByPartitions.oneUniversalPartition()
. In this case a single sorted channel is generated.exec
- executor to perform work inoutputChannelFactory
- factory for partitioned, sorted output channelsintermediateOutputChannelFactory
- factory for intermediate data produced by sorting levelsmaxActiveProcessors
- maximum number of merging processors to execute at once in the providedFrameProcessorExecutor
maxChannelsPerProcessor
- maximum number of channels to merge at once per merging processorrowLimit
- limit to apply during sorting. The limit is merely advisory: the actual number of rows returned may be larger than the limit. The limit is applied across all partitions, not to each partition individually.cancellationId
- cancellation id to use when running processors in the providedFrameProcessorExecutor
.superSorterProgressTracker
- progress tracker
-
-
Method Detail
-
run
public com.google.common.util.concurrent.ListenableFuture<OutputChannels> run()
Starts sorting. Can only be called once. Work is performed in theFrameProcessorExecutor
that was passed to the constructor. Returns a future containing partitioned sorted output channels.
-
stateString
public String stateString()
Returns a string encapsulating the current state of this object.
-
-