org.apache.drill.exec.planner.fragment

Class QueueQueryParallelizer

java.lang.Object

org.apache.drill.exec.planner.fragment.SimpleParallelizer

org.apache.drill.exec.planner.fragment.QueueQueryParallelizer

All Implemented Interfaces:

ParallelizationParameters, QueryParallelizer

public class QueueQueryParallelizer
extends SimpleParallelizer

Paralellizer specialized for managing resources for a query based on Queues. This parallelizer does not deal with increase/decrease of the parallelization of a query plan based on the current cluster state. However, the memory assignment for each operator, minor fragment and major fragment is based on the cluster state and provided queue configuration.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.drill.exec.planner.fragment.SimpleParallelizer

SimpleParallelizer.CountRequiredFragments

Constructor Summary

Constructors

Constructor and Description
QueueQueryParallelizer(boolean memoryPlanning, QueryContext queryContext)

Method Summary

Modifier and Type	Method and Description
void	adjustMemory(PlanningSet planningSet, Set<Wrapper> roots, Collection<CoordinationProtos.DrillbitEndpoint> activeEndpoints) Function called by the SimpleParallelizer to adjust the memory post parallelization.
BiFunction<CoordinationProtos.DrillbitEndpoint,PhysicalOperator,Long>	getMemory()

Methods inherited from class org.apache.drill.exec.planner.fragment.SimpleParallelizer

collectStatsAndParallelizeFragments, generateWorkUnit, generateWorkUnit, getAffinityFactor, getMaxGlobalWidth, getMaxWidthPerNode, getRootFragments, getSliceTarget, getSplitFragments, initFragmentWrappers, prepareFragmentTree, traverse

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

QueueQueryParallelizer

public QueueQueryParallelizer(boolean memoryPlanning,
                              QueryContext queryContext)

Method Detail

getMemory

public BiFunction<CoordinationProtos.DrillbitEndpoint,PhysicalOperator,Long> getMemory()

Specified by:

getMemory in class SimpleParallelizer

adjustMemory

public void adjustMemory(PlanningSet planningSet,
                         Set<Wrapper> roots,
                         Collection<CoordinationProtos.DrillbitEndpoint> activeEndpoints)
                  throws PhysicalOperatorSetupException

Function called by the SimpleParallelizer to adjust the memory post parallelization. The overall logic is to traverse the fragment tree and call the MemoryCalculator on each major fragment. Once the memory is computed, resource requirement are accumulated per drillbit. The total resource requirements are used to select a queue. If the selected queue's resource limit is more/less than the query's requirement than the memory will be re-adjusted.

Specified by:

adjustMemory in class SimpleParallelizer

Parameters:

planningSet - context of the fragments.

roots - root fragments.

activeEndpoints - currently active endpoints.

Throws:

PhysicalOperatorSetupException