execution

Type Members

1. case class AppendColumnsExec(func: (Any) ⇒ Any, deserializer: Expression, serializer: Seq[NamedExpression], child: SparkPlan) extends SparkPlan with UnaryExecNode with ObjectOperator with Product with Serializable

   Applies the given function to each input row, appending the encoded result at the end of the row.

2. case class AppendColumnsWithObjectExec(func: (Any) ⇒ Any, inputSerializer: Seq[NamedExpression], newColumnsSerializer: Seq[NamedExpression], child: SparkPlan) extends SparkPlan with UnaryExecNode with ObjectOperator with Product with Serializable

   An optimized version of AppendColumnsExec, that can be executed on deserialized object directly.

3. trait BaseLimitExec extends SparkPlan with UnaryExecNode with CodegenSupport

   Helper trait which defines methods that are shared by both LocalLimitExec and GlobalLimitExec.

4. abstract class BufferedRowIterator extends AnyRef

5. case class CoGroupExec(func: (Any, Iterator[Any], Iterator[Any]) ⇒ TraversableOnce[Any], keyDeserializer: Expression, leftDeserializer: Expression, rightDeserializer: Expression, leftGroup: Seq[Attribute], rightGroup: Seq[Attribute], leftAttr: Seq[Attribute], rightAttr: Seq[Attribute], outputObjAttr: Attribute, left: SparkPlan, right: SparkPlan) extends SparkPlan with BinaryExecNode with ObjectOperator with Product with Serializable

   Co-groups the data from left and right children, and calls the function with each group and 2 iterators containing all elements in the group from left and right side.

6. class CoGroupedIterator extends Iterator[(InternalRow, Iterator[InternalRow], Iterator[InternalRow])]

   Iterates over GroupedIterators and returns the cogrouped data, i.

7. case class CoalesceExec(numPartitions: Int, child: SparkPlan) extends SparkPlan with UnaryExecNode with Product with Serializable

   Physical plan for returning a new RDD that has exactly numPartitions partitions.

8. class CoalescedPartitioner extends Partitioner

   A Partitioner that might group together one or more partitions from the parent.

9. trait CodegenSupport extends SparkPlan

   An interface for those physical operators that support codegen.

10. case class CollapseCodegenStages(conf: SQLConf) extends Rule[SparkPlan] with Product with Serializable

    Find the chained plans that support codegen, collapse them together as WholeStageCodegen.

11. case class CollectLimitExec(limit: Int, child: SparkPlan) extends SparkPlan with UnaryExecNode with Product with Serializable

    Take the first limit elements and collect them to a single partition.

12. case class DeserializeToObject(deserializer: Expression, outputObjAttr: Attribute, child: SparkPlan) extends SparkPlan with UnaryExecNode with CodegenSupport with Product with Serializable

    Takes the input row from child and turns it into object using the given deserializer expression.

13. case class ExpandExec(projections: Seq[Seq[Expression]], output: Seq[Attribute], child: SparkPlan) extends SparkPlan with UnaryExecNode with CodegenSupport with Product with Serializable

    Apply all of the GroupExpressions to every input row, hence we will get multiple output rows for a input row.

14. case class FilterExec(condition: Expression, child: SparkPlan) extends SparkPlan with UnaryExecNode with CodegenSupport with PredicateHelper with Product with Serializable

    Physical plan for Filter.

15. case class GenerateExec(generator: Generator, join: Boolean, outer: Boolean, output: Seq[Attribute], child: SparkPlan) extends SparkPlan with UnaryExecNode with Product with Serializable

    Applies a Generator to a stream of input rows, combining the output of each into a new stream of rows.

16. case class GlobalLimitExec(limit: Int, child: SparkPlan) extends SparkPlan with BaseLimitExec with Product with Serializable

    Take the first limit elements of the child's single output partition.

17. class GroupedIterator extends Iterator[(InternalRow, Iterator[InternalRow])]

    Iterates over a presorted set of rows, chunking it up by the grouping expression.

18. case class InputAdapter(child: SparkPlan) extends SparkPlan with UnaryExecNode with CodegenSupport with Product with Serializable

    InputAdapter is used to hide a SparkPlan from a subtree that support codegen.

19. case class LocalLimitExec(limit: Int, child: SparkPlan) extends SparkPlan with BaseLimitExec with Product with Serializable

    Take the first limit elements of each child partition, but do not collect or shuffle them.

20. case class MapElementsExec(func: AnyRef, outputObjAttr: Attribute, child: SparkPlan) extends SparkPlan with UnaryExecNode with ObjectOperator with CodegenSupport with Product with Serializable

    Applies the given function to each input object.

21. case class MapGroupsExec(func: (Any, Iterator[Any]) ⇒ TraversableOnce[Any], keyDeserializer: Expression, valueDeserializer: Expression, groupingAttributes: Seq[Attribute], dataAttributes: Seq[Attribute], outputObjAttr: Attribute, child: SparkPlan) extends SparkPlan with UnaryExecNode with ObjectOperator with Product with Serializable

    Groups the input rows together and calls the function with each group and an iterator containing all elements in the group.

22. case class MapPartitionsExec(func: (Iterator[Any]) ⇒ Iterator[Any], outputObjAttr: Attribute, child: SparkPlan) extends SparkPlan with UnaryExecNode with ObjectOperator with Product with Serializable

    Applies the given function to input object iterator.

23. trait ObjectOperator extends SparkPlan

    Helper functions for physical operators that work with user defined objects.

24. case class OutputFakerExec(output: Seq[Attribute], child: SparkPlan) extends SparkPlan with Product with Serializable

    A plan node that does nothing but lie about the output of its child.

25. case class PlanSubqueries(sparkSession: SparkSession) extends Rule[SparkPlan] with Product with Serializable

    Plans scalar subqueries from that are present in the given SparkPlan.

26. case class ProjectExec(projectList: Seq[NamedExpression], child: SparkPlan) extends SparkPlan with UnaryExecNode with CodegenSupport with Product with Serializable

    Physical plan for Project.

27. class QueryExecution extends AnyRef

    The primary workflow for executing relational queries using Spark.

28. class QueryExecutionException extends Exception

29. case class RangeExec(start: Long, step: Long, numSlices: Int, numElements: BigInt, output: Seq[Attribute]) extends SparkPlan with LeafExecNode with CodegenSupport with Product with Serializable

    Physical plan for range (generating a range of 64 bit numbers.

30. case class SampleExec(lowerBound: Double, upperBound: Double, withReplacement: Boolean, seed: Long, child: SparkPlan) extends SparkPlan with UnaryExecNode with CodegenSupport with Product with Serializable

    Physical plan for sampling the dataset.

31. case class ScalarSubquery(executedPlan: SparkPlan, exprId: ExprId) extends SubqueryExpression with Product with Serializable

    A subquery that will return only one row and one column.

32. case class SerializeFromObjectExec(serializer: Seq[NamedExpression], child: SparkPlan) extends SparkPlan with UnaryExecNode with CodegenSupport with Product with Serializable

    Takes the input object from child and turns in into unsafe row using the given serializer expression.

33. class ShuffledRowRDD extends RDD[InternalRow]

    This is a specialized version of org.apache.spark.rdd.ShuffledRDD that is optimized for shuffling rows instead of Java key-value pairs.

34. case class SortExec(sortOrder: Seq[SortOrder], global: Boolean, child: SparkPlan, testSpillFrequency: Int = 0) extends SparkPlan with UnaryExecNode with CodegenSupport with Product with Serializable

    Performs (external) sorting.

35. class SparkOptimizer extends Optimizer

36. abstract class SparkPlan extends QueryPlan[SparkPlan] with Logging with Serializable

    The base class for physical operators.

37. class SparkPlanInfo extends AnyRef

    :: DeveloperApi :: Stores information about a SQL SparkPlan.

38. class SparkPlanner extends SparkStrategies

39. class SparkSqlAstBuilder extends AstBuilder

    Builder that converts an ANTLR ParseTree into a LogicalPlan/Expression/TableIdentifier.

40. class SparkSqlParser extends AbstractSqlParser

    Concrete parser for Spark SQL statements.

41. case class SubqueryExec(name: String, child: SparkPlan) extends SparkPlan with UnaryExecNode with Product with Serializable

    Physical plan for a subquery.

42. case class TakeOrderedAndProjectExec(limit: Int, sortOrder: Seq[SortOrder], projectList: Option[Seq[NamedExpression]], child: SparkPlan) extends SparkPlan with UnaryExecNode with Product with Serializable

    Take the first limit elements as defined by the sortOrder, and do projection if needed.

43. case class UnionExec(children: Seq[SparkPlan]) extends SparkPlan with Product with Serializable

    Physical plan for unioning two plans, without a distinct.

44. final class UnsafeFixedWidthAggregationMap extends AnyRef

45. final class UnsafeKVExternalSorter extends AnyRef

46. case class WholeStageCodegenExec(child: SparkPlan) extends SparkPlan with UnaryExecNode with CodegenSupport with Product with Serializable

    WholeStageCodegen compile a subtree of plans that support codegen together into single Java function.

47. case class WindowExec(windowExpression: Seq[NamedExpression], partitionSpec: Seq[Expression], orderSpec: Seq[SortOrder], child: SparkPlan) extends SparkPlan with UnaryExecNode with Product with Serializable

    This class calculates and outputs (windowed) aggregates over the rows in a single (sorted) partition.