execution

Type Members

1. case class AppendColumns[T, U](func: (T) ⇒ U, tEncoder: ExpressionEncoder[T], uEncoder: ExpressionEncoder[U], newColumns: Seq[Attribute], child: SparkPlan) extends SparkPlan with UnaryNode with Product with Serializable

   Applies the given function to each input row, appending the encoded result at the end of the row.

2. case class BatchPythonEvaluation(udf: PythonUDF, output: Seq[Attribute], child: SparkPlan) extends SparkPlan with Product with Serializable

   Uses PythonRDD to evaluate a PythonUDF, one partition of tuples at a time.

3. case class CacheTableCommand(tableName: String, plan: Option[LogicalPlan], isLazy: Boolean) extends LogicalPlan with RunnableCommand with Product with Serializable

4. case class CoGroup[Key, Left, Right, Result](func: (Key, Iterator[Left], Iterator[Right]) ⇒ TraversableOnce[Result], keyEnc: ExpressionEncoder[Key], leftEnc: ExpressionEncoder[Left], rightEnc: ExpressionEncoder[Right], resultEnc: ExpressionEncoder[Result], output: Seq[Attribute], leftGroup: Seq[Attribute], rightGroup: Seq[Attribute], left: SparkPlan, right: SparkPlan) extends SparkPlan with BinaryNode with Product with Serializable

   Co-groups the data from left and right children, and calls the function with each group and 2 iterators containing all elements in the group from left and right side.

5. class CoGroupedIterator extends Iterator[(InternalRow, Iterator[InternalRow], Iterator[InternalRow])]

   Iterates over GroupedIterators and returns the cogrouped data, i.

6. case class Coalesce(numPartitions: Int, child: SparkPlan) extends SparkPlan with UnaryNode with Product with Serializable

   Return a new RDD that has exactly numPartitions partitions.

7. class CoalescedPartitioner extends Partitioner

   A Partitioner that might group together one or more partitions from the parent.

8. case class ConvertToSafe(child: SparkPlan) extends SparkPlan with UnaryNode with Product with Serializable

   Converts UnsafeRows back into Java-object-based rows.

9. case class ConvertToUnsafe(child: SparkPlan) extends SparkPlan with UnaryNode with Product with Serializable

   Converts Java-object-based rows into UnsafeRows.

10. case class DescribeCommand(child: SparkPlan, output: Seq[Attribute], isExtended: Boolean) extends LogicalPlan with RunnableCommand with Product with Serializable

11. case class DescribeFunction(functionName: String, isExtended: Boolean) extends LogicalPlan with RunnableCommand with Product with Serializable

    A command for users to get the usage of a registered function.

12. case class EvaluatePython(udf: PythonUDF, child: LogicalPlan, resultAttribute: AttributeReference) extends catalyst.plans.logical.UnaryNode with Product with Serializable

    Evaluates a PythonUDF, appending the result to the end of the input tuple.

13. case class Except(left: SparkPlan, right: SparkPlan) extends SparkPlan with BinaryNode with Product with Serializable

    Returns a table with the elements from left that are not in right using the built-in spark subtract function.

14. case class Exchange(newPartitioning: Partitioning, child: SparkPlan, coordinator: Option[ExchangeCoordinator]) extends SparkPlan with UnaryNode with Product with Serializable

    Performs a shuffle that will result in the desired newPartitioning.

15. case class Expand(projections: Seq[Seq[Expression]], output: Seq[Attribute], child: SparkPlan) extends SparkPlan with UnaryNode with Product with Serializable

    Apply the all of the GroupExpressions to every input row, hence we will get multiple output rows for a input row.

16. case class ExplainCommand(logicalPlan: LogicalPlan, output: Seq[Attribute] = ..., extended: Boolean = false) extends LogicalPlan with RunnableCommand with Product with Serializable

    An explain command for users to see how a command will be executed.

17. case class Filter(condition: Expression, child: SparkPlan) extends SparkPlan with UnaryNode with Product with Serializable

18. case class Generate(generator: Generator, join: Boolean, outer: Boolean, output: Seq[Attribute], child: SparkPlan) extends SparkPlan with UnaryNode with Product with Serializable

    Applies a Generator to a stream of input rows, combining the output of each into a new stream of rows.

19. class GroupedIterator extends Iterator[(InternalRow, Iterator[InternalRow])]

    Iterates over a presorted set of rows, chunking it up by the grouping expression.

20. case class Intersect(left: SparkPlan, right: SparkPlan) extends SparkPlan with BinaryNode with Product with Serializable

    Returns the rows in left that also appear in right using the built in spark intersection function.

21. case class Limit(limit: Int, child: SparkPlan) extends SparkPlan with UnaryNode with Product with Serializable

    Take the first limit elements.

22. case class MapGroups[K, T, U](func: (K, Iterator[T]) ⇒ TraversableOnce[U], kEncoder: ExpressionEncoder[K], tEncoder: ExpressionEncoder[T], uEncoder: ExpressionEncoder[U], groupingAttributes: Seq[Attribute], output: Seq[Attribute], child: SparkPlan) extends SparkPlan with UnaryNode with Product with Serializable

    Groups the input rows together and calls the function with each group and an iterator containing all elements in the group.

23. case class MapPartitions[T, U](func: (Iterator[T]) ⇒ Iterator[U], tEncoder: ExpressionEncoder[T], uEncoder: ExpressionEncoder[U], output: Seq[Attribute], child: SparkPlan) extends SparkPlan with UnaryNode with Product with Serializable

    Applies the given function to each input row and encodes the result.

24. case class OutputFaker(output: Seq[Attribute], child: SparkPlan) extends SparkPlan with Product with Serializable

    A plan node that does nothing but lie about the output of its child.

25. case class Project(projectList: Seq[NamedExpression], child: SparkPlan) extends SparkPlan with UnaryNode with Product with Serializable

26. class QueryExecution extends AnyRef

    The primary workflow for executing relational queries using Spark.

27. class QueryExecutionException extends Exception

28. case class Sample(lowerBound: Double, upperBound: Double, withReplacement: Boolean, seed: Long, child: SparkPlan) extends SparkPlan with UnaryNode with Product with Serializable

    Sample the dataset.

29. case class SetCommand(kv: Option[(String, Option[String])]) extends LogicalPlan with RunnableCommand with Logging with Product with Serializable

30. case class ShowFunctions(db: Option[String], pattern: Option[String]) extends LogicalPlan with RunnableCommand with Product with Serializable

    A command for users to list all of the registered functions.

31. case class ShowTablesCommand(databaseName: Option[String]) extends LogicalPlan with RunnableCommand with Product with Serializable

    A command for users to get tables in the given database.

32. class ShuffledRowRDD extends RDD[InternalRow]

    This is a specialized version of org.apache.spark.rdd.ShuffledRDD that is optimized for shuffling rows instead of Java key-value pairs.

33. case class Sort(sortOrder: Seq[SortOrder], global: Boolean, child: SparkPlan, testSpillFrequency: Int = 0) extends SparkPlan with UnaryNode with Product with Serializable

    Performs (external) sorting.

34. abstract class SparkPlan extends QueryPlan[SparkPlan] with Logging with Serializable

    The base class for physical operators.

35. class SparkPlanner extends SparkStrategies

36. class SparkSQLParser extends AbstractSparkSQLParser

    The top level Spark SQL parser.

37. case class TakeOrderedAndProject(limit: Int, sortOrder: Seq[SortOrder], projectList: Option[Seq[NamedExpression]], child: SparkPlan) extends SparkPlan with UnaryNode with Product with Serializable

    Take the first limit elements as defined by the sortOrder, and do projection if needed.

38. case class UncacheTableCommand(tableName: String) extends LogicalPlan with RunnableCommand with Product with Serializable

39. case class Union(children: Seq[SparkPlan]) extends SparkPlan with Product with Serializable

    Union two plans, without a distinct.

40. final class UnsafeFixedWidthAggregationMap extends AnyRef

41. final class UnsafeKVExternalSorter extends AnyRef

42. case class Window(projectList: Seq[Attribute], windowExpression: Seq[NamedExpression], partitionSpec: Seq[Expression], orderSpec: Seq[SortOrder], child: SparkPlan) extends SparkPlan with UnaryNode with Product with Serializable

    This class calculates and outputs (windowed) aggregates over the rows in a single (sorted) partition.