Packages

package root

Definition Classes: root

package org

Definition Classes: root

package apache

Definition Classes: org

package spark

Definition Classes: apache

package sql

Allows the execution of relational queries, including those expressed in SQL using Spark.

Definition Classes: spark

package execution

The physical execution component of Spark SQL.

The physical execution component of Spark SQL. Note that this is a private package. All classes in catalyst are considered an internal API to Spark SQL and are subject to change between minor releases.

Definition Classes: sql

package adaptive

Definition Classes: execution

package aggregate

Definition Classes: execution

package analysis

Definition Classes: execution

DetectAmbiguousSelfJoin

package arrow

Definition Classes: execution

package bucketing

Definition Classes: execution

package columnar

Definition Classes: execution

package command

Definition Classes: execution

package datasources

Definition Classes: execution

package debug

Contains methods for debugging query execution.

Usage:

import org.apache.spark.sql.execution.debug._
sql("SELECT 1").debug()
sql("SELECT 1").debugCodegen()

or for streaming case (structured streaming):

import org.apache.spark.sql.execution.debug._
val query = df.writeStream.<...>.start()
query.debugCodegen()

Note that debug in structured streaming is not supported, because it doesn't make sense for streaming to execute batch once while main query is running concurrently.

Definition Classes: execution

package dynamicpruning

Definition Classes: execution

package exchange

Definition Classes: execution

package joins

Definition Classes: execution

package metric

Definition Classes: execution

package python

Definition Classes: execution

package r

Definition Classes: execution

package stat

Definition Classes: execution

package streaming

Definition Classes: execution

package ui

Definition Classes: execution

package vectorized

Definition Classes: execution

package window

Definition Classes: execution

org.apache.spark.sql.execution

analysis

package analysis

Ordering

Alphabetic

Visibility

Public
All

Value Members

object DetectAmbiguousSelfJoin extends Rule[LogicalPlan]
Detects ambiguous self-joins, so that we can fail the query instead of returning confusing results.
Detects ambiguous self-joins, so that we can fail the query instead of returning confusing results.
Dataset column reference is simply an AttributeReference that is returned by Dataset#col. Most of time we don't need to do anything special, as AttributeReference can point to the column precisely. However, in case of self-join, the analyzer generates AttributeReference with new expr IDs for the right side plan of the join. If the Dataset column reference points to a column in the right side plan of a self-join, users will get unexpected result because the column reference can't match the newly generated AttributeReference.
Note that, this rule removes all the Dataset id related metadata from AttributeReference, so that they don't exist after analyzer.

Packages

analysis 

package analysis

Value Members

Ungrouped

analysis