SparkSession

Value Members

final def !=(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def !=(arg0: Any): Boolean

Definition Classes
Any
final def ##(): Int

Definition Classes
AnyRef → Any
final def ==(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def ==(arg0: Any): Boolean

Definition Classes
Any
def applySchemaToPythonRDD(rdd: RDD[Array[Any]], schema: StructType): DataFrame

Apply a schema defined by the schema to an RDD.
Apply a schema defined by the schema to an RDD. It is only used by PySpark.

Attributes
protected[org.apache.spark.sql]
def applySchemaToPythonRDD(rdd: RDD[Array[Any]], schemaString: String): DataFrame

Apply a schema defined by the schemaString to an RDD.
Apply a schema defined by the schemaString to an RDD. It is only used by PySpark.

Attributes
protected[org.apache.spark.sql]
final def asInstanceOf[T0]: T0

Definition Classes
Any
def baseRelationToDataFrame(baseRelation: BaseRelation): DataFrame

Convert a BaseRelation created for external data sources into a DataFrame.
Convert a BaseRelation created for external data sources into a DataFrame.

Since
2.0.0
def cacheManager: CacheManager

Attributes
protected[org.apache.spark.sql]
lazy val catalog: Catalog

Interface through which the user may create, drop, alter or query underlying databases, tables, functions etc.
Interface through which the user may create, drop, alter or query underlying databases, tables, functions etc.

Since
2.0.0
def clone(): AnyRef

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( ... )
lazy val conf: RuntimeConfig

Runtime configuration interface for Spark.
Runtime configuration interface for Spark.
This is the interface through which the user can get and set all Spark and Hadoop configurations that are relevant to Spark SQL. When getting the value of a config, this defaults to the value set in the underlying SparkContext, if any.

Since
2.0.0
def createDataFrame(rowRDD: RDD[Row], schema: StructType, needsConversion: Boolean): DataFrame

Creates a DataFrame from an RDD[Row].
Creates a DataFrame from an RDD[Row]. User can specify whether the input rows should be converted to Catalyst rows.

Attributes
protected[org.apache.spark.sql]
def createDataFrame(data: List[_], beanClass: Class[_]): DataFrame

Applies a schema to an List of Java Beans.
Applies a schema to an List of Java Beans.
WARNING: Since there is no guaranteed ordering for fields in a Java Bean, SELECT * queries will return the columns in an undefined order.

Since
1.6.0
def createDataFrame(rdd: JavaRDD[_], beanClass: Class[_]): DataFrame

Applies a schema to an RDD of Java Beans.
Applies a schema to an RDD of Java Beans.
WARNING: Since there is no guaranteed ordering for fields in a Java Bean, SELECT * queries will return the columns in an undefined order.

Since
2.0.0
def createDataFrame(rdd: RDD[_], beanClass: Class[_]): DataFrame

Applies a schema to an RDD of Java Beans.
Applies a schema to an RDD of Java Beans.
WARNING: Since there is no guaranteed ordering for fields in a Java Bean, SELECT * queries will return the columns in an undefined order.

Since
2.0.0
def createDataFrame(rows: List[Row], schema: StructType): DataFrame

:: DeveloperApi :: Creates a DataFrame from an java.util.List containing Rows using the given schema.
:: DeveloperApi :: Creates a DataFrame from an java.util.List containing Rows using the given schema. It is important to make sure that the structure of every Row of the provided List matches the provided schema. Otherwise, there will be runtime exception.

Annotations
@DeveloperApi()
Since
2.0.0
def createDataFrame(rowRDD: JavaRDD[Row], schema: StructType): DataFrame

:: DeveloperApi :: Creates a DataFrame from an JavaRDD containing Rows using the given schema.
:: DeveloperApi :: Creates a DataFrame from an JavaRDD containing Rows using the given schema. It is important to make sure that the structure of every Row of the provided RDD matches the provided schema. Otherwise, there will be runtime exception.

Annotations
@DeveloperApi()
Since
2.0.0

def createDataFrame(rowRDD: RDD[Row], schema: StructType): DataFrame

:: DeveloperApi :: Creates a DataFrame from an RDD containing Rows using the given schema.

:: DeveloperApi :: Creates a DataFrame from an RDD containing Rows using the given schema. It is important to make sure that the structure of every Row of the provided RDD matches the provided schema. Otherwise, there will be runtime exception. Example:

import org.apache.spark.sql._
import org.apache.spark.sql.types._
val sparkSession = new org.apache.spark.sql.SparkSession(sc)

val schema =
  StructType(
    StructField("name", StringType, false) ::
    StructField("age", IntegerType, true) :: Nil)

val people =
  sc.textFile("examples/src/main/resources/people.txt").map(
    _.split(",")).map(p => Row(p(0), p(1).trim.toInt))
val dataFrame = sparkSession.createDataFrame(people, schema)
dataFrame.printSchema
// root
// |-- name: string (nullable = false)
// |-- age: integer (nullable = true)

dataFrame.createOrReplaceTempView("people")
sparkSession.sql("select name from people").collect.foreach(println)

Annotations: @DeveloperApi()
Since: 2.0.0

def createDataFrame[A <: Product](data: Seq[A])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[A]): DataFrame

:: Experimental :: Creates a DataFrame from a local Seq of Product.
:: Experimental :: Creates a DataFrame from a local Seq of Product.

Annotations
@Experimental()
Since
2.0.0
def createDataFrame[A <: Product](rdd: RDD[A])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[A]): DataFrame

:: Experimental :: Creates a DataFrame from an RDD of Product (e.
:: Experimental :: Creates a DataFrame from an RDD of Product (e.g. case classes, tuples).

Annotations
@Experimental()
Since
2.0.0
def createDataset[T](data: List[T])(implicit arg0: Encoder[T]): Dataset[T]
def createDataset[T](data: RDD[T])(implicit arg0: Encoder[T]): Dataset[T]
def createDataset[T](data: Seq[T])(implicit arg0: Encoder[T]): Dataset[T]
def createTempView(viewName: String, df: DataFrame, replaceIfExists: Boolean): Unit

Creates a temporary view with a DataFrame.
Creates a temporary view with a DataFrame. The lifetime of this temporary view is tied to this SparkSession.

Attributes
protected[org.apache.spark.sql]
lazy val emptyDataFrame: DataFrame

:: Experimental :: Returns a DataFrame with no rows or columns.
:: Experimental :: Returns a DataFrame with no rows or columns.

Since
2.0.0
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def equals(arg0: Any): Boolean

Definition Classes
AnyRef → Any
def executePlan(plan: LogicalPlan): QueryExecution

Attributes
protected[org.apache.spark.sql]
def executeSql(sql: String): QueryExecution

Attributes
protected[org.apache.spark.sql]
def experimental: ExperimentalMethods

:: Experimental :: A collection of methods that are considered experimental, but can be used to hook into the query planner for advanced functionality.
:: Experimental :: A collection of methods that are considered experimental, but can be used to hook into the query planner for advanced functionality.

Annotations
@Experimental()
Since
2.0.0
def externalCatalog: ExternalCatalog

Attributes
protected[org.apache.spark.sql]
def finalize(): Unit

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
def hashCode(): Int

Definition Classes
AnyRef → Any
object implicits extends SQLImplicits with Serializable

:: Experimental :: (Scala-specific) Implicit methods available in Scala for converting common Scala objects into DataFrames.
def initializeLogIfNecessary(isInterpreter: Boolean): Unit

Attributes
protected
Definition Classes
Logging
def internalCreateDataFrame(catalystRows: RDD[InternalRow], schema: StructType): DataFrame

Creates a DataFrame from an RDD[Row].
Creates a DataFrame from an RDD[Row]. User can specify whether the input rows should be converted to Catalyst rows.

Attributes
protected[org.apache.spark.sql]
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
def isTraceEnabled(): Boolean

Attributes
protected
Definition Classes
Logging
def listener: SQLListener

Attributes
protected[org.apache.spark.sql]
def listenerManager: ExecutionListenerManager

:: Experimental :: An interface to register custom org.apache.spark.sql.util.QueryExecutionListeners that listen for execution metrics.
:: Experimental :: An interface to register custom org.apache.spark.sql.util.QueryExecutionListeners that listen for execution metrics.

Annotations
@Experimental()
Since
2.0.0
def log: Logger

Attributes
protected
Definition Classes
Logging
def logDebug(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logDebug(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logError(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logError(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logInfo(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logInfo(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logName: String

Attributes
protected
Definition Classes
Logging
def logTrace(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logTrace(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logWarning(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logWarning(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def newSession(): SparkSession

Start a new session with isolated SQL configurations, temporary tables, registered functions are isolated, but sharing the underlying SparkContext and cached data.
Start a new session with isolated SQL configurations, temporary tables, registered functions are isolated, but sharing the underlying SparkContext and cached data.
Note: Other than the SparkContext, all shared state is initialized lazily. This method will force the initialization of the shared state to ensure that parent and child sessions are set up with the same shared state. If the underlying catalog implementation is Hive, this will initialize the metastore, which may take some time.

Since
2.0.0
final def notify(): Unit

Definition Classes
AnyRef
final def notifyAll(): Unit

Definition Classes
AnyRef
def parseDataType(dataTypeString: String): DataType

Parses the data type in our internal string representation.
Parses the data type in our internal string representation. The data type string should have the same format as the one generated by toString in scala. It is only used by PySpark.

Attributes
protected[org.apache.spark.sql]
def parseSql(sql: String): LogicalPlan

Attributes
protected[org.apache.spark.sql]
def range(start: Long, end: Long, step: Long, numPartitions: Int): Dataset[Long]

:: Experimental :: Creates a Dataset with a single LongType column named id, containing elements in an range from start to end (exclusive) with an step value, with partition number specified.
:: Experimental :: Creates a Dataset with a single LongType column named id, containing elements in an range from start to end (exclusive) with an step value, with partition number specified.

Annotations
@Experimental()
Since
2.0.0
def range(start: Long, end: Long, step: Long): Dataset[Long]

:: Experimental :: Creates a Dataset with a single LongType column named id, containing elements in an range from start to end (exclusive) with an step value.
:: Experimental :: Creates a Dataset with a single LongType column named id, containing elements in an range from start to end (exclusive) with an step value.

Annotations
@Experimental()
Since
2.0.0
def range(start: Long, end: Long): Dataset[Long]

:: Experimental :: Creates a Dataset with a single LongType column named id, containing elements in an range from start to end (exclusive) with step value 1.
:: Experimental :: Creates a Dataset with a single LongType column named id, containing elements in an range from start to end (exclusive) with step value 1.

Annotations
@Experimental()
Since
2.0.0
def range(end: Long): Dataset[Long]

:: Experimental :: Creates a Dataset with a single LongType column named id, containing elements in an range from 0 to end (exclusive) with step value 1.
:: Experimental :: Creates a Dataset with a single LongType column named id, containing elements in an range from 0 to end (exclusive) with step value 1.

Annotations
@Experimental()
Since
2.0.0
def read: DataFrameReader

:: Experimental :: Returns a DataFrameReader that can be used to read data and streams in as a DataFrame.
:: Experimental :: Returns a DataFrameReader that can be used to read data and streams in as a DataFrame.
```
sparkSession.read.parquet("/path/to/file.parquet")
sparkSession.read.schema(schema).json("/path/to/file.json")
```
Annotations
@Experimental()
Since
2.0.0
lazy val sessionState: SessionState

State isolated across sessions, including SQL configurations, temporary tables, registered functions, and everything else that accepts a org.apache.spark.sql.internal.SQLConf.
State isolated across sessions, including SQL configurations, temporary tables, registered functions, and everything else that accepts a org.apache.spark.sql.internal.SQLConf.

Attributes
protected[org.apache.spark.sql]
def setWrappedContext(sqlContext: SQLContext): Unit

Attributes
protected[org.apache.spark.sql]
lazy val sharedState: SharedState

State shared across sessions, including the SparkContext, cached data, listener, and a catalog that interacts with external systems.
State shared across sessions, including the SparkContext, cached data, listener, and a catalog that interacts with external systems.

Attributes
protected[org.apache.spark.sql]
val sparkContext: SparkContext
def sql(sqlText: String): DataFrame

Executes a SQL query using Spark, returning the result as a DataFrame.
Executes a SQL query using Spark, returning the result as a DataFrame. The dialect that is used for SQL parsing can be configured with 'spark.sql.dialect'.

Since
2.0.0
def stop(): Unit

Stop the underlying SparkContext.
Stop the underlying SparkContext.

Since
2.0.0
def streams: ContinuousQueryManager

Returns a ContinuousQueryManager that allows managing all the ContinuousQueries active on this.
Returns a ContinuousQueryManager that allows managing all the ContinuousQueries active on this.

Since
2.0.0
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
def table(tableIdent: TableIdentifier): DataFrame

Attributes
protected[org.apache.spark.sql]
def table(tableName: String): DataFrame

Returns the specified table as a DataFrame.
Returns the specified table as a DataFrame.

Since
2.0.0
def toString(): String

Definition Classes
AnyRef → Any

def udf: UDFRegistration

A collection of methods for registering user-defined functions (UDF).

The following example registers a Scala closure as UDF:

sparkSession.udf.register("myUDF", (arg1: Int, arg2: String) => arg2 + arg1)

The following example registers a UDF in Java:

sparkSession.udf().register("myUDF",
 new UDF2<Integer, String, String>() {
     @Override
     public String call(Integer arg1, String arg2) {
         return arg2 + arg1;
     }
}, DataTypes.StringType);

Or, to use Java 8 lambda syntax:

sparkSession.udf().register("myUDF",
(Integer arg1, String arg2) -> arg2 + arg1,
DataTypes.StringType);

Since: 2.0.0

final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
def wrapped: SQLContext

Attributes
protected[org.apache.spark.sql]

class SparkSession extends Serializable with Logging

Value Members

final def !=(arg0: AnyRef): Boolean

final def !=(arg0: Any): Boolean

final def ##(): Int

final def ==(arg0: AnyRef): Boolean

final def ==(arg0: Any): Boolean

def applySchemaToPythonRDD(rdd: RDD[Array[Any]], schema: StructType): DataFrame

def applySchemaToPythonRDD(rdd: RDD[Array[Any]], schemaString: String): DataFrame

final def asInstanceOf[T0]: T0

def baseRelationToDataFrame(baseRelation: BaseRelation): DataFrame

def cacheManager: CacheManager

lazy val catalog: Catalog

def clone(): AnyRef

lazy val conf: RuntimeConfig

def createDataFrame(rowRDD: RDD[Row], schema: StructType, needsConversion: Boolean): DataFrame

def createDataFrame(data: List[_], beanClass: Class[_]): DataFrame

def createDataFrame(rdd: JavaRDD[_], beanClass: Class[_]): DataFrame

def createDataFrame(rdd: RDD[_], beanClass: Class[_]): DataFrame

def createDataFrame(rows: List[Row], schema: StructType): DataFrame

def createDataFrame(rowRDD: JavaRDD[Row], schema: StructType): DataFrame

def createDataFrame(rowRDD: RDD[Row], schema: StructType): DataFrame

def createDataFrame[A <: Product](data: Seq[A])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[A]): DataFrame

def createDataFrame[A <: Product](rdd: RDD[A])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[A]): DataFrame

def createDataset[T](data: List[T])(implicit arg0: Encoder[T]): Dataset[T]

def createDataset[T](data: RDD[T])(implicit arg0: Encoder[T]): Dataset[T]

def createDataset[T](data: Seq[T])(implicit arg0: Encoder[T]): Dataset[T]

def createTempView(viewName: String, df: DataFrame, replaceIfExists: Boolean): Unit

lazy val emptyDataFrame: DataFrame

final def eq(arg0: AnyRef): Boolean

def equals(arg0: Any): Boolean

def executePlan(plan: LogicalPlan): QueryExecution

def executeSql(sql: String): QueryExecution

def experimental: ExperimentalMethods

def externalCatalog: ExternalCatalog

def finalize(): Unit

final def getClass(): Class[_]

def hashCode(): Int

object implicits extends SQLImplicits with Serializable

def initializeLogIfNecessary(isInterpreter: Boolean): Unit

def internalCreateDataFrame(catalystRows: RDD[InternalRow], schema: StructType): DataFrame

final def isInstanceOf[T0]: Boolean

def isTraceEnabled(): Boolean

def listener: SQLListener

def listenerManager: ExecutionListenerManager

def log: Logger

def logDebug(msg: ⇒ String, throwable: Throwable): Unit

def logDebug(msg: ⇒ String): Unit

def logError(msg: ⇒ String, throwable: Throwable): Unit

def logError(msg: ⇒ String): Unit

def logInfo(msg: ⇒ String, throwable: Throwable): Unit

def logInfo(msg: ⇒ String): Unit

def logName: String

def logTrace(msg: ⇒ String, throwable: Throwable): Unit

def logTrace(msg: ⇒ String): Unit

def logWarning(msg: ⇒ String, throwable: Throwable): Unit

def logWarning(msg: ⇒ String): Unit

final def ne(arg0: AnyRef): Boolean

def newSession(): SparkSession

final def notify(): Unit

final def notifyAll(): Unit

def parseDataType(dataTypeString: String): DataType

def parseSql(sql: String): LogicalPlan

def range(start: Long, end: Long, step: Long, numPartitions: Int): Dataset[Long]

def range(start: Long, end: Long, step: Long): Dataset[Long]

def range(start: Long, end: Long): Dataset[Long]

def range(end: Long): Dataset[Long]

def read: DataFrameReader

lazy val sessionState: SessionState

def setWrappedContext(sqlContext: SQLContext): Unit

lazy val sharedState: SharedState

val sparkContext: SparkContext

def sql(sqlText: String): DataFrame

def stop(): Unit

def streams: ContinuousQueryManager

final def synchronized[T0](arg0: ⇒ T0): T0

def table(tableIdent: TableIdentifier): DataFrame

def table(tableName: String): DataFrame

def toString(): String