SQLContext

Value Members

1. final def !=(arg0: AnyRef): Boolean

   Definition Classes

   AnyRef

2. final def !=(arg0: Any): Boolean

   Definition Classes

   Any

3. final def ##(): Int

   Definition Classes

   AnyRef → Any

4. final def ==(arg0: AnyRef): Boolean

   Definition Classes

   AnyRef

5. final def ==(arg0: Any): Boolean

   Definition Classes

   Any

6. def applySchemaToPythonRDD(rdd: RDD[Array[Any]], schema: StructType): DataFrame

   Apply a schema defined by the schema to an RDD.

   Apply a schema defined by the schema to an RDD. It is only used by PySpark.

   Attributes

   protected[org.apache.spark.sql]

7. def applySchemaToPythonRDD(rdd: RDD[Array[Any]], schemaString: String): DataFrame

   Apply a schema defined by the schemaString to an RDD.

   Apply a schema defined by the schemaString to an RDD. It is only used by PySpark.

   Attributes

   protected[org.apache.spark.sql]

8. final def asInstanceOf[T0]: T0

   Definition Classes

   Any

9. def baseRelationToDataFrame(baseRelation: BaseRelation): DataFrame

   Convert a BaseRelation created for external data sources into a DataFrame.

   Convert a BaseRelation created for external data sources into a DataFrame.

   Since

   1.3.0

10. def cacheManager: CacheManager

    Attributes

    protected[org.apache.spark.sql]

11. def cacheTable(tableName: String): Unit

    Caches the specified table in-memory.

    Caches the specified table in-memory.

    Since

    1.3.0

12. def clearCache(): Unit

    Removes all cached tables from the in-memory cache.

    Removes all cached tables from the in-memory cache.

    Since

    1.3.0

13. def clone(): AnyRef

    Attributes

    protected[java.lang]

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

14. def conf: SQLConf

    Attributes

    protected[org.apache.spark.sql]

15. def createDataFrame(data: List[_], beanClass: Class[_]): DataFrame

    Applies a schema to an List of Java Beans.

    Applies a schema to an List of Java Beans.

    WARNING: Since there is no guaranteed ordering for fields in a Java Bean, SELECT * queries will return the columns in an undefined order.

    Since

    1.6.0

16. def createDataFrame(rdd: JavaRDD[_], beanClass: Class[_]): DataFrame

    Applies a schema to an RDD of Java Beans.

    Applies a schema to an RDD of Java Beans.

    WARNING: Since there is no guaranteed ordering for fields in a Java Bean, SELECT * queries will return the columns in an undefined order.

    Since

    1.3.0

17. def createDataFrame(rdd: RDD[_], beanClass: Class[_]): DataFrame

    Applies a schema to an RDD of Java Beans.

    Applies a schema to an RDD of Java Beans.

    WARNING: Since there is no guaranteed ordering for fields in a Java Bean, SELECT * queries will return the columns in an undefined order.

    Since

    1.3.0

18. def createDataFrame(rows: List[Row], schema: StructType): DataFrame

    :: DeveloperApi :: Creates a DataFrame from an java.util.List containing Rows using the given schema.

    :: DeveloperApi :: Creates a DataFrame from an java.util.List containing Rows using the given schema. It is important to make sure that the structure of every Row of the provided List matches the provided schema. Otherwise, there will be runtime exception.

    Annotations

    @DeveloperApi()

    Since

    1.6.0

19. def createDataFrame(rowRDD: JavaRDD[Row], schema: StructType): DataFrame

    :: DeveloperApi :: Creates a DataFrame from an JavaRDD containing Rows using the given schema.

    :: DeveloperApi :: Creates a DataFrame from an JavaRDD containing Rows using the given schema. It is important to make sure that the structure of every Row of the provided RDD matches the provided schema. Otherwise, there will be runtime exception.

    Annotations

    @DeveloperApi()

    Since

    1.3.0

20. def createDataFrame(rowRDD: RDD[Row], schema: StructType): DataFrame

    :: DeveloperApi :: Creates a DataFrame from an RDD containing Rows using the given schema.

    :: DeveloperApi :: Creates a DataFrame from an RDD containing Rows using the given schema. It is important to make sure that the structure of every Row of the provided RDD matches the provided schema. Otherwise, there will be runtime exception. Example:

    import org.apache.spark.sql._
    import org.apache.spark.sql.types._
    val sqlContext = new org.apache.spark.sql.SQLContext(sc)
    
    val schema =
      StructType(
        StructField("name", StringType, false) ::
        StructField("age", IntegerType, true) :: Nil)
    
    val people =
      sc.textFile("examples/src/main/resources/people.txt").map(
        _.split(",")).map(p => Row(p(0), p(1).trim.toInt))
    val dataFrame = sqlContext.createDataFrame(people, schema)
    dataFrame.printSchema
    // root
    // |-- name: string (nullable = false)
    // |-- age: integer (nullable = true)
    
    dataFrame.createOrReplaceTempView("people")
    sqlContext.sql("select name from people").collect.foreach(println)

    Annotations

    @DeveloperApi()

    Since

    1.3.0

21. def createDataFrame[A <: Product](data: Seq[A])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[A]): DataFrame

    :: Experimental :: Creates a DataFrame from a local Seq of Product.

    :: Experimental :: Creates a DataFrame from a local Seq of Product.

    Annotations

    @Experimental()

    Since

    1.3.0

22. def createDataFrame[A <: Product](rdd: RDD[A])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[A]): DataFrame

    :: Experimental :: Creates a DataFrame from an RDD of Product (e.

    :: Experimental :: Creates a DataFrame from an RDD of Product (e.g. case classes, tuples).

    Annotations

    @Experimental()

    Since

    1.3.0

23. def createDataset[T](data: List[T])(implicit arg0: Encoder[T]): Dataset[T]

24. def createDataset[T](data: RDD[T])(implicit arg0: Encoder[T]): Dataset[T]

25. def createDataset[T](data: Seq[T])(implicit arg0: Encoder[T]): Dataset[T]

26. def createExternalTable(tableName: String, source: String, schema: StructType, options: Map[String, String]): DataFrame

    :: Experimental :: (Scala-specific) Create an external table from the given path based on a data source, a schema and a set of options.

    :: Experimental :: (Scala-specific) Create an external table from the given path based on a data source, a schema and a set of options. Then, returns the corresponding DataFrame.

    Annotations

    @Experimental()

    Since

    1.3.0

27. def createExternalTable(tableName: String, source: String, schema: StructType, options: Map[String, String]): DataFrame

    :: Experimental :: Create an external table from the given path based on a data source, a schema and a set of options.

    :: Experimental :: Create an external table from the given path based on a data source, a schema and a set of options. Then, returns the corresponding DataFrame.

    Annotations

    @Experimental()

    Since

    1.3.0

28. def createExternalTable(tableName: String, source: String, options: Map[String, String]): DataFrame

    :: Experimental :: (Scala-specific) Creates an external table from the given path based on a data source and a set of options.

    :: Experimental :: (Scala-specific) Creates an external table from the given path based on a data source and a set of options. Then, returns the corresponding DataFrame.

    Annotations

    @Experimental()

    Since

    1.3.0

29. def createExternalTable(tableName: String, source: String, options: Map[String, String]): DataFrame

    :: Experimental :: Creates an external table from the given path based on a data source and a set of options.

    :: Experimental :: Creates an external table from the given path based on a data source and a set of options. Then, returns the corresponding DataFrame.

    Annotations

    @Experimental()

    Since

    1.3.0

30. def createExternalTable(tableName: String, path: String, source: String): DataFrame

    :: Experimental :: Creates an external table from the given path based on a data source and returns the corresponding DataFrame.

    :: Experimental :: Creates an external table from the given path based on a data source and returns the corresponding DataFrame.

    Annotations

    @Experimental()

    Since

    1.3.0

31. def createExternalTable(tableName: String, path: String): DataFrame

    :: Experimental :: Creates an external table from the given path and returns the corresponding DataFrame.

    :: Experimental :: Creates an external table from the given path and returns the corresponding DataFrame. It will use the default data source configured by spark.sql.sources.default.

    Annotations

    @Experimental()

    Since

    1.3.0

32. def dropTempTable(tableName: String): Unit

    Drops the temporary table with the given table name in the catalog.

    Drops the temporary table with the given table name in the catalog. If the table has been cached/persisted before, it's also unpersisted.

    tableName

    the name of the table to be unregistered.

    Since

    1.3.0

33. def emptyDataFrame: DataFrame

    :: Experimental :: Returns a DataFrame with no rows or columns.

    :: Experimental :: Returns a DataFrame with no rows or columns.

    Annotations

    @Experimental()

    Since

    1.3.0

34. final def eq(arg0: AnyRef): Boolean

    Definition Classes

    AnyRef

35. def equals(arg0: Any): Boolean

    Definition Classes

    AnyRef → Any

36. def executePlan(plan: LogicalPlan): QueryExecution

    Attributes

    protected[org.apache.spark.sql]

37. def executeSql(sql: String): QueryExecution

    Attributes

    protected[org.apache.spark.sql]

38. def experimental: ExperimentalMethods

    :: Experimental :: A collection of methods that are considered experimental, but can be used to hook into the query planner for advanced functionality.

    :: Experimental :: A collection of methods that are considered experimental, but can be used to hook into the query planner for advanced functionality.

    Annotations

    @Experimental() @transient()

    Since

    1.3.0

39. def externalCatalog: ExternalCatalog

    Attributes

    protected[org.apache.spark.sql]

40. def finalize(): Unit

    Attributes

    protected[java.lang]

    Definition Classes

    AnyRef

    Annotations

    @throws( classOf[java.lang.Throwable] )

41. def getAllConfs: Map[String, String]

    Return all the configuration properties that have been set (i.

    Return all the configuration properties that have been set (i.e. not the default). This creates a new copy of the config properties in the form of a Map.

    Since

    1.0.0

42. final def getClass(): Class[_]

    Definition Classes

    AnyRef → Any

43. def getConf(key: String, defaultValue: String): String

    Return the value of Spark SQL configuration property for the given key.

    Return the value of Spark SQL configuration property for the given key. If the key is not set yet, return defaultValue.

    Since

    1.0.0

44. def getConf(key: String): String

    Return the value of Spark SQL configuration property for the given key.

    Return the value of Spark SQL configuration property for the given key.

    Since

    1.0.0

45. def hashCode(): Int

    Definition Classes

    AnyRef → Any

46. object implicits extends SQLImplicits with Serializable

    :: Experimental :: (Scala-specific) Implicit methods available in Scala for converting common Scala objects into DataFrames.

47. def initializeLogIfNecessary(isInterpreter: Boolean): Unit

    Attributes

    protected

    Definition Classes

    Logging

48. def isCached(tableName: String): Boolean

    Returns true if the table is currently cached in-memory.

    Returns true if the table is currently cached in-memory.

    Since

    1.3.0

49. final def isInstanceOf[T0]: Boolean

    Definition Classes

    Any

50. val isRootContext: Boolean

51. def isTraceEnabled(): Boolean

    Attributes

    protected

    Definition Classes

    Logging

52. def listener: SQLListener

    Attributes

    protected[org.apache.spark.sql]

53. def listenerManager: ExecutionListenerManager

    An interface to register custom org.apache.spark.sql.util.QueryExecutionListeners that listen for execution metrics.

    An interface to register custom org.apache.spark.sql.util.QueryExecutionListeners that listen for execution metrics.

    Annotations

    @Experimental()

54. def log: Logger

    Attributes

    protected

    Definition Classes

    Logging

55. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    Logging

56. def logDebug(msg: ⇒ String): Unit

    Attributes

    protected

    Definition Classes

    Logging

57. def logError(msg: ⇒ String, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    Logging

58. def logError(msg: ⇒ String): Unit

    Attributes

    protected

    Definition Classes

    Logging

59. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    Logging

60. def logInfo(msg: ⇒ String): Unit

    Attributes

    protected

    Definition Classes

    Logging

61. def logName: String

    Attributes

    protected

    Definition Classes

    Logging

62. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    Logging

63. def logTrace(msg: ⇒ String): Unit

    Attributes

    protected

    Definition Classes

    Logging

64. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    Logging

65. def logWarning(msg: ⇒ String): Unit

    Attributes

    protected

    Definition Classes

    Logging

66. final def ne(arg0: AnyRef): Boolean

    Definition Classes

    AnyRef

67. def newSession(): SQLContext

    Returns a SQLContext as new session, with separated SQL configurations, temporary tables, registered functions, but sharing the same SparkContext, cached data and other things.

    Returns a SQLContext as new session, with separated SQL configurations, temporary tables, registered functions, but sharing the same SparkContext, cached data and other things.

    Since

    1.6.0

68. final def notify(): Unit

    Definition Classes

    AnyRef

69. final def notifyAll(): Unit

    Definition Classes

    AnyRef

70. def parseDataType(dataTypeString: String): DataType

    Parses the data type in our internal string representation.

    Parses the data type in our internal string representation. The data type string should have the same format as the one generated by toString in scala. It is only used by PySpark.

    Attributes

    protected[org.apache.spark.sql]

71. def parseSql(sql: String): LogicalPlan

    Attributes

    protected[org.apache.spark.sql]

72. def range(start: Long, end: Long, step: Long, numPartitions: Int): Dataset[Long]

    :: Experimental :: Creates a Dataset with a single LongType column named id, containing elements in an range from start to end (exclusive) with an step value, with partition number specified.

    :: Experimental :: Creates a Dataset with a single LongType column named id, containing elements in an range from start to end (exclusive) with an step value, with partition number specified.

    Annotations

    @Experimental()

    Since

    2.0.0

73. def range(start: Long, end: Long, step: Long): Dataset[Long]

    :: Experimental :: Creates a Dataset with a single LongType column named id, containing elements in an range from start to end (exclusive) with an step value.

    :: Experimental :: Creates a Dataset with a single LongType column named id, containing elements in an range from start to end (exclusive) with an step value.

    Annotations

    @Experimental()

    Since

    2.0.0

74. def range(start: Long, end: Long): Dataset[Long]

    :: Experimental :: Creates a Dataset with a single LongType column named id, containing elements in an range from start to end (exclusive) with step value 1.

    :: Experimental :: Creates a Dataset with a single LongType column named id, containing elements in an range from start to end (exclusive) with step value 1.

    Annotations

    @Experimental()

    Since

    2.0.0

75. def range(end: Long): Dataset[Long]

    :: Experimental :: Creates a Dataset with a single LongType column named id, containing elements in an range from 0 to end (exclusive) with step value 1.

    :: Experimental :: Creates a Dataset with a single LongType column named id, containing elements in an range from 0 to end (exclusive) with step value 1.

    Annotations

    @Experimental()

    Since

    2.0.0

76. def read: DataFrameReader

    :: Experimental :: Returns a DataFrameReader that can be used to read data and streams in as a DataFrame.

    :: Experimental :: Returns a DataFrameReader that can be used to read data and streams in as a DataFrame.

    sqlContext.read.parquet("/path/to/file.parquet")
    sqlContext.read.schema(schema).json("/path/to/file.json")

    Annotations

    @Experimental()

    Since

    1.4.0

77. def runtimeConf: RuntimeConfig

    Attributes

    protected[org.apache.spark.sql]

78. def sessionState: SessionState

    Attributes

    protected[org.apache.spark.sql]

79. def setConf(key: String, value: String): Unit

    Set the given Spark SQL configuration property.

    Set the given Spark SQL configuration property.

    Since

    1.0.0

80. def setConf(props: Properties): Unit

    Set Spark SQL configuration properties.

    Set Spark SQL configuration properties.

    Since

    1.0.0

81. def sharedState: SharedState

    Attributes

    protected[org.apache.spark.sql]

82. def sparkContext: SparkContext

83. val sparkSession: SparkSession

84. def sql(sqlText: String): DataFrame

    Executes a SQL query using Spark, returning the result as a DataFrame.

    Executes a SQL query using Spark, returning the result as a DataFrame. The dialect that is used for SQL parsing can be configured with 'spark.sql.dialect'.

    Since

    1.3.0

85. def streams: ContinuousQueryManager

    Returns a ContinuousQueryManager that allows managing all the ContinuousQueries active on this context.

    Returns a ContinuousQueryManager that allows managing all the ContinuousQueries active on this context.

    Since

    2.0.0

86. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes

    AnyRef

87. def table(tableName: String): DataFrame

    Returns the specified table as a DataFrame.

    Returns the specified table as a DataFrame.

    Since

    1.3.0

88. def tableNames(databaseName: String): Array[String]

    Returns the names of tables in the given database as an array.

    Returns the names of tables in the given database as an array.

    Since

    1.3.0

89. def tableNames(): Array[String]

    Returns the names of tables in the current database as an array.

    Returns the names of tables in the current database as an array.

    Since

    1.3.0

90. def tables(databaseName: String): DataFrame

    Returns a DataFrame containing names of existing tables in the given database.

    Returns a DataFrame containing names of existing tables in the given database. The returned DataFrame has two columns, tableName and isTemporary (a Boolean indicating if a table is a temporary one or not).

    Since

    1.3.0

91. def tables(): DataFrame

    Returns a DataFrame containing names of existing tables in the current database.

    Returns a DataFrame containing names of existing tables in the current database. The returned DataFrame has two columns, tableName and isTemporary (a Boolean indicating if a table is a temporary one or not).

    Since

    1.3.0

92. def toString(): String

    Definition Classes

    AnyRef → Any

93. def udf: UDFRegistration

    A collection of methods for registering user-defined functions (UDF).

    A collection of methods for registering user-defined functions (UDF).

    The following example registers a Scala closure as UDF:

    sqlContext.udf.register("myUDF", (arg1: Int, arg2: String) => arg2 + arg1)

    The following example registers a UDF in Java:

    sqlContext.udf().register("myUDF",
     new UDF2<Integer, String, String>() {
         @Override
         public String call(Integer arg1, String arg2) {
             return arg2 + arg1;
         }
    }, DataTypes.StringType);

    Or, to use Java 8 lambda syntax:

    sqlContext.udf().register("myUDF",
    (Integer arg1, String arg2) -> arg2 + arg1,
    DataTypes.StringType);

    Since

    1.3.0

94. def uncacheTable(tableName: String): Unit

    Removes the specified table from the in-memory cache.

    Removes the specified table from the in-memory cache.

    Since

    1.3.0

95. final def wait(): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

96. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

97. final def wait(arg0: Long): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )