sources

Type Members

1. case class And(left: Filter, right: Filter) extends Filter with Product with Serializable

   

2. abstract class BaseRelation extends AnyRef

   

   ::DeveloperApi:: Represents a collection of tuples with a known schema.

   ::DeveloperApi:: Represents a collection of tuples with a known schema. Classes that extend BaseRelation must be able to produce the schema of their data in the form of a StructType. Concrete implementation should inherit from one of the descendant Scan classes, which define various abstract methods for execution.

   BaseRelations must also define a equality function that only returns true when the two instances will return the same data. This equality function is used when determining when it is safe to substitute cached results for a given relation.

   Annotations

   @DeveloperApi()

3. class CaseInsensitiveMap extends Map[String, String] with Serializable

   

   Builds a map in which keys are case insensitive

   Builds a map in which keys are case insensitive

   Attributes

   protected[org.apache.spark.sql]

4. trait CatalystScan extends AnyRef

   

   ::Experimental:: An interface for experimenting with a more direct connection to the query planner.

   ::Experimental:: An interface for experimenting with a more direct connection to the query planner. Compared to PrunedFilteredScan, this operator receives the raw expressions from the org.apache.spark.sql.catalyst.plans.logical.LogicalPlan. Unlike the other APIs this interface is NOT designed to be binary compatible across releases and thus should only be used for experimentation.

   Annotations

   @Experimental()

5. trait CreatableRelationProvider extends AnyRef

   

   Annotations

   @DeveloperApi()

6. class DDLException extends Exception

   

   The exception thrown from the DDL parser.

   The exception thrown from the DDL parser.

   Attributes

   protected[org.apache.spark.sql]

7. case class EqualTo(attribute: String, value: Any) extends Filter with Product with Serializable

   

8. abstract class Filter extends AnyRef

   

9. case class GreaterThan(attribute: String, value: Any) extends Filter with Product with Serializable

   

10. case class GreaterThanOrEqual(attribute: String, value: Any) extends Filter with Product with Serializable

    

11. case class In(attribute: String, values: Array[Any]) extends Filter with Product with Serializable

    

12. trait InsertableRelation extends AnyRef

    

    ::DeveloperApi:: A BaseRelation that can be used to insert data into it through the insert method.

    ::DeveloperApi:: A BaseRelation that can be used to insert data into it through the insert method. If overwrite in insert method is true, the old data in the relation should be overwritten with the new data. If overwrite in insert method is false, the new data should be appended.

    InsertableRelation has the following three assumptions. 1. It assumes that the data (Rows in the DataFrame) provided to the insert method exactly matches the ordinal of fields in the schema of the BaseRelation. 2. It assumes that the schema of this relation will not be changed. Even if the insert method updates the schema (e.g. a relation of JSON or Parquet data may have a schema update after an insert operation), the new schema will not be used. 3. It assumes that fields of the data provided in the insert method are nullable. If a data source needs to check the actual nullability of a field, it needs to do it in the insert method.

    Annotations

    @DeveloperApi()

13. case class IsNotNull(attribute: String) extends Filter with Product with Serializable

    

14. case class IsNull(attribute: String) extends Filter with Product with Serializable

    

15. case class LessThan(attribute: String, value: Any) extends Filter with Product with Serializable

    

16. case class LessThanOrEqual(attribute: String, value: Any) extends Filter with Product with Serializable

    

17. case class Not(child: Filter) extends Filter with Product with Serializable

    

18. case class Or(left: Filter, right: Filter) extends Filter with Product with Serializable

    

19. trait PrunedFilteredScan extends AnyRef

    

    ::DeveloperApi:: A BaseRelation that can eliminate unneeded columns and filter using selected predicates before producing an RDD containing all matching tuples as Row objects.

    ::DeveloperApi:: A BaseRelation that can eliminate unneeded columns and filter using selected predicates before producing an RDD containing all matching tuples as Row objects.

    The pushed down filters are currently purely an optimization as they will all be evaluated again. This means it is safe to use them with methods that produce false positives such as filtering partitions based on a bloom filter.

    Annotations

    @DeveloperApi()

20. trait PrunedScan extends AnyRef

    

    ::DeveloperApi:: A BaseRelation that can eliminate unneeded columns before producing an RDD containing all of its tuples as Row objects.

    ::DeveloperApi:: A BaseRelation that can eliminate unneeded columns before producing an RDD containing all of its tuples as Row objects.

    Annotations

    @DeveloperApi()

21. trait RelationProvider extends AnyRef

    

    ::DeveloperApi:: Implemented by objects that produce relations for a specific kind of data source.

    ::DeveloperApi:: Implemented by objects that produce relations for a specific kind of data source. When Spark SQL is given a DDL operation with a USING clause specified (to specify the implemented RelationProvider), this interface is used to pass in the parameters specified by a user.

    Users may specify the fully qualified class name of a given data source. When that class is not found Spark SQL will append the class name DefaultSource to the path, allowing for less verbose invocation. For example, 'org.apache.spark.sql.json' would resolve to the data source 'org.apache.spark.sql.json.DefaultSource'

    A new instance of this class with be instantiated each time a DDL call is made.

    Annotations

    @DeveloperApi()

22. trait SchemaRelationProvider extends AnyRef

    

    ::DeveloperApi:: Implemented by objects that produce relations for a specific kind of data source with a given schema.

    ::DeveloperApi:: Implemented by objects that produce relations for a specific kind of data source with a given schema. When Spark SQL is given a DDL operation with a USING clause specified ( to specify the implemented SchemaRelationProvider) and a user defined schema, this interface is used to pass in the parameters specified by a user.

    Users may specify the fully qualified class name of a given data source. When that class is not found Spark SQL will append the class name DefaultSource to the path, allowing for less verbose invocation. For example, 'org.apache.spark.sql.json' would resolve to the data source 'org.apache.spark.sql.json.DefaultSource'

    A new instance of this class with be instantiated each time a DDL call is made.

    The difference between a RelationProvider and a SchemaRelationProvider is that users need to provide a schema when using a SchemaRelationProvider. A relation provider can inherits both RelationProvider and SchemaRelationProvider if it can support both schema inference and user-specified schemas.

    Annotations

    @DeveloperApi()

23. trait TableScan extends AnyRef

    

    ::DeveloperApi:: A BaseRelation that can produce all of its tuples as an RDD of Row objects.

    ::DeveloperApi:: A BaseRelation that can produce all of its tuples as an RDD of Row objects.

    Annotations

    @DeveloperApi()