sources

Type Members

1. abstract class BaseRelation extends AnyRef

   

   ::DeveloperApi:: Represents a collection of tuples with a known schema.

   ::DeveloperApi:: Represents a collection of tuples with a known schema. Classes that extend BaseRelation must be able to produce the schema of their data in the form of a StructType Concrete implementation should inherit from one of the descendant Scan classes, which define various abstract methods for execution.

   BaseRelations must also define a equality function that only returns true when the two instances will return the same data. This equality function is used when determining when it is safe to substitute cached results for a given relation.

   Annotations

   @DeveloperApi()

2. abstract class CatalystScan extends BaseRelation

   

   ::Experimental:: An interface for experimenting with a more direct connection to the query planner.

   ::Experimental:: An interface for experimenting with a more direct connection to the query planner. Compared to PrunedFilteredScan, this operator receives the raw expressions from the org.apache.spark.sql.catalyst.plans.logical.LogicalPlan. Unlike the other APIs this interface is not designed to be binary compatible across releases and thus should only be used for experimentation.

   Annotations

   @Experimental()

3. case class EqualTo(attribute: String, value: Any) extends Filter with Product with Serializable

   

4. abstract class Filter extends AnyRef

   

5. case class GreaterThan(attribute: String, value: Any) extends Filter with Product with Serializable

   

6. case class GreaterThanOrEqual(attribute: String, value: Any) extends Filter with Product with Serializable

   

7. case class In(attribute: String, values: Array[Any]) extends Filter with Product with Serializable

   

8. case class LessThan(attribute: String, value: Any) extends Filter with Product with Serializable

   

9. case class LessThanOrEqual(attribute: String, value: Any) extends Filter with Product with Serializable

   

10. abstract class PrunedFilteredScan extends BaseRelation

    

    ::DeveloperApi:: A BaseRelation that can eliminate unneeded columns and filter using selected predicates before producing an RDD containing all matching tuples as Row objects.

    ::DeveloperApi:: A BaseRelation that can eliminate unneeded columns and filter using selected predicates before producing an RDD containing all matching tuples as Row objects.

    The pushed down filters are currently purely an optimization as they will all be evaluated again. This means it is safe to use them with methods that produce false positives such as filtering partitions based on a bloom filter.

    Annotations

    @DeveloperApi()

11. abstract class PrunedScan extends BaseRelation

    

    ::DeveloperApi:: A BaseRelation that can eliminate unneeded columns before producing an RDD containing all of its tuples as Row objects.

    ::DeveloperApi:: A BaseRelation that can eliminate unneeded columns before producing an RDD containing all of its tuples as Row objects.

    Annotations

    @DeveloperApi()

12. trait RelationProvider extends AnyRef

    

    ::DeveloperApi:: Implemented by objects that produce relations for a specific kind of data source.

    ::DeveloperApi:: Implemented by objects that produce relations for a specific kind of data source. When Spark SQL is given a DDL operation with a USING clause specified, this interface is used to pass in the parameters specified by a user.

    Users may specify the fully qualified class name of a given data source. When that class is not found Spark SQL will append the class name DefaultSource to the path, allowing for less verbose invocation. For example, 'org.apache.spark.sql.json' would resolve to the data source 'org.apache.spark.sql.json.DefaultSource'

    A new instance of this class with be instantiated each time a DDL call is made.

    Annotations

    @DeveloperApi()

13. abstract class TableScan extends BaseRelation

    

    ::DeveloperApi:: A BaseRelation that can produce all of its tuples as an RDD of Row objects.

    ::DeveloperApi:: A BaseRelation that can produce all of its tuples as an RDD of Row objects.

    Annotations

    @DeveloperApi()