datasources

Type Members

1. case class AnalyzeCreateTable(sparkSession: SparkSession) extends Rule[LogicalPlan] with Product with Serializable

   Analyze CreateTable and do some normalization and checking.

2. class CatalogFileIndex extends FileIndex

   A FileIndex for a metastore catalog table.

3. case class CreateTable(tableDesc: CatalogTable, mode: SaveMode, query: Option[LogicalPlan]) extends LeafNode with Command with Product with Serializable

4. case class CreateTempViewUsing(tableIdent: TableIdentifier, userSpecifiedSchema: Option[StructType], replace: Boolean, global: Boolean, provider: String, options: Map[String, String]) extends LeafNode with RunnableCommand with Product with Serializable

   Create or replace a local/global temporary view with given data source.

5. case class DataSource(sparkSession: SparkSession, className: String, paths: Seq[String] = immutable.this.Nil, userSpecifiedSchema: Option[StructType] = scala.None, partitionColumns: Seq[String] = collection.this.Seq.empty[Nothing], bucketSpec: Option[BucketSpec] = scala.None, options: Map[String, String] = ..., catalogTable: Option[CatalogTable] = scala.None) extends Logging with Product with Serializable

   The main class responsible for representing a pluggable Data Source in Spark SQL.

6. case class DataSourceAnalysis(conf: CatalystConf) extends Rule[LogicalPlan] with Product with Serializable

   Replaces generic operations with specific variants that are designed to work with Spark SQL Data Sources.

7. trait FileFormat extends AnyRef

   Used to read and write data stored in files to/from the InternalRow format.

8. trait FileIndex extends AnyRef

   An interface for objects capable of enumerating the root paths of a relation as well as the partitions of a relation subject to some pruning expressions.

9. case class FilePartition(index: Int, files: Seq[PartitionedFile]) extends Partition with Product with Serializable

   A collection of file blocks that should be read as a single task (possibly from multiple partitioned directories).

10. class FileScanRDD extends RDD[InternalRow]

    An RDD that scans a list of file partitions.

11. abstract class FileStatusCache extends AnyRef

    A cache of the leaf files of partition directories.

12. class FindDataSourceTable extends Rule[LogicalPlan]

    Replaces SimpleCatalogRelation with data source table if its table property contains data source information.

13. class HadoopFileLinesReader extends Iterator[Text] with Closeable

    An adaptor from a PartitionedFile to an Iterator of Text, which are all of the lines in that file.

14. case class HadoopFsRelation(location: FileIndex, partitionSchema: StructType, dataSchema: StructType, bucketSpec: Option[BucketSpec], fileFormat: FileFormat, options: Map[String, String])(sparkSession: SparkSession) extends BaseRelation with FileRelation with Product with Serializable

    Acts as a container for all of the metadata required to read from a datasource.

15. class InMemoryFileIndex extends PartitioningAwareFileIndex

    A FileIndex that generates the list of files to process by recursively listing all the files present in paths.

16. case class InsertIntoDataSourceCommand(logicalRelation: LogicalRelation, query: LogicalPlan, overwrite: OverwriteOptions) extends LeafNode with RunnableCommand with Product with Serializable

    Inserts the results of query in to a relation that extends InsertableRelation.

17. case class InsertIntoHadoopFsRelationCommand(outputPath: Path, staticPartitionKeys: TablePartitionSpec, customPartitionLocations: Map[TablePartitionSpec, String], partitionColumns: Seq[Attribute], bucketSpec: Option[BucketSpec], fileFormat: FileFormat, refreshFunction: (Seq[TablePartitionSpec]) ⇒ Unit, options: Map[String, String], query: LogicalPlan, mode: SaveMode, catalogTable: Option[CatalogTable]) extends LeafNode with RunnableCommand with Product with Serializable

    A command for writing data to a HadoopFsRelation.

18. case class LogicalRelation(relation: BaseRelation, expectedOutputAttributes: Option[Seq[Attribute]] = scala.None, catalogTable: Option[CatalogTable] = scala.None) extends LeafNode with MultiInstanceRelation with Product with Serializable

    Used to link a BaseRelation in to a logical query plan.

19. abstract class OutputWriter extends AnyRef

    OutputWriter is used together with HadoopFsRelation for persisting rows to the underlying file system.

20. abstract class OutputWriterFactory extends Serializable

    A factory that produces OutputWriters.

21. case class PartitionDirectory(values: InternalRow, files: Seq[FileStatus]) extends Product with Serializable

    A collection of data files from a partitioned relation, along with the partition values in the form of an InternalRow.

22. case class PartitionPath(values: InternalRow, path: Path) extends Product with Serializable

    Holds a directory in a partitioned collection of files as well as as the partition values in the form of a Row.

23. case class PartitionSpec(partitionColumns: StructType, partitions: Seq[PartitionPath]) extends Product with Serializable

24. case class PartitionedFile(partitionValues: InternalRow, filePath: String, start: Long, length: Long, locations: Array[String] = ...) extends Product with Serializable

    A part (i.

25. abstract class PartitioningAwareFileIndex extends FileIndex with Logging

    An abstract class that represents FileIndexs that are aware of partitioned tables.

26. case class PreWriteCheck(conf: SQLConf, catalog: SessionCatalog) extends (LogicalPlan) ⇒ Unit with Product with Serializable

    A rule to do various checks before inserting into or writing to a data source table.

27. case class PreprocessTableInsertion(conf: SQLConf) extends Rule[LogicalPlan] with Product with Serializable

    Preprocess the InsertIntoTable plan.

28. class RecordReaderIterator[T] extends Iterator[T] with Closeable

    An adaptor from a Hadoop RecordReader to an Iterator over the values returned.

29. case class RefreshResource(path: String) extends LeafNode with RunnableCommand with Product with Serializable

30. case class RefreshTable(tableIdent: TableIdentifier) extends LeafNode with RunnableCommand with Product with Serializable

31. class ResolveDataSource extends Rule[LogicalPlan]

    Try to replaces UnresolvedRelations with ResolveDataSource.

32. class SQLHadoopMapReduceCommitProtocol extends HadoopMapReduceCommitProtocol with Serializable with Logging

    A variant of HadoopMapReduceCommitProtocol that allows specifying the actual Hadoop output committer using an option specified in SQLConf.

33. abstract class TextBasedFileFormat extends FileFormat

    The base class file format that is based on text file.