org.apache.spark.sql.execution.streaming

MetadataLogFileIndex

class MetadataLogFileIndex extends PartitioningAwareFileIndex

A FileIndex that generates the list of files to processing by reading them from the metadata log files generated by the FileStreamSink.

Linear Supertypes
PartitioningAwareFileIndex, Logging, FileIndex, AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. MetadataLogFileIndex
  2. PartitioningAwareFileIndex
  3. Logging
  4. FileIndex
  5. AnyRef
  6. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Instance Constructors

  1. new MetadataLogFileIndex(sparkSession: SparkSession, path: Path)

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any

  3. final def ##(): Int

    Definition Classes
    AnyRef → Any

  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any

  6. def allFiles(): Seq[FileStatus]

    Definition Classes
    PartitioningAwareFileIndex

  7. final def asInstanceOf[T0]: T0

    Definition Classes
    Any

  8. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  9. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  10. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any

  11. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  12. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any

  13. val hadoopConf: Configuration

    Attributes
    protected
    Definition Classes
    PartitioningAwareFileIndex

  14. def hashCode(): Int

    Definition Classes
    AnyRef → Any

  15. def inferPartitioning(): PartitionSpec

    Attributes
    protected
    Definition Classes
    PartitioningAwareFileIndex

  16. def initializeLogIfNecessary(isInterpreter: Boolean): Unit

    Attributes
    protected
    Definition Classes
    Logging

  17. def inputFiles: Array[String]

    Returns the list of files that will be read when scanning this relation.

    Returns the list of files that will be read when scanning this relation.

    Definition Classes
    PartitioningAwareFileIndex → FileIndex

  18. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any

  19. def isTraceEnabled(): Boolean

    Attributes
    protected
    Definition Classes
    Logging

  20. val leafDirToChildrenFiles: Map[Path, Array[FileStatus]]

    Attributes
    protected
    Definition Classes
    MetadataLogFileIndex → PartitioningAwareFileIndex

  21. val leafFiles: LinkedHashMap[Path, FileStatus]

    Attributes
    protected
    Definition Classes
    MetadataLogFileIndex → PartitioningAwareFileIndex

  22. def listFiles(partitionFilters: Seq[Expression], dataFilters: Seq[Expression]): Seq[PartitionDirectory]

    Returns all valid files grouped into partitions when the data is partitioned.

    Returns all valid files grouped into partitions when the data is partitioned. If the data is unpartitioned, this will return a single partition with no partition values.

    partitionFilters

    The filters used to prune which partitions are returned. These filters must only refer to partition columns and this method will only return files where these predicates are guaranteed to evaluate to true. Thus, these filters will not need to be evaluated again on the returned data.

    dataFilters

    Filters that can be applied on non-partitioned columns. The implementation does not need to guarantee these filters are applied, i.e. the execution engine will ensure these filters are still applied on the returned files.

    Definition Classes
    PartitioningAwareFileIndex → FileIndex

  23. def log: Logger

    Attributes
    protected
    Definition Classes
    Logging

  24. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging

  25. def logDebug(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging

  26. def logError(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging

  27. def logError(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging

  28. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging

  29. def logInfo(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging

  30. def logName: String

    Attributes
    protected
    Definition Classes
    Logging

  31. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging

  32. def logTrace(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging

  33. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging

  34. def logWarning(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging

  35. def metadataOpsTimeNs: Option[Long]

    Returns an optional metadata operation time, in nanoseconds, for listing files.

    Returns an optional metadata operation time, in nanoseconds, for listing files.

    We do file listing in query optimization (in order to get the proper statistics) and we want to account for file listing time in physical execution (as metrics). To do that, we save the file listing time in some implementations and physical execution calls it in this method to update the metrics.

    Definition Classes
    FileIndex

  36. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  37. final def notify(): Unit

    Definition Classes
    AnyRef

  38. final def notifyAll(): Unit

    Definition Classes
    AnyRef

  39. def partitionSchema: StructType

    Schema of the partitioning columns, or the empty schema if the table is not partitioned.

    Schema of the partitioning columns, or the empty schema if the table is not partitioned.

    Definition Classes
    PartitioningAwareFileIndex → FileIndex

  40. def partitionSpec(): PartitionSpec

    Returns the specification of the partitions inferred from the data.

    Returns the specification of the partitions inferred from the data.

    Definition Classes
    MetadataLogFileIndex → PartitioningAwareFileIndex

  41. def refresh(): Unit

    Refresh any cached file listings

    Refresh any cached file listings

    Definition Classes
    MetadataLogFileIndex → FileIndex

  42. def rootPaths: Seq[Path]

    Returns the list of root input paths from which the catalog will get files.

    Returns the list of root input paths from which the catalog will get files. There may be a single root path from which partitions are discovered, or individual partitions may be specified by each path.

    Definition Classes
    MetadataLogFileIndex → FileIndex

  43. def sizeInBytes: Long

    Sum of table file sizes, in bytes

    Sum of table file sizes, in bytes

    Definition Classes
    PartitioningAwareFileIndex → FileIndex

  44. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef

  45. def toString(): String

    Definition Classes
    AnyRef → Any

  46. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  47. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  48. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from PartitioningAwareFileIndex

Inherited from Logging

Inherited from FileIndex

Inherited from AnyRef

Inherited from Any

Ungrouped