Type Members

1.  trait Cache[T] extends AnyRef

   This trait defines interface APIs for a Cache.

   This trait defines interface APIs for a Cache.

   T

   Type of cache entry

2.  class CachingIndexCollectionManager extends IndexCollectionManager

   An IndexCollectionManager which leverages a cache to accelerate fetching indexes, with below properties: - Cache entry is only set/used on the read path (i.e.

   An IndexCollectionManager which leverages a cache to accelerate fetching indexes, with below properties: - Cache entry is only set/used on the read path (i.e. getIndexes() API). - If cache entry is stale, indexes are fetched directly from the system path (as done in the parent class) and cache entry will be refreshed. - If any API is called to add a new index or modify an existing index's status, cache gets cleared.

3.  case class Content(root: Directory, fingerprint: NoOpFingerprint = NoOpFingerprint()) extends Product with Serializable
4.  case class CoveringIndex(properties: Properties) extends Product with Serializable
5.  class CreationTimeBasedIndexCache extends Cache[Seq[IndexLogEntry]]

   An implementation of cache to store indexes that uses entry's last reset time to check validity of cache entry.

   An implementation of cache to store indexes that uses entry's last reset time to check validity of cache entry. Cache entry is stale if it has been in the cache for some (configurable) time.

6.  case class Directory(name: String, files: Seq[FileInfo] = Seq(), subDirs: Seq[Directory] = Seq()) extends Product with Serializable

   Directory is a representation of file system directory.

   Directory is a representation of file system directory. It consists of a name (directory name), a list of files represented by sequence of FileInfo, and a list of subdirectories.

   name

   Directory name.

   files

   List of leaf files in this directory.

   subDirs

   List of sub-directories in this directory.

7.  class FileBasedSignatureProvider extends LogicalPlanSignatureProvider

   FileBasedSignatureProvider provides the logical plan signature based on files in the logical relation.

   FileBasedSignatureProvider provides the logical plan signature based on files in the logical relation. File metadata, eg. size, modification time and path, of each file in the FileIndex will be used to generate the signature. If the logical plan does not have any LogicalRelation operator, no signature is provided.

8.  class FileIdTracker extends AnyRef

   Provides functionality to generate unique file ids for files.

9.  case class FileInfo(name: String, size: Long, modifiedTime: Long, id: Long) extends Product with Serializable
10.  trait FileSystemFactory extends AnyRef
11.  case class Hdfs(properties: Properties) extends Product with Serializable
12.  trait IndexCacheFactory extends AnyRef
13.  class IndexCollectionManager extends IndexManager
14.  case class IndexConfig(indexName: String, indexedColumns: Seq[String], includedColumns: Seq[String] = Seq()) extends Product with Serializable

    IndexConfig specifies the configuration of an index.

    IndexConfig specifies the configuration of an index.

    indexName

    Index name.

    indexedColumns

    Columns from which an index is created.

    includedColumns

    Columns to be included in the index.

15.  trait IndexDataManager extends AnyRef

    Index Directory Structure IDRoot/ _hyperspace_log/ 0 1 ...

    Index Directory Structure IDRoot/ _hyperspace_log/ 0 1 ... n IndexConstants.INDEX_VERSION_DIRECTORY_PREFIX=0/ f1.parquet ... IndexConstants.INDEX_VERSION_DIRECTORY_PREFIX=1/ f1.parquet

16.  trait IndexDataManagerFactory extends AnyRef
17.  class IndexDataManagerImpl extends IndexDataManager
18.  case class IndexLogEntry(name: String, derivedDataset: CoveringIndex, content: Content, source: Source, properties: Map[String, String]) extends LogEntry with Product with Serializable
19.  case class IndexLogEntryTag[T](name: String) extends Product with Serializable
20.  trait IndexLogManager extends AnyRef

    Interface for IndexLogManager which handles log operations.

21.  trait IndexLogManagerFactory extends AnyRef
22.  class IndexLogManagerImpl extends IndexLogManager with Logging
23.  trait IndexManager extends AnyRef

    Interface that contains internal APIs to manage index.

24.  class IndexSignatureProvider extends LogicalPlanSignatureProvider

    IndexSignatureProvider provides signature for a logical plan based on: 1- Files and their properties in LogicalRelation operator(s) of the plan; 2- Overall structure of the plan w.r.t its operators' types.

    IndexSignatureProvider provides signature for a logical plan based on: 1- Files and their properties in LogicalRelation operator(s) of the plan; 2- Overall structure of the plan w.r.t its operators' types.

    FileBasedSignatureProvider is used for (1) and PlanSignatureProvider is used for (2).

    If the plan does not comply with FileBasedSignatureProvider or PlanSignatureProvider requirements for signature computation, then no signature will be provided for the plan.

25.  abstract class LogEntry extends AnyRef
26.  case class LogicalPlanFingerprint(properties: Properties) extends Product with Serializable
27.  trait LogicalPlanSignatureProvider extends AnyRef

    This trait contains the interface that provides the signature of logical plan.

28.  case class NoOpFingerprint() extends Product with Serializable
29.  class PlanSignatureProvider extends LogicalPlanSignatureProvider

    PlanSignatureProvider provides signature for a logical plan based on the type of operators in it.

    PlanSignatureProvider provides signature for a logical plan based on the type of operators in it. A plan needs to have at least one operator so its signature can be generated.

30.  case class Relation(rootPaths: Seq[String], data: Hdfs, dataSchemaJson: String, fileFormat: String, options: Map[String, String]) extends Product with Serializable

    IndexLogEntry-specific Relation that represents the source relation.

    IndexLogEntry-specific Relation that represents the source relation.

    rootPaths

    List of root paths for the source relation.

    data

    Source data for the relation. Hdfs.properties.content captures source data which derived dataset was created from. Hdfs.properties.update captures any updates since the derived dataset was created.

    dataSchemaJson

    Schema in json format.

    fileFormat

    File format name.

    options

    Options to read the source relation.

31.  case class Signature(provider: String, value: String) extends Product with Serializable
32.  case class Source(plan: SparkPlan) extends Product with Serializable
33.  case class SparkPlan(properties: Properties) extends Product with Serializable
34.  case class Update(appendedFiles: Option[Content] = None, deletedFiles: Option[Content] = None) extends Product with Serializable

    Captures any HDFS updates.

    Captures any HDFS updates.

    appendedFiles

    Appended files.

    deletedFiles

    Deleted files.