delta

Type Members

case class CheckpointInstance(version: Long, numParts: Option[Int]) extends Ordered[CheckpointInstance] with Product with Serializable

A class to help with comparing checkpoints with each other, where we may have had concurrent writers that checkpoint with different number of parts.
case class CheckpointMetaData(version: Long, size: Long, parts: Option[Int]) extends Product with Serializable

Records information about a checkpoint.
Records information about a checkpoint.
version
the version of this checkpoint
size
the number of actions in the checkpoint
parts
the number of parts when the checkpoint has multiple parts. None if this is a singular checkpoint
trait Checkpoints extends DeltaLogging
case class CommitStats(startVersion: Long, commitVersion: Long, readVersion: Long, txnDurationMs: Long, commitDurationMs: Long, numAdd: Int, numRemove: Int, bytesNew: Long, numFilesTotal: Long, sizeInBytesTotal: Long, protocol: Protocol, info: CommitInfo, newMetadata: Option[Metadata], numAbsolutePathsInAdd: Int, numDistinctPartitionsInAdd: Int, isolationLevel: String) extends Product with Serializable

Record metrics about a successful commit.
class ConcurrentAppendException extends DeltaConcurrentModificationException

Thrown when files are added that would have been read by the current transaction.
class ConcurrentDeleteDeleteException extends DeltaConcurrentModificationException

Thrown when the current transaction deletes data that was deleted by a concurrent transaction.
class ConcurrentDeleteReadException extends DeltaConcurrentModificationException

Thrown when the current transaction reads data that was deleted by a concurrent transaction.
class ConcurrentTransactionException extends DeltaConcurrentModificationException

Thrown when concurrent transaction both attempt to update the same idempotent transaction.
class ConcurrentWriteException extends DeltaConcurrentModificationException

Thrown when a concurrent transaction has written data after the current transaction read the table.
abstract class DeltaConcurrentModificationException extends ConcurrentModificationException

The basic class for all Tahoe commit conflict exceptions.
case class DeltaConfig[T](key: String, defaultValue: String, fromString: (String) ⇒ T, validationFunction: (T) ⇒ Boolean, helpMessage: String, minimumProtocolVersion: Option[Protocol] = None) extends Product with Serializable
trait DeltaFileFormat extends AnyRef
class DeltaHistoryManager extends DeltaLogging

This class keeps tracks of the version of commits and their timestamps for a Delta table to help with operations like describing the history of a table.
class DeltaLog extends Checkpoints with MetadataCleanup with LogStoreProvider with VerifyChecksum

Used to query the current state of the log as well as modify it by adding new atomic collections of actions.
Used to query the current state of the log as well as modify it by adding new atomic collections of actions.
Internally, this class implements an optimistic concurrency control algorithm to handle multiple readers or writers. Any single read is guaranteed to see a consistent snapshot of the table.
trait DeltaOptionParser extends AnyRef
class DeltaOptions extends DeltaWriteOptions with DeltaReadOptions with Serializable

Options for the Delta data source.
trait DeltaReadOptions extends DeltaOptionParser
case class DeltaTableIdentifier(path: Option[String] = None, table: Option[TableIdentifier] = None) extends Product with Serializable

An identifier for a Delta table containing one of the path or the table identifier.
case class DeltaTimeTravelSpec(timestamp: Option[Expression], version: Option[Long], creationSource: Option[String]) extends Product with Serializable

The specification to time travel a Delta Table to the given timestamp or version.
The specification to time travel a Delta Table to the given timestamp or version.
timestamp
An expression that can be evaluated into a timestamp. The expression cannot be a subquery.
version
The version of the table to time travel to. Must be >= 0.
creationSource
The API used to perform time travel, e.g. atSyntax, dfReader or SQL
trait DeltaWriteOptions extends DeltaWriteOptionsImpl with DeltaOptionParser
trait DeltaWriteOptionsImpl extends DeltaOptionParser
trait DocsPath extends AnyRef
class InitialSnapshot extends Snapshot

An initial snapshot with only metadata specified.
An initial snapshot with only metadata specified. Useful for creating a DataFrame from an existing parquet table during its conversion to delta.
class MetadataChangedException extends DeltaConcurrentModificationException

Thrown when the metadata of the Delta table has changed between the time of read and the time of commit.
trait MetadataCleanup extends DeltaLogging

Cleans up expired Delta table metadata.
class MetadataMismatchErrorBuilder extends AnyRef

A helper class in building a helpful error message in case of metadata mismatches.
class OptimisticTransaction extends OptimisticTransactionImpl with DeltaLogging

Used to perform a set of reads in a transaction and then commit a set of updates to the state of the log.
Used to perform a set of reads in a transaction and then commit a set of updates to the state of the log. All reads from the DeltaLog, MUST go through this instance rather than directly to the DeltaLog otherwise they will not be check for logical conflicts with concurrent updates.
This class is not thread-safe.
trait OptimisticTransactionImpl extends TransactionalWrite

Used to perform a set of reads in a transaction and then commit a set of updates to the state of the log.
Used to perform a set of reads in a transaction and then commit a set of updates to the state of the log. All reads from the DeltaLog, MUST go through this instance rather than directly to the DeltaLog otherwise they will not be check for logical conflicts with concurrent updates.
This trait is not thread-safe.
trait PartitionFiltering extends AnyRef
case class PreprocessTableMerge(conf: SQLConf) extends UpdateExpressionsSupport with Product with Serializable
case class PreprocessTableUpdate(conf: SQLConf) extends UpdateExpressionsSupport with Product with Serializable
class ProtocolChangedException extends DeltaConcurrentModificationException

Thrown when the protocol version has changed between the time of read and the time of commit.
trait RecordChecksum extends DeltaLogging

Record the state of the table as a checksum file along with a commit.
class Snapshot extends StateCache with PartitionFiltering with DeltaFileFormat with DeltaLogging

An immutable snapshot of the state of the log at some delta version.
An immutable snapshot of the state of the log at some delta version. Internally this class manages the replay of actions stored in checkpoint or delta files, given an optional starting snapshot.
After resolving any new actions, it caches the result and collects the following basic information to the driver:
- Protocol Version
- Metadata
- Transaction state
trait UpdateExpressionsSupport extends CastSupport

Trait with helper functions to generate expressions to update target columns, even if they are nested fields.
trait VerifyChecksum extends DeltaLogging

Verify the state of the table using the checksum files.
case class VersionChecksum(tableSizeBytes: Long, numFiles: Long, numMetadata: Long, numProtocol: Long, numTransactions: Long) extends Product with Serializable

Stats calculated within a snapshot, which we store along individual transactions for verification.
Stats calculated within a snapshot, which we store along individual transactions for verification.
tableSizeBytes
The size of the table in bytes
numFiles
Number of AddFile actions in the snapshot
numMetadata
Number of Metadata actions in the snapshot
numProtocol
Number of Protocol actions in the snapshot
numTransactions
Number of SetTransaction actions in the snapshot

Value Members

object CheckpointInstance extends Serializable
object Checkpoints
object DeltaConfigs extends DeltaLogging

Contains list of reservoir configs and validation checks.
object DeltaErrors extends DocsPath with DeltaLogging

A holder object for Delta errors.
object DeltaFullTable

Extractor Object for pulling out the full table scan of a Delta table.
object DeltaHistoryManager extends DeltaLogging

Contains many utility methods that can also be executed on Spark executors.
object DeltaLog extends DeltaLogging
object DeltaOperations

Exhaustive list of operations that can be performed on a Delta table.
Exhaustive list of operations that can be performed on a Delta table. These operations are tracked as the first line in delta logs, and power DESCRIBE HISTORY for Delta tables.
object DeltaOptions extends DeltaLogging with Serializable
object DeltaTable

Extractor Object for pulling out the table scan of a Delta table.
Extractor Object for pulling out the table scan of a Delta table. It could be a full scan or a partial scan.
object DeltaTableIdentifier extends Serializable

Utilities for DeltaTableIdentifier.
object DeltaTableUtils extends PredicateHelper with DeltaLogging
object DeltaTimeTravelSpec extends Serializable
object OptimisticTransaction
object Snapshot extends DeltaLogging
package actions
package commands
package files
package metering
package schema
package sources
package stats
package storage
package util

package delta

Type Members

case class CheckpointInstance(version: Long, numParts: Option[Int]) extends Ordered[CheckpointInstance] with Product with Serializable

case class CheckpointMetaData(version: Long, size: Long, parts: Option[Int]) extends Product with Serializable

trait Checkpoints extends DeltaLogging

class ConcurrentAppendException extends DeltaConcurrentModificationException

class ConcurrentDeleteDeleteException extends DeltaConcurrentModificationException

class ConcurrentDeleteReadException extends DeltaConcurrentModificationException

class ConcurrentTransactionException extends DeltaConcurrentModificationException

class ConcurrentWriteException extends DeltaConcurrentModificationException

abstract class DeltaConcurrentModificationException extends ConcurrentModificationException

case class DeltaConfig[T](key: String, defaultValue: String, fromString: (String) ⇒ T, validationFunction: (T) ⇒ Boolean, helpMessage: String, minimumProtocolVersion: Option[Protocol] = None) extends Product with Serializable

trait DeltaFileFormat extends AnyRef

class DeltaHistoryManager extends DeltaLogging

class DeltaLog extends Checkpoints with MetadataCleanup with LogStoreProvider with VerifyChecksum

trait DeltaOptionParser extends AnyRef

class DeltaOptions extends DeltaWriteOptions with DeltaReadOptions with Serializable

trait DeltaReadOptions extends DeltaOptionParser

case class DeltaTableIdentifier(path: Option[String] = None, table: Option[TableIdentifier] = None) extends Product with Serializable

case class DeltaTimeTravelSpec(timestamp: Option[Expression], version: Option[Long], creationSource: Option[String]) extends Product with Serializable

trait DeltaWriteOptions extends DeltaWriteOptionsImpl with DeltaOptionParser

trait DeltaWriteOptionsImpl extends DeltaOptionParser

trait DocsPath extends AnyRef

class InitialSnapshot extends Snapshot

class MetadataChangedException extends DeltaConcurrentModificationException

trait MetadataCleanup extends DeltaLogging

class MetadataMismatchErrorBuilder extends AnyRef

class OptimisticTransaction extends OptimisticTransactionImpl with DeltaLogging

trait OptimisticTransactionImpl extends TransactionalWrite

trait PartitionFiltering extends AnyRef

case class PreprocessTableMerge(conf: SQLConf) extends UpdateExpressionsSupport with Product with Serializable

case class PreprocessTableUpdate(conf: SQLConf) extends UpdateExpressionsSupport with Product with Serializable

class ProtocolChangedException extends DeltaConcurrentModificationException

trait RecordChecksum extends DeltaLogging

class Snapshot extends StateCache with PartitionFiltering with DeltaFileFormat with DeltaLogging

trait UpdateExpressionsSupport extends CastSupport

trait VerifyChecksum extends DeltaLogging

case class VersionChecksum(tableSizeBytes: Long, numFiles: Long, numMetadata: Long, numProtocol: Long, numTransactions: Long) extends Product with Serializable

Value Members

object CheckpointInstance extends Serializable

object Checkpoints

object DeltaConfigs extends DeltaLogging

object DeltaErrors extends DocsPath with DeltaLogging

object DeltaFullTable

object DeltaHistoryManager extends DeltaLogging

object DeltaLog extends DeltaLogging

object DeltaOperations

object DeltaOptions extends DeltaLogging with Serializable

object DeltaTable

object DeltaTableIdentifier extends Serializable

object DeltaTableUtils extends PredicateHelper with DeltaLogging

object DeltaTimeTravelSpec extends Serializable

object OptimisticTransaction

object Snapshot extends DeltaLogging

package actions

package commands

package files

package metering

package schema

package sources

package stats

package storage

package util

Ungrouped