OptimisticTransaction

Instance Constructors

new OptimisticTransaction(deltaLog: DeltaLog)(implicit clock: Clock)

Creates a new OptimisticTransaction.
Creates a new OptimisticTransaction.
deltaLog
The Delta Log for the table this transaction is modifying.
new OptimisticTransaction(deltaLog: DeltaLog, snapshot: Snapshot)(implicit clock: Clock)

deltaLog
The Delta Log for the table this transaction is modifying.
snapshot
The snapshot that this transaction is reading at.

Value Members

final def !=(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def ##(): Int

Definition Classes
AnyRef → Any
final def ==(arg0: Any): Boolean

Definition Classes
AnyRef → Any
val _spark: SparkSession

Attributes
protected
Definition Classes
OptimisticTransactionImpl
final def asInstanceOf[T0]: T0

Definition Classes
Any
def checkAndRetry(checkVersion: Long, actions: Seq[Action], attemptNumber: Int): Long

Looks at actions that have happened since the txn started and checks for logical conflicts with the read/writes.
Looks at actions that have happened since the txn started and checks for logical conflicts with the read/writes. If no conflicts are found, try to commit again otherwise, throw an exception.

Attributes
protected
Definition Classes
OptimisticTransactionImpl
implicit val clock: Clock

Definition Classes
OptimisticTransaction → OptimisticTransactionImpl
def clone(): AnyRef

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( ... )
def commit(actions: Seq[Action], op: Operation): Long

Modifies the state of the log by adding a new commit that is based on a read at the given lastVersion.
Modifies the state of the log by adding a new commit that is based on a read at the given lastVersion. In the case of a conflict with a concurrent writer this method will throw an exception.
actions
Set of actions to commit
op
Details of operation that is performing this transactional commit

Definition Classes
OptimisticTransactionImpl
Annotations
@throws( ... )
var commitInfo: CommitInfo

Attributes
protected
Definition Classes
OptimisticTransactionImpl
var commitStartNano: Long

Attributes
protected
Definition Classes
OptimisticTransactionImpl
var committed: Boolean

Tracks if this transaction has already committed.
Tracks if this transaction has already committed.

Attributes
protected
Definition Classes
OptimisticTransactionImpl
val deltaLog: DeltaLog

The Delta Log for the table this transaction is modifying.
The Delta Log for the table this transaction is modifying.

Definition Classes
OptimisticTransaction → OptimisticTransactionImpl → TransactionalWrite
var dependsOnFiles: Boolean

Tracks if this transaction depends on any data files.
Tracks if this transaction depends on any data files. This flag must be set if this transaction reads any data explicitly or implicitly (e.g., delete, update and overwrite).

Attributes
protected
Definition Classes
OptimisticTransactionImpl
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def equals(arg0: Any): Boolean

Definition Classes
AnyRef → Any
def filterFiles(filters: Seq[Expression]): Seq[AddFile]

Returns files matching the given predicates.
Returns files matching the given predicates.

Definition Classes
OptimisticTransactionImpl
def filterFiles(): Seq[AddFile]

Returns files matching the given predicates.
Returns files matching the given predicates.

Definition Classes
OptimisticTransactionImpl
def finalize(): Unit

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
def getCommitter(outputPath: Path): DelayedCommitProtocol

Attributes
protected
Definition Classes
TransactionalWrite
def getPartitioningColumns(partitionSchema: StructType, output: Seq[Attribute], colsDropped: Boolean): Seq[Attribute]

Attributes
protected
Definition Classes
TransactionalWrite
var hasWritten: Boolean

Attributes
protected
Definition Classes
TransactionalWrite
def hashCode(): Int

Definition Classes
AnyRef → Any
def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean

Attributes
protected
Definition Classes
Logging
def initializeLogIfNecessary(isInterpreter: Boolean): Unit

Attributes
protected
Definition Classes
Logging
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
def isTraceEnabled(): Boolean

Attributes
protected
Definition Classes
Logging
def log: Logger

Attributes
protected
Definition Classes
Logging
def logConsole(line: String): Unit

Definition Classes
DatabricksLogging
def logDebug(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logDebug(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logError(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logError(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logInfo(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logInfo(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logName: String

Attributes
protected
Definition Classes
Logging
def logTrace(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logTrace(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logWarning(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logWarning(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def makeOutputNullable(output: Seq[Attribute]): Seq[Attribute]

Makes the output attributes nullable, so that we don't write unreadable parquet files.
Makes the output attributes nullable, so that we don't write unreadable parquet files.

Attributes
protected
Definition Classes
TransactionalWrite
def metadata: Metadata

Returns the metadata at the current point in the log.
Returns the metadata at the current point in the log.

Definition Classes
OptimisticTransactionImpl → TransactionalWrite
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
var newMetadata: Option[Metadata]

Stores the updated metadata (if any) that will result from this txn.
Stores the updated metadata (if any) that will result from this txn.

Attributes
protected
Definition Classes
OptimisticTransactionImpl
def normalizeData(data: Dataset[_], partitionCols: Seq[String]): (QueryExecution, Seq[Attribute])

Normalize the schema of the query, and return the QueryExecution to execute.
Normalize the schema of the query, and return the QueryExecution to execute. The output attributes of the QueryExecution may not match the attributes we return as the output schema. This is because streaming queries create IncrementalExecution, which cannot be further modified. We can however have the Parquet writer use the physical plan from IncrementalExecution and the output schema provided through the attributes.

Attributes
protected
Definition Classes
TransactionalWrite
final def notify(): Unit

Definition Classes
AnyRef
final def notifyAll(): Unit

Definition Classes
AnyRef
def postCommit(commitVersion: Long, commitActions: Seq[Action]): Unit

Perform post-commit operations
Perform post-commit operations

Attributes
protected
Definition Classes
OptimisticTransactionImpl
def prepareCommit(actions: Seq[Action], op: Operation): Seq[Action]

Prepare for a commit by doing all necessary pre-commit checks and modifications to the actions.
Prepare for a commit by doing all necessary pre-commit checks and modifications to the actions.
returns
The finalized set of actions.

Attributes
protected
Definition Classes
OptimisticTransactionImpl
val protocol: Protocol

The protocol of the snapshot that this transaction is reading at.
The protocol of the snapshot that this transaction is reading at.

Definition Classes
OptimisticTransactionImpl → TransactionalWrite
val readTxn: ArrayBuffer[String]

Tracks the appIds that have been seen by this transaction.
Tracks the appIds that have been seen by this transaction.

Attributes
protected
Definition Classes
OptimisticTransactionImpl
def readVersion: Long

The version that this transaction is reading from.
The version that this transaction is reading from.

Definition Classes
OptimisticTransactionImpl
def readWholeTable(): Unit

Mark the entire table as tainted by this transaction.
Mark the entire table as tainted by this transaction.

Definition Classes
OptimisticTransactionImpl
def recordDeltaEvent(deltaLog: DeltaLog, opType: String, tags: Map[TagDefinition, String] = Map.empty, data: AnyRef = null): Unit

Used to record the occurrence of a single event or report detailed, operation specific statistics.
Used to record the occurrence of a single event or report detailed, operation specific statistics.

Attributes
protected
Definition Classes
DeltaLogging
def recordDeltaOperation[A](deltaLog: DeltaLog, opType: String, tags: Map[TagDefinition, String] = Map.empty)(thunk: ⇒ A): A

Used to report the duration as well as the success or failure of an operation.
Used to report the duration as well as the success or failure of an operation.

Attributes
protected
Definition Classes
DeltaLogging
def recordEvent(metric: MetricDefinition, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, trimBlob: Boolean = true): Unit

Definition Classes
DatabricksLogging
def recordOperation[S](opType: OpType, opTarget: String = null, extraTags: Map[TagDefinition, String], isSynchronous: Boolean = true, alwaysRecordStats: Boolean = false, allowAuthTags: Boolean = false, killJvmIfStuck: Boolean = false, outputMetric: MetricDefinition = null, silent: Boolean = true)(thunk: ⇒ S): S

Definition Classes
DatabricksLogging
def recordUsage(metric: MetricDefinition, quantity: Double, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, forceSample: Boolean = false, trimBlob: Boolean = true, silent: Boolean = false): Unit

Definition Classes
DatabricksLogging
val snapshot: Snapshot

The snapshot that this transaction is reading at.
The snapshot that this transaction is reading at.

Definition Classes
OptimisticTransaction → OptimisticTransactionImpl → TransactionalWrite
val snapshotMetadata: Metadata

For new tables, fetch global configs as metadata.
For new tables, fetch global configs as metadata.

Definition Classes
OptimisticTransactionImpl
def spark: SparkSession

Attributes
protected
Definition Classes
OptimisticTransactionImpl
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
def toString(): String

Definition Classes
AnyRef → Any
val txnStartNano: Long

Attributes
protected
Definition Classes
OptimisticTransactionImpl
def txnVersion(id: String): Long

Returns the latest version that has committed for the idempotent transaction with given id.
Returns the latest version that has committed for the idempotent transaction with given id.

Definition Classes
OptimisticTransactionImpl
def updateMetadata(metadata: Metadata): Unit

Records an update to the metadata that should be committed with this transaction.
Records an update to the metadata that should be committed with this transaction. Note that this must be done before writing out any files so that file writing and checks happen with the final metadata for the table.
IMPORTANT: It is the responsibility of the caller to ensure that files currently present in the table are still valid under the new metadata.

Definition Classes
OptimisticTransactionImpl
def verifyNewMetadata(metadata: Metadata): Unit

Attributes
protected
Definition Classes
OptimisticTransactionImpl
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
def withStatusCode[T](statusCode: String, defaultMessage: String, data: Map[String, Any] = Map.empty)(body: ⇒ T): T

Report a log to indicate some command is running.
Report a log to indicate some command is running.

Definition Classes
DeltaProgressReporter
def writeFiles(data: Dataset[_], writeOptions: Option[DeltaOptions], isOptimize: Boolean): Seq[AddFile]

Writes out the dataframe after performing schema validation.
Writes out the dataframe after performing schema validation. Returns a list of actions to append these files to the reservoir.

Definition Classes
TransactionalWrite
def writeFiles(data: Dataset[_], isOptimize: Boolean): Seq[AddFile]

Definition Classes
TransactionalWrite
def writeFiles(data: Dataset[_], writeOptions: Option[DeltaOptions]): Seq[AddFile]

Definition Classes
TransactionalWrite
def writeFiles(data: Dataset[_]): Seq[AddFile]

Definition Classes
TransactionalWrite

Related Docs: object OptimisticTransaction | package delta

class OptimisticTransaction extends OptimisticTransactionImpl with DeltaLogging

Instance Constructors

new OptimisticTransaction(deltaLog: DeltaLog)(implicit clock: Clock)

new OptimisticTransaction(deltaLog: DeltaLog, snapshot: Snapshot)(implicit clock: Clock)

Value Members

final def !=(arg0: Any): Boolean

final def ##(): Int

final def ==(arg0: Any): Boolean

val _spark: SparkSession

final def asInstanceOf[T0]: T0

def checkAndRetry(checkVersion: Long, actions: Seq[Action], attemptNumber: Int): Long

implicit val clock: Clock

def clone(): AnyRef

def commit(actions: Seq[Action], op: Operation): Long

var commitInfo: CommitInfo

var commitStartNano: Long

var committed: Boolean

val deltaLog: DeltaLog

var dependsOnFiles: Boolean

final def eq(arg0: AnyRef): Boolean

def equals(arg0: Any): Boolean

def filterFiles(filters: Seq[Expression]): Seq[AddFile]

def filterFiles(): Seq[AddFile]

def finalize(): Unit

final def getClass(): Class[_]

def getCommitter(outputPath: Path): DelayedCommitProtocol

def getPartitioningColumns(partitionSchema: StructType, output: Seq[Attribute], colsDropped: Boolean): Seq[Attribute]

var hasWritten: Boolean

def hashCode(): Int

def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean

def initializeLogIfNecessary(isInterpreter: Boolean): Unit

final def isInstanceOf[T0]: Boolean

def isTraceEnabled(): Boolean

def log: Logger

def logConsole(line: String): Unit

def logDebug(msg: ⇒ String, throwable: Throwable): Unit

def logDebug(msg: ⇒ String): Unit

def logError(msg: ⇒ String, throwable: Throwable): Unit

def logError(msg: ⇒ String): Unit

def logInfo(msg: ⇒ String, throwable: Throwable): Unit

def logInfo(msg: ⇒ String): Unit

def logName: String

def logTrace(msg: ⇒ String, throwable: Throwable): Unit

def logTrace(msg: ⇒ String): Unit

def logWarning(msg: ⇒ String, throwable: Throwable): Unit

def logWarning(msg: ⇒ String): Unit

def makeOutputNullable(output: Seq[Attribute]): Seq[Attribute]

def metadata: Metadata

final def ne(arg0: AnyRef): Boolean

var newMetadata: Option[Metadata]

def normalizeData(data: Dataset[_], partitionCols: Seq[String]): (QueryExecution, Seq[Attribute])

final def notify(): Unit

final def notifyAll(): Unit

def postCommit(commitVersion: Long, commitActions: Seq[Action]): Unit

def prepareCommit(actions: Seq[Action], op: Operation): Seq[Action]

val protocol: Protocol

val readTxn: ArrayBuffer[String]

def readVersion: Long

def readWholeTable(): Unit

def recordDeltaEvent(deltaLog: DeltaLog, opType: String, tags: Map[TagDefinition, String] = Map.empty, data: AnyRef = null): Unit

def recordDeltaOperation[A](deltaLog: DeltaLog, opType: String, tags: Map[TagDefinition, String] = Map.empty)(thunk: ⇒ A): A

def recordEvent(metric: MetricDefinition, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, trimBlob: Boolean = true): Unit

def recordUsage(metric: MetricDefinition, quantity: Double, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, forceSample: Boolean = false, trimBlob: Boolean = true, silent: Boolean = false): Unit

val snapshot: Snapshot

val snapshotMetadata: Metadata

def spark: SparkSession

final def synchronized[T0](arg0: ⇒ T0): T0

def toString(): String

val txnStartNano: Long

def txnVersion(id: String): Long

def updateMetadata(metadata: Metadata): Unit

def verifyNewMetadata(metadata: Metadata): Unit

final def wait(): Unit

final def wait(arg0: Long, arg1: Int): Unit

final def wait(arg0: Long): Unit

def withStatusCode[T](statusCode: String, defaultMessage: String, data: Map[String, Any] = Map.empty)(body: ⇒ T): T

def writeFiles(data: Dataset[_], writeOptions: Option[DeltaOptions], isOptimize: Boolean): Seq[AddFile]

def writeFiles(data: Dataset[_], isOptimize: Boolean): Seq[AddFile]