ActionPipelineContext

ActionPipelineContext contains start and runtime information about a SmartDataLake run.

feed: feed selector of the run
application: application name of the run
executionId: SDLExecutionId of this runs. Contains runId and attemptId. Both stay 1 if state is not enabled.
instanceRegistry: registry of all SmartDataLake objects parsed from the config
referenceTimestamp: timestamp used as reference in certain actions (e.g. HistorizeAction)
appConfig: the command line parameters parsed into a SmartDataLakeBuilderConfig object
runStartTime: start time of the run
attemptStartTime: start time of attempt
simulation: true if this is a simulation run
phase: current execution phase
dataFrameReuseStatistics: Counter how many times a DataFrame of a SparkSubFeed is reused by an Action later in the pipeline. The counter is increased during ExecutionPhase.Init when preparing the SubFeeds for an Action and it is decreased in ExecutionPhase.Exec to unpersist the DataFrame after there is no need for it anymore.
actionsSelected: actions selected for execution by command line parameter --feed-sel
actionsSkipped: actions selected but skipped in current attempt because they already succeeded in a previous attempt.

Annotations: @DeveloperApi()

Linear Supertypes

Serializable, Serializable, Product, Equals, SmartDataLakeLogger, AnyRef, Any

Instance Constructors

new ActionPipelineContext(feed: String, application: String, executionId: SDLExecutionId, instanceRegistry: InstanceRegistry, referenceTimestamp: Option[LocalDateTime] = None, appConfig: SmartDataLakeBuilderConfig, runStartTime: LocalDateTime = LocalDateTime.now(), attemptStartTime: LocalDateTime = LocalDateTime.now(), simulation: Boolean = false, phase: ExecutionPhase = ExecutionPhase.Prepare, dataFrameReuseStatistics: Map[(DataObjectId, Seq[PartitionValues]), Seq[ActionId]] = mutable.Map(), actionsSelected: Seq[ActionId] = Seq(), actionsSkipped: Seq[ActionId] = Seq())

feed
feed selector of the run
application
application name of the run
executionId
SDLExecutionId of this runs. Contains runId and attemptId. Both stay 1 if state is not enabled.
instanceRegistry
registry of all SmartDataLake objects parsed from the config
referenceTimestamp
timestamp used as reference in certain actions (e.g. HistorizeAction)
appConfig
the command line parameters parsed into a SmartDataLakeBuilderConfig object
runStartTime
start time of the run
attemptStartTime
start time of attempt
simulation
true if this is a simulation run
phase
current execution phase
dataFrameReuseStatistics
Counter how many times a DataFrame of a SparkSubFeed is reused by an Action later in the pipeline. The counter is increased during ExecutionPhase.Init when preparing the SubFeeds for an Action and it is decreased in ExecutionPhase.Exec to unpersist the DataFrame after there is no need for it anymore.
actionsSelected
actions selected for execution by command line parameter --feed-sel
actionsSkipped
actions selected but skipped in current attempt because they already succeeded in a previous attempt.

Value Members

final def !=(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def ##(): Int

Definition Classes
AnyRef → Any
final def ==(arg0: Any): Boolean

Definition Classes
AnyRef → Any
val actionsSelected: Seq[ActionId]

actions selected for execution by command line parameter --feed-sel
val actionsSkipped: Seq[ActionId]

actions selected but skipped in current attempt because they already succeeded in a previous attempt.
val appConfig: SmartDataLakeBuilderConfig

the command line parameters parsed into a SmartDataLakeBuilderConfig object
val application: String

application name of the run
final def asInstanceOf[T0]: T0

Definition Classes
Any
val attemptStartTime: LocalDateTime

start time of attempt
def clone(): AnyRef

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( ... )
val dataFrameReuseStatistics: Map[(DataObjectId, Seq[PartitionValues]), Seq[ActionId]]

Counter how many times a DataFrame of a SparkSubFeed is reused by an Action later in the pipeline.
Counter how many times a DataFrame of a SparkSubFeed is reused by an Action later in the pipeline. The counter is increased during ExecutionPhase.Init when preparing the SubFeeds for an Action and it is decreased in ExecutionPhase.Exec to unpersist the DataFrame after there is no need for it anymore.
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
val executionId: SDLExecutionId

SDLExecutionId of this runs.
SDLExecutionId of this runs. Contains runId and attemptId. Both stay 1 if state is not enabled.
val feed: String

feed selector of the run
def finalize(): Unit

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
val instanceRegistry: InstanceRegistry

registry of all SmartDataLake objects parsed from the config
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
lazy val logger: Logger

Attributes
protected
Definition Classes
SmartDataLakeLogger
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def notify(): Unit

Definition Classes
AnyRef
final def notifyAll(): Unit

Definition Classes
AnyRef
var phase: ExecutionPhase

current execution phase
val referenceTimestamp: Option[LocalDateTime]

timestamp used as reference in certain actions (e.g.
timestamp used as reference in certain actions (e.g. HistorizeAction)
val runStartTime: LocalDateTime

start time of the run
val simulation: Boolean

true if this is a simulation run
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )

Related Doc: package workflow

Instance Constructors

Value Members

final def !=(arg0: Any): Boolean

final def ##(): Int

final def ==(arg0: Any): Boolean

val actionsSelected: Seq[ActionId]

val actionsSkipped: Seq[ActionId]

val appConfig: SmartDataLakeBuilderConfig

val application: String

final def asInstanceOf[T0]: T0

val attemptStartTime: LocalDateTime

def clone(): AnyRef

val dataFrameReuseStatistics: Map[(DataObjectId, Seq[PartitionValues]), Seq[ActionId]]

final def eq(arg0: AnyRef): Boolean

val executionId: SDLExecutionId

val feed: String

def finalize(): Unit

final def getClass(): Class[_]

val instanceRegistry: InstanceRegistry

final def isInstanceOf[T0]: Boolean

lazy val logger: Logger

final def ne(arg0: AnyRef): Boolean

final def notify(): Unit

final def notifyAll(): Unit

var phase: ExecutionPhase

val referenceTimestamp: Option[LocalDateTime]

val runStartTime: LocalDateTime

val simulation: Boolean

final def synchronized[T0](arg0: ⇒ T0): T0

final def wait(): Unit

final def wait(arg0: Long, arg1: Int): Unit

final def wait(arg0: Long): Unit

Inherited from Serializable

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from SmartDataLakeLogger

Inherited from AnyRef

Inherited from Any

Ungrouped