Value Members

1.  final def !=(arg0: Any): Boolean

   Definition Classes

   AnyRef → Any

2.  final def ##(): Int

   Definition Classes

   AnyRef → Any

3.  final def ==(arg0: Any): Boolean

   Definition Classes

   AnyRef → Any

4.  val appType: String
5.  val appVersion: String
6.  final def asInstanceOf[T0]: T0

   Definition Classes

   Any

7.  def clone(): AnyRef

   Attributes

   protected[lang]

   Definition Classes

   AnyRef

   Annotations

   @throws( ... ) @native()

8.  final def eq(arg0: AnyRef): Boolean

   Definition Classes

   AnyRef

9.  def equals(arg0: Any): Boolean

   Definition Classes

   AnyRef → Any

10.  final def execActionDAG(actionDAGRun: ActionDAGRun, actionsSelected: Seq[Action], context: ActionPipelineContext, lastStartTime: Option[LocalDateTime] = None)(implicit session: SparkSession): Seq[SubFeed]

    Execute one action DAG iteration and call recursion if streaming mode Must be implemented with tail recursion to avoid stack overflow error for long running streaming jobs.

    Execute one action DAG iteration and call recursion if streaming mode Must be implemented with tail recursion to avoid stack overflow error for long running streaming jobs.

    Annotations

    @tailrec()

11.  def finalize(): Unit

    Attributes

    protected[lang]

    Definition Classes

    AnyRef

    Annotations

    @throws( classOf[java.lang.Throwable] )

12.  final def getClass(): Class[_]

    Definition Classes

    AnyRef → Any

    Annotations

    @native()

13.  def hashCode(): Int

    Definition Classes

    AnyRef → Any

    Annotations

    @native()

14.  def initConfigFromEnvironment: SmartDataLakeBuilderConfig

    Create a new SDL configuration.

    Create a new SDL configuration.

    Could be used in the future to set default values.

    returns

    a new, initialized SmartDataLakeBuilderConfig.

15.  val instanceRegistry: InstanceRegistry

    InstanceRegistry instance

16.  final def isInstanceOf[T0]: Boolean

    Definition Classes

    Any

17.  lazy val logger: Logger

    Attributes

    protected

    Definition Classes

    SmartDataLakeLogger

    Annotations

    @transient()

18.  final def ne(arg0: AnyRef): Boolean

    Definition Classes

    AnyRef

19.  final def notify(): Unit

    Definition Classes

    AnyRef

    Annotations

    @native()

20.  final def notifyAll(): Unit

    Definition Classes

    AnyRef

    Annotations

    @native()

21.  def parseCommandLineArguments(args: Array[String], config: SmartDataLakeBuilderConfig): Option[SmartDataLakeBuilderConfig]

    Parses the supplied (command line) arguments.

    Parses the supplied (command line) arguments.

    This method parses command line arguments and creates the corresponding SmartDataLakeBuilderConfig

    args

    an Array of command line arguments.

    config

    a configuration initialized with default values.

    returns

    a new configuration with default values overwritten from the supplied command line arguments.

22.  val parser: OptionParser[SmartDataLakeBuilderConfig]

    The Parser defines how to extract the options from the command line args.

    The Parser defines how to extract the options from the command line args. Subclasses SmartDataLakeBuilder can define additional options to be extracted.

    Attributes

    protected

23.  def run(appConfig: SmartDataLakeBuilderConfig): Map[RuntimeEventState, Int]

    Run the application with the provided configuration.

    Run the application with the provided configuration.

    appConfig

    Application configuration (parsed from command line).

24.  def startSimulation(appConfig: SmartDataLakeBuilderConfig, initialSubFeeds: Seq[SparkSubFeed], dataObjectsState: Seq[DataObjectState] = Seq())(implicit instanceRegistry: InstanceRegistry, session: SparkSession): (Seq[SparkSubFeed], Map[RuntimeEventState, Int])

    Start a simulation run.

    Start a simulation run. This executes the DAG and returns all subfeeds including the transformed DataFrames. Only prepare and init are executed. All initial subfeeds must be provided as input.

    Note: this only works with SparkActions for now

    appConfig

    application configuration

    initialSubFeeds

    initial subfeeds for DataObjects at the beginning of the DAG

    dataObjectsState

    state for incremental DataObjects

    returns

    tuple of list of final subfeeds and statistics (action count per RuntimeEventState)

25.  final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes

    AnyRef

26.  def toString(): String

    Definition Classes

    AnyRef → Any

27.  final def wait(): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

28.  final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

29.  final def wait(arg0: Long): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws( ... ) @native()