LocalSmartDataLakeBuilder

Value Members

1. final def !=(arg0: Any): Boolean

   

   Definition Classes

   AnyRef → Any

2. final def ##(): Int

   

   Definition Classes

   AnyRef → Any

3. final def ==(arg0: Any): Boolean

   

   Definition Classes

   AnyRef → Any

4. val appType: String

   

   Definition Classes

   SmartDataLakeBuilder

5. val appVersion: String

   

   Definition Classes

   SmartDataLakeBuilder

6. final def asInstanceOf[T0]: T0

   

   Definition Classes

   Any

7. def clone(): AnyRef

   

   Attributes

   protected[java.lang]

   Definition Classes

   AnyRef

   Annotations

   @throws( ... )

8. final def eq(arg0: AnyRef): Boolean

   

   Definition Classes

   AnyRef

9. def equals(arg0: Any): Boolean

   

   Definition Classes

   AnyRef → Any

10. final def execActionDAG(actionDAGRun: ActionDAGRun, actionsSelected: Seq[Action], context: ActionPipelineContext, lastStartTime: Option[LocalDateTime] = None)(implicit session: SparkSession): Seq[SubFeed]

    

    Execute one action DAG iteration and call recursion if streaming mode Must be implemented with tail recursion to avoid stack overflow error for long running streaming jobs.

    Execute one action DAG iteration and call recursion if streaming mode Must be implemented with tail recursion to avoid stack overflow error for long running streaming jobs.

    Definition Classes

    SmartDataLakeBuilder

    Annotations

    @tailrec()

11. def finalize(): Unit

    

    Attributes

    protected[java.lang]

    Definition Classes

    AnyRef

    Annotations

    @throws( classOf[java.lang.Throwable] )

12. final def getClass(): Class[_]

    

    Definition Classes

    AnyRef → Any

13. def hashCode(): Int

    

    Definition Classes

    AnyRef → Any

14. def initConfigFromEnvironment: SmartDataLakeBuilderConfig

    

    Create a new SDL configuration.

    Create a new SDL configuration.

    Could be used in the future to set default values.

    returns

    a new, initialized SmartDataLakeBuilderConfig.

    Definition Classes

    SmartDataLakeBuilder

15. val instanceRegistry: InstanceRegistry

    

    InstanceRegistry instance

    InstanceRegistry instance

    Definition Classes

    SmartDataLakeBuilder

16. final def isInstanceOf[T0]: Boolean

    

    Definition Classes

    Any

17. lazy val logger: Logger

    

    Attributes

    protected

    Definition Classes

    SmartDataLakeLogger

18. def main(args: Array[String]): Unit

    

    Entry-Point of the application.

    Entry-Point of the application.

    args

    Command-line arguments.

19. final def ne(arg0: AnyRef): Boolean

    

    Definition Classes

    AnyRef

20. final def notify(): Unit

    

    Definition Classes

    AnyRef

21. final def notifyAll(): Unit

    

    Definition Classes

    AnyRef

22. def parseCommandLineArguments(args: Array[String], config: SmartDataLakeBuilderConfig): Option[SmartDataLakeBuilderConfig]

    

    Parses the supplied (command line) arguments.

    Parses the supplied (command line) arguments.

    This method parses command line arguments and creates the corresponding SmartDataLakeBuilderConfig

    args

    an Array of command line arguments.

    config

    a configuration initialized with default values.

    returns

    a new configuration with default values overwritten from the supplied command line arguments.

    Definition Classes

    SmartDataLakeBuilder

23. val parser: OptionParser[SmartDataLakeBuilderConfig]

    

    The Parser defines how to extract the options from the command line args.

    The Parser defines how to extract the options from the command line args. Subclasses SmartDataLakeBuilder can define additional options to be extracted.

    Attributes

    protected

    Definition Classes

    SmartDataLakeBuilder

24. def run(appConfig: SmartDataLakeBuilderConfig): Map[RuntimeEventState, Int]

    

    Run the application with the provided configuration.

    Run the application with the provided configuration.

    appConfig

    Application configuration (parsed from command line).

    Definition Classes

    SmartDataLakeBuilder

25. def startSimulation(appConfig: SmartDataLakeBuilderConfig, initialSubFeeds: Seq[SparkSubFeed], dataObjectsState: Seq[DataObjectState] = Seq())(implicit instanceRegistry: InstanceRegistry, session: SparkSession): (Seq[SparkSubFeed], Map[RuntimeEventState, Int])

    

    Start a simulation run.

    Start a simulation run. This executes the DAG and returns all subfeeds including the transformed DataFrames. Only prepare and init are executed. All initial subfeeds must be provided as input.

    Note: this only works with SparkActions for now

    appConfig

    application configuration

    initialSubFeeds

    initial subfeeds for DataObjects at the beginning of the DAG

    dataObjectsState

    state for incremental DataObjects

    returns

    tuple of list of final subfeeds and statistics (action count per RuntimeEventState)

    Definition Classes

    SmartDataLakeBuilder

26. final def synchronized[T0](arg0: ⇒ T0): T0

    

    Definition Classes

    AnyRef

27. def toString(): String

    

    Definition Classes

    AnyRef → Any

28. final def wait(): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

29. final def wait(arg0: Long, arg1: Int): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

30. final def wait(arg0: Long): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )