dataflow

Type Members

1. type ActionResult = Seq[Option[Any]]

   

2. sealed case class CachePostAction[T, C](run: (Option[T], C) ⇒ Option[T], labelToIntercept: String) extends PostAction[T, C] with Product with Serializable

   

3. trait DataFlow[C] extends Logging

   

   Defines a state of the data flow.

   Defines a state of the data flow. State is defined by the inputs that are ready to be consumed and actions that need to be executed. In most of the BAU cases, initial state of the data flow has no inputs, as they need to be produced by the actions. When an action finishes, it can produce 0 or N outputs, to create next state of the data flow, that action is removed from the data flow and its outputs are added as inputs into the flow. This state transitioning will enable restarts of the flow from any point or debug/exploratory runs with already existing/manufactured/captured/materialised inputs.

   Also inputs are useful for unit testing, as they give access to all intermediate outputs of actions.

   C

   the type of context which we pass to the actions

4. trait DataFlowAction[C] extends AnyRef

   

   An action to be performed as part of a data flow.

   An action to be performed as part of a data flow. Actions declare what input labels they expect in order to start execution (0 .. N) and can produce outputs associated with the output labels (0 .. N). Executors will use these labels to execute to schedule the actions sequentially or in parallel.

   C

   the type of the context of the flow in which this action runs

5. trait DataFlowActionState extends AnyRef

   

   State of the action.

6. case class DataFlowActionTags(tags: Set[String], dependentOnTags: Set[String]) extends Product with Serializable

   

   Represents the tag state on a given action

   Represents the tag state on a given action

   tags

   Tags belonging to this action

   dependentOnTags

   Tags this action is dependent on

7. class DataFlowEntities extends AnyRef

   

   Maintains data flow entities (the inputs and outputs of data flow actions).

   Maintains data flow entities (the inputs and outputs of data flow actions). Every entity has a label which must be unique across the data flow.

8. class DataFlowException extends RuntimeException

   

9. trait DataFlowExecutor[C] extends AnyRef

   

   Created by Alexei Perelighin on 11/01/18.

10. case class DataFlowTagState(activeTags: Set[String], activeDependentOnTags: Set[String], taggedActions: Map[String, DataFlowActionTags]) extends Product with Serializable

    

    Represents the tag state on a DataFlow

    Represents the tag state on a DataFlow

    activeTags

    Tags currently active on the flow (i.e. within the tag() context)

    activeDependentOnTags

    Tag dependencies currently active on the flow (i.e. within the tagDependency() context)

    taggedActions

    Mapping of actions to their applied tag state

11. class EmptyFlowContext extends AnyRef

    

12. sealed case class Executed() extends DataFlowActionState with Product with Serializable

    

    Action was executed and can not be executed again.

13. sealed case class ExpectedInputIsEmpty(ready: Seq[String], notReady: Seq[String]) extends DataFlowActionState with Product with Serializable

    

    Can not be executed, as expected input is present, but is empty

14. trait FlowReporter[C] extends AnyRef

    

15. class InterceptorAction[C] extends DataFlowAction[C]

    

    This action can be added to the flow over an existing action, it will be scheduled instead of it and can override or intercept the behaviour of an action.

    This action can be added to the flow over an existing action, it will be scheduled instead of it and can override or intercept the behaviour of an action. This is useful when additional behaviours need to be added. Examples: register as spark temp views for sql, logging filtering, persisting to disk, dredging, etc.

    Created by Alexei Perelighin on 23/02/2018.

    C

    the type of the context of the flow in which this action runs

16. class NoReportingFlowReporter[C] extends FlowReporter[C]

    

17. sealed abstract class PostAction[T, C] extends AnyRef

    

18. case class PostActionInterceptor[T, C](toIntercept: DataFlowAction[C], postActions: Seq[PostAction[T, C]]) extends InterceptorAction[C] with Logging with Product with Serializable

    

19. sealed case class ReadyToRun(ready: Seq[String]) extends DataFlowActionState with Product with Serializable

    

    Action is ready to run.

20. sealed case class RequiresInput(ready: Seq[String], notReady: Seq[String]) extends DataFlowActionState with Product with Serializable

    

    Action can not be executed as it requires more inputs to be available.

21. class SequentialDataFlowExecutor[C] extends DataFlowExecutor[C] with Logging

    

    Created by Alexei Perelighin 2017/12/27

    Created by Alexei Perelighin 2017/12/27

    Executes one action at a time wihtout trying to parallelize them.

    C

    the type of context which we pass to the actions

22. class SimpleDataFlow[C] extends DataFlow[C]

    

23. sealed case class TransformPostAction[T, C](run: (Option[T], C) ⇒ Option[T], labelToIntercept: String) extends PostAction[T, C] with Product with Serializable