spark

Type Members

1. case class LabelCommitDefinition(basePath: String, timestampFolder: Option[String] = None, partitions: Seq[String] = Seq.empty, connection: Option[HadoopDBConnector] = None) extends Product with Serializable

   

2. class SimpleAction[C] extends DataFlowAction[C]

   

   Instances of this class build a bridge between OOP part of the Waimak engine and functional definition of the data flow.

   Instances of this class build a bridge between OOP part of the Waimak engine and functional definition of the data flow.

   Created by Alexei Perelighin on 03/11/17.

   C

   the type of the context of the flow in which this action runs

3. class SimpleSparkDataFlow extends SparkDataFlow with Logging

   

   Created by Alexei Perelighin on 22/12/17.

4. trait SparkDataFlow extends DataFlow[SparkFlowContext] with Logging

   

   Introduces spark session into the data flows

5. trait SparkDataFlowAction extends DataFlowAction[SparkFlowContext]

   

6. case class SparkFlowContext(spark: SparkSession) extends Product with Serializable

   

   Context required in a Spark data flow (SparkSession and FileSystem)

   Context required in a Spark data flow (SparkSession and FileSystem)

   Created by Vicky Avison on 23/02/2018.

   spark

   the SparkSession

7. class SparkSimpleAction extends SimpleAction[SparkFlowContext]

   

   Spark specific simple action, that sets spark specific generics.