app

Type Members

1. case class AllApps(apps: Seq[String]) extends Product with Serializable

   

2. trait BaseEnv extends Env

   

   Environment which provides a base path into which the application can write its data Unless overridden, paths will be of the form {uri}/data/{environment}/{project}/{branch} where environment is the logical environment (e.g.

   Environment which provides a base path into which the application can write its data Unless overridden, paths will be of the form {uri}/data/{environment}/{project}/{branch} where environment is the logical environment (e.g. dev, test), project is the name of the application and branch is the Git branch

   N.B when environment is 'prod', the branch is omitted from the path as we assume it will always be master

   e.g. hdfs:///data/dev/my_project/feature_abc, hdfs:///data/prod/my_project

3. trait Env extends Logging

   

   Environment defining a sandbox in which an application can write

4. case class EnvironmentAction(action: String, appClassName: String) extends Product with Serializable

   

5. trait HiveEnv extends BaseEnv

   

   Environment which provides databases.

   Environment which provides databases. By default, there will be a single database of the form {environment}_{project}_{branch} where environment is the logical environment (e.g. dev, test), project is the name of the application and branch is the Git branch

   N.B when environment is 'prod', the branch is omitted from the database name as we assume it will always be master

   e.g. dev_my_project_feature_abc, prod_my_project

6. case class SingleAppConfig(appClassName: String, dependencies: Seq[String] = Nil) extends Product with Serializable

   

7. abstract class SparkApp[E <: Env] extends AnyRef

   

   During the development lifecycle of Spark applications, it is useful to create sandbox environments comprising paths and Hive databases etc.

   During the development lifecycle of Spark applications, it is useful to create sandbox environments comprising paths and Hive databases etc. which are tied to specific logical environments (e.g. dev, test, prod) and feature development (i.e Git branches). e.g. when working on a feature called new_feature for a project called my_project, the application should write its data to paths under /data/dev/my_project/new_feature/ and create tables in a database called dev_my_project_new_feature (actual implementation of what these environments should look like can be defined by extending Env or one of its subclasses - the final implementation should be a case class whose values define the environment i.e env, branch etc.)

   This is a generic Spark Application which uses an implementation of Env to generate application-specific configuration and subsequently parse this configuration into a case class to be used for the application logic.

   E

   the type of the Env implementation (must be a case class)

8. abstract class WaimakApp[E <: Env] extends SparkApp[E]

   

   This is a SparkApp specifically for applications using Waimak

   This is a SparkApp specifically for applications using Waimak

   E

   the type of the Env implementation (must be a case class)