Default Spark configuration values to use for the application
Default Spark configuration values to use for the application
the environment
a map containing default Spark configuration
Run the application for given environment and configuration case classes
Run the application for given environment and configuration case classes
the SparkSession
the environment
Cleans up the environment associated with this application
Cleans up the environment associated with this application
the SparkSession
the prefix for keys in the SparkConf needed by the Env implementation
Create the environment associated with this application
Create the environment associated with this application
the SparkSession
the prefix for keys in the SparkConf needed by the Env implementation
Parses configuration in the SparkSession into the environment case class (type E)
Parses configuration in the SparkSession into the environment case class (type E)
the SparkSession
the prefix for keys in the SparkConf needed by the Env implementation
a parsed case class of type E
Runs the application
During the development lifecycle of Spark applications, it is useful to create sandbox environments comprising paths and Hive databases etc. which are tied to specific logical environments (e.g. dev, test, prod) and feature development (i.e Git branches). e.g. when working on a feature called new_feature for a project called my_project, the application should write its data to paths under /data/dev/my_project/new_feature/ and create tables in a database called dev_my_project_new_feature (actual implementation of what these environments should look like can be defined by extending Env or one of its subclasses - the final implementation should be a case class whose values define the environment i.e env, branch etc.)
This is a generic Spark Application which uses an implementation of Env to generate application-specific configuration and subsequently parse this configuration into a case class to be used for the application logic.
the type of the Env implementation (must be a case class)