Allows multiple Spark applications to be run in a single main method whilst obeying configured dependency constraints.
The following configuration values should be present in the SparkSession:
spark.waimak.apprunner.apps: a comma-delimited list of the names (identifiers) of all of the applications being run
(e.g. myapp1,myapp2)
spark.waimak.apprunner.{appname}.appClassName: for each application, the application class to use (must extend SparkApp)
(e.g. spark.waimak.apprunner.myapp1.appClassName = com.example.MyWaimakApp)
spark.waimak.apprunner.{appname}.dependencies: for each application, an optional comma-delimited list of dependencies.
If omitted, the application will have no dependencies and will not wait for other apps to finish before starting execution.
Dependencies must match the names provided in spark.waimak.apprunner.apps
(e.g. spark.waimak.apprunner.myapp1.dependencies = myapp2)
The Env implementation used by the provided SparkApp implementation expects configuration values prefixed with:
spark.waimak.environment.{appname}.
Allows multiple Spark applications to be run in a single main method whilst obeying configured dependency constraints. The following configuration values should be present in the SparkSession:
spark.waimak.apprunner.apps: a comma-delimited list of the names (identifiers) of all of the applications being run (e.g. myapp1,myapp2)
spark.waimak.apprunner.{appname}.appClassName: for each application, the application class to use (must extend SparkApp) (e.g. spark.waimak.apprunner.myapp1.appClassName = com.example.MyWaimakApp)
spark.waimak.apprunner.{appname}.dependencies: for each application, an optional comma-delimited list of dependencies. If omitted, the application will have no dependencies and will not wait for other apps to finish before starting execution. Dependencies must match the names provided in spark.waimak.apprunner.apps (e.g. spark.waimak.apprunner.myapp1.dependencies = myapp2)
The Env implementation used by the provided SparkApp implementation expects configuration values prefixed with: spark.waimak.environment.{appname}.