Checks the config and returns an error message, or null if the config is fine.
Checks the config and returns an error message, or null if the config is fine. The error message will be returned to the user as a 404 HTTP error code.
The main class that carries out the Spark job.
The main class that carries out the Spark job. The results will be converted to JSON and emitted (but NOT persisted).
This is the entry point for a Spark Job Server to execute Spark jobs.
This is the entry point for a Spark Job Server to execute Spark jobs. This function should create or reuse RDDs and return the result at the end, which the Job Server will cache or display.
a SparkContext or similar for the job. May be reused across jobs.
the Typesafe Config object passed into the job request
the job result
This method is called by the job server to allow jobs to validate their input and reject invalid job requests.
This method is called by the job server to allow jobs to validate their input and reject invalid job requests. If SparkJobInvalid is returned, then the job server returns 400 to the user. NOTE: this method should return very quickly. If it responds slowly then the job server may time out trying to start this job.
either SparkJobValid or SparkJobInvalid
A class to make Java jobs easier to write. In Java: public class MySparkJob extends JavaSparkJob {