PythonJob

Value Members

1. final def !=(arg0: Any): Boolean

   

   Definition Classes

   AnyRef → Any

2. final def ##(): Int

   

   Definition Classes

   AnyRef → Any

3. final def ==(arg0: Any): Boolean

   

   Definition Classes

   AnyRef → Any

4. final def asInstanceOf[T0]: T0

   

   Definition Classes

   Any

5. def clone(): AnyRef

   

   Attributes

   protected[java.lang]

   Definition Classes

   AnyRef

   Annotations

   @throws( ... )

6. val eggPath: String

   

7. final def eq(arg0: AnyRef): Boolean

   

   Definition Classes

   AnyRef

8. def finalize(): Unit

   

   Attributes

   protected[java.lang]

   Definition Classes

   AnyRef

   Annotations

   @throws( classOf[java.lang.Throwable] )

9. final def getClass(): Class[_]

   

   Definition Classes

   AnyRef → Any

10. final def isInstanceOf[T0]: Boolean

    

    Definition Classes

    Any

11. val logger: Logger

    

12. val modulePath: String

    

13. final def ne(arg0: AnyRef): Boolean

    

    Definition Classes

    AnyRef

14. final def notify(): Unit

    

    Definition Classes

    AnyRef

15. final def notifyAll(): Unit

    

    Definition Classes

    AnyRef

16. val py4JImports: Seq[String]

    

17. def runJob(sc: X, runtime: JobEnvironment, data: Config): Any

    

    This is the entry point for a Spark Job Server to execute Python jobs.

    This is the entry point for a Spark Job Server to execute Python jobs. It calls a Python subprocess to execute the relevant Python Job class.

    sc

    a SparkContext or similar for the job. May be reused across jobs.

    runtime

    the JobEnvironment containing run time information pertaining to the job and context.

    data

    not used for Python jobs

    returns

    the job result

    Definition Classes

    PythonJob → SparkJobBase

18. final def synchronized[T0](arg0: ⇒ T0): T0

    

    Definition Classes

    AnyRef

19. def validate(sc: X, runtime: JobEnvironment, config: Config): Or[Config, Every[ValidationProblem]]

    

    To support a useful validate method here for Python jobs we would have call two python processes, one for validate and one for runJob.

    To support a useful validate method here for Python jobs we would have call two python processes, one for validate and one for runJob. However this is inefficient and it would mean having to convert JobData into a Java Object and then back out to a Python Object for runJob.

    So for Python Jobs this simply returns indicating the job is valid. Validation by the underlying Python class will be performed within the subprocess called during runJob.

    sc

    a SparkContext or similar for the job. May be reused across jobs.

    runtime

    the JobEnvironment containing run time information pertaining to the job and context.

    config

    the Typesafe Config object passed into the job request

    returns

    Always returns the jobConfig, so it will be passed on to runJob as the job data.

    Definition Classes

    PythonJob → SparkJobBase

20. final def wait(): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

21. final def wait(arg0: Long, arg1: Int): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

22. final def wait(arg0: Long): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )