JobEndpoint
Related Doc:
package python
The contextConfig, which is a Typesafe Config object, serialized to HOCON, so that it can be deserialized by the Python implementation of Typesafe Config (pyhocon) in the subprocess.
Method for python subprocess to retrieve job data in run stage.
Method for python subprocess to retrieve job data in run stage.
the jobData if it exists, else null
Returns the list of py4J imports as a Java list, which is the necessary type to have the list auto-converted to a python list.
The jobConfig, which is a Typesafe Config object, serialized to HOCON, so that it can be deserialized by the Python implementation of Typesafe Config (pyhocon) in the subprocess.
Method for the python subprocess to return jobdata after validate stage
Method for the python subprocess to return jobdata after validate stage
the job data to be held for the run stage
Receives the result from the python subprocess.
Receives the result from the python subprocess.
The result of the process. Although this can be Any, if it cannot
be serialized by Spray Json then an error will occur when attempting
to return this result to the client.
Receives the list of validation problems from the python subprocess.
Receives the list of validation problems from the python subprocess.
A list of problems. This is of type java.util.ArrayList[String]
because this is the format that Python lists are converted to
by Py4J. This is converted to a Scala Seq internally.
The target for all communications between the Spark Job Server and the underlying python subprocess.
When a Py4J gateway is created, it is passed an object of type
Anywhich is available to the python process as a field on the gateway namedendpoint. Since the object is of typeAny, it is up to the python program to know which members to expect the endpoint to have.The Spark Job Server python subprocess assumes the endpoint to be an implementation of this Trait, and attempts to access fields and methods accordingly.