Package com.pulumi.gcp.dataproc.outputs

Class JobPysparkConfig

  • public final class JobPysparkConfig
extends java.lang.Object

    • Method Detail

      • archiveUris

        public java.util.List<java.lang.String> archiveUris()
        Returns:
        HCFS URIs of archives to be extracted in the working directory of .jar, .tar, .tar.gz, .tgz, and .zip.

      • args

        public java.util.List<java.lang.String> args()
        Returns:
        The arguments to pass to the driver.

      • fileUris

        public java.util.List<java.lang.String> fileUris()
        Returns:
        HCFS URIs of files to be copied to the working directory of Python drivers and distributed tasks. Useful for naively parallel tasks.

      • jarFileUris

        public java.util.List<java.lang.String> jarFileUris()
        Returns:
        HCFS URIs of jar files to add to the CLASSPATHs of the Python driver and tasks.

      • mainPythonFileUri

        public java.lang.String mainPythonFileUri()
        Returns:
        The HCFS URI of the main Python file to use as the driver. Must be a .py file.

      • properties

        public java.util.Map<java.lang.String,​java.lang.String> properties()
        Returns:
        A mapping of property names to values, used to configure PySpark. Properties that conflict with values set by the Cloud Dataproc API may be overwritten. Can include properties set in `/etc/spark/conf/spark-defaults.conf` and classes in user code. * `logging_config.driver_log_levels`- (Required) The per-package log levels for the driver. This may include 'root' package name to configure rootLogger. Examples: 'com.google = FATAL', 'root = INFO', 'org.apache = DEBUG'

      • pythonFileUris

        public java.util.List<java.lang.String> pythonFileUris()
        Returns:
        HCFS file URIs of Python files to pass to the PySpark framework. Supported file types: .py, .egg, and .zip.