public final class SparkPackageUtils extends Object
Modifier and Type | Field and Description |
---|---|
static String |
LOCALIZED_CONF_DIR |
static String |
SPARK_YARN_MODE |
Modifier and Type | Method and Description |
---|---|
static Set<File> |
getLocalPySparkArchives(SparkCompat sparkCompat)
Returns the set of PySpark archive files.
|
static Set<File> |
getLocalSparkLibrary(SparkCompat sparkCompat)
Returns the set of jar files for the spark library.
|
static Map<String,String> |
getSparkClientEnv()
Returns the environment for the Spark client container.
|
static Properties |
getSparkDefaultConf()
Loads the spark-defaults.conf based on the environment.
|
static Map<String,String> |
getSparkEnv()
Returns the Spark environment setup via the start up script.
|
static void |
prepareSparkResources(SparkCompat sparkCompat,
LocationFactory locationFactory,
File tempDir,
Map<String,LocalizeResource> localizeResources,
Map<String,String> env)
Prepares the resources that need to be localized to the Spark client container.
|
public static final String SPARK_YARN_MODE
public static final String LOCALIZED_CONF_DIR
public static Set<File> getLocalSparkLibrary(SparkCompat sparkCompat)
public static Set<File> getLocalPySparkArchives(SparkCompat sparkCompat)
public static void prepareSparkResources(SparkCompat sparkCompat, LocationFactory locationFactory, File tempDir, Map<String,LocalizeResource> localizeResources, Map<String,String> env) throws IOException
sparkCompat
- the spark version to prepare forlocationFactory
- the location factory for uploading filestempDir
- a temporary directory for file creationlocalizeResources
- A map from localized name to LocalizeResource
for this method to updateenv
- the environment map to updateIOException
- if failed to prepare the spark resourcespublic static Map<String,String> getSparkEnv()
public static Map<String,String> getSparkClientEnv()
public static Properties getSparkDefaultConf()
Properties
object representing Spark default configurations.Copyright © 2018 Cask Data, Inc. Licensed under the Apache License, Version 2.0.