public final class SparkRuntimeUtils extends Object
Modifier and Type | Field and Description |
---|---|
static String |
CDAP_SPARK_EXECUTION_SERVICE_URI |
static co.cask.cdap.common.lang.FilterClassLoader.Filter |
SPARK_PROGRAM_CLASS_LOADER_FILTER |
Constructor and Description |
---|
SparkRuntimeUtils() |
Modifier and Type | Method and Description |
---|---|
static File |
createConfArchive(org.apache.spark.SparkConf sparkConf,
String propertiesEntryName,
String confDirPath,
String outputZipPath)
Creates a zip file which contains a serialized
Properties with a given zip entry name, together with
all files under the given directory. |
static Map<String,File> |
getLocalizedResources(File dir,
org.apache.spark.SparkConf sparkConf)
Retrieves the names of localized resources in the given config and constructs a map from the resource name
to local files with the resource names as the file names in the given directory.
|
static Cancellable |
initSparkMain()
Initialize a Spark main() method.
|
static void |
setLocalizedResources(Set<String> localizedResourcesNames,
Map<String,String> configs)
Saves the names of localized resources to the given config.
|
static <T> scala.collection.parallel.mutable.ParArray<T> |
setTaskSupport(scala.collection.parallel.mutable.ParArray<T> parArray)
Sets the
TaskSupport for the given Scala ParArray to ThreadPoolTaskSupport . |
public static final String CDAP_SPARK_EXECUTION_SERVICE_URI
public static final co.cask.cdap.common.lang.FilterClassLoader.Filter SPARK_PROGRAM_CLASS_LOADER_FILTER
public static File createConfArchive(org.apache.spark.SparkConf sparkConf, String propertiesEntryName, String confDirPath, String outputZipPath)
Properties
with a given zip entry name, together with
all files under the given directory. This is called from Client.createConfArchive() as a workaround for the
SPARK-13441 bug.sparkConf
- the SparkConf
to savepropertiesEntryName
- name of the zip entry for the propertiesconfDirPath
- directory to scan for files to include in the zip fileoutputZipPath
- output filepublic static <T> scala.collection.parallel.mutable.ParArray<T> setTaskSupport(scala.collection.parallel.mutable.ParArray<T> parArray)
TaskSupport
for the given Scala ParArray
to ThreadPoolTaskSupport
.
This method is mainly used by SparkRunnerClassLoader
to set the TaskSupport
for the
parallel array used inside the DStreamGraph
class in spark to avoid thread leakage after the
Spark program execution finished.public static void setLocalizedResources(Set<String> localizedResourcesNames, Map<String,String> configs)
public static Map<String,File> getLocalizedResources(File dir, org.apache.spark.SparkConf sparkConf)
public static Cancellable initSparkMain()
Cancellable
for releasing resources.Copyright © 2018 Cask Data, Inc. Licensed under the Apache License, Version 2.0.