A B C D E F G H I K L M N O P R S T V W

A

AbstractSparkPythonUtil - Class in co.cask.cdap.app.runtime.spark.python

Abstract base utility class for PySpark.

AbstractSparkPythonUtil() - Constructor for class co.cask.cdap.app.runtime.spark.python.AbstractSparkPythonUtil

AbstractSparkSubmitter - Class in co.cask.cdap.app.runtime.spark.submit

Provides common implementation for different SparkSubmitter.

AbstractSparkSubmitter() - Constructor for class co.cask.cdap.app.runtime.spark.submit.AbstractSparkSubmitter

addCompiler(SparkCompiler, Closeable) - Method in class co.cask.cdap.app.runtime.spark.dynamic.SparkCompilerCleanupManager

Adds a SparkCompiler.

addMaster(Map<String, String>, ImmutableList.Builder<String>) - Method in class co.cask.cdap.app.runtime.spark.submit.AbstractSparkSubmitter

Add the --master argument for the Spark submission.

addMaster(Map<String, String>, ImmutableList.Builder<String>) - Method in class co.cask.cdap.app.runtime.spark.submit.DistributedSparkSubmitter

addMaster(Map<String, String>, ImmutableList.Builder<String>) - Method in class co.cask.cdap.app.runtime.spark.submit.LocalSparkSubmitter

addURL(URL) - Method in class org.apache.spark.util.MutableURLClassLoader

addURLs(Seq<URL>) - Method in class co.cask.cdap.app.runtime.spark.dynamic.SparkClassFileHandler

Adds a set of URLs to the class file search path.

B

beforeSubmit() - Method in class co.cask.cdap.app.runtime.spark.submit.AbstractSparkSubmitter

Called before submitting the Spark job.

beforeSubmit() - Method in class co.cask.cdap.app.runtime.spark.submit.DistributedSparkSubmitter

C

CDAP_SPARK_EXECUTION_SERVICE_URI - Static variable in class co.cask.cdap.app.runtime.spark.SparkRuntimeUtils

close() - Method in class co.cask.cdap.app.runtime.spark.dynamic.SparkCompilerCleanupManager

Performs cleanup for all SparkCompiler managed by this manager.

co.cask.cdap.app.runtime.spark - package co.cask.cdap.app.runtime.spark

co.cask.cdap.app.runtime.spark.classloader - package co.cask.cdap.app.runtime.spark.classloader

co.cask.cdap.app.runtime.spark.distributed - package co.cask.cdap.app.runtime.spark.distributed

co.cask.cdap.app.runtime.spark.dynamic - package co.cask.cdap.app.runtime.spark.dynamic

co.cask.cdap.app.runtime.spark.preview - package co.cask.cdap.app.runtime.spark.preview

co.cask.cdap.app.runtime.spark.python - package co.cask.cdap.app.runtime.spark.python

co.cask.cdap.app.runtime.spark.serializer - package co.cask.cdap.app.runtime.spark.serializer

co.cask.cdap.app.runtime.spark.stream - package co.cask.cdap.app.runtime.spark.stream

co.cask.cdap.app.runtime.spark.submit - package co.cask.cdap.app.runtime.spark.submit

completed(BasicWorkflowToken) - Method in class co.cask.cdap.app.runtime.spark.distributed.SparkExecutionClient

Sends a execution completed request to the execution service.

completed(HttpRequest, HttpResponder, String, String) - Method in class co.cask.cdap.app.runtime.spark.distributed.SparkExecutionService.SparkControllerHandler

Handles execution completion request from the running Spark program.

count(String, int) - Method in class co.cask.cdap.app.runtime.spark.SparkRuntimeContext

count(String, int) - Method in class co.cask.cdap.app.runtime.spark.SparkUserMetrics

create() - Static method in class co.cask.cdap.app.runtime.spark.SparkClassLoader

Creates a new SparkClassLoader from the execution context.

createConfArchive(SparkConf, String, String, String) - Static method in class co.cask.cdap.app.runtime.spark.SparkRuntimeUtils

Creates a zip file which contains a serialized Properties with a given zip entry name, together with all files under the given directory.

createJavaExecutionContext(SparkExecutionContext) - Method in class co.cask.cdap.app.runtime.spark.SparkClassLoader

Creates a new instance of JavaSparkExecutionContext by wrapping the given SparkExecutionContext.

createProgramClassLoaderParent() - Method in class co.cask.cdap.app.runtime.spark.distributed.DistributedSparkProgramRunner

createProgramController(TwillController, ProgramDescriptor, RunId) - Method in class co.cask.cdap.app.runtime.spark.distributed.DistributedSparkProgramRunner

createProgramInvocationClassLoader() - Method in class co.cask.cdap.app.runtime.spark.SparkRuntimeContext

createProgramRunner(ProgramType, ProgramRuntimeProvider.Mode, Injector) - Method in class co.cask.cdap.app.runtime.spark.SparkProgramRuntimeProvider

D

destroy(HandlerContext) - Method in class co.cask.cdap.app.runtime.spark.dynamic.SparkClassFileHandler

DistributedSparkProgramRunner - Class in co.cask.cdap.app.runtime.spark.distributed

A ProgramRunner for launching Spark program in distributed mode.

DistributedSparkProgramRunner(SparkCompat, TwillRunner, YarnConfiguration, CConfiguration, TokenSecureStoreRenewer, Impersonator, LocationFactory) - Constructor for class co.cask.cdap.app.runtime.spark.distributed.DistributedSparkProgramRunner

DistributedSparkSubmitter - Class in co.cask.cdap.app.runtime.spark.submit

A SparkSubmitter to submit Spark job that runs on cluster.

DistributedSparkSubmitter(Configuration, LocationFactory, String, SparkRuntimeContext, String) - Constructor for class co.cask.cdap.app.runtime.spark.submit.DistributedSparkSubmitter

E

equals(Object) - Method in class co.cask.cdap.app.runtime.spark.distributed.SparkCommand

executor() - Method in class co.cask.cdap.app.runtime.spark.distributed.SparkDriverService

F

findFromContext() - Static method in class co.cask.cdap.app.runtime.spark.SparkClassLoader

Finds the SparkClassLoader from the context ClassLoader hierarchy.

G

gauge(String, long) - Method in class co.cask.cdap.app.runtime.spark.SparkRuntimeContext

gauge(String, long) - Method in class co.cask.cdap.app.runtime.spark.SparkUserMetrics

get() - Static method in class co.cask.cdap.app.runtime.spark.SparkRuntimeContextProvider

Returns the current SparkRuntimeContext.

getApplicationSpecification() - Method in class co.cask.cdap.app.runtime.spark.SparkRuntimeContextConfig

getAuthenticationContext() - Method in class co.cask.cdap.app.runtime.spark.SparkRuntimeContext

Returns the AuthenticationContext that can be used for this program.

getAuthenticationContext(TaskAttemptContext) - Method in class co.cask.cdap.app.runtime.spark.stream.SparkStreamInputFormat

getAuthorizationEnforcer() - Method in class co.cask.cdap.app.runtime.spark.SparkRuntimeContext

Returns the AuthorizationEnforcer that can be used for this program.

getAuthorizationEnforcer(TaskAttemptContext) - Method in class co.cask.cdap.app.runtime.spark.stream.SparkStreamInputFormat

getBaseURI() - Method in class co.cask.cdap.app.runtime.spark.distributed.SparkExecutionService

Returns the base URI for talking to this service remotely through HTTP.

getBody() - Method in class co.cask.cdap.app.runtime.spark.SerializableStreamEvent

getCConfiguration() - Method in class co.cask.cdap.app.runtime.spark.SparkRuntimeContext

Returns the CDAP CConfiguration used for the execution.

getClassFile(HttpRequest, HttpResponder) - Method in class co.cask.cdap.app.runtime.spark.dynamic.SparkClassFileHandler

REST endpoint for the Spark executor nodes to load classes remotely.

getCommand() - Method in class co.cask.cdap.app.runtime.spark.distributed.SparkCommand

getConfiguration() - Method in class co.cask.cdap.app.runtime.spark.SparkRuntimeContext

Returns the Configuration used for the execution.

getConfiguration() - Method in class co.cask.cdap.app.runtime.spark.SparkRuntimeContextConfig

Returns the configuration.

getDatasetFramework() - Method in class co.cask.cdap.app.runtime.spark.SparkRuntimeContext

Returns the DatasetFramework used for this execution.

getHeaders() - Method in class co.cask.cdap.app.runtime.spark.SerializableStreamEvent

getHostname() - Method in class co.cask.cdap.app.runtime.spark.SparkRuntimeContext

Returns the hostname of the current container.

getLocalizedResources(File, SparkConf) - Static method in class co.cask.cdap.app.runtime.spark.SparkRuntimeUtils

Retrieves the names of localized resources in the given config and constructs a map from the resource name to local files with the resource names as the file names in the given directory.

getLocalPySparkArchives(SparkCompat) - Static method in class co.cask.cdap.app.runtime.spark.SparkPackageUtils

Returns the set of PySpark archive files.

getLocalSparkLibrary(SparkCompat) - Static method in class co.cask.cdap.app.runtime.spark.SparkPackageUtils

Returns the set of jar files for the spark library.

getLoggingContext() - Method in class co.cask.cdap.app.runtime.spark.SparkRuntimeContext

Returns the LoggingContext representing the program.

getName() - Method in class co.cask.cdap.app.runtime.spark.preview.SparkDataTracer

getOptions() - Method in class co.cask.cdap.app.runtime.spark.distributed.SparkCommand

getPluginArchive() - Method in class co.cask.cdap.app.runtime.spark.SparkRuntimeContextConfig

getPluginProperties(String) - Method in class co.cask.cdap.app.runtime.spark.SparkPluginContext

getPluginProperties(String, MacroEvaluator) - Method in class co.cask.cdap.app.runtime.spark.SparkPluginContext

getProgramClassLoader() - Method in class co.cask.cdap.app.runtime.spark.SparkClassLoader

Returns the program ClassLoader.

getProgramId() - Method in class co.cask.cdap.app.runtime.spark.SparkRuntimeContextConfig

getProgramOptions() - Method in class co.cask.cdap.app.runtime.spark.SparkRuntimeContextConfig

getRuntimeContext() - Method in class co.cask.cdap.app.runtime.spark.SparkClassLoader

Returns the SparkRuntimeContext.

getSecureData(String, String) - Method in class co.cask.cdap.app.runtime.spark.SparkSecureStore

getServiceURL(String, String) - Method in class co.cask.cdap.app.runtime.spark.SparkServiceDiscoverer

getServiceURL(String) - Method in class co.cask.cdap.app.runtime.spark.SparkServiceDiscoverer

getSparkClientEnv() - Static method in class co.cask.cdap.app.runtime.spark.SparkPackageUtils

Returns the environment for the Spark client container.

getSparkDefaultConf() - Static method in class co.cask.cdap.app.runtime.spark.SparkPackageUtils

Loads the spark-defaults.conf based on the environment.

getSparkEnv() - Static method in class co.cask.cdap.app.runtime.spark.SparkPackageUtils

Returns the Spark environment setup via the start up script.

getSparkExecutionContext(boolean) - Method in class co.cask.cdap.app.runtime.spark.SparkClassLoader

Returns the SparkExecutionContext.

getSparkSpecification() - Method in class co.cask.cdap.app.runtime.spark.SparkRuntimeContext

Returns the SparkSpecification of the spark program of this context.

getStreamAdmin() - Method in class co.cask.cdap.app.runtime.spark.SparkRuntimeContext

Returns the StreamAdmin used for this execution.

getSubmitConf() - Method in class co.cask.cdap.app.runtime.spark.submit.AbstractSparkSubmitter

Returns configs that are specific to the submission context.

getSubmitConf() - Method in class co.cask.cdap.app.runtime.spark.submit.DistributedSparkSubmitter

getTimestamp() - Method in class co.cask.cdap.app.runtime.spark.SerializableStreamEvent

getTransaction(int, long, TimeUnit) - Method in class co.cask.cdap.app.runtime.spark.SparkTransactionClient

Returns the Transaction for the given stage.

getTransaction(HttpRequest, HttpResponder, int) - Method in class co.cask.cdap.app.runtime.spark.SparkTransactionHandler

Handler method to get a serialized Transaction for the given stage.

getTransactionSystemClient() - Method in class co.cask.cdap.app.runtime.spark.SparkRuntimeContext

Returns the TransactionSystemClient for this execution.

getURLs() - Method in class org.apache.spark.util.MutableURLClassLoader

getWorkflowInfo() - Method in class co.cask.cdap.app.runtime.spark.SparkRuntimeContext

Returns the WorkflowProgramInfo if the spark program is running inside a workflow.

getWorkflowProgramInfo() - Method in class co.cask.cdap.app.runtime.spark.SparkRuntimeContextConfig

H

hashCode() - Method in class co.cask.cdap.app.runtime.spark.distributed.SparkCommand

HCONF_ATTR_CLUSTER_MODE - Static variable in class co.cask.cdap.app.runtime.spark.SparkRuntimeContextConfig

Configuration key for boolean value to tell whether Spark program is executed on a cluster or not.

HCONF_ATTR_CREDENTIALS_UPDATE_INTERVAL_MS - Static variable in class co.cask.cdap.app.runtime.spark.SparkRuntimeContextConfig

heartbeat(BasicWorkflowToken) - Method in class co.cask.cdap.app.runtime.spark.distributed.SparkExecutionClient

Sends a heartbeat request to the execution service.

heartbeat(HttpRequest, HttpResponder, String, String) - Method in class co.cask.cdap.app.runtime.spark.distributed.SparkExecutionService.SparkControllerHandler

Handles heartbeat request from the running Spark program.

I

info(String, Object) - Method in class co.cask.cdap.app.runtime.spark.preview.SparkDataTracer

initSparkMain() - Static method in class co.cask.cdap.app.runtime.spark.SparkRuntimeUtils

Initialize a Spark main() method.

isEnabled() - Method in class co.cask.cdap.app.runtime.spark.preview.SparkDataTracer

isLocal() - Method in class co.cask.cdap.app.runtime.spark.SparkRuntimeContextConfig

Returns true if in local mode.

isSupported(ProgramType, CConfiguration) - Method in class co.cask.cdap.app.runtime.spark.SparkProgramRuntimeProvider

K

KryoDecoder - Class in co.cask.cdap.app.runtime.spark.serializer

A Decoder for reading data from Kryo.

KryoDecoder(Input) - Constructor for class co.cask.cdap.app.runtime.spark.serializer.KryoDecoder

KryoEncoder - Class in co.cask.cdap.app.runtime.spark.serializer

A Encoder for writing data to Kryo.

KryoEncoder(Output) - Constructor for class co.cask.cdap.app.runtime.spark.serializer.KryoEncoder

L

launch(String, String[]) - Static method in class co.cask.cdap.app.runtime.spark.distributed.SparkContainerLauncher

Launches the given main class.

listSecureData(String) - Method in class co.cask.cdap.app.runtime.spark.SparkSecureStore

loadClass(String, boolean) - Method in class co.cask.cdap.app.runtime.spark.classloader.SparkRunnerClassLoader

loadPluginClass(String) - Method in class co.cask.cdap.app.runtime.spark.SparkPluginContext

LOCALIZED_CONF_DIR - Static variable in class co.cask.cdap.app.runtime.spark.SparkPackageUtils

LocalSparkSubmitter - Class in co.cask.cdap.app.runtime.spark.submit

A SparkSubmitter to submit Spark job that runs in the same local process.

LocalSparkSubmitter() - Constructor for class co.cask.cdap.app.runtime.spark.submit.LocalSparkSubmitter

M

MutableURLClassLoader - Class in org.apache.spark.util

This class is for Spark 1.3+ execution.

MutableURLClassLoader(URL[], ClassLoader) - Constructor for class org.apache.spark.util.MutableURLClassLoader

Constructor called from Spark framework.

N

needIntercept(String) - Method in class co.cask.cdap.app.runtime.spark.classloader.SparkContainerClassLoader

newPluginInstance(String) - Method in class co.cask.cdap.app.runtime.spark.SparkPluginContext

newPluginInstance(String, MacroEvaluator) - Method in class co.cask.cdap.app.runtime.spark.SparkPluginContext

O

onCompleted(boolean) - Method in class co.cask.cdap.app.runtime.spark.submit.AbstractSparkSubmitter

onCompleted(boolean) - Method in class co.cask.cdap.app.runtime.spark.submit.DistributedSparkSubmitter

org.apache.spark.util - package org.apache.spark.util

P

prepareSparkResources(SparkCompat, LocationFactory, File, Map<String, LocalizeResource>, Map<String, String>) - Static method in class co.cask.cdap.app.runtime.spark.SparkPackageUtils

Prepares the resources that need to be localized to the Spark client container.

R

read(Kryo, Input, Class<Schema>) - Method in class co.cask.cdap.app.runtime.spark.serializer.SchemaSerializer

read(Kryo, Input, Class<StructuredRecord>) - Method in class co.cask.cdap.app.runtime.spark.serializer.StructuredRecordSerializer

readBool() - Method in class co.cask.cdap.app.runtime.spark.serializer.KryoDecoder

readBytes() - Method in class co.cask.cdap.app.runtime.spark.serializer.KryoDecoder

readDouble() - Method in class co.cask.cdap.app.runtime.spark.serializer.KryoDecoder

readExternal(ObjectInput) - Method in class co.cask.cdap.app.runtime.spark.preview.SparkDataTracer

readExternal(ObjectInput) - Method in class co.cask.cdap.app.runtime.spark.SerializableStreamEvent

readExternal(ObjectInput) - Method in class co.cask.cdap.app.runtime.spark.SparkPluginContext

readExternal(ObjectInput) - Method in class co.cask.cdap.app.runtime.spark.SparkSecureStore

readExternal(ObjectInput) - Method in class co.cask.cdap.app.runtime.spark.SparkServiceDiscoverer

readExternal(ObjectInput) - Method in class co.cask.cdap.app.runtime.spark.SparkUserMetrics

readFloat() - Method in class co.cask.cdap.app.runtime.spark.serializer.KryoDecoder

readInt() - Method in class co.cask.cdap.app.runtime.spark.serializer.KryoDecoder

readLong() - Method in class co.cask.cdap.app.runtime.spark.serializer.KryoDecoder

readNull() - Method in class co.cask.cdap.app.runtime.spark.serializer.KryoDecoder

readString() - Method in class co.cask.cdap.app.runtime.spark.serializer.KryoDecoder

removeCompiler(SparkCompiler) - Method in class co.cask.cdap.app.runtime.spark.dynamic.SparkCompilerCleanupManager

Removes a SparkCompiler with resource cleanup through the Closeable provided in the SparkCompilerCleanupManager.addCompiler(SparkCompiler, Closeable) method.

removeURLs(Iterable<? extends URL>) - Method in class co.cask.cdap.app.runtime.spark.dynamic.SparkClassFileHandler

Removes a set of jar files.

report() - Method in class co.cask.cdap.app.runtime.spark.SparkMetricsSink

rewriteClass(String, InputStream) - Method in class co.cask.cdap.app.runtime.spark.classloader.SparkClassRewriter

rewriteClass(String, InputStream) - Method in class co.cask.cdap.app.runtime.spark.classloader.SparkContainerClassLoader

run() - Method in class co.cask.cdap.app.runtime.spark.distributed.SparkDriverService

run() - Method in class co.cask.cdap.app.runtime.spark.SparkCredentialsUpdater

run(LineageDatasetContext) - Method in interface co.cask.cdap.app.runtime.spark.SparkTxRunnable

Provides a LineageDatasetContext to get instances of Datasets.

S

SchemaSerializer - Class in co.cask.cdap.app.runtime.spark.serializer

A Kryo Serializer for Schema.

SchemaSerializer() - Constructor for class co.cask.cdap.app.runtime.spark.serializer.SchemaSerializer

SerializableStreamEvent - Class in co.cask.cdap.app.runtime.spark

A StreamEvent that can be serialized.

SerializableStreamEvent() - Constructor for class co.cask.cdap.app.runtime.spark.SerializableStreamEvent

SerializableStreamEvent(StreamEvent) - Constructor for class co.cask.cdap.app.runtime.spark.SerializableStreamEvent

set(SparkRuntimeContext, File) - Method in class co.cask.cdap.app.runtime.spark.SparkRuntimeContextConfig

Sets configurations based on the given context.

setGatewayCallbackPort(GatewayServer, int) - Static method in class co.cask.cdap.app.runtime.spark.python.SparkPythonUtil

Updates the python callback port in the GatewayServer.

setLocalizedResources(Set<String>, Map<String, String>) - Static method in class co.cask.cdap.app.runtime.spark.SparkRuntimeUtils

Saves the names of localized resources to the given config.

setTaskSupport(ParArray<T>) - Static method in class co.cask.cdap.app.runtime.spark.SparkRuntimeUtils

Sets the TaskSupport for the given Scala ParArray to ThreadPoolTaskSupport.

setupLaunchConfig(DistributedProgramRunner.LaunchConfig, Program, ProgramOptions, CConfiguration, Configuration, File) - Method in class co.cask.cdap.app.runtime.spark.distributed.DistributedSparkProgramRunner

shutDown() - Method in class co.cask.cdap.app.runtime.spark.distributed.SparkDriverService

shutDown() - Method in class co.cask.cdap.app.runtime.spark.distributed.SparkExecutionService

shutDown() - Method in class co.cask.cdap.app.runtime.spark.SparkCredentialsUpdater

shutdownNow() - Method in class co.cask.cdap.app.runtime.spark.distributed.SparkExecutionService

skipBytes() - Method in class co.cask.cdap.app.runtime.spark.serializer.KryoDecoder

skipDouble() - Method in class co.cask.cdap.app.runtime.spark.serializer.KryoDecoder

skipFloat() - Method in class co.cask.cdap.app.runtime.spark.serializer.KryoDecoder

skipString() - Method in class co.cask.cdap.app.runtime.spark.serializer.KryoDecoder

Spark2ProgramRuntimeProvider - Class in co.cask.cdap.app.runtime.spark

Spark2 program runtime provider.

Spark2ProgramRuntimeProvider() - Constructor for class co.cask.cdap.app.runtime.spark.Spark2ProgramRuntimeProvider

SPARK_PROGRAM_CLASS_LOADER_FILTER - Static variable in class co.cask.cdap.app.runtime.spark.SparkRuntimeUtils

SPARK_YARN_MODE - Static variable in class co.cask.cdap.app.runtime.spark.SparkPackageUtils

SparkClassFileHandler - Class in co.cask.cdap.app.runtime.spark.dynamic

A HttpHandler to serve class file to Spark executor.

SparkClassFileHandler() - Constructor for class co.cask.cdap.app.runtime.spark.dynamic.SparkClassFileHandler

SparkClassLoader - Class in co.cask.cdap.app.runtime.spark

ClassLoader being used in Spark execution context.

SparkClassLoader(SparkRuntimeContext) - Constructor for class co.cask.cdap.app.runtime.spark.SparkClassLoader

Creates a new SparkClassLoader with the given SparkRuntimeContext.

SparkClassRewriter - Class in co.cask.cdap.app.runtime.spark.classloader

A ClassRewriter for rewriting Spark related classes.

SparkClassRewriter(Function<String, URL>, boolean) - Constructor for class co.cask.cdap.app.runtime.spark.classloader.SparkClassRewriter

SparkCommand - Class in co.cask.cdap.app.runtime.spark.distributed

Command used by SparkExecutionService.

SparkCommand(String) - Constructor for class co.cask.cdap.app.runtime.spark.distributed.SparkCommand

SparkCommand(String, Map<String, String>) - Constructor for class co.cask.cdap.app.runtime.spark.distributed.SparkCommand

SparkCompilerCleanupManager - Class in co.cask.cdap.app.runtime.spark.dynamic

Manage SparkCompiler cleanup process.

SparkCompilerCleanupManager() - Constructor for class co.cask.cdap.app.runtime.spark.dynamic.SparkCompilerCleanupManager

SparkContainerClassLoader - Class in co.cask.cdap.app.runtime.spark.classloader

The ClassLoader for Spark containers in distributed mode.

SparkContainerClassLoader(URL[], ClassLoader) - Constructor for class co.cask.cdap.app.runtime.spark.classloader.SparkContainerClassLoader

Creates a new instance for the following set of URL.

SparkContainerLauncher - Class in co.cask.cdap.app.runtime.spark.distributed

This class launches Spark YARN containers with classes loaded through the SparkContainerClassLoader.

SparkContainerLauncher() - Constructor for class co.cask.cdap.app.runtime.spark.distributed.SparkContainerLauncher

SparkControllerHandler() - Constructor for class co.cask.cdap.app.runtime.spark.distributed.SparkExecutionService.SparkControllerHandler

SparkCredentialsUpdater - Class in co.cask.cdap.app.runtime.spark

This class is responsible for writing credentials to a location that Spark executor is expecting for secure credentials update for long running process.

SparkCredentialsUpdater(Supplier<Credentials>, Location, String, long, long, int) - Constructor for class co.cask.cdap.app.runtime.spark.SparkCredentialsUpdater

SparkDataTracer - Class in co.cask.cdap.app.runtime.spark.preview

A Externalizable implementation of DataTracer used in Spark program execution.

SparkDataTracer() - Constructor for class co.cask.cdap.app.runtime.spark.preview.SparkDataTracer

SparkDataTracer(SparkRuntimeContext, String) - Constructor for class co.cask.cdap.app.runtime.spark.preview.SparkDataTracer

Constructor.

SparkDriverService - Class in co.cask.cdap.app.runtime.spark.distributed

A service that runs in the Spark Driver process in distributed mode.

SparkDriverService(URI, SparkRuntimeContext) - Constructor for class co.cask.cdap.app.runtime.spark.distributed.SparkDriverService

SparkExecutionClient - Class in co.cask.cdap.app.runtime.spark.distributed

A client to the SparkExecutionService.

SparkExecutionClient(URI, ProgramRunId) - Constructor for class co.cask.cdap.app.runtime.spark.distributed.SparkExecutionClient

SparkExecutionService - Class in co.cask.cdap.app.runtime.spark.distributed

The HTTP service for communicating with the SparkMainWrapper running in the driver for controlling lifecycle management.

SparkExecutionService(LocationFactory, String, ProgramRunId, WorkflowToken) - Constructor for class co.cask.cdap.app.runtime.spark.distributed.SparkExecutionService

SparkExecutionService.SparkControllerHandler - Class in co.cask.cdap.app.runtime.spark.distributed

The HttpHandler for communicating with the Spark driver.

SparkMetricsSink - Class in co.cask.cdap.app.runtime.spark

A Sink which collects Metrics from Spark program and used SparkMetricsReporter to report it to MetricsContext

This full qualified name of this class is given to spark through the metrics configuration file.

SparkMetricsSink(Properties, MetricRegistry, SecurityManager) - Constructor for class co.cask.cdap.app.runtime.spark.SparkMetricsSink

SparkPackageUtils - Class in co.cask.cdap.app.runtime.spark

A utility class to help determine Spark supports and locating Spark jars and PySpark libraries.

SparkPluginContext - Class in co.cask.cdap.app.runtime.spark

A Externalizable implementation of PluginContext used in Spark program execution.

SparkPluginContext() - Constructor for class co.cask.cdap.app.runtime.spark.SparkPluginContext

Constructor.

SparkProgramRuntimeProvider - Class in co.cask.cdap.app.runtime.spark

A ProgramRuntimeProvider that provides runtime system support for ProgramType.SPARK program.

SparkProgramRuntimeProvider(SparkCompat) - Constructor for class co.cask.cdap.app.runtime.spark.SparkProgramRuntimeProvider

SparkPythonUtil - Class in co.cask.cdap.app.runtime.spark.python

Utility class to provide methods for PySpark integration.

SparkRunnerClassLoader - Class in co.cask.cdap.app.runtime.spark.classloader

A special ClassLoader for defining and loading all cdap-spark-core classes and Spark classes.

SparkRunnerClassLoader(URL[], ClassLoader, boolean) - Constructor for class co.cask.cdap.app.runtime.spark.classloader.SparkRunnerClassLoader

SparkRuntimeContext - Class in co.cask.cdap.app.runtime.spark

Context to be used at Spark runtime to provide common functionality that are needed at both the driver and the executors.

SparkRuntimeContextConfig - Class in co.cask.cdap.app.runtime.spark

Helper class for getting and setting configurations that are needed for recreating the runtime context in each Spark executor.

SparkRuntimeContextProvider - Class in co.cask.cdap.app.runtime.spark

Helper class for locating or creating SparkRuntimeContext from the execution context.

SparkRuntimeUtils - Class in co.cask.cdap.app.runtime.spark

Util class for common functions needed for Spark implementation.

SparkRuntimeUtils() - Constructor for class co.cask.cdap.app.runtime.spark.SparkRuntimeUtils

SparkSecureStore - Class in co.cask.cdap.app.runtime.spark

A Externalizable implementation of SecureStore used in Spark program execution.

SparkSecureStore() - Constructor for class co.cask.cdap.app.runtime.spark.SparkSecureStore

Constructor.

SparkServiceDiscoverer - Class in co.cask.cdap.app.runtime.spark

A Externalizable implementation of ServiceDiscoverer used in Spark program execution.

SparkServiceDiscoverer() - Constructor for class co.cask.cdap.app.runtime.spark.SparkServiceDiscoverer

Constructor.

SparkStreamInputFormat<K,V> - Class in co.cask.cdap.app.runtime.spark.stream

StreamInputFomat for Spark jobs.

SparkStreamInputFormat() - Constructor for class co.cask.cdap.app.runtime.spark.stream.SparkStreamInputFormat

SparkSubmitter - Interface in co.cask.cdap.app.runtime.spark.submit

Interface to provide abstraction for submitting a Spark program to the Spark framework for execution.

SparkTransactionClient - Class in co.cask.cdap.app.runtime.spark

Client class to interact with SparkTransactionHandler through HTTP.

SparkTransactionClient(URI) - Constructor for class co.cask.cdap.app.runtime.spark.SparkTransactionClient

SparkTransactionClient(URI, long) - Constructor for class co.cask.cdap.app.runtime.spark.SparkTransactionClient

SparkTransactionHandler - Class in co.cask.cdap.app.runtime.spark

Provides transaction management for Spark job and stage executors.

SparkTxRunnable - Interface in co.cask.cdap.app.runtime.spark

An interface for executing task inside transaction.

SparkUserMetrics - Class in co.cask.cdap.app.runtime.spark

A Externalizable implementation of Metrics used in Spark program execution.

SparkUserMetrics() - Constructor for class co.cask.cdap.app.runtime.spark.SparkUserMetrics

Constructor.

start() - Method in class co.cask.cdap.app.runtime.spark.SparkMetricsSink

startPy4jGateway(Path) - Static method in class co.cask.cdap.app.runtime.spark.python.AbstractSparkPythonUtil

Starts a Py4j gateway server.

startUp() - Method in class co.cask.cdap.app.runtime.spark.distributed.SparkDriverService

startUp() - Method in class co.cask.cdap.app.runtime.spark.distributed.SparkExecutionService

startUp() - Method in class co.cask.cdap.app.runtime.spark.SparkCredentialsUpdater

STOP - Static variable in class co.cask.cdap.app.runtime.spark.distributed.SparkCommand

Command for stopping the spark execution.

stop() - Method in class co.cask.cdap.app.runtime.spark.SparkMetricsSink

StructuredRecordSerializer - Class in co.cask.cdap.app.runtime.spark.serializer

A Kryo Serializer for StructuredRecord.

StructuredRecordSerializer() - Constructor for class co.cask.cdap.app.runtime.spark.serializer.StructuredRecordSerializer

submit(SparkRuntimeContext, Map<String, String>, List<LocalizeResource>, URI, V) - Method in class co.cask.cdap.app.runtime.spark.submit.AbstractSparkSubmitter

submit(SparkRuntimeContext, Map<String, String>, List<LocalizeResource>, URI, V) - Method in interface co.cask.cdap.app.runtime.spark.submit.SparkSubmitter

Submits a Spark job to the Spark framework.

T

toString() - Method in class co.cask.cdap.app.runtime.spark.SparkRuntimeContext

triggerShutdown() - Method in class co.cask.cdap.app.runtime.spark.distributed.SparkDriverService

triggerShutdown() - Method in class co.cask.cdap.app.runtime.spark.submit.AbstractSparkSubmitter

Invoked for stopping the Spark job explicitly.

triggerShutdown() - Method in class co.cask.cdap.app.runtime.spark.submit.DistributedSparkSubmitter

triggerShutdown() - Method in class co.cask.cdap.app.runtime.spark.submit.LocalSparkSubmitter

V

validateOptions(Program, ProgramOptions) - Method in class co.cask.cdap.app.runtime.spark.distributed.DistributedSparkProgramRunner

W

write(Kryo, Output, Schema) - Method in class co.cask.cdap.app.runtime.spark.serializer.SchemaSerializer

write(Kryo, Output, StructuredRecord) - Method in class co.cask.cdap.app.runtime.spark.serializer.StructuredRecordSerializer

writeBool(boolean) - Method in class co.cask.cdap.app.runtime.spark.serializer.KryoEncoder

writeBytes(byte[]) - Method in class co.cask.cdap.app.runtime.spark.serializer.KryoEncoder

writeBytes(byte[], int, int) - Method in class co.cask.cdap.app.runtime.spark.serializer.KryoEncoder

writeBytes(ByteBuffer) - Method in class co.cask.cdap.app.runtime.spark.serializer.KryoEncoder

writeConfig(File) - Static method in class co.cask.cdap.app.runtime.spark.SparkMetricsSink

Writes a properties file which is used to config Spark Metrics in SparkConf.

writeCredentials(Location) - Method in class co.cask.cdap.app.runtime.spark.distributed.SparkExecutionClient

Sends a write credentials request to the execution service.

writeCredentials(HttpRequest, HttpResponder, String, String) - Method in class co.cask.cdap.app.runtime.spark.distributed.SparkExecutionService.SparkControllerHandler

Handles request for Credentials.

writeDouble(double) - Method in class co.cask.cdap.app.runtime.spark.serializer.KryoEncoder

writeExternal(ObjectOutput) - Method in class co.cask.cdap.app.runtime.spark.preview.SparkDataTracer

writeExternal(ObjectOutput) - Method in class co.cask.cdap.app.runtime.spark.SerializableStreamEvent

writeExternal(ObjectOutput) - Method in class co.cask.cdap.app.runtime.spark.SparkPluginContext

writeExternal(ObjectOutput) - Method in class co.cask.cdap.app.runtime.spark.SparkSecureStore

writeExternal(ObjectOutput) - Method in class co.cask.cdap.app.runtime.spark.SparkServiceDiscoverer

writeExternal(ObjectOutput) - Method in class co.cask.cdap.app.runtime.spark.SparkUserMetrics

writeFloat(float) - Method in class co.cask.cdap.app.runtime.spark.serializer.KryoEncoder

writeInt(int) - Method in class co.cask.cdap.app.runtime.spark.serializer.KryoEncoder

writeLong(long) - Method in class co.cask.cdap.app.runtime.spark.serializer.KryoEncoder

writeNull() - Method in class co.cask.cdap.app.runtime.spark.serializer.KryoEncoder

writeString(String) - Method in class co.cask.cdap.app.runtime.spark.serializer.KryoEncoder

A B C D E F G H I K L M N O P R S T V W