public final class DistributedSparkProgramRunner extends DistributedProgramRunner implements ProgramClassLoaderProvider
ProgramRunner
for launching Spark
program in distributed mode. It starts
a YARN application to act as the Spark client. A second YARN application will be launched
by Spark framework as the actual Spark program execution.DistributedProgramRunner.LaunchConfig
cConf, hConf, secureStoreRenewer
Constructor and Description |
---|
DistributedSparkProgramRunner(SparkCompat sparkComat,
TwillRunner twillRunner,
YarnConfiguration hConf,
co.cask.cdap.common.conf.CConfiguration cConf,
TokenSecureStoreRenewer tokenSecureStoreRenewer,
co.cask.cdap.security.impersonation.Impersonator impersonator,
LocationFactory locationFactory) |
Modifier and Type | Method and Description |
---|---|
ClassLoader |
createProgramClassLoaderParent() |
ProgramController |
createProgramController(TwillController twillController,
ProgramDescriptor programDescriptor,
RunId runId) |
protected void |
setupLaunchConfig(DistributedProgramRunner.LaunchConfig launchConfig,
Program program,
ProgramOptions options,
co.cask.cdap.common.conf.CConfiguration cConf,
Configuration hConf,
File tempDir) |
protected void |
validateOptions(Program program,
ProgramOptions options) |
beforeLaunch, createEventHandler, run
@Inject public DistributedSparkProgramRunner(SparkCompat sparkComat, TwillRunner twillRunner, YarnConfiguration hConf, co.cask.cdap.common.conf.CConfiguration cConf, TokenSecureStoreRenewer tokenSecureStoreRenewer, co.cask.cdap.security.impersonation.Impersonator impersonator, LocationFactory locationFactory)
public ProgramController createProgramController(TwillController twillController, ProgramDescriptor programDescriptor, RunId runId)
createProgramController
in class DistributedProgramRunner
protected void validateOptions(Program program, ProgramOptions options)
validateOptions
in class DistributedProgramRunner
protected void setupLaunchConfig(DistributedProgramRunner.LaunchConfig launchConfig, Program program, ProgramOptions options, co.cask.cdap.common.conf.CConfiguration cConf, Configuration hConf, File tempDir) throws IOException
setupLaunchConfig
in class DistributedProgramRunner
IOException
public ClassLoader createProgramClassLoaderParent()
createProgramClassLoaderParent
in interface ProgramClassLoaderProvider
Copyright © 2018 Cask Data, Inc. Licensed under the Apache License, Version 2.0.