io.cdap.cdap.api.spark

Interface SparkConfigurer

All Superinterfaces:

DatasetConfigurer, FeatureFlagsProvider, PluginConfigurer, ProgramConfigurer

@Beta
public interface SparkConfigurer
extends ProgramConfigurer, DatasetConfigurer, PluginConfigurer

Configurer for configuring Spark.

Method Summary

Modifier and Type	Method and Description
void	setClientResources(Resources resources) Sets the resources requirement for the Spark client process.
void	setDriverResources(Resources resources) Sets the resources requirement for the Spark driver process.
void	setExecutorResources(Resources resources) Sets the resources requirement for the Spark executor processes.
void	setMainClassName(String className) Sets the Spark program class name in specification.

Methods inherited from interface io.cdap.cdap.api.ProgramConfigurer

setDescription, setName, setProperties

Methods inherited from interface io.cdap.cdap.api.feature.FeatureFlagsProvider

isFeatureEnabled

Methods inherited from interface io.cdap.cdap.api.DatasetConfigurer

addDatasetModule, addDatasetType, createDataset, createDataset, createDataset, createDataset

Methods inherited from interface io.cdap.cdap.api.plugin.PluginConfigurer

evaluateMacros, evaluateMacros, usePlugin, usePlugin, usePluginClass, usePluginClass

Method Detail

setMainClassName

void setMainClassName(String className)

Sets the Spark program class name in specification. The provided class should have a static void main(String[]) method. It can also implements the io.cdap.cdap.api.spark.SparkMain interface for Spark program written in Scala or the io.cdap.cdap.api.spark.JavaSparkMain for Spark program written in Java, as defined in the cdap-api-spark* module.

Parameters:

className - the fully qualified name of Spark program class.

setClientResources

void setClientResources(Resources resources)

Sets the resources requirement for the Spark client process. It is the process where the ProgramLifecycle.initialize(RuntimeContext) and ProgramLifecycle.destroy() methods get executed.

setDriverResources

void setDriverResources(Resources resources)

Sets the resources requirement for the Spark driver process.

setExecutorResources

void setExecutorResources(Resources resources)

Sets the resources requirement for the Spark executor processes.