kotlin-spark-api_3.2.0_2.13/org.jetbrains.kotlinx.spark.api/withSpark

withSpark

inline fun withSpark( props: Map<String, Any> = emptyMap(), master: String = SparkConf().get("spark.master", "local[*]"), appName: String = "Kotlin Spark Sample", logLevel: SparkLogLevel = ERROR, func: KSparkSession.() -> Unit)

Wrapper for spark creation which allows setting different spark params.

Parameters

props

spark options, value types are runtime-checked for type-correctness

master

Sets the Spark master URL to connect to, such as "local" to run locally, "local4" to run locally with 4 cores, or "spark://master:7077" to run on a Spark standalone cluster. By default, it tries to get the system value "spark.master", otherwise it uses "local*"

appName

Sets a name for the application, which will be shown in the Spark web UI. If no application name is set, a randomly generated name will be used.

logLevel

Control our logLevel. This overrides any user-defined log settings.

func

function which will be executed in context of KSparkSession (it means that this inside block will point to KSparkSession)

@JvmOverloads

inline fun withSpark( builder: SparkSession.Builder, logLevel: SparkLogLevel = ERROR, func: KSparkSession.() -> Unit)

Wrapper for spark creation which allows setting different spark params.

Parameters

builder

A SparkSession.Builder object, configured how you want.

logLevel

Control our logLevel. This overrides any user-defined log settings.

func

function which will be executed in context of KSparkSession (it means that this inside block will point to KSparkSession)

@JvmOverloads

inline fun withSpark( sparkConf: SparkConf, logLevel: SparkLogLevel = ERROR, func: KSparkSession.() -> Unit)

Wrapper for spark creation which copies params from sparkConf.

Parameters

sparkConf

Sets a list of config options based on this.

logLevel

Control our logLevel. This overrides any user-defined log settings.

func

function which will be executed in context of KSparkSession (it means that this inside block will point to KSparkSession)