GlobalConfig

Global configuration options

kryoClasses: classes to register for spark kryo serialization
sparkOptions: spark options
enableHive: enable hive for spark session
memoryLogTimer: enable periodic memory usage logging, see detailed configuration MemoryLogTimerConfig
shutdownHookLogger: enable shutdown hook logger to trace shutdown cause
stateListeners: Define state listeners to be registered for receiving events of the execution of SmartDataLake job
sparkUDFs: Define UDFs to be registered in spark session. The registered UDFs are available in Spark SQL transformations and expression evaluation, e.g. configuration of ExecutionModes.
pythonUDFs: Define UDFs in python to be registered in spark session. The registered UDFs are available in Spark SQL transformations but not for expression evaluation.
secretProviders: Define SecretProvider's to be registered.
allowOverwriteAllPartitionsWithoutPartitionValues: Configure a list of exceptions for partitioned DataObject id's, which are allowed to overwrite the all partitions of a table if no partition values are set. This is used to override/avoid a protective error when using SDLSaveMode.OverwriteOptimized|OverwritePreserveDirectories. Define it as a list of DataObject id's.
runtimeDataNumberOfExecutionsToKeep: Number of Executions to keep runtime data for in streaming mode (default = 10). Must be bigger than 1.
synchronousStreamingTriggerIntervalSec: Trigger interval for synchronous actions in streaming mode in seconds (default = 60 seconds) The synchronous actions of the DAG will be executed with this interval if possile. Note that for asynchronous actions there are separate settings, e.g. SparkStreamingMode.triggerInterval.

Linear Supertypes

Serializable, Serializable, Product, Equals, SmartDataLakeLogger, AnyRef, Any

Instance Constructors

new GlobalConfig(kryoClasses: Option[Seq[String]] = None, sparkOptions: Option[Map[String, String]] = None, enableHive: Boolean = true, memoryLogTimer: Option[MemoryLogTimerConfig] = None, shutdownHookLogger: Boolean = false, stateListeners: Seq[StateListenerConfig] = Seq(), sparkUDFs: Option[Map[String, SparkUDFCreatorConfig]] = None, pythonUDFs: Option[Map[String, PythonUDFCreatorConfig]] = None, secretProviders: Option[Map[String, SecretProviderConfig]] = None, allowOverwriteAllPartitionsWithoutPartitionValues: Seq[DataObjectId] = Seq(), runtimeDataNumberOfExecutionsToKeep: Int = 10, synchronousStreamingTriggerIntervalSec: Int = 60)

kryoClasses
classes to register for spark kryo serialization
sparkOptions
spark options
enableHive
enable hive for spark session
memoryLogTimer
enable periodic memory usage logging, see detailed configuration MemoryLogTimerConfig
shutdownHookLogger
enable shutdown hook logger to trace shutdown cause
stateListeners
Define state listeners to be registered for receiving events of the execution of SmartDataLake job
sparkUDFs
Define UDFs to be registered in spark session. The registered UDFs are available in Spark SQL transformations and expression evaluation, e.g. configuration of ExecutionModes.
pythonUDFs
Define UDFs in python to be registered in spark session. The registered UDFs are available in Spark SQL transformations but not for expression evaluation.
secretProviders
Define SecretProvider's to be registered.
allowOverwriteAllPartitionsWithoutPartitionValues
Configure a list of exceptions for partitioned DataObject id's, which are allowed to overwrite the all partitions of a table if no partition values are set. This is used to override/avoid a protective error when using SDLSaveMode.OverwriteOptimized|OverwritePreserveDirectories. Define it as a list of DataObject id's.
runtimeDataNumberOfExecutionsToKeep
Number of Executions to keep runtime data for in streaming mode (default = 10). Must be bigger than 1.
synchronousStreamingTriggerIntervalSec
Trigger interval for synchronous actions in streaming mode in seconds (default = 60 seconds) The synchronous actions of the DAG will be executed with this interval if possile. Note that for asynchronous actions there are separate settings, e.g. SparkStreamingMode.triggerInterval.

Value Members

final def !=(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def ##(): Int

Definition Classes
AnyRef → Any
final def ==(arg0: Any): Boolean

Definition Classes
AnyRef → Any
val allowOverwriteAllPartitionsWithoutPartitionValues: Seq[DataObjectId]

Configure a list of exceptions for partitioned DataObject id's, which are allowed to overwrite the all partitions of a table if no partition values are set.
Configure a list of exceptions for partitioned DataObject id's, which are allowed to overwrite the all partitions of a table if no partition values are set. This is used to override/avoid a protective error when using SDLSaveMode.OverwriteOptimized|OverwritePreserveDirectories. Define it as a list of DataObject id's.
final def asInstanceOf[T0]: T0

Definition Classes
Any
def clone(): AnyRef

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( ... )
def createSparkSession(appName: String, master: Option[String], deployMode: Option[String] = None): SparkSession

Create a spark session using settings from this global config
val enableHive: Boolean

enable hive for spark session
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def finalize(): Unit

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
val kryoClasses: Option[Seq[String]]

classes to register for spark kryo serialization
lazy val logger: Logger

Attributes
protected
Definition Classes
SmartDataLakeLogger
val memoryLogTimer: Option[MemoryLogTimerConfig]

enable periodic memory usage logging, see detailed configuration MemoryLogTimerConfig
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def notify(): Unit

Definition Classes
AnyRef
final def notifyAll(): Unit

Definition Classes
AnyRef
val pythonUDFs: Option[Map[String, PythonUDFCreatorConfig]]

Define UDFs in python to be registered in spark session.
Define UDFs in python to be registered in spark session. The registered UDFs are available in Spark SQL transformations but not for expression evaluation.
val runtimeDataNumberOfExecutionsToKeep: Int

Number of Executions to keep runtime data for in streaming mode (default = 10).
Number of Executions to keep runtime data for in streaming mode (default = 10). Must be bigger than 1.
val secretProviders: Option[Map[String, SecretProviderConfig]]

Define SecretProvider's to be registered.
val shutdownHookLogger: Boolean

enable shutdown hook logger to trace shutdown cause
val sparkOptions: Option[Map[String, String]]

spark options
val sparkUDFs: Option[Map[String, SparkUDFCreatorConfig]]

Define UDFs to be registered in spark session.
Define UDFs to be registered in spark session. The registered UDFs are available in Spark SQL transformations and expression evaluation, e.g. configuration of ExecutionModes.
val stateListeners: Seq[StateListenerConfig]

Define state listeners to be registered for receiving events of the execution of SmartDataLake job
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
val synchronousStreamingTriggerIntervalSec: Int

Trigger interval for synchronous actions in streaming mode in seconds (default = 60 seconds) The synchronous actions of the DAG will be executed with this interval if possile.
Trigger interval for synchronous actions in streaming mode in seconds (default = 60 seconds) The synchronous actions of the DAG will be executed with this interval if possile. Note that for asynchronous actions there are separate settings, e.g. SparkStreamingMode.triggerInterval.
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )

Related Docs: object GlobalConfig | package app

Instance Constructors

Value Members

final def !=(arg0: Any): Boolean

final def ##(): Int

final def ==(arg0: Any): Boolean

val allowOverwriteAllPartitionsWithoutPartitionValues: Seq[DataObjectId]

final def asInstanceOf[T0]: T0

def clone(): AnyRef

def createSparkSession(appName: String, master: Option[String], deployMode: Option[String] = None): SparkSession

val enableHive: Boolean

final def eq(arg0: AnyRef): Boolean

def finalize(): Unit

final def getClass(): Class[_]

final def isInstanceOf[T0]: Boolean

val kryoClasses: Option[Seq[String]]

lazy val logger: Logger

val memoryLogTimer: Option[MemoryLogTimerConfig]

final def ne(arg0: AnyRef): Boolean

final def notify(): Unit

final def notifyAll(): Unit

val pythonUDFs: Option[Map[String, PythonUDFCreatorConfig]]

val runtimeDataNumberOfExecutionsToKeep: Int

val secretProviders: Option[Map[String, SecretProviderConfig]]

val shutdownHookLogger: Boolean

val sparkOptions: Option[Map[String, String]]

val sparkUDFs: Option[Map[String, SparkUDFCreatorConfig]]

val stateListeners: Seq[StateListenerConfig]

final def synchronized[T0](arg0: ⇒ T0): T0

val synchronousStreamingTriggerIntervalSec: Int

final def wait(): Unit

final def wait(arg0: Long, arg1: Int): Unit

final def wait(arg0: Long): Unit

Inherited from Serializable

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from SmartDataLakeLogger

Inherited from AnyRef

Inherited from Any

Ungrouped