package app
- Alphabetic
- Public
- All
Type Members
-
class
DefaultSmartDataLakeBuilder extends SmartDataLakeBuilder
Default Smart Data Lake Command Line Application.
Default Smart Data Lake Command Line Application.
Implementation Note: This must be a class and not an object in order to be found by reflection in DatabricksSmartDataLakeBuilder
-
case class
GlobalConfig(kryoClasses: Option[Seq[String]] = None, sparkOptions: Option[Map[String, String]] = None, enableHive: Boolean = true, memoryLogTimer: Option[MemoryLogTimerConfig] = None, shutdownHookLogger: Boolean = false, stateListeners: Seq[StateListenerConfig] = Seq()) extends SmartDataLakeLogger with Product with Serializable
Global configuration options
Global configuration options
- kryoClasses
classes to register for spark kryo serialization
- sparkOptions
spark options
- enableHive
enable hive for spark session
- memoryLogTimer
enable periodic memory usage logging, see detailled configuration MemoryLogTimerConfig
- shutdownHookLogger
enable shutdown hook logger to trace shutdown cause
-
case class
MemoryLogTimerConfig(intervalSec: Int, logLinuxMem: Boolean = true, logLinuxCGroupMem: Boolean = false, logBuffers: Boolean = false) extends Product with Serializable
Configuration for periodic memory usage logging
Configuration for periodic memory usage logging
- intervalSec
interval in seconds between memory usage logs
- logLinuxMem
enable logging linux memory
- logLinuxCGroupMem
enable logging details about linux cgroup memory
- logBuffers
enable logging details about different jvm buffers
-
abstract
class
SmartDataLakeBuilder extends SmartDataLakeLogger
Abstract Smart Data Lake Command Line Application.
-
case class
SmartDataLakeBuilderConfig(feedSel: String = null, applicationName: Option[String] = None, configuration: Option[String] = None, master: Option[String] = None, deployMode: Option[String] = None, username: Option[String] = None, kerberosDomain: Option[String] = None, keytabPath: Option[File] = None, partitionValues: Option[Seq[PartitionValues]] = None, multiPartitionValues: Option[Seq[PartitionValues]] = None, parallelism: Int = 1, statePath: Option[String] = None, overrideJars: Option[Seq[String]] = None, test: Option[TestMode.Value] = None) extends Product with Serializable
This case class represents a default configuration for the App.
This case class represents a default configuration for the App. It is populated by parsing command-line arguments. It also specifies default values.
- feedSel
Regex pattern to select the feed to execute.
- applicationName
Application name.
- configuration
A configuration file or a directory containing configuration files.
- master
The Spark master URL passed to SparkContext when in local mode.
- deployMode
The Spark deploy mode passed to SparkContext when in local mode.
- username
Kerberos user name (
username
@kerberosDomain
) for local mode.- kerberosDomain
Kerberos domain (
username
@kerberosDomain
) for local mode.- keytabPath
Path to Kerberos keytab file for local mode.
- test
Run in test mode:
- "config": validate configuration
- "dry-run": execute "prepare" and "init" phase to check environment
-
trait
StateListener extends AnyRef
Interface to notify interested parties about action results & metric
-
case class
StateListenerConfig(className: String, options: Option[Map[String, String]] = None) extends Product with Serializable
Configuration to notify interested parties about action results & metric
Configuration to notify interested parties about action results & metric
- className
fully qualified class name of class implementing StateListener interface. The class needs a constructor with one parameter
options: Map[String,String]
.- options
Options are passed to StateListener constructor.
Value Members
-
object
DatabricksSmartDataLakeBuilder extends SmartDataLakeBuilder
Databricks Smart Data Lake Command Line Application.
Databricks Smart Data Lake Command Line Application.
As there is an old version of config-*.jar deployed on Databricks, this special App uses a ChildFirstClassLoader to override it in the classpath.
- object DefaultSmartDataLakeBuilder
- object GlobalConfig extends Serializable
-
object
LocalSmartDataLakeBuilder extends SmartDataLakeBuilder
Smart Data Lake Builder application for local mode.
Smart Data Lake Builder application for local mode.
Sets master to local[*] and deployMode to client by default.
- object TestMode extends Enumeration