Class

com.coxautodata.waimak.dataflow.spark

SparkFlowContext

Related Doc: package spark

Permalink

case class SparkFlowContext(spark: SparkSession) extends FlowContext with Product with Serializable

Context required in a Spark data flow (SparkSession and FileSystem)

Created by Vicky Avison on 23/02/2018.

spark

the SparkSession

Linear Supertypes
Serializable, Serializable, Product, Equals, FlowContext, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. SparkFlowContext
  2. Serializable
  3. Serializable
  4. Product
  5. Equals
  6. FlowContext
  7. AnyRef
  8. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new SparkFlowContext(spark: SparkSession)

    Permalink

    spark

    the SparkSession

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  5. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  6. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  7. lazy val fileSystem: FileSystem

    Permalink
  8. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  9. def getBoolean(key: String, defaultValue: Boolean): Boolean

    Permalink

    Get a configuration value as a boolean, falling back to a default if not set

    Get a configuration value as a boolean, falling back to a default if not set

    Definition Classes
    FlowContext
  10. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
  11. def getDouble(key: String, defaultValue: Double): Double

    Permalink

    Get a configuration value as a double, falling back to a default if not set

    Get a configuration value as a double, falling back to a default if not set

    Definition Classes
    FlowContext
  12. def getInt(key: String, defaultValue: Int): Int

    Permalink

    Get a configuration value as an integer, falling back to a default if not set

    Get a configuration value as an integer, falling back to a default if not set

    Definition Classes
    FlowContext
  13. def getLong(key: String, defaultValue: Long): Long

    Permalink

    Get a configuration value as a long, falling back to a default if not set

    Get a configuration value as a long, falling back to a default if not set

    Definition Classes
    FlowContext
  14. def getOption(key: String): Option[String]

    Permalink

    Get a configuration value, returning a None if not set

    Get a configuration value, returning a None if not set

    Definition Classes
    SparkFlowContextFlowContext
  15. def getString(key: String, defaultValue: String): String

    Permalink

    Get a configuration value as a string, falling back to a default if not set

    Get a configuration value as a string, falling back to a default if not set

    Definition Classes
    FlowContext
  16. def getStringList(key: String, defaultValue: List[String]): List[String]

    Permalink

    Get a configuration value as a list of strings with "," separator, falling back to a default if not set

    Get a configuration value as a list of strings with "," separator, falling back to a default if not set

    Definition Classes
    FlowContext
  17. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  18. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  19. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
  20. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
  21. def reportActionFinished(action: DataFlowAction): Unit

    Permalink
    Definition Classes
    SparkFlowContextFlowContext
  22. def reportActionStarted(action: DataFlowAction): Unit

    Permalink
    Definition Classes
    SparkFlowContextFlowContext
  23. def setPoolIntoContext(poolName: String): Unit

    Permalink
    Definition Classes
    SparkFlowContextFlowContext
  24. val spark: SparkSession

    Permalink

    the SparkSession

  25. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  26. val uriUsed: String

    Permalink
  27. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  28. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  29. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from Serializable

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from FlowContext

Inherited from AnyRef

Inherited from Any

Ungrouped