SparkInterceptorActions

Value Members

1. final def !=(arg0: Any): Boolean

   

   Definition Classes

   AnyRef → Any

2. final def ##(): Int

   

   Definition Classes

   AnyRef → Any

3. final def ==(arg0: Any): Boolean

   

   Definition Classes

   AnyRef → Any

4. final def asInstanceOf[T0]: T0

   

   Definition Classes

   Any

5. def cacheAsParquet(labels: String*): SparkDataFlow

   

   Creates a persistent snapshot into the staging folder of the spark data flow and substitutes the dataset behind the label with the one opened from the stored version.

   Creates a persistent snapshot into the staging folder of the spark data flow and substitutes the dataset behind the label with the one opened from the stored version.

   It will not trigger for labels whose datasets are empty.

   labels

   - list of labels to cache

6. def cacheAsPartitionedParquet(partitions: Seq[String], repartition: Boolean = true)(labels: String*): SparkDataFlow

   

   Creates a persistent snapshot into the staging folder of the spark data flow and substitutes the dataset behind the label with the one opened from the stored version.

   Creates a persistent snapshot into the staging folder of the spark data flow and substitutes the dataset behind the label with the one opened from the stored version.

   It will not trigger for labels whose datasets are empty.

   labels

   - list of labels to snapshot

7. def clone(): AnyRef

   

   Attributes

   protected[java.lang]

   Definition Classes

   AnyRef

   Annotations

   @throws( ... )

8. final def eq(arg0: AnyRef): Boolean

   

   Definition Classes

   AnyRef

9. def equals(arg0: Any): Boolean

   

   Definition Classes

   AnyRef → Any

10. def finalize(): Unit

    

    Attributes

    protected[java.lang]

    Definition Classes

    AnyRef

    Annotations

    @throws( classOf[java.lang.Throwable] )

11. final def getClass(): Class[_]

    

    Definition Classes

    AnyRef → Any

12. def hashCode(): Int

    

    Definition Classes

    AnyRef → Any

13. def inPlaceTransform(label: String)(post: (Dataset[_]) ⇒ Dataset[_]): SparkDataFlow

    

    Applies a transformation to the label's data set and replaces it.

    Applies a transformation to the label's data set and replaces it.

    Multiple intercept action can be chained. Like post -> post -> snapshot.

14. final def isInstanceOf[T0]: Boolean

    

    Definition Classes

    Any

15. def isTraceEnabled(): Boolean

    

    Attributes

    protected

    Definition Classes

    Logging

16. def logAndReturn[A](a: A, msg: String, level: Level): A

    

    Takes a value of type A and a msg to log, returning a and logging the message at the desired level

    Takes a value of type A and a msg to log, returning a and logging the message at the desired level

    returns

    a

    Definition Classes

    Logging

17. def logAndReturn[A](a: A, message: (A) ⇒ String, level: Level): A

    

    Takes a value of type A and a function message from A to String, logs the value of invoking message(a) at the level described by the level parameter

    Takes a value of type A and a function message from A to String, logs the value of invoking message(a) at the level described by the level parameter

    returns

    a

    Definition Classes

    Logging

    Example:

    1. logAndReturn(1, (num: Int) => s"number: $num", Info)
       // In the log we would see a log corresponding to "number 1"

18. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    

    Attributes

    protected

    Definition Classes

    Logging

19. def logDebug(msg: ⇒ String): Unit

    

    Attributes

    protected

    Definition Classes

    Logging

20. def logError(msg: ⇒ String, throwable: Throwable): Unit

    

    Attributes

    protected

    Definition Classes

    Logging

21. def logError(msg: ⇒ String): Unit

    

    Attributes

    protected

    Definition Classes

    Logging

22. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    

    Attributes

    protected

    Definition Classes

    Logging

23. def logInfo(msg: ⇒ String): Unit

    

    Attributes

    protected

    Definition Classes

    Logging

24. def logName: String

    

    Attributes

    protected

    Definition Classes

    Logging

25. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    

    Attributes

    protected

    Definition Classes

    Logging

26. def logTrace(msg: ⇒ String): Unit

    

    Attributes

    protected

    Definition Classes

    Logging

27. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    

    Attributes

    protected

    Definition Classes

    Logging

28. def logWarning(msg: ⇒ String): Unit

    

    Attributes

    protected

    Definition Classes

    Logging

29. final def ne(arg0: AnyRef): Boolean

    

    Definition Classes

    AnyRef

30. final def notify(): Unit

    

    Definition Classes

    AnyRef

31. final def notifyAll(): Unit

    

    Definition Classes

    AnyRef

32. def sparkCache(labels: String*): SparkDataFlow

    

    Cache multiple labels using using Spark's in-built caching mechanism

    Cache multiple labels using using Spark's in-built caching mechanism

    labels

    - list of labels to cache

33. def sparkCacheSingle(label: String, partitions: Option[Int] = None, storageLevel: StorageLevel = StorageLevel.MEMORY_AND_DISK): SparkDataFlow

    

    Cache a single label using Spark's in-built caching mechanism

    Cache a single label using Spark's in-built caching mechanism

    label

    the label to cache

    partitions

    optionally, the number of partitions to partition the dataset by before caching (will invoke a .repartition call)

    storageLevel

    the StorageLevel to use

34. final def synchronized[T0](arg0: ⇒ T0): T0

    

    Definition Classes

    AnyRef

35. def toString(): String

    

    Definition Classes

    AnyRef → Any

36. final def wait(): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

37. final def wait(arg0: Long, arg1: Int): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

38. final def wait(arg0: Long): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )