com.holdenkarau.spark.testing

JavaStreamingSuiteBase

Related Doc: package testing

class JavaStreamingSuiteBase extends JavaSuiteBase with StreamingSuiteCommon

This is the base trait for Spark Streaming testsuite. This provides basic functionality to run user-defined set of input on user-defined stream operations, and verify the output. This implementation is designed to work with JUnit for java users.

Note: this always uses the manual clock

Linear Supertypes
StreamingSuiteCommon, Logging, JavaSuiteBase, SharedJavaSparkContext, SparkContextProvider, AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. JavaStreamingSuiteBase
  2. StreamingSuiteCommon
  3. Logging
  4. JavaSuiteBase
  5. SharedJavaSparkContext
  6. SparkContextProvider
  7. AnyRef
  8. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Instance Constructors

  1. new JavaStreamingSuiteBase()

Value Members

  1. final def !=(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any

  2. final def ##(): Int

    Definition Classes
    AnyRef → Any

  3. final def ==(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any

  4. def actuallyWait: Boolean

    Definition Classes
    StreamingSuiteCommon

  5. final def asInstanceOf[T0]: T0

    Definition Classes
    Any

  6. def batchDuration: Duration

    Definition Classes
    StreamingSuiteCommon

  7. lazy val checkpointDir: String

    Definition Classes
    StreamingSuiteCommon

  8. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  9. def compareArrays[U](i1: Array[U], i2: Array[U]): Unit

    Utility wrapper around assertArrayEquals that resolves the types

    Utility wrapper around assertArrayEquals that resolves the types

    Definition Classes
    JavaSuiteBase

  10. def conf: SparkConf

    Definition Classes
    JavaStreamingSuiteBase → StreamingSuiteCommon → SharedJavaSparkContext → SparkContextProvider

  11. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  12. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any

  13. val eventuallyTimeout: Timeout

    Definition Classes
    StreamingSuiteCommon

  14. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  15. def framework: String

    Definition Classes
    StreamingSuiteCommon

  16. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any

  17. def hashCode(): Int

    Definition Classes
    AnyRef → Any

  18. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any

  19. def isTraceEnabled(): Boolean

    Attributes
    protected
    Definition Classes
    Logging

  20. def jsc(): JavaSparkContext

    Definition Classes
    SharedJavaSparkContext

  21. def log: Logger

    Attributes
    protected
    Definition Classes
    Logging

  22. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging

  23. def logDebug(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging

  24. def logError(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging

  25. def logError(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging

  26. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging

  27. def logInfo(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging

  28. def logName: String

    Attributes
    protected
    Definition Classes
    Logging

  29. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging

  30. def logTrace(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging

  31. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging

  32. def logWarning(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging

  33. def master: String

    Definition Classes
    StreamingSuiteCommon

  34. def maxWaitTimeMillis: Int

    Definition Classes
    StreamingSuiteCommon

  35. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  36. final def notify(): Unit

    Definition Classes
    AnyRef

  37. final def notifyAll(): Unit

    Definition Classes
    AnyRef

  38. def numInputPartitions: Int

    Definition Classes
    StreamingSuiteCommon

  39. def runBefore(): Unit

    Definition Classes
    SharedJavaSparkContext

  40. def sc(): SparkContext

    Definition Classes
    SharedJavaSparkContext → SparkContextProvider

  41. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef

  42. def testOperation[U, V, W](input1: List[List[U]], input2: List[List[V]], operation: Function2[JavaDStream[U], JavaDStream[V], JavaDStream[W]], expectedOutput: List[List[W]], ordered: Boolean): Unit

    Test binary DStream operation with two lists of inputs, with number of batches to run same as the number of input values.

    Test binary DStream operation with two lists of inputs, with number of batches to run same as the number of input values. The size of the two input lists Should be the same. You can simulate the input batch as a List of values or as null to simulate empty batch.

    input1

    First sequence of input collections

    input2

    Second sequence of input collections

    operation

    Binary DStream operation to be applied to the 2 inputs

    expectedOutput

    Sequence of expected output collections

    ordered

    Compare output values with expected output values within the same output batch ordered or unOrdered. Comparing doubles may not work well in case of unordered.

  43. def testOperation[U, V, W](input1: List[List[U]], input2: List[List[V]], operation: Function2[JavaDStream[U], JavaDStream[V], JavaDStream[W]], expectedOutput: List[List[W]]): Unit

    Test binary DStream operation with two lists of inputs, with number of batches to run same as the number of input values.

    Test binary DStream operation with two lists of inputs, with number of batches to run same as the number of input values. The size of the two input lists Should be the same. You can simulate the input batch as a List of values or as null to simulate empty batch.

    input1

    First sequence of input collections

    input2

    Second sequence of input collections

    operation

    Binary DStream operation to be applied to the 2 inputs

    expectedOutput

    Sequence of expected output collections

  44. def testOperation[U, V](input: List[List[U]], operation: Function[JavaDStream[U], JavaDStream[V]], expectedOutput: List[List[V]], ordered: Boolean): Unit

    Test unary DStream operation with a list of inputs, with number of batches to run same as the number of input values.

    Test unary DStream operation with a list of inputs, with number of batches to run same as the number of input values. You can simulate the input batch as a List of values or as null to simulate empty batch.

    input

    Sequence of input collections

    operation

    Binary DStream operation to be applied to the 2 inputs

    expectedOutput

    Sequence of expected output collections

    ordered

    Compare output values with expected output values within the same output batch ordered or unordered. Comparing doubles may not work well in case of unordered.

  45. def testOperation[U, V](input: List[List[U]], operation: Function[JavaDStream[U], JavaDStream[V]], expectedOutput: List[List[V]]): Unit

    Test unary DStream operation with a list of inputs, with number of batches to run same as the number of input values.

    Test unary DStream operation with a list of inputs, with number of batches to run same as the number of input values. You can simulate the input batch as a List of values or as null to simulate empty batch.

    input

    Sequence of input collections

    operation

    Binary DStream operation to be applied to the 2 inputs

    expectedOutput

    Sequence of expected output collections

  46. def toString(): String

    Definition Classes
    AnyRef → Any

  47. def useManualClock: Boolean

    Definition Classes
    StreamingSuiteCommon

  48. def verifyOutput[V](output: Seq[Seq[V]], expectedOutput: Seq[Seq[V]], ordered: Boolean)(implicit arg0: ClassTag[V]): Unit

    Verify whether the output values after running a DStream operation is same as the expected output values, by comparing the output collections either as lists (order matters) or sets (order does not matter)

  49. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  50. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  51. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from StreamingSuiteCommon

Inherited from Logging

Inherited from JavaSuiteBase

Inherited from SharedJavaSparkContext

Inherited from SparkContextProvider

Inherited from AnyRef

Inherited from Any

Ungrouped