Utility wrapper around assertArrayEquals that resolves the types
Utility wrapper around assertArrayEquals that resolves the types
Setup work to be called when creating a new SparkContext.
Setup work to be called when creating a new SparkContext. Default implementation currently sets a checkpoint directory.
This _should_ be called by the context provider automatically.
Test binary DStream operation with two lists of inputs, with number of batches to run same as the number of input values.
Test binary DStream operation with two lists of inputs, with number of batches to run same as the number of input values. The size of the two input lists should be equal.
Each input micro-batch is a list of values or as null to simulate empty batch.
First sequence of input collections
Second sequence of input collections
Binary DStream operation to be applied to the 2 inputs
Sequence of expected output collections
Compare output values with expected output values within the same output batch ordered or unOrdered. Comparing doubles may not work well in case of unordered.
Test binary DStream operation with two lists of inputs, with number of batches to run same as the number of input values.
Test binary DStream operation with two lists of inputs, with number of batches to run same as the number of input values. The size of the two input lists should be equal.
Each input micro-batch is a list of values or as null to simulate empty batch.
First sequence of input collections
Second sequence of input collections
Binary DStream operation to be applied to the 2 inputs
Sequence of expected output collections
Test unary DStream operation with a list of inputs, with number of batches to run same as the number of input values.
Test unary DStream operation with a list of inputs, with number of batches to run same as the number of input values.
Each input micro-batch is a list of values or as null to simulate empty batch.
Sequence of input collections
Binary DStream operation to be applied to the 2 inputs
Sequence of expected output collections
Compare output values with expected output values within the same output batch ordered or unordered. Comparing doubles may not work well in case of unordered.
Test unary DStream operation with a list of inputs, with number of batches to run same as the number of input values.
Test unary DStream operation with a list of inputs, with number of batches to run same as the number of input values.
Each input micro-batch is a list of values or as null to simulate empty batch.
Sequence of input collections
Binary DStream operation to be applied to the 2 inputs
Sequence of expected output collections
Verify whether the output values after running a DStream operation is same as the expected output values, by comparing the output collections either as lists (order matters) or sets (order does not matter)
This is the base trait for Spark Streaming testsuite. This provides basic functionality to run user-defined set of input on user-defined stream operations, and verify the output matches as expected.
This implementation is designed to work with JUnit for java users.
Note: this always uses the manual clock to control Spark Streaming's batches.