Test binary DStream operation with two lists of inputs, with number of batches to run same as the number of input values.
Test binary DStream operation with two lists of inputs, with number of batches to run same as the number of input values. The size of the two input lists Should be the same. You can simulate the input batch as a List of values or as null to simulate empty batch.
First sequence of input collections
Second sequence of input collections
Binary DStream operation to be applied to the 2 inputs
Sequence of expected output collections
Compare output values with expected output values within the same output batch ordered or unOrdered. Comparing doubles may not work well in case of unordered.
Test unary DStream operation with a list of inputs, with number of batches to run same as the number of input values.
Test unary DStream operation with a list of inputs, with number of batches to run same as the number of input values. You can simulate the input batch as a List of values or as null to simulate empty batch.
Sequence of input collections
Binary DStream operation to be applied to the 2 inputs
Sequence of expected output collections
Compare output values with expected output values within the same output batch ordered or unordered. Comparing doubles may not work well in case of unordered.
Test binary DStream and RDD operation with two lists of inputs, with number of batches to run same as the number of input values corresponding to the DStream.
Test binary DStream and RDD operation with two lists of inputs, with number of batches to run same as the number of input values corresponding to the DStream. You can simulate the input batch as a List of values or as null to simulate empty batch.
Sequence of input collections corresponding to the DStream
Sequence of input values corresponding to the RDD
Binary DStream and RDD operation to be applied to the 2 inputs
Sequence of expected output collections
Compare output values with expected output values within the same output batch ordered or unOrdered. Comparing doubles may not work well in case of unordered.
Verify whether the output values after running a DStream operation is same as the expected output values, by comparing the output collections either as lists (order matters) or sets (order does not matter)
Verify whether the output values after running a DStream operation is same as the expected output values, by comparing the output collections either as lists (order matters) or sets (order does not matter)
Compare output values with expected output values within the same output batch ordered or unordered. Comparing doubles may not work well in case of unordered.
Please use the afterAll(ConfigMap) method of trait BeforeAndAfterAllConfigMap instead.
Please use the beforeAll(ConfigMap) method of trait BeforeAndAfterAllConfigMap instead.
This is the base trait for Spark Streaming testsuites. This provides basic functionality to run user-defined set of input on user-defined stream operations, and verify the output.