testing
Trait with ScalaTest Matchers for SCollections specific to BigTable output.
Trait for unit testing pipelines.
Trait for unit testing pipelines.
A simple test might look like this:
class SimplePipelineTest extends PipelineSpec { "A simple pipeline" should "sum integers" in { runWithContext { sc => sc.parallelize(Seq(1, 2, 3)).sum should containSingleValue (6) } } }
Trait with utility methods for unit testing pipelines.
Trait that enforces JobTest.Builder.run is called.
Trait with ScalaTest Matchers for SCollections.
Enhanced version of ScioContext with streaming methods.
Set up a Scio job for end-to-end unit testing.
Set up a Scio job for end-to-end unit testing. To be used in a PipelineSpec. For example:
import com.spotify.scio.testing._ class WordCountTest extends PipelineSpec { // Mock input data, mock distributed cache and expected result val inData = Seq("a b c d e", "a b a b") val distCache = Map(1 -> "Jan", 2 -> "Feb", 3 -> "Mar") val expected = Seq("a: 3", "b: 3", "c: 1", "d: 1", "e: 1") // Test specification "WordCount" should "work" in { JobTest("com.spotify.scio.examples.WordCount") // Or the type safe version // JobTest[com.spotify.scio.examples.WordCount.type] // Command line arguments .args("--input=in.txt", "--output=out.txt") // Mock input data .input(TextIO("in.txt"), inData) // Mock distributed cache .distCache(DistCacheIO("gs://dataflow-samples/samples/misc/months.txt"), distCache) // Verify output .output(TextIO("out.txt")) { actual => actual should containInAnyOrder (expected) } // Run job test .run() } }
Create a TestStream.Builder instance.