testing
Trait with ScalaTest Matchers for SCollections specific to Bigtable output.
Trait for unit testing pipelines.
Trait for unit testing pipelines.
A simple test might look like this:
class SimplePipelineTest extends PipelineSpec { "A simple pipeline" should "sum integers" in { runWithContext { sc => sc.parallelize(Seq(1, 2, 3)).sum should containSingleValue (6) } } }
Trait with utility methods for unit testing pipelines.
Trait that enforces JobTest.Builder.run is called.
Trait with ScalaTest Matchers for SCollections.
Trait for unit testing ScioIO.
Enhanced version of ScioContext with streaming methods.
Set up a Scio job for end-to-end unit testing.
Set up a Scio job for end-to-end unit testing. To be used in a PipelineSpec. For example:
import com.spotify.scio.testing._ class WordCountTest extends PipelineSpec { // Mock input data, mock distributed cache and expected result val inData = Seq("a b c d e", "a b a b") val distCache = Map(1 -> "Jan", 2 -> "Feb", 3 -> "Mar") val expected = Seq("a: 3", "b: 3", "c: 1", "d: 1", "e: 1") // Test specification "WordCount" should "work" in { JobTest("com.spotify.scio.examples.WordCount") // Or the type safe version // JobTest[com.spotify.scio.examples.WordCount.type] // Command line arguments .args("--input=in.txt", "--output=out.txt") // Mock input data .input(TextIO("in.txt"), inData) // Mock distributed cache .distCache(DistCacheIO("gs://dataflow-samples/samples/misc/months.txt"), distCache) // Verify output .output(TextIO("out.txt")) { actual => actual should containInAnyOrder (expected) } // Run job test .run() } }
Create a TestStream.Builder instance.