Immutable, strict, finite sequence of values that supports efficient index-based random access of elements, is memory efficient for all sizes, and avoids unnecessary copying.

Chunks can be created from a variety of collection types using methods on the Chunk companion (e.g., Chunk.array, Chunk.seq, Chunk.vector).

Chunks can be appended via the ++ method. The returned chunk is a composite of the input chunks -- that is, there's no copying of the source chunks. For example, Chunk(1, 2) ++ Chunk(3, 4) ++ Chunk(5, 6) returns a Chunk.Queue(Chunk(1, 2), Chunk(3, 4), Chunk(5, 6)). As a result, indexed based lookup of an appended chunk is amortized O(log2(number of underlying chunks)). In the worst case, where each constituent chunk has size 1, indexed lookup is O(log2(size)). To restore O(1) lookup, call compact, which copies all the underlying chunk elements to a single array backed chunk. Note compact requires a ClassTag of the element type.

Alternatively, a collection of chunks can be directly copied to a new array backed chunk via Chunk.concat(chunks). Like compact, Chunk.concat requires a ClassTag for the element type.

Various subtypes of Chunk are exposed for efficiency reasons:

• Chunk.Singleton
• Chunk.ArraySlice
• Chunk.Queue

In particular, calling .toArraySlice on a chunk returns a Chunk.ArraySlice, which provides access to the underlying backing array, along with an offset and length, referring to a slice of that array.

Supports building a result of type Out from zero or more Chunk[A].

This is similar to the standard library collection builders but optimized for building a collection from a stream.

The companion object provides implicit conversions (methods starting with supports), which adapts various collections to the Collector trait.

Provides compilation of a Stream[F, O] to a G[*].

In the most common case, F = G = IO or another "fully featured" effect type. However, there are other common instantiations like F = Pure, G = Id, which allows compiling a Stream[Pure, A] in to pure values.

For the common case where F = G, the target implicit constructor provides an instance of Compiler[F, F] -- target requires a Compiler.Target[F] instance. The Compiler.Target[F] is a super charged MonadErrorThrow[F], providing additional capabilities needed for stream compilation. Compiler.Target[F] instances are given for all F[_] which have:

• Concurrent[F] instances
• both MonadCancelThrow[F] and Sync[F] intances
• only Sync[F] instances Support for stream interruption requires compilation to an effect which has a Concurrent instance.

Indicates that a stream evaluates no effects but unlike Pure, may raise errors.

Uninhabited.

A Stream[Fallible,O] can be safely converted to a Stream[F,O] for all F via s.lift[F], provided an ApplicativeError[F, Throwable] is available.

A stream transformation represented as a function from stream to stream.

Pipes are typically applied with the through operation on Stream.

A stream transformation that combines two streams in to a single stream, represented as a function from two streams to a single stream.

Pipe2s are typically applied with the through2 operation on Stream.

A purely functional data structure that describes a process. This process may evaluate actions in an effect type F, emit any number of output values of type O (or None), and may a) terminate with a single result of type R; or b) terminate abnormally by raising (inside the effect F) an exception, or c) terminate because it was cancelled by another process, or d) not terminate.

Like types from other effect libraries, pulls are pure and immutable values. They preserve referential transparency.

Chunking

The output values of a pull are emitted not one by one, but in chunks. A Chunk is an immutable sequence with constant-time indexed lookup. For example, a pull p: Pull[F, Byte, R] internally operates and emits Chunk[Byte] values, which can wrap unboxed byte arrays -- avoiding boxing/unboxing costs. The Pull API provides mechanisms for working at both the chunk level and the individual element level. Generally, working at the chunk level will result in better performance but at the cost of more complex implementations

A pull only emits non-empty chunks.

However, chunks are not merely an operational matter of efficiency. Each pull is emitted from a chunk atomically, which is to say, any errors or interruptions in a pull can only happen between chunks, not within a chunk. For instance, if creating a new chunk of values fails (raises an uncaught exception) while creating an intermediate value, then it fails to create the entire chunk and previous values are discarded.

Evaluation

Like other functional effect types (e.g. cats.effect.IO), a pull describes a _process_ or _computation_. It is not a running process nor a handle for the result of a spawned, running process, like scala.concurrent.Future.

A pull can be converted to a stream and then compiled to an effectful value. For a Pull[F, O, Unit], the result of compilation is a combination, via the monad instance of F, of all the actions in the effect F present in the pull. The result of that F action is the result of combining the outputs emitted by the pull, in the order it emits them, using a _fold_ function. Depending on that function, outputs may be collected into a list (or vector or array or ...), combined together into a single value, or just discarded altogether (by _draining_ the pull).

Compilation is pull-based, rather than push-based (hence the name of the datatype). It is the compilation process itself, that determines when the evaluation of each single effect can proceed or is held back. Effects and outputs later in the pull are not performed or emitted, _unless and until_ the compilation calls for them.

Resource scoping

The effects in a Pull may operate on resources, which must be retained during the execution of the pull, may be shared by several pulls, and must be properly finalised when no longer needed, regardless of whether the pull completed successfully or not. A pull tracks its resources using scopes, which register how many pulls are actively using each resource, and finalises resources when no longer used.

Some operations of the Pull API can be used to introduce new resource scopes, or resource boundaries.

Functional typeclasses

The Pull data structure is a "free" implementation of Monad and has an instance for cats.effect.kernel.Sync.

For any types F[_] and O, a Pull[F, O, *] holds the following laws:

• pure >=> f == f
• f >=> pure == f
• (f >=> g) >=> h == f >=> (g >=> h) where f >=> g is defined as a => a flatMap f flatMap g
• handleErrorWith(raiseError(e))(f) == f(e)

Indicates that a stream evaluates no effects.

Because Stream is covariant, A Stream[Pure,O] is also an instance of Stream[F,O] for all F.

This should not be confused with cats.Id, which provides an alternative encoding of pure streams, namely Stream[Id, O]. The difference is that Stream[Id, O] achieves purity by using an effect type whose evaluation is a no-op, whereas Stream[Pure, O] achieves purity by using an effect type that has no instances and therefore cannot be instantiated in the first place.

Witnesses that F supports raising throwables.

An instance of RaiseThrowable is available for any F which has an ApplicativeError[F, Throwable] instance. Alternatively, an instance is available for the uninhabited type Fallible.

A stateful transformation of the elements of a stream.

A scan is primarily represented as a function (S, I) => (S, Chunk[O]). Scans also have an initial state value of type S and the ability to emit elements upon completion via a function S => Chunk[O].

A scan is built up incrementally via various combinators and then converted to a pipe via .toPipe. For example, s.through(Scan.lift(identity).toPipe) == s.

A scan is much less powerful than a pull. Scans cannot evaluate effects or terminate early. These limitations allow combinators that are not possible on pulls though. For example, the first method converts a Scan[S, I, O] to a Scan[S, (I, A), (O, A)]. Critically, this method relies on the ability to feed a single I to the original scan and collect the resulting O values, pairing each O with the A that was paired with I.

A stream producing output of type O and which may evaluate F effects.

- Purely functional a value of type Stream[F, O] _describes_ an effectful computation. A function that returns a Stream[F, O] builds a _description_ of an effectful computation, but does not perform them. The methods of the Stream class derive new descriptions from others. This is similar to how effect types like cats.effect.IO and monix.Task build descriptions of computations.

- Pull: to evaluate a stream, a consumer pulls its values from it, by repeatedly performing one pull step at a time. Each step is a F-effectful computation that may yield some O values (or none), and a stream from which to continue pulling. The consumer controls the evaluation of the stream, which effectful operations are performed, and when.

- Non-Strict: stream evaluation only pulls from the stream a prefix large enough to compute its results. Thus, although a stream may yield an unbounded number of values or, after successfully yielding several values, either raise an error or hang up and never yield any value, the consumer need not reach those points of failure. For the same reason, in general, no effect in F is evaluated unless and until the consumer needs it.

- Abstract: a stream needs not be a plain finite list of fixed effectful computations in F. It can also represent an input or output connection through which data incrementally arrives. It can represent an effectful computation, such as reading the system's time, that can be re-evaluated as often as the consumer of the stream requires.

Special properties for streams

There are some special properties or cases of streams:

• A stream is finite if we can reach the end after a limited number of pull steps, which may yield a finite number of values. It is empty if it terminates and yields no values.
• A singleton stream is a stream that ends after yielding one single value.
• A pure stream is one in which the F is Pure, which indicates that it evaluates no effects.
• A never stream is a stream that never terminates and never yields any value.

Pure Streams and operations

We can sometimes think of streams, naively, as lists of O elements with F-effects. This is particularly true for pure streams, which are instances of Stream which use the Pure effect type. We can convert every pure and finite stream into a List[O] using the .toList method. Also, we can convert pure infinite streams into instances of the Stream[O] class from the Scala standard library.

A method of the Stream class is pure if it can be applied to pure streams. Such methods are identified in that their signature includes no type-class constraint (or implicit parameter) on the F method. Pure methods in Stream[F, O] can be projected naturally to methods in the List class, which means that applying the stream's method and converting the result to a list gets the same result as first converting the stream to a list, and then applying list methods.

Some methods that project directly to list are map, filter, takeWhile, etc. There are other methods, like exists or find, that in the List class they return a value or an Option, but their stream counterparts return an (either empty or singleton) stream. Other methods, like zipWithPrevious, have a more complicated but still pure translation to list methods.

Type-Class instances and laws of the Stream Operations

Laws (using infix syntax):

append forms a monoid in conjunction with empty:

• empty append s == s and s append empty == s.
• (s1 append s2) append s3 == s1 append (s2 append s3)

And cons is consistent with using ++ to prepend a single chunk:

• s.cons(c) == Stream.chunk(c) ++ s

Stream.raiseError propagates until being caught by handleErrorWith:

• Stream.raiseError(e) handleErrorWith h == h(e)
• Stream.raiseError(e) ++ s == Stream.raiseError(e)
• Stream.raiseError(e) flatMap f == Stream.raiseError(e)

Stream forms a monad with emit and flatMap:

• Stream.emit >=> f == f (left identity)
• f >=> Stream.emit === f (right identity - note weaker equality notion here)
• (f >=> g) >=> h == f >=> (g >=> h) (associativity) where Stream.emit(a) is defined as chunk(Chunk.singleton(a)) and f >=> g is defined as a => a flatMap f flatMap g

The monad is the list-style sequencing monad:

• (a ++ b) flatMap f == (a flatMap f) ++ (b flatMap f)
• Stream.empty flatMap f == Stream.empty

Technical notes

Note: since the chunk structure of the stream is observable, and s flatMap Stream.emit produces a stream of singleton chunks, the right identity law uses a weaker notion of equality, === which normalizes both sides with respect to chunk structure:

(s1 === s2) = normalize(s1) == normalize(s2) where == is full equality (a == b iff f(a) is identical to f(b) for all f)

normalize(s) can be defined as s.flatMap(Stream.emit), which just produces a singly-chunked stream from any input stream s.

For instance, for a stream s and a function f: A => B, - the result of s.map(f) is a Stream with the same _chunking_ as the s; whereas... - the result of s.flatMap(x => S.emit(f(x))) is a Stream structured as a sequence of singleton chunks. The latter is using the definition of map that is derived from the Monad instance.

This is not unlike equality for maps or sets, which is defined by which elements they contain, not by how these are spread between a tree's branches or a hashtable buckets. However, a Stream structure can be _observed_ through the chunks method, so two streams "_equal_" under that notion may give different results through this method.

Note: For efficiency Stream.map function operates on an entire chunk at a time and preserves chunk structure, which differs from the map derived from the monad (s map f == s flatMap (f andThen Stream.emit)) which would produce singleton chunk. In particular, if f throws errors, the chunked version will fail on the first chunk with an error, while the unchunked version will fail on the first element with an error. Exceptions in pure code like this are strongly discouraged.

Alias for Nothing which works better with type inference.