scodec
package scodec
Combinator library for working with binary data.
The primary abstraction of this library is Codec, which provides the ability to encode/decode values to/from binary.
There are more general abstractions though, such as Encoder and Decoder. There's also GenCodec which extends
both Encoder
and Decoder
but allows the types to vary. Given these more general abstractions, a Codec[A]
can be
represented as a GenCodec[A, A]
.
The more general abstractions are important because they allow operations on codecs that would not otherwise be possible.
For example, given a Codec[A]
, mapping a function A => B
over the codec yields a GenCodec[A, B]
. Without the
more general abstractions, map
is impossible to define (e.g., how would codec.map(f).encode(b)
be implemented?).
Given a GenCodec[A, B]
, the encoding functionality can be ignored by treating it as a Decoder[B]
, or the encoding
type can be changed via contramap
. If after further transformations, the two types to GenCodec
are equal, we can
reconstitute a Codec
from the GenCodec
by calling fuse
.
See the codecs package object for pre-defined codecs for many common data types and combinators for building larger codecs out of smaller ones.
For the categorically minded, note the following:
Decoder
is a monadEncoder
is a contravariant functorGenCodec
is a profunctorCodec
is an invariant functor
- Source
- package.scala
- Alphabetic
- By Inheritance
- scodec
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Type Members
-
sealed abstract
class
Attempt[+A] extends Product with Serializable
Right biased
Either[Err, A]
.Right biased
Either[Err, A]
.An
Attempt
is either anAttempt.Successful
or anAttempt.Failure
. Attempts can be created by callingAttempt.successful
orAttempt.failure
, as well as converting from anOption
viafromOption
. -
trait
Codec[A] extends GenCodec[A, A]
Supports encoding a value of type
A
to aBitVector
and decoding aBitVector
to a value ofA
.Supports encoding a value of type
A
to aBitVector
and decoding aBitVector
to a value ofA
.Not every value of
A
can be encoded to a bit vector and similarly, not every bit vector can be decoded to a value of typeA
. Hence, both encode and decode return either an error or the result. Furthermore, decode returns the remaining bits in the bit vector that it did not use in decoding.There are various ways to create instances of
Codec
. The trait can be implemented directly or one of the constructor methods in the companion can be used (e.g.,apply
). Most of the methods onCodec
create return a new codec that has been transformed in some way. For example, the xmap method converts aCodec[A]
to aCodec[B]
given two functions,A => B
andB => A
.One of the simplest transformation methods is
def withContext(context: String): Codec[A]
, which pushes the specified context string in to any errors (i.e.,Err
s) returned from encode or decode.See the methods on this trait for additional transformation types.
See the codecs package object for pre-defined codecs for many common data types and combinators for building larger codecs out of smaller ones.
Tuple Codecs
The
~
operator supports combining aCodec[A]
and aCodec[B]
in to aCodec[(A, B)]
.For example:
val codec: Codec[Int ~ Int ~ Int] = uint8 ~ uint8 ~ uint8
Codecs generated with
~
result in left nested tuples. These left nested tuples can be pulled back apart by pattern matching with~
. For example:Codec.decode(uint8 ~ uint8 ~ uint8, bytes) map { case a ~ b ~ c => a + b + c }
Alternatively, a function of N arguments can be lifted to a function of left-nested tuples. For example:
val add3 = (_: Int) + (_: Int) + (_: Int) Codec.decode(uint8 ~ uint8 ~ uint8, bytes) map add3
Similarly, a left nested tuple can be created with the
~
operator. This is useful when creating the tuple structure to pass to encode. For example:(uint8 ~ uint8 ~ uint8).encode(1 ~ 2 ~ 3)
Tuple based codecs are of limited use compared to
HList
based codecs, which is discussed later.Note: this design is heavily based on Scala's parser combinator library and the syntax it provides.
flatZip
Sometimes when combining codecs, a latter codec depends on a formerly decoded value. The
flatZip
method is important in these types of situations -- it represents a dependency between the left hand side and right hand side. Its signature isdef flatZip[B](f: A => Codec[B]): Codec[(A, B)]
. This is similar toflatMap
except the return type isCodec[(A, B)]
instead ofDecoder[B]
.Consider a binary format of an 8-bit unsigned integer indicating the number of bytes following it. To implement this with
flatZip
, we could write:val x: Codec[(Int, ByteVector)] = uint8 flatZip { numBytes => bytes(numBytes) } val y: Codec[ByteVector] = x.xmap[ByteVector]({ case (_, bv) => bv }, bv => (bv.size, bv))
In this example,
x
is aCodec[(Int, ByteVector)]
but we do not need the size directly in the model because it is redundant with the size stored in theByteVector
. Hence, we remove theInt
byxmap
-ping overx
. The notion of removing redundant data from models comes up frequently. Note: there is a combinator that expresses this pattern more succinctly --variableSizeBytes(uint8, bytes)
.HList Codecs
HList
s are similar to tuples in that they represent the product of an arbitrary number of types. That is, the size of anHList
is known at compile time and the type of each element is also known at compile time. For more information onHList
s in general, see Shapeless.Codec
makes heavy use ofHList
s. The primary operation is extending aCodec[L]
for someL <: HList
to aCodec[A :: L]
. For example:val uint8: Codec[Int] = ... val string: Codec[String] = ... val codec: Codec[Int :: Int :: String] = uint8 :: uint8 :: string
The
::
method is sort of like cons-ing on to theHList
but it is doing so *inside* theCodec
type. The resulting codec encodes values by passing each component of theHList
to the corresponding codec and concatenating all of the results.There are various methods on this trait that only work on
Codec[L]
for someL <: HList
. Besides the aforementioned::
method, there are others like:::
,flatPrepend
,flatConcat
, etc. One particularly useful method isdropUnits
, which removes anyUnit
values from theHList
.Given a
Codec[X0 :: X1 :: ... Xn :: HNil]
and a case class with typesX0
toXn
in the same order, theHList
codec can be turned in to a case class codec via theas
method. For example:case class Point(x: Int, y: Int, z: Int) val threeInts: Codec[Int :: Int :: Int :: HNil] = uint8 :: uint8 :: uint8 val point: Codec[Point] = threeInts.as[Point]
flatPrepend
The
HList
analog toflatZip
isflatPrepend
. It has the signature:def flatPrepend[L <: HList](f: A => Codec[L]): Codec[A :: L]
It forms a codec of
A
consed on toL
when called on aCodec[A]
and passed a functionA => Codec[L]
. Note that the specified function must return anHList
based codec. Implementing our example from earlier usingflatPrepend
:val x: Codec[Int :: ByteVector :: HNil] = uint8 flatPrepend { numBytes => bytes(numBytes).hlist }
In this example,
bytes(numBytes)
returns aCodec[ByteVector]
so we called.hlist
on it to lift it in to aCodec[ByteVector :: HNil]
.There are similar methods for flat appending and flat concating.
Coproduct Codecs
Given some ordered list of types, potentially with duplicates, a value of the
HList
of those types has a value for *every* type in the list. In other words, anHList
represents having anX0
ANDX1
AND ... ANDXN
. ACoproduct
for the same list of types represents having a value for *one* of those types. In other words, aCoproduct
represents having anX0
ORX1
OR ... ORXN
. This is somewhat imprecise because a coproduct can tell us exactly whichXi
we have, even in the presence of duplicate types.A coproduct can also be thought of as an
Either
that has an unlimited number of choices instead of just 2 choices.Shapeless represents coproducts in a similar way as
HList
s. A coproduct type is built using the:+:
operator with a sentinal value ofCNil
. For example, anInt
orLong
orString
is represented as the coproduct type:Int :+: Long :+: String :+: CNil
For more information on coproducts in general, see Shapeless.
Like
HList
based codecs, scodec supportsCoproduct
based codecs by coopting syntax from Shapeless. Specifically, the:+:
operator is used:val builder = uint8 :+: int64 :+: utf8
Unlike
HList
based codecs, the result of:+:
is not a codec but rather a codecs.CoproductCodecBuilder. Having a list of types and a codec for each is not sufficient to build a coproduct codec. We also need to describe how each entry in the coproduct is differentiated from the other entries. There are a number of ways to do this and each way changes the binary format significantly. See the docs onCoproductCodecBuilder
for details.Derived Codecs
Codecs for case classes and sealed class hierarchies can often be automatically derived.
Consider this example:
import scodec.codecs.implicits._ case class Point(x: Int, y: Int, z: Int) Codec[Point].encode(Point(1, 2, 3))
In this example, no explicit codec was defined for
Point
yetCodec[Point]
successfully created one. It did this by "reflecting" over the structure ofPoint
and looking up a codec for each component type (note: no runtime reflection is performed - rather, this is implemented using macro-based compile time reflection). In this case, there are three components, each of typeInt
, so the compiler first looked for an implicitCodec[Int]
. It then combined eachCodec[Int]
using anHList
based codec and finally converted theHList
codec to aCodec[Point]
. It found the implicitCodec[Int]
instances due to the import ofscodec.codecs.implicits._
. Furthermore, if there was an error encoding or decoding a field, the field name (i.e., x, y, or z) is included as context on theErr
returned.This works similarly for sealed class hierarchies -- each subtype is internally represented as a member of a coproduct. There must be the following implicits in scope however:
Discriminated[A, D]
for some discriminator typeD
, which provides theCodec[D]
to use for encoding/decoding the discriminatorDiscriminator[A, X, D]
for each subtypeX
ofA
, which provides the discriminator value for typeX
Codec[X]
for each subtypeX
ofA
Full examples are available in the test directory of this project.
Implicit Codecs
If authoring combinators that require implicit codec arguments, use
shapeless.Lazy[Codec[A]]
instead ofCodec[A]
. This prevents the occurrence of diverging implicit expansion errors. -
type
CodecTransformation = ~>[Codec, Codec]
Universally quantified transformation of a
Codec
to aCodec
. -
final
case class
DecodeResult[+A](value: A, remainder: BitVector) extends Product with Serializable
Result of a decoding operation, which consists of the decoded value and the remaining bits that were not consumed by decoding.
-
trait
Decoder[+A] extends AnyRef
Supports decoding a value of type
A
from aBitVector
. -
trait
DecoderFunctions extends AnyRef
Provides functions for working with decoders.
-
trait
Encoder[-A] extends AnyRef
Supports encoding a value of type
A
to aBitVector
. -
trait
EncoderFunctions extends AnyRef
Provides functions for working with encoders.
-
implicit final
class
EnrichedCoproductDecoder[C <: Coproduct] extends AnyVal
Provides methods specific to decoders of Shapeless coproducts.
-
implicit final
class
EnrichedCoproductEncoder[C <: Coproduct] extends AnyVal
Provides methods specific to encoders of Shapeless coproducts.
-
implicit final
class
EnrichedHList[L <: HList] extends AnyVal
Provides additional methods on
HList
s. -
trait
Err extends AnyRef
Describes an error.
Describes an error.
An error has a message and a list of context identifiers that provide insight into where an error occurs in a large structure.
This type is not sealed so that codecs can return domain specific subtypes and dispatch on those subtypes.
-
trait
GenCodec[-A, +B] extends Encoder[A] with Decoder[B]
Generalized codec that allows the type to encode to vary from the type to decode.
-
implicit final
class
HListCodecEnrichedWithHListSupport[L <: HList] extends AnyVal
Provides common operations on a
Codec[HList]
. -
final
case class
SizeBound(lowerBound: Long, upperBound: Option[Long]) extends Product with Serializable
Bounds the size, in bits, of the binary encoding of a codec -- i.e., it provides a lower bound and an upper bound on the size of bit vectors returned as a result of encoding.
Bounds the size, in bits, of the binary encoding of a codec -- i.e., it provides a lower bound and an upper bound on the size of bit vectors returned as a result of encoding.
- lowerBound
Minimum number of bits
- upperBound
Maximum number of bits
-
abstract
class
Transform[F[_]] extends AnyRef
Typeclass that describes type constructors that support the
exmap
operation. -
implicit
class
TransformSyntax[F[_], A] extends AnyRef
Provides method syntax for working with a type constructor that has a Transform typeclass instance.
-
abstract
class
Transformer[A, B] extends AnyRef
Witness operation that supports transforming an
F[A]
to anF[B]
for allF
which have aTransform
instance available.Witness operation that supports transforming an
F[A]
to anF[B]
for allF
which have aTransform
instance available.- Annotations
- @implicitNotFound( ... )
- implicit final class Tuple2CodecSupport[A] extends AnyVal
-
implicit final
class
ValueCodecEnrichedWithGenericSupport[A] extends AnyVal
Provides syntax related to generic programming for codecs of any type.
-
implicit final
class
ValueCodecEnrichedWithHListSupport[A] extends AnyVal
Provides
HList
related syntax for codecs of any type. -
sealed abstract
class
DecodingContext[A] extends AnyRef
Provides the ability to sequence decoding operations such that the remainder of an operation is fed in to the input of the next operation.
Provides the ability to sequence decoding operations such that the remainder of an operation is fed in to the input of the next operation. This is useful when using codecs in for comprehensions for decoding purposes.
Note: this is a domain specific fail fast state monad.
- Annotations
- @deprecated
- Deprecated
(Since version 1.8.2) Use flatMap on Codec or Decoder, which provides equivalent functionality
Value Members
-
object
Attempt extends Serializable
Companion for Attempt.
-
object
BuildInfo extends Product with Serializable
This object was generated by sbt-buildinfo.
-
object
Codec extends EncoderFunctions with DecoderFunctions
Companion for Codec.
-
object
CodecTransformation extends Serializable
Companion for CodecTransformation.
-
object
Decoder extends DecoderFunctions
Companion for Decoder.
-
object
DecodingContext
Provides constructors for
DecodingContext
. -
object
Encoder extends EncoderFunctions
Companion for Encoder.
-
object
Err
Companion for Err.
-
object
GenCodec extends EncoderFunctions with DecoderFunctions
Companion for GenCodec.
-
object
SizeBound extends Serializable
Companion for SizeBound.
-
object
Transform
Companion for Transform.
-
object
Transformer
Companion for Transformer.