qscript

Type Members

1. sealed abstract class Binary[T[_[_]], A] extends MapFunc[T, A]

   

2. final case class BucketField[T[_[_]], A](src: A, value: FreeMap[T], name: FreeMap[T]) extends QScriptBucket[T, A] with Product with Serializable

   

3. final case class BucketIndex[T[_[_]], A](src: A, value: FreeMap[T], index: FreeMap[T]) extends QScriptBucket[T, A] with Product with Serializable

   

4. trait Bucketable[F[_]] extends Serializable

   

5. sealed abstract class DeadEnd extends AnyRef

   

6. final case class Drop[T[_[_]], A](src: A, from: FreeQS[T], count: FreeQS[T]) extends QScriptCore[T, A] with Product with Serializable

   

7. trait ElideBuckets[F[_]] extends Serializable

   

8. trait ElideBucketsInstances extends ElideBucketsInstances0

   

9. trait ElideBucketsInstances0 extends AnyRef

   

10. final case class EquiJoin[T[_[_]], A](src: A, lBranch: FreeQS[T], rBranch: FreeQS[T], lKey: FreeMap[T], rKey: FreeMap[T], f: JoinType, combine: JoinFunc[T]) extends Product with Serializable

    

    This is an optional component of QScript that can be used instead of ThetaJoin.

    This is an optional component of QScript that can be used instead of ThetaJoin. It’s easier to implement, but more restricted (where ThetaJoin has an arbitrary predicate to determin if a pair of records should be combined, EquiJoin has an expression on each side that is compared with simple equality).

11. type EquiQScript[T[_[_]], A] = Coproduct[[β]EquiJoin[T, β], [β]Coproduct[Read, [β]Coproduct[[β]QScriptCore[T, β], [β]Coproduct[[β]Const[DeadEnd, β], [β]SourcedPathable[T, β], β], β], β], A]

    

    A variant with a simpler join type.

    A variant with a simpler join type. A backend can choose to operate on this structure by applying the equiJoinsOnly transformation. Backends without true join support will likely find it easier to work with this than to handle full ThetaJoins.

12. final case class Filter[T[_[_]], A](src: A, f: FreeMap[T]) extends QScriptCore[T, A] with Product with Serializable

    

    Eliminates some values from a dataset, based on the result of FilterFunc.

13. type FreeMap[T[_[_]]] = Free[[β]MapFunc[T, β], Unit]

    

14. type FreeQS[T[_[_]]] = Free[[β]Coproduct[[β]QScriptBucket[T, β], [β]Coproduct[[β]ThetaJoin[T, β], [β]Coproduct[[β]QScriptCore[T, β], [β]Coproduct[[β]Const[DeadEnd, β], [β]SourcedPathable[T, β], β], β], β], β], Unit]

    

15. type FreeUnit[F[_]] = Free[F, Unit]

    

16. final case class GroupBy[T[_[_]], A](src: A, values: FreeMap[T], bucket: FreeMap[T]) extends QScriptBucket[T, A] with Product with Serializable

    

17. trait Helpers[T[_[_]]] extends AnyRef

    

18. type JoinFunc[T[_[_]]] = Free[[β]MapFunc[T, β], JoinSide]

    

19. sealed trait JoinSide extends AnyRef

    

20. sealed abstract class JoinType extends AnyRef

    

21. final case class LeftShift[T[_[_]], A](src: A, struct: FreeMap[T], repair: JoinFunc[T]) extends SourcedPathable[T, A] with Product with Serializable

    

    Flattens nested structure, converting each value into a data set, which are then unioned.

    Flattens nested structure, converting each value into a data set, which are then unioned.

    struct is an expression that evaluates to an array or object, which is then “exploded” into multiple values. repair is applied across the new set, integrating the exploded values into the original set.

22. final case class LeftShiftBucket[T[_[_]], A](src: A, struct: FreeMap[T], repair: JoinFunc[T], bucketShift: FreeMap[T]) extends QScriptBucket[T, A] with Product with Serializable

    

23. final case class Map[T[_[_]], A](src: A, f: FreeMap[T]) extends SourcedPathable[T, A] with Product with Serializable

    

    A data-level transformation.

24. sealed abstract class MapFunc[T[_[_]], A] extends AnyRef

    

25. trait Mergeable[A] extends Serializable

    

26. trait Normalizable[F[_]] extends Serializable

    

27. trait NormalizableInstances extends NormalizableInstances0

    

28. trait NormalizableInstances0 extends AnyRef

    

29. final case class Nullary[T[_[_]], A](ejson: T[EJson]) extends MapFunc[T, A] with Product with Serializable

    

30. class Optimize[T[_[_]]] extends Helpers[T]

    

31. type Pathable[T[_[_]], A] = Coproduct[[β]Const[DeadEnd, β], [β]SourcedPathable[T, β], A]

    

32. type QScript[T[_[_]], A] = Coproduct[[β]ThetaJoin[T, β], [β]Coproduct[Read, [β]Coproduct[[β]QScriptCore[T, β], [β]Coproduct[[β]Const[DeadEnd, β], [β]SourcedPathable[T, β], β], β], β], A]

    

    This is the primary form seen by a backend.

    This is the primary form seen by a backend. It contains reads of files.

33. sealed abstract class QScriptBucket[T[_[_]], A] extends AnyRef

    

34. type QScriptCommon[T[_[_]], A] = Coproduct[Read, [β]Coproduct[[β]QScriptCore[T, β], [β]Coproduct[[β]Const[DeadEnd, β], [β]SourcedPathable[T, β], β], β], A]

    

    These nodes exist in all QScript structures that a backend sees.

35. sealed abstract class QScriptCore[T[_[_]], A] extends AnyRef

    

36. type QScriptInternal[T[_[_]], A] = Coproduct[[β]QScriptBucket[T, β], [β]Coproduct[[β]ThetaJoin[T, β], [β]Coproduct[[β]QScriptCore[T, β], [β]Coproduct[[β]Const[DeadEnd, β], [β]SourcedPathable[T, β], β], β], β], A]

    

37. type QScriptPrim[T[_[_]], A] = Coproduct[[β]QScriptCore[T, β], [β]Coproduct[[β]Const[DeadEnd, β], [β]SourcedPathable[T, β], β], A]

    

    These are the operations included in all forms of QScript.

38. type QScriptPure[T[_[_]], A] = Coproduct[[β]ThetaJoin[T, β], [β]Coproduct[[β]QScriptCore[T, β], [β]Coproduct[[β]Const[DeadEnd, β], [β]SourcedPathable[T, β], β], β], A]

    

    This is the target of the core compiler.

    This is the target of the core compiler. Normalization is applied to this structure, and it contains no Read or EquiJoin.

39. final case class Read[A](src: A, path: AbsFile[Sandboxed]) extends Product with Serializable

    

    A backend-resolved Root, which is now a path.

40. final case class Reduce[T[_[_]], A, N <: Nat](src: A, bucket: FreeMap[T], reducers: Sized[List[ReduceFunc[FreeMap[T]]], Succ[N]], repair: Free[[β]MapFunc[T, β], Fin[Succ[N]]]) extends QScriptCore[T, A] with Product with Serializable

    

    Performs a reduction over a dataset, with the dataset partitioned by the result of the MapFunc.

    Performs a reduction over a dataset, with the dataset partitioned by the result of the MapFunc. So, rather than many-to-one, this is many-to-fewer.

    bucket partitions the values into buckets based on the result of the expression, reducers applies the provided reduction to each expression, and repair finally turns those reduced expressions into a final value.

41. sealed trait ReduceFunc[A] extends AnyRef

    

42. final case class Sort[T[_[_]], A](src: A, bucket: FreeMap[T], order: List[(FreeMap[T], SortDir)]) extends QScriptCore[T, A] with Product with Serializable

    

    Sorts values within a bucket.

    Sorts values within a bucket. This could be represented with LeftShift(Map(_.sort, Reduce(_ :: _, ???)) but backends tend to provide sort directly, so this avoids backends having to recognize the pattern. We could provide an algebra (Sort :+: QScript)#λ => QScript so that a backend without a native sort could eliminate this node.

43. sealed trait SortDir extends AnyRef

    

44. sealed abstract class SourcedPathable[T[_[_]], A] extends AnyRef

    

45. final case class SquashBucket[T[_[_]], A](src: A) extends QScriptBucket[T, A] with Product with Serializable

    

46. final case class SrcMerge[A, B](src: A, left: B, right: B) extends Product with Serializable

    

47. final case class Take[T[_[_]], A](src: A, from: FreeQS[T], count: FreeQS[T]) extends QScriptCore[T, A] with Product with Serializable

    

48. sealed abstract class Ternary[T[_[_]], A] extends MapFunc[T, A]

    

49. final case class ThetaJoin[T[_[_]], A](src: A, lBranch: FreeQS[T], rBranch: FreeQS[T], on: JoinFunc[T], f: JoinType, combine: JoinFunc[T]) extends Product with Serializable

    

    Applies a function across two datasets, in the cases where the JoinFunc evaluates to true.

    Applies a function across two datasets, in the cases where the JoinFunc evaluates to true. The branches represent the divergent operations applied to some common src. Each branch references the src exactly once. (Since no constructor has more than one recursive component, it’s guaranteed that neither side references the src _more_ than once.)

    This case represents a full θJoin, but we could have an algebra that rewites it as Filter(_, EquiJoin(...)) to simplify behavior for the backend.

50. class Transform[T[_[_]], F[_]] extends Helpers[T]

    

51. sealed abstract class Unary[T[_[_]], A] extends MapFunc[T, A]

    

52. final case class Union[T[_[_]], A](src: A, lBranch: FreeQS[T], rBranch: FreeQS[T]) extends SourcedPathable[T, A] with Product with Serializable

    

    Creates a new dataset, |a|+|b|, containing all of the entries from each of the input sets, without any indication of which set they came from

    Creates a new dataset, |a|+|b|, containing all of the entries from each of the input sets, without any indication of which set they came from

    This could be handled as another join type, the anti-join (T[EJson] \/ T[EJson] => T[EJson], specifically as _.merge), with the condition being κ(true),