qscript

Type Members

1. sealed abstract class Binary[T[_[_]], A] extends MapFunc[T, A]

   

2. final case class BucketField[T[_[_]], A](src: A, value: FreeMap[T], name: FreeMap[T]) extends ProjectBucket[T, A] with Product with Serializable

   

3. final case class BucketIndex[T[_[_]], A](src: A, value: FreeMap[T], index: FreeMap[T]) extends ProjectBucket[T, A] with Product with Serializable

   

4. trait Bucketable[F[_]] extends AnyRef

   

5. type Bucketing[T[_[_]], A] = Coproduct[[β]QScriptBucket[T, β], [β]ProjectBucket[T, β], A]

   

6. sealed abstract class DeadEnd extends AnyRef

   

7. trait Diggable[F[_]] extends Serializable

   

8. final case class Drop[T[_[_]], A](src: A, from: FreeQS[T], count: FreeQS[T]) extends QScriptCore[T, A] with Product with Serializable

   

9. trait ElideBuckets[F[_]] extends Serializable

   

10. trait ElideBucketsInstances extends ElideBucketsInstances0

    

11. trait ElideBucketsInstances0 extends AnyRef

    

12. final case class EquiJoin[T[_[_]], A](src: A, lBranch: FreeQS[T], rBranch: FreeQS[T], lKey: FreeMap[T], rKey: FreeMap[T], f: JoinType, combine: JoinFunc[T]) extends Product with Serializable

    

    This is an optional component of QScript that can be used instead of ThetaJoin.

    This is an optional component of QScript that can be used instead of ThetaJoin. It’s easier to implement, but more restricted (where ThetaJoin has an arbitrary predicate to determin if a pair of records should be combined, EquiJoin has an expression on each side that is compared with simple equality).

13. type EquiQScript[T[_[_]], A] = Coproduct[[β]EquiJoin[T, β], [β]Coproduct[Read, [β]Coproduct[[β]QScriptCore[T, β], [β]Coproduct[[β]Const[DeadEnd, β], [β]SourcedPathable[T, β], β], β], β], A]

    

    A variant with a simpler join type.

    A variant with a simpler join type. A backend can choose to operate on this structure by applying the equiJoinsOnly transformation. Backends without true join support will likely find it easier to work with this than to handle full ThetaJoins.

14. final case class Filter[T[_[_]], A](src: A, f: FreeMap[T]) extends QScriptCore[T, A] with Product with Serializable

    

    Eliminates some values from a dataset, based on the result of FilterFunc.

15. type FreeMap[T[_[_]]] = Free[[β]MapFunc[T, β], Unit]

    

16. type FreeQS[T[_[_]]] = Free[[β]Coproduct[[β]QScriptBucket[T, β], [β]Coproduct[[β]ProjectBucket[T, β], [β]Coproduct[[β]ThetaJoin[T, β], [β]Coproduct[[β]QScriptCore[T, β], [β]Coproduct[[β]Const[DeadEnd, β], [β]SourcedPathable[T, β], β], β], β], β], β], Unit]

    

17. type FreeUnit[F[_]] = Free[F, Unit]

    

18. final case class GroupBy[T[_[_]], A](src: A, values: FreeMap[T], bucket: FreeMap[T]) extends QScriptBucket[T, A] with Product with Serializable

    

19. trait Helpers[T[_[_]]] extends AnyRef

    

20. type JoinFunc[T[_[_]]] = Free[[β]MapFunc[T, β], JoinSide]

    

21. sealed trait JoinSide extends AnyRef

    

22. sealed abstract class JoinType extends AnyRef

    

23. final case class LeftShift[T[_[_]], A](src: A, struct: FreeMap[T], repair: JoinFunc[T]) extends SourcedPathable[T, A] with Product with Serializable

    

    Flattens nested structure, converting each value into a data set, which are then unioned.

    Flattens nested structure, converting each value into a data set, which are then unioned.

    struct is an expression that evaluates to an array or object, which is then “exploded” into multiple values. repair is applied across the new set, integrating the exploded values into the original set.

24. final case class LeftShiftBucket[T[_[_]], A](src: A, struct: FreeMap[T], repair: JoinFunc[T], bucketShift: FreeMap[T]) extends QScriptBucket[T, A] with Product with Serializable

    

25. final case class Map[T[_[_]], A](src: A, f: FreeMap[T]) extends QScriptCore[T, A] with Product with Serializable

    

    A data-level transformation.

26. sealed abstract class MapFunc[T[_[_]], A] extends AnyRef

    

27. trait Mergeable[A] extends Serializable

    

28. trait Normalizable[F[_]] extends Serializable

    

29. trait NormalizableInstances extends NormalizableInstances0

    

30. trait NormalizableInstances0 extends AnyRef

    

31. final case class Nullary[T[_[_]], A](ejson: T[EJson]) extends MapFunc[T, A] with Product with Serializable

    

32. class Optimize[T[_[_]]] extends Helpers[T]

    

33. type Pathable[T[_[_]], A] = Coproduct[[β]Const[DeadEnd, β], [β]SourcedPathable[T, β], A]

    

34. sealed abstract class ProjectBucket[T[_[_]], A] extends AnyRef

    

    Projections are technically dimensional (i.e., QScript) operations.

    Projections are technically dimensional (i.e., QScript) operations. However, to a filesystem, they are merely Map operations. So, we use these components while building the QScript plan and they are then used in static path processing, but they are replaced with equivalent MapFuncs before being processed by the filesystem.

35. type QSState[A] = IndexedStateT[[β]\/[PlannerError, β], NameGen, NameGen, A]

    

36. type QScript[T[_[_]], A] = Coproduct[[β]ThetaJoin[T, β], [β]Coproduct[Read, [β]Coproduct[[β]QScriptCore[T, β], [β]Coproduct[[β]Const[DeadEnd, β], [β]SourcedPathable[T, β], β], β], β], A]

    

    This is the primary form seen by a backend.

    This is the primary form seen by a backend. It contains reads of files.

37. sealed abstract class QScriptBucket[T[_[_]], A] extends AnyRef

    

38. type QScriptCommon[T[_[_]], A] = Coproduct[Read, [β]Coproduct[[β]QScriptCore[T, β], [β]Coproduct[[β]Const[DeadEnd, β], [β]SourcedPathable[T, β], β], β], A]

    

    These nodes exist in all QScript structures that a backend sees.

39. sealed abstract class QScriptCore[T[_[_]], A] extends AnyRef

    

40. type QScriptInternal[T[_[_]], A] = Coproduct[[β]QScriptBucket[T, β], [β]Coproduct[[β]ProjectBucket[T, β], [β]Coproduct[[β]ThetaJoin[T, β], [β]Coproduct[[β]QScriptCore[T, β], [β]Coproduct[[β]Const[DeadEnd, β], [β]SourcedPathable[T, β], β], β], β], β], A]

    

41. type QScriptPrim[T[_[_]], A] = Coproduct[[β]QScriptCore[T, β], [β]Coproduct[[β]Const[DeadEnd, β], [β]SourcedPathable[T, β], β], A]

    

    These are the operations included in all forms of QScript.

42. type QScriptProject[T[_[_]], A] = Coproduct[[β]ProjectBucket[T, β], [β]Coproduct[[β]ThetaJoin[T, β], [β]Coproduct[[β]QScriptCore[T, β], [β]Coproduct[[β]Const[DeadEnd, β], [β]SourcedPathable[T, β], β], β], β], A]

    

43. type QScriptPure[T[_[_]], A] = Coproduct[[β]ThetaJoin[T, β], [β]Coproduct[[β]QScriptCore[T, β], [β]Coproduct[[β]Const[DeadEnd, β], [β]SourcedPathable[T, β], β], β], A]

    

    This is the target of the core compiler.

    This is the target of the core compiler. Normalization is applied to this structure, and it contains no Read or EquiJoin.

44. final case class Read[A](src: A, path: AbsFile[Sandboxed]) extends Product with Serializable

    

    A backend-resolved Root, which is now a path.

45. final case class Reduce[T[_[_]], A, N <: Nat](src: A, bucket: FreeMap[T], reducers: Sized[List[ReduceFunc[FreeMap[T]]], Succ[N]], repair: Free[[β]MapFunc[T, β], Fin[Succ[N]]]) extends QScriptCore[T, A] with Product with Serializable

    

    Performs a reduction over a dataset, with the dataset partitioned by the result of the MapFunc.

    Performs a reduction over a dataset, with the dataset partitioned by the result of the MapFunc. So, rather than many-to-one, this is many-to-fewer.

    bucket partitions the values into buckets based on the result of the expression, reducers applies the provided reduction to each expression, and repair finally turns those reduced expressions into a final value.

46. sealed trait ReduceFunc[A] extends AnyRef

    

47. final case class Sort[T[_[_]], A](src: A, bucket: FreeMap[T], order: List[(FreeMap[T], SortDir)]) extends QScriptCore[T, A] with Product with Serializable

    

    Sorts values within a bucket.

    Sorts values within a bucket. This could be represented with LeftShift(Map(_.sort, Reduce(_ :: _, ???)) but backends tend to provide sort directly, so this avoids backends having to recognize the pattern. We could provide an algebra (Sort :+: QScript)#λ => QScript so that a backend without a native sort could eliminate this node.

48. sealed trait SortDir extends AnyRef

    

49. sealed abstract class SourcedPathable[T[_[_]], A] extends AnyRef

    

50. final case class SquashBucket[T[_[_]], A](src: A) extends QScriptBucket[T, A] with Product with Serializable

    

51. final case class SrcMerge[A, B](src: A, left: B, right: B) extends Product with Serializable

    

52. final case class Take[T[_[_]], A](src: A, from: FreeQS[T], count: FreeQS[T]) extends QScriptCore[T, A] with Product with Serializable

    

53. sealed abstract class Ternary[T[_[_]], A] extends MapFunc[T, A]

    

54. final case class ThetaJoin[T[_[_]], A](src: A, lBranch: FreeQS[T], rBranch: FreeQS[T], on: JoinFunc[T], f: JoinType, combine: JoinFunc[T]) extends Product with Serializable

    

    Applies a function across two datasets, in the cases where the JoinFunc evaluates to true.

    Applies a function across two datasets, in the cases where the JoinFunc evaluates to true. The branches represent the divergent operations applied to some common src. Each branch references the src exactly once. (Since no constructor has more than one recursive component, it’s guaranteed that neither side references the src _more_ than once.)

    This case represents a full θJoin, but we could have an algebra that rewites it as Filter(_, EquiJoin(...)) to simplify behavior for the backend.

55. class Transform[T[_[_]], F[_]] extends Helpers[T]

    

56. sealed abstract class Unary[T[_[_]], A] extends MapFunc[T, A]

    

57. final case class Union[T[_[_]], A](src: A, lBranch: FreeQS[T], rBranch: FreeQS[T]) extends SourcedPathable[T, A] with Product with Serializable

    

    Creates a new dataset, |a|+|b|, containing all of the entries from each of the input sets, without any indication of which set they came from

    Creates a new dataset, |a|+|b|, containing all of the entries from each of the input sets, without any indication of which set they came from

    This could be handled as another join type, the anti-join (T[EJson] \/ T[EJson] => T[EJson], specifically as _.merge), with the condition being κ(true),