qscript
Rewrites adjacent nodes.
This extracts statically-known paths from QScript queries to make it easier for connectors to map queries to their own filesystems.
This is an optional component of QScript that can be used instead of ThetaJoin.
This is an optional component of QScript that can be used instead of ThetaJoin. It’s easier to implement, but more restricted (where ThetaJoin has an arbitrary predicate to determine if a pair of records should be combined, EquiJoin has an expression on each side that is compared with simple equality).
Converts any {Shifted}Read containing a directory to a union of all the files in that directory.
Extracts paths of particular type from QScript, collecting them in the
provided ApplicativePlus.
Eliminates some values from a dataset, based on the result of f (which
must evaluate to a boolean value for each element in the set).
This is like scalaz.Inject, but for injecting an arbitrary coproduct
where all of the components are in the OUT coproduct in any order.
This is like scalaz.Inject, but for injecting an arbitrary coproduct
where all of the components are in the OUT coproduct in any order.
It _may_ be unprincipled (otherwise, why not allow scalaz.Inject to work this way directly?) But it is temporarily necessary in order to “inject” our more constrained versions of QScript into QScriptTotal.
Flattens nested structure, converting each value into a data set, which are then unioned.
Flattens nested structure, converting each value into a data set, which are then unioned.
struct is an expression that evaluates to an array or object, which is
then “exploded” into multiple values. idStatus indicates what each of
those exploded values should look like (either just the value, just the “id”
(i.e., the key or index), or a 2-element array of key and value). repair
is applied across the new set, integrating the exploded values into the
original set.
E.g., in:
LeftShift(x,
ProjectField(SrcHole, "bar"),
ExcludeId,
ConcatMaps(LeftSide, MakeMap("bar", RightSide)))
If x, then
that’s what LeftSide is. And RightSide is values like 1, 2, and
{ bar: 1 }3, because that’s what you get from flattening the struct.So then our
right-biased quasar.qscript.MapFuncs.ConcatMaps says to concat
{ foo: 7, bar: [1, 2, 3] } with , resulting in
{ foo: 7, bar: 2 }{ foo: 7, bar: 1 } (then again with and
x{ foo: 7, bar: 3 }, finishing up the handling of that one element in the
original () dataset.
A data-level transformation.
Projections are technically dimensional (i.e., QScript) operations.
Projections are technically dimensional (i.e., QScript) operations. However, to a filesystem, they are merely Map operations. So, we use these components while building the QScript plan and they are then used in static path processing, but they are replaced with equivalent MapFuncs before being processed by the filesystem.
QScript that has not gone through Read conversion.
QScript that has gone through Read conversion.
QScript that has gone through Read conversion.
NB: Once QScriptTotal goes away, this could become parametric in the path type.
QScript that has gone through Read conversion and shifted conversion.
QScript that has gone through Read conversion and shifted conversion.
NB: Once QScriptTotal goes away, this could become parametric in the path type.
This type is _only_ used for join branch-like structures.
This type is _only_ used for join branch-like structures. It’s an unfortunate consequence of not having mutually-recursive data structures. Once we do, this can go away. It should _not_ be used in other situations.
NB: We're using the "alias" method of building the coproduct here as it provides a modest reduction in compilation time (~15%) for this module.
A backend-resolved Root, which is now a path.
Performs a reduction over a dataset, with the dataset partitioned by the result of the bucket MapFunc.
Performs a reduction over a dataset, with the dataset partitioned by the result of the bucket MapFunc. So, rather than many-to-one, this is many-to-fewer.
bucket partitions the values into buckets based on the result of the
expression, reducers applies the provided reduction to each expression,
and repair finally turns those reduced expressions into a final value.
ReduceIndex is guaranteed to be a valid index into reducers.
This optional transformation changes the semantics of Read.
Like ShiftRead, but only applies when the path is a directory, leaving file reads untouched.
Similar to Read, but returns a dataset with an entry for each record.
Replaces ThetaJoin with EquiJoin, which is often more feasible for connectors to implement.
This optional transformation changes the semantics of Read.
Sorts values within a bucket.
Sorts values within a bucket. This can be an _unstable_ sort, but the
elements of order must be stably sorted.
Chooses a subset of values from a dataset, given a count.
Centralizes the knowledge of T[_[_]] as well as certain type classes required for many operations.
Centralizes the knowledge of T[_[_]] as well as certain type classes required for many operations. This is for compilation performance.
Applies a function across two datasets, in the cases where the JoinFunc evaluates to true.
Applies a function across two datasets, in the cases where the JoinFunc evaluates to true. The branches represent the divergent operations applied to some common src. Each branch references the src exactly once. (Since no constructor has more than one recursive component, it’s guaranteed that neither side references the src _more_ than once.)
This case represents a full θJoin, but we could have an algebra that rewrites it as Filter(_, EquiJoin(...)) to simplify behavior for the backend.
Creates a new dataset that contains the elements from the datasets created by each branch.
Creates a new dataset that contains the elements from the datasets created by each branch. Duplicate values should be eliminated.
A placeholder value that can appear in plans, but will never be referenced in the result.
A placeholder value that can appear in plans, but will never be referenced in the result. We consider this a wart. It should be implemented as an arbitrary value (of cardinality 1) with minimal cost to generate (since it will simply be discarded).
Drops the first count elements from a dataset.
The top level of a filesystem.
The top level of a filesystem. During compilation this represents /, but
in the structure a backend sees, it represents the mount point.
Similar to Take, but keeps a random sampling of elements.
Drops all elements after the first count elements from a dataset.
A variant of repeatedly that works with Inject instances.
The various representations of an arbitrary query, as seen by the filesystem connectors, along with the operations for dealing with them.
There are a few patterns that are worth noting: -
(src: A, ..., lBranch: FreeQS[T], rBranch: FreeQS[T], ...)– used in operations that combine multiple data sources (notably joins and unions). This holds the divergent parts of the data sources in the branches, with SrcHole indicating a reference back to the commonsrcof the two branches. There is not required to be a SrcHole. -Free[F, A]– we use this structure as a restricted form of variable binding, whereFis some pattern functor, andAis some enumeration that has a specific referent. E.g., FreeMap is a recursive structure of MapFunc that has a single “variable”, SrcHole, which (usually) refers to thesrcparameter of that operation. JoinFunc, FreeQS, and therepairparameter to Reduce behave similarly. - We use the type parameterQS[_]to indicate QScript, as well as the type parametersIN[_]andOUT[_]to indicate the input and output coproducts in transformations where they can be different.