This does the AlsoProducer logic of making ensure
a part of the
flow, but not this output.
This makes sure that the output FlowToPipe[T] produces a TypedPipe[T] with only times in the given time interval.
Like pipeFactory, but allows the output of the factory to be mapped.
Like pipeFactory, but allows the output of the factory to be mapped.
Useful when using TextLine, for example, where the lines need to be parsed before you can extract the timestamps.
Memoize the inner reader This is not a performance optimization, but a correctness one applicable to some cases (namely any function that mutates the FlowDef or does IO).
Memoize the inner reader This is not a performance optimization, but a correctness one applicable to some cases (namely any function that mutates the FlowDef or does IO). Though we are working in a referentially transparent manner, the application of the function inside the PipeFactory (the Reader) mutates the FlowDef. For a fixed PipeFactory, we only want to mutate a given FlowDef once. If we memoize with this function, it guarantees that the PipeFactory is idempotent.
Given a constructor function, computes the maximum available range of time or gives an error.
Given a constructor function, computes the maximum available range of time or gives an error.
Works by calling validateTaps on the Mappable, so if that does not work correctly this will be incorrect.
Like pipeFactory, but allows the output of the factory to be mapped to an optional value.
Like pipeFactory, but allows the output of the factory to be mapped to an optional value.
Useful when using TextLine, for example, where the lines need to be parsed before you can extract the timestamps.
This uses minify to find the smallest subset we can run.
This uses minify to find the smallest subset we can run. If you don't want this behavior, then use pipeFactoryExact which either produces all the DateRange or the whole job fails.
Use this method to interop with existing scalding code Note this may return a smaller DateRange than you ask for If you need an exact DateRange see toPipeExact.
Use this method to interop with existing scalding code that expects to schedule an exact DateRange or fail.