scalding

Type Members

1. trait ExternalService[K, +V] extends Service[K, V]

   This represents a service that is *external* to the current job.

2. type FactoryInput = (Interval[Timestamp], Mode)

   The Platform recursively passes this input around to describe a step forward: requested input time span, and scalding Mode

3. type FailureReason = String

   These are printed/logged only when we can't make any progress

4. type FlowInput = (FlowDef, Mode)

   When it is time to run build the final flow, this is what scalding needs.

   When it is time to run build the final flow, this is what scalding needs. It is modified in the Reader[FlowInput, T]

5. case class FlowPlanException(errs: List[FailureReason]) extends Exception with Product with Serializable

   Puts a type on the final exception thrown by the scalding Planner

6. type FlowProducer[+T] = Reader[(FlowDef, Mode), T]

   This is a function that modifies a flow to return T generally T will be some kind of TypedPipe

7. type FlowToPipe[+T] = Reader[(FlowDef, Mode), TimedPipe[T]]

   We so commonly talk about producing TimedPipe we define this

8. sealed trait InternalService[K, +V] extends Service[K, V]

   This represents a join against data that is materialized by a store in the current job

9. type KeyValuePipe[+K, +V] = TypedPipe[(Timestamp, (K, V))]

10. class Memo[T] extends Serializable

11. type PipeFactory[+T] = StateWithError[(Interval[Timestamp], Mode), List[FailureReason], FlowToPipe[T]]

    We are usually producing Pipes in the Planner

12. class PipeFactoryOps[+T] extends AnyRef

13. type PlannerOutput[+T] = StateWithError[(Interval[Timestamp], Mode), List[FailureReason], T]

    The recursive planner produces these objects which are Monads

14. class Scalding extends Platform[Scalding] with Serializable

15. trait ScaldingExecutionConfig extends ChillExecutionConfig[Scalding]

16. sealed trait Service[K, +V] extends Serializable

17. trait Sink[T] extends AnyRef

18. trait Store[K, V] extends Serializable

19. case class StoreService[K, V](store: BatchedStore[K, V]) extends InternalService[K, V] with Store[K, V] with Product with Serializable

20. type TimedPipe[+T] = TypedPipe[(Timestamp, T)]

    How we represent the streams in scalding

21. type Try[+T] = Either[List[FailureReason], T]

22. case class WriteDot(filename: String) extends Product with Serializable

    Use this option to write the logical graph that cascading produces before Map/Reduce planning.

23. case class WriteStepsDot(filename: String) extends Product with Serializable

    Use this option to write map/reduce graph that cascading produces Use the job name as the key