tasksystem
A trait that declares all of the dependency management operators for Tasks and their friends, to ensure consistency in implementation between the various objects that implement them.
Represents a link from one Dependable to another, which can have dependencies wired up to it, and will vector the dependencies to the appropriate sub-dependencies.
Represents a group of Dependable objects such that depedency operations on a DependencyGroup are transmitted to all contained Dependables.
A task that returns either the left or right task based on a deferred choice.
A trait for all tasks that required a specific number of cores and a specific amount of memory.
A trait for all tasks that required a specific number of cores and a specific amount of memory. The amount can be decided any time prior to takeResources() being called, but must be a single set of values.
A task that can execute a method in the Jvm, and does not generate any new tasks.
Determines if a the task failed because it ran out of memory by looking for various strings in the log file.
A trait for all tasks that wish to linearly increase their memory upon each retry.
A trait for all tasks that wish to linearly increase their memory upon each retry.
By default retries until the maximum system memory is reached, and increases by the initial amount of memory required.
A trait for all tasks that wish to double their memory upon each retry.
A trait to facilitate retrying a task with more memory.
Simple trait to retry a given number of times
Trivial No-Op task that runs inside the JVM and does nothing.
A task that runs and does nothing.
Trait for tasks that would like to be able to pipe data using stdin and stdout.
Trait for tasks that would like to be able to pipe data using stdin and stdout. Tasks
are still responsible for managing their input and output - so if tasks can also read
and write from files, they must have the input set to Io.Stdin and/or output to Io.Stdout
before connecting them with pipes.
See also the traits PipeIn and PipeOut which provide a simplified way to implement a piping task that only receives on or only emits to a pipe.
a symbolic type representing the kind of data the task can accept
a symbolic type representing the kind of data the task emits
A simplified trait for sink tasks that can receive data via a pipe, but cannot pipe onwards.
A simplified trait for generate tasks that can pipe out, but cannot receive data from a pipe.
Simple trait to track tasks within a pipeline
A task that can execute a set of commands in its own process, and does not generate any new tasks.
A trait to facilitate retry a task when it has failed.
Traits that isolates methods about how Tasks interact with the Scheduler, and allows for multiple implementations independent from the Task hierarchy.
Executes a command, with a set of arguments, in a shell.
A simplified In JVM task that hides the need to return an Int exit code and instead uses the raising of exceptions or lack thereof to indicate failure and success.
Base class for all tasks, multi-tasks, and workflows.
Base class for all tasks, multi-tasks, and workflows.
Once a task is constructed, it has the following evolution: 1. Any tasks on which it depends are added (see ==>). 2. When all tasks on which it is dependent have completed, the getTasks method is called to create a set of tasks. This task becomes dependent on any task that is returned that is not itself. 3. When all newly dependent tasks from #2 are complete, as well as this task, the onComplete method is called to perform any light-weight modification of this task. 4. If a task failed during execution or within onComplete, the retry method will be called until the task no longer wishes to retry or succeeds.
A task that should be directly executed or scheduled.
A task that should be directly executed or scheduled.
A few things about unit tasks: - a task can depend on data from tasks on which it is dependent. See Callbacks. - a task extending this class should return only one task in its getTasks method. See Task.getTasks. - a task can perform any final logic dependent on the resources with which it is scheduled. See Scheduler!.schedule*.
When a unit task gets scheduled, the dagr.core.execsystem.Scheduler.schedule method will be called to allow any final logic based on the resources this task was scheduled with. This is in addition to the steps listed in Task.
An object that can be implicitly converted to from a None when using Option[Dependable].
Object for creating linker tasks that transfer state between other tasks at runtime.
Companion object to provide a couple of helper constructors for ShellCommand.
Companion object for SimpleInJvmTask that provides helpful factory methods.
Utility methods to aid in working with a task.
A task that returns either the left or right task based on a deferred choice. The choice function is not evaluated until all dependencies have been met and the
EitherTaskneeds to make a decision about which task to return from