package cuttle
Core cuttle concepts are defined here.
- A CuttleProject is basically a Workflow to execute.
- Workflows are directed acyclic graphs of Jobs.
- Scheduler are defined for a given Scheduling mechanism.
- Executions are created by a Scheduler for a given Job and SchedulingContext.
- Executor handles the SideEffects execution.
- SideEffects are plain asynchronous Scala functions and can use ExecutionPlatforms to access underlying resources.
- Alphabetic
- By Inheritance
- cuttle
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Type Members
-
class
CancellationListener
extends AnyRef
Allows to unsubscribe a cancel listener.
Allows to unsubscribe a cancel listener. It is sometimes useful for long running executions to dettach the cancel listener to not leak any memory.
-
sealed
trait
Completed
extends AnyRef
Used a return type for the Job side effect instead of
Unit
.Used a return type for the Job side effect instead of
Unit
.We use
Future[Completed]
instead ofFuture[Unit]
to avoid mistakes in user code because of value discarding. -
class
CuttleProject
[S <: Scheduling] extends AnyRef
A cuttle project is a workflow to execute with the appropriate scheduler.
A cuttle project is a workflow to execute with the appropriate scheduler. See the CuttleProject companion object to create projects.
- S
The type of Scheduling used by the project (for example TimeSeries).
-
case class
DBLocation
(host: String, port: Int) extends Product with Serializable
Configuration of JDBC endpoint.
Configuration of JDBC endpoint.
- host
JDBC driver host
- port
JDBC driver port
-
case class
DatabaseConfig
(locations: Seq[DBLocation], database: String, username: String, password: String) extends Product with Serializable
Configuration for the MySQL database used by Cuttle.
Configuration for the MySQL database used by Cuttle.
- locations
sequence of JDBC endpoints
- database
JDBC database
- username
JDBC username
- password
JDBC password
-
case class
Execution
[S <: Scheduling](id: String, job: Job[S], context: S.Context, streams: ExecutionStreams, platforms: Seq[ExecutionPlatform], executionContext: ExecutionContext, cuttleProject: CuttleProject[S]) extends Product with Serializable
Executions are created by the Scheduler.
Executions are created by the Scheduler.
- id
The unique id for this execution. Guaranteed to be unique.
- job
The Job for which this execution has been created.
- context
The SchedulingContext for which this execution has been created.
- streams
The execution streams are scoped stdout, stderr for the execution.
- platforms
The available ExecutionPlatforms for this execution.
- executionContext
The scoped
scala.concurrent.ExecutionContext
for this execution.- cuttleProject
The CuttleProject in which this execution has been created.
-
trait
ExecutionPlatform
extends AnyRef
An ExecutionPlatform provides controlled access to shared resources.
-
trait
ExecutionStreams
extends AnyRef
The scoped output streams for an Execution.
The scoped output streams for an Execution. Allows the execution to log its output.
-
class
Executor
[S <: Scheduling] extends MetricProvider
An Executor is responsible to actually execute the SideEffect functions for the given Executions.
-
case class
Job
[S <: Scheduling](id: String, scheduling: S, name: String = "", description: String = "", tags: Set[Tag] = Set.empty[Tag])(effect: SideEffect[S]) extends Workflow[S] with Product with Serializable
The job SideEffect is the most important part as it represents the real job logic to execute.
The job SideEffect is the most important part as it represents the real job logic to execute. A job is defined for a given Scheduling, for example it can be a TimeSeries job. Jobs are also Workflow with a single vertice.
- S
The kind of Scheduling used by this job.
- id
the internal job id. It will be sued to track the job state in the database, so it must not change over time otherwise the job will be seen as a new one by the scheduler.
- scheduling
The scheduling configuration for the job. For example a TimeSeries job can be configured to be hourly or daily, etc.
- name
The job name as displayed in the UI.
- description
The job description as displayed in the UI.
- tags
The job tags used to filter jobs in the UI.
- effect
The job side effect, representing the real job execution.
-
trait
Logger
extends AnyRef
A logger used to output internal informations.
-
trait
RetryStrategy
extends AnyRef
The strategy to use to retry stuck executions.
The strategy to use to retry stuck executions.
When an Execution fails, the Executor keeps a track of this failure in a list of recently failed Executions.
If the Scheduler asks for the same Execution (same here is defined as an execution for the same Job and the same SchedulingContext) during the
retryWindow
defined by this strategy, it will be considered by the Executor as a stuck execution and delayed for the duration computed by the configured RetryStrategy. -
trait
Scheduler
[S <: Scheduling] extends MetricProvider
A scheduler interpret a Workflow and instanciate Executions for all defined Jobs.
A scheduler interpret a Workflow and instanciate Executions for all defined Jobs. For example, a typical cuttle Scheduler is the TimeSeries scheduler that executes the graph for each time partition.
- S
The king of Scheduling managed by this Scheduler.
-
trait
Scheduling
extends AnyRef
Represent a type of scheduling.
Represent a type of scheduling. A typical cuttle scheduling is TimeSeries for example, that is a scheduling based on a calendar.
-
trait
SchedulingContext
extends AnyRef
A scheduling context is the input given to each Execution that will be created for a given Scheduling.
A scheduling context is the input given to each Execution that will be created for a given Scheduling. For example, for a TimeSeries scheduling the context contain the time partition for which the job is running.
-
type
SideEffect[S <: Scheduling] = (Execution[S]) ⇒ Future[Completed]
The side effect function represents the real job execution.
The side effect function represents the real job execution. It returns a
Future[Completed]
to indicate the execution result (we use Completed here instead ofUnit
to avoid automatic value discarding, but Completed do not maintain additional state).The cuttle Executor ensures that a scheduled side effect for a given SchedulingContext will be run a least once, but cannot garantee that it will be run execactly once. That's why the side effect function must be idempotent, meaning that if executed for the same SchedulingContext it must produce the same result.
A failed future means a failed execution.
-
case class
Tag
(name: String, description: String = "") extends Product with Serializable
Allow to tag a job.
Allow to tag a job. Tags can be used in the UI/API to filter jobs and more easily retrieve them.
- name
Tag name as displayed in the UI.
- description
Description as displayed in the UI.
-
trait
Workflow
[S <: Scheduling] extends AnyRef
The workflow to be run by cuttle.
The workflow to be run by cuttle. A workflow is defined for a given Scheduling, for example it can be a TimeSeries workflow.
- S
The kind of Scheduling used by this workflow.
-
type
XA = Transactor[IOLite]
Doobie transactor.
Doobie transactor. See https://github.com/tpolecat/doobie.
Value Members
-
implicit
val
logger: Logger { def logMe(message: => String,level: String): Unit }
Default implicit logger that output everything to stdout
-
implicit
def
scopedExecutionContext(implicit execution: Execution[_]): ExecutionContext
Automatically provide a scala
scala.concurrent.ExecutionContext
for a given Execution. -
object
Auth
The cuttle API is private for any write operation while it is publicly open for any read only operation.
The cuttle API is private for any write operation while it is publicly open for any read only operation. It allows to make it easy to build tooling that monitor any running cuttle scheduler while restricting access to potentially dangerous operations.
The UI access itself requires authentication.
-
object
Completed
extends Completed with Product with Serializable
The object to use to successfully complete a job side effect.
The object to use to successfully complete a job side effect.
Future.successful(Completed)
-
object
CuttleProject
Create new projects.
-
object
DatabaseConfig
extends Serializable
Utilities for DatabaseConfig.
-
object
ExecutionCancelled
extends RuntimeException
Used to fail an execution Future when the execution has been cancelled.
-
object
Metrics
Expose cuttle metrics via the prometheus protocol.
-
object
RetryStrategy
Built-in RetryStrategy
-
object
SchedulingContext
Utilities for SchedulingContext
-
object
Workflow
Utilities for Workflow.