package cuttle
Core cuttle concepts are defined here.
- A CuttleProject is basically a Workflow to execute.
- Workflows are directed acyclic graphs of Jobs.
- Scheduler is defined for a given Scheduling mechanism.
- Executions are created by a Scheduler for a given Job and SchedulingContext.
- Executor handles the SideEffects execution.
- SideEffects are plain asynchronous Scala functions and can use com.criteo.cuttle.ExecutionPlatforms to access underlying resources.
- Alphabetic
- By Inheritance
- cuttle
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Type Members
-
class
CancellationListener extends AnyRef
Allows to unsubscribe a cancel listener.
Allows to unsubscribe a cancel listener. It is sometimes useful for long running executions to dettach the cancel listener to not leak any memory.
-
sealed
trait
Completed extends AnyRef
Used a return type for the Job side effect instead of
Unit
.Used a return type for the Job side effect instead of
Unit
.We use
Future[Completed]
instead ofFuture[Unit]
to avoid mistakes in user code because of value discarding. -
case class
DBLocation(host: String, port: Int) extends Product with Serializable
Configuration of JDBC endpoint.
Configuration of JDBC endpoint.
- host
JDBC driver host
- port
JDBC driver port
-
case class
DatabaseConfig(locations: Seq[DBLocation], database: String, username: String, password: String) extends Product with Serializable
Configuration for the MySQL database used by Cuttle.
Configuration for the MySQL database used by Cuttle.
- locations
sequence of JDBC endpoints
- database
JDBC database
- username
JDBC username
- password
JDBC password
- case class DoobieLogsHandler(logger: Logger) extends Product with Serializable
-
case class
Execution[S <: Scheduling](id: String, job: Job[S], context: S.Context, streams: ExecutionStreams, platforms: Seq[ExecutionPlatform], projectName: String, projectVersion: String, previousFailures: List[ExecutionLog])(implicit executionContext: SideEffectThreadPool) extends Product with Serializable
Executions are created by the Scheduler.
Executions are created by the Scheduler.
- id
The unique id for this execution. Guaranteed to be unique.
- job
The Job for which this execution has been created.
- context
The SchedulingContext for which this execution has been created.
- streams
The execution streams are scoped stdout, stderr for the execution.
- platforms
The available ExecutionPlatforms for this execution.
- executionContext
The scoped
scala.concurrent.ExecutionContext
for this execution.
- case class ExecutionLog(id: String, job: String, startTime: Option[Instant], endTime: Option[Instant], context: Json, status: ExecutionStatus, failing: Option[FailingJob] = None, waitingSeconds: Int) extends Product with Serializable
-
trait
ExecutionPlatform extends AnyRef
An ExecutionPlatform provides controlled access to shared resources.
- sealed trait ExecutionStatus extends AnyRef
-
trait
ExecutionStreams extends AnyRef
The scoped output streams for an Execution.
The scoped output streams for an Execution. Allows the execution to log its output.
-
class
Executor[S <: Scheduling] extends MetricProvider[S]
An Executor is responsible to actually execute the SideEffect functions for the given Executions.
-
case class
Job[S <: Scheduling](id: String, scheduling: S, name: String = "", description: String = "", tags: Set[Tag] = Set.empty[Tag])(effect: SideEffect[S]) extends Product with Serializable
The job SideEffect is the most important part as it represents the real job logic to execute.
The job SideEffect is the most important part as it represents the real job logic to execute. A job is defined for a given Scheduling, for example it can be a TimeSeries job. Jobs are also Workflow with a single vertice.
- S
The kind of Scheduling used by this job.
- id
The internal job id. It will be sued to track the job state in the database, so it must not change over time otherwise the job will be seen as a new one by the scheduler. That id, being technical, should only use valid characters such as [a-zA-Z0-9_-.]
- scheduling
The scheduling configuration for the job. For example a TimeSeries job can be configured to be hourly or daily, etc.
- name
The job name as displayed in the UI.
- description
The job description as displayed in the UI.
- tags
The job tags used to filter jobs in the UI.
- effect
The job side effect, representing the real job execution.
-
trait
Logger extends AnyRef
A logger used to output internal informations.
-
trait
RetryStrategy extends AnyRef
The strategy to use to retry stuck executions.
The strategy to use to retry stuck executions.
When an Execution fails, the Executor keeps a track of this failure in a list of recently failed Executions.
If the Scheduler asks for the same Execution (same here is defined as an execution for the same Job and the same SchedulingContext) during the
retryWindow
defined by this strategy, it will be considered by the Executor as a stuck execution and delayed for the duration computed by the configured RetryStrategy. -
trait
Scheduler[S <: Scheduling] extends MetricProvider[S]
A scheduler interpret a Workflow and instanciate Executions for all defined Jobs.
A scheduler interpret a Workflow and instanciate Executions for all defined Jobs. For example, a typical cuttle Scheduler is the TimeSeries scheduler that executes the graph for each time partition.
- S
The king of Scheduling managed by this Scheduler.
-
trait
Scheduling extends AnyRef
Represent a type of scheduling.
Represent a type of scheduling. A typical cuttle scheduling is TimeSeries for example, that is a scheduling based on a calendar.
-
trait
SchedulingContext extends AnyRef
A scheduling context is the input given to each Execution that will be created for a given Scheduling.
A scheduling context is the input given to each Execution that will be created for a given Scheduling. For example, for a TimeSeries scheduling the context contain the time partition for which the job is running.
-
type
SideEffect[S <: Scheduling] = (Execution[S]) ⇒ Future[Completed]
The side effect function represents the real job execution.
The side effect function represents the real job execution. It returns a
Future[Completed]
to indicate the execution result (we use Completed here instead ofUnit
to avoid automatic value discarding, but Completed do not maintain additional state).The cuttle Executor ensures that a scheduled side effect for a given SchedulingContext will be run a least once, but cannot guarantee that it will be run exactly once. That's why the side effect function must be idempotent, meaning that if executed for the same SchedulingContext it must produce the same result.
A failed future means a failed execution.
-
case class
Tag(name: String, description: String = "") extends Product with Serializable
Allow to tag a job.
Allow to tag a job. Tags can be used in the UI/API to filter jobs and more easily retrieve them.
- name
Tag name as displayed in the UI.
- description
Description as displayed in the UI.
-
trait
Workload[S <: Scheduling] extends AnyRef
Represent the workload of a Cuttle project, ie.
Represent the workload of a Cuttle project, ie. the list of jobs to be scheduled in some ways.
-
type
XA = Transactor[IO]
Doobie transactor.
Doobie transactor. See https://github.com/tpolecat/doobie.
Value Members
-
implicit
def
scopedExecutionContext(implicit execution: Execution[_]): ExecutionContext
Automatically provide a scala
scala.concurrent.ExecutionContext
for a given Execution. -
object
Auth
The cuttle API is private for any write operation while it is publicly open for any read only operation.
The cuttle API is private for any write operation while it is publicly open for any read only operation. It allows to make it easy to build tooling that monitor any running cuttle scheduler while restricting access to potentially dangerous operations.
The UI access itself requires authentication.
-
object
Completed extends Completed with Product with Serializable
The object to use to successfully complete a job side effect.
The object to use to successfully complete a job side effect.
Future.successful(Completed)
- object Database
-
object
DatabaseConfig extends Serializable
Utilities for DatabaseConfig.
-
object
ExecutionCancelled extends RuntimeException
Used to fail an execution Future when the execution has been cancelled.
- object ExecutionPlatform
- object ExecutionStatus
- object Executor
-
object
Job extends Product with Serializable
Companion object for Job.
-
object
Metrics
Expose cuttle metrics via the prometheus protocol.
-
object
RetryStrategy
Built-in RetryStrategy
-
object
SchedulingContext
Utilities for SchedulingContext
- object Tag extends Serializable
- object ThreadPools