Packages

  • package root
    Definition Classes
    root
  • package com
    Definition Classes
    root
  • package criteo
    Definition Classes
    com
  • package cuttle

    Core cuttle concepts are defined here.

    Core cuttle concepts are defined here.

    Definition Classes
    criteo
  • package platforms

    Defines useful ExecutionPlatforms:

    Defines useful ExecutionPlatforms:

    • local to fork process locally in a managed way.
    • http to make HTTP calls with rate limiting.
    Definition Classes
    cuttle
  • package http

    Allow to make HTTP calls in a managed way with rate limiting.

    Allow to make HTTP calls in a managed way with rate limiting.

    It uses lolhttp as HTTP client.

    Definition Classes
    platforms
  • HttpPlatform
  • package local

    Allow to fork process locally in a managed way.

    Allow to fork process locally in a managed way.

    It provides the exec interpolator that you can use to fork shell scripts:

    exec"""hdfs dfs -cp /from /to"""()

    The script your provide will be forked into another process. Note that if you provide several commands separated by ;' only the first one will be forked and the other ones will be ignored.

    If you really need to run several commands, you can use one of the followed ways:

    1. Chain multiple execs in a for-comprehension. Because of platform mechanism it will require several platform allocations to run completely.

    2. Wrap it in one sh -c' _In this case we cannot guarantee that all process spawned inside this fork will be successfully killed when we kill the parent process._

    Definition Classes
    platforms

package http

Allow to make HTTP calls in a managed way with rate limiting.

It uses lolhttp as HTTP client.

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. http
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Type Members

  1. case class HttpPlatform (maxConcurrentRequests: Int, rateLimits: Seq[(String, RateLimit)]) extends ExecutionPlatform with Product with Serializable

    Allow to make HTTP calls in a managed way with rate limiting.

    Allow to make HTTP calls in a managed way with rate limiting. Globally the platform limits the number of concurrent requests on the platform. Additionnaly a rate limiter must be defined for each host allowed to be called by this platform.

    Example:

    platforms.http.HttpPlatform(
  maxConcurrentRequests = 10,
  rateLimits = Seq(
    .*[.]criteo[.](pre)?prod([:][0-9]+)?" -> platforms.http.HttpPlatform.RateLimit(100, per = SECONDS),
    google.com -> platforms.http.HttpPlatform.RateLimit(1, per = SECONDS)
  )
),

    While being rate limited, the Job Execution is seen as WAITING in the UI.

Value Members

  1. object HttpPlatform extends Serializable

    Access to the HttpPlatform.

Inherited from AnyRef

Inherited from Any

Ungrouped