package streamlets
- Alphabetic
- Public
- All
Type Members
- sealed trait AccessMode extends AnyRef
- final case class BooleanConfigParameter(key: String, description: String = "", defaultValue: Option[Boolean] = None) extends ConfigParameter with Product with Serializable
- case class BootstrapServersForTopicNotFound(topic: Topic) extends Exception with Product with Serializable
-
trait
Codec[T] extends Serializable
- Annotations
- @implicitNotFound( ... )
-
trait
CodecInlet[T] extends Inlet
A handle to read and deserialize data into elements of type
T
. -
trait
CodecOutlet[T] extends Outlet
A handle to serialize elements of type
T
into a partitioned stream. -
trait
ConfigParameter extends AnyRef
Describes arguments that have to be supplied when deploying a Cloudflow application.
Describes arguments that have to be supplied when deploying a Cloudflow application. Each streamlet in an application can require zero or more arguments. The configuration parameter contains information for the end user entering the values and logic to validate those values so they are entered correctly.
Streamlet developers can create their own configuration parameters by deriving from ConfigParameter, the parameter can use validation logic from a fixed set of types derived from ValidationType.
The developer can also re-use any of the following predefined configuration parameter types:
- IntegerConfigParameter
- StringConfigParameter
- DoubleConfigParameter
- BooleanConfigParameter
- RegExpConfigParameter
- DurationConfigParameter
- MemorySizeConfigParameter
Each of these pre-defined parameters uses one of the following validation types:
- BooleanValidationType
- IntegerValidationType
- DoubleValidationType
- RegexpValidationType
- DurationValidationType
- MemorySizeValidationType
Example of how to create a custom configuration parameter:
final case class MilitaryTimeParameter(key: String, defaultValue: Option[String] = None) extends ConfigParameter[String] { val description: String = "This parameter validates that the users enter the time in 24h format." val validation = RegexpValidation("^(0[0-9]|1[0-9]|2[0-3]|[0-9]):[0-5][0-9]$") def toDescriptor = ConfigParameterDescriptor(key,description,validation,defaultValue) }
Example on how to use a configuration parameter in a streamlet:
class RecordSumFlow extends AkkaStreamlet { val recordsInWindowParameter = IntegerConfigParameter("records-in-window","This value describes how many records of data should be processed together, default 65 KB", Some(64 * 1024)) override def configParameters = Set(recordsInWindowParameter) val inlet = AvroInlet[Metric]("metric") val outlet = AvroOutlet[SummedMetric]("summed-metric") val shape = StreamletShape.withInlets(inlet).withOutlets(outlet) def createLogic = new RunnableGraphStreamletLogic() { val recordsInWindow = streamletConfig.getInt(recordsInWindowParameter.key) def sumRecords(records: Seq[Metric]) : SummedMetric {....} def flow = FlowWithOffsetContext[Metric].grouped(recordsInWindow).map(sumRecords) def runnableGraph() = atLeastOnceSource(inlet).via(flow).to(atLeastOnceSink(outlet)) } }
- case class DecodeException(msg: String, cause: Throwable) extends RuntimeException with NoStackTrace with Product with Serializable
- final case class DoubleConfigParameter(key: String, description: String = "", defaultValue: Option[Double] = None) extends ConfigParameter with Product with Serializable
-
sealed abstract
class
Dun extends Serializable
Typically used together with
Future
to signal completion but there is no actual value completed.Typically used together with
Future
to signal completion but there is no actual value completed. More clearly signals intent thanUnit
and is available both from Scala and Java (whichUnit
is not).Adopted from akka to avoid
streamlets
depending on akka. - case class DurationConfigParameter(key: String, description: String = "", defaultValue: Option[String] = None) extends ConfigParameter with Product with Serializable
-
final
case class
ExceptionAcc(exceptions: Vector[Throwable]) extends Exception with Product with Serializable
An exception to return when the runner returns an accumulated list of distinct exceptions.
-
trait
Inlet extends StreamletPort
A handle to read data according to a schema.
- final case class IntegerConfigParameter(key: String, description: String = "", defaultValue: Option[Int] = None) extends ConfigParameter with Product with Serializable
- case class LoadedStreamlet(streamlet: Streamlet[StreamletContext], config: StreamletDefinition) extends Product with Serializable
- final case class MemorySizeConfigParameter(key: String, description: String = "", defaultValue: Option[String] = None) extends ConfigParameter with Product with Serializable
-
trait
Outlet extends StreamletPort
A handle to write data according to a schema.
-
case class
PortMapping(port: String, topic: Topic) extends Product with Serializable
Mapping between the port name and the topic
- final case class RegExpConfigParameter(key: String, description: String, pattern: String, defaultValue: Option[String] = None) extends ConfigParameter with Product with Serializable
- final case class RegexpValidationType(regExpPattern: String) extends ValidationType with Product with Serializable
-
final
case class
SchemaDefinition(name: String, schema: String, fingerprint: String, format: String) extends Product with Serializable
Describes the schema.
Describes the schema.
name
should be the unique name of the schema.schema
is a string representation of the schema itself. (In the case of the avro format this is a json document) Thefingerprint
is a consistent hash of the schema. Theformat
specifies the format of the schema. Unique names should be used for different formats. (In the case of Avro, format is "avro") - abstract class Streamlet[Context <: StreamletContext] extends AnyRef
-
trait
StreamletAttribute extends AnyRef
Describes that a streamlet requires specific configuration by the cloudflow platform.
Describes that a streamlet requires specific configuration by the cloudflow platform.
The configuration value will be added by the platform under the
configPath
path in the Streamlet configuration. - trait StreamletContext extends AnyRef
- case class StreamletContextData(appId: String, appVersion: String, portMappings: Map[String, Topic], volumeMounts: Option[List[VolumeMount]] = None, config: Config) extends Product with Serializable
- class StreamletContextException extends Exception
- case class StreamletDefinition(appId: String, appVersion: String, streamletRef: String, streamletClass: String, portMappings: List[PortMapping], volumeMounts: List[VolumeMount], config: Config) extends Product with Serializable
-
trait
StreamletExecution extends AnyRef
A handle to the running Streamlet.
-
trait
StreamletLoader extends AnyRef
Functions to load a streamlet from its configuration through reflection.
- trait StreamletLogic[Context <: StreamletContext] extends Serializable
-
trait
StreamletPort extends AnyRef
A named port handle handle to read or write data according to a schema.
-
trait
StreamletRuntime extends AnyRef
A simple marker trait to provide the name of the "runtime" supported by a streamlet, e.g.
A simple marker trait to provide the name of the "runtime" supported by a streamlet, e.g. "akka", "spark", etc.
Implementations will usually be provided by a runtime support library such as cloudflow-akka, cloudflow-spark and cloudflow-flink.
- trait StreamletShape extends AnyRef
- final case class StringConfigParameter(key: String, description: String = "", defaultValue: Option[String] = None) extends ConfigParameter with Product with Serializable
- final case class Topic(id: String, config: Config) extends Product with Serializable
- case class TopicForPortNotFoundException(port: StreamletPort, streamletDefinition: StreamletDefinition) extends Exception with Product with Serializable
- sealed trait ValidationType extends AnyRef
- case class VolumeMount(name: String, path: String, accessMode: AccessMode) extends Product with Serializable
Value Members
- object AkkaClusterAttribute extends StreamletAttribute with Product with Serializable
-
object
BooleanConfigParameter extends Serializable
Describes a boolean configuration parameter.
- object BooleanValidationType extends ValidationType with Product with Serializable
- object BootstrapInfo
- object ClassOps
-
object
DoubleConfigParameter extends Serializable
Describes a 64 bit floating point number.
- object DoubleValidationType extends ValidationType with Product with Serializable
- object Dun extends Dun with Product with Serializable
-
object
DurationConfigParameter extends Serializable
Describes a time duration as defined by https://github.com/lightbend/config/blob/master/HOCON.md#duration-format
Describes a time duration as defined by https://github.com/lightbend/config/blob/master/HOCON.md#duration-format
The following units can be used with DurationConfigParameter
ns, nano, nanos, nanosecond, nanoseconds us, micro, micros, microsecond, microseconds ms, milli, millis, millisecond, milliseconds s, second, seconds m, minute, minutes h, hour, hours d, day, days
Example on how to extract a duration value using config:
val parameter = DurationDurationConfigParameter("TTL","Time to live of packets sent out","2 m") ... val duration = config.getDuration(parameter.key)
- object DurationValidationType extends ValidationType with Product with Serializable
-
object
IntegerConfigParameter extends Serializable
Describes an integer configuration parameter.
- object IntegerValidationType extends ValidationType with Product with Serializable
-
object
MemorySizeConfigParameter extends Serializable
Describes a memory quantity as defined by https://github.com/lightbend/config/blob/master/HOCON.md#size-in-bytes-format
Describes a memory quantity as defined by https://github.com/lightbend/config/blob/master/HOCON.md#size-in-bytes-format
The following units can be used with MemorySizeConfigParameter
"B", "b", "byte", "bytes", "kB", "kilobyte", "kilobytes", "MB", "megabyte", "megabytes", "GB", "gigabyte", "gigabytes", "TB", "terabyte", "terabytes", "PB", "petabyte", "petabytes", "EB", "exabyte", "exabytes", "ZB", "zettabyte", "zettabytes", "YB", "yottabyte", "yottabytes", "K", "k", "Ki", "KiB", "kibibyte", "kibibytes", "M", "m", "Mi", "MiB", "mebibyte", "mebibytes", "G", "g", "Gi", "GiB", "gibibyte", "gibibytes", "T", "t", "Ti", "TiB", "tebibyte", "tebibytes", "P", "p", "Pi", "PiB", "pebibyte", "pebibytes", "E", "e", "Ei", "EiB", "exbibyte", "exbibytes", "Z", "z", "Zi", "ZiB", "zebibyte", "zebibytes", "Y", "y", "Yi", "YiB", "yobibyte", "yobibytes",
Example on how to extract a duration value using config:
val parameter = MemorySizeMemorySizeConfigParameter("max-size-of-log","Maximum size of log before wrapping","8M") ... val memorySize = config.getMemorySize(parameter.key)
- object MemorySizeValidationType extends ValidationType with Product with Serializable
- object ReadOnlyMany extends AccessMode with Product with Serializable
- object ReadWriteMany extends AccessMode with Product with Serializable
-
object
RegExpConfigParameter extends Serializable
Describes a parameter that can be used for custom validation.
Describes a parameter that can be used for custom validation.
Note: - This is a limited set of regular expressions compared to regular expressions supported by Java/Scala. - These regular expressions are only used for validation so capture groups are not supported.
The type of regular expressions that can be used is described in: https://github.com/google/re2/wiki/Syntax
Example:
val durationParameter = RegExpConfigParameter("window-duration","The duration of a sampling window", """((\d{1,2}h\s?)?(\d{1,2}m\s?)?(\d{1,2}s\s?)?)|\d{1,2}""",Some("1m 20s"))
-
object
RoundRobinPartitioner extends (Any) ⇒ String with Serializable
A round-robin partitioning function.
A round-robin partitioning function. Elements written to a CodecOutlet that uses this partitioner will be distributed in round-robin fashion across the topic partitions.
-
object
ServerAttribute extends StreamletAttribute with Product with Serializable
Describes that a Streamlet is a server, which will want to listen on a container port.
-
object
StreamletContextDataJsonSupport extends DefaultJsonProtocol
Helper object for creating an instance of StreamletContextData from JSON.
- object StreamletDefinition extends Serializable
- object StreamletShape
-
object
StringConfigParameter extends Serializable
Describes a UTF8 string
- object Topic extends Serializable
- object VolumeMount extends Serializable