package streamlets
- Alphabetic
- Public
- Protected
Type Members
- sealed trait AccessMode extends AnyRef
- final case class BooleanConfigParameter(key: String, description: String = "", defaultValue: Option[Boolean] = None) extends ConfigParameter with Product with Serializable
- case class BootstrapServersForTopicNotFound(topic: Topic) extends Exception with Product with Serializable
- trait Codec[T] extends Serializable
- Annotations
- @implicitNotFound("Cannot find Codec typeclass for ${T}")
- trait CodecInlet[T] extends Inlet
A handle to read and deserialize data into elements of type
T
. - trait CodecOutlet[T] extends Outlet
A handle to serialize elements of type
T
into a partitioned stream. - trait ConfigParameter extends AnyRef
Describes arguments that have to be supplied when deploying a Cloudflow application.
Describes arguments that have to be supplied when deploying a Cloudflow application. Each streamlet in an application can require zero or more arguments. The configuration parameter contains information for the end user entering the values and logic to validate those values so they are entered correctly.
Streamlet developers can create their own configuration parameters by deriving from ConfigParameter, the parameter can use validation logic from a fixed set of types derived from ValidationType.
The developer can also re-use any of the following predefined configuration parameter types:
- IntegerConfigParameter
- StringConfigParameter
- DoubleConfigParameter
- BooleanConfigParameter
- RegExpConfigParameter
- DurationConfigParameter
- MemorySizeConfigParameter
Each of these pre-defined parameters uses one of the following validation types:
- BooleanValidationType
- IntegerValidationType
- DoubleValidationType
- RegexpValidationType
- DurationValidationType
- MemorySizeValidationType
Example of how to create a custom configuration parameter:
final case class MilitaryTimeParameter(key: String, defaultValue: Option[String] = None) extends ConfigParameter[String] { val description: String = "This parameter validates that the users enter the time in 24h format." val validation = RegexpValidation("^(0[0-9]|1[0-9]|2[0-3]|[0-9]):[0-5][0-9]$") def toDescriptor = ConfigParameterDescriptor(key,description,validation,defaultValue) }
Example on how to use a configuration parameter in a streamlet:
class RecordSumFlow extends AkkaStreamlet { val recordsInWindowParameter = IntegerConfigParameter("records-in-window","This value describes how many records of data should be processed together, default 65 KB", Some(64 * 1024)) override def configParameters = Set(recordsInWindowParameter) val inlet = AvroInlet[Metric]("metric") val outlet = AvroOutlet[SummedMetric]("summed-metric") val shape = StreamletShape.withInlets(inlet).withOutlets(outlet) def createLogic = new RunnableGraphStreamletLogic() { val recordsInWindow = streamletConfig.getInt(recordsInWindowParameter.key) def sumRecords(records: Seq[Metric]) : SummedMetric {....} def flow = FlowWithOffsetContext[Metric].grouped(recordsInWindow).map(sumRecords) def runnableGraph() = atLeastOnceSource(inlet).via(flow).to(atLeastOnceSink(outlet)) } }
- final case class DoubleConfigParameter(key: String, description: String = "", defaultValue: Option[Double] = None) extends ConfigParameter with Product with Serializable
- sealed abstract class Dun extends Serializable
Typically used together with
Future
to signal completion but there is no actual value completed.Typically used together with
Future
to signal completion but there is no actual value completed. More clearly signals intent thanUnit
and is available both from Scala and Java (whichUnit
is not).Adopted from akka to avoid
streamlets
depending on akka. - case class DurationConfigParameter(key: String, description: String = "", defaultValue: Option[String] = None) extends ConfigParameter with Product with Serializable
- final case class ExceptionAcc(exceptions: Vector[Throwable]) extends Exception with Product with Serializable
An exception to return when the runner returns an accumulated list of distinct exceptions.
- trait Inlet extends StreamletPort
A handle to read data according to a schema.
- final case class IntegerConfigParameter(key: String, description: String = "", defaultValue: Option[Int] = None) extends ConfigParameter with Product with Serializable
- case class LoadedStreamlet(streamlet: Streamlet[StreamletContext], config: StreamletDefinition) extends Product with Serializable
- final case class MemorySizeConfigParameter(key: String, description: String = "", defaultValue: Option[String] = None) extends ConfigParameter with Product with Serializable
- trait Outlet extends StreamletPort
A handle to write data according to a schema.
- case class PortMapping(port: String, topic: Topic) extends Product with Serializable
Mapping between the port name and the topic
- final case class RegExpConfigParameter(key: String, description: String, pattern: String, defaultValue: Option[String] = None) extends ConfigParameter with Product with Serializable
- final case class RegexpValidationType(regExpPattern: String) extends ValidationType with Product with Serializable
- final case class SchemaDefinition(name: String, schema: String, fingerprint: String, format: String) extends Product with Serializable
Describes the schema.
Describes the schema.
name
should be the unique name of the schema.schema
is a string representation of the schema itself. (In the case of the avro format this is a json document) Thefingerprint
is a consistent hash of the schema. Theformat
specifies the format of the schema. Unique names should be used for different formats. (In the case of Avro, format is "avro") - abstract class Streamlet[Context <: StreamletContext] extends AnyRef
- trait StreamletAttribute extends AnyRef
Describes that a streamlet requires specific configuration by the cloudflow platform.
Describes that a streamlet requires specific configuration by the cloudflow platform.
The configuration value will be added by the platform under the
configPath
path in the Streamlet configuration. - trait StreamletContext extends AnyRef
- case class StreamletContextData(appId: String, appVersion: String, portMappings: Map[String, Topic], volumeMounts: Option[List[VolumeMount]] = None, config: Config) extends Product with Serializable
- class StreamletContextException extends Exception
- case class StreamletDefinition(appId: String, appVersion: String, streamletRef: String, streamletClass: String, portMappings: List[PortMapping], volumeMounts: List[VolumeMount], config: Config) extends Product with Serializable
- trait StreamletExecution extends AnyRef
A handle to the running Streamlet.
- trait StreamletLoader extends AnyRef
Functions to load a streamlet from its configuration through reflection.
- trait StreamletLogic[Context <: StreamletContext] extends Serializable
- trait StreamletPort extends AnyRef
A named port handle handle to read or write data according to a schema.
- trait StreamletRuntime extends AnyRef
A simple marker trait to provide the name of the "runtime" supported by a streamlet, e.g.
A simple marker trait to provide the name of the "runtime" supported by a streamlet, e.g. "akka", "spark", etc.
Implementations will usually be provided by a runtime support library such as cloudflow-akka, cloudflow-spark and cloudflow-flink.
- trait StreamletShape extends AnyRef
- final case class StringConfigParameter(key: String, description: String = "", defaultValue: Option[String] = None) extends ConfigParameter with Product with Serializable
- final case class Topic(id: String, config: Config) extends Product with Serializable
- case class TopicForPortNotFoundException(port: StreamletPort, streamletDefinition: StreamletDefinition) extends Exception with Product with Serializable
- sealed trait ValidationType extends AnyRef
- case class VolumeMount(name: String, path: String, accessMode: AccessMode) extends Product with Serializable
Value Members
- case object AkkaClusterAttribute extends StreamletAttribute with Product with Serializable
- object BooleanConfigParameter extends Serializable
Describes a boolean configuration parameter.
- case object BooleanValidationType extends ValidationType with Product with Serializable
- object BootstrapInfo
- object ClassOps
- object CodecInlet
- object DoubleConfigParameter extends Serializable
Describes a 64 bit floating point number.
- case object DoubleValidationType extends ValidationType with Product with Serializable
- case object Dun extends Dun with Product with Serializable
- object DurationConfigParameter extends Serializable
Describes a time duration as defined by https://github.com/lightbend/config/blob/master/HOCON.md#duration-format
Describes a time duration as defined by https://github.com/lightbend/config/blob/master/HOCON.md#duration-format
The following units can be used with DurationConfigParameter
ns, nano, nanos, nanosecond, nanoseconds us, micro, micros, microsecond, microseconds ms, milli, millis, millisecond, milliseconds s, second, seconds m, minute, minutes h, hour, hours d, day, days
Example on how to extract a duration value using config:
val parameter = DurationDurationConfigParameter("TTL","Time to live of packets sent out","2 m") ... val duration = config.getDuration(parameter.key)
- case object DurationValidationType extends ValidationType with Product with Serializable
- object IntegerConfigParameter extends Serializable
Describes an integer configuration parameter.
- case object IntegerValidationType extends ValidationType with Product with Serializable
- object MemorySizeConfigParameter extends Serializable
Describes a memory quantity as defined by https://github.com/lightbend/config/blob/master/HOCON.md#size-in-bytes-format
Describes a memory quantity as defined by https://github.com/lightbend/config/blob/master/HOCON.md#size-in-bytes-format
The following units can be used with MemorySizeConfigParameter
"B", "b", "byte", "bytes", "kB", "kilobyte", "kilobytes", "MB", "megabyte", "megabytes", "GB", "gigabyte", "gigabytes", "TB", "terabyte", "terabytes", "PB", "petabyte", "petabytes", "EB", "exabyte", "exabytes", "ZB", "zettabyte", "zettabytes", "YB", "yottabyte", "yottabytes", "K", "k", "Ki", "KiB", "kibibyte", "kibibytes", "M", "m", "Mi", "MiB", "mebibyte", "mebibytes", "G", "g", "Gi", "GiB", "gibibyte", "gibibytes", "T", "t", "Ti", "TiB", "tebibyte", "tebibytes", "P", "p", "Pi", "PiB", "pebibyte", "pebibytes", "E", "e", "Ei", "EiB", "exbibyte", "exbibytes", "Z", "z", "Zi", "ZiB", "zebibyte", "zebibytes", "Y", "y", "Yi", "YiB", "yobibyte", "yobibytes",
Example on how to extract a duration value using config:
val parameter = MemorySizeMemorySizeConfigParameter("max-size-of-log","Maximum size of log before wrapping","8M") ... val memorySize = config.getMemorySize(parameter.key)
- case object MemorySizeValidationType extends ValidationType with Product with Serializable
- case object ReadOnlyMany extends AccessMode with Product with Serializable
- case object ReadWriteMany extends AccessMode with Product with Serializable
- object RegExpConfigParameter extends Serializable
Describes a parameter that can be used for custom validation.
Describes a parameter that can be used for custom validation.
Note: - This is a limited set of regular expressions compared to regular expressions supported by Java/Scala. - These regular expressions are only used for validation so capture groups are not supported.
The type of regular expressions that can be used is described in: https://github.com/google/re2/wiki/Syntax
Example:
val durationParameter = RegExpConfigParameter("window-duration","The duration of a sampling window", """((\d{1,2}h\s?)?(\d{1,2}m\s?)?(\d{1,2}s\s?)?)|\d{1,2}""",Some("1m 20s"))
- object RoundRobinPartitioner extends (Any) => String with Serializable
A round-robin partitioning function.
A round-robin partitioning function. Elements written to a CodecOutlet that uses this partitioner will be distributed in round-robin fashion across the topic partitions.
- case object ServerAttribute extends StreamletAttribute with Product with Serializable
Describes that a Streamlet is a server, which will want to listen on a container port.
- object StreamletContextDataJsonSupport extends DefaultJsonProtocol
Helper object for creating an instance of StreamletContextData from JSON.
- object StreamletDefinition extends Serializable
- object StreamletShape
- object StringConfigParameter extends Serializable
Describes a UTF8 string
- object Topic extends Serializable
- object VolumeMount extends Serializable