Interface to implement custom source where data is read into the system.
Default Task container for org.apache.gearpump.streaming.source.DataSource that reads from DataSource in batch See org.apache.gearpump.streaming.source.DataSourceProcessor for its usage
Default Task container for org.apache.gearpump.streaming.source.DataSource that reads from DataSource in batch See org.apache.gearpump.streaming.source.DataSourceProcessor for its usage
DataSourceTask calls:
DataSource.open()
in onStart
and pass in
org.apache.gearpump.streaming.task.TaskContext
and application start timeDataSource.read()
in each onNext
, which reads a batch of messagesDataSource.close()
in onStop
TimeStampFilter filters out messages which have obsolete (smaller) timestamp.
Utility that helps user to create a DAG starting with DataSourceTask user should pass in a DataSource
Utility that helps user to create a DAG starting with DataSourceTask user should pass in a DataSource
Here is an example to build a DAG that reads from Kafka source followed by word count
val source = new KafkaSource() val sourceProcessor = DataSourceProcessor(source, 1) val split = Processor[Split](1) val sum = Processor[Sum](1) val dag = sourceProcessor ~> split ~> sum
Interface to implement custom source where data is read into the system. a DataSource could be a message queue like kafka or simply data generation source.
An example would be like
subclass is required to be serializable