Create an input stream that pulls message from a Aliyun ONS stream.
Create an input stream that pulls message from a Aliyun ONS stream.
JavaStreamingContext jssc = ...; String cid = "ConsumerID"; String topic = "sample-topic"; String subExpression = "*"; String accessKeyId = "kj7aY*******UYx6"; String accessKeySecret = "AiNMAlxz*************1PxaPaL8t"; static class ReadMessage implements Function<Message, Byte[]> { @Override public Byte[] call(Message msg) { return msg.getBody; } JavaReceiverInputDStream<Byte[]> onsStream = OnsUtils.createStream( ssc, cId, topic, subExpression, accessKeyId, accessKeySecret, StorageLevel.MEMORY_AND_DISK_2, ReadMessage); onsStream.foreachRDD(rdd => { ... });
Java streamingContext object.
Name of a set of consumers.
Which topic to subscribe.
Which tag to subscribe.
Aliyun Access Key ID.
Aliyun Access Key Secret.
Storage level to use for storing the received objects. StorageLevel.MEMORY_AND_DISK_2 is recommended.
Extract information from ONS message.
Create an input stream that pulls message from a Aliyun ONS stream.
Create an input stream that pulls message from a Aliyun ONS stream.
val ssc: StreamingSparkContext = ... val cid = "ConsumerID" val topic = "sample-topic" val subExpression = "*" val accessKeyId = "kj7aY*******UYx6" val accessKeySecret = "AiNMAlxz*************1PxaPaL8t" def func: Message => Array[Byte] = msg => msg.getBody val onsStream = OnsUtils.createStream( ssc, cId, topic, subExpression, accessKeyId, accessKeySecret, StorageLevel.MEMORY_AND_DISK_2, func) onsStream.foreachRDD(rdd => { ... })
StreamingContext object.
Name of a set of consumers.
Which topic to subscribe.
Which tag to subscribe.
Aliyun Access Key ID.
Aliyun Access Key Secret.
Storage level to use for storing the received objects. StorageLevel.MEMORY_AND_DISK_2 is recommended.
Extract information from ONS message.
Create an input stream that pulls message from a Aliyun ONS stream.
Create an input stream that pulls message from a Aliyun ONS stream.
val ssc: StreamingSparkContext = ... val properties = new Properties() properties.setProperty("topic", topic) properties.setProperty("subExpression", tags) // more properties see [[com.aliyun.openservices.ons.api.PropertyKeyConst]] properties.setProperty("ConsumerId", consumerId) properties.setProperty("AccessKey", accessKeyId) properties.setProperty("SecretKey", accessKeySecret) // set for internet-region, and make sure the public network of spark cluster node is valid properties.setProperty("NAMESRV_ADDR", nameSrvAddr) def func: Message => Array[Byte] = msg => msg.getBody val onsStream = OnsUtils.createStream( ssc, properties, StorageLevel.MEMORY_AND_DISK_2, func) onsStream.foreachRDD(rdd => { ... })
StreamingContext object.
ons consumer properties.
Storage level to use for storing the received objects. StorageLevel.MEMORY_AND_DISK_2 is recommended.
Extract information from ONS message.
Create an union input stream that pulls message from a Aliyun ONS stream.
Create an union input stream that pulls message from a Aliyun ONS stream.
val ssc: StreamingSparkContext = ... val consumerIdTopicTags = Array(("ConsumerID1", "sample-topic1", "*")) val accessKeyId = "kj7aY*******UYx6" val accessKeySecret = "AiNMAlxz*************1PxaPaL8t" def func: Message => Array[Byte] = msg => msg.getBody val onsStream = OnsUtils.createStream( ssc, accessKeyId, accessKeySecret, StorageLevel.MEMORY_AND_DISK_2, func) onsStream.foreachRDD(rdd => { ... })
StreamingContext object.
Trituple(consumerId, topic, tag).
Aliyun Access Key ID.
Aliyun Access Key Secret.
Storage level to use for storing the received objects. StorageLevel.MEMORY_AND_DISK_2 is recommended.
Extract information from ONS message.
Various utility classes for working with Aliyun ONS.