Create loghub DStream.
Create loghub DStream.
StreamingContext.
The name of LogService
project.
The name of logStore.
The group name of loghub consumer. All consumer process which has the same group name will consumer specific logStore together.
The endpoint of loghub.
The number of receivers.
The Aliyun Access Key Id.
The Aliyun Access Key Secret.
Storage level to use for storing the received objects.
Set user defined cursor type.
Set user defined cursor position (Unix Timestamp).
Whether to force to set consume position as the
mLoghubCursorStartTime
.
Create loghub DStream.
Create loghub DStream.
StreamingContext.
The name of LogService
project.
The name of logStore.
The group name of loghub consumer. All consumer process which has the same group name will consumer specific logStore together.
The endpoint of loghub.
The Aliyun Access Key Id.
The Aliyun Access Key Secret.
Storage level to use for storing the received objects.
Set user defined cursor type.
Set user defined cursor position (Unix Timestamp).
Whether to force to set consume position as the
mLoghubCursorStartTime
.
Create loghub DStream.
Create loghub DStream.
val loghubProject = "sample-project" val logStream = "sample-logstore" val loghubGroupName = "sample-group" val endpoint = "cn-hangzhou-intranet.sls.aliyuncs.com" val accessKeyId = "kj7aY*******UYx6" val accessKeySecret = "AiNMAlxz*************1PxaPaL8t" val numReceivers = 2 val batchInterval = Milliseconds(5 * 1000) val conf = new SparkConf().setAppName("Test Loghub") val ssc = new StreamingContext(conf, batchInterval) val loghubStream = LoghubUtils.createStream( ssc, loghubProject, logStream, loghubGroupName, endpoint, numReceivers, accessKeyId, accessKeySecret, StorageLevel.MEMORY_AND_DISK)
StreamingContext.
The name of LogService
project.
The name of logStore.
The group name of loghub consumer. All consumer process which has the same group name will consumer specific logStore together.
The endpoint of loghub.
The number of receivers.
The Aliyun Access Key Id.
The Aliyun Access Key Secret.
Storage level to use for storing the received objects.
val loghubProject = "sample-project" val logStream = "sample-logstore" val loghubGroupName = "sample-group" val endpoint = "cn-hangzhou-intranet.sls.aliyuncs.com" val accessKeyId = "kj7aY*******UYx6" val accessKeySecret = "AiNMAlxz*************1PxaPaL8t" val batchInterval = Milliseconds(5 * 1000) val conf = new SparkConf().setAppName("Test Loghub") val ssc = new StreamingContext(conf, batchInterval) val loghubStream = LoghubUtils.createStream( ssc, loghubProject, logStream, loghubGroupName, endpoint, accessKeyId, accessKeySecret, StorageLevel.MEMORY_AND_DISK)
StreamingContext.
The name of LogService
project.
The name of logStore.
The group name of loghub consumer. All consumer process which has the same group name will consumer specific logStore together.
The endpoint of loghub.
The Aliyun Access Key Id.
The Aliyun Access Key Secret.
Storage level to use for storing the received objects.
Various utility classes for working with Aliyun LogService.