Object

org.apache.spark.streaming.aliyun.logservice

LoghubUtils

Related Doc: package logservice

Permalink

object LoghubUtils

Various utility classes for working with Aliyun LogService.

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. LoghubUtils
  2. AnyRef
  3. Any
  1. Hide All
  2. Show all
Visibility
  1. Public
  2. All

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  5. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  6. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  7. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  8. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  9. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
  10. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  11. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  12. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  13. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
  14. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
  15. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  16. def toString(): String

    Permalink
    Definition Classes
    AnyRef → Any
  17. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  18. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  19. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Deprecated Value Members

  1. def createDirectStream(jssc: JavaStreamingContext, project: String, logStore: String, mConsumerGroup: String, accessKeyId: String, accessKeySecret: String, endpoint: String, zkParams: HashMap[String, String], mode: LogHubCursorPosition, cursorStartTime: Long): JavaInputDStream[String]

    Permalink

    Create direct loghub DStream.

    Create direct loghub DStream.

    Set cursorStartTime a valid value when using LogHubCursorPosition.SPECIAL_TIMER_CURSOR mode at the first time with current mConsumerGroup

    jssc

    StreamingContext.

    project

    The name of LogService project.

    logStore

    The name of logStore.

    mConsumerGroup

    The group name of loghub consumer. All consumer process which has the same group name will consumer specific logStore together.

    accessKeyId

    The Aliyun Access Key Id.

    accessKeySecret

    The Aliyun Access Key Secret.

    endpoint

    The endpoint of loghub.

    zkParams

    Zookeeper parameters.

    mode

    Set user defined cursor type.

    cursorStartTime

    Set user defined cursor position (Unix Timestamp).

    Annotations
    @Experimental() @deprecated
    Deprecated

    (Since version 1.8.0)

  2. def createDirectStream(jssc: JavaStreamingContext, project: String, logStore: String, mConsumerGroup: String, accessKeyId: String, accessKeySecret: String, endpoint: String, zkParams: HashMap[String, String], mode: LogHubCursorPosition): JavaInputDStream[String]

    Permalink

    Create direct loghub DStream.

    Create direct loghub DStream.

    jssc

    JavaStreamingContext.

    project

    The name of LogService project.

    logStore

    The name of logStore.

    mConsumerGroup

    The group name of loghub consumer. All consumer process which has the same group name will consumer specific logStore together.

    accessKeyId

    The Aliyun Access Key Id.

    accessKeySecret

    The Aliyun Access Key Secret.

    endpoint

    The endpoint of loghub.

    zkParams

    Zookeeper parameters.

    mode

    Set user defined cursor type.

    Annotations
    @Experimental() @deprecated
    Deprecated

    (Since version 1.8.0)

  3. def createDirectStream(ssc: StreamingContext, project: String, logStore: String, mConsumerGroup: String, accessKeyId: String, accessKeySecret: String, endpoint: String, zkParams: Map[String, String], mode: LogHubCursorPosition, cursorStartTime: Long): DStream[String]

    Permalink

    Create direct loghub DStream.

    Create direct loghub DStream.

    Set cursorStartTime a valid value when using LogHubCursorPosition.SPECIAL_TIMER_CURSOR mode at the first time with current mConsumerGroup

    ssc

    StreamingContext.

    project

    The name of LogService project.

    logStore

    The name of logStore.

    mConsumerGroup

    The group name of loghub consumer. All consumer process which has the same group name will consumer specific logStore together.

    accessKeyId

    The Aliyun Access Key Id.

    accessKeySecret

    The Aliyun Access Key Secret.

    endpoint

    The endpoint of loghub.

    zkParams

    Zookeeper parameters.

    mode

    Set user defined cursor type.

    cursorStartTime

    Set user defined cursor position (Unix Timestamp).

    Annotations
    @Experimental() @deprecated
    Deprecated

    (Since version 1.8.0)

  4. def createDirectStream(ssc: StreamingContext, project: String, logStore: String, mConsumerGroup: String, accessKeyId: String, accessKeySecret: String, endpoint: String, zkParams: Map[String, String], mode: LogHubCursorPosition): DStream[String]

    Permalink

    Create direct loghub DStream.

    Create direct loghub DStream.

    ssc

    StreamingContext.

    project

    The name of LogService project.

    logStore

    The name of logStore.

    mConsumerGroup

    The group name of loghub consumer. All consumer process which has the same group name will consumer specific logStore together.

    accessKeyId

    The Aliyun Access Key Id.

    accessKeySecret

    The Aliyun Access Key Secret.

    endpoint

    The endpoint of loghub.

    zkParams

    Zookeeper parameters.

    mode

    Set user defined cursor type.

    Annotations
    @Experimental() @deprecated
    Deprecated

    (Since version 1.8.0)

  5. def createRDD(sc: SparkContext, project: String, logStore: String, accessId: String, accessKey: String, endpoint: String, startTime: Long, endTime: Long = 1, parallelismInShard: Int = 1): RDD[String]

    Permalink
    Annotations
    @deprecated
    Deprecated

    (Since version 1.8.0)

  6. def createStream(jssc: JavaStreamingContext, logServiceProject: String, logStoreName: String, loghubConsumerGroupName: String, numReceivers: Int, storageLevel: StorageLevel, cursorPosition: LogHubCursorPosition, mLoghubCursorStartTime: Int, forceSpecial: Boolean): JavaDStream[Array[Byte]]

    Permalink

    Create loghub DStream.

    Create loghub DStream.

    jssc

    JavaStreamingContext.

    logServiceProject

    The name of LogService project.

    logStoreName

    The name of logStore.

    loghubConsumerGroupName

    The group name of loghub consumer. All consumer process which has the same group name will consumer specific logStore together.

    numReceivers

    The number of receivers.

    storageLevel

    Storage level to use for storing the received objects.

    cursorPosition

    Set user defined cursor type.

    mLoghubCursorStartTime

    Set user defined cursor position (Unix Timestamp).

    forceSpecial

    Whether to force to set consume position as the mLoghubCursorStartTime.

    Annotations
    @deprecated
    Deprecated

    (Since version 1.8.0)

  7. def createStream(jssc: JavaStreamingContext, logServiceProject: String, logStoreName: String, loghubConsumerGroupName: String, loghubEndpoint: String, numReceivers: Int, accessKeyId: String, accessKeySecret: String, storageLevel: StorageLevel, cursorPosition: LogHubCursorPosition, mLoghubCursorStartTime: Int, forceSpecial: Boolean): JavaDStream[Array[Byte]]

    Permalink

    Create loghub DStream.

    Create loghub DStream.

    jssc

    JavaStreamingContext.

    logServiceProject

    The name of LogService project.

    logStoreName

    The name of logStore.

    loghubConsumerGroupName

    The group name of loghub consumer. All consumer process which has the same group name will consumer specific logStore together.

    loghubEndpoint

    The endpoint of loghub.

    numReceivers

    The number of receivers.

    accessKeyId

    The Aliyun Access Key Id.

    accessKeySecret

    The Aliyun Access Key Secret.

    storageLevel

    Storage level to use for storing the received objects.

    cursorPosition

    Set user defined cursor type.

    mLoghubCursorStartTime

    Set user defined cursor position (Unix Timestamp).

    forceSpecial

    Whether to force to set consume position as the mLoghubCursorStartTime.

    Annotations
    @deprecated
    Deprecated

    (Since version 1.8.0)

  8. def createStream(jssc: JavaStreamingContext, logServiceProject: String, logStoreName: String, loghubConsumerGroupName: String, storageLevel: StorageLevel, cursorPosition: LogHubCursorPosition, mLoghubCursorStartTime: Int, forceSpecial: Boolean): JavaDStream[Array[Byte]]

    Permalink

    Create loghub DStream.

    Create loghub DStream.

    jssc

    JavaStreamingContext.

    logServiceProject

    The name of LogService project.

    logStoreName

    The name of logStore.

    loghubConsumerGroupName

    The group name of loghub consumer. All consumer process which has the same group name will consumer specific logStore together.

    storageLevel

    Storage level to use for storing the received objects.

    cursorPosition

    Set user defined cursor type.

    mLoghubCursorStartTime

    Set user defined cursor position (Unix Timestamp).

    forceSpecial

    Whether to force to set consume position as the mLoghubCursorStartTime.

    Annotations
    @deprecated
    Deprecated

    (Since version 1.8.0)

  9. def createStream(jssc: JavaStreamingContext, logServiceProject: String, logStoreName: String, loghubConsumerGroupName: String, loghubEndpoint: String, accessKeyId: String, accessKeySecret: String, storageLevel: StorageLevel, cursorPosition: LogHubCursorPosition, mLoghubCursorStartTime: Int, forceSpecial: Boolean): JavaDStream[Array[Byte]]

    Permalink

    Create loghub DStream.

    Create loghub DStream.

    jssc

    JavaStreamingContext.

    logServiceProject

    The name of LogService project.

    logStoreName

    The name of logStore.

    loghubConsumerGroupName

    The group name of loghub consumer. All consumer process which has the same group name will consumer specific logStore together.

    loghubEndpoint

    The endpoint of loghub.

    accessKeyId

    The Aliyun Access Key Id.

    accessKeySecret

    The Aliyun Access Key Secret.

    storageLevel

    Storage level to use for storing the received objects.

    cursorPosition

    Set user defined cursor type.

    mLoghubCursorStartTime

    Set user defined cursor position (Unix Timestamp).

    forceSpecial

    Whether to force to set consume position as the mLoghubCursorStartTime.

    Annotations
    @deprecated
    Deprecated

    (Since version 1.8.0)

  10. def createStream(jssc: JavaStreamingContext, logServiceProject: String, logStoreName: String, loghubConsumerGroupName: String, numReceivers: Int, storageLevel: StorageLevel): JavaDStream[Array[Byte]]

    Permalink

    Create loghub DStream.

    Create loghub DStream.

    jssc

    JavaStreamingContext.

    logServiceProject

    The name of LogService project.

    logStoreName

    The name of logStore.

    loghubConsumerGroupName

    The group name of loghub consumer. All consumer process which has the same group name will consumer specific logStore together.

    numReceivers

    The number of receivers.

    storageLevel

    Storage level to use for storing the received objects.

    Annotations
    @deprecated
    Deprecated

    (Since version 1.8.0)

  11. def createStream(jssc: JavaStreamingContext, logServiceProject: String, logStoreName: String, loghubConsumerGroupName: String, loghubEndpoint: String, numReceivers: Int, accessKeyId: String, accessKeySecret: String, storageLevel: StorageLevel): JavaDStream[Array[Byte]]

    Permalink

    Create loghub DStream.

    Create loghub DStream.

    jssc

    JavaStreamingContext.

    logServiceProject

    The name of LogService project.

    logStoreName

    The name of logStore.

    loghubConsumerGroupName

    The group name of loghub consumer. All consumer process which has the same group name will consumer specific logStore together.

    loghubEndpoint

    The endpoint of loghub.

    numReceivers

    The number of receivers.

    accessKeyId

    The Aliyun Access Key Id.

    accessKeySecret

    The Aliyun Access Key Secret.

    storageLevel

    Storage level to use for storing the received objects.

    Annotations
    @deprecated
    Deprecated

    (Since version 1.8.0)

  12. def createStream(jssc: JavaStreamingContext, logServiceProject: String, logStoreName: String, loghubConsumerGroupName: String, storageLevel: StorageLevel): JavaReceiverInputDStream[Array[Byte]]

    Permalink

    Create loghub DStream.

    Create loghub DStream.

    jssc

    JavaStreamingContext.

    logServiceProject

    The name of LogService project.

    logStoreName

    The name of logStore.

    loghubConsumerGroupName

    The group name of loghub consumer. All consumer process which has the same group name will consumer specific logStore together.

    storageLevel

    Storage level to use for storing the received objects.

    Annotations
    @deprecated
    Deprecated

    (Since version 1.8.0)

  13. def createStream(jssc: JavaStreamingContext, logServiceProject: String, logStoreName: String, loghubConsumerGroupName: String, loghubEndpoint: String, accessKeyId: String, accessKeySecret: String, storageLevel: StorageLevel): JavaReceiverInputDStream[Array[Byte]]

    Permalink

    Create loghub DStream.

    Create loghub DStream.

    jssc

    JavaStreamingContext.

    logServiceProject

    The name of LogService project.

    logStoreName

    The name of logStore.

    loghubConsumerGroupName

    The group name of loghub consumer. All consumer process which has the same group name will consumer specific logStore together.

    loghubEndpoint

    The endpoint of loghub.

    accessKeyId

    The Aliyun Access Key Id.

    accessKeySecret

    The Aliyun Access Key Secret.

    storageLevel

    Storage level to use for storing the received objects.

    Annotations
    @deprecated
    Deprecated

    (Since version 1.8.0)

  14. def createStream(ssc: StreamingContext, logServiceProject: String, logStoreName: String, loghubConsumerGroupName: String, numReceivers: Int, storageLevel: StorageLevel, cursorPosition: LogHubCursorPosition, mLoghubCursorStartTime: Int, forceSpecial: Boolean): DStream[Array[Byte]]

    Permalink

    Create loghub DStream.

    Create loghub DStream.

    ssc

    StreamingContext.

    logServiceProject

    The name of LogService project.

    logStoreName

    The name of logStore.

    loghubConsumerGroupName

    The group name of loghub consumer. All consumer process which has the same group name will consumer specific logStore together.

    numReceivers

    The number of receivers.

    storageLevel

    Storage level to use for storing the received objects.

    cursorPosition

    Set user defined cursor type.

    mLoghubCursorStartTime

    Set user defined cursor position (Unix Timestamp).

    forceSpecial

    Whether to force to set consume position as the mLoghubCursorStartTime.

    Annotations
    @deprecated
    Deprecated

    (Since version 1.8.0)

  15. def createStream(ssc: StreamingContext, logServiceProject: String, logStoreName: String, loghubConsumerGroupName: String, loghubEndpoint: String, numReceivers: Int, accessKeyId: String, accessKeySecret: String, storageLevel: StorageLevel, cursorPosition: LogHubCursorPosition, mLoghubCursorStartTime: Int, forceSpecial: Boolean): DStream[Array[Byte]]

    Permalink

    Create loghub DStream.

    Create loghub DStream.

    ssc

    StreamingContext.

    logServiceProject

    The name of LogService project.

    logStoreName

    The name of logStore.

    loghubConsumerGroupName

    The group name of loghub consumer. All consumer process which has the same group name will consumer specific logStore together.

    loghubEndpoint

    The endpoint of loghub.

    numReceivers

    The number of receivers.

    accessKeyId

    The Aliyun Access Key Id.

    accessKeySecret

    The Aliyun Access Key Secret.

    storageLevel

    Storage level to use for storing the received objects.

    cursorPosition

    Set user defined cursor type.

    mLoghubCursorStartTime

    Set user defined cursor position (Unix Timestamp).

    forceSpecial

    Whether to force to set consume position as the mLoghubCursorStartTime.

    Annotations
    @deprecated
    Deprecated

    (Since version 1.8.0)

  16. def createStream(ssc: StreamingContext, logServiceProject: String, logStoreName: String, loghubConsumerGroupName: String, storageLevel: StorageLevel, cursorPosition: LogHubCursorPosition, mLoghubCursorStartTime: Int, forceSpecial: Boolean): ReceiverInputDStream[Array[Byte]]

    Permalink

    Create loghub DStream.

    Create loghub DStream.

    ssc

    StreamingContext.

    logServiceProject

    The name of LogService project.

    logStoreName

    The name of logStore.

    loghubConsumerGroupName

    The group name of loghub consumer. All consumer process which has the same group name will consumer specific logStore together.

    storageLevel

    Storage level to use for storing the received objects.

    cursorPosition

    Set user defined cursor type.

    mLoghubCursorStartTime

    Set user defined cursor position (Unix Timestamp).

    forceSpecial

    Whether to force to set consume position as the mLoghubCursorStartTime.

    Annotations
    @deprecated
    Deprecated

    (Since version 1.8.0)

  17. def createStream(ssc: StreamingContext, logServiceProject: String, logStoreName: String, loghubConsumerGroupName: String, loghubEndpoint: String, accessKeyId: String, accessKeySecret: String, storageLevel: StorageLevel, cursorPosition: LogHubCursorPosition, mLoghubCursorStartTime: Int, forceSpecial: Boolean): ReceiverInputDStream[Array[Byte]]

    Permalink

    Create loghub DStream.

    Create loghub DStream.

    ssc

    StreamingContext.

    logServiceProject

    The name of LogService project.

    logStoreName

    The name of logStore.

    loghubConsumerGroupName

    The group name of loghub consumer. All consumer process which has the same group name will consumer specific logStore together.

    loghubEndpoint

    The endpoint of loghub.

    accessKeyId

    The Aliyun Access Key Id.

    accessKeySecret

    The Aliyun Access Key Secret.

    storageLevel

    Storage level to use for storing the received objects.

    cursorPosition

    Set user defined cursor type.

    mLoghubCursorStartTime

    Set user defined cursor position (Unix Timestamp).

    forceSpecial

    Whether to force to set consume position as the mLoghubCursorStartTime.

    Annotations
    @deprecated
    Deprecated

    (Since version 1.8.0)

  18. def createStream(ssc: StreamingContext, logServiceProject: String, logStoreName: String, loghubConsumerGroupName: String, numReceivers: Int, storageLevel: StorageLevel): DStream[Array[Byte]]

    Permalink

    Create loghub DStream.

    Create loghub DStream.

    val loghubProject = "sample-project"
    val logStream = "sample-logstore"
    val loghubGroupName = "sample-group"
    val numReceivers = 2
    val batchInterval = Milliseconds(5 * 1000)
    
    val conf = new SparkConf().setAppName("Test Loghub")
    val ssc = new StreamingContext(conf, batchInterval)
    val loghubStream = LoghubUtils.createStream(
      ssc,
      loghubProject,
      logStream,
      loghubGroupName,
      numReceivers,
      StorageLevel.MEMORY_AND_DISK)
    ssc

    StreamingContext.

    logServiceProject

    The name of LogService project.

    logStoreName

    The name of logStore.

    loghubConsumerGroupName

    The group name of loghub consumer. All consumer process which has the same group name will consumer specific logStore together.

    numReceivers

    The number of receivers.

    storageLevel

    Storage level to use for storing the received objects.

    Annotations
    @deprecated
    Deprecated

    (Since version 1.8.0)

  19. def createStream(ssc: StreamingContext, logServiceProject: String, logStoreName: String, loghubConsumerGroupName: String, loghubEndpoint: String, numReceivers: Int, accessKeyId: String, accessKeySecret: String, storageLevel: StorageLevel): DStream[Array[Byte]]

    Permalink

    Create loghub DStream.

    Create loghub DStream.

    val loghubProject = "sample-project"
    val logStream = "sample-logstore"
    val loghubGroupName = "sample-group"
    val endpoint = "cn-hangzhou-intranet.sls.aliyuncs.com"
    val accessKeyId = "kj7aY*******UYx6"
    val accessKeySecret = "AiNMAlxz*************1PxaPaL8t"
    val numReceivers = 2
    val batchInterval = Milliseconds(5 * 1000)
    
    val conf = new SparkConf().setAppName("Test Loghub")
    val ssc = new StreamingContext(conf, batchInterval)
    val loghubStream = LoghubUtils.createStream(
      ssc,
      loghubProject,
      logStream,
      loghubGroupName,
      endpoint,
      numReceivers,
      accessKeyId,
      accessKeySecret,
      StorageLevel.MEMORY_AND_DISK)
    ssc

    StreamingContext.

    logServiceProject

    The name of LogService project.

    logStoreName

    The name of logStore.

    loghubConsumerGroupName

    The group name of loghub consumer. All consumer process which has the same group name will consumer specific logStore together.

    loghubEndpoint

    The endpoint of loghub.

    numReceivers

    The number of receivers.

    accessKeyId

    The Aliyun Access Key Id.

    accessKeySecret

    The Aliyun Access Key Secret.

    storageLevel

    Storage level to use for storing the received objects.

    Annotations
    @deprecated
    Deprecated

    (Since version 1.8.0)

  20. def createStream(ssc: StreamingContext, logServiceProject: String, logStoreName: String, loghubConsumerGroupName: String, storageLevel: StorageLevel): ReceiverInputDStream[Array[Byte]]

    Permalink

    val loghubProject = "sample-project"
    val logStream = "sample-logstore"
    val loghubGroupName = "sample-group"
    val batchInterval = Milliseconds(5 * 1000)
    
    val conf = new SparkConf().setAppName("Test Loghub")
    val ssc = new StreamingContext(conf, batchInterval)
    val loghubStream = LoghubUtils.createStream(
      ssc,
      loghubProject,
      logStream,
      loghubGroupName,
      StorageLevel.MEMORY_AND_DISK)
    ssc

    StreamingContext.

    logServiceProject

    The name of LogService project.

    logStoreName

    The name of logStore.

    loghubConsumerGroupName

    The group name of loghub consumer. All consumer process which has the same group name will consumer specific logStore together.

    storageLevel

    Storage level to use for storing the received objects.

    Annotations
    @deprecated
    Deprecated

    (Since version 1.8.0)

  21. def createStream(ssc: StreamingContext, logServiceProject: String, logStoreName: String, loghubConsumerGroupName: String, loghubEndpoint: String, accessKeyId: String, accessKeySecret: String, storageLevel: StorageLevel): ReceiverInputDStream[Array[Byte]]

    Permalink

    val loghubProject = "sample-project"
    val logStream = "sample-logstore"
    val loghubGroupName = "sample-group"
    val endpoint = "cn-hangzhou-intranet.sls.aliyuncs.com"
    val accessKeyId = "kj7aY*******UYx6"
    val accessKeySecret = "AiNMAlxz*************1PxaPaL8t"
    val batchInterval = Milliseconds(5 * 1000)
    
    val conf = new SparkConf().setAppName("Test Loghub")
    val ssc = new StreamingContext(conf, batchInterval)
    val loghubStream = LoghubUtils.createStream(
      ssc,
      loghubProject,
      logStream,
      loghubGroupName,
      endpoint,
      accessKeyId,
      accessKeySecret,
      StorageLevel.MEMORY_AND_DISK)
    ssc

    StreamingContext.

    logServiceProject

    The name of LogService project.

    logStoreName

    The name of logStore.

    loghubConsumerGroupName

    The group name of loghub consumer. All consumer process which has the same group name will consumer specific logStore together.

    loghubEndpoint

    The endpoint of loghub.

    accessKeyId

    The Aliyun Access Key Id.

    accessKeySecret

    The Aliyun Access Key Secret.

    storageLevel

    Storage level to use for storing the received objects.

    Annotations
    @deprecated
    Deprecated

    (Since version 1.8.0)

Inherited from AnyRef

Inherited from Any

Ungrouped