Class

com.coxautodata.waimak.azure.table

AzureTableMultiWriter

Related Doc: package table

Permalink

class AzureTableMultiWriter extends Logging

Azure tables are accessed using HTTP/HTTPS, with maximum batch size of 100 records per Azure Table partition key. Therefore there is a lot of lag between the batches and spending one spark executor per connection will be very wasteful. To work around it, this class will open multiple threads each having its own connection to an Azure Table and will push data from the blocking queue.

Created by Alexei Perelighin on 2018/03/01

Linear Supertypes
Logging, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. AzureTableMultiWriter
  2. Logging
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new AzureTableMultiWriter(table: String, connection: String, threadNum: Int, timeoutMs: Long, retryDelayMs: Seq[Long])

    Permalink

    table

    - Azure Table name

    connection

    - Azure Table connection string

    threadNum

    - Number of threads/connections.

    timeoutMs

    - Timeout in milliseconds for an Azure Table operation

    retryDelayMs

    - Delay in milliseconds to wait between successive retries of an Azure Table operation in case of failure or timeout. Number of elements in the sequence indicates how number of retries for the operation.

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  5. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  6. val connection: String

    Permalink

    - Azure Table connection string

  7. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  8. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  9. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  10. def finish: Int

    Permalink

    When there will be no more data pushed into the queue, producer must call it.

    When there will be no more data pushed into the queue, producer must call it. It will wait for the threads to push all of the pages into the Azure Table and for all threads to finish.

    returns

    - number of records pushed by all threads

  11. var futures: Seq[Future[Int]]

    Permalink
    Attributes
    protected
  12. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
  13. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  14. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  15. def isTraceEnabled(): Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  16. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  17. def logDebug(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  18. def logError(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  19. def logError(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  20. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  21. def logInfo(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  22. def logName: String

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  23. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  24. def logTrace(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  25. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  26. def logWarning(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  27. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  28. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
  29. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
  30. val queue: BlockingQueue[Seq[DynamicTableEntity]]

    Permalink

    Multiple data producers will push pages of at most 100 records into the queue.

    Multiple data producers will push pages of at most 100 records into the queue. There will be no validation of the pages size by this class, so code that produces pages must take care of that.

  31. def retry[T](f: () ⇒ Future[T], timeoutMs: Long, delaysMs: Seq[Long]): T

    Permalink

    T

    - Return type of the future

    f

    - Function that produces a Future

    timeoutMs

    - Timeout in milliseconds for the future to complete

    delaysMs

    - Delay in milliseconds to wait between successive retries of the future in case of failure or timeout. Number of elements in the sequence indicates how number of retries for the future.

  32. val retryDelayMs: Seq[Long]

    Permalink

    - Delay in milliseconds to wait between successive retries of an Azure Table operation in case of failure or timeout.

    - Delay in milliseconds to wait between successive retries of an Azure Table operation in case of failure or timeout. Number of elements in the sequence indicates how number of retries for the operation.

  33. def run(): Unit

    Permalink

    Initialises the writer threads, which will wait for pages of records to be placed into the queue.

    Initialises the writer threads, which will wait for pages of records to be placed into the queue.

    This method does not wait for completion and does not block, exits immediately after initialisation.

  34. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  35. val table: String

    Permalink

    - Azure Table name

  36. val thereWillBeMoreData: AtomicBoolean

    Permalink
    Attributes
    protected
  37. val threadFailed: AtomicBoolean

    Permalink
  38. val threadNum: Int

    Permalink

    - Number of threads/connections.

  39. val threadPool: ExecutorService

    Permalink
  40. val timeoutMs: Long

    Permalink

    - Timeout in milliseconds for an Azure Table operation

  41. def toString(): String

    Permalink
    Definition Classes
    AnyRef → Any
  42. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  43. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  44. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from Logging

Inherited from AnyRef

Inherited from Any

Ungrouped