SparkAzureTables

Value Members

1. final def !=(arg0: Any): Boolean

   

   Definition Classes

   AnyRef → Any

2. final def ##(): Int

   

   Definition Classes

   AnyRef → Any

3. final def ==(arg0: Any): Boolean

   

   Definition Classes

   AnyRef → Any

4. final def asInstanceOf[T0]: T0

   

   Definition Classes

   Any

5. def azureRepartition(inLabel: String)(outLabel: String): SparkDataFlow

   

   Azure Table batch operations allow only 100 records per batch per partition, it is not possible to mix records from different azure table partitions in one batch.

   Azure Table batch operations allow only 100 records per batch per partition, it is not possible to mix records from different azure table partitions in one batch. This will correctly repartition and sort the dataset for it to be pushed efficiently into Azure Table.

   The reorganised data set can be saved into storage and than read back for writing to azure table.

6. def clone(): AnyRef

   

   Attributes

   protected[java.lang]

   Definition Classes

   AnyRef

   Annotations

   @throws( ... )

7. final def eq(arg0: AnyRef): Boolean

   

   Definition Classes

   AnyRef

8. def equals(arg0: Any): Boolean

   

   Definition Classes

   AnyRef → Any

9. def finalize(): Unit

   

   Attributes

   protected[java.lang]

   Definition Classes

   AnyRef

   Annotations

   @throws( classOf[java.lang.Throwable] )

10. final def getClass(): Class[_]

    

    Definition Classes

    AnyRef → Any

11. def hashCode(): Int

    

    Definition Classes

    AnyRef → Any

12. final def isInstanceOf[T0]: Boolean

    

    Definition Classes

    Any

13. def isTraceEnabled(): Boolean

    

    Attributes

    protected

    Definition Classes

    Logging

14. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    

    Attributes

    protected

    Definition Classes

    Logging

15. def logDebug(msg: ⇒ String): Unit

    

    Attributes

    protected

    Definition Classes

    Logging

16. def logError(msg: ⇒ String, throwable: Throwable): Unit

    

    Attributes

    protected

    Definition Classes

    Logging

17. def logError(msg: ⇒ String): Unit

    

    Attributes

    protected

    Definition Classes

    Logging

18. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    

    Attributes

    protected

    Definition Classes

    Logging

19. def logInfo(msg: ⇒ String): Unit

    

    Attributes

    protected

    Definition Classes

    Logging

20. def logName: String

    

    Attributes

    protected

    Definition Classes

    Logging

21. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    

    Attributes

    protected

    Definition Classes

    Logging

22. def logTrace(msg: ⇒ String): Unit

    

    Attributes

    protected

    Definition Classes

    Logging

23. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    

    Attributes

    protected

    Definition Classes

    Logging

24. def logWarning(msg: ⇒ String): Unit

    

    Attributes

    protected

    Definition Classes

    Logging

25. val mandatoryFields: Set[String]

    

26. final def ne(arg0: AnyRef): Boolean

    

    Definition Classes

    AnyRef

27. final def notify(): Unit

    

    Definition Classes

    AnyRef

28. final def notifyAll(): Unit

    

    Definition Classes

    AnyRef

29. final def synchronized[T0](arg0: ⇒ T0): T0

    

    Definition Classes

    AnyRef

30. def toString(): String

    

    Definition Classes

    AnyRef → Any

31. def validateColumns(label: String, ds: Dataset[_]): Unit

    

    Attributes

    protected

32. final def wait(): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

33. final def wait(arg0: Long, arg1: Int): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

34. final def wait(arg0: Long): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

35. def writeAzureTable(connection: String, tableName: Option[String] = None, threadNum: Int = 50, timeoutMs: Long = 10000, retryDelayMs: Seq[Long] = Seq(1000, 2000, 4000, 8000))(label: String): SparkDataFlow

    

    Writes all records from the dataset behind a label into an Azure Table.

    Writes all records from the dataset behind a label into an Azure Table.

    connection

    connection string to Azure storage account

    tableName

    name of the azure table, if different to the label name. If None, then label name will be used for azure table name

    threadNum

    number of threads to push with from each executor. Default is 50.

    timeoutMs

    timeout in milliseconds for an Azure Table operation

    retryDelayMs

    delay in milliseconds to wait between successive retries of an Azure Table operation in case of failure or timeout. Number of elements in the sequence indicates how number of retries for the operation.

    label

    waimak label