- Azure Table name
- Azure Table connection string
- Number of threads/connections.
- Timeout in milliseconds for an Azure Table operation
- Delay in milliseconds to wait between successive retries of an Azure Table operation in case of failure or timeout. Number of elements in the sequence indicates how number of retries for the operation.
- Azure Table connection string
When there will be no more data pushed into the queue, producer must call it.
When there will be no more data pushed into the queue, producer must call it. It will wait for the threads to push all of the pages into the Azure Table and for all threads to finish.
- number of records pushed by all threads
Multiple data producers will push pages of at most 100 records into the queue.
Multiple data producers will push pages of at most 100 records into the queue. There will be no validation of the pages size by this class, so code that produces pages must take care of that.
- Return type of the future
- Function that produces a Future
- Timeout in milliseconds for the future to complete
- Delay in milliseconds to wait between successive retries of the future in case of failure or timeout. Number of elements in the sequence indicates how number of retries for the future.
- Delay in milliseconds to wait between successive retries of an Azure Table operation in case of failure or timeout.
- Delay in milliseconds to wait between successive retries of an Azure Table operation in case of failure or timeout. Number of elements in the sequence indicates how number of retries for the operation.
Initialises the writer threads, which will wait for pages of records to be placed into the queue.
Initialises the writer threads, which will wait for pages of records to be placed into the queue.
This method does not wait for completion and does not block, exits immediately after initialisation.
- Azure Table name
- Number of threads/connections.
- Timeout in milliseconds for an Azure Table operation
Azure tables are accessed using HTTP/HTTPS, with maximum batch size of 100 records per Azure Table partition key. Therefore there is a lot of lag between the batches and spending one spark executor per connection will be very wasteful. To work around it, this class will open multiple threads each having its own connection to an Azure Table and will push data from the blocking queue.
Created by Alexei Perelighin on 2018/03/01