io.greptime

Interface BulkStreamWriter

All Superinterfaces:

AutoCloseable

public interface BulkStreamWriter
extends AutoCloseable

BulkStreamWriter is a specialized interface for efficiently writing data to the server in bulk operations.

Each BulkStreamWriter is associated with a TableBufferRoot instance that manages off-heap memory. The workflow involves first obtaining the TableBufferRoot instance and populating it with data. This data resides in off-heap memory and is only transmitted to the server when writeNext() is called. Upon calling writeNext(), all data in the TableBufferRoot is sent as a batch, and the TableBufferRoot is automatically cleared for the next set of data.

As a streaming interface, BulkStreamWriter allows you to repeat this cycle (populate data, call writeNext()) multiple times for efficient batch processing. When all data has been transmitted, you must call completed() to properly close the stream and ensure any server-side errors are properly reported. Additionally, you should call the close() method to ensure all related resources are properly released, though this happens automatically when using try-with-resources.

Example usage:

 try (BulkStreamWriter bulkStreamWriter = greptimeDB.bulkStreamWriter(schema)) { // auto close in try-with-resources
     // Write 1000 times, each time write 100000 rows
     for (int i = 0; i < 1000; i++) {
         long start = System.currentTimeMillis();
         Table.TableBufferRoot table = bulkStreamWriter.tableBufferRoot();
         for (int j = 0; j < 100000; j++) {
             // with 100000 cardinality
             Object[] row = generateOneRow(100000);
             table.addRow(row);
         }
         // Complete the table; adding rows is no longer permitted.
         table.complete();

         LOG.info("Prepare data, time cost: {}ms", System.currentTimeMillis() - start);

         start = System.currentTimeMillis();
         CompletableFuture<Integer> future = bulkStreamWriter.writeNext();
         Integer result = future.get();
         LOG.info("Wrote rows: {}, time cost: {}ms", result, System.currentTimeMillis() - start);
     }

     bulkStreamWriter.completed();
 }
 

Method Summary

Modifier and Type	Method and Description
void	completed() Completes the bulk write operation by signaling the end of transmission and waits for the server to finish processing the data.
default boolean	isStreamReady() If the stream is not ready, calling writeNext() will block until the stream becomes ready for writing, potentially using a busy-wait mechanism.
Table.TableBufferRoot	tableBufferRoot(int columnBufferSize) Returns the TableBufferRoot instance associated with this writer.
CompletableFuture<Integer>	writeNext() Writes current table data to the stream.

Methods inherited from interface java.lang.AutoCloseable

close

Method Detail

tableBufferRoot

Table.TableBufferRoot tableBufferRoot(int columnBufferSize)

Returns the TableBufferRoot instance associated with this writer. The TableBufferRoot provides direct access to the underlying memory where table data is stored for efficient bulk operations.

Parameters:

columnBufferSize - the buffer size for each column

Returns:

a table buffer root

writeNext

CompletableFuture<Integer> writeNext()
                              throws Exception

Writes current table data to the stream.

Returns:

a future that completes with the number of rows affected

Throws:

Exception - if an error occurs

completed

void completed()
        throws Exception

Completes the bulk write operation by signaling the end of transmission and waits for the server to finish processing the data. This method must be called to ensure all data is properly written and to receive any errors that may have occurred during the operation.

Throws:

Exception - if an error occurs

isStreamReady

default boolean isStreamReady()

If the stream is not ready, calling writeNext() will block until the stream becomes ready for writing, potentially using a busy-wait mechanism.

Returns:

true if the stream is ready to write data, false otherwise