Class MultiFileTransfer<Request extends MultiFileTransfer.ChunkRequest>
- All Implemented Interfaces:
java.io.Closeable
,java.lang.AutoCloseable
public abstract class MultiFileTransfer<Request extends MultiFileTransfer.ChunkRequest>
extends java.lang.Object
implements java.io.Closeable
The sender/requestor can send up to maxConcurrentFileChunks
file chunk requests without waiting for responses. Since the recovery
target can receive file chunks out of order, it has to buffer those file chunks in memory and only flush to disk when there's no gap.
To ensure the recover target never buffers more than maxConcurrentFileChunks
file chunks, we allow the sender/requestor to send
only up to maxConcurrentFileChunks
file chunk requests from the last flushed (and acknowledged) file chunk. We leverage the local
checkpoint tracker for this purpose. We generate a new sequence number and assign it to each file chunk request before sending; then mark
that sequence number as processed when we receive a response for the corresponding file chunk request. With the local checkpoint tracker,
we know the last acknowledged-flushed file-chunk is a file chunk whose requestSeqId
equals to the local checkpoint because the
recover target can flush all file chunks up to the local checkpoint.
When the number of un-replied file chunk requests reaches the limit (i.e. the gap between the max_seq_no and the local checkpoint is
greater than maxConcurrentFileChunks
), the sending/requesting thread will abort its execution. That process will be resumed by
one of the networking threads which receive/handle the responses of the current pending file chunk requests. This process will continue
until all chunk requests are sent/responded.
-
Nested Class Summary
Nested Classes Modifier and Type Class Description static interface
MultiFileTransfer.ChunkRequest
-
Constructor Summary
Constructors Modifier Constructor Description protected
MultiFileTransfer(org.apache.logging.log4j.Logger logger, ThreadContext threadContext, ActionListener<java.lang.Void> listener, int maxConcurrentFileChunks, java.util.List<StoreFileMetaData> files)
-
Method Summary
Modifier and Type Method Description protected abstract void
executeChunkRequest(Request request, ActionListener<java.lang.Void> listener)
protected abstract void
handleError(StoreFileMetaData md, java.lang.Exception e)
protected abstract Request
nextChunkRequest(StoreFileMetaData md)
protected abstract void
onNewFile(StoreFileMetaData md)
This method is called when starting sending/requesting a new file.void
start()
-
Constructor Details
-
MultiFileTransfer
protected MultiFileTransfer(org.apache.logging.log4j.Logger logger, ThreadContext threadContext, ActionListener<java.lang.Void> listener, int maxConcurrentFileChunks, java.util.List<StoreFileMetaData> files)
-
-
Method Details
-
start
public final void start() -
onNewFile
This method is called when starting sending/requesting a new file. Subclasses should override this method to reset the file offset or close the previous file and open a new file if needed.- Throws:
java.io.IOException
-
nextChunkRequest
- Throws:
java.io.IOException
-
executeChunkRequest
protected abstract void executeChunkRequest(Request request, ActionListener<java.lang.Void> listener) -
handleError
protected abstract void handleError(StoreFileMetaData md, java.lang.Exception e) throws java.lang.Exception- Throws:
java.lang.Exception
-