Package ai.djl.serving.wlm
Class TemporaryBatchAggregator<I,O>
java.lang.Object
ai.djl.serving.wlm.TemporaryBatchAggregator<I,O>
a batch aggregator that terminates after a maximum idle time.
-
Field Summary
Fields -
Constructor Summary
ConstructorsConstructorDescriptionTemporaryBatchAggregator
(WorkerPoolConfig<I, O> wpc, LinkedBlockingDeque<WorkerJob<I, O>> jobQueue) a batch aggregator that terminates after a maximum idle time. -
Method Summary
Modifier and TypeMethodDescriptionprotected void
Poll the queue and return a list of Input Objects for the model.boolean
Checks if thisBatchAggregator
and the thread can be shutdown or if this aggregator waits for more data.Fills in the list with a batch of jobs.void
Completes the job with an error.void
Sends to response to all waiting clients.
-
Field Details
-
batchSize
protected int batchSize -
maxBatchDelayMicros
protected long maxBatchDelayMicros -
wjs
-
jobQueue
-
-
Constructor Details
-
TemporaryBatchAggregator
public TemporaryBatchAggregator(WorkerPoolConfig<I, O> wpc, LinkedBlockingDeque<WorkerJob<I, O>> jobQueue) a batch aggregator that terminates after a maximum idle time.- Parameters:
wpc
- the workerPoolConfig to run for.jobQueue
- reference to external job queue for polling.
-
-
Method Details
-
pollBatch
Fills in the list with a batch of jobs.- Returns:
- a list of jobs read by this batch interation.
- Throws:
InterruptedException
- if interrupted
-
isFinished
public boolean isFinished()Checks if thisBatchAggregator
and the thread can be shutdown or if this aggregator waits for more data.- Returns:
- true if we can shutdown the thread. for example when max idle time exceeded in temporary batch aggregator.
-
getRequest
Poll the queue and return a list of Input Objects for the model.- Returns:
- list of input objects to pass to the model.
- Throws:
InterruptedException
- if thread gets interrupted while waiting for new data in the queue.
-
sendResponse
public void sendResponse()Sends to response to all waiting clients. -
sendError
Completes the job with an error.- Parameters:
error
- the exception
-
drainTo
- Throws:
InterruptedException
-