Package ai.djl.serving.wlm
Class PermanentBatchAggregator<I,O>
java.lang.Object
ai.djl.serving.wlm.PermanentBatchAggregator<I,O>
a batch aggregator that never terminates by itself. the surrounding thread has to be interrupted
by sending an interrupt signal.
-
Field Summary
Fields -
Constructor Summary
ConstructorsConstructorDescriptionPermanentBatchAggregator
(WorkerPoolConfig<I, O> wpc, LinkedBlockingDeque<WorkerJob<I, O>> jobQueue) Constructs aPermanentBatchAggregator
instance. -
Method Summary
Modifier and TypeMethodDescriptionprotected void
Poll the queue and return a list of Input Objects for the model.boolean
Checks if thisBatchAggregator
and the thread can be shutdown or if this aggregator waits for more data.Fills in the list with a batch of jobs.void
Completes the job with an error.void
Sends to response to all waiting clients.
-
Field Details
-
batchSize
protected int batchSize -
maxBatchDelayMicros
protected long maxBatchDelayMicros -
wjs
-
jobQueue
-
-
Constructor Details
-
PermanentBatchAggregator
public PermanentBatchAggregator(WorkerPoolConfig<I, O> wpc, LinkedBlockingDeque<WorkerJob<I, O>> jobQueue) Constructs aPermanentBatchAggregator
instance.- Parameters:
wpc
- the workerPoolConfig to use.jobQueue
- the job queue for polling data from.
-
-
Method Details
-
pollBatch
Fills in the list with a batch of jobs.- Returns:
- a list of jobs read by this batch interation.
- Throws:
InterruptedException
- if interrupted
-
isFinished
public boolean isFinished()Checks if thisBatchAggregator
and the thread can be shutdown or if this aggregator waits for more data.- Returns:
- true if we can shutdown the thread. for example when max idle time exceeded in temporary batch aggregator.
-
getRequest
Poll the queue and return a list of Input Objects for the model.- Returns:
- list of input objects to pass to the model.
- Throws:
InterruptedException
- if thread gets interrupted while waiting for new data in the queue.
-
sendResponse
public void sendResponse()Sends to response to all waiting clients. -
sendError
Completes the job with an error.- Parameters:
error
- the exception
-
drainTo
- Throws:
InterruptedException
-