Package ai.djl.serving.wlm

Class TemporaryBatchAggregator<I,​O>

  • public class TemporaryBatchAggregator<I,​O>
extends java.lang.Object
    a batch aggregator that terminates after a maximum idle time.

    • Constructor Summary

      Constructors 
      Constructor Description
      TemporaryBatchAggregator​(ModelInfo<I,​O> model, java.util.concurrent.LinkedBlockingDeque<WorkerJob<I,​O>> jobQueue)
      a batch aggregator that terminates after a maximum idle time.

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      protected void drainTo​(java.util.List<WorkerJob<I,​O>> list, int maxDelay)  
      java.util.List<I> getRequest()
      Poll the queue and return a list of Input Objects for the model.
      boolean isFinished()
      Checks if this BatchAggregator and the thread can be shutdown or if this aggregator waits for more data.
      protected java.util.List<WorkerJob<I,​O>> pollBatch()
      Fills in the list with a batch of jobs.
      void sendError​(java.lang.Throwable error)
      Completes the job with an error.
      void sendResponse​(java.util.List<O> outputs)
      Sends to response to all waiting clients.

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Field Detail

      • batchSize

        protected int batchSize

      • maxBatchDelay

        protected int maxBatchDelay

      • wjs

        protected java.util.List<WorkerJob<I,​O>> wjs

      • jobQueue

        protected java.util.concurrent.LinkedBlockingDeque<WorkerJob<I,​O>> jobQueue

    • Constructor Detail

      • TemporaryBatchAggregator

        public TemporaryBatchAggregator​(ModelInfo<I,​O> model,
                                java.util.concurrent.LinkedBlockingDeque<WorkerJob<I,​O>> jobQueue)
        a batch aggregator that terminates after a maximum idle time.
        Parameters:
        model - the model to run for.
        jobQueue - reference to external job queue for polling.

    • Method Detail

      • pollBatch

        protected java.util.List<WorkerJob<I,​O>> pollBatch()
                                                  throws java.lang.InterruptedException
        Fills in the list with a batch of jobs.
        Returns:
        a list of jobs read by this batch interation.
        Throws:
        java.lang.InterruptedException - if interrupted

      • isFinished

        public boolean isFinished()
        Checks if this BatchAggregator and the thread can be shutdown or if this aggregator waits for more data.
        Returns:
        true if we can shutdown the thread. for example when max idle time exceeded in temporary batch aggregator.

      • getRequest

        public java.util.List<I> getRequest()
                             throws java.lang.InterruptedException
        Poll the queue and return a list of Input Objects for the model.
        Returns:
        list of input objects to pass to the model.
        Throws:
        java.lang.InterruptedException - if thread gets interrupted while waiting for new data in the queue.

      • sendResponse

        public void sendResponse​(java.util.List<O> outputs)
        Sends to response to all waiting clients.
        Parameters:
        outputs - list of model-outputs in same order as the input objects.

      • sendError

        public void sendError​(java.lang.Throwable error)
        Completes the job with an error.
        Parameters:
        error - the exception

      • drainTo

        protected void drainTo​(java.util.List<WorkerJob<I,​O>> list,
                       int maxDelay)
                throws java.lang.InterruptedException
        Throws:
        java.lang.InterruptedException