ai.djl.serving.wlm

Class WorkLoadManager

java.lang.Object

ai.djl.serving.wlm.WorkLoadManager

public class WorkLoadManager
extends java.lang.Object

WorkLoadManager is responsible to manage the work load of worker thread. the manage scales up/down the required amount of worker threads per model.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
class	WorkLoadManager.WorkerPool Manages the work load for a single model.

Constructor Summary

Constructors

Constructor and Description
WorkLoadManager() Constructs a WorkLoadManager instance.

Method Summary

Modifier and Type	Method and Description
int	getNumRunningWorkers(ModelInfo modelInfo) Returns the number of running workers of a model.
int	getQueueLength(ModelInfo modelInfo) Returns the current number of request in the queue.
WorkLoadManager.WorkerPool	getWorkerPoolForModel(ModelInfo modelInfo) Returns the WorkLoadManager.WorkerPool for a model.
java.util.List<WorkerThread>	getWorkers(ModelInfo modelInfo) Returns the workers for the specific model.
java.util.concurrent.CompletableFuture<ai.djl.modality.Output>	runJob(Job job) Adds an inference job to the job queue of the next free worker.
void	unregisterModel(ModelInfo model) Removes a model from management.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

WorkLoadManager

public WorkLoadManager()

Constructs a WorkLoadManager instance.

Method Detail

getWorkers

public java.util.List<WorkerThread> getWorkers(ModelInfo modelInfo)

Returns the workers for the specific model.

Parameters:

modelInfo - the name of the model we are looking for.

Returns:

the list of workers responsible to handle predictions for this model.

unregisterModel

public void unregisterModel(ModelInfo model)

Removes a model from management.

Parameters:

model - the model to remove

runJob

public java.util.concurrent.CompletableFuture<ai.djl.modality.Output> runJob(Job job)

Adds an inference job to the job queue of the next free worker. scales up worker if necessary.

Parameters:

job - an inference job to be executed.

Returns:

true if submit success, false otherwise.

getNumRunningWorkers

public int getNumRunningWorkers(ModelInfo modelInfo)

Returns the number of running workers of a model. running workers are workers which are not stopped, in error or scheduled to scale down.

Parameters:

modelInfo - the model we are interested in.

Returns:

number of running workers.

getQueueLength

public int getQueueLength(ModelInfo modelInfo)

Returns the current number of request in the queue.

Parameters:

modelInfo - the model

Returns:

the current number of request in the queue

getWorkerPoolForModel

public WorkLoadManager.WorkerPool getWorkerPoolForModel(ModelInfo modelInfo)

Returns the WorkLoadManager.WorkerPool for a model.

Parameters:

modelInfo - the model to get the worker pool for

Returns:

the WorkLoadManager.WorkerPool