A B C E G H I J L M O P R S T U V W

A

ai.djl.serving.wlm - package ai.djl.serving.wlm

Contains the model server backend which manages worker threads and executes jobs on models.

ai.djl.serving.wlm.util - package ai.djl.serving.wlm.util

Contains utilities to support the WorkLoadManager.

B

build() - Method in class ai.djl.serving.wlm.WorkerThread.Builder

Builds the WorkerThread with the provided data.

builder() - Static method in class ai.djl.serving.wlm.WorkerThread

Creates a builder to build a WorkerThread.

C

cleanup() - Method in class ai.djl.serving.wlm.WorkLoadManager.WorkerPool

removes all stopped workers and workers in state error from the pool.

close() - Method in class ai.djl.serving.wlm.ModelInfo

configureModelBatch(int, int) - Method in class ai.djl.serving.wlm.ModelInfo

Sets a new batchSize and returns a new configured ModelInfo object.

configurePool(int) - Method in class ai.djl.serving.wlm.ModelInfo

Sets new configuration for the workerPool backing this model and returns a new configured ModelInfo object.

E

equals(Object) - Method in class ai.djl.serving.wlm.ModelInfo

G

generate() - Method in class ai.djl.serving.wlm.WorkerIdGenerator

generate a new worker id.

getBatchSize() - Method in class ai.djl.serving.wlm.ModelInfo

Returns the configured batch size.

getBegin() - Method in class ai.djl.serving.wlm.Job

Returns the job begin time.

getDefaultWorkers(NDManager, String, int) - Method in class ai.djl.serving.wlm.util.WlmConfigManager

Returns the default number of workers for a new registered model.

getFuture() - Method in class ai.djl.serving.wlm.util.WorkerJob

Returns the future for the job.

getGpuId() - Method in class ai.djl.serving.wlm.WorkerThread

Returns the gpu id used by the thread.

getInput() - Method in class ai.djl.serving.wlm.Job

Returns the input data.

getInstance() - Static method in class ai.djl.serving.wlm.util.WlmConfigManager

Returns the singleton ConfigManager instance.

getJob() - Method in class ai.djl.serving.wlm.util.WorkerJob

Returns the Job.

getJobQueue() - Method in class ai.djl.serving.wlm.WorkLoadManager.WorkerPool

Returns the JobQueue for this model.

getMaxBatchDelay() - Method in class ai.djl.serving.wlm.ModelInfo

Returns the maximum delay in milliseconds to aggregate a batch.

getMaxIdleTime() - Method in class ai.djl.serving.wlm.ModelInfo

returns the configured maxIdleTime of workers.

getMaxWorkers() - Method in class ai.djl.serving.wlm.WorkLoadManager.WorkerPool

Returns the maximum number of workers for a model.

getMinWorkers() - Method in class ai.djl.serving.wlm.WorkLoadManager.WorkerPool

Returns the minimum number of workers for a model.

getModel() - Method in class ai.djl.serving.wlm.Job

Returns the model that associated with this job.

getModel() - Method in class ai.djl.serving.wlm.ModelInfo

Returns the loaded ZooModel.

getModelDir() - Method in class ai.djl.serving.wlm.ModelInfo

Returns the model cache directory.

getModelName() - Method in class ai.djl.serving.wlm.ModelInfo

Returns the model name.

getNumRunningWorkers(ModelInfo) - Method in class ai.djl.serving.wlm.WorkLoadManager

Returns the number of running workers of a model.

getQueueLength(ModelInfo) - Method in class ai.djl.serving.wlm.WorkLoadManager

Returns the current number of request in the queue.

getQueueSize() - Method in class ai.djl.serving.wlm.ModelInfo

Returns the configured size of the workers queue.

getScheduled() - Method in class ai.djl.serving.wlm.Job

Returns the job scheduled time.

getStartTime() - Method in class ai.djl.serving.wlm.WorkerThread

Returns the thread start time.

getState() - Method in class ai.djl.serving.wlm.WorkerThread

Returns the worker state.

getVersion() - Method in class ai.djl.serving.wlm.ModelInfo

Returns the model version.

getWorkerId() - Method in class ai.djl.serving.wlm.WorkerThread

Returns the worker thread ID.

getWorkerPoolForModel(ModelInfo) - Method in class ai.djl.serving.wlm.WorkLoadManager

Returns the WorkLoadManager.WorkerPool for a model.

getWorkers(ModelInfo) - Method in class ai.djl.serving.wlm.WorkLoadManager

Returns the workers for the specific model.

getWorkers() - Method in class ai.djl.serving.wlm.WorkLoadManager.WorkerPool

Returns a list of worker thread.

H

hashCode() - Method in class ai.djl.serving.wlm.ModelInfo

I

inferModelNameFromUrl(String) - Static method in class ai.djl.serving.wlm.ModelInfo

Infer model name form model URL in case model name is not provided.

isDebug() - Method in class ai.djl.serving.wlm.util.WlmConfigManager

Returns if debug is enabled.

isFinished() - Method in class ai.djl.serving.wlm.PermanentBatchAggregator

Checks if this BatchAggregator and the thread can be shutdown or if this aggregator waits for more data.

isFinished() - Method in class ai.djl.serving.wlm.TemporaryBatchAggregator

Checks if this BatchAggregator and the thread can be shutdown or if this aggregator waits for more data.

isFixPoolThread() - Method in class ai.djl.serving.wlm.WorkerThread

check if this worker is instantiate is one of the fix threads of a pool.

isRunning() - Method in class ai.djl.serving.wlm.WorkerThread

Returns true if the worker thread is running.

J

Job - Class in ai.djl.serving.wlm

A class represents an inference job.

Job(ModelInfo, Input) - Constructor for class ai.djl.serving.wlm.Job

Constructs a new Job instance.

L

log() - Method in class ai.djl.serving.wlm.WorkLoadManager.WorkerPool

Logs the current state of this WorkerPool when level "Debug" is enabled.

M

ModelInfo - Class in ai.djl.serving.wlm

A class represent a loaded model and it's metadata.

ModelInfo(String, String, ZooModel<Input, Output>, int, int, int, int) - Constructor for class ai.djl.serving.wlm.ModelInfo

Constructs a new ModelInfo instance.

O

optAggregator(BatchAggregator) - Method in class ai.djl.serving.wlm.WorkerThread.Builder

Sets a BatchAggregator which overrides the instantiated default BatchAggregator.

optFixPoolThread(boolean) - Method in class ai.djl.serving.wlm.WorkerThread.Builder

Sets if the workerThread should be part of the fixed pool.

P

PermanentBatchAggregator - Class in ai.djl.serving.wlm

a batch aggregator that never terminates by itself.

PermanentBatchAggregator(ModelInfo, LinkedBlockingDeque<WorkerJob>) - Constructor for class ai.djl.serving.wlm.PermanentBatchAggregator

Constructs a PermanentBatchAggregator instance.

pollBatch() - Method in class ai.djl.serving.wlm.PermanentBatchAggregator

Fills in the list with a batch of jobs.

pollBatch() - Method in class ai.djl.serving.wlm.TemporaryBatchAggregator

Fills in the list with a batch of jobs.

preBuildProcessing() - Method in class ai.djl.serving.wlm.WorkerThread.Builder

R

run() - Method in class ai.djl.serving.wlm.WorkerThread

runJob(Job) - Method in class ai.djl.serving.wlm.WorkLoadManager

Adds an inference job to the job queue of the next free worker.

S

scaleWorkers(String, int, int) - Method in class ai.djl.serving.wlm.WorkLoadManager.WorkerPool

Sets new worker capcities for this model.

self() - Method in class ai.djl.serving.wlm.WorkerThread.Builder

Returns self reference to this builder.

setDebug(boolean) - Method in class ai.djl.serving.wlm.util.WlmConfigManager

Sets debug mode.

setJobQueue(LinkedBlockingDeque<WorkerJob>) - Method in class ai.djl.serving.wlm.WorkerThread.Builder

Sets the jobQueue used to poll for new jobs.

setModel(ModelInfo) - Method in class ai.djl.serving.wlm.WorkerThread.Builder

Sets the ModelInfo the thread will be responsible for.

setScheduled() - Method in class ai.djl.serving.wlm.Job

Marks the job has been scheduled.

shutdown(WorkerState) - Method in class ai.djl.serving.wlm.WorkerThread

Shuts down the worker thread.

T

TemporaryBatchAggregator - Class in ai.djl.serving.wlm

a batch aggregator that terminates after a maximum idle time.

TemporaryBatchAggregator(ModelInfo, LinkedBlockingDeque<WorkerJob>) - Constructor for class ai.djl.serving.wlm.TemporaryBatchAggregator

a batch aggregator that terminates after a maximum idle time.

toString() - Method in class ai.djl.serving.wlm.ModelInfo

U

unregisterModel(ModelInfo) - Method in class ai.djl.serving.wlm.WorkLoadManager

Removes a model from management.

V

validate() - Method in class ai.djl.serving.wlm.WorkerThread.Builder

valueOf(String) - Static method in enum ai.djl.serving.wlm.WorkerState

Returns the enum constant of this type with the specified name.

values() - Static method in enum ai.djl.serving.wlm.WorkerState

Returns an array containing the constants of this enum type, in the order they are declared.

W

WlmCapacityException - Exception in ai.djl.serving.wlm.util

Thrown to throttle when a job is run but the job queue capacity is exceeded.

WlmCapacityException(String) - Constructor for exception ai.djl.serving.wlm.util.WlmCapacityException

Constructs a WlmCapacityException with the specified detail message.

WlmCapacityException(String, Throwable) - Constructor for exception ai.djl.serving.wlm.util.WlmCapacityException

Constructs a WlmCapacityException with the specified detail message and cause.

WlmConfigManager - Class in ai.djl.serving.wlm.util

This manages some configurations used by the WorkLoadManager.

WlmConfigManager() - Constructor for class ai.djl.serving.wlm.util.WlmConfigManager

WlmException - Exception in ai.djl.serving.wlm.util

Thrown when an exception occurs inside the WorkLoadManager.

WlmException(String) - Constructor for exception ai.djl.serving.wlm.util.WlmException

Constructs a WlmException with the specified detail message.

WlmException(String, Throwable) - Constructor for exception ai.djl.serving.wlm.util.WlmException

Constructs a WlmException with the specified detail message and cause.

WlmShutdownException - Exception in ai.djl.serving.wlm.util

Thrown when a job is run but all workers are shutdown.

WlmShutdownException(String) - Constructor for exception ai.djl.serving.wlm.util.WlmShutdownException

Constructs a WlmShutdownException with the specified detail message.

WlmShutdownException(String, Throwable) - Constructor for exception ai.djl.serving.wlm.util.WlmShutdownException

Constructs a WlmShutdownException with the specified detail message and cause.

WorkerIdGenerator - Class in ai.djl.serving.wlm

class to generate an unique worker id.

WorkerIdGenerator() - Constructor for class ai.djl.serving.wlm.WorkerIdGenerator

WorkerJob - Class in ai.djl.serving.wlm.util

A Job containing metadata from the WorkLoadManager.

WorkerJob(Job, CompletableFuture<Output>) - Constructor for class ai.djl.serving.wlm.util.WorkerJob

Constructs a new WorkerJob.

WorkerPool(ModelInfo) - Constructor for class ai.djl.serving.wlm.WorkLoadManager.WorkerPool

Construct and initial data structure.

WorkerState - Enum in ai.djl.serving.wlm

An enum represents state of a worker.

WorkerThread - Class in ai.djl.serving.wlm

The WorkerThread is the worker managed by the WorkLoadManager.

WorkerThread.Builder - Class in ai.djl.serving.wlm

A Builder to construct a WorkerThread.

WorkLoadManager - Class in ai.djl.serving.wlm

WorkLoadManager is responsible to manage the work load of worker thread.

WorkLoadManager() - Constructor for class ai.djl.serving.wlm.WorkLoadManager

Constructs a WorkLoadManager instance.

WorkLoadManager.WorkerPool - Class in ai.djl.serving.wlm

Manages the work load for a single model.

A B C E G H I J L M O P R S T U V W