ai.djl.inference

Class Predictor<I,O>

java.lang.Object

ai.djl.inference.Predictor<I,O>

Type Parameters:

I - the input type

O - the output type

All Implemented Interfaces:

java.lang.AutoCloseable

public class Predictor<I,O>
extends java.lang.Object
implements java.lang.AutoCloseable

The Predictor interface provides a session for model inference.

You can use a Predictor, with a specified Translator, to perform inference on a Model. The following is example code that uses Predictor:

 Model model = Model.load(modelDir, modelName);

 // User must implement Translator interface, read Translator for detail.
 Translator<String, String> translator = new MyTranslator();

 try (Predictor<String, String> predictor = model.newPredictor(translator)) {
   String result = predictor.predict("What's up");
 }
 

See the tutorials on:

• Inference with a custom trained model
• Inference with a model zoo model
• Inference with an MXNet model

See Also:

Model, Translator, The guide on memory management, The guide on running multi-threaded inference, The guide on inference performance optimization

Field Summary

Fields

Modifier and Type	Field and Description
protected Block	block
protected NDManager	manager
protected Metrics	metrics
protected ParameterStore	parameterStore

Constructor Summary

Constructors

Constructor and Description
Predictor(Model model, Translator<I,O> translator, Device device, boolean copy) Creates a new instance of BasePredictor with the given Model and Translator.

Method Summary

Modifier and Type	Method and Description
java.util.List<O>	batchPredict(java.util.List<I> inputs) Predicts a batch for inference.
void	close()
protected void	finalize()
O	predict(I input) Predicts an item for inference.
protected NDList	predictInternal(TranslatorContext ctx, NDList ndList) Predicts an item for inference.
void	setMetrics(Metrics metrics) Attaches a Metrics param to use for benchmark.

Methods inherited from class java.lang.Object

clone, equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

manager

protected NDManager manager

metrics

protected Metrics metrics

block

protected Block block

parameterStore

protected ParameterStore parameterStore

Constructor Detail

Predictor

public Predictor(Model model,
                 Translator<I,O> translator,
                 Device device,
                 boolean copy)

Creates a new instance of BasePredictor with the given Model and Translator.

Parameters:

model - the model on which the predictions are based

translator - the translator to be used

device - the device for prediction

copy - whether to copy the parameters to the parameter store. If the device changes, it will copy regardless

Method Detail

predict

public O predict(I input)
          throws TranslateException

Predicts an item for inference.

Parameters:

input - the input

Returns:

the output object defined by the user

Throws:

TranslateException - if an error occurs during prediction

predictInternal

protected NDList predictInternal(TranslatorContext ctx,
                                 NDList ndList)
                          throws TranslateException

Predicts an item for inference.

Parameters:

ctx - the context for the Predictor.

ndList - the input NDList

Returns:

the output NDList

Throws:

TranslateException - if an error occurs during prediction

batchPredict

public java.util.List<O> batchPredict(java.util.List<I> inputs)
                               throws TranslateException

Predicts a batch for inference.

Parameters:

inputs - a list of inputs

Returns:

a list of output objects defined by the user

Throws:

TranslateException - if an error occurs during prediction

setMetrics

public void setMetrics(Metrics metrics)

Attaches a Metrics param to use for benchmark.

Parameters:

metrics - the Metrics class

close

public void close()

Specified by:

close in interface java.lang.AutoCloseable

finalize

protected void finalize()
                 throws java.lang.Throwable

Overrides:

finalize in class java.lang.Object

Throws:

java.lang.Throwable