Package ai.djl.inference
Class Predictor<I,O>
- java.lang.Object
-
- ai.djl.inference.Predictor<I,O>
-
- Type Parameters:
I
- the input typeO
- the output type
- All Implemented Interfaces:
java.lang.AutoCloseable
public class Predictor<I,O> extends java.lang.Object implements java.lang.AutoCloseable
ThePredictor
interface provides a session for model inference.You can use a
Predictor
, with a specifiedTranslator
, to perform inference on aModel
. The following is example code that usesPredictor
:Model model = Model.load(modelDir, modelName); // User must implement Translator interface, read
Translator
for detail. Translator<String, String> translator = new MyTranslator(); try (Predictor<String, String> predictor = model.newPredictor(translator)) { String result = predictor.predict("What's up"); }See the tutorials on:
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description protected class
Predictor.PredictorContext
-
Field Summary
Fields Modifier and Type Field Description protected Block
block
protected NDManager
manager
protected Metrics
metrics
protected Model
model
protected ParameterStore
parameterStore
protected boolean
prepared
protected long
timestamp
protected Translator<I,O>
translator
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description java.util.List<O>
batchPredict(java.util.List<I> inputs)
Predicts a batch for inference.void
close()
protected void
finalize()
O
predict(I input)
Predicts an item for inference.protected NDList
predictInternal(TranslatorContext ctx, NDList ndList)
Predicts an item for inference.void
setMetrics(Metrics metrics)
Attaches a Metrics param to use for benchmark.StreamingTranslator.StreamOutput<O>
streamingPredict(I input)
Predicts an item for inference.boolean
supportsStreaming()
Returns true if streaming is supported by the predictor, block, and translator.
-
-
-
Field Detail
-
translator
protected Translator<I,O> translator
-
timestamp
protected long timestamp
-
prepared
protected boolean prepared
-
model
protected Model model
-
manager
protected NDManager manager
-
metrics
protected Metrics metrics
-
block
protected Block block
-
parameterStore
protected ParameterStore parameterStore
-
-
Constructor Detail
-
Predictor
public Predictor(Model model, Translator<I,O> translator, Device device, boolean copy)
- Parameters:
model
- the model on which the predictions are basedtranslator
- the translator to be useddevice
- the device for predictioncopy
- whether to copy the parameters to the parameter store. If the device changes, it will copy regardless
-
-
Method Detail
-
predict
public O predict(I input) throws TranslateException
Predicts an item for inference.- Parameters:
input
- the input- Returns:
- the output object defined by the user
- Throws:
TranslateException
- if an error occurs during prediction
-
predictInternal
protected NDList predictInternal(TranslatorContext ctx, NDList ndList) throws TranslateException
Predicts an item for inference.- Parameters:
ctx
- the context for thePredictor
.ndList
- the inputNDList
- Returns:
- the output
NDList
- Throws:
TranslateException
- if an error occurs during prediction
-
batchPredict
public java.util.List<O> batchPredict(java.util.List<I> inputs) throws TranslateException
Predicts a batch for inference.- Parameters:
inputs
- a list of inputs- Returns:
- a list of output objects defined by the user
- Throws:
TranslateException
- if an error occurs during prediction
-
streamingPredict
public StreamingTranslator.StreamOutput<O> streamingPredict(I input) throws TranslateException
Predicts an item for inference.- Parameters:
input
- the input- Returns:
- the output object defined by the user
- Throws:
TranslateException
- if an error occurs during prediction
-
supportsStreaming
public boolean supportsStreaming()
Returns true if streaming is supported by the predictor, block, and translator.- Returns:
- true if streaming is supported by the predictor, block, and translator
-
setMetrics
public void setMetrics(Metrics metrics)
Attaches a Metrics param to use for benchmark.- Parameters:
metrics
- the Metrics class
-
close
public void close()
- Specified by:
close
in interfacejava.lang.AutoCloseable
-
finalize
protected void finalize() throws java.lang.Throwable
- Overrides:
finalize
in classjava.lang.Object
- Throws:
java.lang.Throwable
-
-