Package ai.djl.inference.streaming

Interface StreamingBlock

All Superinterfaces:

Block

All Known Implementing Classes:

PointwiseFeedForwardBlock, SequentialBlock

public interface StreamingBlock
extends Block

A Block possessing the additional streaming forward capabilities used by Predictor.streamingPredict(Object).

Method Summary

Modifier and Type	Method	Description
default Stream<NDList>	forwardStream(ParameterStore parameterStore, NDList inputs, boolean training)	Applies the operating function of the block once, but returns the result in chunks.
default Stream<NDList>	forwardStream(ParameterStore parameterStore, NDList inputs, boolean training, ai.djl.util.PairList<String,Object> params)	Applies the operating function of the block once, but returns the result in chunks.
Iterator<NDList>	forwardStreamIter(ParameterStore parameterStore, NDList inputs, boolean training, ai.djl.util.PairList<String,Object> params)	Applies the operating function of the block once, but returns the result in chunks.

Methods inherited from interface ai.djl.nn.Block

cast, clear, describeInput, forward, forward, forward, freezeParameters, freezeParameters, getChildren, getDirectParameters, getInputShapes, getOutputDataTypes, getOutputShapes, getOutputShapes, getParameters, initialize, isInitialized, loadParameters, saveParameters, setInitializer, setInitializer, setInitializer

Method Details

forwardStream

default Stream<NDList> forwardStream(ParameterStore parameterStore,
 NDList inputs,
 boolean training)

Applies the operating function of the block once, but returns the result in chunks. This method should only be called on blocks that are initialized.

Parameters:

parameterStore - the parameter store

inputs - the input NDList

training - true for a training forward pass (turn on dropout and layerNorm)

Returns:

the output of the forward pass

forwardStream

default Stream<NDList> forwardStream(ParameterStore parameterStore,
 NDList inputs,
 boolean training,
 ai.djl.util.PairList<String,Object> params)

Applies the operating function of the block once, but returns the result in chunks. This method should only be called on blocks that are initialized.

Parameters:

parameterStore - the parameter store

inputs - the input NDList

training - true for a training forward pass (turn on dropout and layerNorm)

params - optional parameters

Returns:

the output of the forward pass

forwardStreamIter

Iterator<NDList> forwardStreamIter(ParameterStore parameterStore,
 NDList inputs,
 boolean training,
 ai.djl.util.PairList<String,Object> params)

Applies the operating function of the block once, but returns the result in chunks. This method should only be called on blocks that are initialized.

Parameters:

parameterStore - the parameter store

inputs - the input NDList

training - true for a training forward pass (turn on dropout and layerNorm)

params - optional parameters

Returns:

the output of the forward pass