Package ai.djl.inference.streaming

Interface StreamingBlock

All Superinterfaces:

Block

All Known Implementing Classes:

PointwiseFeedForwardBlock, SequentialBlock

public interface StreamingBlock
extends Block

A Block possessing the additional streaming forward capabilities used by Predictor.streamingPredict(Object).

Method Summary

Modifier and Type	Method	Description
default java.util.stream.Stream<NDList>	forwardStream(ParameterStore parameterStore, NDList inputs, boolean training)	Applies the operating function of the block once, but returns the result in chunks.
default java.util.stream.Stream<NDList>	forwardStream(ParameterStore parameterStore, NDList inputs, boolean training, ai.djl.util.PairList<java.lang.String,java.lang.Object> params)	Applies the operating function of the block once, but returns the result in chunks.
java.util.Iterator<NDList>	forwardStreamIter(ParameterStore parameterStore, NDList inputs, boolean training, ai.djl.util.PairList<java.lang.String,java.lang.Object> params)	Applies the operating function of the block once, but returns the result in chunks.

Methods inherited from interface ai.djl.nn.Block

cast, clear, describeInput, forward, forward, forward, freezeParameters, freezeParameters, getChildren, getDirectParameters, getInputShapes, getOutputDataTypes, getOutputShapes, getOutputShapes, getParameters, initialize, isInitialized, loadParameters, saveParameters, setInitializer, setInitializer, setInitializer

Method Detail

forwardStream

default java.util.stream.Stream<NDList> forwardStream(ParameterStore parameterStore,
                                                      NDList inputs,
                                                      boolean training)

Applies the operating function of the block once, but returns the result in chunks. This method should only be called on blocks that are initialized.

Parameters:

parameterStore - the parameter store

inputs - the input NDList

training - true for a training forward pass (turn on dropout and layerNorm)

Returns:

the output of the forward pass

forwardStream

default java.util.stream.Stream<NDList> forwardStream(ParameterStore parameterStore,
                                                      NDList inputs,
                                                      boolean training,
                                                      ai.djl.util.PairList<java.lang.String,java.lang.Object> params)

Applies the operating function of the block once, but returns the result in chunks. This method should only be called on blocks that are initialized.

Parameters:

parameterStore - the parameter store

inputs - the input NDList

training - true for a training forward pass (turn on dropout and layerNorm)

params - optional parameters

Returns:

the output of the forward pass

forwardStreamIter

java.util.Iterator<NDList> forwardStreamIter(ParameterStore parameterStore,
                                             NDList inputs,
                                             boolean training,
                                             ai.djl.util.PairList<java.lang.String,java.lang.Object> params)

Applies the operating function of the block once, but returns the result in chunks. This method should only be called on blocks that are initialized.

Parameters:

parameterStore - the parameter store

inputs - the input NDList

training - true for a training forward pass (turn on dropout and layerNorm)

params - optional parameters

Returns:

the output of the forward pass