ai.djl.nn.transformer

Class BertMaskedLanguageModelBlock

java.lang.Object

ai.djl.nn.AbstractBlock

ai.djl.nn.transformer.BertMaskedLanguageModelBlock

All Implemented Interfaces:

Block

public class BertMaskedLanguageModelBlock
extends AbstractBlock

Block for the bert masked language task.

Field Summary

Fields inherited from class ai.djl.nn.AbstractBlock

children, inputNames, inputShapes, parameters, parameterShapeCallbacks, version

Constructor Summary

Constructors

Constructor and Description
BertMaskedLanguageModelBlock(BertBlock bertBlock, java.util.function.Function<NDArray,NDArray> hiddenActivation) Creates a new block that applies the masked language task.

Method Summary

Modifier and Type	Method and Description
NDArray	forward(ParameterStore ps, NDArray sequenceOutput, NDArray maskedIndices, NDArray embeddingTable, boolean training) Calculates the result of the masked language task.
NDList	forward(ParameterStore ps, NDList inputs, boolean training) Applies the operating function of the block once.
protected NDList	forwardInternal(ParameterStore ps, NDList inputs, boolean training, ai.djl.util.PairList<java.lang.String,java.lang.Object> params)
static NDArray	gatherFromIndices(NDArray sequences, NDArray indices) Given a 3D array of shape (B, S, E) and a 2D array of shape (B, I) returns the flattened lookup result of shape (B * I * E).
Shape[]	getOutputShapes(NDManager manager, Shape[] inputShapes) Returns the expected output shapes of the block for the specified input shapes.
void	initializeChildBlocks(NDManager manager, DataType dataType, Shape... inputShapes) Initializes the Child blocks of this block.

Methods inherited from class ai.djl.nn.AbstractBlock

addChildBlock, addParameter, addParameter, addParameter, beforeInitialize, cast, clear, describeInput, forward, getChildren, getDirectParameters, getParameters, getParameterShape, initialize, isInitialized, loadMetadata, loadParameters, readInputShapes, saveInputShapes, saveMetadata, saveParameters, setInitializer, setInitializer, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface ai.djl.nn.Block

forward, validateLayout

Constructor Detail

BertMaskedLanguageModelBlock

public BertMaskedLanguageModelBlock(BertBlock bertBlock,
                                    java.util.function.Function<NDArray,NDArray> hiddenActivation)

Creates a new block that applies the masked language task.

Parameters:

bertBlock - the bert block to create the task for

hiddenActivation - the activation to use for the hidden layer

Method Detail

gatherFromIndices

public static NDArray gatherFromIndices(NDArray sequences,
                                        NDArray indices)

Given a 3D array of shape (B, S, E) and a 2D array of shape (B, I) returns the flattened lookup result of shape (B * I * E).

Parameters:

sequences - Sequences of embeddings

indices - Indices into the sequences. The indices are relative within each sequence, i.e. [[0, 1],[0, 1]] would return the first two elements of two sequences.

Returns:

The flattened result of gathering elements from the sequences

initializeChildBlocks

public void initializeChildBlocks(NDManager manager,
                                  DataType dataType,
                                  Shape... inputShapes)

Initializes the Child blocks of this block. You need to override this method if your subclass has child blocks. Used to determine the correct input shapes for child blocks based on the requested input shape for this block.

Overrides:

initializeChildBlocks in class AbstractBlock

Parameters:

manager - the manager to use for initialization

dataType - the requested data type

inputShapes - the expected input shapes for this block

forwardInternal

protected NDList forwardInternal(ParameterStore ps,
                                 NDList inputs,
                                 boolean training,
                                 ai.djl.util.PairList<java.lang.String,java.lang.Object> params)

Specified by:

forwardInternal in class AbstractBlock

forward

public NDList forward(ParameterStore ps,
                      NDList inputs,
                      boolean training)

Applies the operating function of the block once. This method should be called only on blocks that are initialized.

Parameters:

ps - the parameter store

inputs - the input NDList

training - true for a training forward pass

Returns:

the output of the forward pass

forward

public NDArray forward(ParameterStore ps,
                       NDArray sequenceOutput,
                       NDArray maskedIndices,
                       NDArray embeddingTable,
                       boolean training)

Calculates the result of the masked language task.

Parameters:

ps - the parameter store

sequenceOutput - The sequence output of the bert model (B, S, E)

maskedIndices - The indices of the tokens masked for pretraining (B, I)

embeddingTable - The embedding table of the bert model (D, E)

training - true=apply dropout etc.

Returns:

the log probabilities for each dictionary item for each masked token, size (B * I, D)

getOutputShapes

public Shape[] getOutputShapes(NDManager manager,
                               Shape[] inputShapes)

Returns the expected output shapes of the block for the specified input shapes.

Parameters:

manager - an NDManager

inputShapes - the shapes of the inputs

Returns:

the expected output shapes of the block