ai.djl.modality.nlp.bert

Class BertToken

java.lang.Object

ai.djl.modality.nlp.bert.BertToken

public class BertToken
extends java.lang.Object

BertToken contains all the information for Bert model after encoding question and paragraph.

Constructor Summary

Constructors

Constructor and Description
BertToken(java.util.List<java.lang.String> tokens, java.util.List<java.lang.Long> tokenType, java.util.List<java.lang.Long> attentionMask, int validLength) Creates an instance of BertToken which includes information for Bert model.

Method Summary

Modifier and Type	Method and Description
java.util.List<java.lang.Long>	getAttentionMask() Gets the mask to avoid performing attention on padding token indices.
java.util.List<java.lang.String>	getTokens() Gets the indices of input sequence tokens in the vocabulary.
java.util.List<java.lang.Long>	getTokenTypes() Gets segment token indices to indicate first and second portions of the inputs.
int	getValidLength() Gets the length of the original sentence which has question and paragraph.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BertToken

public BertToken(java.util.List<java.lang.String> tokens,
                 java.util.List<java.lang.Long> tokenType,
                 java.util.List<java.lang.Long> attentionMask,
                 int validLength)

Creates an instance of BertToken which includes information for Bert model.

Parameters:

tokens - indices of input sequence tokens in the vocabulary.

tokenType - segment token indices to indicate first and second portions of the inputs.

attentionMask - mask to avoid performing attention on padding token indices.

validLength - length that indicates the original input sequence.

Method Detail

getTokens

public java.util.List<java.lang.String> getTokens()

Gets the indices of input sequence tokens in the vocabulary.

Returns:

indices of input sequence tokens

getTokenTypes

public java.util.List<java.lang.Long> getTokenTypes()

Gets segment token indices to indicate first and second portions of the inputs.

Returns:

segment token indices

getAttentionMask

public java.util.List<java.lang.Long> getAttentionMask()

Gets the mask to avoid performing attention on padding token indices.

Returns:

mask that performs attention on non-padding token indices

getValidLength

public int getValidLength()

Gets the length of the original sentence which has question and paragraph.

Returns:

length of the original sentence which has question and paragraph