Package ai.djl.modality.nlp.bert

Class BertToken

java.lang.Object

ai.djl.modality.nlp.bert.BertToken

public class BertToken
extends Object

BertToken contains all the information for Bert model after encoding question and paragraph.

Constructor Summary

Constructors

Constructor	Description
BertToken(List<String> tokens, List<Long> tokenType, List<Long> attentionMask, int validLength)	Creates an instance of BertToken which includes information for Bert model.

Method Summary

Modifier and Type	Method	Description
List<Long>	getAttentionMask()	Gets the mask to avoid performing attention on padding token indices.
List<String>	getTokens()	Gets the indices of input sequence tokens in the vocabulary.
List<Long>	getTokenTypes()	Gets segment token indices to indicate first and second portions of the inputs.
int	getValidLength()	Gets the length of the original sentence which has question and paragraph.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

BertToken

public BertToken(List<String> tokens,
 List<Long> tokenType,
 List<Long> attentionMask,
 int validLength)

Creates an instance of BertToken which includes information for Bert model.

Parameters:

tokens - indices of input sequence tokens in the vocabulary.

tokenType - segment token indices to indicate first and second portions of the inputs.

attentionMask - mask to avoid performing attention on padding token indices.

validLength - length that indicates the original input sequence.

Method Details

getTokens

public List<String> getTokens()

Gets the indices of input sequence tokens in the vocabulary.

Returns:

indices of input sequence tokens

getTokenTypes

public List<Long> getTokenTypes()

Gets segment token indices to indicate first and second portions of the inputs.

Returns:

segment token indices

getAttentionMask

public List<Long> getAttentionMask()

Gets the mask to avoid performing attention on padding token indices.

Returns:

mask that performs attention on non-padding token indices

getValidLength

public int getValidLength()

Gets the length of the original sentence which has question and paragraph.

Returns:

length of the original sentence which has question and paragraph