Package ai.djl.huggingface.tokenizers

Class Encoding

java.lang.Object

ai.djl.huggingface.tokenizers.Encoding

public class Encoding
extends Object

A class holds token encoding information.

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	Encoding(long[] ids, long[] typeIds, String[] tokens, long[] wordIds, long[] attentionMask, long[] specialTokenMask, CharSpan[] charTokenSpans, boolean exceedMaxLength, Encoding[] overflowing)

Method Summary

Modifier and Type	Method	Description
boolean	exceedMaxLength()	Returns if tokens exceed max length.
long[]	getAttentionMask()	Returns the attention masks.
CharSpan[]	getCharTokenSpans()	Returns char token spans.
long[]	getIds()	Returns the token ids.
Encoding[]	getOverflowing()	Returns an array of overflowing encodings.
long[]	getSpecialTokenMask()	Returns the special token masks.
String[]	getTokens()	Returns the tokens.
long[]	getTypeIds()	Returns the token type ids.
long[]	getWordIds()	Returns the word ids.
ai.djl.ndarray.NDList	toNDList(ai.djl.ndarray.NDManager manager, boolean withTokenType)	Returns the NDList representation of the encoding.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

Encoding

protected Encoding(long[] ids,
 long[] typeIds,
 String[] tokens,
 long[] wordIds,
 long[] attentionMask,
 long[] specialTokenMask,
 CharSpan[] charTokenSpans,
 boolean exceedMaxLength,
 Encoding[] overflowing)

Method Details

toNDList

public ai.djl.ndarray.NDList toNDList(ai.djl.ndarray.NDManager manager,
 boolean withTokenType)

Returns the NDList representation of the encoding.

Parameters:

manager - the NDManager to create the NDList

withTokenType - true to include the token type id

Returns:

the NDList

getIds

public long[] getIds()

Returns the token ids.

Returns:

the token ids

getTypeIds

public long[] getTypeIds()

Returns the token type ids.

Returns:

the token type ids

getTokens

public String[] getTokens()

Returns the tokens.

Returns:

the tokens

getWordIds

public long[] getWordIds()

Returns the word ids.

Returns:

the word ids

getAttentionMask

public long[] getAttentionMask()

Returns the attention masks.

Returns:

the attention masks

getSpecialTokenMask

public long[] getSpecialTokenMask()

Returns the special token masks.

Returns:

the special token masks

getCharTokenSpans

public CharSpan[] getCharTokenSpans()

Returns char token spans.

Returns:

char token spans

exceedMaxLength

public boolean exceedMaxLength()

Returns if tokens exceed max length.

Returns:

true if tokens exceed max length

getOverflowing

public Encoding[] getOverflowing()

Returns an array of overflowing encodings.

Returns:

the array of overflowing encodings