Package ai.djl.huggingface.tokenizers

Class Encoding

java.lang.Object

ai.djl.huggingface.tokenizers.Encoding

public class Encoding
extends java.lang.Object

A class holds token encoding information.

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	Encoding(long[] ids, long[] typeIds, java.lang.String[] tokens, long[] wordIds, long[] attentionMask, long[] specialTokenMask, CharSpan[] charTokenSpans, Encoding[] overflowing)

Method Summary

Modifier and Type	Method	Description
long[]	getAttentionMask()	Returns the attention masks.
CharSpan[]	getCharTokenSpans()	Returns char token spans.
long[]	getIds()	Returns the token ids.
Encoding[]	getOverflowing()	Returns an array of overflowing encodings.
long[]	getSpecialTokenMask()	Returns the special token masks.
java.lang.String[]	getTokens()	Returns the tokens.
long[]	getTypeIds()	Returns the token type ids.
long[]	getWordIds()	Returns the word ids.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Encoding

protected Encoding(long[] ids,
                   long[] typeIds,
                   java.lang.String[] tokens,
                   long[] wordIds,
                   long[] attentionMask,
                   long[] specialTokenMask,
                   CharSpan[] charTokenSpans,
                   Encoding[] overflowing)

Method Detail

getIds

public long[] getIds()

Returns the token ids.

Returns:

the token ids

getTypeIds

public long[] getTypeIds()

Returns the token type ids.

Returns:

the token type ids

getTokens

public java.lang.String[] getTokens()

Returns the tokens.

Returns:

the tokens

getWordIds

public long[] getWordIds()

Returns the word ids.

Returns:

the word ids

getAttentionMask

public long[] getAttentionMask()

Returns the attention masks.

Returns:

the attention masks

getSpecialTokenMask

public long[] getSpecialTokenMask()

Returns the special token masks.

Returns:

the special token masks

getCharTokenSpans

public CharSpan[] getCharTokenSpans()

Returns char token spans.

Returns:

char token spans

getOverflowing

public Encoding[] getOverflowing()

Returns an array of overflowing encodings.

Returns:

the array of overflowing encodings