ai.djl.modality.nlp

Interface Vocabulary

All Known Implementing Classes:

DefaultVocabulary

public interface Vocabulary

Vocabulary is a collection of tokens. The primary purpose of a vocabulary is the map a token to an index.

Method Summary

Modifier and Type	Method and Description
boolean	contains(java.lang.String token) Check if the vocabulary contains a token.
long	getIndex(java.lang.String token) Returns the index of the given token.
java.lang.String	getToken(long index) Returns the token corresponding to the given index.
long	size() Returns the size of the Vocabulary.

Method Detail

getToken

java.lang.String getToken(long index)

Returns the token corresponding to the given index.

Parameters:

index - the index

Returns:

the token corresponding to the given index

contains

boolean contains(java.lang.String token)

Check if the vocabulary contains a token.

Parameters:

token - String token to be checked

Returns:

whether this vocabulary contains the token

getIndex

long getIndex(java.lang.String token)

Returns the index of the given token.

Parameters:

token - the token

Returns:

the index of the given token.

size

long size()

Returns the size of the Vocabulary.

Returns:

the size of the Vocabulary