Package ai.djl.modality.nlp
Interface Vocabulary
- All Known Implementing Classes:
DefaultVocabulary
public interface Vocabulary
Vocabulary
is a collection of tokens. The primary purpose of a vocabulary is the map a
token to an index.-
Method Summary
Modifier and TypeMethodDescriptionboolean
Check if the vocabulary contains a token.long
Returns the index of the given token.getToken
(long index) Returns the token corresponding to the given index.long
size()
Returns the size of theVocabulary
.
-
Method Details
-
getToken
Returns the token corresponding to the given index.- Parameters:
index
- the index- Returns:
- the token corresponding to the given index
-
contains
Check if the vocabulary contains a token.- Parameters:
token
- String token to be checked- Returns:
- whether this vocabulary contains the token
-
getIndex
Returns the index of the given token.- Parameters:
token
- the token- Returns:
- the index of the given token.
-
size
long size()Returns the size of theVocabulary
.- Returns:
- the size of the
Vocabulary
-