Package ai.djl.modality.nlp
Class DefaultVocabulary
java.lang.Object
ai.djl.modality.nlp.DefaultVocabulary
- All Implemented Interfaces:
Vocabulary
The default implementation of Vocabulary.
-
Nested Class Summary
Modifier and TypeClassDescriptionstatic final class
Builder class that is used to build theDefaultVocabulary
. -
Constructor Summary
ConstructorDescriptionCreates aDefaultVocabulary
object with aDefaultVocabulary.Builder
.DefaultVocabulary
(List<String> tokens) Creates aDefaultVocabulary
object with the given list of tokens. -
Method Summary
Modifier and TypeMethodDescriptionstatic DefaultVocabulary.Builder
builder()
Creates a new builder to build aDefaultVocabulary
.boolean
Check if the vocabulary contains a token.long
Returns the index of the given token.getToken
(long index) Returns the token corresponding to the given index.long
size()
Returns the size of theVocabulary
.
-
Constructor Details
-
DefaultVocabulary
Creates aDefaultVocabulary
object with the given list of tokens.- Parameters:
tokens
- theList
of tokens to build the vocabulary with
-
DefaultVocabulary
Creates aDefaultVocabulary
object with aDefaultVocabulary.Builder
.- Parameters:
builder
- theDefaultVocabulary.Builder
to build the vocabulary with
-
-
Method Details
-
contains
Check if the vocabulary contains a token.- Specified by:
contains
in interfaceVocabulary
- Parameters:
token
- String token to be checked- Returns:
- whether this vocabulary contains the token
-
getToken
Returns the token corresponding to the given index.- Specified by:
getToken
in interfaceVocabulary
- Parameters:
index
- the index- Returns:
- the token corresponding to the given index
-
getIndex
Returns the index of the given token.- Specified by:
getIndex
in interfaceVocabulary
- Parameters:
token
- the token- Returns:
- the index of the given token.
-
size
public long size()Returns the size of theVocabulary
.- Specified by:
size
in interfaceVocabulary
- Returns:
- the size of the
Vocabulary
-
builder
Creates a new builder to build aDefaultVocabulary
.- Returns:
- a new builder
-