ai.djl.modality.nlp

Class Vocabulary

java.lang.Object

ai.djl.modality.nlp.Vocabulary

public class Vocabulary
extends java.lang.Object

Vocabulary is a collection of tokens. The primary purpose of a vocabulary is the map a token to an index.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	Vocabulary.VocabularyBuilder Builder class that is used to build the Vocabulary.

Constructor Summary

Constructors

Constructor and Description
Vocabulary(Vocabulary.VocabularyBuilder builder) Create a Vocabulary object with ta Vocabulary.VocabularyBuilder.

Method Summary

Modifier and Type	Method and Description
long	getIndex(java.lang.String token) Returns the index of the given token.
java.lang.String	getToken(int index) Returns the token corresponding to the given index.
java.lang.String	getUnknownToken() Return a String used for unseen or rarely-seen tokens.
boolean	isKnownToken(java.lang.String token) Returns whether the given token is a known word.
int	size() Returns the size of the Vocabulary.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Vocabulary

public Vocabulary(Vocabulary.VocabularyBuilder builder)

Create a Vocabulary object with ta Vocabulary.VocabularyBuilder.

Parameters:

builder - the Vocabulary.VocabularyBuilder to build the vocabulary with

Method Detail

isKnownToken

public boolean isKnownToken(java.lang.String token)

Returns whether the given token is a known word.

Parameters:

token - the token

Returns:

whether the given token is a known word

getUnknownToken

public java.lang.String getUnknownToken()

Return a String used for unseen or rarely-seen tokens.

Returns:

the String used for unseen or rarely-seen tokens

getToken

public java.lang.String getToken(int index)

Returns the token corresponding to the given index.

Parameters:

index - the index

Returns:

the token corresponding to the given index

getIndex

public long getIndex(java.lang.String token)

Returns the index of the given token.

Parameters:

token - the token

Returns:

the index of the given token

size

public int size()

Returns the size of the Vocabulary.

Returns:

the size of the Vocabulary