ai.djl.modality.nlp

Class Vocabulary.VocabularyBuilder

java.lang.Object

ai.djl.modality.nlp.Vocabulary.VocabularyBuilder

Enclosing class:

Vocabulary

public static class Vocabulary.VocabularyBuilder
extends java.lang.Object

Builder class that is used to build the Vocabulary.

Field Summary

Fields

Modifier and Type	Field and Description
protected int	minFrequency
protected java.util.Set<java.lang.String>	reservedTokens
protected java.util.List<java.util.List<java.lang.String>>	sentences
protected java.lang.String	unknownToken

Constructor Summary

Constructors

Constructor and Description
VocabularyBuilder()

Method Summary

Modifier and Type	Method and Description
Vocabulary.VocabularyBuilder	add(java.util.List<java.lang.String> sentence) Adds the given sentence to the Vocabulary.
Vocabulary.VocabularyBuilder	addAll(java.util.List<java.util.List<java.lang.String>> sentences) Adds the given list of sentences to the Vocabulary.
Vocabulary	build() Builds the Vocabulary object with the set arguments.
Vocabulary.VocabularyBuilder	optMinFrequency(int minFrequency) Sets the optional parameter that specifies the minimum frequency to consider a token to be part of the Vocabulary.
Vocabulary.VocabularyBuilder	optReservedTokens(java.util.Collection<java.lang.String> reservedTokens) Sets the optional parameter that sets the list of reserved tokens.
Vocabulary.VocabularyBuilder	optUnknownToken(java.lang.String unknownToken) Sets the optional parameter that specifies the unknown token's string value.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

sentences

protected java.util.List<java.util.List<java.lang.String>> sentences

reservedTokens

protected java.util.Set<java.lang.String> reservedTokens

minFrequency

protected int minFrequency

unknownToken

protected java.lang.String unknownToken

Constructor Detail

VocabularyBuilder

public VocabularyBuilder()

Method Detail

optMinFrequency

public Vocabulary.VocabularyBuilder optMinFrequency(int minFrequency)

Sets the optional parameter that specifies the minimum frequency to consider a token to be part of the Vocabulary. Defaults to 10.

Parameters:

minFrequency - the minimum frequency to consider a token to be part of the Vocabulary

Returns:

this VocabularyBuilder

optUnknownToken

public Vocabulary.VocabularyBuilder optUnknownToken(java.lang.String unknownToken)

Sets the optional parameter that specifies the unknown token's string value.

Parameters:

unknownToken - the string value of the unknown token

Returns:

this VocabularyBuilder

optReservedTokens

public Vocabulary.VocabularyBuilder optReservedTokens(java.util.Collection<java.lang.String> reservedTokens)

Sets the optional parameter that sets the list of reserved tokens.

Parameters:

reservedTokens - the list of reserved tokens

Returns:

this VocabularyBuilder

add

public Vocabulary.VocabularyBuilder add(java.util.List<java.lang.String> sentence)

Adds the given sentence to the Vocabulary.

Parameters:

sentence - the sentence to be added

Returns:

this VocabularyBuilder

addAll

public Vocabulary.VocabularyBuilder addAll(java.util.List<java.util.List<java.lang.String>> sentences)

Adds the given list of sentences to the Vocabulary.

Parameters:

sentences - the list of sentences to be added

Returns:

this VocabularyBuilder

build

public Vocabulary build()

Builds the Vocabulary object with the set arguments.

Returns:

the Vocabulary object built