ai.djl.modality.nlp

Class SimpleVocabulary.Builder

java.lang.Object

ai.djl.modality.nlp.SimpleVocabulary.Builder

Enclosing class:

SimpleVocabulary

public static final class SimpleVocabulary.Builder
extends java.lang.Object

Builder class that is used to build the SimpleVocabulary.

Method Summary

Modifier and Type	Method and Description
SimpleVocabulary.Builder	add(java.util.List<java.lang.String> sentence) Adds the given sentence to the SimpleVocabulary.
SimpleVocabulary.Builder	addAll(java.util.List<java.util.List<java.lang.String>> sentences) Adds the given list of sentences to the SimpleVocabulary.
SimpleVocabulary.Builder	addFromCustomizedFile(java.net.URL url, java.util.function.Function<java.net.URL,java.util.List<java.lang.String>> lambda) Adds a customized vocabulary to the SimpleVocabulary.
SimpleVocabulary.Builder	addFromTextFile(java.nio.file.Path path) Adds a text vocabulary to the SimpleVocabulary.
SimpleVocabulary.Builder	addFromTextFile(java.net.URL url) Adds a text vocabulary to the SimpleVocabulary.
SimpleVocabulary	build() Builds the SimpleVocabulary object with the set arguments.
SimpleVocabulary.Builder	optMinFrequency(int minFrequency) Sets the optional parameter that specifies the minimum frequency to consider a token to be part of the SimpleVocabulary.
SimpleVocabulary.Builder	optReservedTokens(java.util.Collection<java.lang.String> reservedTokens) Sets the optional parameter that sets the list of reserved tokens.
SimpleVocabulary.Builder	optUnknownToken(java.lang.String unknownToken) Sets the optional parameter that specifies the unknown token's string value.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

optMinFrequency

public SimpleVocabulary.Builder optMinFrequency(int minFrequency)

Sets the optional parameter that specifies the minimum frequency to consider a token to be part of the SimpleVocabulary. Defaults to 10.

Parameters:

minFrequency - the minimum frequency to consider a token to be part of the SimpleVocabulary

Returns:

this VocabularyBuilder

optUnknownToken

public SimpleVocabulary.Builder optUnknownToken(java.lang.String unknownToken)

Sets the optional parameter that specifies the unknown token's string value.

Parameters:

unknownToken - the string value of the unknown token

Returns:

this VocabularyBuilder

optReservedTokens

public SimpleVocabulary.Builder optReservedTokens(java.util.Collection<java.lang.String> reservedTokens)

Sets the optional parameter that sets the list of reserved tokens.

Parameters:

reservedTokens - the list of reserved tokens

Returns:

this VocabularyBuilder

add

public SimpleVocabulary.Builder add(java.util.List<java.lang.String> sentence)

Adds the given sentence to the SimpleVocabulary.

Parameters:

sentence - the sentence to be added

Returns:

this VocabularyBuilder

addAll

public SimpleVocabulary.Builder addAll(java.util.List<java.util.List<java.lang.String>> sentences)

Adds the given list of sentences to the SimpleVocabulary.

Parameters:

sentences - the list of sentences to be added

Returns:

this VocabularyBuilder

addFromTextFile

public SimpleVocabulary.Builder addFromTextFile(java.nio.file.Path path)
                                         throws java.io.IOException

Adds a text vocabulary to the SimpleVocabulary.

   Example text file(vocab.txt):
   token1
   token2
   token3
   will be mapped to index of 0 1 2
 

Parameters:

path - the path to the text file

Returns:

this VocabularyBuilder

Throws:

java.io.IOException - if failed to read vocabulary file

addFromTextFile

public SimpleVocabulary.Builder addFromTextFile(java.net.URL url)
                                         throws java.io.IOException

Adds a text vocabulary to the SimpleVocabulary.

Parameters:

url - the text file url

Returns:

this VocabularyBuilder

Throws:

java.io.IOException - if failed to read vocabulary file

addFromCustomizedFile

public SimpleVocabulary.Builder addFromCustomizedFile(java.net.URL url,
                                                      java.util.function.Function<java.net.URL,java.util.List<java.lang.String>> lambda)

Adds a customized vocabulary to the SimpleVocabulary.

Parameters:

url - the text file url

lambda - the function to parse the vocabulary file

Returns:

this VocabularyBuilder

build

public SimpleVocabulary build()

Builds the SimpleVocabulary object with the set arguments.

Returns:

the SimpleVocabulary object built