ai.djl.modality.nlp.preprocess

Interface Tokenizer

All Known Implementing Classes:

SimpleTokenizer

public interface Tokenizer

Tokenizer interface provides the ability to break-down sentences into embeddable tokens.

Method Summary

Modifier and Type	Method and Description
java.lang.String	buildSentence(java.util.List<java.lang.String> tokens) Combines a list of tokens to form a sentence.
java.util.List<java.lang.String>	tokenize(java.lang.String sentence) Breaks down the given sentence into a list of tokens that can be represented by embeddings.

Method Detail

tokenize

java.util.List<java.lang.String> tokenize(java.lang.String sentence)

Breaks down the given sentence into a list of tokens that can be represented by embeddings.

Parameters:

sentence - the sentence to tokenize

Returns:

a List of tokens

buildSentence

java.lang.String buildSentence(java.util.List<java.lang.String> tokens)

Combines a list of tokens to form a sentence.

Parameters:

tokens - the List of tokens

Returns:

the sentence built from the given tokens