-
Interface Summary
Interface |
Description |
TextProcessor |
TextProcessor allows applying pre-processing to input tokens for natural language
applications.
|
Tokenizer |
Tokenizer interface provides the ability to break-down sentences into embeddable tokens.
|
-
Class Summary
Class |
Description |
LowerCaseConvertor |
LowerCaseConvertor converts every character of the input tokens to it's respective lower
case character.
|
PunctuationSeparator |
PunctuationSeparator converts every character of the input tokens to it's respective
lower case character.
|
SentenceLengthNormalizer |
SentenceLengthNormalizer normalizes the length of all the input sentences to the
specified number of tokens.
|
SimpleTokenizer |
SimpleTokenizer is an implementation of the Tokenizer interface that converts
sentences into token by splitting them by a given delimiter.
|
Package ai.djl.modality.nlp.preprocess Description
Contains utility classes for natural language pre-processing tasks.