ai.djl.modality.nlp.preprocess

Class TextTruncator

java.lang.Object

ai.djl.modality.nlp.preprocess.TextTruncator

All Implemented Interfaces:

TextProcessor

public class TextTruncator
extends java.lang.Object
implements TextProcessor

TextProcessor that truncates text to a maximum size.

Constructor Summary

Constructors

Constructor and Description
TextTruncator(int maxSize) Constructs a TextTruncator.

Method Summary

Modifier and Type	Method and Description
java.util.List<java.lang.String>	preprocess(java.util.List<java.lang.String> tokens) Applies the preprocessing defined to the given input tokens.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TextTruncator

public TextTruncator(int maxSize)

Constructs a TextTruncator.

Parameters:

maxSize - the size to limit the text to

Method Detail

preprocess

public java.util.List<java.lang.String> preprocess(java.util.List<java.lang.String> tokens)

Applies the preprocessing defined to the given input tokens.

Specified by:

preprocess in interface TextProcessor

Parameters:

tokens - the tokens created after the input text is tokenized

Returns:

the preprocessed tokens