Package ai.djl.modality.nlp.preprocess

Class TextTerminator

java.lang.Object

ai.djl.modality.nlp.preprocess.TextTerminator

All Implemented Interfaces:

TextProcessor

public class TextTerminator
extends Object
implements TextProcessor

A TextProcessor that adds a beginning of string and end of string token.

Constructor Summary

Constructors

Constructor	Description
TextTerminator()	Constructs a default TextTerminator.
TextTerminator(boolean addBosToken, boolean addEosToken)	Constructs a TextTerminator using the default tokens.
TextTerminator(boolean addBosToken, boolean addEosToken, String bosToken, String eosToken)	Constructs a TextTerminator.

Method Summary

Modifier and Type	Method	Description
List<String>	preprocess(List<String> tokens)	Applies the preprocessing defined to the given input tokens.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

TextTerminator

public TextTerminator()

Constructs a default TextTerminator.

TextTerminator

public TextTerminator(boolean addBosToken,
 boolean addEosToken)

Constructs a TextTerminator using the default tokens.

Parameters:

addBosToken - true to add a beginning of text token

addEosToken - true to add an end of text token

TextTerminator

public TextTerminator(boolean addBosToken,
 boolean addEosToken,
 String bosToken,
 String eosToken)

Constructs a TextTerminator.

Parameters:

addBosToken - true to add a beginning of text token

addEosToken - true to add an end of text token

bosToken - the token to add to the beginning of the text

eosToken - the token to add to the end of the text

Method Details

preprocess

public List<String> preprocess(List<String> tokens)

Applies the preprocessing defined to the given input tokens.

Specified by:

preprocess in interface TextProcessor

Parameters:

tokens - the tokens created after the input text is tokenized

Returns:

the preprocessed tokens