Package ai.djl.modality.nlp.preprocess

Class TextTerminator

java.lang.Object

ai.djl.modality.nlp.preprocess.TextTerminator

All Implemented Interfaces:

TextProcessor

public class TextTerminator
extends java.lang.Object
implements TextProcessor

A TextProcessor that adds a beginning of string and end of string token.

Constructor Summary

Constructors

Constructor	Description
TextTerminator()	Constructs a default TextTerminator.
TextTerminator(boolean addBosToken, boolean addEosToken)	Constructs a TextTerminator using the default tokens.
TextTerminator(boolean addBosToken, boolean addEosToken, java.lang.String bosToken, java.lang.String eosToken)	Constructs a TextTerminator.

Method Summary

Modifier and Type	Method	Description
java.util.List<java.lang.String>	preprocess(java.util.List<java.lang.String> tokens)	Applies the preprocessing defined to the given input tokens.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TextTerminator

public TextTerminator()

Constructs a default TextTerminator.

TextTerminator

public TextTerminator(boolean addBosToken,
                      boolean addEosToken)

Constructs a TextTerminator using the default tokens.

Parameters:

addBosToken - true to add a beginning of text token

addEosToken - true to add an end of text token

TextTerminator

public TextTerminator(boolean addBosToken,
                      boolean addEosToken,
                      java.lang.String bosToken,
                      java.lang.String eosToken)

Constructs a TextTerminator.

Parameters:

addBosToken - true to add a beginning of text token

addEosToken - true to add an end of text token

bosToken - the token to add to the beginning of the text

eosToken - the token to add to the end of the text

Method Detail

preprocess

public java.util.List<java.lang.String> preprocess(java.util.List<java.lang.String> tokens)

Applies the preprocessing defined to the given input tokens.

Specified by:

preprocess in interface TextProcessor

Parameters:

tokens - the tokens created after the input text is tokenized

Returns:

the preprocessed tokens