ai.djl.modality.nlp.preprocess

Class SentenceLengthNormalizer

java.lang.Object

ai.djl.modality.nlp.preprocess.SentenceLengthNormalizer

All Implemented Interfaces:

TextProcessor

public class SentenceLengthNormalizer
extends java.lang.Object
implements TextProcessor

SentenceLengthNormalizer normalizes the length of all the input sentences to the specified number of tokens.

If the number of tokens in the input sentence is higher than the fixed length, the sentence is truncated to the fixed number. If the number of tokens in the input sentence is fewer than the fixed sentence length, padding tokens are inserted to make the length equal to the sentence length.

Constructor Summary

Constructors

Constructor and Description
SentenceLengthNormalizer() Creates a TextProcessor that normalizes the length of the input.
SentenceLengthNormalizer(int sentenceLength, boolean addEosBosTokens) Creates a TextProcessor that normalizes the length of the input to the given sentence length.
SentenceLengthNormalizer(int sentenceLength, boolean addEosBosTokens, java.lang.String paddingToken, java.lang.String eosToken, java.lang.String bosToken) Creates a TextProcessor that normalizes the length of the input to the given sentence length.

Method Summary

Modifier and Type	Method and Description
int	getLastValidLength() Returns the valid length of the sentence that was last served as input to preprocess(List).
java.util.List<java.lang.String>	preprocess(java.util.List<java.lang.String> tokens) Applies the preprocessing defined to the given input tokens.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SentenceLengthNormalizer

public SentenceLengthNormalizer()

Creates a TextProcessor that normalizes the length of the input.

SentenceLengthNormalizer

public SentenceLengthNormalizer(int sentenceLength,
                                boolean addEosBosTokens)

Creates a TextProcessor that normalizes the length of the input to the given sentence length.

Parameters:

sentenceLength - the sentence length

addEosBosTokens - whether to add Eos and Bos tokens before normalizing sentence length

SentenceLengthNormalizer

public SentenceLengthNormalizer(int sentenceLength,
                                boolean addEosBosTokens,
                                java.lang.String paddingToken,
                                java.lang.String eosToken,
                                java.lang.String bosToken)

Creates a TextProcessor that normalizes the length of the input to the given sentence length.

Parameters:

sentenceLength - the sentence length

addEosBosTokens - whether to add Eos and Bos tokens before normalizing sentence length

paddingToken - the padding token to be used if the number of tokens in the input is less than sentence length

eosToken - the end of sentence token

bosToken - the begining of sentence token

Method Detail

preprocess

public java.util.List<java.lang.String> preprocess(java.util.List<java.lang.String> tokens)

Applies the preprocessing defined to the given input tokens.

Specified by:

preprocess in interface TextProcessor

Parameters:

tokens - the tokens created after the input text is tokenized

Returns:

the preprocessed tokens

getLastValidLength

public int getLastValidLength()

Returns the valid length of the sentence that was last served as input to preprocess(List). If no sentences preprocess before calling this method, it will -1.

Returns:

the valid length of the sentence that was last preprocessed