public class PunctuationSeparator extends java.lang.Object implements TextProcessor
PunctuationSeparator
separates punctuation into a separate token.Constructor and Description |
---|
PunctuationSeparator() |
Modifier and Type | Method and Description |
---|---|
java.util.List<java.lang.String> |
preprocess(java.util.List<java.lang.String> tokens)
Applies the preprocessing defined to the given input tokens.
|
public java.util.List<java.lang.String> preprocess(java.util.List<java.lang.String> tokens)
preprocess
in interface TextProcessor
tokens
- the tokens created after the input text is tokenized