Class Tokenizer
- java.lang.Object
-
- com.aspectran.core.context.expr.token.Tokenizer
-
public class Tokenizer extends java.lang.Object
The Class Tokenizer.Created: 2008. 03. 29 AM 1:55:03
-
-
Constructor Summary
Constructors Constructor Description Tokenizer()
-
Method Summary
All Methods Static Methods Concrete Methods Modifier and Type Method Description static Token[]
optimize(Token[] tokens)
Returns an array of tokens that is optimized.static java.util.List<Token>
tokenize(java.lang.CharSequence input, boolean textTrim)
Returns a list of tokens that contains tokenized string.
-
-
-
Method Detail
-
tokenize
public static java.util.List<Token> tokenize(java.lang.CharSequence input, boolean textTrim)
Returns a list of tokens that contains tokenized string.- Parameters:
input
- the string to tokenizetextTrim
- whether to trim text- Returns:
- a list of tokens
-
-