Package com.yahoo.language.process
Class SpecialTokens
java.lang.Object
com.yahoo.language.process.SpecialTokens
An immutable list of special tokens - strings which should override the normal tokenizer semantics
and be tokenized into a single token. Special tokens are case insensitive.
- Author:
- bratseth
-
Nested Class Summary
Modifier and TypeClassDescriptionstatic final class
An immutable special token -
Constructor Summary
-
Method Summary
Modifier and TypeMethodDescriptionasMap()
Returns the tokens of this as an immutable map from token to replacement.static SpecialTokens
empty()
name()
Returns the name of this special tokens listReturns the special token starting at the start of the given string, or null if no special token starts at this string
-
Constructor Details
-
SpecialTokens
-
-
Method Details
-
name
Returns the name of this special tokens list -
asMap
Returns the tokens of this as an immutable map from token to replacement. Tokens which do not have a replacement token maps to themselves. -
tokenize
Returns the special token starting at the start of the given string, or null if no special token starts at this string- Parameters:
string
- the string to search for a special token at the start positionsubstring
- true to allow the special token to be followed by a character which does not mark the end of a token
-
empty
-