Package com.yahoo.language.process
Class SpecialTokenRegistry
java.lang.Object
com.yahoo.language.process.SpecialTokenRegistry
Immutable named lists of "special tokens" - strings which should override the normal tokenizer semantics
and be tokenized into a single token.
- Author:
- bratseth
-
Constructor Summary
ConstructorDescriptionCreates an empty special token registrySpecialTokenRegistry
(com.yahoo.vespa.configdefinition.SpecialtokensConfig config) Create a special token registry from a configuration object.SpecialTokenRegistry
(List<SpecialTokens> specialTokensList) -
Method Summary
Modifier and TypeMethodDescriptiongetSpecialTokens
(String name) Returns the list of special tokens for a given name.
-
Constructor Details
-
SpecialTokenRegistry
public SpecialTokenRegistry()Creates an empty special token registry -
SpecialTokenRegistry
public SpecialTokenRegistry(com.yahoo.vespa.configdefinition.SpecialtokensConfig config) Create a special token registry from a configuration object. -
SpecialTokenRegistry
-
-
Method Details
-
getSpecialTokens
Returns the list of special tokens for a given name.- Parameters:
name
- the name of the special tokens to return null, the empty string or the string "default" returns the default ones- Returns:
- a read-only list of SpecialToken instances, an empty list if this name has no special tokens
-