@API(value=EXPERIMENTAL) public class DefaultTextTokenizer extends Object implements TextTokenizer
TextTokenizer.TokenizerMode
Modifier and Type | Field and Description |
---|---|
static String |
NAME
The name of the default tokenizer.
|
GLOBAL_MIN_VERSION
Modifier and Type | Method and Description |
---|---|
int |
getMaxVersion()
Get the maximum supported version.
|
String |
getName()
Get the name for this tokenizer.
|
static DefaultTextTokenizer |
instance()
Get this class's singleton.
|
Iterator<String> |
tokenize(String text,
int version,
TextTokenizer.TokenizerMode mode)
Tokenize the text based on whitespace.
|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
getMinVersion, tokenizeToList, tokenizeToMap, validateVersion
@Nonnull public static final String NAME
@Nonnull public static DefaultTextTokenizer instance()
@Nonnull public Iterator<String> tokenize(@Nonnull String text, int version, @Nonnull TextTokenizer.TokenizerMode mode)
tokenize
in interface TextTokenizer
text
- source text to splitversion
- version of the tokenizer to use to split the textmode
- ignored as this tokenizer operates the same way at index and query time@Nonnull public String getName()
getName
in interface TextTokenizer
public int getMaxVersion()
getMaxVersion
in interface TextTokenizer