public final class LowerCaseTokenizer extends LetterTokenizer
Note: this does a decent job for most European languages, but does a terrible job for some Asian languages, where words are not separated by spaces.
You must specify the required Version
compatibility when creating
LowerCaseTokenizer
:
CharTokenizer
uses an int based API to normalize and
detect token characters. See CharTokenizer.isTokenChar(int)
and
CharTokenizer.normalize(int)
for details.AttributeSource.AttributeFactory, AttributeSource.State
Constructor and Description |
---|
LowerCaseTokenizer(Version matchVersion,
AttributeSource.AttributeFactory factory,
Reader in)
Construct a new LowerCaseTokenizer using a given
AttributeSource.AttributeFactory . |
LowerCaseTokenizer(Version matchVersion,
Reader in)
Construct a new LowerCaseTokenizer.
|
end, incrementToken, reset
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, restoreState, toString
public LowerCaseTokenizer(Version matchVersion, Reader in)
matchVersion
- Lucene version to match See abovein
- the input to split up into tokenspublic LowerCaseTokenizer(Version matchVersion, AttributeSource.AttributeFactory factory, Reader in)
AttributeSource.AttributeFactory
.Copyright © 2010 - 2020 Adobe. All Rights Reserved