Class HyphenationCompoundWordTokenFilter
- java.lang.Object
-
- org.apache.lucene.util.AttributeSource
-
- org.apache.lucene.analysis.TokenStream
-
- org.apache.lucene.analysis.TokenFilter
-
- org.apache.lucene.analysis.compound.CompoundWordTokenFilterBase
-
- org.apache.lucene.analysis.compound.HyphenationCompoundWordTokenFilter
-
- All Implemented Interfaces:
java.io.Closeable
,java.lang.AutoCloseable
public class HyphenationCompoundWordTokenFilter extends CompoundWordTokenFilterBase
ATokenFilter
that decomposes compound words found in many Germanic languages."Donaudampfschiff" becomes Donau, dampf, schiff so that you can find "Donaudampfschiff" even when you only enter "schiff". It uses a hyphenation grammar and a word dictionary to achieve this.
You must specify the required
Version
compatibility when creating CompoundWordTokenFilterBase:- As of 3.1, CompoundWordTokenFilterBase correctly handles Unicode 4.0 supplementary characters in strings and char arrays provided as compound word dictionaries.
-
-
Nested Class Summary
-
Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource
AttributeSource.AttributeFactory, AttributeSource.State
-
-
Field Summary
-
Fields inherited from class org.apache.lucene.analysis.compound.CompoundWordTokenFilterBase
DEFAULT_MAX_SUBWORD_SIZE, DEFAULT_MIN_SUBWORD_SIZE, DEFAULT_MIN_WORD_SIZE
-
-
Constructor Summary
Constructors Constructor Description HyphenationCompoundWordTokenFilter(Version matchVersion, TokenStream input, HyphenationTree hyphenator)
Create a HyphenationCompoundWordTokenFilter with no dictionary.HyphenationCompoundWordTokenFilter(Version matchVersion, TokenStream input, HyphenationTree hyphenator, int minWordSize, int minSubwordSize, int maxSubwordSize)
Create a HyphenationCompoundWordTokenFilter with no dictionary.HyphenationCompoundWordTokenFilter(Version matchVersion, TokenStream input, HyphenationTree hyphenator, CharArraySet dictionary)
Creates a newHyphenationCompoundWordTokenFilter
instance.HyphenationCompoundWordTokenFilter(Version matchVersion, TokenStream input, HyphenationTree hyphenator, CharArraySet dictionary, int minWordSize, int minSubwordSize, int maxSubwordSize, boolean onlyLongestMatch)
Creates a newHyphenationCompoundWordTokenFilter
instance.
-
Method Summary
All Methods Static Methods Concrete Methods Modifier and Type Method Description static HyphenationTree
getHyphenationTree(java.io.File hyphenationFile)
Create a hyphenator treestatic HyphenationTree
getHyphenationTree(java.lang.String hyphenationFilename)
Create a hyphenator treestatic HyphenationTree
getHyphenationTree(org.xml.sax.InputSource hyphenationSource)
Create a hyphenator tree-
Methods inherited from class org.apache.lucene.analysis.compound.CompoundWordTokenFilterBase
incrementToken, reset
-
Methods inherited from class org.apache.lucene.analysis.TokenFilter
close, end
-
Methods inherited from class org.apache.lucene.util.AttributeSource
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, restoreState, toString
-
-
-
-
Constructor Detail
-
HyphenationCompoundWordTokenFilter
public HyphenationCompoundWordTokenFilter(Version matchVersion, TokenStream input, HyphenationTree hyphenator, CharArraySet dictionary)
Creates a newHyphenationCompoundWordTokenFilter
instance.- Parameters:
matchVersion
- Lucene version to enable correct Unicode 4.0 behavior in the dictionaries if Version > 3.0. See CompoundWordTokenFilterBase for details.input
- theTokenStream
to processhyphenator
- the hyphenation pattern tree to use for hyphenationdictionary
- the word dictionary to match against.
-
HyphenationCompoundWordTokenFilter
public HyphenationCompoundWordTokenFilter(Version matchVersion, TokenStream input, HyphenationTree hyphenator, CharArraySet dictionary, int minWordSize, int minSubwordSize, int maxSubwordSize, boolean onlyLongestMatch)
Creates a newHyphenationCompoundWordTokenFilter
instance.- Parameters:
matchVersion
- Lucene version to enable correct Unicode 4.0 behavior in the dictionaries if Version > 3.0. See CompoundWordTokenFilterBase for details.input
- theTokenStream
to processhyphenator
- the hyphenation pattern tree to use for hyphenationdictionary
- the word dictionary to match against.minWordSize
- only words longer than this get processedminSubwordSize
- only subwords longer than this get to the output streammaxSubwordSize
- only subwords shorter than this get to the output streamonlyLongestMatch
- Add only the longest matching subword to the stream
-
HyphenationCompoundWordTokenFilter
public HyphenationCompoundWordTokenFilter(Version matchVersion, TokenStream input, HyphenationTree hyphenator, int minWordSize, int minSubwordSize, int maxSubwordSize)
Create a HyphenationCompoundWordTokenFilter with no dictionary.
-
HyphenationCompoundWordTokenFilter
public HyphenationCompoundWordTokenFilter(Version matchVersion, TokenStream input, HyphenationTree hyphenator)
Create a HyphenationCompoundWordTokenFilter with no dictionary.
-
-
Method Detail
-
getHyphenationTree
public static HyphenationTree getHyphenationTree(java.lang.String hyphenationFilename) throws java.io.IOException
Create a hyphenator tree- Parameters:
hyphenationFilename
- the filename of the XML grammar to load- Returns:
- An object representing the hyphenation patterns
- Throws:
java.io.IOException
- If there is a low-level I/O error.
-
getHyphenationTree
public static HyphenationTree getHyphenationTree(java.io.File hyphenationFile) throws java.io.IOException
Create a hyphenator tree- Parameters:
hyphenationFile
- the file of the XML grammar to load- Returns:
- An object representing the hyphenation patterns
- Throws:
java.io.IOException
- If there is a low-level I/O error.
-
getHyphenationTree
public static HyphenationTree getHyphenationTree(org.xml.sax.InputSource hyphenationSource) throws java.io.IOException
Create a hyphenator tree- Parameters:
hyphenationSource
- the InputSource pointing to the XML grammar- Returns:
- An object representing the hyphenation patterns
- Throws:
java.io.IOException
- If there is a low-level I/O error.
-
-