Class WordDelimiterFilter
- java.lang.Object
-
- org.apache.lucene.util.AttributeSource
-
- org.apache.lucene.analysis.TokenStream
-
- org.apache.lucene.analysis.TokenFilter
-
- org.apache.lucene.analysis.miscellaneous.WordDelimiterFilter
-
- All Implemented Interfaces:
java.io.Closeable
,java.lang.AutoCloseable
public final class WordDelimiterFilter extends TokenFilter
Splits words into subwords and performs optional transformations on subword groups. Words are split into subwords with the following rules:- split on intra-word delimiters (by default, all non alpha-numeric
characters):
"Wi-Fi"
→"Wi", "Fi"
- split on case transitions:
"PowerShot"
→"Power", "Shot"
- split on letter-number transitions:
"SD500"
→"SD", "500"
- leading and trailing intra-word delimiters on each subword are ignored:
"//hello---there, 'dude'"
→"hello", "there", "dude"
- trailing "'s" are removed for each subword:
"O'Neil's"
→"O", "Neil"
- Note: this step isn't performed in a separate filter because of possible subword combinations.
- combinations="0" causes no subword combinations:
"PowerShot"
→0:"Power", 1:"Shot"
(0 and 1 are the token positions) - combinations="1" means that in addition to the subwords, maximum runs of
non-numeric subwords are catenated and produced at the same position of the
last subword in the run:
"PowerShot"
→0:"Power", 1:"Shot" 1:"PowerShot"
"A's+B's&C's"
-gt;0:"A", 1:"B", 2:"C", 2:"ABC"
"Super-Duper-XL500-42-AutoCoder!"
→0:"Super", 1:"Duper", 2:"XL", 2:"SuperDuperXL", 3:"500" 4:"42", 5:"Auto", 6:"Coder", 6:"AutoCoder"
WordDelimiterFilter
is to help match words with different subword delimiters. For example, if the source text contained "wi-fi" one may want "wifi" "WiFi" "wi-fi" "wi+fi" queries to all match. One way of doing so is to specify combinations="1" in the analyzer used for indexing, and combinations="0" (the default) in the analyzer used for querying. Given that the currentStandardTokenizer
immediately removes many intra-word delimiters, it is recommended that this filter be used after a tokenizer that does not do this (such asWhitespaceTokenizer
).
-
-
Nested Class Summary
-
Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource
AttributeSource.AttributeFactory, AttributeSource.State
-
-
Field Summary
Fields Modifier and Type Field Description static int
ALPHA
static int
ALPHANUM
static int
CATENATE_ALL
Causes all subword parts to be catenated:static int
CATENATE_NUMBERS
Causes maximum runs of word parts to be catenated:static int
CATENATE_WORDS
Causes maximum runs of word parts to be catenated:static int
DIGIT
static int
GENERATE_NUMBER_PARTS
Causes number subwords to be generated:static int
GENERATE_WORD_PARTS
Causes parts of words to be generated:static int
LOWER
static int
PRESERVE_ORIGINAL
Causes original words are preserved and added to the subword list (Defaults to false)static int
SPLIT_ON_CASE_CHANGE
If not set, causes case changes to be ignored (subwords will only be generated given SUBWORD_DELIM tokens)static int
SPLIT_ON_NUMERICS
If not set, causes numeric changes to be ignored (subwords will only be generated given SUBWORD_DELIM tokens).static int
STEM_ENGLISH_POSSESSIVE
Causes trailing "'s" to be removed for each subwordstatic int
SUBWORD_DELIM
static int
UPPER
-
Constructor Summary
Constructors Constructor Description WordDelimiterFilter(TokenStream in, byte[] charTypeTable, int configurationFlags, CharArraySet protWords)
Creates a new WordDelimiterFilterWordDelimiterFilter(TokenStream in, int configurationFlags, CharArraySet protWords)
Creates a new WordDelimiterFilter usingWordDelimiterIterator.DEFAULT_WORD_DELIM_TABLE
as its charTypeTable
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description boolean
incrementToken()
Consumers (i.e.,IndexWriter
) use this method to advance the stream to the next token.void
reset()
This method is called by a consumer before it begins consumption usingTokenStream.incrementToken()
.-
Methods inherited from class org.apache.lucene.analysis.TokenFilter
close, end
-
Methods inherited from class org.apache.lucene.util.AttributeSource
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, restoreState, toString
-
-
-
-
Field Detail
-
LOWER
public static final int LOWER
- See Also:
- Constant Field Values
-
UPPER
public static final int UPPER
- See Also:
- Constant Field Values
-
DIGIT
public static final int DIGIT
- See Also:
- Constant Field Values
-
SUBWORD_DELIM
public static final int SUBWORD_DELIM
- See Also:
- Constant Field Values
-
ALPHA
public static final int ALPHA
- See Also:
- Constant Field Values
-
ALPHANUM
public static final int ALPHANUM
- See Also:
- Constant Field Values
-
GENERATE_WORD_PARTS
public static final int GENERATE_WORD_PARTS
Causes parts of words to be generated: "PowerShot" => "Power" "Shot"- See Also:
- Constant Field Values
-
GENERATE_NUMBER_PARTS
public static final int GENERATE_NUMBER_PARTS
Causes number subwords to be generated: "500-42" => "500" "42"- See Also:
- Constant Field Values
-
CATENATE_WORDS
public static final int CATENATE_WORDS
Causes maximum runs of word parts to be catenated: "wi-fi" => "wifi"- See Also:
- Constant Field Values
-
CATENATE_NUMBERS
public static final int CATENATE_NUMBERS
Causes maximum runs of word parts to be catenated: "wi-fi" => "wifi"- See Also:
- Constant Field Values
-
CATENATE_ALL
public static final int CATENATE_ALL
Causes all subword parts to be catenated: "wi-fi-4000" => "wifi4000"- See Also:
- Constant Field Values
-
PRESERVE_ORIGINAL
public static final int PRESERVE_ORIGINAL
Causes original words are preserved and added to the subword list (Defaults to false) "500-42" => "500" "42" "500-42"- See Also:
- Constant Field Values
-
SPLIT_ON_CASE_CHANGE
public static final int SPLIT_ON_CASE_CHANGE
If not set, causes case changes to be ignored (subwords will only be generated given SUBWORD_DELIM tokens)- See Also:
- Constant Field Values
-
SPLIT_ON_NUMERICS
public static final int SPLIT_ON_NUMERICS
If not set, causes numeric changes to be ignored (subwords will only be generated given SUBWORD_DELIM tokens).- See Also:
- Constant Field Values
-
STEM_ENGLISH_POSSESSIVE
public static final int STEM_ENGLISH_POSSESSIVE
Causes trailing "'s" to be removed for each subword "O'Neil's" => "O", "Neil"- See Also:
- Constant Field Values
-
-
Constructor Detail
-
WordDelimiterFilter
public WordDelimiterFilter(TokenStream in, byte[] charTypeTable, int configurationFlags, CharArraySet protWords)
Creates a new WordDelimiterFilter- Parameters:
in
- TokenStream to be filteredcharTypeTable
- table containing character typesconfigurationFlags
- Flags configuring the filterprotWords
- If not null is the set of tokens to protect from being delimited
-
WordDelimiterFilter
public WordDelimiterFilter(TokenStream in, int configurationFlags, CharArraySet protWords)
Creates a new WordDelimiterFilter usingWordDelimiterIterator.DEFAULT_WORD_DELIM_TABLE
as its charTypeTable- Parameters:
in
- TokenStream to be filteredconfigurationFlags
- Flags configuring the filterprotWords
- If not null is the set of tokens to protect from being delimited
-
-
Method Detail
-
incrementToken
public boolean incrementToken() throws java.io.IOException
Description copied from class:TokenStream
Consumers (i.e.,IndexWriter
) use this method to advance the stream to the next token. Implementing classes must implement this method and update the appropriateAttributeImpl
s with the attributes of the next token.The producer must make no assumptions about the attributes after the method has been returned: the caller may arbitrarily change it. If the producer needs to preserve the state for subsequent calls, it can use
AttributeSource.captureState()
to create a copy of the current attribute state.This method is called for every token of a document, so an efficient implementation is crucial for good performance. To avoid calls to
AttributeSource.addAttribute(Class)
andAttributeSource.getAttribute(Class)
, references to allAttributeImpl
s that this stream uses should be retrieved during instantiation.To ensure that filters and consumers know which attributes are available, the attributes must be added during instantiation. Filters and consumers are not required to check for availability of attributes in
TokenStream.incrementToken()
.- Specified by:
incrementToken
in classTokenStream
- Returns:
- false for end of stream; true otherwise
- Throws:
java.io.IOException
-
reset
public void reset() throws java.io.IOException
This method is called by a consumer before it begins consumption usingTokenStream.incrementToken()
.Resets this stream to a clean state. Stateful implementations must implement this method so that they can be reused, just as if they had been created fresh.
If you override this method, always call
super.reset()
, otherwise some internal state will not be correctly reset (e.g.,Tokenizer
will throwIllegalStateException
on further usage).NOTE: The default implementation chains the call to the input TokenStream, so be sure to call
super.reset()
when overriding this method.- Overrides:
reset
in classTokenFilter
- Throws:
java.io.IOException
-
-