org.dizitart.no2.fulltext

Interface TextTokenizer

All Known Implementing Classes:

BaseTextTokenizer, EnglishTextTokenizer, UniversalTextTokenizer

public interface TextTokenizer

A stop-word based string tokenizer.

Since:

1.0

See Also:

TextIndexingService, EnglishTextTokenizer, NitriteBuilder.textTokenizer(TextTokenizer)

Method Summary

Modifier and Type	Method and Description
java.util.Set<java.lang.String>	stopWords() Gets all stop-words for a language.
java.util.Set<java.lang.String>	tokenize(java.lang.String text) Tokenize a text and discards all stop-words from it.

Method Detail

tokenize

java.util.Set<java.lang.String> tokenize(java.lang.String text)
                                  throws java.io.IOException

Tokenize a text and discards all stop-words from it.

Parameters:

text - the text to tokenize

Returns:

the set of tokens.

Throws:

java.io.IOException - if a low-level I/O error occurs.

stopWords

java.util.Set<java.lang.String> stopWords()

Gets all stop-words for a language.

Returns:

the set of all stop-words.