Package weka.core.tokenizers

Class Summary

Class	Description
AlphabeticTokenizer	Alphabetic string tokenizer, tokens are to be formed only from contiguous alphabetic sequences.
CharacterDelimitedTokenizer	Abstract superclass for tokenizers that take characters as delimiters.
CharacterNGramTokenizer	Splits a string into an n-gram with min and max grams.
NGramTokenizer	Splits a string into an n-gram with min and max grams.
Tokenizer	A superclass for all tokenizer algorithms.
WordTokenizer	A simple tokenizer that is using the java.util.StringTokenizer class to tokenize the strings.