cc.mallet.pipe

Class SimpleTokenizer

java.lang.Object

cc.mallet.pipe.Pipe

cc.mallet.pipe.SimpleTokenizer

All Implemented Interfaces:

AlphabetCarrying, Serializable

public class SimpleTokenizer
extends Pipe

A simple unicode tokenizer that accepts sequences of letters as tokens.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected HashSet<String>	stoplist
static int	USE_DEFAULT_ENGLISH_STOPLIST
static int	USE_EMPTY_STOPLIST

Constructor Summary

Constructors

Constructor and Description
SimpleTokenizer(File stopfile)
SimpleTokenizer(HashSet<String> stoplist)
SimpleTokenizer(int languageFlag)

Method Summary

Modifier and Type	Method and Description
SimpleTokenizer	deepClone()
Instance	pipe(Instance instance) Really this should be 'protected', but isn't for historical reasons.
void	stop(String word)

Methods inherited from class cc.mallet.pipe.Pipe

alphabetsMatch, getAlphabet, getAlphabets, getDataAlphabet, getInstanceId, getTargetAlphabet, instanceFrom, instancesFrom, instancesFrom, isDataAlphabetSet, isTargetProcessing, newIteratorFrom, preceedingPipeDataAlphabetNotification, preceedingPipeTargetAlphabetNotification, precondition, readResolve, setDataAlphabet, setOrCheckDataAlphabet, setOrCheckTargetAlphabet, setTargetAlphabet, setTargetProcessing

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

USE_EMPTY_STOPLIST

public static final int USE_EMPTY_STOPLIST

See Also:

Constant Field Values

USE_DEFAULT_ENGLISH_STOPLIST

public static final int USE_DEFAULT_ENGLISH_STOPLIST

See Also:

Constant Field Values

stoplist

protected HashSet<String> stoplist

Constructor Detail

SimpleTokenizer

public SimpleTokenizer(int languageFlag)

SimpleTokenizer

public SimpleTokenizer(File stopfile)

SimpleTokenizer

public SimpleTokenizer(HashSet<String> stoplist)

Method Detail

deepClone

public SimpleTokenizer deepClone()

stop

public void stop(String word)

pipe

public Instance pipe(Instance instance)

Description copied from class: Pipe

Really this should be 'protected', but isn't for historical reasons.

Overrides:

pipe in class Pipe