gate.creole.annic.apache.lucene.analysis.standard
Class StandardTokenizer
java.lang.Object
gate.creole.annic.apache.lucene.analysis.TokenStream
gate.creole.annic.apache.lucene.analysis.Tokenizer
gate.creole.annic.apache.lucene.analysis.standard.StandardTokenizer
- All Implemented Interfaces:
- StandardTokenizerConstants
public class StandardTokenizer
- extends Tokenizer
- implements StandardTokenizerConstants
A grammar-based tokenizer constructed with JavaCC.
This should be a good tokenizer for most European-language documents.
Many applications have specific tokenizer needs. If this tokenizer does
not suit your application, please consider copying this source code
directory to your project and maintaining your own grammar-based tokenizer.
Fields inherited from class gate.creole.annic.apache.lucene.analysis.Tokenizer |
input |
Fields inherited from interface gate.creole.annic.apache.lucene.analysis.standard.StandardTokenizerConstants |
ACRONYM, ALPHA, ALPHANUM, APOSTROPHE, CJK, COMPANY, DEFAULT, DIGIT, EMAIL, EOF, HAS_DIGIT, HOST, LETTER, NOISE, NUM, P, tokenImage |
Methods inherited from class gate.creole.annic.apache.lucene.analysis.Tokenizer |
close |
Methods inherited from class java.lang.Object |
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
token_source
public StandardTokenizerTokenManager token_source
token
public Token token
jj_nt
public Token jj_nt
StandardTokenizer
public StandardTokenizer(Reader reader)
- Constructs a tokenizer for this Reader.
StandardTokenizer
public StandardTokenizer(CharStream stream)
StandardTokenizer
public StandardTokenizer(StandardTokenizerTokenManager tm)
next
public final Token next()
throws ParseException,
IOException
- Returns the next token in the stream, or null at EOS.
The returned token's type is set to an element of StandardTokenizerConstants.tokenImage
.
- Specified by:
next
in class TokenStream
- Throws:
ParseException
IOException
ReInit
public void ReInit(CharStream stream)
ReInit
public void ReInit(StandardTokenizerTokenManager tm)
getNextToken
public final Token getNextToken()
getToken
public final Token getToken(int index)
generateParseException
public ParseException generateParseException()
enable_tracing
public final void enable_tracing()
disable_tracing
public final void disable_tracing()