public abstract class Tokenizer extends TokenStream
This is an abstract class; subclasses must override TokenStream.incrementToken()
NOTE: Subclasses overriding TokenStream.incrementToken()
must
call AttributeSource.clearAttributes()
before
setting attributes.
AttributeSource.AttributeFactory, AttributeSource.State
Modifier and Type | Method and Description |
---|---|
void |
close()
Releases resources associated with this stream.
|
void |
reset()
This method is called by a consumer before it begins consumption using
TokenStream.incrementToken() . |
void |
setReader(Reader input)
Expert: Set a new reader on the Tokenizer.
|
end, incrementToken
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, restoreState, toString
public void close() throws IOException
If you override this method, always call super.close()
, otherwise
some internal state will not be correctly reset (e.g., Tokenizer
will
throw IllegalStateException
on reuse).
NOTE:
The default implementation closes the input Reader, so
be sure to call super.close()
when overriding this method.
close
in interface Closeable
close
in interface AutoCloseable
close
in class TokenStream
IOException
public final void setReader(Reader input) throws IOException
IOException
public void reset() throws IOException
TokenStream
TokenStream.incrementToken()
.
Resets this stream to a clean state. Stateful implementations must implement this method so that they can be reused, just as if they had been created fresh.
If you override this method, always call super.reset()
, otherwise
some internal state will not be correctly reset (e.g., Tokenizer
will
throw IllegalStateException
on further usage).
reset
in class TokenStream
IOException
Copyright © 2010 - 2020 Adobe. All Rights Reserved