Delimiting terminal symbols. No whitespace is skipped during this tokenization.
Lookahead is applied when needed to distinguish between different terminal symbols starting with
the same character.
When using this object, make sure that a "-" symbol requires a symbol separator (whitespace or comments)
if it follows a QName or NCName. The same holds for a "." symbol. Also, if a "." follows or precedes a
numeric literal, it requires a symbol separator. (Other than that, symbol separators are only needed
for non-delimiting terminal symbols.)
Delimiting terminal symbols. No whitespace is skipped during this tokenization. Lookahead is applied when needed to distinguish between different terminal symbols starting with the same character.
When using this object, make sure that a "-" symbol requires a symbol separator (whitespace or comments) if it follows a QName or NCName. The same holds for a "." symbol. Also, if a "." follows or precedes a numeric literal, it requires a symbol separator. (Other than that, symbol separators are only needed for non-delimiting terminal symbols.)