Package io.debezium.text
Interface TokenStream.Tokenizer
- All Known Implementing Classes:
Strings.RegExSplitter
,TableIdParser.TableIdTokenizer
,TokenStream.BasicTokenizer
- Enclosing class:
- TokenStream
public static interface TokenStream.Tokenizer
Interface for a Tokenizer component responsible for processing the characters in a
TokenStream.CharacterStream
and constructing
the appropriate TokenStream.Token
objects.-
Method Summary
Modifier and TypeMethodDescriptionvoid
tokenize
(TokenStream.CharacterStream input, TokenStream.Tokens tokens) Process the supplied characters and construct the appropriateTokenStream.Token
objects.
-
Method Details
-
tokenize
Process the supplied characters and construct the appropriateTokenStream.Token
objects.- Parameters:
input
- the character input stream; never nulltokens
- the factory forTokenStream.Token
objects, which records the order in which the tokens are created- Throws:
ParsingException
- if there is an error while processing the character stream (e.g., a quote is not closed, etc.)
-