Package io.debezium.text
Interface TokenStream.Tokenizer
-
- All Known Implementing Classes:
Strings.RegExSplitter
,TableIdParser.TableIdTokenizer
,TokenStream.BasicTokenizer
- Enclosing class:
- TokenStream
public static interface TokenStream.Tokenizer
Interface for a Tokenizer component responsible for processing the characters in aTokenStream.CharacterStream
and constructing the appropriateTokenStream.Token
objects.
-
-
Method Summary
All Methods Instance Methods Abstract Methods Modifier and Type Method Description void
tokenize(TokenStream.CharacterStream input, TokenStream.Tokens tokens)
Process the supplied characters and construct the appropriateTokenStream.Token
objects.
-
-
-
Method Detail
-
tokenize
void tokenize(TokenStream.CharacterStream input, TokenStream.Tokens tokens) throws ParsingException
Process the supplied characters and construct the appropriateTokenStream.Token
objects.- Parameters:
input
- the character input stream; never nulltokens
- the factory forTokenStream.Token
objects, which records the order in which the tokens are created- Throws:
ParsingException
- if there is an error while processing the character stream (e.g., a quote is not closed, etc.)
-
-