Package io.debezium.text

Interface TokenStream.Tokenizer

All Known Implementing Classes:

Strings.RegExSplitter, TableIdParser.TableIdTokenizer, TokenStream.BasicTokenizer

Enclosing class:

TokenStream

public static interface TokenStream.Tokenizer

Interface for a Tokenizer component responsible for processing the characters in a TokenStream.CharacterStream and constructing the appropriate TokenStream.Token objects.

Method Summary

Modifier and Type	Method	Description
void	tokenize(TokenStream.CharacterStream input, TokenStream.Tokens tokens)	Process the supplied characters and construct the appropriate TokenStream.Token objects.

Method Details

tokenize

void tokenize(TokenStream.CharacterStream input,
 TokenStream.Tokens tokens)
       throws ParsingException

Process the supplied characters and construct the appropriate TokenStream.Token objects.

Parameters:

input - the character input stream; never null

tokens - the factory for TokenStream.Token objects, which records the order in which the tokens are created

Throws:

ParsingException - if there is an error while processing the character stream (e.g., a quote is not closed, etc.)