Package io.debezium.text

Class TokenStream.BasicTokenizer

java.lang.Object

io.debezium.text.TokenStream.BasicTokenizer

All Implemented Interfaces:

TokenStream.Tokenizer

Enclosing class:

TokenStream

public static class TokenStream.BasicTokenizer
extends Object
implements TokenStream.Tokenizer

A basic TokenStream.Tokenizer implementation that ignores whitespace but includes tokens for individual symbols, the period ('.'), single-quoted strings, double-quoted strings, whitespace-delimited words, and optionally comments.

Note this Tokenizer may not be appropriate in many situations, but is provided merely as a convenience for those situations that happen to be able to use it.

Field Summary

Fields

Modifier and Type	Field	Description
static final int	COMMENT	The token type for tokens that consist of all the characters between "/*" and "*/" or between "//" and the next line terminator (e.g., '\n', '\r' or "\r\n").
static final int	DECIMAL	The token type for tokens that consist of an individual '.' character.
static final int	DOUBLE_QUOTED_STRING	The token type for tokens that consist of all the characters within double-quotes.
static final int	SINGLE_QUOTED_STRING	The token type for tokens that consist of all the characters within single-quotes.
static final int	SYMBOL	The token type for tokens that consist of an individual "symbol" character.
private final boolean	useComments
static final int	WORD	The token type for tokens that represent an unquoted string containing a character sequence made up of non-whitespace and non-symbol characters.

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	BasicTokenizer(boolean useComments)

Method Summary

Modifier and Type	Method	Description
void	tokenize(TokenStream.CharacterStream input, TokenStream.Tokens tokens)	Process the supplied characters and construct the appropriate TokenStream.Token objects.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

WORD

public static final int WORD

The token type for tokens that represent an unquoted string containing a character sequence made up of non-whitespace and non-symbol characters.

See Also:

• Constant Field Values

SYMBOL

public static final int SYMBOL

The token type for tokens that consist of an individual "symbol" character. The set of characters includes: -(){}*,;+%?$[]!invalid input: '<'>|=:

See Also:

• Constant Field Values

DECIMAL

public static final int DECIMAL

The token type for tokens that consist of an individual '.' character.

See Also:

• Constant Field Values

SINGLE_QUOTED_STRING

public static final int SINGLE_QUOTED_STRING

The token type for tokens that consist of all the characters within single-quotes. Single quote characters are included if they are preceded (escaped) by a '\' character.

See Also:

• Constant Field Values

DOUBLE_QUOTED_STRING

public static final int DOUBLE_QUOTED_STRING

The token type for tokens that consist of all the characters within double-quotes. Double quote characters are included if they are preceded (escaped) by a '\' character.

See Also:

• Constant Field Values

COMMENT

public static final int COMMENT

The token type for tokens that consist of all the characters between "/*" and "*/" or between "//" and the next line terminator (e.g., '\n', '\r' or "\r\n").

See Also:

• Constant Field Values

useComments

private final boolean useComments

Constructor Details

BasicTokenizer

protected BasicTokenizer(boolean useComments)

Method Details

tokenize

public void tokenize(TokenStream.CharacterStream input,
 TokenStream.Tokens tokens)
              throws ParsingException

Description copied from interface: TokenStream.Tokenizer

Process the supplied characters and construct the appropriate TokenStream.Token objects.

Specified by:

tokenize in interface TokenStream.Tokenizer

Parameters:

input - the character input stream; never null

tokens - the factory for TokenStream.Token objects, which records the order in which the tokens are created

Throws:

ParsingException - if there is an error while processing the character stream (e.g., a quote is not closed, etc.)