Package io.debezium.text
Interface TokenStream.Tokens
- All Known Implementing Classes:
TokenStream.CaseInsensitiveTokenFactory
,TokenStream.CaseSensitiveTokenFactory
,TokenStream.TokenFactory
- Enclosing class:
- TokenStream
public static interface TokenStream.Tokens
A factory for Token objects, used by a
TokenStream.Tokenizer
to create tokens in the correct order.-
Method Summary
Modifier and TypeMethodDescriptiondefault void
Create a single-character token at the supplied index in the character stream.default void
Create a single- or multi-character token with the characters in the range given by the starting and ending index in the character stream.void
Create a single- or multi-character token with the supplied type and with the characters in the range given by the starting and ending index in the character stream.
-
Method Details
-
addToken
Create a single-character token at the supplied index in the character stream. The token type is set to 0, meaning this is equivalent to callingaddToken(index,index+1)
oraddToken(index,index+1,0)
.- Parameters:
position
- the position (line and column numbers) of this new token; may not be nullindex
- the index of the character to appear in the token; must be a valid index in the stream
-
addToken
Create a single- or multi-character token with the characters in the range given by the starting and ending index in the character stream. The character at the ending index is not included in the token (as this is standard practice when using 0-based indexes). The token type is set to 0, meaning this is equivalent to callingaddToken(startIndex,endIndex,0)
.- Parameters:
position
- the position (line and column numbers) of this new token; may not be nullstartIndex
- the index of the first character to appear in the token; must be a valid index in the streamendIndex
- the index just past the last character to appear in the token; must be a valid index in the stream
-
addToken
Create a single- or multi-character token with the supplied type and with the characters in the range given by the starting and ending index in the character stream. The character at the ending index is not included in the token (as this is standard practice when using 0-based indexes).- Parameters:
position
- the position (line and column numbers) of this new token; may not be nullstartIndex
- the index of the first character to appear in the token; must be a valid index in the streamendIndex
- the index just past the last character to appear in the token; must be a valid index in the streamtype
- the type of the token
-