Class TokenStream
- java.lang.Object
-
- io.debezium.text.TokenStream
-
@NotThreadSafe public class TokenStream extends Object
A foundation for basic parsers that tokenize input content and allows parsers to easily access and use those tokens. ATokenStream
object literally represents the stream ofTokenStream.Token
objects that each represent a word, symbol, comment or other lexically-relevant piece of information. This simple framework makes it very easy to create a parser that walks through (or "consumes") the tokens in the order they appear and do something useful with that content (usually creating another representation of the content, such as some domain-specific Abstract Syntax Tree or object model).The parts
This simple framework consists of a couple of pieces that fit together to do the whole job of parsing input content.
The
TokenStream.Tokenizer
is responsible for consuming the character-level input content and constructingTokenStream.Token
objects for the different words, symbols, or other meaningful elements contained in the content. Each Token object is a simple object that records the character(s) that make up the token's value, but it does this in a very lightweight and efficient way by pointing to the original character stream. Each token can be assigned a parser-specific integral token type that may make it easier to do quickly figure out later in the process what kind of information each token represents. The general idea is to keep the Tokenizer logic very simple, and very oftenTokenStream.Tokenizer
s will merely look for the different kinds of characters (e.g., symbols, letters, digits, etc.) as well as things like quoted strings and comments. However,TokenStream.Tokenizer
s are never called by the parser, but instead are always given to the TokenStream that then calls the Tokenizer at the appropriate time.The
TokenStream
is supplied the input content, a Tokenizer implementation, and a few options. Its job is to prepare the content for processing, call the Tokenizer implementation to create the series of Token objects, and then provide an interface for walking through and consuming the tokens. This interface makes it possible to discover the value and type of the current token, and consume the current token and move to the next token. Plus, the interface has been designed to make the code that works with the tokens to be as readable as possible.The final component in this framework is the Parser. The parser is really any class that takes as input the content to be parsed and that outputs some meaningful information. The parser will do this by defining the Tokenizer, constructing a TokenStream object, and then using the TokenStream to walk through the sequence of Tokens and produce some meaningful representation of the content. Parsers can create instances of some object model, or they can create a domain-specific Abstract Syntax Tree representation.
The benefit of breaking the responsibility along these lines is that the TokenStream implementation is able to encapsulate quite a bit of very tedious and very useful functionality, while still allowing a lot of flexibility as to what makes up the different tokens. It also makes the parser very easy to write and read (and thus maintain), without placing very many restrictions on how that logic is to be defined. Plus, because the TokenStream takes responsibility for tracking the positions of every token (including line and column numbers), it can automatically produce meaningful errors.
Consuming tokens
A parser works with the tokens on the TokenStream using a variety of methods:
- The
start()
method must be called before any of the other methods. It performs initialization and tokenization, and prepares the internal state by finding the first token and setting an internal current token reference. - The
hasNext()
method can be called repeatedly to determine if there is another token after the current token. This is often useful when an unknown number of tokens is to be processed, and behaves very similarly to theIterator.hasNext()
method. - The
consume()
method returns thevalue
of the current token and moves the current token pointer to the next available token. - The
consume(String)
andconsume(char)
methods look at the current token and ensure the token'svalue
matches the value supplied as a method parameter, or they throw aParsingException
if the values don't match. Theconsume(int)
method works similarly, except that it attempts to match the token'stype
. And, theconsume(String, String...)
is a convenience method that is equivalent to callingconsume(String)
for each of the arguments. - The
canConsume(String)
andcanConsume(char)
methods look at the current token and check whether the token'svalue
matches the value supplied as a method parameter. If there is a match, the method advances the current token reference and returns true. Otherwise, the current token does not match and the method returns false without advancing the current token reference or throwing a ParsingException. Similarly, thecanConsume(int)
method checks the token'stype
rather than the value, consuming the token and returning true if there is a match, or just returning false if there is no match. ThecanConsume(String, String...)
method determines whether all of the supplied values can be consumed in the given order. - The
matches(String)
andmatches(char)
methods look at the current token and check whether the token'svalue
matches the value supplied as a method parameter. The method then returns whether there was a match, but does not advance the current token pointer. Similarly, thematches(int)
method checks the token'stype
rather than the value. Thematches(String, String...)
method is a convenience method that is equivalent to callingmatches(String)
for each of the arguments, and thematches(int, int...)
method is a convenience method that is equivalent to callingmatches(int)
for each of the arguments.
- The
matchesAnyOf(String, String...)
methods look at the current token and check whether the token'svalue
matches at least one of the values supplied as method parameters. The method then returns whether there was a match, but does not advance the current token pointer. Similarly, thematchesAnyOf(int, int...)
method checks the token'stype
rather than the value.With these methods, it's very easy to create a parser that looks at the current token to decide what to do, and then consume that token, and repeat this process.
Example parser
Here is an example of a very simple parser that parses very simple and limited SQL
SELECT
andDELETE
statements, such asSELECT * FROM Customers
orSELECT Name, StreetAddress AS Address, City, Zip FROM Customers
orDELETE FROM Customers WHERE Zip=12345
:public class SampleSqlSelectParser { public List<Statement> parse( String ddl ) { TokenStream tokens = new TokenStream(ddl, new SqlTokenizer(), false); List<Statement> statements = new LinkedList<Statement>(); token.start(); while (tokens.hasNext()) { if (tokens.matches("SELECT")) { statements.add(parseSelect(tokens)); } else { statements.add(parseDelete(tokens)); } } return statements; } protected Select parseSelect( TokenStream tokens ) throws ParsingException { tokens.consume("SELECT"); List<Column> columns = parseColumns(tokens); tokens.consume("FROM"); String tableName = tokens.consume(); return new Select(tableName, columns); } protected List<Column> parseColumns( TokenStream tokens ) throws ParsingException { List<Column> columns = new LinkedList<Column>(); if (tokens.matches('*')) { tokens.consume(); // leave the columns empty to signal wildcard } else { // Read names until we see a ',' do { String columnName = tokens.consume(); if (tokens.canConsume("AS")) { String columnAlias = tokens.consume(); columns.add(new Column(columnName, columnAlias)); } else { columns.add(new Column(columnName, null)); } } while (tokens.canConsume(',')); } return columns; } protected Delete parseDelete( TokenStream tokens ) throws ParsingException { tokens.consume("DELETE", "FROM"); String tableName = tokens.consume(); tokens.consume("WHERE"); String lhs = tokens.consume(); tokens.consume('='); String rhs = tokens.consume(); return new Delete(tableName, new Criteria(lhs, rhs)); } } public abstract class Statement { ... } public class Query extends Statement { ... } public class Delete extends Statement { ... } public class Column { ... }
This example shows an idiomatic way of writing a parser that is stateless and thread-safe. Theparse(...)
method takes the input as a parameter, and returns the domain-specific representation that resulted from the parsing. All other methods are utility methods that simply encapsulate common logic or make the code more readable.In the example, the
parse(...)
first creates a TokenStream object (using a Tokenizer implementation that is not shown), and then loops as long as there are more tokens to read. As it loops, if the next token is "SELECT", the parser calls theparseSelect(...)
method which immediately consumes a "SELECT" token, the names of the columns separated by commas (or a '*' if there all columns are to be selected), a "FROM" token, and the name of the table being queried. TheparseSelect(...)
method returns aSelect
object, which then added to the list of statements in theparse(...)
method. The parser handles the "DELETE" statements in a similar manner.Case sensitivity
Very often grammars to not require the case of keywords to match. This can make parsing a challenge, because all combinations of case need to be used. The TokenStream framework provides a very simple solution that requires no more effort than providing a boolean parameter to the constructor.
When a
false
value is provided for the thecaseSensitive
parameter, the TokenStream performs all matching operations as if each token's value were in uppercase only. This means that the arguments supplied to thematch(...)
,canConsume(...)
, andconsume(...)
methods should be upper-cased. Note that the actual value of each token remains the actual case as it appears in the input.Of course, when the TokenStream is created with a
true
value for thecaseSensitive
parameter, the matching is performed using the actual value as it appears in the input contentWhitespace
Many grammars are independent of lines breaks or whitespace, allowing a lot of flexibility when writing the content. The TokenStream framework makes it very easy to ignore line breaks and whitespace. To do so, the Tokenizer implementation must simply not include the line break character sequences and whitespace in the token ranges. Since none of the tokens contain whitespace, the parser never has to deal with them.
Of course, many parsers will require that some whitespace be included. For example, whitespace within a quoted string may be needed by the parser. In this case, the Tokenizer should simply include the whitespace characters in the tokens.
Writing a Tokenizer
Each parser will likely have its own
TokenStream.Tokenizer
implementation that contains the parser-specific logic about how to break the content into token objects. Generally, the easiest way to do this is to simply iterate through the character sequence passed into thetokenize(...)
method, and use a switch statement to decide what to do.Here is the code for a very basic Tokenizer implementation that ignores whitespace, line breaks and Java-style (multi-line and end-of-line) comments, while constructing single tokens for each quoted string.
public class BasicTokenizer implements Tokenizer { public void tokenize(CharacterStream input, Tokens tokens) throws ParsingException { while (input.hasNext()) { char c = input.next(); switch (c) { case ' ': case '\t': case '\n': case '\r': // Just skip these whitespace characters ... break; case '-': case '(': case ')': case '{': case '}': case '*': case ',': case ';': case '+': case '%': case '?': case '$': case '[': case ']': case '!': case '<': case '>': case '|': case '=': case ':': tokens.addToken(input.index(), input.index() + 1, SYMBOL); break; case '.': tokens.addToken(input.index(), input.index() + 1, DECIMAL); break; case '\"': int startIndex = input.index(); Position startingPosition = input.position(); boolean foundClosingQuote = false; while (input.hasNext()) { c = input.next(); if (c == '\\' && input.isNext('"')) { c = input.next(); // consume the ' character since it is escaped } else if (c == '"') { foundClosingQuote = true; break; } } if (!foundClosingQuote) { throw new ParsingException(startingPosition, "No matching closing double quote found"); } int endIndex = input.index() + 1; // beyond last character read tokens.addToken(startIndex, endIndex, DOUBLE_QUOTED_STRING); break; case '\'': startIndex = input.index(); startingPosition = input.position(); foundClosingQuote = false; while (input.hasNext()) { c = input.next(); if (c == '\\' && input.isNext('\'')) { c = input.next(); // consume the ' character since it is escaped } else if (c == '\'') { foundClosingQuote = true; break; } } if (!foundClosingQuote) { throw new ParsingException(startingPosition, "No matching closing single quote found"); } endIndex = input.index() + 1; // beyond last character read tokens.addToken(startIndex, endIndex, SINGLE_QUOTED_STRING); break; case '/': startIndex = input.index(); if (input.isNext('/')) { // End-of-line comment ... boolean foundLineTerminator = false; while (input.hasNext()) { c = input.next(); if (c == '\n' || c == '\r') { foundLineTerminator = true; break; } } endIndex = input.index(); // the token won't include the '\n' or '\r' character(s) if (!foundLineTerminator) ++endIndex; // must point beyond last char if (c == '\r' && input.isNext('\n')) input.next(); if (useComments) { tokens.addToken(startIndex, endIndex, COMMENT); } } else if (input.isNext('*')) { // Multi-line comment ... while (input.hasNext() && !input.isNext('*', '/')) { c = input.next(); } if (input.hasNext()) input.next(); // consume the '*' if (input.hasNext()) input.next(); // consume the '/' if (useComments) { endIndex = input.index() + 1; // the token will include the '/' and '*' characters tokens.addToken(startIndex, endIndex, COMMENT); } } else { // just a regular slash ... tokens.addToken(startIndex, startIndex + 1, SYMBOL); } break; default: startIndex = input.index(); // Read until another whitespace/symbol/decimal/slash is found while (input.hasNext() && !(input.isNextWhitespace() || input.isNextAnyOf("/.-(){}*,;+%?$[]!<>|=:"))) { c = input.next(); } endIndex = input.index() + 1; // beyond last character that was included tokens.addToken(startIndex, endIndex, WORD); } } } }
TokenStream.Tokenizer
s with exactly this behavior can actually be created using thebasicTokenizer(boolean)
method. So while this very basic implementation is not meant to be used in all situations, it may be useful in some situations.- Author:
- Randall Hauch, Horia Chiorean, Daniel Kelleher
- The
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description static class
TokenStream.BasicTokenizer
A basicTokenStream.Tokenizer
implementation that ignores whitespace but includes tokens for individual symbols, the period ('.'), single-quoted strings, double-quoted strings, whitespace-delimited words, and optionally comments.protected class
TokenStream.CaseInsensitiveToken
class
TokenStream.CaseInsensitiveTokenFactory
protected class
TokenStream.CaseSensitiveToken
An immutableTokenStream.Token
that implements matching using case-sensitive logic.class
TokenStream.CaseSensitiveTokenFactory
static class
TokenStream.CharacterArrayStream
An implementation ofTokenStream.CharacterStream
that works with a single character array.static interface
TokenStream.CharacterStream
Interface used by aTokenStream.Tokenizer
to iterate through the characters in the content input to theTokenStream
.static class
TokenStream.Marker
An opaque marker for a position within the token stream.static interface
TokenStream.Token
The interface defining a token, which references the characters in the actual input character stream.protected class
TokenStream.TokenFactory
static interface
TokenStream.Tokenizer
Interface for a Tokenizer component responsible for processing the characters in aTokenStream.CharacterStream
and constructing the appropriateTokenStream.Token
objects.static interface
TokenStream.Tokens
A factory for Token objects, used by aTokenStream.Tokenizer
to create tokens in the correct order.
-
Field Summary
Fields Modifier and Type Field Description static int
ANY_TYPE
A constant that can be used with thematches(int)
,matches(int, int...)
,consume(int)
, andcanConsume(int)
methods to signal that any token type is allowed to be matched.static String
ANY_VALUE
A constant that can be used with thematches(String)
,matches(String, String...)
,consume(String)
,consume(String, String...)
,canConsume(String)
andcanConsume(String, String...)
methods to signal that any value is allowed to be matched.private boolean
caseSensitive
private boolean
completed
private TokenStream.Token
currentToken
private char[]
inputContent
protected String
inputString
private ListIterator<TokenStream.Token>
tokenIterator
This class navigates the Token objects using this iterator.private TokenStream.Tokenizer
tokenizer
private List<TokenStream.Token>
tokens
-
Constructor Summary
Constructors Constructor Description TokenStream(String content, TokenStream.Tokenizer tokenizer, boolean caseSensitive)
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description boolean
advance(TokenStream.Marker marker)
Advance the stream back to the position described by the supplied marker.static TokenStream.BasicTokenizer
basicTokenizer(boolean includeComments)
Obtain a basicTokenStream.Tokenizer
implementation that ignores whitespace but includes tokens for individual symbols, the period ('.'), single-quoted strings, double-quoted strings, whitespace-delimited words, and optionally comments.boolean
canConsume(char expected)
Attempt to consume this current token if it matches the expected value, and return whether this method was indeed able to consume the token.boolean
canConsume(int expectedType)
Attempt to consume this current token if it matches the expected token type, and return whether this method was indeed able to consume the token.boolean
canConsume(int type, String expected)
Attempt to consume this current token if it matches the expected value, and return whether this method was indeed able to consume the token.boolean
canConsume(int type, String currentExpected, String... expectedForNextTokens)
Attempt to consume this current token and the next tokens if and only if they match the expected type and values, and return whether this method was indeed able to consume all of the supplied tokens.boolean
canConsume(Iterable<String> nextTokens)
Attempt to consume this current token and the next tokens if and only if they match the expected values, and return whether this method was indeed able to consume all of the supplied tokens.boolean
canConsume(String expected)
Attempt to consume this current token if it matches the expected value, and return whether this method was indeed able to consume the token.boolean
canConsume(String[] nextTokens)
Attempt to consume this current token and the next tokens if and only if they match the expected values, and return whether this method was indeed able to consume all of the supplied tokens.boolean
canConsume(String currentExpected, String... expectedForNextTokens)
Attempt to consume this current token and the next tokens if and only if they match the expected values, and return whether this method was indeed able to consume all of the supplied tokens.boolean
canConsumeAnyOf(int[] typeOptions)
Attempt to consume the next token if it matches one of the supplied types.boolean
canConsumeAnyOf(int firstTypeOption, int... additionalTypeOptions)
Attempt to consume the next token if it matches one of the supplied types.boolean
canConsumeAnyOf(Iterable<String> options)
Attempt to consume the next token if it matches one of the supplied values.boolean
canConsumeAnyOf(String[] options)
Attempt to consume the next token if it matches one of the supplied values.boolean
canConsumeAnyOf(String firstOption, String... additionalOptions)
Attempt to consume the next token if it matches one of the supplied values.boolean
canConsumeBoolean(BooleanConsumer consumer)
Attempt to consume this current token if it can be parsed as a boolean, and return whether this method was indeed able to consume the token.boolean
canConsumeInteger(IntConsumer consumer)
Attempt to consume this current token if it can be parsed as an integer, and return whether this method was indeed able to consume the token.boolean
canConsumeLong(LongConsumer consumer)
Attempt to consume this current token if it can be parsed as a long, and return whether this method was indeed able to consume the token.boolean
canConsumeWord(String expected)
Attempt to consume this current token if it isTokenStream.BasicTokenizer.WORD
and it matches the expected value, and return whether this method was indeed able to consume the token.boolean
canConsumeWords(String currentExpected, String... expectedForNextTokens)
Attempt to consume this current token and the next tokens if and only if they are ofTokenStream.BasicTokenizer.WORD
and match the expected values, and return whether this method was indeed able to consume all of the supplied tokens.String
consume()
Return the value of this token and move to the next token.TokenStream
consume(char expected)
Attempt to consume this current token as long as it matches the expected character, or throw an exception if the token does not match.TokenStream
consume(int expectedType)
Attempt to consume this current token as long as it matches the expected character, or throw an exception if the token does not match.TokenStream
consume(Iterable<String> nextTokens)
Attempt to consume this current token as the next tokens as long as they match the expected values, or throw an exception if the token does not match.TokenStream
consume(String expected)
Attempt to consume this current token as long as it matches the expected value, or throw an exception if the token does not match.TokenStream
consume(String[] nextTokens)
Attempt to consume this current token as the next tokens as long as they match the expected values, or throw an exception if the token does not match.TokenStream
consume(String expected, String... expectedForNextTokens)
Attempt to consume this current token as the next tokens as long as they match the expected values, or throw an exception if the token does not match.String
consumeAnyOf(int... typeOptions)
Consume and return the next token that must match one of the supplied values.String
consumeAnyOf(String... options)
Consume and return the next token that must match one of the supplied values.TokenStream
consumeThrough(char expected)
Attempt to consume all tokens until the specified token is consumed, and then stop.TokenStream
consumeThrough(char expected, char skipMatchingTokens)
Attempt to consume all tokens until the specified token is consumed, and then stop.TokenStream
consumeThrough(String expected)
Attempt to consume all tokens until the specified token is consumed, and then stop.TokenStream
consumeThrough(String expected, String skipMatchingTokens)
Attempt to consume all tokens until the specified token is consumed, and then stop.TokenStream
consumeUntil(char expected)
Attempt to consume all tokens until the specified token is found, and then stop before consuming that token.TokenStream
consumeUntil(char expected, char skipMatchingTokens)
Attempt to consume all tokens until the specified token is found, and then stop before consuming that token.TokenStream
consumeUntil(String expected)
Attempt to consume all tokens until the specified token is found, and then stop before consuming that token.TokenStream
consumeUntil(String expected, String... skipMatchingTokens)
Attempt to consume all tokens until the specified token is found, and then stop before consuming that token.TokenStream
consumeUntilEndOrOneOf(String... stopTokens)
Consume the token stream until one of the stop tokens or the end of the stream is found.(package private) TokenStream.Token
currentToken()
Get the current token.(package private) String
generateFragment()
(package private) static String
generateFragment(String content, int indexOfProblem, int charactersToIncludeBeforeAndAfter, String highlightText)
Utility method to generate a highlighted fragment of a particular point in the stream.protected String
getContentBetween(int startIndex, Position end)
String
getContentBetween(Position starting, Position end)
Gets the content string starting at the first position (inclusive) and continuing up to the end position (exclusive).String
getContentBetween(TokenStream.Marker starting, Position end)
Gets the content string starting at the specified marker (inclusive) and continuing up to the end position (exclusive).String
getContentFrom(TokenStream.Marker starting)
Gets the content string starting at the specified marker (inclusive) and continuing up to the next position (exclusive).String
getInputString()
boolean
hasNext()
Determine if this stream has another token to be consumed.protected List<TokenStream.Token>
initializeTokens(List<TokenStream.Token> tokens)
Method to allow subclasses to pre-process the set of tokens and return the correct tokens to use.TokenStream.Marker
mark()
Obtain a marker that records the current position so that the stream can berewind(Marker)
back to the mark even after having been advanced beyond the mark.boolean
matches(char expected)
Determine if the current token matches the expected value.boolean
matches(int expectedType)
Determine if the current token matches the expected token type.boolean
matches(int[] typesForNextTokens)
Determine if the next few tokens have the supplied types.boolean
matches(int currentExpectedType, int... expectedTypeForNextTokens)
Determine if the next few tokens have the supplied types.boolean
matches(int type, String expected)
Determine if the current token matches the expected type and a value.boolean
matches(Iterable<String> nextTokens)
Determine if the next few tokens match the expected values.boolean
matches(String expected)
Determine if the current token matches the expected value.boolean
matches(String[] nextTokens)
Determine if the next few tokens match the expected values.boolean
matches(String currentExpected, String... expectedForNextTokens)
Determine if the next few tokens match the expected values.boolean
matchesAnyOf(int[] typeOptions)
Determine if the next token have one of the supplied types.boolean
matchesAnyOf(int firstTypeOption, int... additionalTypeOptions)
Determine if the next token have one of the supplied types.boolean
matchesAnyOf(int type, String firstOption, String... additionalOptions)
Determine if the next token matches one of the supplied values of the expected type.boolean
matchesAnyOf(Iterable<String> options)
Determine if the next token matches one of the supplied values.boolean
matchesAnyOf(String[] options)
Determine if the next token matches one of the supplied values.boolean
matchesAnyOf(String firstOption, String... additionalOptions)
Determine if the next token matches one of the supplied values.boolean
matchesAnyWordOf(String firstOption, String... additionalOptions)
Determine if the next token matches one of the supplied values of the typeTokenStream.BasicTokenizer.WORD
boolean
matchesWord(String expected)
Determine if the current token isTokenStream.BasicTokenizer.WORD
and matches the expected value.private void
moveToNextToken()
private void
moveToNextToken(List<TokenStream.Token> newTokens)
Position
nextPosition()
Get the position of the next (or current) token.String
peek()
Position
previousPosition()
Get the position of the previous token.Position
previousPosition(int count)
Get the position of a token earlier in the stream from the current position.TokenStream.Token
previousToken(int count)
Get the previous token.void
rewind()
Method to allow tokens to be re-used from the start without re-tokenizing content.boolean
rewind(TokenStream.Marker marker)
Reset the stream back to the position described by the supplied marker.TokenStream
start()
Begin the token stream, including (if required) the tokenization of the input content.protected void
throwNoMoreContent()
String
toString()
-
-
-
Field Detail
-
ANY_VALUE
public static final String ANY_VALUE
A constant that can be used with thematches(String)
,matches(String, String...)
,consume(String)
,consume(String, String...)
,canConsume(String)
andcanConsume(String, String...)
methods to signal that any value is allowed to be matched.Note that this exact instance must be used; an equivalent string will not work.
- See Also:
- Constant Field Values
-
ANY_TYPE
public static final int ANY_TYPE
A constant that can be used with thematches(int)
,matches(int, int...)
,consume(int)
, andcanConsume(int)
methods to signal that any token type is allowed to be matched.- See Also:
- Constant Field Values
-
inputString
protected final String inputString
-
inputContent
private final char[] inputContent
-
caseSensitive
private final boolean caseSensitive
-
tokenizer
private final TokenStream.Tokenizer tokenizer
-
tokens
private List<TokenStream.Token> tokens
-
tokenIterator
private ListIterator<TokenStream.Token> tokenIterator
This class navigates the Token objects using this iterator. However, because it very often needs to access the "current token" in the "consume(...)" and "canConsume(...)" and "matches(...)" methods, the class caches a "current token" and makes this iterator point to the 2nd token.T1 T2 T3 T4 T5 ˆ ˆ ˆ | | | | | +- The position of the tokenIterator, where tokenIterator.hasNext() will return T3 | +---- The token referenced by currentToken +-------- The logical position of the TokenStream object, where the "consume()" would return T2
-
currentToken
private TokenStream.Token currentToken
-
completed
private boolean completed
-
-
Constructor Detail
-
TokenStream
public TokenStream(String content, TokenStream.Tokenizer tokenizer, boolean caseSensitive)
-
-
Method Detail
-
start
public TokenStream start() throws ParsingException
Begin the token stream, including (if required) the tokenization of the input content.- Returns:
- this object for easy method chaining; never null
- Throws:
ParsingException
- if an error occurs during tokenization of the content
-
initializeTokens
protected List<TokenStream.Token> initializeTokens(List<TokenStream.Token> tokens)
Method to allow subclasses to pre-process the set of tokens and return the correct tokens to use. The default behavior is to simply return the supplied tokens.- Parameters:
tokens
- the tokens- Returns:
- list of tokens.
-
rewind
public void rewind()
Method to allow tokens to be re-used from the start without re-tokenizing content.
-
mark
public TokenStream.Marker mark()
Obtain a marker that records the current position so that the stream can berewind(Marker)
back to the mark even after having been advanced beyond the mark.- Returns:
- the marker; never null
- Throws:
IllegalStateException
- if this method was called before the stream wasstarted
NoSuchElementException
- if there are no more tokens
-
rewind
public boolean rewind(TokenStream.Marker marker)
Reset the stream back to the position described by the supplied marker. This method does nothing if the mark is invalid. For example, it is not possible to advance the token stream beyond the current position.- Parameters:
marker
- the marker- Returns:
- true if the token stream was reset, or false if the marker was invalid
- See Also:
advance(Marker)
-
advance
public boolean advance(TokenStream.Marker marker)
Advance the stream back to the position described by the supplied marker. This method does nothing if the mark is invalid. For example, it is not possible to rewind the token stream beyond the current position.- Parameters:
marker
- the marker- Returns:
- true if the token stream was advanced, or false if the marker was invalid
- See Also:
rewind(Marker)
-
previousPosition
public Position previousPosition()
Get the position of the previous token.- Returns:
- the previous token's position; never null
- Throws:
IllegalStateException
- if this method was called before the stream wasstarted
NoSuchElementException
- if there is no previous token
-
previousPosition
public Position previousPosition(int count)
Get the position of a token earlier in the stream from the current position.- Parameters:
count
- the number of tokens before the current position (e.g., 1 for the previous position)- Returns:
- the previous token's position; never null
- Throws:
IllegalStateException
- if this method was called before the stream wasstarted
NoSuchElementException
- if there is no previous token
-
nextPosition
public Position nextPosition()
Get the position of the next (or current) token.- Returns:
- the current token's position; never null
- Throws:
IllegalStateException
- if this method was called before the stream wasstarted
NoSuchElementException
- if there is no previous token
-
consume
public String consume() throws ParsingException, IllegalStateException
Return the value of this token and move to the next token.- Returns:
- the value of the current token
- Throws:
ParsingException
- if there is no such token to consumeIllegalStateException
- if this method was called before the stream wasstarted
-
throwNoMoreContent
protected void throwNoMoreContent() throws ParsingException
- Throws:
ParsingException
-
peek
public String peek() throws IllegalStateException
- Throws:
IllegalStateException
-
consume
public TokenStream consume(String expected) throws ParsingException, IllegalStateException
Attempt to consume this current token as long as it matches the expected value, or throw an exception if the token does not match.The
ANY_VALUE
constant can be used in the expected values as a wildcard.- Parameters:
expected
- the expected value of the current token- Returns:
- this token stream instance so callers can chain together methods; never null
- Throws:
ParsingException
- if the current token doesn't match the supplied valueIllegalStateException
- if this method was called before the stream wasstarted
-
consume
public TokenStream consume(char expected) throws ParsingException, IllegalStateException
Attempt to consume this current token as long as it matches the expected character, or throw an exception if the token does not match.- Parameters:
expected
- the expected character of the current token- Returns:
- this token stream instance so callers can chain together methods; never null
- Throws:
ParsingException
- if the current token doesn't match the supplied valueIllegalStateException
- if this method was called before the stream wasstarted
-
consume
public TokenStream consume(int expectedType) throws ParsingException, IllegalStateException
Attempt to consume this current token as long as it matches the expected character, or throw an exception if the token does not match.The
ANY_TYPE
constant can be used in the expected values as a wildcard.- Parameters:
expectedType
- the expected token type of the current token- Returns:
- this token stream instance so callers can chain together methods; never null
- Throws:
ParsingException
- if the current token doesn't match the supplied valueIllegalStateException
- if this method was called before the stream wasstarted
-
consume
public TokenStream consume(String expected, String... expectedForNextTokens) throws ParsingException, IllegalStateException
Attempt to consume this current token as the next tokens as long as they match the expected values, or throw an exception if the token does not match.The
ANY_VALUE
constant can be used in the expected values as a wildcard.- Parameters:
expected
- the expected value of the current tokenexpectedForNextTokens
- the expected values of the following tokens- Returns:
- this token stream instance so callers can chain together methods; never null
- Throws:
ParsingException
- if the current token doesn't match the supplied valueIllegalStateException
- if this method was called before the stream wasstarted
-
consume
public TokenStream consume(String[] nextTokens) throws ParsingException, IllegalStateException
Attempt to consume this current token as the next tokens as long as they match the expected values, or throw an exception if the token does not match.The
ANY_VALUE
constant can be used in the expected values as a wildcard.- Parameters:
nextTokens
- the expected values for the next tokens- Returns:
- this token stream instance so callers can chain together methods; never null
- Throws:
ParsingException
- if the current token doesn't match the supplied valueIllegalStateException
- if this method was called before the stream wasstarted
-
consume
public TokenStream consume(Iterable<String> nextTokens) throws ParsingException, IllegalStateException
Attempt to consume this current token as the next tokens as long as they match the expected values, or throw an exception if the token does not match.The
ANY_VALUE
constant can be used in the expected values as a wildcard.- Parameters:
nextTokens
- the expected values for the next tokens- Returns:
- this token stream instance so callers can chain together methods; never null
- Throws:
ParsingException
- if the current token doesn't match the supplied valueIllegalStateException
- if this method was called before the stream wasstarted
-
consumeAnyOf
public String consumeAnyOf(int... typeOptions) throws IllegalStateException
Consume and return the next token that must match one of the supplied values.- Parameters:
typeOptions
- the options for the type of the current token- Returns:
- the token that was consumed and that matches one of the supplied options
- Throws:
ParsingException
- if the current token doesn't match the supplied valueIllegalStateException
- if this method was called before the stream wasstarted
-
consumeAnyOf
public String consumeAnyOf(String... options) throws IllegalStateException
Consume and return the next token that must match one of the supplied values.- Parameters:
options
- the additional options for the value of the current token- Returns:
- the token that was consumed and that matches one of the supplied options
- Throws:
ParsingException
- if the current token doesn't match the supplied valueIllegalStateException
- if this method was called before the stream wasstarted
-
consumeThrough
public TokenStream consumeThrough(char expected) throws ParsingException, IllegalStateException
Attempt to consume all tokens until the specified token is consumed, and then stop. If it is not found, then the token stream is left untouched and a ParsingException is thrown.The
ANY_VALUE
constant can be used in the expected values as a wildcard.- Parameters:
expected
- the token that is to be found- Returns:
- this token stream instance so callers can chain together methods; never null
- Throws:
ParsingException
- if the specified token cannot be foundIllegalStateException
- if this method was called before the stream wasstarted
-
consumeThrough
public TokenStream consumeThrough(char expected, char skipMatchingTokens) throws ParsingException, IllegalStateException
Attempt to consume all tokens until the specified token is consumed, and then stop. If it is not found, then the token stream is left untouched and a ParsingException is thrown.The
ANY_VALUE
constant can be used in the expected values as a wildcard.- Parameters:
expected
- the token that is to be foundskipMatchingTokens
- the token that, if found, should result in skippingexpected
once for each occurrence ofskipMatchingTokens
; may be null- Returns:
- this token stream instance so callers can chain together methods; never null
- Throws:
ParsingException
- if the specified token cannot be foundIllegalStateException
- if this method was called before the stream wasstarted
-
consumeThrough
public TokenStream consumeThrough(String expected) throws ParsingException, IllegalStateException
Attempt to consume all tokens until the specified token is consumed, and then stop. If it is not found, then the token stream is left untouched and a ParsingException is thrown.The
ANY_VALUE
constant can be used in the expected values as a wildcard.- Parameters:
expected
- the token that is to be found- Returns:
- this token stream instance so callers can chain together methods; never null
- Throws:
ParsingException
- if the specified token cannot be foundIllegalStateException
- if this method was called before the stream wasstarted
-
consumeThrough
public TokenStream consumeThrough(String expected, String skipMatchingTokens) throws ParsingException, IllegalStateException
Attempt to consume all tokens until the specified token is consumed, and then stop. If it is not found, then the token stream is left untouched and a ParsingException is thrown.The
ANY_VALUE
constant can be used in the expected values as a wildcard.- Parameters:
expected
- the token that is to be foundskipMatchingTokens
- the token that, if found, should result in skippingexpected
once for each occurrence ofskipMatchingTokens
; may be null- Returns:
- this token stream instance so callers can chain together methods; never null
- Throws:
ParsingException
- if the specified token cannot be foundIllegalStateException
- if this method was called before the stream wasstarted
-
consumeUntil
public TokenStream consumeUntil(char expected) throws ParsingException, IllegalStateException
Attempt to consume all tokens until the specified token is found, and then stop before consuming that token. If it is not found, then the token stream is left untouched and a ParsingException is thrown.The
ANY_VALUE
constant can be used in the expected values as a wildcard.- Parameters:
expected
- the token that is to be found- Returns:
- this token stream instance so callers can chain together methods; never null
- Throws:
ParsingException
- if the specified token cannot be foundIllegalStateException
- if this method was called before the stream wasstarted
-
consumeUntil
public TokenStream consumeUntil(char expected, char skipMatchingTokens) throws ParsingException, IllegalStateException
Attempt to consume all tokens until the specified token is found, and then stop before consuming that token. If it is not found, then the token stream is left untouched and a ParsingException is thrown.The
ANY_VALUE
constant can be used in the expected values as a wildcard.- Parameters:
expected
- the token that is to be foundskipMatchingTokens
- the token that, if found, should result in skippingexpected
once for each occurrence ofskipMatchingTokens
- Returns:
- this token stream instance so callers can chain together methods; never null
- Throws:
ParsingException
- if the specified token cannot be foundIllegalStateException
- if this method was called before the stream wasstarted
-
consumeUntil
public TokenStream consumeUntil(String expected) throws ParsingException, IllegalStateException
Attempt to consume all tokens until the specified token is found, and then stop before consuming that token. If it is not found, then the token stream is left untouched and a ParsingException is thrown.The
ANY_VALUE
constant can be used in the expected values as a wildcard.- Parameters:
expected
- the token that is to be found- Returns:
- this token stream instance so callers can chain together methods; never null
- Throws:
ParsingException
- if the specified token cannot be foundIllegalStateException
- if this method was called before the stream wasstarted
-
consumeUntil
public TokenStream consumeUntil(String expected, String... skipMatchingTokens) throws ParsingException, IllegalStateException
Attempt to consume all tokens until the specified token is found, and then stop before consuming that token. If it is not found, then the token stream is left untouched and a ParsingException is thrown.The
ANY_VALUE
constant can be used in the expected values as a wildcard.- Parameters:
expected
- the token that is to be foundskipMatchingTokens
- the token that, if found, should result in skippingexpected
once for each occurrence ofskipMatchingTokens
; may be null- Returns:
- this token stream instance so callers can chain together methods; never null
- Throws:
ParsingException
- if the specified token cannot be foundIllegalStateException
- if this method was called before the stream wasstarted
-
consumeUntilEndOrOneOf
public TokenStream consumeUntilEndOrOneOf(String... stopTokens) throws ParsingException, IllegalStateException
Consume the token stream until one of the stop tokens or the end of the stream is found.- Parameters:
stopTokens
- the stop tokens; may not be null- Returns:
- this token stream instance so callers can chain together methods; never null
- Throws:
ParsingException
- if none of the specified tokens can be foundIllegalStateException
- if this method was called before the stream wasstarted
-
canConsumeInteger
public boolean canConsumeInteger(IntConsumer consumer) throws IllegalStateException
Attempt to consume this current token if it can be parsed as an integer, and return whether this method was indeed able to consume the token.- Parameters:
consumer
- the function that should be called with the integer value if the current token token could be parsed- Returns:
- true if the current token did match and was consumed, or false if the current token did not match and therefore was not consumed
- Throws:
IllegalStateException
- if this method was called before the stream wasstarted
-
canConsumeBoolean
public boolean canConsumeBoolean(BooleanConsumer consumer) throws IllegalStateException
Attempt to consume this current token if it can be parsed as a boolean, and return whether this method was indeed able to consume the token.- Parameters:
consumer
- the function that should be called with the boolean value if the current token token could be parsed- Returns:
- true if the current token did match and was consumed, or false if the current token did not match and therefore was not consumed
- Throws:
IllegalStateException
- if this method was called before the stream wasstarted
-
canConsumeLong
public boolean canConsumeLong(LongConsumer consumer) throws IllegalStateException
Attempt to consume this current token if it can be parsed as a long, and return whether this method was indeed able to consume the token.- Parameters:
consumer
- the function that should be called with the long value if the current token token could be parsed- Returns:
- true if the current token did match and was consumed, or false if the current token did not match and therefore was not consumed
- Throws:
IllegalStateException
- if this method was called before the stream wasstarted
-
canConsume
public boolean canConsume(String expected) throws IllegalStateException
Attempt to consume this current token if it matches the expected value, and return whether this method was indeed able to consume the token.The
ANY_VALUE
constant can be used in the expected value as a wildcard.- Parameters:
expected
- the expected value of the current token- Returns:
- true if the current token did match and was consumed, or false if the current token did not match and therefore was not consumed
- Throws:
IllegalStateException
- if this method was called before the stream wasstarted
-
canConsume
public boolean canConsume(int type, String expected) throws IllegalStateException
Attempt to consume this current token if it matches the expected value, and return whether this method was indeed able to consume the token.The
ANY_VALUE
constant can be used in the expected value as a wildcard.- Parameters:
type
- the expected type of the current tokenexpected
- the expected value of the current token- Returns:
- true if the current token did match and was consumed, or false if the current token did not match and therefore was not consumed
- Throws:
IllegalStateException
- if this method was called before the stream wasstarted
-
canConsumeWord
public boolean canConsumeWord(String expected) throws IllegalStateException
Attempt to consume this current token if it isTokenStream.BasicTokenizer.WORD
and it matches the expected value, and return whether this method was indeed able to consume the token.The
ANY_VALUE
constant can be used in the expected value as a wildcard.- Parameters:
expected
- the expected value of the current token- Returns:
- true if the current token did match and was consumed, or false if the current token did not match and therefore was not consumed
- Throws:
IllegalStateException
- if this method was called before the stream wasstarted
-
canConsume
public boolean canConsume(char expected) throws IllegalStateException
Attempt to consume this current token if it matches the expected value, and return whether this method was indeed able to consume the token.- Parameters:
expected
- the expected value of the current token token- Returns:
- true if the current token did match and was consumed, or false if the current token did not match and therefore was not consumed
- Throws:
IllegalStateException
- if this method was called before the stream wasstarted
-
canConsume
public boolean canConsume(int expectedType) throws IllegalStateException
Attempt to consume this current token if it matches the expected token type, and return whether this method was indeed able to consume the token.The
ANY_TYPE
constant can be used in the expected type as a wildcard.- Parameters:
expectedType
- the expected token type of the current token- Returns:
- true if the current token did match and was consumed, or false if the current token did not match and therefore was not consumed
- Throws:
IllegalStateException
- if this method was called before the stream wasstarted
-
canConsume
public boolean canConsume(String currentExpected, String... expectedForNextTokens) throws IllegalStateException
Attempt to consume this current token and the next tokens if and only if they match the expected values, and return whether this method was indeed able to consume all of the supplied tokens.This is not the same as calling
canConsume(String)
for each of the supplied arguments, since this method ensures that all of the supplied values can be consumed.This method is equivalent to calling the following:
if (tokens.matches(currentExpected, expectedForNextTokens)) { tokens.consume(currentExpected, expectedForNextTokens); }
The
ANY_VALUE
constant can be used in the expected values as a wildcard.- Parameters:
currentExpected
- the expected value of the current tokenexpectedForNextTokens
- the expected values fo the following tokens- Returns:
- true if the current token did match and was consumed, or false if the current token did not match and therefore was not consumed
- Throws:
IllegalStateException
- if this method was called before the stream wasstarted
-
canConsume
public boolean canConsume(int type, String currentExpected, String... expectedForNextTokens) throws IllegalStateException
Attempt to consume this current token and the next tokens if and only if they match the expected type and values, and return whether this method was indeed able to consume all of the supplied tokens.This is not the same as calling
#canConsume(type String)
for each of the supplied arguments, since this method ensures that all of the supplied values can be consumed.This method is equivalent to calling the following:
if (tokens.matches(currentExpected, expectedForNextTokens) && tokens.matches(type, type, ...)) { tokens.consume(currentExpected, expectedForNextTokens); }
The
ANY_VALUE
constant can be used in the expected values as a wildcard.- Parameters:
type
- the expect type of the tokenscurrentExpected
- the expected value of the current tokenexpectedForNextTokens
- the expected values fo the following tokens- Returns:
- true if the current token did match and was consumed, or false if the current token did not match and therefore was not consumed
- Throws:
IllegalStateException
- if this method was called before the stream wasstarted
-
canConsumeWords
public boolean canConsumeWords(String currentExpected, String... expectedForNextTokens) throws IllegalStateException
Attempt to consume this current token and the next tokens if and only if they are ofTokenStream.BasicTokenizer.WORD
and match the expected values, and return whether this method was indeed able to consume all of the supplied tokens.- Parameters:
currentExpected
- the expected value of the current tokenexpectedForNextTokens
- the expected values fo the following tokens- Returns:
- true if the current token did match and was consumed, or false if the current token did not match and therefore was not consumed
- Throws:
IllegalStateException
- if this method was called before the stream wasstarted
- See Also:
canConsume(int, String, String...)
-
canConsume
public boolean canConsume(String[] nextTokens) throws IllegalStateException
Attempt to consume this current token and the next tokens if and only if they match the expected values, and return whether this method was indeed able to consume all of the supplied tokens.This is not the same as calling
canConsume(String)
for each of the supplied arguments, since this method ensures that all of the supplied values can be consumed.This method is equivalent to calling the following:
if (tokens.matches(currentExpected, expectedForNextTokens)) { tokens.consume(currentExpected, expectedForNextTokens); }
The
ANY_VALUE
constant can be used in the expected values as a wildcard.- Parameters:
nextTokens
- the expected values of the next tokens- Returns:
- true if the current token did match and was consumed, or false if the current token did not match and therefore was not consumed
- Throws:
IllegalStateException
- if this method was called before the stream wasstarted
-
canConsume
public boolean canConsume(Iterable<String> nextTokens) throws IllegalStateException
Attempt to consume this current token and the next tokens if and only if they match the expected values, and return whether this method was indeed able to consume all of the supplied tokens.This is not the same as calling
canConsume(String)
for each of the supplied arguments, since this method ensures that all of the supplied values can be consumed.This method is equivalent to calling the following:
if (tokens.matches(currentExpected, expectedForNextTokens)) { tokens.consume(currentExpected, expectedForNextTokens); }
The
ANY_VALUE
constant can be used in the expected values as a wildcard.- Parameters:
nextTokens
- the expected values of the next tokens- Returns:
- true if the current token did match and was consumed, or false if the current token did not match and therefore was not consumed
- Throws:
IllegalStateException
- if this method was called before the stream wasstarted
-
canConsumeAnyOf
public boolean canConsumeAnyOf(String firstOption, String... additionalOptions) throws IllegalStateException
Attempt to consume the next token if it matches one of the supplied values.- Parameters:
firstOption
- the first option for the value of the current tokenadditionalOptions
- the additional options for the value of the current token- Returns:
- true if the current token's value did match one of the supplied options, or false otherwise
- Throws:
IllegalStateException
- if this method was called before the stream wasstarted
-
canConsumeAnyOf
public boolean canConsumeAnyOf(String[] options) throws IllegalStateException
Attempt to consume the next token if it matches one of the supplied values.- Parameters:
options
- the options for the value of the current token- Returns:
- true if the current token's value did match one of the suplied options, or false otherwise
- Throws:
IllegalStateException
- if this method was called before the stream wasstarted
-
canConsumeAnyOf
public boolean canConsumeAnyOf(Iterable<String> options) throws IllegalStateException
Attempt to consume the next token if it matches one of the supplied values.- Parameters:
options
- the options for the value of the current token- Returns:
- true if the current token's value did match one of the supplied options, or false otherwise
- Throws:
IllegalStateException
- if this method was called before the stream wasstarted
-
canConsumeAnyOf
public boolean canConsumeAnyOf(int firstTypeOption, int... additionalTypeOptions) throws IllegalStateException
Attempt to consume the next token if it matches one of the supplied types.- Parameters:
firstTypeOption
- the first option for the type of the current tokenadditionalTypeOptions
- the additional options for the type of the current token- Returns:
- true if the current token's type matched one of the supplied options, or false otherwise
- Throws:
IllegalStateException
- if this method was called before the stream wasstarted
-
canConsumeAnyOf
public boolean canConsumeAnyOf(int[] typeOptions) throws IllegalStateException
Attempt to consume the next token if it matches one of the supplied types.- Parameters:
typeOptions
- the options for the type of the current token- Returns:
- true if the current token's type matched one of the supplied options, or false otherwise
- Throws:
IllegalStateException
- if this method was called before the stream wasstarted
-
matches
public boolean matches(String expected) throws IllegalStateException
Determine if the current token matches the expected value.The
ANY_VALUE
constant can be used as a wildcard.- Parameters:
expected
- the expected value of the current token- Returns:
- true if the current token did match, or false if the current token did not match
- Throws:
IllegalStateException
- if this method was called before the stream wasstarted
-
matches
public boolean matches(int type, String expected) throws IllegalStateException
Determine if the current token matches the expected type and a value.The
ANY_VALUE
constant can be used as a wildcard.- Parameters:
type
- the expected type of the curent tokenexpected
- the expected value of the current token- Returns:
- true if the current token did match, or false if the current token did not match
- Throws:
IllegalStateException
- if this method was called before the stream wasstarted
-
matchesWord
public boolean matchesWord(String expected) throws IllegalStateException
Determine if the current token isTokenStream.BasicTokenizer.WORD
and matches the expected value.The
ANY_VALUE
constant can be used as a wildcard.- Parameters:
expected
- the expected value of the current token- Returns:
- true if the current token did match, or false if the current token did not match
- Throws:
IllegalStateException
- if this method was called before the stream wasstarted
-
matches
public boolean matches(char expected) throws IllegalStateException
Determine if the current token matches the expected value.- Parameters:
expected
- the expected value of the current token token- Returns:
- true if the current token did match, or false if the current token did not match
- Throws:
IllegalStateException
- if this method was called before the stream wasstarted
-
matches
public boolean matches(int expectedType) throws IllegalStateException
Determine if the current token matches the expected token type.- Parameters:
expectedType
- the expected token type of the current token- Returns:
- true if the current token did match, or false if the current token did not match
- Throws:
IllegalStateException
- if this method was called before the stream wasstarted
-
matches
public boolean matches(String currentExpected, String... expectedForNextTokens) throws IllegalStateException
Determine if the next few tokens match the expected values.The
ANY_VALUE
constant can be used in the expected values as a wildcard.- Parameters:
currentExpected
- the expected value of the current tokenexpectedForNextTokens
- the expected values for the following tokens- Returns:
- true if the tokens did match, or false otherwise
- Throws:
IllegalStateException
- if this method was called before the stream wasstarted
-
matches
public boolean matches(String[] nextTokens) throws IllegalStateException
Determine if the next few tokens match the expected values.The
ANY_VALUE
constant can be used in the expected values as a wildcard.- Parameters:
nextTokens
- the expected value of the next tokens- Returns:
- true if the tokens did match, or false otherwise
- Throws:
IllegalStateException
- if this method was called before the stream wasstarted
-
matches
public boolean matches(Iterable<String> nextTokens) throws IllegalStateException
Determine if the next few tokens match the expected values.The
ANY_VALUE
constant can be used in the expected values as a wildcard.- Parameters:
nextTokens
- the expected value of the next tokens- Returns:
- true if the tokens did match, or false otherwise
- Throws:
IllegalStateException
- if this method was called before the stream wasstarted
-
matches
public boolean matches(int currentExpectedType, int... expectedTypeForNextTokens) throws IllegalStateException
Determine if the next few tokens have the supplied types.The
ANY_TYPE
constant can be used in the expected values as a wildcard.- Parameters:
currentExpectedType
- the expected type of the current tokenexpectedTypeForNextTokens
- the expected type for the following tokens- Returns:
- true if the tokens did match, or false otherwise
- Throws:
IllegalStateException
- if this method was called before the stream wasstarted
-
matches
public boolean matches(int[] typesForNextTokens) throws IllegalStateException
Determine if the next few tokens have the supplied types.The
ANY_TYPE
constant can be used in the expected values as a wildcard.- Parameters:
typesForNextTokens
- the expected type for each of the next tokens- Returns:
- true if the tokens did match, or false otherwise
- Throws:
IllegalStateException
- if this method was called before the stream wasstarted
-
matchesAnyOf
public boolean matchesAnyOf(String firstOption, String... additionalOptions) throws IllegalStateException
Determine if the next token matches one of the supplied values.- Parameters:
firstOption
- the first option for the value of the current tokenadditionalOptions
- the additional options for the value of the current token- Returns:
- true if the current token's value did match one of the supplied options, or false otherwise
- Throws:
IllegalStateException
- if this method was called before the stream wasstarted
-
matchesAnyOf
public boolean matchesAnyOf(int type, String firstOption, String... additionalOptions) throws IllegalStateException
Determine if the next token matches one of the supplied values of the expected type.- Parameters:
type
- the expected type of tokensfirstOption
- the first option for the value of the current tokenadditionalOptions
- the additional options for the value of the current token- Returns:
- true if the current token's value did match one of the supplied options, or false otherwise
- Throws:
IllegalStateException
- if this method was called before the stream wasstarted
-
matchesAnyWordOf
public boolean matchesAnyWordOf(String firstOption, String... additionalOptions) throws IllegalStateException
Determine if the next token matches one of the supplied values of the typeTokenStream.BasicTokenizer.WORD
- Parameters:
firstOption
- the first option for the value of the current tokenadditionalOptions
- the additional options for the value of the current token- Returns:
- true if the current token's value did match one of the supplied options, or false otherwise
- Throws:
IllegalStateException
- if this method was called before the stream wasstarted
-
matchesAnyOf
public boolean matchesAnyOf(String[] options) throws IllegalStateException
Determine if the next token matches one of the supplied values.- Parameters:
options
- the options for the value of the current token- Returns:
- true if the current token's value did match one of the supplied options, or false otherwise
- Throws:
IllegalStateException
- if this method was called before the stream wasstarted
-
matchesAnyOf
public boolean matchesAnyOf(Iterable<String> options) throws IllegalStateException
Determine if the next token matches one of the supplied values.- Parameters:
options
- the options for the value of the current token- Returns:
- true if the current token's value did match one of the supplied options, or false otherwise
- Throws:
IllegalStateException
- if this method was called before the stream wasstarted
-
matchesAnyOf
public boolean matchesAnyOf(int firstTypeOption, int... additionalTypeOptions) throws IllegalStateException
Determine if the next token have one of the supplied types.- Parameters:
firstTypeOption
- the first option for the type of the current tokenadditionalTypeOptions
- the additional options for the type of the current token- Returns:
- true if the current token's type matched one of the supplied options, or false otherwise
- Throws:
IllegalStateException
- if this method was called before the stream wasstarted
-
matchesAnyOf
public boolean matchesAnyOf(int[] typeOptions) throws IllegalStateException
Determine if the next token have one of the supplied types.- Parameters:
typeOptions
- the options for the type of the current token- Returns:
- true if the current token's type matched one of the supplied options, or false otherwise
- Throws:
IllegalStateException
- if this method was called before the stream wasstarted
-
hasNext
public boolean hasNext()
Determine if this stream has another token to be consumed.- Returns:
- true if there is another token ready for consumption, or false otherwise
- Throws:
IllegalStateException
- if this method was called before the stream wasstarted
-
moveToNextToken
private void moveToNextToken(List<TokenStream.Token> newTokens)
-
moveToNextToken
private void moveToNextToken()
-
currentToken
final TokenStream.Token currentToken() throws IllegalStateException, NoSuchElementException
Get the current token.- Returns:
- the current token; never null
- Throws:
IllegalStateException
- if this method was called before the stream wasstarted
NoSuchElementException
- if there are no more tokens
-
getContentFrom
public String getContentFrom(TokenStream.Marker starting)
Gets the content string starting at the specified marker (inclusive) and continuing up to the next position (exclusive).- Parameters:
starting
- the marker describing a point in the stream; may not be null- Returns:
- the content string; never null
-
getContentBetween
public String getContentBetween(TokenStream.Marker starting, Position end)
Gets the content string starting at the specified marker (inclusive) and continuing up to the end position (exclusive).- Parameters:
starting
- the marker describing a point in the stream; may not be nullend
- the position located directly after the returned content string; can be null, which means end of content- Returns:
- the content string; never null
-
getContentBetween
public String getContentBetween(Position starting, Position end)
Gets the content string starting at the first position (inclusive) and continuing up to the end position (exclusive).- Parameters:
starting
- the position marking the beginning of the desired content string; may not be nullend
- the position located directly after the returned content string; can be null, which means end of content- Returns:
- the content string; never null
-
previousToken
public final TokenStream.Token previousToken(int count) throws IllegalStateException, NoSuchElementException
Get the previous token. This does not modify the state.- Parameters:
count
- the number of tokens back from the current position that this method should return- Returns:
- the previous token; never null
- Throws:
IllegalStateException
- if this method was called before the stream wasstarted
NoSuchElementException
- if there is no previous token
-
generateFragment
String generateFragment()
-
generateFragment
static String generateFragment(String content, int indexOfProblem, int charactersToIncludeBeforeAndAfter, String highlightText)
Utility method to generate a highlighted fragment of a particular point in the stream.- Parameters:
content
- the content from which the fragment should be taken; may not be nullindexOfProblem
- the index of the problem point that should be highlighted; must be a valid index in the contentcharactersToIncludeBeforeAndAfter
- the maximum number of characters before and after the problem point to include in the fragmenthighlightText
- the text that should be included in the fragment at the problem point to highlight the location, or an empty string if there should be no highlighting- Returns:
- the highlighted fragment; never null
-
basicTokenizer
public static TokenStream.BasicTokenizer basicTokenizer(boolean includeComments)
Obtain a basicTokenStream.Tokenizer
implementation that ignores whitespace but includes tokens for individual symbols, the period ('.'), single-quoted strings, double-quoted strings, whitespace-delimited words, and optionally comments.Note that the resulting Tokenizer may not be appropriate in many situations, but is provided merely as a convenience for those situations that happen to be able to use it.
- Parameters:
includeComments
- true if the comments should be retained and be included in the token stream, or false if comments should be stripped and not included in the token stream- Returns:
- the tokenizer; never null
-
getInputString
public String getInputString()
- Returns:
- a string to be parsed
-
-