LexToken

parsley.errors.tokenextractors.LexToken

See theLexToken companion object

This extractor mixin provides an implementation for ErrorBuilder.unexpectedToken when mixed into an error builder: it will try and parse the residual input to identify a valid lexical token to report.

When parsing a grammar that as a dedicated lexical distinction, it is nice to be able to report problematic tokens relevant to that grammar as opposed to generic input lifted straight from the input stream. The easiest way of doing this would be having a pre-lexing pass and parsing based on tokens, but this is deliberately not how Parsley is designed. Instead, this extractor can try and parse the remaining input to try and identify a token on demand.

If the lexicalError flag of the unexpectedToken method is not set, which would indicate a problem within a token reported by a classical lexer and not the parser, the extractor will try to parse each of the provided tokens in turn: whichever is the longest matched of these tokens will be reported as the problematic one (this can be changed by overriding selectToken). For best effect, these tokens should not consume whitespace (which would otherwise be included at the end of the token!): this means that, if using the Lexer class, the functionality in nonlexeme should be used. If one of the givens tokens cannot be parsed, the input until the next valid parsable token (or end of input) is returned as a Token.Raw.

Currently, if lexicalError is true, this extractor will just return the next character as the problematic item (this may be changed by overriding the extractItem method).

Attributes

Since: 4.0.0
Companion: object
Source: LexToken.scala
Graph
Supertypes: class Object

trait Matchable

class Any
Self type: ErrorBuilder[_]

Members list

Value members

Abstract methods

The tokens that should be recognised by this extractor: each parser should return the intended name of the token exactly as it should appear in the Named token.

This should include a whitespace parser for "unexpected whitespace".

Attributes

Since: 4.0.0
Note: with the exception of the whitespace parser, these tokens should not consume trailing (and certainly not leading) whitespace: if using definitions from parsley.token.Lexer functionality, the nonlexeme versions of the tokens should be used.
Source: LexToken.scala

Concrete methods

If the parser failed during the parsing of a token, this function extracts the problematic item from the remaining input.

The default behaviour mimics SingleChar.

Attributes

Since: 4.0.0
Source: LexToken.scala

If the extractor is successful in identifying tokens that can be parsed from the residual input, this function will select one of them to report back.

The default behaviour is to take the longest matched token (i.e. the one with the largest paired position). In case of a tie, the first token is chosen: this means that more specific tokens should be put sooner in the tokens list.

Value parameters

matchedToks: the list of tokens successfully parsed, along with the position at the end of that parse (careful: this position starts back at (1, 1), not where the original parser left off!)

Attributes

Returns: the chosen token and position pair
Since: 4.0.0
Note: the matchedToks list is guaranteed to be non-empty
Source: LexToken.scala

Attributes

See also: ErrorBuilder.unexpectedToken
Definition Classes
Source: LexToken.scala

In this article

Generated with