Create a lexer for "raw" tokens.
Create a lexer for "raw" tokens.
rawTokenise
Convert the given Pilar source code into a list of "raw" tokens.
Convert the given Pilar source code into a list of "raw" tokens.
This includes whitespace and comment tokens. No NEWLINE or NEWLINES tokens are inferred. The final token will be of type EOF.
Convert the given Pilar source code into a list of tokens.
Convert the given Pilar source code into a list of tokens.
whitespace and comments are absorbed into the token they precede. The final token will be of type EOF.