LookaheadScanner

class LookaheadScanner() extends Scanner
class Scanner
trait TokenData
class Object
trait Matchable
class Any

Type members

Inherited classlikes

class LookaheadScanner() extends Scanner
Inherited from:
Scanner

Value members

Concrete methods

Definition Classes

Inherited methods

def adjustSepRegions(lastToken: Token): Unit
Inherited from:
Scanner
def charLitOr(op: => Token): Unit

Parse character literal if current character is followed by ', or follow with given op and return a symbol literal token

Parse character literal if current character is followed by ', or follow with given op and return a symbol literal token

Inherited from:
Scanner
Inherited from:
Scanner
Inherited from:
Scanner

Return a list of all the comment positions

Return a list of all the comment positions

Inherited from:
Scanner
Inherited from:
TokenData
protected def decodeUni: Boolean

Switch whether unicode should be decoded

Switch whether unicode should be decoded

Inherited from:
CharArrayReader
Inherited from:
Scanner
def error(msg: String, off: Offset): Unit

Generate an error at the given offset

Generate an error at the given offset

Inherited from:
ScannerCommon
Inherited from:
ScannerCommon
Inherited from:
Scanner
inline def finishNamed(): Unit

Finish an IDENTIFIER with this.name.

Finish an IDENTIFIER with this.name.

Inherited from:
ScannerCommon
def finishNamedToken(idtoken: Token, target: TokenData): Unit

Clear buffer and set name and token. If target is different from this, don't treat identifiers as end tokens.

Clear buffer and set name and token. If target is different from this, don't treat identifiers as end tokens.

Inherited from:
ScannerCommon

Returns the closest docstring preceding the position supplied

Returns the closest docstring preceding the position supplied

Inherited from:
Scanner
def getc(): Char
Inherited from:
CharArrayReader
def handleNewLine(lastToken: Token): Unit

Handle newlines, possibly inserting an INDENT, OUTDENT, NEWLINE, or NEWLINES token in front of the current token. This depends on whether indentation is significant or not.

Handle newlines, possibly inserting an INDENT, OUTDENT, NEWLINE, or NEWLINES token in front of the current token. This depends on whether indentation is significant or not.

Indentation is significant if indentSyntax is set, and we are not inside a {...}, [...], (...), case ... => pair, nor in a if/while condition (i.e. currentRegion is empty).

There are three rules:

  1. Insert NEWLINE or NEWLINES if

    • the closest enclosing sepRegion is { ... } or for ... do/yield, or we are on the toplevel, i.e. currentRegion is empty, and
    • the previous token can end a statement, and
    • the current token can start a statement, and
    • the current token is not a leading infix operator, and
    • if indentation is significant then the current token starts at the current indentation width or to the right of it.

    The inserted token is NEWLINES if the current token is preceded by a whitespace line, or NEWLINE otherwise.

  2. Insert INDENT if

    • indentation is significant, and
    • the last token can start an indentation region.
    • the indentation of the current token is strictly greater than the previous indentation width, or the two widths are the same and the current token is one of : or match.

    The following tokens can start an indentation region:

    : = => <- if then else while do try catch finally for yield match

    Inserting an INDENT starts a new indentation region with the indentation of the current token as indentation width.

  3. Insert OUTDENT if

    • indentation is significant, and
    • the indentation of the current token is strictly less than the previous indentation width,
    • the current token is not a leading infix operator.

    Inserting an OUTDENT closes an indentation region. In this case, issue an error if the indentation of the current token does not match the indentation of some previous line in an enclosing indentation region.

    If a token is inserted and consumed, the original source token is still considered to start a new line, so the process that inserts an OUTDENT might repeat several times.

Indentation widths are strings consisting of spaces and tabs, ordered by the prefix relation. I.e. a <= b iff b.startsWith(a). If indentation is significant it is considered an error if the current indentation width and the indentation of the current token are incomparable.

Inherited from:
Scanner

Is the current token in a position where a modifier is allowed?

Is the current token in a position where a modifier is allowed?

Inherited from:
Scanner

signal an error where the input ended in the middle of a token

signal an error where the input ended in the middle of a token

Inherited from:
ScannerCommon

The indentation width of the given offset.

The indentation width of the given offset.

Inherited from:
Scanner
def insert(token: Token, offset: Int): Unit

Insert token at assumed offset in front of current one.

Insert token at assumed offset in front of current one.

Inherited from:
Scanner

Is current token first one after a newline?

Is current token first one after a newline?

Inherited from:
TokenData
Inherited from:
TokenData
Inherited from:
CharArrayReader

Is token a COLON, after having converted COLONEOL to COLON? The conversion means that indentation is not significant after : anymore. So, warning: this is a side-effecting operation.

Is token a COLON, after having converted COLONEOL to COLON? The conversion means that indentation is not significant after : anymore. So, warning: this is a side-effecting operation.

Inherited from:
TokenData
Inherited from:
Scanner
def isIdent(name: Name): Boolean
Inherited from:
TokenData
Inherited from:
TokenData
def isLeadingInfixOperator(nextWidth: IndentWidth, inConditional: Boolean): Boolean

A leading symbolic or backquoted identifier is treated as an infix operator if

A leading symbolic or backquoted identifier is treated as an infix operator if

  • it does not follow a blank line, and
  • it is followed by at least one whitespace character and a token that can start an expression.
  • if the operator appears on its own line, the next line must have at least the same indentation width as the operator. See pos/i12395 for a test where this matters. If a leading infix operator is found and the source version is 3.0-migration, emit a change warning.
Inherited from:
Scanner
Inherited from:
TokenData
Inherited from:
TokenData
Inherited from:
TokenData
Inherited from:
TokenData
Inherited from:
Scanner
Inherited from:
TokenData

Is last character a unicode escape \uxxxx?

Is last character a unicode escape \uxxxx?

Inherited from:
CharArrayReader
Inherited from:
Scanner
Inherited from:
Scanner

The next token after this one. The token is computed via fetchToken, so complex two word tokens such as CASECLASS are not recognized. Newlines and indent/unindent tokens are skipped.

The next token after this one. The token is computed via fetchToken, so complex two word tokens such as CASECLASS are not recognized. Newlines and indent/unindent tokens are skipped.

Inherited from:
Scanner
Inherited from:
CharArrayReader

A new reader that takes off at the current character position

A new reader that takes off at the current character position

Inherited from:
CharArrayReader
Inherited from:
Scanner
final def nextChar(): Unit

Advance one character; reducing CR;LF pairs to just LF

Advance one character; reducing CR;LF pairs to just LF

Inherited from:
CharArrayReader
final def nextRawChar(): Unit

Advance one character, leaving CR;LF pairs intact. This is for use in multi-line strings, so there are no "potential line ends" here.

Advance one character, leaving CR;LF pairs intact. This is for use in multi-line strings, so there are no "potential line ends" here.

Inherited from:
CharArrayReader

Produce next token, filling TokenData fields of Scanner.

Produce next token, filling TokenData fields of Scanner.

Inherited from:
Scanner
Inherited from:
Scanner
Inherited from:
Scanner

Insert an token if next token closes an indentation region. Exception: continue if indentation region belongs to a match and next token is case.

Insert an token if next token closes an indentation region. Exception: continue if indentation region belongs to a match and next token is case.

Inherited from:
Scanner
  • Join CASE + CLASS => CASECLASS, CASE + OBJECT => CASEOBJECT SEMI + ELSE => ELSE, COLON + => COLONEOL
  • Insert missing OUTDENTs at EOF
Inherited from:
Scanner
Inherited from:
Scanner
final def printState(): Unit
Inherited from:
Scanner
protected def putChar(c: Char): Unit

append Unicode character to "litBuf" buffer

append Unicode character to "litBuf" buffer

Inherited from:
ScannerCommon
def reset(): Unit
Inherited from:
Scanner
def resume(lastTokenData: TokenData): Unit
Inherited from:
Scanner

Clear buffer and set string

Clear buffer and set string

Inherited from:
ScannerCommon
Inherited from:
Scanner
def skip(): Unit

Skip on error to next safe point.

Skip on error to next safe point.

Inherited from:
Scanner
final def skipParens(multiple: Boolean): Unit

Skip matching pairs of (...) or [...] parentheses.

Skip matching pairs of (...) or [...] parentheses.

Inherited from:
Scanner
def skipToken[T](result: T): T
Inherited from:
Scanner

read next token and return last offset

read next token and return last offset

Inherited from:
Scanner
Inherited from:
ScannerCommon
def spaceTabMismatchMsg(lastWidth: IndentWidth, nextWidth: IndentWidth): String
Inherited from:
Scanner
override def toString: String

Returns a string representation of the object.

Returns a string representation of the object.

The default representation is platform dependent.

Returns:

a string representation of the object.

Definition Classes
Scanner -> Any
Inherited from:
Scanner
def toToken(identifier: SimpleName): Token

The token for given name. Either IDENTIFIER or a keyword.

The token for given name. Either IDENTIFIER or a keyword.

Inherited from:
Scanner

Inherited fields

var base: Int

the base of a number

the base of a number

Inherited from:
TokenData
val buf: Array[Char]
Inherited from:
ScannerCommon
var ch: Char

the last read character

the last read character

Inherited from:
CharArrayReader

The offset one past the last read character

The offset one past the last read character

Inherited from:
CharArrayReader

The current region. This is initially an Indented region with zero indentation width.

The current region. This is initially an Indented region with zero indentation width.

Inherited from:
Scanner
Inherited from:
Scanner

the last error offset

the last error offset

Inherited from:
ScannerCommon
Inherited from:
Scanner
Inherited from:
Scanner

The offset before the last read character

The offset before the last read character

Inherited from:
CharArrayReader

the offset of the character following the token preceding this one

the offset of the character following the token preceding this one

Inherited from:
TokenData

the offset of the newline immediately preceding the token, or -1 if token is not preceded by a newline.

the offset of the newline immediately preceding the token, or -1 if token is not preceded by a newline.

Inherited from:
TokenData

The start offset of the current line

The start offset of the current line

Inherited from:
CharArrayReader
protected val litBuf: CharBuffer

A character buffer for literals

A character buffer for literals

Inherited from:
ScannerCommon

the name of an identifier

the name of an identifier

Inherited from:
TokenData
Inherited from:
Scanner

We need one token lookahead and one token history

We need one token lookahead and one token history

Inherited from:
Scanner
Inherited from:
Scanner

the offset of the first character of the current token

the offset of the first character of the current token

Inherited from:
TokenData
Inherited from:
Scanner
Inherited from:
Scanner
Inherited from:
Scanner
Inherited from:
Scanner
override val startFrom: Offset
Inherited from:
Scanner

the string value of a literal

the string value of a literal

Inherited from:
TokenData

the next token

the next token

Inherited from:
TokenData