Scanner

class Scanner(source: SourceFile, val startFrom: Offset, profile: Profile, allowIndent: Boolean)(using x$5: Context) extends ScannerCommon
trait TokenData
class Object
trait Matchable
class Any

Type members

Inherited classlikes

Value members

Concrete methods

def adjustSepRegions(lastToken: Token): Unit
def charLitOr(op: => Token): Unit

Parse character literal if current character is followed by ', or follow with given op and return a symbol literal token

Parse character literal if current character is followed by ', or follow with given op and return a symbol literal token

Return a list of all the comment positions

Return a list of all the comment positions

Returns the closest docstring preceding the position supplied

Returns the closest docstring preceding the position supplied

def handleNewLine(lastToken: Token): Unit

Handle newlines, possibly inserting an INDENT, OUTDENT, NEWLINE, or NEWLINES token in front of the current token. This depends on whether indentation is significant or not.

Handle newlines, possibly inserting an INDENT, OUTDENT, NEWLINE, or NEWLINES token in front of the current token. This depends on whether indentation is significant or not.

Indentation is significant if indentSyntax is set, and we are not inside a {...}, [...], (...), case ... => pair, nor in a if/while condition (i.e. currentRegion is empty).

There are three rules:

  1. Insert NEWLINE or NEWLINES if

    • the closest enclosing sepRegion is { ... } or for ... do/yield, or we are on the toplevel, i.e. currentRegion is empty, and
    • the previous token can end a statement, and
    • the current token can start a statement, and
    • the current token is not a leading infix operator, and
    • if indentation is significant then the current token starts at the current indentation width or to the right of it.

    The inserted token is NEWLINES if the current token is preceded by a whitespace line, or NEWLINE otherwise.

  2. Insert INDENT if

    • indentation is significant, and
    • the last token can start an indentation region.
    • the indentation of the current token is strictly greater than the previous indentation width, or the two widths are the same and the current token is one of : or match.

    The following tokens can start an indentation region:

    : = => <- if then else while do try catch finally for yield match

    Inserting an INDENT starts a new indentation region with the indentation of the current token as indentation width.

  3. Insert OUTDENT if

    • indentation is significant, and
    • the indentation of the current token is strictly less than the previous indentation width,
    • the current token is not a leading infix operator.

    Inserting an OUTDENT closes an indentation region. In this case, issue an error if the indentation of the current token does not match the indentation of some previous line in an enclosing indentation region.

    If a token is inserted and consumed, the original source token is still considered to start a new line, so the process that inserts an OUTDENT might repeat several times.

Indentation widths are strings consisting of spaces and tabs, ordered by the prefix relation. I.e. a <= b iff b.startsWith(a). If indentation is significant it is considered an error if the current indentation width and the indentation of the current token are incomparable.

Is the current token in a position where a modifier is allowed?

Is the current token in a position where a modifier is allowed?

The indentation width of the given offset.

The indentation width of the given offset.

def insert(token: Token, offset: Int): Unit

Insert token at assumed offset in front of current one.

Insert token at assumed offset in front of current one.

def isLeadingInfixOperator(nextWidth: IndentWidth, inConditional: Boolean): Boolean

A leading symbolic or backquoted identifier is treated as an infix operator if

A leading symbolic or backquoted identifier is treated as an infix operator if

  • it does not follow a blank line, and
  • it is followed by at least one whitespace character and a token that can start an expression.
  • if the operator appears on its own line, the next line must have at least the same indentation width as the operator. See pos/i12395 for a test where this matters. If a leading infix operator is found and the source version is 3.0-migration, emit a change warning.

The next token after this one. The token is computed via fetchToken, so complex two word tokens such as CASECLASS are not recognized. Newlines and indent/unindent tokens are skipped. Restriction: lookahead is illegal if the current token is INTERPOLATIONID

The next token after this one. The token is computed via fetchToken, so complex two word tokens such as CASECLASS are not recognized. Newlines and indent/unindent tokens are skipped. Restriction: lookahead is illegal if the current token is INTERPOLATIONID

Produce next token, filling TokenData fields of Scanner.

Produce next token, filling TokenData fields of Scanner.

def observeColonEOL(inTemplate: Boolean): Unit

Insert an token if next token closes an indentation region. Exception: continue if indentation region belongs to a match and next token is case.

Insert an token if next token closes an indentation region. Exception: continue if indentation region belongs to a match and next token is case.

def postProcessToken(lastToken: Token, lastName: SimpleName): Unit
  • Join CASE + CLASS => CASECLASS, CASE + OBJECT => CASEOBJECT SEMI + ELSE => ELSE, COLON following id/)/] => COLONfollow
  • Insert missing OUTDENTs at EOF
final def printState(): Unit
def reset(): Unit
def resume(lastTokenData: TokenData): Unit
def skip(): Unit

Skip on error to next safe point.

Skip on error to next safe point.

final def skipParens(multiple: Boolean): Unit

Skip matching pairs of (...) or [...] parentheses.

Skip matching pairs of (...) or [...] parentheses.

read next token and return last offset

read next token and return last offset

def skipToken[T](result: T): T
def spaceTabMismatchMsg(lastWidth: IndentWidth, nextWidth: IndentWidth): String
override def toString: String

Returns a string representation of the object.

Returns a string representation of the object.

The default representation is platform dependent.

Returns:

a string representation of the object.

Definition Classes
Any
def toToken(identifier: SimpleName): Token

The token for given name. Either IDENTIFIER or a keyword.

The token for given name. Either IDENTIFIER or a keyword.

Inherited methods

Inherited from:
TokenData
protected def decodeUni: Boolean

Switch whether unicode should be decoded

Switch whether unicode should be decoded

Inherited from:
CharArrayReader
def error(msg: String, off: Offset): Unit

Generate an error at the given offset

Generate an error at the given offset

Inherited from:
ScannerCommon
Inherited from:
ScannerCommon
inline def finishNamed(): Unit

Finish an IDENTIFIER with this.name.

Finish an IDENTIFIER with this.name.

Inherited from:
ScannerCommon
def finishNamedToken(idtoken: Token, target: TokenData): Unit

Clear buffer and set name and token. If target is different from this, don't treat identifiers as end tokens.

Clear buffer and set name and token. If target is different from this, don't treat identifiers as end tokens.

Inherited from:
ScannerCommon
def getc(): Char
Inherited from:
CharArrayReader

signal an error where the input ended in the middle of a token

signal an error where the input ended in the middle of a token

Inherited from:
ScannerCommon

Is current token first one after a newline?

Is current token first one after a newline?

Inherited from:
TokenData
Inherited from:
TokenData
Inherited from:
CharArrayReader
Inherited from:
TokenData
def isIdent(name: Name): Boolean
Inherited from:
TokenData
Inherited from:
TokenData
Inherited from:
TokenData
Inherited from:
TokenData
Inherited from:
TokenData
Inherited from:
TokenData
Inherited from:
TokenData

Is last character a unicode escape \uxxxx?

Is last character a unicode escape \uxxxx?

Inherited from:
CharArrayReader
Inherited from:
CharArrayReader

A new reader that takes off at the current character position

A new reader that takes off at the current character position

Inherited from:
CharArrayReader
final def nextChar(): Unit

Advance one character; reducing CR;LF pairs to just LF

Advance one character; reducing CR;LF pairs to just LF

Inherited from:
CharArrayReader
final def nextRawChar(): Unit

Advance one character, leaving CR;LF pairs intact. This is for use in multi-line strings, so there are no "potential line ends" here.

Advance one character, leaving CR;LF pairs intact. This is for use in multi-line strings, so there are no "potential line ends" here.

Inherited from:
CharArrayReader
protected def putChar(c: Char): Unit

append Unicode character to "litBuf" buffer

append Unicode character to "litBuf" buffer

Inherited from:
ScannerCommon

Clear buffer and set string

Clear buffer and set string

Inherited from:
ScannerCommon
Inherited from:
ScannerCommon

Concrete fields

The current region. This is initially an Indented region with zero indentation width.

The current region. This is initially an Indented region with zero indentation width.

We need one token lookahead and one token history

We need one token lookahead and one token history

override val startFrom: Offset

Inherited fields

var base: Int

the base of a number

the base of a number

Inherited from:
TokenData
val buf: Array[Char]
Inherited from:
ScannerCommon
var ch: Char

the last read character

the last read character

Inherited from:
CharArrayReader

The offset one past the last read character

The offset one past the last read character

Inherited from:
CharArrayReader

the last error offset

the last error offset

Inherited from:
ScannerCommon

The offset before the last read character

The offset before the last read character

Inherited from:
CharArrayReader

the offset of the character following the token preceding this one

the offset of the character following the token preceding this one

Inherited from:
TokenData

the offset of the newline immediately preceding the token, or -1 if token is not preceded by a newline.

the offset of the newline immediately preceding the token, or -1 if token is not preceded by a newline.

Inherited from:
TokenData

The start offset of the current line

The start offset of the current line

Inherited from:
CharArrayReader
protected val litBuf: CharBuffer

A character buffer for literals

A character buffer for literals

Inherited from:
ScannerCommon

the name of an identifier

the name of an identifier

Inherited from:
TokenData

the offset of the first character of the current token

the offset of the first character of the current token

Inherited from:
TokenData

the string value of a literal

the string value of a literal

Inherited from:
TokenData

the next token

the next token

Inherited from:
TokenData