Class Scanner
java.lang.Object
org.aspectj.org.eclipse.jdt.internal.compiler.parser.Scanner
- All Implemented Interfaces:
TerminalTokens
- Direct Known Subclasses:
CompletionScanner
,JavadocScanner
,RecoveryScanner
,SelectionScanner
IMPORTANT NOTE: Internal Scanner implementation. It is mirrored in
org.eclipse.jdt.core.compiler public package where it is API.
The mirror implementation is using the backward compatible ITerminalSymbols constant
definitions (stable with 2.0), whereas the internal implementation uses TerminalTokens
which constant values reflect the latest parser generation state.
-
Nested Class Summary
Modifier and TypeClassDescriptionstatic final record
static interface
static final record
-
Field Summary
Modifier and TypeFieldDescriptionstatic final String
static final int
int
boolean
boolean
static int
int
int[]
int[]
int[]
long
boolean
static final int
char
int
boolean
static final String
int
boolean
int
char[][]
int[][]
char[][]
char[][]
static final int
static final int
static final char[]
static final String
int
protected boolean
boolean
static final String
static final String
static final String
static final String
static final String
static final String
static final String
static final String
static final String
static final String
static final String
static final String
static final String
static final String
boolean
int
protected int
int[]
int
static final int
static final int
protected boolean
protected int
protected int
static final String
boolean
boolean
boolean
static final int
protected org.aspectj.org.eclipse.jdt.internal.compiler.parser.Scanner.ScanContext
boolean
boolean
char[]
long
static final int
int
static final char
static final int
static final char[]
static final int
char[][]
char[][]
protected int
boolean
boolean
static final String
boolean
static final String
static final String
static final String
boolean
boolean
boolean[]
boolean
char[]
int
Fields inherited from interface org.aspectj.org.eclipse.jdt.internal.compiler.parser.TerminalTokens
TokenNameabstract, TokenNameafter, TokenNameAND, TokenNameAND_AND, TokenNameAND_EQUAL, TokenNamearound, TokenNameARROW, TokenNameaspect, TokenNameassert, TokenNameAT, TokenNameAT308, TokenNameAT308DOTDOTDOT, TokenNamebefore, TokenNameBeginCaseElement, TokenNameBeginCaseExpr, TokenNameBeginIntersectionCast, TokenNameBeginLambda, TokenNameBeginTypeArguments, TokenNameboolean, TokenNamebreak, TokenNamebyte, TokenNamecase, TokenNamecatch, TokenNamechar, TokenNameCharacterLiteral, TokenNameclass, TokenNameCOLON, TokenNameCOLON_COLON, TokenNameCOMMA, TokenNameCOMMENT_BLOCK, TokenNameCOMMENT_JAVADOC, TokenNameCOMMENT_LINE, TokenNameconst, TokenNamecontinue, TokenNamedeclare, TokenNamedefault, TokenNameDIVIDE, TokenNameDIVIDE_EQUAL, TokenNamedo, TokenNameDOT, TokenNamedouble, TokenNameDoubleLiteral, TokenNameElidedSemicolonAndRightBrace, TokenNameELLIPSIS, TokenNameelse, TokenNameenum, TokenNameEOF, TokenNameEQUAL, TokenNameEQUAL_EQUAL, TokenNameERROR, TokenNameexports, TokenNameextends, TokenNamefalse, TokenNamefinal, TokenNamefinally, TokenNamefloat, TokenNameFloatingPointLiteral, TokenNamefor, TokenNamegoto, TokenNameGREATER, TokenNameGREATER_EQUAL, TokenNameIdentifier, TokenNameif, TokenNameimplements, TokenNameimport, TokenNameinstanceof, TokenNameint, TokenNameIntegerLiteral, TokenNameinterface, TokenNameLBRACE, TokenNameLBRACKET, TokenNameLEFT_SHIFT, TokenNameLEFT_SHIFT_EQUAL, TokenNameLESS, TokenNameLESS_EQUAL, TokenNamelong, TokenNameLongLiteral, TokenNameLPAREN, TokenNameMINUS, TokenNameMINUS_EQUAL, TokenNameMINUS_MINUS, TokenNamemodule, TokenNameMULTIPLY, TokenNameMULTIPLY_EQUAL, TokenNamenative, TokenNamenew, TokenNamenon_sealed, TokenNameNOT, TokenNameNOT_EQUAL, TokenNameNotAToken, TokenNamenull, TokenNameopen, TokenNameopens, TokenNameOR, TokenNameOR_EQUAL, TokenNameOR_OR, TokenNamepackage, TokenNamePLUS, TokenNamePLUS_EQUAL, TokenNamePLUS_PLUS, TokenNamepointcut, TokenNameprivate, TokenNameprivileged, TokenNameprotected, TokenNameprovides, TokenNamepublic, TokenNameQUESTION, TokenNameRBRACE, TokenNameRBRACKET, TokenNameREMAINDER, TokenNameREMAINDER_EQUAL, TokenNamerequires, TokenNameRestrictedIdentifierpermits, TokenNameRestrictedIdentifierrecord, TokenNameRestrictedIdentifiersealed, TokenNameRestrictedIdentifierWhen, TokenNameRestrictedIdentifierYield, TokenNamereturn, TokenNameRIGHT_SHIFT, TokenNameRIGHT_SHIFT_EQUAL, TokenNameRPAREN, TokenNameSEMICOLON, TokenNameshort, TokenNameSingleQuoteStringLiteral, TokenNamestatic, TokenNamestrictfp, TokenNameStringLiteral, TokenNameStringTemplate, TokenNamesuper, TokenNameswitch, TokenNamesynchronized, TokenNameTextBlock, TokenNameTextBlockTemplate, TokenNamethis, TokenNamethrow, TokenNamethrows, TokenNameto, TokenNametransient, TokenNametransitive, TokenNametrue, TokenNametry, TokenNameTWIDDLE, TokenNameUNDERSCORE, TokenNameUNSIGNED_RIGHT_SHIFT, TokenNameUNSIGNED_RIGHT_SHIFT_EQUAL, TokenNameuses, TokenNamevoid, TokenNamevolatile, TokenNamewhile, TokenNameWHITESPACE, TokenNamewith, TokenNameXOR, TokenNameXOR_EQUAL
-
Constructor Summary
ConstructorDescriptionScanner()
Scanner
(boolean tokenizeComments, boolean tokenizeWhiteSpace, boolean checkNonExternalizedStringLiterals, long sourceLevel, char[][] taskTags, char[][] taskPriorities, boolean isTaskCaseSensitive) Scanner
(boolean tokenizeComments, boolean tokenizeWhiteSpace, boolean checkNonExternalizedStringLiterals, long sourceLevel, char[][] taskTags, char[][] taskPriorities, boolean isTaskCaseSensitive, boolean isPreviewEnabled) Scanner
(boolean tokenizeComments, boolean tokenizeWhiteSpace, boolean checkNonExternalizedStringLiterals, long sourceLevel, long complianceLevel, char[][] taskTags, char[][] taskPriorities, boolean isTaskCaseSensitive, boolean isPreviewEnabled) -
Method Summary
Modifier and TypeMethodDescriptionprotected boolean
final boolean
atEnd()
protected final boolean
void
checkTaskTag
(int commentStart, int commentEnd) protected int
disambiguateArrowWithCaseExpr
(Scanner scanner, int retToken) int
fastForward
(Statement unused) char[]
final String
char[]
protected char[][]
int
char[]
getCurrentTokenInRange
(int start, int end) char[]
char[]
int
final String
boolean[]
final int
getLineEnd
(int lineNumber) final int[]
final int
getLineNumber
(int position) Search the line number corresponding to a specific positionfinal int
getLineStart
(int lineNumber) Search the source position corresponding to the beginning of a given line number Line numbers are 1-based, and relative to the scanner initialPosition.final int
final boolean
getNextChar
(char testedChar) final int
getNextChar
(char testedChar1, char testedChar2) final boolean
final boolean
getNextCharAsDigit
(int radix) boolean
boolean
final int
protected int
Overridable hook, to allow CompletionScanner to hide a faked identifier token.int
protected int
void
NLSTag[]
final char[]
final char[]
char[]
protected static InvalidInputException
protected static InvalidInputException
protected static InvalidInputException
protected static InvalidInputException
protected static InvalidInputException
protected static InvalidInputException
protected static InvalidInputException
protected static InvalidInputException
protected static InvalidInputException
protected static InvalidInputException
static InvalidInputException
protected static InvalidInputException
static InvalidInputException
invalidToken
(int token) protected static InvalidInputException
protected static InvalidInputException
protected static InvalidInputException
protected boolean
protected boolean
static boolean
isIdentifier
(int token) boolean
static boolean
isKeyword
(int token) static boolean
isLiteral
(int token) static boolean
isRestrictedKeyword
(int token) final void
final void
final void
final boolean
protected boolean
protected final boolean
mayBeAtCasePattern
(int token) protected final boolean
protected final boolean
protected int
processSingleQuotes
(boolean checkIfUnicode) final void
final void
void
recordComment
(int token) void
resetTo
(int begin, int end) Reposition the scanner on some portion of the original source.void
resetTo
(int begin, int end, boolean isModuleInfo) void
resetTo
(int begin, int end, boolean isModuleInfo, org.aspectj.org.eclipse.jdt.internal.compiler.parser.Scanner.ScanContext context) Reposition the scanner on some portion of the original source.protected final void
protected int
protected int
protected final boolean
protected final boolean
int
int
int
scanIdentifierOrKeywordAj
(int kind) int
int
scanNumber
(boolean dotPrefix) void
setActiveParser
(ConflictedParser parser) final void
setSource
(char[] sourceString) final void
setSource
(char[] contents, CompilationResult compilationResult) final void
setSource
(CompilationResult compilationResult) toString()
toStringAction
(int act) void
ungetToken
(int unambiguousToken) void
unicodeInitializeBuffer
(int length) void
void
unicodeStore
(char character) protected static InvalidInputException
protected static InvalidInputException
protected static InvalidInputException
protected void
updateCase
(int token)
-
Field Details
-
sourceLevel
public long sourceLevel -
complianceLevel
public long complianceLevel -
useAssertAsAnIndentifier
public boolean useAssertAsAnIndentifier -
containsAssertKeyword
public boolean containsAssertKeyword -
previewEnabled
public boolean previewEnabled -
useEnumAsAnIndentifier
public boolean useEnumAsAnIndentifier -
recordLineSeparator
public boolean recordLineSeparator -
currentCharacter
public char currentCharacter -
startPosition
public int startPosition -
currentPosition
public int currentPosition -
initialPosition
public int initialPosition -
eofPosition
public int eofPosition -
skipComments
public boolean skipComments -
tokenizeComments
public boolean tokenizeComments -
tokenizeWhiteSpace
public boolean tokenizeWhiteSpace -
source
public char[] source -
withoutUnicodeBuffer
public char[] withoutUnicodeBuffer -
withoutUnicodePtr
public int withoutUnicodePtr -
unicodeAsBackSlash
public boolean unicodeAsBackSlash -
scanningFloatLiteral
public boolean scanningFloatLiteral -
COMMENT_ARRAYS_SIZE
public static int COMMENT_ARRAYS_SIZE -
commentStops
public int[] commentStops -
commentStarts
public int[] commentStarts -
commentTagStarts
public int[] commentTagStarts -
commentPtr
public int commentPtr -
lastCommentLinePosition
public int lastCommentLinePosition -
foundTaskTags
public char[][] foundTaskTags -
foundTaskMessages
public char[][] foundTaskMessages -
foundTaskPriorities
public char[][] foundTaskPriorities -
foundTaskPositions
public int[][] foundTaskPositions -
foundTaskCount
public int foundTaskCount -
taskTags
public char[][] taskTags -
taskPriorities
public char[][] taskPriorities -
isTaskCaseSensitive
public boolean isTaskCaseSensitive -
diet
public boolean diet -
lineEnds
public int[] lineEnds -
linePtr
public int linePtr -
wasAcr
public boolean wasAcr -
fakeInModule
public boolean fakeInModule -
caseStartPosition
public int caseStartPosition -
scanContext
protected org.aspectj.org.eclipse.jdt.internal.compiler.parser.Scanner.ScanContext scanContext -
insideModuleInfo
protected boolean insideModuleInfo -
END_OF_SOURCE
- See Also:
-
INVALID_HEXA
- See Also:
-
INVALID_OCTAL
- See Also:
-
INVALID_CHARACTER_CONSTANT
- See Also:
-
INVALID_ESCAPE
- See Also:
-
INVALID_INPUT
- See Also:
-
INVALID_TEXTBLOCK
- See Also:
-
INVALID_UNICODE_ESCAPE
- See Also:
-
INVALID_FLOAT
- See Also:
-
INVALID_LOW_SURROGATE
- See Also:
-
INVALID_HIGH_SURROGATE
- See Also:
-
NULL_SOURCE_STRING
- See Also:
-
UNTERMINATED_STRING
- See Also:
-
UNTERMINATED_TEXT_BLOCK
- See Also:
-
UNTERMINATED_COMMENT
- See Also:
-
INVALID_CHAR_IN_STRING
- See Also:
-
INVALID_DIGIT
- See Also:
-
INVALID_BINARY
- See Also:
-
BINARY_LITERAL_NOT_BELOW_17
- See Also:
-
ILLEGAL_HEXA_LITERAL
- See Also:
-
INVALID_UNDERSCORE
- See Also:
-
UNDERSCORES_IN_LITERALS_NOT_BELOW_17
- See Also:
-
TAG_PREFIX
public static final char[] TAG_PREFIX -
TAG_PREFIX_LENGTH
public static final int TAG_PREFIX_LENGTH -
TAG_POSTFIX
public static final char TAG_POSTFIX- See Also:
-
TAG_POSTFIX_LENGTH
public static final int TAG_POSTFIX_LENGTH- See Also:
-
IDENTITY_COMPARISON_TAG
public static final char[] IDENTITY_COMPARISON_TAG -
validIdentityComparisonLines
public boolean[] validIdentityComparisonLines -
checkUninternedIdentityComparison
public boolean checkUninternedIdentityComparison -
nlsTagsPtr
protected int nlsTagsPtr -
checkNonExternalizedStringLiterals
public boolean checkNonExternalizedStringLiterals -
lastPosition
protected int lastPosition -
returnOnlyGreater
public boolean returnOnlyGreater -
insideRecovery
public boolean insideRecovery -
nextToken
protected int nextToken -
multiCaseLabelComma
protected boolean multiCaseLabelComma -
RoundBracket
public static final int RoundBracket- See Also:
-
SquareBracket
public static final int SquareBracket- See Also:
-
CurlyBracket
public static final int CurlyBracket- See Also:
-
BracketKinds
public static final int BracketKinds- See Also:
-
LOW_SURROGATE_MIN_VALUE
public static final int LOW_SURROGATE_MIN_VALUE- See Also:
-
HIGH_SURROGATE_MIN_VALUE
public static final int HIGH_SURROGATE_MIN_VALUE- See Also:
-
HIGH_SURROGATE_MAX_VALUE
public static final int HIGH_SURROGATE_MAX_VALUE- See Also:
-
LOW_SURROGATE_MAX_VALUE
public static final int LOW_SURROGATE_MAX_VALUE- See Also:
-
textBlockOffset
protected int textBlockOffset
-
-
Constructor Details
-
Scanner
public Scanner() -
Scanner
public Scanner(boolean tokenizeComments, boolean tokenizeWhiteSpace, boolean checkNonExternalizedStringLiterals, long sourceLevel, long complianceLevel, char[][] taskTags, char[][] taskPriorities, boolean isTaskCaseSensitive, boolean isPreviewEnabled) -
Scanner
public Scanner(boolean tokenizeComments, boolean tokenizeWhiteSpace, boolean checkNonExternalizedStringLiterals, long sourceLevel, char[][] taskTags, char[][] taskPriorities, boolean isTaskCaseSensitive, boolean isPreviewEnabled) -
Scanner
public Scanner(boolean tokenizeComments, boolean tokenizeWhiteSpace, boolean checkNonExternalizedStringLiterals, long sourceLevel, char[][] taskTags, char[][] taskPriorities, boolean isTaskCaseSensitive)
-
-
Method Details
-
atEnd
public final boolean atEnd() -
checkTaskTag
- Throws:
InvalidInputException
-
getCurrentIdentifierSource
public char[] getCurrentIdentifierSource() -
getCurrentTokenEndPosition
public int getCurrentTokenEndPosition() -
getCurrentTokenSource
public char[] getCurrentTokenSource() -
getCurrentTokenString
-
getCurrentTokenInRange
public char[] getCurrentTokenInRange(int start, int end) -
getCurrentTokenSourceString
public char[] getCurrentTokenSourceString() -
scanForTextBlockBeginning
protected final boolean scanForTextBlockBeginning() -
scanForTextBlockClose
- Throws:
InvalidInputException
-
getCurrentTextBlock
public char[] getCurrentTextBlock() -
getCurrentTextBlockAsLines
protected char[][] getCurrentTextBlockAsLines() -
getCurrentStringLiteral
-
getRawTokenSource
public final char[] getRawTokenSource() -
getRawTokenSourceEnd
public final char[] getRawTokenSourceEnd() -
getCurrentTokenStartPosition
public int getCurrentTokenStartPosition() -
getLineEnd
public final int getLineEnd(int lineNumber) -
getLineEnds
public final int[] getLineEnds() -
getLineStart
public final int getLineStart(int lineNumber) Search the source position corresponding to the beginning of a given line number Line numbers are 1-based, and relative to the scanner initialPosition. Character positions are 0-based. e.g. getLineStart(1) --> 0 indicates that the first line starts at character 0. In case the given line number is inconsistent, answers -1.- Parameters:
lineNumber
- int- Returns:
- int
-
getNextChar
public final int getNextChar() -
getNextCharWithBoundChecks
public final int getNextCharWithBoundChecks() -
getNextChar
public final boolean getNextChar(char testedChar) -
getNextChar
public final int getNextChar(char testedChar1, char testedChar2) -
getNextCharAsDigit
- Throws:
InvalidInputException
-
getNextCharAsDigit
public final boolean getNextCharAsDigit(int radix) -
getNextCharAsJavaIdentifierPartWithBoundCheck
public boolean getNextCharAsJavaIdentifierPartWithBoundCheck() -
getNextCharAsJavaIdentifierPart
public boolean getNextCharAsJavaIdentifierPart() -
scanIdentifier
- Throws:
InvalidInputException
-
ungetToken
public void ungetToken(int unambiguousToken) -
updateCase
protected void updateCase(int token) -
getNextToken
- Throws:
InvalidInputException
-
getNextToken0
- Throws:
InvalidInputException
-
processSingleQuotes
- Throws:
InvalidInputException
-
scanForStringLiteral
- Throws:
InvalidInputException
-
scanForTextBlock
- Throws:
InvalidInputException
-
getNextUnicodeChar
- Throws:
InvalidInputException
-
getNLSTags
-
getIdentityComparisonLines
public boolean[] getIdentityComparisonLines() -
getSource
public char[] getSource() -
isFirstTag
protected boolean isFirstTag() -
jumpOverMethodBody
public final void jumpOverMethodBody() -
jumpOverEmbeddedExpression
public final void jumpOverEmbeddedExpression() -
jumpOverBody
public final void jumpOverBody() -
jumpOverUnicodeWhiteSpace
- Throws:
InvalidInputException
-
isInModuleDeclaration
public boolean isInModuleDeclaration() -
areRestrictedModuleKeywordsActive
protected boolean areRestrictedModuleKeywordsActive() -
pushLineSeparator
public final void pushLineSeparator() -
pushUnicodeLineSeparator
public final void pushUnicodeLineSeparator() -
recordComment
public void recordComment(int token) -
resetTo
public void resetTo(int begin, int end) Reposition the scanner on some portion of the original source. The given endPosition is the last valid position. Beyond this position, the scanner will answer EOF tokens (ITerminalSymbols.TokenNameEOF
).- Parameters:
begin
- the given start positionend
- the given end position
-
resetTo
public void resetTo(int begin, int end, boolean isModuleInfo) -
resetTo
public void resetTo(int begin, int end, boolean isModuleInfo, org.aspectj.org.eclipse.jdt.internal.compiler.parser.Scanner.ScanContext context) Reposition the scanner on some portion of the original source. The given endPosition is the last valid position. Beyond this position, the scanner will answer EOF tokens (ITerminalSymbols.TokenNameEOF
).- Parameters:
begin
- the given start positionend
- the given end positionisModuleInfo
- if true apply rules for restricted keywords even without a connection to a properly configured parsercontext
- The scan context to use for restricted keyword support, use null to compute
-
scanEscapeCharacter
- Throws:
InvalidInputException
-
scanIdentifierOrKeywordWithBoundCheck
public int scanIdentifierOrKeywordWithBoundCheck() -
scanIdentifierOrKeyword
public int scanIdentifierOrKeyword() -
scanIdentifierOrKeywordAj
public int scanIdentifierOrKeywordAj(int kind) -
scanNumber
- Throws:
InvalidInputException
-
getLineNumber
public final int getLineNumber(int position) Search the line number corresponding to a specific position- Parameters:
position
- int- Returns:
- int
-
setSource
public final void setSource(char[] sourceString) -
setSource
-
setSource
-
toString
-
toStringAction
-
unicodeInitializeBuffer
public void unicodeInitializeBuffer(int length) -
unicodeStore
public void unicodeStore() -
unicodeStore
public void unicodeStore(char character) -
isIdentifier
public static boolean isIdentifier(int token) -
isLiteral
public static boolean isLiteral(int token) -
isKeyword
public static boolean isKeyword(int token) -
mayBeAtCasePattern
protected final boolean mayBeAtCasePattern(int token) -
maybeAtLambdaOrCast
protected final boolean maybeAtLambdaOrCast() -
maybeAtReferenceExpression
protected final boolean maybeAtReferenceExpression() -
atTypeAnnotation
protected final boolean atTypeAnnotation() -
setActiveParser
-
isRestrictedKeyword
public static boolean isRestrictedKeyword(int token) -
disambiguateArrowWithCaseExpr
-
mayBeAtCaseLabelExpr
protected boolean mayBeAtCaseLabelExpr() -
isAtAssistIdentifier
protected boolean isAtAssistIdentifier() -
fastForward
-
getNextNotFakedToken
Overridable hook, to allow CompletionScanner to hide a faked identifier token.- Throws:
InvalidInputException
-
invalidCharacter
-
invalidCharInString
-
unterminatedString
-
invalidUnicodeEscape
-
invalidLowSurrogate
-
invalidHighSurrogate
-
unterminatedComment
-
unterminatedTextBlock
-
invalidEof
-
invalidUnderscore
-
invalidUnderscoresInLiterals
-
invalidEscape
-
invalidHexa
-
illegalHexaLiteral
-
invalidFloat
-
invalidBinaryLiteral
-
invalidBinary
-
invalidToken
-
invalidInput
-