public final class Token extends Object
| Modifier and Type | Field and Description |
|---|---|
static int |
LENGTH_MASK
We use 28 bits for the position and 28 bits for the length of the token.
|
| Modifier and Type | Method and Description |
|---|---|
static int |
descLength(long token)
Extract token length from a token descriptor.
|
static int |
descPosition(long token)
Extract token position from a token descriptor.
|
static TokenType |
descType(long token)
Extract token type from a token descriptor.
|
static long |
recast(long token,
TokenType newType)
Change the token to use a new type.
|
static long |
toDesc(TokenType type,
int position,
int length)
Create a compact form of token information.
|
static String |
toString(Source source,
long token)
String conversion of token
|
static String |
toString(Source source,
long token,
boolean verbose)
Return a string representation of a token.
|
static long |
withDelimiter(long token)
Normally returns the token itself, except in case of string tokens which report their
position past their opening delimiter and thus need to have position and length adjusted.
|
public static final int LENGTH_MASK
public static long toDesc(TokenType type, int position, int length)
type - Type of token.position - Start position of the token in the source.length - Length of the token.public static int descPosition(long token)
token - Token descriptor.public static long withDelimiter(long token)
token - Token descriptor.public static int descLength(long token)
token - Token descriptor.public static TokenType descType(long token)
token - Token descriptor.public static long recast(long token,
TokenType newType)
token - The original token.newType - The new token type.public static String toString(Source source, long token, boolean verbose)
source - Token source.token - Token descriptor.verbose - True to include details.