com.oracle.js.parser

Class Token

java.lang.Object

com.oracle.js.parser.Token

public final class Token
extends Object

A token is a 64 bit long value that represents a basic parse/lex unit. This class provides static methods to manipulate lexer tokens.

Field Summary

Fields

Modifier and Type	Field and Description
static int	LENGTH_MASK We use 28 bits for the position and 28 bits for the length of the token.

Method Summary

Modifier and Type	Method and Description
static int	descLength(long token) Extract token length from a token descriptor.
static int	descPosition(long token) Extract token position from a token descriptor.
static TokenType	descType(long token) Extract token type from a token descriptor.
static long	recast(long token, TokenType newType) Change the token to use a new type.
static long	toDesc(TokenType type, int position, int length) Create a compact form of token information.
static String	toString(Source source, long token) String conversion of token
static String	toString(Source source, long token, boolean verbose) Return a string representation of a token.
static long	withDelimiter(long token) Normally returns the token itself, except in case of string tokens which report their position past their opening delimiter and thus need to have position and length adjusted.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LENGTH_MASK

public static final int LENGTH_MASK

We use 28 bits for the position and 28 bits for the length of the token. This limits the maximal length of code we can handle to 2 ^ 28 - 1 bytes.

See Also:

Constant Field Values

Method Detail

toDesc

public static long toDesc(TokenType type,
                          int position,
                          int length)

Create a compact form of token information.

Parameters:

type - Type of token.

position - Start position of the token in the source.

length - Length of the token.

Returns:

Token descriptor.

descPosition

public static int descPosition(long token)

Extract token position from a token descriptor.

Parameters:

token - Token descriptor.

Returns:

Start position of the token in the source.

withDelimiter

public static long withDelimiter(long token)

Normally returns the token itself, except in case of string tokens which report their position past their opening delimiter and thus need to have position and length adjusted.

Parameters:

token - Token descriptor.

Returns:

same or adjusted token.

descLength

public static int descLength(long token)

Extract token length from a token descriptor.

Parameters:

token - Token descriptor.

Returns:

Length of the token.

descType

public static TokenType descType(long token)

Extract token type from a token descriptor.

Parameters:

token - Token descriptor.

Returns:

Type of token.

recast

public static long recast(long token,
                          TokenType newType)

Change the token to use a new type.

Parameters:

token - The original token.

newType - The new token type.

Returns:

The recast token.

toString

public static String toString(Source source,
                              long token,
                              boolean verbose)

Return a string representation of a token.

Parameters:

source - Token source.

token - Token descriptor.

verbose - True to include details.

Returns:

String representation.

toString

public static String toString(Source source,
                              long token)

String conversion of token

Parameters:

source - the source

token - the token

Returns:

token as string