java.lang.Object
- gw.internal.gosu.parser.SourceCodeTokenizerInternal

```
public final class SourceCodeTokenizerInternal
extends Object
```
This class is adapted from java.io.SourceCodeTokenizer. It adds the notion of operator to the mix. You can define your own operators or use the default set of operators, which are taken from the Java Language Spec. It also captures state information for use by our parser e.g., current token location, line number and column.

Field Summary

Fields
Modifier and Type	Field	Description
`boolean`	`_bUnterminatedComment`
`boolean`	`_bUnterminatedString`
`int`	`_iInvalidCharPos`
`int`	`_iType`
`Keyword`	`_keyword`
`String`	`_strValue`
`protected static int`	`CT_ALPHA`
`protected static int`	`CT_BITSHIFT_OPERATOR`
`protected static int`	`CT_CHARQUOTE`
`protected static int`	`CT_COMMENT`
`protected static int`	`CT_DIGIT`
`protected static int`	`CT_OPERATOR`
`protected static int`	`CT_QUOTE`
`protected static int`	`CT_WHITESPACE`

Constructor Summary

Constructors
Constructor	Description
`SourceCodeTokenizerInternal(SourceCodeReader reader)`
`SourceCodeTokenizerInternal(SourceCodeReader reader, ITokenizerInstructor instructor)`
`SourceCodeTokenizerInternal(Reader reader)`
`SourceCodeTokenizerInternal(CharSequence sourceCode)`

Method Summary

All Methods Static Methods Instance Methods Concrete Methods
Modifier and Type	Method	Description
`void`	`bitshiftOperatorChars(int iLow, int iHigh)`
`void`	`charQuoteChar(int ch)`
`void`	`commentChar(int ch)`
`Token`	`copy()`
`void`	`eolIsSignificant(boolean bFlag)`
`static String[]`	`getBitshiftOperators()`
`static List<String>`	`getDefaultBindingOperators()`
`static String[]`	`getDefaultOperators()`
`Token`	`getEofToken()`
`ITokenizerInstructor`	`getInstructor()`
`int`	`getLineNumber()`
`int`	`getLineOffset()`
`SourceCodeReader`	`getReader()`
`String`	`getSource()`
`int`	`getTokenColumn()`
`int`	`getTokenEnd()`
`Stack<Token>`	`getTokens()`
`int`	`getTokenStart()`
`int`	`getType()`
`protected void`	`incrementLineNumber()`
`boolean`	`isAnalyzingDirective()`
`boolean`	`isAnalyzingSeparately()`
`boolean`	`isCommentsSignificant()`
`boolean`	`isEOF()`
`boolean`	`isNOTHING()`
`boolean`	`isOperator(String strOperator)`
`boolean`	`isParseDotsAsOperators()`
`boolean`	`isSupportsKeywords()`
`boolean`	`isUnterminatedComment()`
`boolean`	`isUnterminatedString()`
`boolean`	`isWhitespaceSignificant()`
`void`	`lowerCaseMode(boolean bLowerCaseMode)`
`int`	`nextToken()`
`void`	`operatorChars(int iLow, int iHigh)`
`void`	`operators(String[] astrOperators)`
`void`	`ordinaryChar(int ch)`
`void`	`ordinaryChars(int iLow, int iHigh)`
`void`	`parseNumbers()`
`DocCommentBlock`	`popLastComment()`
`void`	`quoteChar(int ch)`
`protected int`	`readWithInstructions(int c)`
`void`	`reset()`
`void`	`reset(SourceCodeReader reader)`
`void`	`reset(Reader reader)`
`void`	`setCommentsSignificant(boolean bCommentsSignificant)`
`void`	`setInstructor(ITokenizerInstructor instructor)`
`void`	`setParseDotsAsOperators(boolean parseDotsAsOperators)`
`void`	`setSupportsKeywords(boolean supportsKeywords)`
`void`	`setWhitespaceSignificant(boolean bWhitespaceSignificant)`
`String`	`toString()`
`void`	`whitespaceChars(int iLow, int iHigh)`
`void`	`wordChars(int iLow, int iHigh)`

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

CT_WHITESPACE
```
protected static final int CT_WHITESPACE
```
See Also:

Constant Field Values

CT_DIGIT
```
protected static final int CT_DIGIT
```
See Also:

Constant Field Values

CT_ALPHA
```
protected static final int CT_ALPHA
```
See Also:

Constant Field Values

CT_QUOTE
```
protected static final int CT_QUOTE
```
See Also:

Constant Field Values

CT_COMMENT
```
protected static final int CT_COMMENT
```
See Also:

Constant Field Values

CT_OPERATOR
```
protected static final int CT_OPERATOR
```
See Also:

Constant Field Values

CT_BITSHIFT_OPERATOR
```
protected static final int CT_BITSHIFT_OPERATOR
```
See Also:

Constant Field Values

CT_CHARQUOTE
```
protected static final int CT_CHARQUOTE
```
See Also:

Constant Field Values

_iType
```
public int _iType
```

_keyword
```
public Keyword _keyword
```

_strValue
```
public String _strValue
```

_iInvalidCharPos
```
public int _iInvalidCharPos
```

_bUnterminatedString
```
public boolean _bUnterminatedString
```

_bUnterminatedComment
```
public boolean _bUnterminatedComment
```

Constructor Detail

SourceCodeTokenizerInternal

public SourceCodeTokenizerInternal(CharSequence sourceCode)

SourceCodeTokenizerInternal

public SourceCodeTokenizerInternal(Reader reader)

SourceCodeTokenizerInternal

public SourceCodeTokenizerInternal(SourceCodeReader reader)

SourceCodeTokenizerInternal

public SourceCodeTokenizerInternal(SourceCodeReader reader,
                                   ITokenizerInstructor instructor)

Method Detail

copy
```
public Token copy()
```

reset
```
public void reset()
```

reset
```
public void reset(Reader reader)
```

reset

public void reset(SourceCodeReader reader)

getReader
```
public SourceCodeReader getReader()
```

getSource
```
public String getSource()
```

getInstructor

public ITokenizerInstructor getInstructor()

setInstructor

public void setInstructor(ITokenizerInstructor instructor)

isWhitespaceSignificant

public boolean isWhitespaceSignificant()

setWhitespaceSignificant

public void setWhitespaceSignificant(boolean bWhitespaceSignificant)

isCommentsSignificant

public boolean isCommentsSignificant()

setCommentsSignificant

public void setCommentsSignificant(boolean bCommentsSignificant)

getLineNumber
```
public int getLineNumber()
```

getLineOffset
```
public int getLineOffset()
```

incrementLineNumber
```
protected void incrementLineNumber()
```

getTokenColumn
```
public int getTokenColumn()
```

wordChars

public void wordChars(int iLow,
                      int iHigh)

whitespaceChars

public void whitespaceChars(int iLow,
                            int iHigh)

ordinaryChars

public void ordinaryChars(int iLow,
                          int iHigh)

ordinaryChar
```
public void ordinaryChar(int ch)
```

getDefaultOperators

public static String[] getDefaultOperators()

getDefaultBindingOperators

public static List<String> getDefaultBindingOperators()

getBitshiftOperators

public static String[] getBitshiftOperators()

operators

public void operators(String[] astrOperators)

operatorChars

public void operatorChars(int iLow,
                          int iHigh)

bitshiftOperatorChars

public void bitshiftOperatorChars(int iLow,
                                  int iHigh)

isOperator

public boolean isOperator(String strOperator)

commentChar
```
public void commentChar(int ch)
```

quoteChar
```
public void quoteChar(int ch)
```

charQuoteChar
```
public void charQuoteChar(int ch)
```

parseNumbers
```
public void parseNumbers()
```

eolIsSignificant

public void eolIsSignificant(boolean bFlag)

lowerCaseMode

public void lowerCaseMode(boolean bLowerCaseMode)

isUnterminatedString
```
public boolean isUnterminatedString()
```

isUnterminatedComment

public boolean isUnterminatedComment()

setParseDotsAsOperators

public void setParseDotsAsOperators(boolean parseDotsAsOperators)

isParseDotsAsOperators

public boolean isParseDotsAsOperators()

popLastComment

public DocCommentBlock popLastComment()

getTokenStart
```
public int getTokenStart()
```

getTokenEnd
```
public int getTokenEnd()
```

readWithInstructions

protected int readWithInstructions(int c)
                            throws IOException

Throws:: IOException

isAnalyzingSeparately

public boolean isAnalyzingSeparately()

isAnalyzingDirective
```
public boolean isAnalyzingDirective()
```

getTokens
```
public final Stack<Token> getTokens()
```

nextToken

public int nextToken()
              throws IOException

Throws:: IOException

toString
```
public String toString()
```
Overrides:

toString in class Object

getType
```
public int getType()
```

isEOF
```
public boolean isEOF()
```

isNOTHING
```
public boolean isNOTHING()
```

getEofToken
```
public Token getEofToken()
```

isSupportsKeywords
```
public boolean isSupportsKeywords()
```

setSupportsKeywords

public void setSupportsKeywords(boolean supportsKeywords)

Class SourceCodeTokenizerInternal

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

CT_WHITESPACE

CT_DIGIT

CT_ALPHA

CT_QUOTE

CT_COMMENT

CT_OPERATOR

CT_BITSHIFT_OPERATOR

CT_CHARQUOTE

_iType

_keyword

_strValue

_iInvalidCharPos

_bUnterminatedString

_bUnterminatedComment

Constructor Detail

SourceCodeTokenizerInternal

SourceCodeTokenizerInternal

SourceCodeTokenizerInternal

SourceCodeTokenizerInternal

Method Detail

copy

reset

reset

reset

getReader

getSource

getInstructor

setInstructor

isWhitespaceSignificant

setWhitespaceSignificant

isCommentsSignificant

setCommentsSignificant

getLineNumber

getLineOffset

incrementLineNumber

getTokenColumn

wordChars

whitespaceChars

ordinaryChars

ordinaryChar

getDefaultOperators

getDefaultBindingOperators

getBitshiftOperators

operators

operatorChars

bitshiftOperatorChars

isOperator

commentChar

quoteChar

charQuoteChar

parseNumbers

eolIsSignificant

lowerCaseMode

isUnterminatedString

isUnterminatedComment

setParseDotsAsOperators

isParseDotsAsOperators

popLastComment

getTokenStart

getTokenEnd

readWithInstructions

isAnalyzingSeparately

isAnalyzingDirective

getTokens

nextToken

toString

getType

isEOF

isNOTHING

getEofToken

isSupportsKeywords

setSupportsKeywords