```
public class Tokenizer
extends Object
```
Tokenizer for string literals inside a rule set document.

All Methods Instance Methods Concrete Methods
Modifier and Type	Method	Description
`boolean`	`atEof()`
`void`	`consumeDirectIndexed(Consumer<Integer> consumer)`
`void`	`consumeIdentifier(Consumer<String> consumer)`
`void`	`consumeIndexed(BiConsumer<String,Integer> consumer)`
`void`	`consumeNamedAccess(BiConsumer<String,String> consumer)`
`void`	`consumeReferenceAccess(Consumer<String> consumer)`
`void`	`expectAtEof(String state)`
`boolean`	`isDirectIndexedAccess()`
`boolean`	`isIdentifier()`
`boolean`	`isIndexedAccess()`
`boolean`	`isNamedAccess()`
`boolean`	`isReference()`
`boolean`	`matches(software.amazon.awssdk.codegen.poet.rules2.Tokenizer.TokenKind... kinds)`
`software.amazon.awssdk.codegen.poet.rules2.Tokenizer.Token`	`next()`
`software.amazon.awssdk.codegen.poet.rules2.Tokenizer.Token`	`peek()`

public software.amazon.awssdk.codegen.poet.rules2.Tokenizer.Token peek()

public boolean matches(software.amazon.awssdk.codegen.poet.rules2.Tokenizer.TokenKind... kinds)

public void consumeIndexed(BiConsumer<String,Integer> consumer)

public boolean isDirectIndexedAccess()

public void consumeDirectIndexed(Consumer<Integer> consumer)

public void consumeNamedAccess(BiConsumer<String,String> consumer)

public void consumeIdentifier(Consumer<String> consumer)

public void expectAtEof(String state)

public void consumeReferenceAccess(Consumer<String> consumer)

Class Tokenizer