Package net.sourceforge.pmd.cpd

Interface Tokenizer

All Known Implementing Classes:

AntlrTokenizer, AnyTokenizer, JavaCCTokenizer, TokenizerBase

public interface Tokenizer

Tokenizes a source file into tokens consumable by CPD.

Method Summary

Modifier and Type	Method	Description
static void	tokenize(Tokenizer tokenizer, TextDocument textDocument, Tokens tokens)	Wraps a call to tokenize(TextDocument, TokenFactory) to properly create and close the token factory.
void	tokenize(TextDocument document, TokenFactory tokens)	Tokenize the source code and record tokens using the provided token factory.

Method Detail

tokenize

void tokenize(TextDocument document,
              TokenFactory tokens)
       throws IOException

Tokenize the source code and record tokens using the provided token factory.

Throws:

IOException

tokenize

static void tokenize(Tokenizer tokenizer,
                     TextDocument textDocument,
                     Tokens tokens)
              throws IOException

Wraps a call to tokenize(TextDocument, TokenFactory) to properly create and close the token factory.

Throws:

IOException