Package net.sourceforge.pmd.cpd

Class PythonTokenizer

java.lang.Object

net.sourceforge.pmd.cpd.impl.JavaCCTokenizer

net.sourceforge.pmd.cpd.PythonTokenizer

All Implemented Interfaces:

Tokenizer

public class PythonTokenizer
extends JavaCCTokenizer

The Python tokenizer.

Field Summary

Fields inherited from interface net.sourceforge.pmd.cpd.Tokenizer

DEFAULT_SKIP_BLOCKS_PATTERN, IGNORE_ANNOTATIONS, IGNORE_IDENTIFIERS, IGNORE_LITERALS, IGNORE_USINGS, OPTION_IGNORE_LITERAL_SEQUENCES, OPTION_SKIP_BLOCKS, OPTION_SKIP_BLOCKS_PATTERN

Constructor Summary

Constructors

Constructor	Description
PythonTokenizer()

Method Summary

Modifier and Type	Method	Description
protected String	getImage(JavaccToken token)
protected TokenManager<JavaccToken>	makeLexerImpl(CharStream sourceCode)
protected JavaccTokenDocument.TokenDocumentBehavior	tokenBehavior()

Methods inherited from class net.sourceforge.pmd.cpd.impl.JavaCCTokenizer

getLexerForSource, getTokenFilter, processToken, tokenize

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

PythonTokenizer

public PythonTokenizer()

Method Detail

makeLexerImpl

protected TokenManager<JavaccToken> makeLexerImpl(CharStream sourceCode)

Specified by:

makeLexerImpl in class JavaCCTokenizer

tokenBehavior

protected JavaccTokenDocument.TokenDocumentBehavior tokenBehavior()

Overrides:

tokenBehavior in class JavaCCTokenizer

getImage

protected String getImage(JavaccToken token)

Overrides:

getImage in class JavaCCTokenizer