weka.attributeSelection

Class ClassifierSubsetEval

java.lang.Object

weka.attributeSelection.ASEvaluation

weka.attributeSelection.HoldOutSubsetEvaluator

weka.attributeSelection.ClassifierSubsetEval

All Implemented Interfaces:

Serializable, ErrorBasedMeritEvaluator, SubsetEvaluator, CapabilitiesHandler, CapabilitiesIgnorer, CommandlineRunnable, OptionHandler, RevisionHandler

public class ClassifierSubsetEval
extends HoldOutSubsetEvaluator
implements OptionHandler, ErrorBasedMeritEvaluator

Classifier subset evaluator:

Evaluates attribute subsets on training data or a seperate hold out testing set. Uses a classifier to estimate the 'merit' of a set of attributes.

Valid options are:

 -B <classifier>
  class name of the classifier to use for accuracy estimation.
  Place any classifier options LAST on the command line
  following a "--". eg.:
   -B weka.classifiers.bayes.NaiveBayes ... -- -K
  (default: weka.classifiers.rules.ZeroR)

 -T
  Use the training data to estimate accuracy.

 -H <filename>
  Name of the hold out/test set to 
  estimate accuracy on.

 -percentage-split
  Perform a percentage split on the training data.
  Use in conjunction with -T.

 -P
  Split percentage to use (default = 90).

 -S
  Random seed for percentage split (default = 1).

 -E <DEFAULT|ACC|RMSE|MAE|F-MEAS|AUC|AUPRC|CORR-COEFF>
  Performance evaluation measure to use for selecting attributes.
  (Default = default: accuracy for discrete class and rmse for numeric class)

 -IRclass <label | index>
  Optional class value (label or 1-based index) to use in conjunction with
  IR statistics (f-meas, auc or auprc). Omitting this option will use
  the class-weighted average.

 
 Options specific to scheme weka.classifiers.rules.ZeroR:
 

 -output-debug-info
  If set, classifier is run in debug mode and
  may output additional info to the console

 -do-not-check-capabilities
  If set, classifier capabilities are not checked before classifier is built
  (use with caution).

 -num-decimal-places
  The number of decimal places for the output of numbers in the model (default 2).

 -batch-size
  The desired batch size for batch prediction  (default 100).

Version:

$Revision: 10332 $

Author:

Mark Hall ([email protected])

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
static int	EVAL_ACCURACY
static int	EVAL_AUC
static int	EVAL_AUPRC
static int	EVAL_CORRELATION
static int	EVAL_DEFAULT
static int	EVAL_FMEASURE
static int	EVAL_MAE
static int	EVAL_PLUGIN
static int	EVAL_RMSE
static Tag[]	TAGS_EVALUATION Holds all tags for metrics

Constructor Summary

Constructors

Constructor and Description
ClassifierSubsetEval()

Method Summary

Modifier and Type	Method and Description
void	buildEvaluator(Instances data) Generates a attribute evaluator.
String	classifierTipText() Returns the tip text for this property
double	evaluateSubset(BitSet subset) Evaluates a subset of attributes
double	evaluateSubset(BitSet subset, Instance holdOut, boolean retrain) Evaluates a subset of attributes with respect to a single instance.
double	evaluateSubset(BitSet subset, Instances holdOut) Evaluates a subset of attributes with respect to a set of instances.
String	evaluationMeasureTipText() Returns the tip text for this property
Capabilities	getCapabilities() Returns the capabilities of this evaluator.
Classifier	getClassifier() Get the classifier used as the base learner.
SelectedTag	getEvaluationMeasure() Gets the currently set performance evaluation measure used for selecting attributes for the decision table
File	getHoldOutFile() Gets the file that holds hold out/test instances.
String	getIRClassValue() Get the class value (label or index) to use with IR metric evaluation of subsets.
String[]	getOptions() Gets the current settings of ClassifierSubsetEval
String	getRevision() Returns the revision string.
int	getSeed() Get the random seed used to randomize the data before performing a percentage split
String	getSplitPercent() Get the split percentage to use
boolean	getUsePercentageSplit() Get whether to perform a percentage split on the training data for evaluation
boolean	getUseTraining() Get if training data is to be used instead of hold out/test data
String	globalInfo() Returns a string describing this attribute evaluator
String	holdOutFileTipText() Returns the tip text for this property
String	IRClassValueTipText() Returns the tip text for this property
Enumeration<Option>	listOptions() Returns an enumeration describing the available options.
static void	main(String[] args) Main method for testing this class.
String	seedTipText() Returns the tip text for this property
void	setClassifier(Classifier newClassifier) Set the classifier to use for accuracy estimation
void	setEvaluationMeasure(SelectedTag newMethod) Sets the performance evaluation measure to use for selecting attributes for the decision table
void	setHoldOutFile(File h) Set the file that contains hold out/test instances
void	setIRClassValue(String val) Set the class value (label or index) to use with IR metric evaluation of subsets.
void	setOptions(String[] options) Parses a given list of options.
void	setSeed(int s) Set the random seed used to randomize the data before performing a percentage split
void	setSplitPercent(String sp) Set the split percentage to use
void	setUsePercentageSplit(boolean p) Set whether to perform a percentage split on the training data for evaluation
void	setUseTraining(boolean t) Set if training data is to be used instead of hold out/test data
String	splitPercentTipText() Returns the tip text for this property
String	toString() Returns a string describing classifierSubsetEval
String	usePercentageSplitTipText() Returns the tip text for this property
String	useTrainingTipText() Returns the tip text for this property

Methods inherited from class weka.attributeSelection.ASEvaluation

clean, doNotCheckCapabilitiesTipText, forName, getDoNotCheckCapabilities, makeCopies, postExecution, postProcess, preExecution, run, runEvaluator, setDoNotCheckCapabilities

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

EVAL_DEFAULT

public static final int EVAL_DEFAULT

See Also:

Constant Field Values

EVAL_ACCURACY

public static final int EVAL_ACCURACY

See Also:

Constant Field Values

EVAL_RMSE

public static final int EVAL_RMSE

See Also:

Constant Field Values

EVAL_MAE

public static final int EVAL_MAE

See Also:

Constant Field Values

EVAL_FMEASURE

public static final int EVAL_FMEASURE

See Also:

Constant Field Values

EVAL_AUC

public static final int EVAL_AUC

See Also:

Constant Field Values

EVAL_AUPRC

public static final int EVAL_AUPRC

See Also:

Constant Field Values

EVAL_CORRELATION

public static final int EVAL_CORRELATION

See Also:

Constant Field Values

EVAL_PLUGIN

public static final int EVAL_PLUGIN

See Also:

Constant Field Values

TAGS_EVALUATION

public static final Tag[] TAGS_EVALUATION

Holds all tags for metrics

Constructor Detail

ClassifierSubsetEval

public ClassifierSubsetEval()

Method Detail

globalInfo

public String globalInfo()

Returns a string describing this attribute evaluator

Returns:

a description of the evaluator suitable for displaying in the explorer/experimenter gui

listOptions

public Enumeration<Option> listOptions()

Returns an enumeration describing the available options.

Specified by:

listOptions in interface OptionHandler

Returns:

an enumeration of all the available options.

setOptions

public void setOptions(String[] options)
                throws Exception

Parses a given list of options.

Valid options are:

 -B <classifier>
  class name of the classifier to use for accuracy estimation.
  Place any classifier options LAST on the command line
  following a "--". eg.:
   -B weka.classifiers.bayes.NaiveBayes ... -- -K
  (default: weka.classifiers.rules.ZeroR)

 -T
  Use the training data to estimate accuracy.

 -H <filename>
  Name of the hold out/test set to 
  estimate accuracy on.

 -percentage-split
  Perform a percentage split on the training data.
  Use in conjunction with -T.

 -P
  Split percentage to use (default = 90).

 -S
  Random seed for percentage split (default = 1).

 -E <DEFAULT|ACC|RMSE|MAE|F-MEAS|AUC|AUPRC|CORR-COEFF>
  Performance evaluation measure to use for selecting attributes.
  (Default = default: accuracy for discrete class and rmse for numeric class)

 -IRclass <label | index>
  Optional class value (label or 1-based index) to use in conjunction with
  IR statistics (f-meas, auc or auprc). Omitting this option will use
  the class-weighted average.

 
 Options specific to scheme weka.classifiers.rules.ZeroR:
 

 -output-debug-info
  If set, classifier is run in debug mode and
  may output additional info to the console

 -do-not-check-capabilities
  If set, classifier capabilities are not checked before classifier is built
  (use with caution).

 -num-decimal-places
  The number of decimal places for the output of numbers in the model (default 2).

 -batch-size
  The desired batch size for batch prediction  (default 100).

Specified by:

setOptions in interface OptionHandler

Parameters:

options - the list of options as an array of strings

Throws:

Exception - if an option is not supported

seedTipText

public String seedTipText()

Returns the tip text for this property

Returns:

tip text for this property suitable for displaying in the explorer/experimenter gui

setSeed

public void setSeed(int s)

Set the random seed used to randomize the data before performing a percentage split

Parameters:

s - the seed to use

getSeed

public int getSeed()

Get the random seed used to randomize the data before performing a percentage split

Returns:

the seed to use

usePercentageSplitTipText

public String usePercentageSplitTipText()

Returns the tip text for this property

Returns:

tip text for this property suitable for displaying in the explorer/experimenter gui

setUsePercentageSplit

public void setUsePercentageSplit(boolean p)

Set whether to perform a percentage split on the training data for evaluation

Parameters:

p - true if a percentage split is to be performed

getUsePercentageSplit

public boolean getUsePercentageSplit()

Get whether to perform a percentage split on the training data for evaluation

Returns:

true if a percentage split is to be performed

splitPercentTipText

public String splitPercentTipText()

Returns the tip text for this property

Returns:

tip text for this property suitable for displaying in the explorer/experimenter gui

setSplitPercent

public void setSplitPercent(String sp)

Set the split percentage to use

Parameters:

sp - the split percentage to use

getSplitPercent

public String getSplitPercent()

Get the split percentage to use

Returns:

the split percentage to use

setIRClassValue

public void setIRClassValue(String val)

Set the class value (label or index) to use with IR metric evaluation of subsets. Leaving this unset will result in the class weighted average for the IR metric being used.

Parameters:

val - the class label or 1-based index of the class label to use when evaluating subsets with an IR metric

getIRClassValue

public String getIRClassValue()

Get the class value (label or index) to use with IR metric evaluation of subsets. Leaving this unset will result in the class weighted average for the IR metric being used.

Returns:

the class label or 1-based index of the class label to use when evaluating subsets with an IR metric

IRClassValueTipText

public String IRClassValueTipText()

Returns the tip text for this property

Returns:

tip text for this property suitable for displaying in the explorer/experimenter gui

evaluationMeasureTipText

public String evaluationMeasureTipText()

Returns the tip text for this property

Returns:

tip text for this property suitable for displaying in the explorer/experimenter gui

getEvaluationMeasure

public SelectedTag getEvaluationMeasure()

Gets the currently set performance evaluation measure used for selecting attributes for the decision table

Returns:

the performance evaluation measure

setEvaluationMeasure

public void setEvaluationMeasure(SelectedTag newMethod)

Sets the performance evaluation measure to use for selecting attributes for the decision table

Parameters:

newMethod - the new performance evaluation metric to use

classifierTipText

public String classifierTipText()

Returns the tip text for this property

Returns:

tip text for this property suitable for displaying in the explorer/experimenter gui

setClassifier

public void setClassifier(Classifier newClassifier)

Set the classifier to use for accuracy estimation

Parameters:

newClassifier - the Classifier to use.

getClassifier

public Classifier getClassifier()

Get the classifier used as the base learner.

Returns:

the classifier used as the classifier

holdOutFileTipText

public String holdOutFileTipText()

Returns the tip text for this property

Returns:

tip text for this property suitable for displaying in the explorer/experimenter gui

getHoldOutFile

public File getHoldOutFile()

Gets the file that holds hold out/test instances.

Returns:

File that contains hold out instances

setHoldOutFile

public void setHoldOutFile(File h)

Set the file that contains hold out/test instances

Parameters:

h - the hold out file

useTrainingTipText

public String useTrainingTipText()

Returns the tip text for this property

Returns:

tip text for this property suitable for displaying in the explorer/experimenter gui

getUseTraining

public boolean getUseTraining()

Get if training data is to be used instead of hold out/test data

Returns:

true if training data is to be used instead of hold out data

setUseTraining

public void setUseTraining(boolean t)

Set if training data is to be used instead of hold out/test data

Parameters:

t - true if training data is to be used instead of hold out data

getOptions

public String[] getOptions()

Gets the current settings of ClassifierSubsetEval

Specified by:

getOptions in interface OptionHandler

Returns:

an array of strings suitable for passing to setOptions()

getCapabilities

public Capabilities getCapabilities()

Returns the capabilities of this evaluator.

Specified by:

getCapabilities in interface CapabilitiesHandler

Overrides:

getCapabilities in class ASEvaluation

Returns:

the capabilities of this evaluator

See Also:

Capabilities

buildEvaluator

public void buildEvaluator(Instances data)
                    throws Exception

Generates a attribute evaluator. Has to initialize all fields of the evaluator that are not being set via options.

Specified by:

buildEvaluator in class ASEvaluation

Parameters:

data - set of instances serving as training data

Throws:

Exception - if the evaluator has not been generated successfully

evaluateSubset

public double evaluateSubset(BitSet subset)
                      throws Exception

Evaluates a subset of attributes

Specified by:

evaluateSubset in interface SubsetEvaluator

Parameters:

subset - a bitset representing the attribute subset to be evaluated

Returns:

the error rate

Throws:

Exception - if the subset could not be evaluated

evaluateSubset

public double evaluateSubset(BitSet subset,
                             Instances holdOut)
                      throws Exception

Evaluates a subset of attributes with respect to a set of instances. Calling this function overrides any test/hold out instances set from setHoldOutFile.

Specified by:

evaluateSubset in class HoldOutSubsetEvaluator

Parameters:

subset - a bitset representing the attribute subset to be evaluated

holdOut - a set of instances (possibly separate and distinct from those use to build/train the evaluator) with which to evaluate the merit of the subset

Returns:

the "merit" of the subset on the holdOut data

Throws:

Exception - if the subset cannot be evaluated

evaluateSubset

public double evaluateSubset(BitSet subset,
                             Instance holdOut,
                             boolean retrain)
                      throws Exception

Evaluates a subset of attributes with respect to a single instance. Calling this function overides any hold out/test instances set through setHoldOutFile.

Specified by:

evaluateSubset in class HoldOutSubsetEvaluator

Parameters:

subset - a bitset representing the attribute subset to be evaluated

holdOut - a single instance (possibly not one of those used to build/train the evaluator) with which to evaluate the merit of the subset

retrain - true if the classifier should be retrained with respect to the new subset before testing on the holdOut instance.

Returns:

the "merit" of the subset on the holdOut instance

Throws:

Exception - if the subset cannot be evaluated

toString

public String toString()

Returns a string describing classifierSubsetEval

Overrides:

toString in class Object

Returns:

the description as a string

getRevision

public String getRevision()

Returns the revision string.

Specified by:

getRevision in interface RevisionHandler

Overrides:

getRevision in class ASEvaluation

Returns:

the revision

main

public static void main(String[] args)

Main method for testing this class.

Parameters:

args - the options