org.linqs.psl.evaluation.statistics

Class Evaluator

java.lang.Object

org.linqs.psl.evaluation.statistics.Evaluator

Direct Known Subclasses:

AUCEvaluator, CategoricalEvaluator, ContinuousEvaluator, DiscreteEvaluator

public abstract class Evaluator
extends Object

Compute some metric (or set of metrics) for some predicted and labeled data. Every evaluator will have some "representative metric" that is usually set through config options. For example, a discrete Evaluator may have F1 as its representative metric. This is so that more automated methods like search-based weight learning and CLI evaluation can easily access all types of metrics through a common interface. The evaluator will also have to provide some indication as to if higher values for the representative metric is better than lower values. In addition to the representative metric, subclasses classes will usually provide their own methods to access specific metrics. One of the compute methods must be called before attempting to get statistics.

Field Summary

Fields

Modifier and Type	Field and Description
protected boolean	closeTruth
protected boolean	includeObserved

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	Evaluator()

Method Summary

Modifier and Type	Method and Description
void	compute(Database rvDB, Database truthDB, StandardPredicate predicate)
void	compute(Database rvDB, Database truthDB, StandardPredicate predicate, boolean rvDBCached) A convenience call for those who don't want to create a training map directly.
abstract void	compute(TrainingMap data) One of the main computation method.
abstract void	compute(TrainingMap data, StandardPredicate predicate) One of the main computation method.
abstract String	getAllStats() Get a string that contains the full range of stats that this Evaluator can provide.
abstract double	getBestRepScore() The maximum (i.e.
boolean	getCloseTruth()
boolean	getIncludeObserved()
Iterable<Map.Entry<GroundAtom,GroundAtom>>	getMap(TrainingMap trainingMap) Get the full mapping of target atoms to truth atoms.
double	getNormalizedMaxRepMetric() Combine getBestRepScore() with isHigherRepBetter() so that the values from getNormalizedRepMetric are always less than or equal to this value.
double	getNormalizedRepMetric() Combine getRepMetric() with isHigherRepBetter() so that higher values that come out of this method are always better.
abstract double	getRepMetric() The representative (rep) metric is the metric that was chosen to be the representative for this evaluator.
Iterable<GroundAtom>	getTargets(TrainingMap trainingMap) Get the full collection of target atoms.
abstract boolean	isHigherRepBetter() Is a higher value for the current representative metric better?
void	setCloseTruth(boolean closeTruth)
void	setIncludeObserved(boolean includeObserved)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

includeObserved

protected boolean includeObserved

closeTruth

protected boolean closeTruth

Constructor Detail

Evaluator

protected Evaluator()

Method Detail

getIncludeObserved

public boolean getIncludeObserved()

setIncludeObserved

public void setIncludeObserved(boolean includeObserved)

getCloseTruth

public boolean getCloseTruth()

setCloseTruth

public void setCloseTruth(boolean closeTruth)

compute

public abstract void compute(TrainingMap data)

One of the main computation method. This must be called before any of the metric retrieval methods. Only values in the TrainingMap are computed over.

compute

public abstract void compute(TrainingMap data,
                             StandardPredicate predicate)

One of the main computation method. This must be called before any of the metric retrieval methods. Only values in the TrainingMap matching the given predicate are computed over.

getRepMetric

public abstract double getRepMetric()

The representative (rep) metric is the metric that was chosen to be the representative for this evaluator. This metric is chosen via config.

isHigherRepBetter

public abstract boolean isHigherRepBetter()

Is a higher value for the current representative metric better?

getNormalizedRepMetric

public double getNormalizedRepMetric()

Combine getRepMetric() with isHigherRepBetter() so that higher values that come out of this method are always better.

getNormalizedMaxRepMetric

public double getNormalizedMaxRepMetric()

Combine getBestRepScore() with isHigherRepBetter() so that the values from getNormalizedRepMetric are always less than or equal to this value.

getBestRepScore

public abstract double getBestRepScore()

The maximum (i.e. the best) value that the representative (rep) can take for this evaluator.

getAllStats

public abstract String getAllStats()

Get a string that contains the full range of stats that this Evaluator can provide. compute() should have been called first.

compute

public void compute(Database rvDB,
                    Database truthDB,
                    StandardPredicate predicate,
                    boolean rvDBCached)

A convenience call for those who don't want to create a training map directly. If the random variable database is already fully cached (ie a PAM has already been used on it (like if it has been used in inference)) then don't rebuild the cache.

compute

public void compute(Database rvDB,
                    Database truthDB,
                    StandardPredicate predicate)

getMap

public Iterable<Map.Entry<GroundAtom,GroundAtom>> getMap(TrainingMap trainingMap)

Get the full mapping of target atoms to truth atoms. What constitutes a full mapping depends on includeObserved and closeTruth. Note that certain configurations may result in truth results being returned that include an UnmanagedAtom (atoms that are not managed by an atom manager). This is not an inherently bad or erroneous situation, the caller should just be concious of this.

getTargets

public Iterable<GroundAtom> getTargets(TrainingMap trainingMap)

Get the full collection of target atoms. What constitutes a full collection depends on includeObserved.