edu.stanford.nlp.stats

Class EquivalenceClassEval<IN,OUT>

java.lang.Object

edu.stanford.nlp.stats.EquivalenceClassEval<IN,OUT>

public class EquivalenceClassEval<IN,OUT>
extends java.lang.Object

A class for calculating precision and recall statistics based on comparisons between two Collections. Allows flexible specification of:

• The criterion by which to evaluate whether two Objects are equivalent for purposes of precision and recall calculation (specified by an EquivalenceClassEval.EqualityChecker instance)
• The criterion by which Objects are grouped into equivalence classes for purposes of calculating subclass precision and recall (specified by an EquivalenceClasser instance)
• Evaluation is set-based or bag-based (by default, it is set-based). For example, if a gold collection has {a,a,b} and a guess collection has {a,b}, then recall is 100% in set-based evaluation, but is 66.67% in bag-based evaluation.

Note that for set-based evaluation, sets are always constructed using object equality, NOT equality on the basis of an EquivalenceClassEval.EqualityChecker if one is given. If set-based evaluation were conducted on the basis of an EqualityChecker, then there would be indeterminacy when it did not subsume the EquivalenceClasser, if one was given. For example, if objects of the form X:y were equivalence-classed by the left criterion and evaluated for equality on the right, then set-based evaluation based on the equality checker would be indeterminate for a collection of {A:a,B:a} because it would be unclear whether to use the first or second element of the collection.

Author:

Roger Levy, Sarah Spikes (sdspikes@cs.stanford.edu) Attempt at templatization... this may be a failure.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	EquivalenceClassEval.EqualityChecker<T> A strategy-type interface for specifying an equality criterion for pairs of Objects.
static interface	EquivalenceClassEval.Factory<IN,OUT>

Field Summary

Fields

Modifier and Type	Field and Description
protected boolean	bagEval
static EquivalenceClassEval.EqualityChecker	DEFAULT_CHECKER A default equality checker that uses Object.equals(java.lang.Object) to determine equality.
static EquivalenceClasser	NULL_EQUIVALENCE_CLASSER Maps all objects to the equivalence class null

Constructor Summary

Constructors

Constructor and Description
EquivalenceClassEval() Specifies a default EquivalenceClassEval, using Object.equals(java.lang.Object) as equality criterion and grouping all items into the "null" equivalence class for reporting purposes
EquivalenceClassEval(EquivalenceClasser<IN,OUT> eq) Specifies an EquivalenceClassEval using Object.equals(java.lang.Object) as equality criterion and grouping all items according to the EquivalenceClasser argument.
EquivalenceClassEval(EquivalenceClasser<IN,OUT> eq, EquivalenceClassEval.EqualityChecker<IN> e) Specifies an EquivalenceClassEval using the Eval.EqualityChecker argument as equality criterion and grouping all items according to the EquivalenceClasser argument.
EquivalenceClassEval(EquivalenceClasser<IN,OUT> eq, EquivalenceClassEval.EqualityChecker<IN> e, java.lang.String summaryName) Specifies an EquivalenceClassEval using the Eval.EqualityChecker argument as equality criterion and grouping all items according to the EquivalenceClasser argument.
EquivalenceClassEval(EquivalenceClasser<IN,OUT> eq, java.lang.String name) Specifies an EquivalenceClassEval using Object.equals(java.lang.Object) as equality criterion and grouping all items according to the EquivalenceClasser argument.
EquivalenceClassEval(EquivalenceClassEval.EqualityChecker<IN> e) Specifies an EquivalenceClassEval using the Eval.EqualityChecker argument as equality criterion and grouping all items into a single equivalence class for reporting statistics.

Method Summary

Modifier and Type	Method and Description
static <T> EquivalenceClassEval.EqualityChecker<T>	defaultChecker()
void	display() Displays the cumulative results of the evaluation to System.out.
void	display(java.io.PrintWriter pw) Displays the cumulative results of the evaluation.
void	displayLast() Displays the results of the previous Collection pair evaluation to System.out.
void	displayLast(java.io.PrintWriter pw) Displays the results of the previous Collection pair evaluation.
void	eval(java.util.Collection<IN> guesses, java.util.Collection<IN> golds) Adds a round of evaluation between guesses and golds Collections to the tabulated statistics of the evaluation.
void	eval(java.util.Collection<IN> guesses, java.util.Collection<IN> golds, java.io.PrintWriter pw)
static <E> Counter<E>	f1(Counter<E> precision, Counter<E> recall)
static double	f1(double precision, double recall)
double	f1(OUT key)
EquivalenceClassEval.Factory<IN,OUT>	factory() returns a new EquivalenceClassEval.Factory instance that vends new EquivalenceClassEval instances with settings like this
ClassicCounter<OUT>	lastF1()
double	lastF1(OUT key)
ClassicCounter<OUT>	lastNumGolds()
double	lastNumGolds(OUT key)
ClassicCounter<OUT>	lastNumGoldsCorrect()
ClassicCounter<OUT>	lastNumGuessed()
double	lastNumGuessed(OUT key)
ClassicCounter<OUT>	lastNumGuessedCorrect()
ClassicCounter<OUT>	lastPrecision()
double	lastPrecision(OUT key)
ClassicCounter<OUT>	lastRecall()
double	lastRecall(OUT key)
static void	main(java.lang.String[] args)
static <T,U> EquivalenceClasser<T,U>	nullEquivalenceClasser()
double	precision(OUT key)
double	recall(OUT key)
protected static <T> void	removeItem(T o, java.util.Collection<T> c, edu.stanford.nlp.stats.EquivalenceClassEval.Eval.CollectionContainsChecker<T> checker)
void	setBagEval(boolean bagEval) If bagEval is set to true, then multiple instances of the same item will not be merged.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

bagEval

protected boolean bagEval

NULL_EQUIVALENCE_CLASSER

public static final EquivalenceClasser NULL_EQUIVALENCE_CLASSER

Maps all objects to the equivalence class null

DEFAULT_CHECKER

public static final EquivalenceClassEval.EqualityChecker DEFAULT_CHECKER

A default equality checker that uses Object.equals(java.lang.Object) to determine equality.

Constructor Detail

EquivalenceClassEval

public EquivalenceClassEval()

Specifies a default EquivalenceClassEval, using Object.equals(java.lang.Object) as equality criterion and grouping all items into the "null" equivalence class for reporting purposes

EquivalenceClassEval

public EquivalenceClassEval(EquivalenceClasser<IN,OUT> eq)

Specifies an EquivalenceClassEval using Object.equals(java.lang.Object) as equality criterion and grouping all items according to the EquivalenceClasser argument.

EquivalenceClassEval

public EquivalenceClassEval(EquivalenceClassEval.EqualityChecker<IN> e)

Specifies an EquivalenceClassEval using the Eval.EqualityChecker argument as equality criterion and grouping all items into a single equivalence class for reporting statistics.

EquivalenceClassEval

public EquivalenceClassEval(EquivalenceClasser<IN,OUT> eq,
                            java.lang.String name)

Specifies an EquivalenceClassEval using Object.equals(java.lang.Object) as equality criterion and grouping all items according to the EquivalenceClasser argument.

EquivalenceClassEval

public EquivalenceClassEval(EquivalenceClasser<IN,OUT> eq,
                            EquivalenceClassEval.EqualityChecker<IN> e)

Specifies an EquivalenceClassEval using the Eval.EqualityChecker argument as equality criterion and grouping all items according to the EquivalenceClasser argument.

EquivalenceClassEval

public EquivalenceClassEval(EquivalenceClasser<IN,OUT> eq,
                            EquivalenceClassEval.EqualityChecker<IN> e,
                            java.lang.String summaryName)

Specifies an EquivalenceClassEval using the Eval.EqualityChecker argument as equality criterion and grouping all items according to the EquivalenceClasser argument.

Method Detail

setBagEval

public void setBagEval(boolean bagEval)

If bagEval is set to true, then multiple instances of the same item will not be merged. For example, gold (a,a,b) against guess (a,b) will be scored as 100% precision and 66.67% recall. It is false by default.

nullEquivalenceClasser

public static final <T,U> EquivalenceClasser<T,U> nullEquivalenceClasser()

eval

public void eval(java.util.Collection<IN> guesses,
                 java.util.Collection<IN> golds)

Adds a round of evaluation between guesses and golds Collections to the tabulated statistics of the evaluation.

eval

public void eval(java.util.Collection<IN> guesses,
                 java.util.Collection<IN> golds,
                 java.io.PrintWriter pw)

Parameters:

guesses - Collection of guessed objects

golds - Collection of gold-standard objects

pw - PrintWriter to print eval stats

removeItem

protected static <T> void removeItem(T o,
                                     java.util.Collection<T> c,
                                     edu.stanford.nlp.stats.EquivalenceClassEval.Eval.CollectionContainsChecker<T> checker)

display

public void display()

Displays the cumulative results of the evaluation to System.out.

display

public void display(java.io.PrintWriter pw)

Displays the cumulative results of the evaluation.

displayLast

public void displayLast()

Displays the results of the previous Collection pair evaluation to System.out.

displayLast

public void displayLast(java.io.PrintWriter pw)

Displays the results of the previous Collection pair evaluation.

precision

public double precision(OUT key)

recall

public double recall(OUT key)

lastPrecision

public double lastPrecision(OUT key)

lastPrecision

public ClassicCounter<OUT> lastPrecision()

lastRecall

public double lastRecall(OUT key)

lastRecall

public ClassicCounter<OUT> lastRecall()

lastNumGuessed

public double lastNumGuessed(OUT key)

lastNumGuessed

public ClassicCounter<OUT> lastNumGuessed()

lastNumGuessedCorrect

public ClassicCounter<OUT> lastNumGuessedCorrect()

lastNumGolds

public double lastNumGolds(OUT key)

lastNumGolds

public ClassicCounter<OUT> lastNumGolds()

lastNumGoldsCorrect

public ClassicCounter<OUT> lastNumGoldsCorrect()

f1

public double f1(OUT key)

lastF1

public double lastF1(OUT key)

lastF1

public ClassicCounter<OUT> lastF1()

f1

public static double f1(double precision,
                        double recall)

f1

public static <E> Counter<E> f1(Counter<E> precision,
                                Counter<E> recall)

main

public static void main(java.lang.String[] args)

defaultChecker

public static final <T> EquivalenceClassEval.EqualityChecker<T> defaultChecker()

factory

public EquivalenceClassEval.Factory<IN,OUT> factory()

returns a new EquivalenceClassEval.Factory instance that vends new EquivalenceClassEval instances with settings like this