edu.stanford.nlp.stats
Class EquivalenceClassEval<IN,OUT>

java.lang.Object
  extended by edu.stanford.nlp.stats.EquivalenceClassEval<IN,OUT>

public class EquivalenceClassEval<IN,OUT>
extends java.lang.Object

A class for calculating precision and recall statistics based on comparisons between two Collections. Allows flexible specification of:

Note that for set-based evaluation, sets are always constructed using object equality, NOT equality on the basis of an EquivalenceClassEval.EqualityChecker if one is given. If set-based evaluation were conducted on the basis of an EqualityChecker, then there would be indeterminacy when it did not subsume the EquivalenceClasser, if one was given. For example, if objects of the form X:y were equivalence-classed by the left criterion and evaluated for equality on the right, then set-based evaluation based on the equality checker would be indeterminate for a collection of {A:a,B:a} because it would be unclear whether to use the first or second element of the collection.

Author:
Roger Levy, Sarah Spikes (sdspikes@cs.stanford.edu) Attempt at templatization... this may be a failure.

Nested Class Summary
static interface EquivalenceClassEval.EqualityChecker<T>
          A strategy-type interface for specifying an equality criterion for pairs of Objects.
static interface EquivalenceClassEval.Factory<IN,OUT>
           
 
Field Summary
protected  boolean bagEval
           
static EquivalenceClassEval.EqualityChecker DEFAULT_CHECKER
          A default equality checker that uses Object.equals(java.lang.Object) to determine equality.
static EquivalenceClasser NULL_EQUIVALENCE_CLASSER
          Maps all objects to the equivalence class null
 
Constructor Summary
EquivalenceClassEval()
          Specifies a default EquivalenceClassEval, using Object.equals(java.lang.Object) as equality criterion and grouping all items into the "null" equivalence class for reporting purposes
EquivalenceClassEval(EquivalenceClasser<IN,OUT> eq)
          Specifies an EquivalenceClassEval using Object.equals(java.lang.Object) as equality criterion and grouping all items according to the EquivalenceClasser argument.
EquivalenceClassEval(EquivalenceClasser<IN,OUT> eq, EquivalenceClassEval.EqualityChecker<IN> e)
          Specifies an EquivalenceClassEval using the Eval.EqualityChecker argument as equality criterion and grouping all items according to the EquivalenceClasser argument.
EquivalenceClassEval(EquivalenceClasser<IN,OUT> eq, EquivalenceClassEval.EqualityChecker<IN> e, java.lang.String summaryName)
          Specifies an EquivalenceClassEval using the Eval.EqualityChecker argument as equality criterion and grouping all items according to the EquivalenceClasser argument.
EquivalenceClassEval(EquivalenceClasser<IN,OUT> eq, java.lang.String name)
          Specifies an EquivalenceClassEval using Object.equals(java.lang.Object) as equality criterion and grouping all items according to the EquivalenceClasser argument.
EquivalenceClassEval(EquivalenceClassEval.EqualityChecker<IN> e)
          Specifies an EquivalenceClassEval using the Eval.EqualityChecker argument as equality criterion and grouping all items into a single equivalence class for reporting statistics.
 
Method Summary
static
<T> EquivalenceClassEval.EqualityChecker<T>
defaultChecker()
           
 void display()
          Displays the cumulative results of the evaluation to System.out.
 void display(java.io.PrintWriter pw)
          Displays the cumulative results of the evaluation.
 void displayLast()
          Displays the results of the previous Collection pair evaluation to System.out.
 void displayLast(java.io.PrintWriter pw)
          Displays the results of the previous Collection pair evaluation.
 void eval(java.util.Collection<IN> guesses, java.util.Collection<IN> golds)
          Adds a round of evaluation between guesses and golds Collections to the tabulated statistics of the evaluation.
 void eval(java.util.Collection<IN> guesses, java.util.Collection<IN> golds, java.io.PrintWriter pw)
           
static
<E> Counter<E>
f1(Counter<E> precision, Counter<E> recall)
           
static double f1(double precision, double recall)
           
 double f1(OUT key)
           
 EquivalenceClassEval.Factory<IN,OUT> factory()
          returns a new EquivalenceClassEval.Factory instance that vends new EquivalenceClassEval instances with settings like this
 ClassicCounter<OUT> lastF1()
           
 double lastF1(OUT key)
           
 ClassicCounter<OUT> lastNumGolds()
           
 double lastNumGolds(OUT key)
           
 ClassicCounter<OUT> lastNumGoldsCorrect()
           
 ClassicCounter<OUT> lastNumGuessed()
           
 double lastNumGuessed(OUT key)
           
 ClassicCounter<OUT> lastNumGuessedCorrect()
           
 ClassicCounter<OUT> lastPrecision()
           
 double lastPrecision(OUT key)
           
 ClassicCounter<OUT> lastRecall()
           
 double lastRecall(OUT key)
           
static void main(java.lang.String[] args)
           
static
<T,U> EquivalenceClasser<T,U>
nullEquivalenceClasser()
           
 double precision(OUT key)
           
 double recall(OUT key)
           
protected static
<T> void
removeItem(T o, java.util.Collection<T> c, edu.stanford.nlp.stats.EquivalenceClassEval.Eval.CollectionContainsChecker<T> checker)
           
 void setBagEval(boolean bagEval)
          If bagEval is set to true, then multiple instances of the same item will not be merged.
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

bagEval

protected boolean bagEval

NULL_EQUIVALENCE_CLASSER

public static final EquivalenceClasser NULL_EQUIVALENCE_CLASSER
Maps all objects to the equivalence class null


DEFAULT_CHECKER

public static final EquivalenceClassEval.EqualityChecker DEFAULT_CHECKER
A default equality checker that uses Object.equals(java.lang.Object) to determine equality.

Constructor Detail

EquivalenceClassEval

public EquivalenceClassEval()
Specifies a default EquivalenceClassEval, using Object.equals(java.lang.Object) as equality criterion and grouping all items into the "null" equivalence class for reporting purposes


EquivalenceClassEval

public EquivalenceClassEval(EquivalenceClasser<IN,OUT> eq)
Specifies an EquivalenceClassEval using Object.equals(java.lang.Object) as equality criterion and grouping all items according to the EquivalenceClasser argument.


EquivalenceClassEval

public EquivalenceClassEval(EquivalenceClassEval.EqualityChecker<IN> e)
Specifies an EquivalenceClassEval using the Eval.EqualityChecker argument as equality criterion and grouping all items into a single equivalence class for reporting statistics.


EquivalenceClassEval

public EquivalenceClassEval(EquivalenceClasser<IN,OUT> eq,
                            java.lang.String name)
Specifies an EquivalenceClassEval using Object.equals(java.lang.Object) as equality criterion and grouping all items according to the EquivalenceClasser argument.


EquivalenceClassEval

public EquivalenceClassEval(EquivalenceClasser<IN,OUT> eq,
                            EquivalenceClassEval.EqualityChecker<IN> e)
Specifies an EquivalenceClassEval using the Eval.EqualityChecker argument as equality criterion and grouping all items according to the EquivalenceClasser argument.


EquivalenceClassEval

public EquivalenceClassEval(EquivalenceClasser<IN,OUT> eq,
                            EquivalenceClassEval.EqualityChecker<IN> e,
                            java.lang.String summaryName)
Specifies an EquivalenceClassEval using the Eval.EqualityChecker argument as equality criterion and grouping all items according to the EquivalenceClasser argument.

Method Detail

setBagEval

public void setBagEval(boolean bagEval)
If bagEval is set to true, then multiple instances of the same item will not be merged. For example, gold (a,a,b) against guess (a,b) will be scored as 100% precision and 66.67% recall. It is false by default.


nullEquivalenceClasser

public static final <T,U> EquivalenceClasser<T,U> nullEquivalenceClasser()

eval

public void eval(java.util.Collection<IN> guesses,
                 java.util.Collection<IN> golds)
Adds a round of evaluation between guesses and golds Collections to the tabulated statistics of the evaluation.


eval

public void eval(java.util.Collection<IN> guesses,
                 java.util.Collection<IN> golds,
                 java.io.PrintWriter pw)
Parameters:
guesses - Collection of guessed objects
golds - Collection of gold-standard objects
pw - PrintWriter to print eval stats

removeItem

protected static <T> void removeItem(T o,
                                     java.util.Collection<T> c,
                                     edu.stanford.nlp.stats.EquivalenceClassEval.Eval.CollectionContainsChecker<T> checker)

display

public void display()
Displays the cumulative results of the evaluation to System.out.


display

public void display(java.io.PrintWriter pw)
Displays the cumulative results of the evaluation.


displayLast

public void displayLast()
Displays the results of the previous Collection pair evaluation to System.out.


displayLast

public void displayLast(java.io.PrintWriter pw)
Displays the results of the previous Collection pair evaluation.


precision

public double precision(OUT key)

recall

public double recall(OUT key)

lastPrecision

public double lastPrecision(OUT key)

lastPrecision

public ClassicCounter<OUT> lastPrecision()

lastRecall

public double lastRecall(OUT key)

lastRecall

public ClassicCounter<OUT> lastRecall()

lastNumGuessed

public double lastNumGuessed(OUT key)

lastNumGuessed

public ClassicCounter<OUT> lastNumGuessed()

lastNumGuessedCorrect

public ClassicCounter<OUT> lastNumGuessedCorrect()

lastNumGolds

public double lastNumGolds(OUT key)

lastNumGolds

public ClassicCounter<OUT> lastNumGolds()

lastNumGoldsCorrect

public ClassicCounter<OUT> lastNumGoldsCorrect()

f1

public double f1(OUT key)

lastF1

public double lastF1(OUT key)

lastF1

public ClassicCounter<OUT> lastF1()

f1

public static double f1(double precision,
                        double recall)

f1

public static <E> Counter<E> f1(Counter<E> precision,
                                Counter<E> recall)

main

public static void main(java.lang.String[] args)

defaultChecker

public static final <T> EquivalenceClassEval.EqualityChecker<T> defaultChecker()

factory

public EquivalenceClassEval.Factory<IN,OUT> factory()
returns a new EquivalenceClassEval.Factory instance that vends new EquivalenceClassEval instances with settings like this



Stanford NLP Group