edu.stanford.nlp.parser.metrics
Class Evalb

java.lang.Object
  extended by edu.stanford.nlp.parser.metrics.AbstractEval
      extended by edu.stanford.nlp.parser.metrics.Evalb
Direct Known Subclasses:
Evalb.CBEval

public class Evalb
extends AbstractEval

A Java re-implementation of the evalb bracket scoring metric (Collins, 1997) that accepts Unicode input. "Collinization" should be performed on input trees prior to invoking the package programmatically. "Collinization" refers to normalization of trees for things not counted in evaluation, such as equivalencing PRT and ADVP, which has standardly been done in English evaluation. A main method is provided that performs Collinization according to language specific settings.

This implementation assumes that the guess/gold input files are of equal length, and have one tree per line.

This implementation was last validated against EVALB20080701 (http://nlp.cs.nyu.edu/evalb/) by Spence Green on 22 Jan. 2010. Notwithstanding this, Sekine and collins' EVALB script has been the common standard for constituency evaluation of parsers for the last decade. We always validate any numbers we report with it, and we suggest that you do the same.

Author:
Dan Klein, Spence Green

Nested Class Summary
static class Evalb.CBEval
           
 
Nested classes/interfaces inherited from class edu.stanford.nlp.parser.metrics.AbstractEval
AbstractEval.CatErrorEval, AbstractEval.RuleErrorEval, AbstractEval.ScoreEval
 
Field Summary
static Map<String,Integer> optionArgDefs
           
 
Fields inherited from class edu.stanford.nlp.parser.metrics.AbstractEval
curF1, num, runningAverages, str
 
Constructor Summary
Evalb(String str, boolean runningAverages)
           
 
Method Summary
 void evaluate(Tree guess, Tree gold, PrintWriter pw)
           
static void main(String[] args)
          Run the Evalb scoring metric on guess/gold input.
protected  Set<Constituent> makeObjects(Tree tree)
          evalb only evaluates phrasal categories, thus constituents() does not return objects for terminals and pre-terminals.
 
Methods inherited from class edu.stanford.nlp.parser.metrics.AbstractEval
display, display, evaluate, evaluate, getEvalbF1, getEvalbF1Percent, getExact, getExactPercent, getLastF1, getNum, getSentAveF1, precision
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

optionArgDefs

public static final Map<String,Integer> optionArgDefs
Constructor Detail

Evalb

public Evalb(String str,
             boolean runningAverages)
Method Detail

makeObjects

protected Set<Constituent> makeObjects(Tree tree)
evalb only evaluates phrasal categories, thus constituents() does not return objects for terminals and pre-terminals.

Specified by:
makeObjects in class AbstractEval

evaluate

public void evaluate(Tree guess,
                     Tree gold,
                     PrintWriter pw)
Overrides:
evaluate in class AbstractEval

main

public static void main(String[] args)
Run the Evalb scoring metric on guess/gold input. The default language is English.

Parameters:
args -


Stanford NLP Group