Software > Entailment-based MT Evaluation

Entailment-based MT evaluation

About | Questions | Mailing lists | Download | Release history

About

This is a system for MT adequacy evaluation based on an RTE (Recognition of Textual Entailment) engine. It uses the NIST MetricsMATR 2008 XML input and output specifications (see the MetricsMATR evaluation plan).

S. Pado, M. Galley, D. Jurafsky, and C. Manning. 2009. Textual Entailment Features for Machine Translation Evaluation. Proceedings of WMT 2009.
S. Pado, M. Galley, D. Jurafsky, and C. Manning. 2009. Robust Machine Translation Evaluation with Entailment Features. Proceedings of ACL 2009.

Note: the current download is essentially the Stanford submission to MetricsMATR 2008.
It does not yet include the improvements reported on in the WMT and ACL 2009 papers.
We have a long term aim of making a more current version available, but this is not currently being worked on, and there is no schedule for release.

This system is not the full Stanford RTE system (which isn't currently distributed). It is not set up to work as an RTE system. It is only available as java archives (jar files), not as source code.

System requirements: The system requires Java 1.6. and 8 GB of memory.

Questions

For documentation, take a look at the included README and the paper.

Mailing Lists

We have 3 mailing lists for the MT evaluation system, all of which are shared with other JavaNLP tools (with the exclusion of the parser). Each address is at @lists.stanford.edu:

1. java-nlp-user This is the best list to post to in order to ask questions, make announcements, or for discussion among JavaNLP users. You have to subscribe to be able to use it. Join the list (via a webpage) or by emailing java-nlp-user-join@lists.stanford.edu. (Leave the subject and message body empty.) You can also look at the list archives.
2. java-nlp-announce This list will be used only to announce new versions of Stanford JavaNLP tools. So it will be very low volume (expect 1-3 message a year). Join the list by emailing java-nlp-announce-join@lists.stanford.edu. (Leave the subject and message body empty.)
3. java-nlp-support This list goes only to the software maintainers. It's a good address for licensing questions, etc. For general use and support questions, you're better off joining and using java-nlp-user. You cannot join java-nlp-support, but you can mail questions to java-nlp-support@lists.stanford.edu.

Download

Download the README

Download entailment-based MT evaluation system

The system download is a 700 MB gzipped tar file (mainly consisting of included linguistic resources). When you unpack the tar file, you should have everything needed.

Release History

Version	Date	Description
1.0	2008-10-31	Initial release