This is a system for MT adequacy evaluation based on an RTE (Recognition of Textual Entailment) engine. It uses the NIST MetricsMATR 2008 XML input and output specifications (see the MetricsMATR evaluation plan).
S. Pado, M. Galley, D. Jurafsky, and C. Manning. 2009. Textual Entailment Features for Machine Translation Evaluation. Proceedings of WMT 2009.Note: the current download is essentially the Stanford submission to MetricsMATR 2008.
S. Pado, M. Galley, D. Jurafsky, and C. Manning. 2009. Robust Machine Translation Evaluation with Entailment Features. Proceedings of ACL 2009.
This system is not the full Stanford RTE system (which isn't currently distributed). It is not set up to work as an RTE system. It is only available as java archives (jar files), not as source code.
System requirements: The system requires Java 1.6. and 8 GB of memory.
For documentation, take a look at the included README and the paper.
We have 3 mailing lists for the MT evaluation system, all of which are shared
with other JavaNLP tools (with the exclusion of the parser). Each address is
java-nlp-user This is the best list to post to in order
to ask questions, make announcements, or for discussion among JavaNLP
users. You have to subscribe to be able to use it.
Join the list
(via a webpage) or by emailing
email@example.com. (Leave the
subject and message body empty.) You can also
the list archives.
java-nlp-announce This list will be used only to announce
new versions of Stanford JavaNLP tools. So it will be very low volume (expect 1-3
message a year). Join the list by emailing
firstname.lastname@example.org. (Leave the
subject and message body empty.)
java-nlp-support This list goes only to the software
maintainers. It's a good address for licensing questions, etc. For
general use and support questions, you're better off joining and using
You cannot join
java-nlp-support, but you can mail questions to
The system download is a 700 MB gzipped tar file (mainly consisting of included linguistic resources). When you unpack the tar file, you should have everything needed.