Robust Textual Inference (PASCAL RTE and AQUAINT KB Eval)

Data matrix | Web tools | Talks and papers | RTE-1 | KB Eval | RTE-2 | RTE-3 | Wiki

Data Files (original and with our analyses)

Web Based Tools

• Alignment
  • Table Annotator
  • Scoring
  • Current Scores

Various slide presentations & discussion notes

• ?/??/05? (Chris): Stanford @ KBEval: a presentation at KBEval meeting (?)
• ?/??/05? (Chris): PASCAL RTE: an introduction aimed at AQUAINT people
• ?/??/05? (Chris): Robust Textual Inference: a talk given at MIT
• ?/??/06? (Chris): Robust Textual Inference: a talk for CS faculty lunch (?)
• ?/??/06? (Chris): Robust Textual Inference: a talk given at Yahoo! Research
• ?/??/06? (Chris): Robust Local Textual Inference: a talk given at U. Chicago
• 1/18/06 (Bill): Ideas for future directions: natural logic and tree edit models
• 1/26/06 (Bill): Intro to natural logic: a talk for NLP lunch
• 1/30/06 (Bill): Robust Textual Inference: a presentation for Ruth Gee's site visit
• 3/2/06 (Bill): 3 slides about RTE: for NLP lunch "teaser" talks
• 3/10/06 (Bill): Computational Models of Textual Inference: for SemFest '06
• 3/21/06 (Dan J): Information containment, consistency, and contradiction: for GALE meeting (loosely related)
• 3/24/06 (Bill): Natural Logic and Textual Inference: for AQUINAS meeting
• RTE 2006 workshop paper
• 4/10/06 (Marie): RTE2 workshop talk: Learning to distinguish valid textual entailments
• 5/1/06 (Marie): RTE 2006 workshop review: 5 best systems
• 5/15/06 (Bill): Lexical resource investigations, including ablation studies and comparisons of score distributions.
• 5/22/06 (Bill): Concept probability models and derived lexical similarity measures: Resnik, Jiang-Conrath, and Lin.
• 6/5/06 (Bill): Learning to recognize features of valid textual entailments
  Bill MacCartney, Trond Grenager, Marie-Catherine de Marneffe, Daniel Cer, Christopher D. Manning
  Presented at NAACL-06

PASCAL RTE-3 (2007) information

• RTE-3 webpage

PASCAL RTE-3 (2007) Stanford system

• Quick start instructions (2007 edition) Feb 2007!
• RTE3dev hand alignment tasks Feb 2007!

PASCAL RTE-2 (2006) information

• RTE-2 webpage

AQUAINT2 KB Eval (2005) information

• AQUAINT KB Eval guide
• KB Eval XSchema
• XSLT to produce HTML
• XML sample file
• XSD Schema file modifying original KB Eval Schema (with extra fields, what we use internally)
• XSD Schema file for storing info about each passage ("docInfo" schema)
• Stylesheet for RTEEngine ("infoFile") Output to HTML
• (Mainly outdated) Feb 2005 discussion meeting notes and sample data
  • Meeting summary
  • Discussion slides
  • PARC examples: txt, pdf with explanatory text
  • ICSI/UTD examples
  • LCC examples: XML, HTML, annotation guide
  • Brandeis
  • Cyc
  • MIT
  • ASU
  • ISI
• KB Eval data reannotation guidelines and log

AQUAINT2 KB Eval (2005) Stanford system and discussion notes

• Stanford RTE architecture diagrams
• Ideas for representing PARC sentences as first order logic
• Example of using Otter to decide a PARC sentence.
• Summary of ideas for recursive text inferer
• A system primer, which was done in Sep 2005, and is now probably too old to be useful
• KBE results: Accuracy/CWS table for learnt feature weights

PASCAL RTE-1 (2005) information

• PASCAL Recognizing Textual Entailment website, which includes the workshop proceedings, links to data sets, etc. The data is also available on this page, below.
• MSR annotations for RTE data (examples answerable based on syntax alone / syntax + thesaurus alone): tarfile (Rajat)

PASCAL RTE-1 (2005) Stanford submitted system and related stuff

• RTE Workshop presentation (ppt)
• RTE Workshop paper
• Rajat Raina et al. AAAI 2005 paper on weighted abductive inference
• Aria Haghighi et al. EMNLP 2005 paper on graph subsumption inference
• Rion Snow's RTE graphs (outdated).
• Rajat's system
  • Rajat dev set one output
  • dev set 1 error analysis
  • Rajat dev set two output
  • Explore the rest of Rajat's stuff on your own: dev set 1, dev set 2
• Aria URLs: I'm not quite sure how these all relate date wise:
  • Bag of dependencies output dev1
  • Simple edge matching dev1
  • Full graph matching dev1
  • Bag of words dev1
  • end of Jan 2005 dev1
  • end of Jan 2005 dev1 error analysis
  • dev2 output
  • dev1 Feb 2005
• Dev set 2 error analysis:
  • analysis: Open Office, HTML
  • explanation: Open Office, HTML

Links to other lexical resources and competitions

• DIRT online demo
• Microsoft Research Paraphrase Corpus
• XWN: Extended WordNet
• VerbNet (2.1), framesets, mapping from VerbNet 2.2 to FrameNet 1.5, and a unified verb index (Martha Palmer et al., Colorado/U. Penn), and SemLink.
• NomBank. (Oct 2005: 80% of data released!)
• Interpretation As Abduction by Jerry Hobbs et al. -- or you can find more from his homepage.
• Answer Validation Exercise at CLEF 2006.
• Surface grammatical relations annotation:
  • John Carroll et al.'s English Parser Evaluation Corpus
  • SUSANNE corpus, etc.
  • PARC 700 Dependency Bank

http://nlp.stanford.edu/projects/infer/ Christopher Manning -- Last modified: Mon Feb 5 11:07:06 PST 2007 [manning]