Below are 20 justifications for RTE-style problems, provided as sample data for the RTE3 Optional Pilot Task. They have been judged at NIST for each of usability and correctness on a 5 point scale, with larger numbers being better scores. If the usability is too low, correctness is not judged. This is shown as an "X".
Sample 1 Usability: 5 Correctness: 4 Text: North Korea opened relations with Belgium and Spain. Hypothesis: Spain and Korea started ambassadorial relations. Entailed?: Yes Response: Yes Justification: Assuming that "Korea" means "North Korea", "relations" means "diplomatic relations" Because Countries with diplomatic relations exchange ambassadors.
Sample 2 Usability: 5 Correctness: 5 Text: Yoko Ono unveiled a bronze statue of her late husband, John Lennon, to complete the official renaming of England's Liverpool Airport as Liverpool John Lennon Airport. Hypothesis: Yoko Ono is a widow. Entailed?: Unkown Response: Unknown Justification: Yes: Assuming that a widow is a woman whose husband has died and who has not married again And that Yoko Ono has not remarried. No: Assuming that a widow is a woman whose husband has died and who has not married again And that Yoko Ono has remarried.
Sample 3 Usability: 4 Correctness: 5 Text: "Mars is considered the red planet because of its bright red color. " Hypothesis: "Mars is called 'the red planet'. Entailed?: Yes Response: Yes Justification: Yes! I know that another way of saying "Y is considered X" is "Y is called X". In this case, X = the planet, and Y = the Mars. Thus here: If Mars is considered the planet, then Mars is called the planet. Hence: Mars is called 'the red planet'.
Sample 4 Usability: 4 Correctness: 5 Text: "The magazine said the number of billionaires worldwide increased by 102 people in 2006 to 793, a record number, largely due to bullish global stock markets. Their total net worth jumped 18 percent to $2.6 trillion." Hypothesis: "The number of billionaires increases." Entailed?: Yes Response: Yes Justification: Yes! I know that another way of saying "Y is increased by X" is "Y increases by X" In this case, X = the number, and Y = 102 people. Thus here: If the number is increased by 102 people, then the number increases by 102 people. Hence: The number of billionaires increases.
Sample 5 Usability: 5 Correctness: 4 Text: "El-Nashar was detained July 14 in Cairo after Britain notified Egyptian authorities that it suspected he may have had links to some of the attackers." Hypothesis: "El-Nashar was arrested in Egypt." Entailed?: Yes Response: Yes Justification: Yes! I know that another way of saying "X is detained in Y" is "X is arrested in Y". In this case, X = El-Nashar and Y = Egypt. Thus here: If El-Nashar is detained in Egypt then El-Nashar is arrested in Egypt. Also "Cairo" and "Egypt" seem like they might to match. Hence: El-Nashar was arrested in Egypt.
Sample 6 Usability: 3 Correctness: 1 Text: "In 1969, he drew up the report proposing the expulsion from the party of the Manifesto group. In 1984, after Berlinguer's death, Natta was elected as party secretary." Hypothesis: "Berlinguer succeeded Natta." Entailed? No Response: Yes Justification: Yes! I know that another way of saying "X is elected as Y" is "Y succeeds X". In this case, Y = Berlinguer and X = Natta. Thus here: If Natta is elected as Berlinguer, then Berlingua succeeds Natta. Also "Berlingua" and "secretary" seem like they might match, as Berlingua is a secretary. Hence: Berlinguer succeeded Natta.
Sample 7 Usability: 4 Correctness: 2 Text: "So far Sony BMG has not released a list of how many of its CDs are protected or how many have been sold." Hypothesis: "Sony BMG sells protected CDs." Entailed?: Yes Response: Yes Justification: Yes! I know that another way of saying "Y releases X" is "Y sells X". In this case, X = a list and Y = Sony Thus here: If Sony releases a list, then Sony sells a list. Also "list" and "CDs" seem like they might match. Hence: Sony BMG sells protected CDs.
Sample 8 Usability: 4 Correctness: 2 Text: "U.S. District Judge Leonie Brinkema accepted would-be hijacker Zacarias Moussaoui's guilty pleas on six charges of conspiracy laid against him for his plotting to participate in the Sept. 11 attacks. Moussaoui, who had to be restrained at one point during the trial in an Alexandria, Virginia courthouse, was polite and quiet during the day's appearance." Hypothesis: "Moussaoui participated in the Sept. 11 attacks." Entailed?: No Response: Yes Justification: Yes! The text says a hijacker participates in the Sept. 11 attacks. "hijacker" and "Moussaoui" seem like they might match. Hence: Moussaoui participated in the Sept. 11 attacks.
Sample 9 Usability: 5 Correctness: 5 Text: "Traditionally, the Brahui of the Raisani tribe are in charge of the law and order situation through the Pass area. This tribe is still living in present day Balochistan in Pakistan." Hypothesis: "The Raisani tribe resides in Pakistan." Entailed? Yes Response: Yes Justification: Yes! Because: -- 1. "Pakistan" matches "Pakistan" -- 2. "live" matches "reside" -- 3. "tribe" matches "tribe" -- 4. "Raisani" matches "Raisani" Hence: The Raisani tribe resides in Pakistan.
Sample 10 Usability: 1 Correctness: X Text: "Edison decided to call "his" invention the Kinetoscope, combining the Greek root words "kineto" (movement), and "scopos" ("to view")." Hypothesis: "Edison invented the Kinetoscope." Entailed?: Yes Response: Yes Justification: Yes! Because: 1. FORCE-PAIR-MATCH: The matching of the text item _Call75037 to the query item _Kinetoscope75047 was successfully explained by the FORCE-PAIR-MATCH match operator. Forced match of text variable _Call75037 to query variable _Kinetoscope75047 as there were no other options left. Semantic distance in WordNet is 8 links apart. Best WordNet path from call_n1 to Kinetoscope_n1 found: 1. |call_n1| 2. |telephone_n2| 3. |telecom_n1| 4. |medium_n1| 5. |means_n2| 6. |instrumentation_n1| 7. |device_n1| 8. |Kinetoscope_n1| 2. LIN-RULE: The Lin rule N:obj:VV:subj:N -> N:obj:V V:subj:N was applied to transform the text input triples. BINDINGS: -- 1. |_Call75037| to |_Kinetoscope75047| -- 2. _W75218 to |_Invent75045| -- 3. |_Edison75044| to |_Edison75046| PLAUSIBILITY: 0.48000002 Hence: Edison invented the Kinetoscope.
Sample 11 Usability: 3 Correctness: 2 Text: US Steel could even have a technical advantage over Nucor since one new method of steel making it is considering, thin strip casting, may produce higher quality steel than the Nucor thin slab technique. Hypothesis: US Steel may invest in strip casting. Entailed?: Yes Response: Yes Justification: There is a reasonably good match between the hypothesis and the text Names (persons, organizations, locations) in the hypothesis are found in the text. Modal.weakYes : possible -> possible is an allowed inference pattern
Sample 12 Usability: 4 Correctness: 5 Text: Maximo Rene Mendez in an apparent smuggling scheme died after doctors disconnected life-support equipment. Hypothesis: A cocaine-imbibing Miami man had life support removed. Entailed?: Unknown Response: Unknown Justification: There is a poor match between the hypothesis and the text Adjunct.addPosCxt : the hypothesis added the modifier word "Miami", which isn't present in the hypothesis (suggesting non-entailment) Names (persons, organizations, locations) in the hypothesis were NOT found in the text: Miami
Sample 13 Usability: 4 Correctness: 3 Text: Thousands of Romanians participated in rallies on Monday to campaign for the release of three Romanian journalists kidnapped last month in Iraq. Crowds swelled in size after a weekend of smaller rallies sponsored by Romanian media workers and the Arab community in Romania. Hypothesis: Romania sent 730 troops in Iraq. Entailed?: Unknown Response: No Justification: There is a poor match between the hypothesis and the text Numeric.mismatch : NUMBER mismatch: '730.0' does not match '3.0'
Sample 14 Usability: 3 Correctness: 4 Text: The sale was made to pay Yukos' US$ 27.5 billion tax bill, Yuganskneftegaz was originally sold for US$ 9.4 billion to a little known company Baikalfinansgroup which was later bought by the Russian state-owned oil company Rosneft. Hypothesis: Baikalfinansgroup was sold to Rosneft. Entailed?: Yes Response: Yes Justification: We can assume that 'an organization, a little known company Baikalfinansgroup' mentioned in the text refers to an organization. We can also infer that a little known company Baikalfinansgroup is Baikalfinansgroup. From the text we learn that tax bill, Yuganskneftegaz sold to a little known company Baikalfinansgroup which was bought owned. On that basis we can assume that Baikalfinansgroup was sold to Rosneft. We can conclude that Baikalfinansgroup was sold to Rosneft.
Sample 15 Usability: 5 Correctness: 5 Text: He also referred to the "illegal" arrest on 31 May of Mexican Professor Maria Eugenia Ochoa Garcia, whom the Salvadoran government accused of having connections with the Salvadoran guerrillas. Hypothesis: Maria Eugenia Ochoa Garcia was arrested in May. Entailed?: Yes Response: Yes Justification: The text mentions 'the " illegal " arrest'. We can infer that Maria Eugenia Ochoa Garcia was arrested. We can assume that dates 'a date, 31 May' and 'May' specify the same time. We conclude that Maria Eugenia Ochoa Garcia was arrested in May.
Sample 16 Usability: 5 Correctness: 5 Text: They named themselves for Saint Joan of Arc. The brigade began with 17 women, but soon grew to 135 members. Its mission was to obtain money, weapons, provisions, and information for the combatant men. Many smuggled weapons into the combat zones by carrying them in carts filled with grain or cement. Hypothesis: Saint Joan of Arc's brigade got weapons for combatants. Entailed: Yes Response: Yes Justification: The text mentions 'weapons, provisions, and information for the combatant men'. We can assume that weapons, provisions, and information for the combatant men are weapons. We can assume that 'combatant men' and 'combatants' are the same. We can conclude that Saint Joan of Arc's brigade got weapons for combatants.
Sample 17 Usability: 5 Correctness: 5 Text: As leaders gather in Argentina ahead of this weekends regional talks, Hugo Chavez, Venezuela's populist president, is using an energy windfall to win friends and promote his vision of 21st-century socialism. Hypothesis: Chavez is a follower of socialism. Entailed?: Yes Response: Yes Justification: In the text 'a human, Hugo Chavez' is mentioned. We can assume that Hugo Chavez is Chavez. In the text 'a date, 21st-century socialism' is mentioned. We can infer that 21st-century socialism refers to socialism. We conclude that Chavez is a follower of socialism.
Sample 18 Usability: 5 Correctness: 5 Text: Italian film-maker, Fellini was awarded an honorary Oscar for lifetime achievement. He died on October 31, 1993. Hypothesis: An Italian director is awarded an honorary Oscar. Entailed?: Yes Response: Yes Justification: It follows immediately from the first sentence in the text that the following conjunction holds: Fellini was Italian, was a film-maker, and was awarded an honorary Oscar. Since from any conjunction each of the elements conjoined can be inferred, it can be inferred in this case that Fellini was a film-maker. Since if x is a film-maker, x is a director, it follows that Fellini was a director. Assembling a new conjunction, we can deduce that Fellini was Italian, was a director, and was awarded an honorary Oscar. Generalizing, we can from this deduce that an Italian director was awarded an honorary Oscar.
Sample 19 Usability: 5 Correctness: 5 Text: Hughes loved his wife, Gracia, and was absolutely obsessed with his little daughter Elicia. Hypothesis: Hughes was married to Gracia. Entailed?: Yes Response: Yes Justification: We can deduce from the premise that Hughes loved his wife, Gracia. (This is just one of the two conjuncts in the premise, after all.) We know that if x loved y, where y is the wife of x, then y was the wife of x. We can therefore deduce that Gracia was the wife of Hughes. We also know that if x was the wife of y, then y was married to x. Thus we can deduce that Hughes was married to Gracia.
Sample 20 Usability: 5 Correctness: 1 Text: "Hughes loved his wife, Gracia, and was absolutely obsessed with his little daughter Elicia." Hypothesis: "Hughes was married to Gracia." Entailed?: Yes Response: Yes Justification: Yes! I know that another way of saying "X loves Y" is "X is married to Y". In this case, X = Hughes and Y = Gracia. Thus here: If Hughes loves Gracia then Hughes is married to Gracia. Hence: Hughes was married to Gracia.
Return to RTE3 Optional Pilot Task page