Subject: Re: KB evaluation Date: Mon, 17 Jan 2005 13:54:57 -0800 From: Jerry Hobbs Organization: USC/ISI To: Jean-Michel Pomarede Jean-Mi, Here, several months late, is a write-up of the ideas I was discussing with you on the way to the airport in Tampa about the NL inference evaluation problem. The issue I was concerned about in PASCAL, and perhaps even more in PARC's elaboration of it, was that by simply drawing inferences unconstrained out of our heads, we would tend to bias the evaluation specifically to the kinds of inferences each site's systems could handle. To get around this, or at least to add one kind of constraint, my proposal was this: Every word or morpheme in a text presents four kinds of problems: 1. It refers to something -- an entity, relation, event, etc. What does it refer to? 2. It conveys a predication, that is, a predicate applied to arguments. What are the arguments of the predication and how are the predicate and its arguments congruent? How do they represent an appropriate combination? 3. Often the predication as stated is more general than the predication it is intended to convey. What is the more specific predication that is conveyed? 4. Often the predication stands in an implicit causal/contrastive/etc. relation to other predications conveyed by the text. What are these implicit relations? The method I proposed to constrain the questions asked was to pick a text and to ask a question associated with or triggered by every Nth word or morpheme, for some N. I've illustrated this below. Here I've added symbols for the information conveyed by the adjacency of sentences, the coherence relations, R1, R2, ..., although they don't figure in the example below. I've picked N = 10; that is, for every 10th word (approximately), I've tried to formulate a question related to what that word conveys, of one of the four types listed above. These are true-false questions, so I flipped a coin 10 times, and got TFTTFTFTFF. So those are the answers. The word that triggered the entailment/inference is bracketed in *'s and the entailment/inference is indented with >. The text is a Reuters news article from last October. Anyway, this will give you some sense of the idea. I'm not sure how successful it would be, but there it is for discussion. -- Jerry -------------------------------------------------------------------- TEXT: With a threatened boycott of its historic presidential election firmly relegated to the background, Afghanistan took first steps on Tuesday to begin counting votes. R1 During the day, the Afghan-U.N. Joint Election Management Body will decide which ballots are suspicious and how to deal with them. R2 The actual count will begin on Wednesday morning, election officials said. R3 President Hamid Karzai's chief rival, Yunus Qanuni, said on Monday that he and several other candidates had decided to withdraw the call to boycott Saturday's landmark election that was issued after suspicions emerged of illegal multiple voting. R4 "We want unity in this election, not a boycott," he told reporters. R5 "The people want it and we appreciate their feelings." R6 For the Tajik commander, a hero of the resistance to Soviet occupation and the hardline Taliban regime, to talk of popular sentiment shows how much Afghanistan has changed in recent years. --------------------------------------------------------------------- TEXT FRAGMENTS with ENTAILMENTS: 1. With a threatened boycott of its historic presidential election firmly *relegated* to the background, Afghanistan took first steps on Tuesday to begin counting votes. > A boycott of Afghanistan's presidential election is not likely. 2. With a threatened boycott its historic presidential election firmly relegated to the background, Afghanistan took first steps on Tuesday *to* begin counting votes. > Afghanistan has not yet begun to count the votes. 3. During the day, the Afghan-*U.N.* Joint Election Management Body will decide which ballots are suspicious and how to deal with them. > The United Nations is working with the Afghanis to manage the > election. 4. During the day, the Afghan-U.N. Joint Election Management Body will decide which ballots are *suspicious* and how to deal with them. > Some ballots may be invalid. 5. During the day, the Afghan-U.N. Joint Election Management Body will decide which ballots are suspicious and how to deal with them. The actual *count* will begin on Wednesday morning, election officials said. > Election officials will be counted beginning on Wednesday. 6. With a threatened boycott its historic presidential election firmly relegated to the background, Afghanistan took first steps on Tuesday to begin counting votes. ... *President* Hamid Karzai's chief rival, Yunus Qanuni, said on Monday that he and several other candidates had decided to withdraw the call to boycott Saturday's landmark election that was issued after suspicions emerged of illegal multiple voting. > Hamid Karzai is president of Afghanistan. 7. President Hamid Karzai's chief rival, Yunus Qanuni, said on Monday that *he* and several other candidates had decided to withdraw the call to boycott Saturday's landmark election that was issued after suspicions emerged of illegal multiple voting. > Hamid Karzai has decided to withdraw the call to boycott the > election. 8. President Hamid Karzai's chief rival, Yunus Qanuni, said on Monday that he and several other candidates had decided to withdraw *the* call to boycott Saturday's landmark election that was issued after suspicions emerged of illegal multiple voting. > A call to boycott the election was issued. 9. President Hamid Karzai's chief rival, Yunus Qanuni, said on Monday that he and several other candidates had decided to withdraw the call to boycott Saturday's landmark election that was *issued* after suspicions emerged of illegal multiple voting. > Someone issued an election. 10. "We *want* unity in this election, not a boycott," he told reporters. > He and other candidates want a boycott.