Applying First-Order Theorem Proving to PARC Data-set

Some of the sentences can be decided directly by FOL:

(1) Q: Some students came to school by car.
    A: Did any students come to school?
       YES, STRICT, LINGUISTIC

EXISTS 1[Student(1) & School(2) & Came(3, 1, 2) & By-Car(3)]

EXISTS 1[Student(1) & School(2) & Came(3, 1, 2)]

(2) Q: No students came to school by car.
    A: Did any students come to school?
       UNKNOWN, STRICT, LINGUISTIC

~ EXISTS 1[Student(1) & School(2) & Came(3, 1, 2) & By-Car(3)]

EXISTS 1[Student(1) & School(2) & Came(3, 1, 2)]

(3) Q: John drove legally.
    A: Did John drive?
       YES, STRICT, LINGUISTIC

EXISTS 1, 2, 3[John(1) & Drive(2, 1) & Legally(3, 2)]

EXISTS 1, 2[John(1) & Drive(2, 1)]

(16) Q: No US congressman has visited Iraq since the war ended.
     A: Has Jones, a US Congressman, visited Iraq after the war ended?
        NO, STRICT, LINGUISTIC

~EXISTS 1, ..., 5 [US-Congressman(1) & Visit(2, 1, 3) & Iraq(3) & Since(2, 4) & War(5) & End(4, 5)]

EXISTS 1, ..., 5 [Jones(1) & US-Congressman(1) & Visit(2, 1, 3) & Iraq(3) & Since(2, 4) & War(5) & End(4, 5)]

(33) Q: All companies have to file annual reports.
     A: Do all Fortune 500 companies have to file annual reports?
        YES, STRICT, LINGUISTIC

ALL 1 EXISTS 2, 3, 4 [Company(1) --> Have-To(1, 2) & File(2, 1, 3) & Reports(3) & Annual(4, 3)]

ALL 1 EXISTS 2, 3, 4 [Company(1) & Fortune-500(1) --> Have-To(1, 2) & File(2, 1, 3) & Reports(4, 3)]

Others require some background knowledge. (9) requires knowing that if you visited someplace, you must have gone there.

(9) Q: The president visited Iraq in September.
    A: Has the president gone to Iraq?
       YES, STRICT, LINGUISTIC

EXISTS 1, 2, 3, 4[President(1) & Iraq(2) & Visit(3, 1, 2) & September(4) & In(3, 4)]

EXISTS 1, 2, 3[President(1) & Iraq(2) & Go(3, 1, 2)]

Assumption: ALL a, b, c[Visit(a, b, c) --> EXISTS d[Go(d, b, c)]]

I'm not sure here whether “last year” modifies “arrived” or “September”, so I assume it modifies “September”, to be consisted with phrases like “in September 1995”. This example requires some understanding of temporal events.

(11) Q: Jones arrived in Paris in September last year.
     A: Did Jones arrive in Paris last year?
        YES, STRICT, LINGUISTIC

EXISTS 1, 2, 3, 4, 5[Jones(1) & Arrive(2, 1, 3) & Paris(3) & In(2, 4) & September(4) & Last-Year(5) & Of(4, 5)]

EXISTS 1, 2, 3[Jones(1) & Arrive(2, 1, 3) & Paris(3) & Last-Year(2)]

Assumptions:

ALL a, b, c, d, e[Event(a) & Month(b) & Year(c) & In(a, b) & Of(b, c) --> In(a, c)]

ALL a, b, c[Arrive(a, b, c) --> Event(a)]

ALL a[September(a) --> Month(a)]

ALL a[Last-Year(a) --> Year(a)]

“After” and “before” are opposites:

(15) Q: The president left after the diplomat arrived.
     A: Did the diplomat arrive before the president leave?
        YES, STRICT, LINGUISTIC

President(1) & Leave(2, 1) & After(2, 3) & Diplomat(4) & Arrive(PARC Sentences 23, 4)

Diplomat(1) & Arrive(2, 1) & Before(2, 3) & President(4) & Leave(3, 4)

Assumption: ALL a, b[After(a, b) --> Before(b, a)]

Frames.

The PARC set also heavily tests frames. Here, we have to somehow represent that if the New York Times reports something, it isn't necessarily true. Using plain FOL, we might translate these sentences as the following, and wrongly conclude TRUE:

(43) Q: The New York Times reported that Hanssen sold fbi secrets to
        the Russians and could face the death penalty. 
     A: Did Hanssen sell fbi secrets to the Russians? 
        UNKNOWN, STRICT, LINGUISTIC

EXISTS 1, ..., 8[New-York-Times(1) & Reported(2, 1, 9) & And(9, 3, 7) & Hanssen(4) & Sell(3, 4, 5, 6) & FBI-Secrets(5) & Russians(6) & Could(7) & Face(7, 4, 8) & Death-Penalty(8)]

EXISTS 1, ..., 4[Hanssen(1) & Sell(2, 1, 3, 4) & FBI-Secrets(3) & Russians(4)]

The problem is that “Sell”, “Could”, and “Face” aren't marked as being in a frame that doesn't imply truth. Maybe we could represent this with something like:

EXISTS 1, ..., 8[New-York-Times(1) & Reported(2, 1, 9) & Reported:And(9, 3, 7) & Hanssen(4) & Reported:Sell(3, 4, 5, 6) & FBI-Secrets(5) & Russians(6) & Reported:Could(7) & Reported:Could:Face(7, 4, 8) & Death-Penalty(8)]

This way, the prover won't automatically conclude “Sell” from “Reported:Sell”. It's somewhat subtle which parts of the arguments to “Reported” aren't necessarily true. For example, in (44), it is correct to conclude that someone sold centrifuges to North Korea, apparently since Bush's saying it was Khan who sold them presupposes that they were sold. The representation should end up as something like the following:

(44) Q: Bush said that it was Khan who sold centrifuges to North Korea.
     A: Were centrifuges sold to North Korea?
        YES, STRICT, LINGUISTIC

EXISTS 1, ..., 8[Bush(1) & Say(2, 1, 3) & Say:Be(3, 4, 5) & Khan(4) & Sell(6, 5, 7, 8) & Centrifuges(7) & North-Korea(8)]

EXISTS 1, ..., 4[Centrifuges(1) & Sell(2, 3, 1, 4) & North-Korea(4)]

What if some sources are credible enough that we could believe everything they say? We could add assumptions like:

ALL a, b, c[Say(a, b, c) & Credible(b) & Say:X(c, ...) --> X]

Or, a weaker version:

ALL a, b, c[Say(a, b, c) & Say:X(c, ...) --> Likely:X]

This next example is even more subtle. Here, we can conclude that Bin Laden was not seen in Tora Bora, since that fact is in the “false that” frame. I don't know what the FOL representation would look like.

(61) Q: It is false that Bin Laden was seen in Tora Bora.
     A: Was Bin Laden seen in Tora Bora?
        NO, STRICT, LINGUISTIC

The tricky part is, in this made-up sentence, the conclusion seems to follow, again because it is presupposed that someone sold the centrifuges.

Q: It is false that it was Khan who sold centrifuges to North Korea.

A: Someone sold centrifuges to North Korea.

YES, STRICT, LINGUISTIC

Here are all of the sentences, divided into categories.

Syntax and Propositional Logic Only

(3) Q: John drove legally.
    A: Did John drive?
       YES, STRICT, LINGUISTIC

(4) Q: John drove predictably.
    A: Did John drive?
       YES, STRICT, LINGUISTIC

(5) Q: Legally, John could drive.
    A: Did John drive?
       UNKNOWN, STRICT, LINGUISTIC
(6) Q: Predictably, John drove.
    A: Did John drive?
       YES, STRICT, LINGUISTIC
(10) Q: Jones has visited Iraq.
     A: Did Jones visit Iraq in September?
        UNKNOWN, STRICT, LINGUISTIC

(29) Q: The room was full of intelligent women.
     A: Was the room full of women?
        YES, STRICT, LINGUISTIC

(30) Q: The room was full of women.
     A: Was the room full of intelligent women?
        UNKNOWN, STRICT, LINGUISTIC
(38) Q: The US troops stayed in Iraq although the war was over.
     A: Was the war over?
        YES, STRICT, LINGUISTIC
(39) Q: Since it was cold, he closed the window.
     A: Was it cold?
        YES, STRICT, LINGUISTIC

(40) Q: John didn't visit us after he returned from Spain.
     A: Did John return from Spain?
        YES, STRICT LINGUISTIC

(41) Q: Hanssen, who sold FBI secrets to the Russians, could face the
        death penalty.
     A: Did Hanssen sell FBI secrets to the Russians? 
        YES, STRICT, LINGUISTIC

Quantifiers

(1) Q: Some students came to school by car.
    A: Did any students come to school?
       YES, STRICT, LINGUISTIC

(2) Q: No students came to school by car.
    A: Did any students come to school?
       UNKNOWN, STRICT, LINGUISTIC
(16) Q: No US congressman has visited Iraq since the war ended.
     A: Has Jones, a US Congressman, visited Iraq after the war ended?
        NO, STRICT, LINGUISTIC

(17) Q: No US congressman has visited Iraq since the war.
     A: Did Jones, a US Congressman, visit Iraq before the war?
        UNKNOWN, STRICT, LINGUISTIC

(18) Q: No US congressman visited Iraq until the war.
     A: Did any US congressman visit Iraq before the war?
        NO, STRICT, LINGUISTIC
(19) Q: Some students arrived at the school on Sunday.
     A: Were there any students at the school on Sunday?
        YES, STRICT, LINGUISTIC

(20) Q: No students arrived at the school on Sunday.
     A: Were there any students at the school on Sunday?
        UNKNOWN, STRICT, LINGUISTIC

(21) Q: There were no students at the school on Sunday.
     A: Did any students arrive at the school on Sunday?
        NO, STRICT, LINGUISTIC
(31) Q: Children are not admitted to the theatre.
     A: Are small children admitted to the theatre?
        NO, STRICT, LINGUISTIC

(32) Q: Small children are not admitted to the theatre.
     A: Are children admitted to the theatre?
        UNKNOWN, STRICT, LINGUISTIC
(33) Q: All companies have to file annual reports.
     A: Do all Fortune 500 companies have to file annual reports?
        YES, STRICT, LINGUISTIC

(34) Q: All Fortune 500 companies have to file annual reports.
     A: Do all companies have to file annual reports?
        UNKNOWN, STRICT, LINGUISTIC

(35) Q: All companies have to file annual reports to the sec.
     A: Do all companies have to file annual reports?
        YES, STRICT, LINGUISTIC

(36) Q: All companies have to file annual reports.
     A: Do all companies have to file annual reports to the sec.
        UNKNOWN, STRICT, LINGUISTIC

(37) Q: No delegates finished the report.
     A: Did any delegate finish the report on time?
        NO, STRICT, LINGUISTIC
(74) Q: Many soldiers were killed in the ambush.
     A: Were all soldiers killed in the ambush?
        NO, PLAUSIBLE, LINGUISTIC

Synonyms/Hypernyms

Verb Tense

(22) Q: The diplomat left Baghdad last week.
     A: Has the diplomat been to Baghdad?
        YES, STRICT, LINGUISTIC


(23) Q: The diplomat will arrive in Baghdad next week.
     A: Has the diplomat been to Baghdad?
        UNKNOWN, STRICT, LINGUISTIC

Negation

(16) Q: No US congressman has visited Iraq since the war ended.
     A: Has Jones, a US Congressman, visited Iraq after the war ended?
        NO, STRICT, LINGUISTIC

(17) Q: No US congressman has visited Iraq since the war.
     A: Did Jones, a US Congressman, visit Iraq before the war?
        UNKNOWN, STRICT, LINGUISTIC

(18) Q: No US congressman visited Iraq until the war.
     A: Did any US congressman visit Iraq before the war?
        NO, STRICT, LINGUISTIC
(20) Q: No students arrived at the school on Sunday.
     A: Were there any students at the school on Sunday?
        UNKNOWN, STRICT, LINGUISTIC

(21) Q: There were no students at the school on Sunday.
     A: Did any students arrive at the school on Sunday?
        NO, STRICT, LINGUISTIC
(61) Q: It is false that Bin Laden was seen in Tora Bora.
     A: Was Bin Laden seen in Tora Bora?
        NO, STRICT, LINGUISTIC

More Lexical Knowledge

(7) Q: The technician cooled the room.
    A: Did the technician lower the temperature of the room?
       YES, STRICT, LINGUISTIC

(8) Q: The technician raised the temperature of the room.
    A: Did the technician cool the room?
       NO, STRICT, LINGUISTIC

(9) Q: The president visited Iraq in September.
    A: Has the president gone to Iraq?
       YES, STRICT, LINGUISTIC
(11) Q: Jones arrived in Paris in September last year.
     A: Did Jones arrive in Paris last year?
        YES, STRICT, LINGUISTIC

(12) Q: Jones arrived in Paris in September last year.
     A: Did Jones arrive in Paris in September?
        UNKNOWN, STRICT, LINGUISTIC

(13) Q: Jones arrived on a Sunday in September.
     A: Did Jones arrive on a Sunday?
        YES, STRICT, LINGUISTIC

(14) Q: Jones arrived on a Sunday in September.
     A: Did Jones arrive in September?
        YES, STRICT, LINGUISTIC
(15) Q: The president left after the diplomat arrived.
     A: Did the diplomat arrive before the president leave?
        YES, STRICT, LINGUISTIC

(16) Q: No US congressman has visited Iraq since the war ended.
     A: Has Jones, a US Congressman, visited Iraq after the war ended?
        NO, STRICT, LINGUISTIC

(17) Q: No US congressman has visited Iraq since the war.
     A: Did Jones, a US Congressman, visit Iraq before the war?
        UNKNOWN, STRICT, LINGUISTIC

(18) Q: No US congressman visited Iraq until the war.
     A: Did any US congressman visit Iraq before the war?
        NO, STRICT, LINGUISTIC
(22) Q: The diplomat left Baghdad last week.
     A: Has the diplomat been to Baghdad?
        YES, STRICT, LINGUISTIC


(23) Q: The diplomat will arrive in Baghdad next week.
     A: Has the diplomat been to Baghdad?
        UNKNOWN, STRICT, LINGUISTIC

(24) Q: The president knows that the diplomat left Baghdad.
     A: Has the diplomat been to Baghdad?
        YES, STRICT, LINGUISTIC

(25) Q: The president hasn't gone to Iraq since the diplomat left Baghdad.
     A: Has the diplomat been to Baghdad?
        YES, STRICT, LINGUISTIC

(26) Q: The president hasn't gone to Iraq since the diplomat left Baghdad.
     A: Has the president been to Iraq?
        UNKNOWN, STRICT, LINGUISTIC
(27) Q: The diplomat didn't manage to leave Baghdad.
     A: Has the diplomat been to Baghdad?
        YES, STRICT, LINGUISTIC

(28) Q: The diplomat hasn't managed to leave Baghdad.
     A: Is the diplomat in Baghdad now?
        YES, STRICT, LINGUISTIC
(60) Q: We met in September during the feast.
     A: Did the feast take place in September?
        YES, STRICT, LINGUISTIC

(64) Q: Tony Hall left Amman on Sunday.
     A: Was Tony Hall in Amman on Sunday?
        YES, STRICT, LINGUISTIC

(65) Q: Tony Hall left Amman on Sunday.
     A: Was Tony Hall in Amman on Saturday?
        UNKNOWN, STRICT, LINGUISTIC

(66) Q: Khan sold 10 centrifuges to North Korea.
     A: Did North Korea buy 10 centrifuges?
        YES, STRICT, LINGUISTIC

Frames

(24) Q: The president knows that the diplomat left Baghdad.
     A: Has the diplomat been to Baghdad?
        YES, STRICT, LINGUISTIC

(25) Q: The president hasn't gone to Iraq since the diplomat left Baghdad.
     A: Has the diplomat been to Baghdad?
        YES, STRICT, LINGUISTIC

(26) Q: The president hasn't gone to Iraq since the diplomat left Baghdad.
     A: Has the president been to Iraq?
        UNKNOWN, STRICT, LINGUISTIC
(27) Q: The diplomat didn't manage to leave Baghdad.
     A: Has the diplomat been to Baghdad?
        YES, STRICT, LINGUISTIC

(28) Q: The diplomat hasn't managed to leave Baghdad.
     A: Is the diplomat in Baghdad now?
        YES, STRICT, LINGUISTIC
	(42) Q: The New York Times reported that Hanssen, who sold fbi secrets to the Russians, could face the death penalty. 
     A: Did Hanssen sell fbi secrets to the Russians?
        YES, STRICT, LINGUISTIC

(43) Q: The New York Times reported that Hanssen sold fbi secrets to
        the Russians and could face the death penalty. 
     A: Did Hanssen sell fbi secrets to the Russians? 
        UNKNOWN, STRICT, LINGUISTIC

(44) Q: Bush said that it was Khan who sold centrifuges to North Korea.
     A: Were centrifuges sold to North Korea?
        YES, STRICT, LINGUISTIC

(45) Q: Bush said that Khan sold centrifuges to North Korea.
     A: Were centrifuges sold to North Korea?
        UNKNOWN, STRICT, LINGUISTIC
(46) Q: What we found in Iraq was rusted shrapnel.
     A: Did we find anything in Iraq?
        YES, STRICT, LINGUISTIC

(47) Q: The fact that Bin Laden was in Tora Bora lead to the suspicion
        that the Afghan campaign was mismanaged.
     A: Was Bin Laden in To(61) Q: It is false that Bin Laden was seen in Tora Bora.
     A: Was Bin Laden seen in Tora Bora?
        NO, STRICT, LINGUISTIC
ra Bora?
        YES, STRICT, LINGUISTIC

(48) Q: The fact that Bin Laden was in Tora Bora lead to the suspicion
        that the Afghan campaign was mismanaged. 
     A: Was the Afghan campaign mismanaged?
        UNKNOWN, STRICT, LINGUISTIC

(49) Q: The paper concluded that the election had been rigged.
     A: Was the election rigged?
        UNKNOWN, STRICT, LINGUISTIC

(50) Q: Ames was, as the press reported, a successful spy.
     A: Was Ames a successful spy?
        YES, STRICT, LINGUISTIC

(51) Q: The press reported that Ames was a successful spy.
     A: Was Ames a successful spy?
        UNKNOWN, STRICT, LINGUISTIC

(52) Q: The US forgot that the Afghans speak several different languages.
     A: Do the Afghans speak several different languages?
        YES, STRICT, LINGUISTIC

(53) Q: Bush realized that the US Army had to be transformed to meet new
        threats.
     A:  Did the US Army have to be transformed to meet new threats?
         YES, STRICT, LINGUISTIC

(54) Q: Bush didn't realize that Afghanistan is land-locked.
     A: Is Afghanistan land-locked?
        YES, STRICT, LINGUISTIC

(55) Q: There is a belief that the US will invade Syria.
     A: Will the US invade Syria?
        UNKNOWN, STRICT, LINGUISTIC

(56) Q: It is not surprising that Bush has the lead in Ohio.
     A: Does Bush have the lead in Ohio?
        YES, STRICT, LINGUISTIC

(57) Q: It is not likely that Bush has the lead in Ohio.
     A: Does Bush have the lead in Ohio?
        UNKNOWN, STRICT, LINGUISTIC

(61) Q: It is false that Bin Laden was seen in Tora Bora.
     A: Was Bin Laden seen in Tora Bora?
        NO, STRICT, LINGUISTIC
(62) Q: It follows that Bin Laden was in Tora Bora.
     A: Was Bin Laden in Tora Bora?
        YES, STRICT, LINGUISTIC

(63) Q: It is likely that Bin Laden was in Tora Bora.
     A: Was Bin Laden in Tora Bora?
        UNKNOWN, STRICT, LINGUISTIC
(67) Q: The US invasion of Afghanistan prevented Al-Qaida from attacking Ryad
        in 2002.
     A: Did Al-Qaida attack Ryad in 2002?
        NO, STRICT, LINGUISTIC

(68) Q: The administration managed to track down the perpetrators.
     A: Did the administration track down the perpetrators?
        YES, STRICT, LINGUISTIC

(69) Q: The administration didn't manage to track down the perpetrators.
     A: Did the administration track down the perpetrators?
        NO, STRICT, LINGUISTIC

(70) Q: Bush didn't have the time to read the report.
     A: Did Bush read the report?
        NO, STRICT, LINGUISTIC

(71) Q: Bush had the time to read the report.
     A: Did Bush read the report?
        YES, PLAUSIBLE, LINGUISTIC

(72) Q: The president wasn't able to attend the meeting.
     A: Did the president attend the meeting?
        NO, STRICT, LINGUISTIC

(73) Q: The president was able to attend to meeting.
     A: Did the president attend the meeting?
        YES, PLAUSIBLE, LINGUISTIC

Plausibility
(73) Q: The president was able to attend to meeting.
     A: Did the president attend the meeting?
        YES, PLAUSIBLE, LINGUISTIC

(74) Q: Many soldiers were killed in the ambush.
     A: Were all soldiers killed in the ambush?
        NO, PLAUSIBLE, LINGUISTIC

(75) Q: The man had $20 in his pocket.
     A: Did the man have $40 in his pocket?
        NO, PLAUSIBLE, LINGUISTIC
Other
(58) Q: Kerry knew that Edwards would accept the nomination.
     A: Did Kerry know whether Edwards would accept the nomination?
        YES, STRICT, LINGUISTIC

(59) Q: Tom knows that Naples is in Campania.
     A: Does Tom know where Naples is?
        YES, STRICT, LINGUISTIC
(75) Q: The man had $20 in his pocket.
     A: Did the man have $40 in his pocket?
        NO, PLAUSIBLE, LINGUISTIC

(76) Q: The man had $20 in his pocket.
     A: Did the man have $10 in his pocket?
        YES, STRICT, LINGUISTIC