The Imperfective Paradox in Large Language Models
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2026 · paper · Computational linguistics
Introduces ImperfectiveNLI, a dataset testing whether language models grasp that past progressive sentences entail completion for activities but not goal-oriented events. Models show a bias toward assuming completion, which prompting only partly reduces.
Checked: PDF from aclanthology.org, fetched 5 Oct 2026. Page numbers are those of that PDF.
9 findings
Substance Table 6 gives a Group C "False" rate of "0.33" for Llama-3.1-8B-Instruct under Counterfactual, but Table 3 gives Acc C 0.97 and TBR_C 0.00 for the same cell, so the three label shares would sum to 1.30.
- p. 16 Table 6, Counterfactual, Llama-3.1-8B-Instruct
0.33
- p. 6 Table 3, Counterfactual, Llama-3.1-8B-Instruct, Acc C / TBR_C
0.97
- p. 16 Table 6, Counterfactual, Llama-3.1-8B-Instruct
Llama-3.1-8B-Instruct 0.33
- p. 6 Table 3, Counterfactual Llama row
0.35 0.00 0.97 0.00 0.00 0.00
Judge's note: Read Table 6 (p16), Table 3 (p6) and Tables 17/18 (p19) on the images. Group C gold label is Unknown, so Acc C is the Unknown share, TBR_C is the True share (Eq. 1) and Table 6 is the False share; they cannot exceed 1 together. Other rows are consistent: Llama DAP 0.36+0.45+0.19 = 1.00, Mistral zero-shot 0.02+0.97+0.01 = 1.00. For Llama Counterfactual the residual is 0.03. The printed 0.33 matches the Motion to Goal subclass residual alone (Table 17: 0.67, Table 18: 0.00), so a subclass value or a typo for 0.03 is likely. It also conflicts with the adjacent text "very few models in very few cases predicted False". No different subset or run is described for Table 6. The reviewers' GLM-4 point is handled in C021.
Second check: another model agreed
Report a problem with this finding
Substance Table 6 gives a Group C "False" rate of "0.17" for glm-4-9b-chat-hf under Counterfactual, but Table 3 gives Acc C 0.73 and TBR_C 0.20 for the same cell, so the three label shares would sum to 1.10.
- p. 16 Table 6, Counterfactual, glm-4-9b-chat-hf
0.17
- p. 6 Table 3, Counterfactual, glm-4-9b-chat-hf, Acc C / TBR_C
0.73
- p. 16 Table 6, Counterfactual, glm-4
glm-4-9b-chat-hf 0.17
- p. 6 Table 3, Counterfactual glm
0.23 0.66 0.73 0.24 0.20 0.04
Only the authors can say what is right here.
Judge's note: Read Table 6 (p16), Table 3 (p6) and Tables 17/18 (p19) on the images. Unknown 0.73 plus True 0.20 leaves at most 0.07 for False; Table 6 prints 0.17. A 0.10 excess is beyond rounding, and unparsed outputs would lower the sum, not raise it. Other rows in Table 6 are consistent with Table 3 (see C020). The printed 0.17 is close to the Motion to Goal subclass residual (Table 17: 0.67, Table 18: 0.17, residual 0.16). The intended value (about 0.07) can only be confirmed by the authors. Kept separate from C020 because it is a different table cell.
Second check: another model agreed
Report a problem with this finding
Substance Section 5.4 states "∆Motion−Creation ≈ +28% in Group A and ≈ +13% in Group C", but Figure 3 shows Group C averages of 0.16 (Motion to Goal) and 0.09 (Creation), a gap of about 7 points, not 13.
- p. 8 Section 5.4, 'The Resilience of Motion vs. The Creation Penalty'
≈ +28% in Group A and ≈ +13% in Group C
- p. 8 Figure 3, Group C bars (Creation, Motion to Goal)
0.09
- p. 8 Section 5.4, 'The Resilience of Motion vs. The Creation Penalty'
+28% in Group A and ≈ +13% in Group C.
- p. 8 Figure 3, Group C bar labels
0.16
Judge's note: Read the Figure 3 bar labels and the sentence on the p8 image, and Table 8 on the p18 image. The Group A gap (0.46 − 0.18) reproduces the stated +28%. The Group C gap is 0.07 from the figure and 0.066 from the seven-model means of Table 8 (Motion 0.159, Creation 0.093). No alternative reading gives 13: the relative difference is about 78%, and the zero-shot TBR gap from Table 9 is about 6 points. Footnote 4 does not explain it.
Second check: another model agreed
Report a problem with this finding
Reference or label The evaluated model "deepseek-llm-7b-chat (DeepSeek-AI, 2024)" is cited to "DeepSeek-AI. 2024. Deepseek-v3 technical report. Preprint, arXiv:2412.19437.", a report whose title names a different model.
- p. 4 Section 4.1 Models
deepseek-llm-7b-chat
- p. 11 References, DeepSeek-AI 2024
Deepseek-v3 technical report
- p. 4 Section 4.1 Models
deepseek-llm-7b-chat (DeepSeek-AI, 2024)
- p. 11 References
DeepSeek-AI. 2024. Deepseek-v3 technical report.
Only the authors can say what is right here.
Judge's note: Rests on outside knowledge: that DeepSeek-V3 is a different model from the 7B DeepSeek-LLM chat model, and that a more fitting reference exists. On the page, the Section 4.1 citation and the reference entry read exactly as quoted, and the entry's own title names 'Deepseek-v3', not the 7B model used. The authors should say which source was intended.
Second check: another model agreed
Report a problem with this finding
Wording Section 3 gives "Motion to Goal (8)" and zero-shot Tables 7-9 show eighths (0.12, 0.25, 0.38, 0.62, 0.75, 0.88), but the Motion to Goal column of Tables 10-18 shows 0.17, 0.33, 0.67, 0.83, which are sixths and cannot occur with 8 items.
- p. 3 Section 3, semantic classes
Motion to Goal (8)
- p. 18 Table 10, Motion to Goal column
0.83
- p. 19 Table 13, Motion to Goal column
0.67
- p. 18 Table 11, Motion to Goal column
0.17
Only the authors can say what is right here.
Judge's note: Read Tables 7-12 on the p18 image and Tables 13-18 on the p19 image. Every Motion to Goal entry in Tables 7-9 is a multiple of 1/8; every entry in Tables 10-18 is a multiple of 1/6 (0, 0.17, 0.33, 0.50, 0.67, 0.83, 1.00). Appendix A (p14) lists 8 Motion to Goal verbs. I found no statement in the text about excluded items, unparseable outputs or a different class assignment for the DAP/CoT/Counterfactual runs. Only the authors can say which denominator is right, so this is a query.
Second check: another model agreed
Report a problem with this finding
Wording Table 4 caption defines agreement as instances where "the scores from the two evaluators differed by at most 1 point", while Section 3 ("three native English speakers as evaluators") and Appendix C ("We recruited 3 native English speakers") report three evaluators.
- p. 15 Table 4 caption
the two evaluators differed by at most 1 point
- p. 15 Appendix C, first paragraph
We recruited 3 native En-
- p. 4 Section 3, Quality Assurance
three native English speakers as evaluators
- p. 15 Appendix C, first paragraph
We recruited 3 native En- glish speakers
Only the authors can say what is right here.
Judge's note: Read p4 and p15 images. Section 3 and Appendix C both say three evaluators. The Table 4 caption says "the two evaluators". The Appendix C body text speaks of "the percentage of sample pairs where the score difference between evaluators is within a tolerance of 1", which does not say how three raters were paired or whether each item was rated by only two. No statement elsewhere explains it. Only the authors can say how agreement was computed.
Second check: another model agreed
Report a problem with this finding
Wording Appendix E says the trade-off analysis is extended "to the remaining five models", but 7 models are evaluated and Figure 1 covers only Llama-3.1, leaving six; Figures 5-9 cover Mistral, Yi, Gemma-2, GLM-4 and DeepSeek, and Qwen2.5-7B has no trade-off figure.
- p. 16 Appendix E, 'Trade-offs of the Prompts across Models'
offs in prompting to the remaining five models
- p. 7 Section 5.2, end of Counterfactual paragraph
ses and visualization for other models are provided
- p. 4 Section 4.1
7 instruction-tuned open-weight models
- p. 7 Section 5.2, end of Counterfactuals paragraph
Detailed analy- ses and visualization for other models are provided in Appendix E.
Only the authors can say what is right here.
Judge's note: Section 4.1 (p4 image) lists 7 models and Table 3 (p6) has 7 rows per prompt. Figure 1 is Llama-3.1 (Section 5.2 text), and Section 5.2 promises "visualization for other models" in Appendix E. Page 17 shows five figures (5-9): Mistral, Yi, Gemma-2, GLM-4, DeepSeek. Qwen2.5-7B is missing and no reason is given anywhere in the text. Either the count or the figure set needs the authors' clarification.
Second check: another model agreed
Report a problem with this finding
Wording "We observe a clear monotonic relationship between model scale and both evaluation metrics." is not borne out by Table 5: TBR is 1.00, 0.53, 0.61, 0.09, 0.16 and ΔAA is 0.00, 0.44, 0.37, 0.83, 0.81 across 1.5B to 72B, so neither is monotonic.
- p. 7 Section 5.3, second paragraph
We observe a clear monotonic relationship be-
- p. 16 Table 5, rows 7B and 14B (TBR_C, ΔAA)
0.61
- p. 7 Section 5.3, second paragraph
We observe a clear monotonic relationship be- tween model scale and both evaluation metrics.
- p. 16 Table 5, TBR_C and ΔAA columns
14B 0.24 0.86 0.39 0.98 0.61 0.37
Judge's note: Read Table 5 on the p16 image, Figure 2 on the p8 image and the sentence on the p7 image. TBR rises from 7B to 14B and from 32B to 72B; ΔAA falls at the same two steps. Figure 2 shows the same zig-zag, and the same section says ΔAA 'decreases slightly to 0.81' at 72B. 'Monotonic' is literally false; the overall trend and the reported correlations are unaffected, so this is a wording matter.
Second check: another model agreed
Report a problem with this finding
Wording Appendix E says Mistral-7B, Yi-1.5, Gemma-2 and GLM-4 "exhibit strong, statistically significant inverse correlations", but the figure captions print p = 0.07 (Mistral Group D), p = 0.06 (Gemma-2 Group C) and "Group D: r = −0.79, p = 0.21" (GLM-4).
- p. 16 Appendix E, 'Trade-offs of the Prompts across Models'
exhibit strong, statistically significant
- p. 17 Figure 8 caption (GLM-4)
Group D: r = −0.79, p =
- p. 17 Figure 5 caption (Mistral)
r = −0.93, p = 0.07
- p. 16 Appendix E, 'Trade-offs of the Prompts across Models'
exhibit strong, statistically significant inverse correlations
Only the authors can say what is right here.
Judge's note: Read the Appendix E paragraph on p16 and the captions of Figures 5-8 on p17. Printed p values: Mistral C 0.05, D 0.07; Yi C 0.03, D 0.05; Gemma C 0.06, D 0.05; GLM C 0.03, D 0.21. The paper states no significance threshold, and the Figure 1 caption itself treats p = 0.09 as "less consistently". The p values and the claim come from the same correlations (the text quotes the same r thresholds as the captions), so the wording "statistically significant" for all four models does not match the printed p values, most clearly GLM-4 Group D. The strict "r < −0.79" point is handled in C025.
Second check: another model agreed
Report a problem with this finding
8 minor (typos and the like; most checked by the judge only)
Wording Example verbs given for the classes, "(e.g., arrive, enter)" for Motion to Goal (Section 3 and again in Section 5.4, "Verbs of Motion to Goal (e.g., “arrive”, “enter”)") and "(e.g., melt, break)" for Change of State, are not in the Appendix A lists: "Motion to Goal: [’climb’, ’cross’, ’hike’, ’fly’, ’mail’, ’deliver’, ’catch’, ’locate’]"; 'break' is absent from Change of State.
- p. 3 Section 3, description of semantic classes
arrive, enter
- p. 3 Section 3, description of semantic classes
melt, break
- p. 8 Section 5.4, results on Motion to Goal
Verbs of Motion to Goal (e.g., “arrive”, “en-
- p. 14 Appendix A, Motion to Goal list
Motion to Goal: [’climb’, ’cross’, ’hike’, ’fly’,
Judge's note: Checked the p14 image: the Motion to Goal list has 8 verbs with neither arrive nor enter, and Change of State has 'crack' but no 'break'. The other examples (melt, build, knit, eat, burn) are in the lists. The examples are introduced with 'e.g.' and can be read as illustrating the linguistic class, so Section 3 alone is defensible. The Section 5.4 use, attached to measured results, is the mildly misleading one. Minor consistency point.
Second check: another model agreed
Report a problem with this finding
Wording Table 3 gives Qwen2.5-7B-Instruct DAP Acc A as "0.35", but Table 10's class values "0.34 0.33 0.28 1.00" weighted by the stated class sizes 44/9/39/8 give about 0.37.
- p. 6 Table 3, DAP, Qwen2.5-7B
Qwen2.5-7B-Instruct 0.35 0.97 0.89 0.72 0.09 0.63
- p. 18 Table 10, Qwen2.5-7B-Instruct
Qwen2.5-7B-Instruct 0.34 0.33 0.28 1.00
- p. 6 Table 3, DAP, Qwen2.5-7B-Instruct
0.35 0.97 0.89 0.72 0.09 0.63
Only the authors can say what is right here.
Judge's note: Read Table 3 (p6 image) and Table 10 (p18 image). With the stated sizes the weighted count is 15+3+11+8 = 37 of 100, against 0.35 printed. This looks like a consequence of the C002 anomaly rather than a separate error: if Motion to Goal has 6 items in Table 10 (as its 0.83/0.33 entries indicate), the count is 35, matching 0.35. The Llama row behaves the same way (61 with 8 Motion items against 0.60 printed; 59 with 6, plus up to 2 unassigned). Best raised with the authors together with C002.
Second check: another model agreed
Report a problem with this finding
Wording The association between Qwen2.5 model size and ΔAA is given as "r = 0.94, p = 0.01" in Section 5.3 and as "the linear scaling trend (r = 0.79)" in Appendix E, with no statement of how size was scaled in either case.
- p. 8 Section 5.3
r = 0.94, p =
- p. 16 Appendix E, 'Qwen2.5 with Different Scales'
(r = 0.79)
Only the authors can say what is right here.
Judge's note: Read both passages on the p8 and p16 images and recomputed from Table 5. Pearson r on raw parameter count (1.5, 7, 14, 32, 72) is about 0.79 (r² ≈ 0.63), matching the appendix. On log size I get roughly 0.95 for ΔAA and −0.93 for TBR, close to the main-text values, so the two figures probably use different x-scalings. The paper states neither, so a reader sees two r values for what reads as one relationship. Query to the authors on which scale each r uses; I make no claim that either number is wrong.
Second check: another model agreed
Report a problem with this finding
Wording Prompt P4 requires a JSON with "exactly two keys": "reasoning" and "label", but its Example Format shows { "possible_interruptions": ["..."], "label": ... } with no "reasoning" key.
- p. 15 Appendix B, P4 Counterfactual Simulation Prompt, Output Requirement
"reasoning": A string explaining your step-by-step
- p. 15 Appendix B, P4 Counterfactual Simulation Prompt, Example Format
{ "possible_interruptions": ["..."],
Only the authors can say what is right here.
Judge's note: Read the P4 box on the p15 image. The Output Requirement text is identical to P3's (keys "reasoning" and "label"), while the example format uses "possible_interruptions" (a list) and "label". The printed prompt is internally inconsistent. It may faithfully reproduce the prompt actually sent to the models, so the authors should say whether the write-up or the prompt carries the inconsistency.
Second check: another model agreed
Report a problem with this finding
Typo Missing article in two places: "uncover following critical insights" (p2) and "We report following metrics:" (p5); both should read "the following".
- p. 2 Introduction
uncover following critical insights
- p. 5 Section 4.3
We report following metrics:
Judge's note: Both phrases read on the crop images exactly as quoted. Purely grammatical.
Second check: none (judge only)
Report a problem with this finding
Typo Appendix E names the neutral label "Unsure" ("the models would either select “Unsure” (the correct predictions for uncertainty)"), whereas the label is "Unknown" in Section 3, Table 2 and every prompt.
- p. 16 Appendix E, 'The Predictions of “False” in Group C'
“Unsure” (the correct predictions for uncertainty)
- p. 4 Section 3, label mapping
“True”, “False”, and “Un-
- p. 4 Section 3, label mapping
“True”, “False”, and “Un- known”
- p. 16 Appendix E, 'The Predictions of False in Group C'
the models would either select “Unsure” (the correct predictions for uncertainty)
Judge's note: Read p16 and p4 images and the prompts on p15. Section 3 defines the labels as "True", "False" and "Unknown"; Table 2 and prompts P2-P4 use "Unknown". "Unsure" appears in quotation marks as a label name only in this Appendix E paragraph. Same label is clearly meant; evident fix is "Unknown".
Second check: none (judge only)
Report a problem with this finding
Typo Cosmetic inconsistencies: Table 3 caption lists "Zero-shot, CoT, DAP, and Counterfactual" while the rows run Zero-shot, DAP, CoT, Counterfactual; Table 6 prints the row label "COT" (elsewhere "CoT"); Appendix E says models would select "“Unsure” (the correct predictions for uncertainty)" although the label is "Unknown".
- p. 6 Table 3 caption
under Zero-shot, CoT, DAP, and Counterfactual prompting strategies
- p. 6 Table 3 caption
Zero-shot, CoT, DAP, and Counterfactual prompting strategies
- p. 16 Table 6
COT
- p. 16 Appendix E, False predictions
“Unsure” (the correct predictions for uncertainty)
Judge's note: Checked Table 3 and its caption on the p6 image, and Table 6 and the Appendix E paragraph on the p16 image. All three points are as described. The caption order and capitalisation are purely cosmetic. 'Unsure' differs from the 'Unknown' label defined in Section 3 and the prompts, but the meaning is clear. No values are affected.
Second check: none (judge only)
Report a problem with this finding
Typo Footnote 3 lacks a space: "the activity“running”".
- p. 7 Footnote 3
the activity“running”
Judge's note: Visible on the crop image: no space between 'activity' and the opening quotation mark.
Second check: none (judge only)