Memory Efficiency and Resource-Rational Encoding in Sentence Processing
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2026 · paper · Computational linguistics
Constrains working memory in Transformer language models by injecting noise into hidden representations and training under limits on encoding precision. Such constraints improve alignment with human reading times and make context representations more compressed and categorical.
Checked: PDF from aclanthology.org, fetched 5 Oct 2026. Page numbers are those of that PDF.
4 findings
Wording Abstract: "We find that explicit WM constraints improve the model’s alignment with human reading times" and Conclusion: "We show that explicit WM constraints improve the model’s alignment with human RTs" are unqualified, whereas Figure 2a and footnote 18 ("the enhanced predictive power is not observed in eye-tracking total time, nor in Maze RT") show the gain only for SPR NSC at λ = 0.01 and a mild trend for Provo first fixation.
- p. 1 Abstract
explicit WM constraints improve the model’s
- p. 9 Section 8 Conclusion
that explicit WM constraints improve the model’s
- p. 6 Figure 2a and footnote 18
However, the enhanced predictive power is not observed
- p. 9 8 Conclusion
We show that explicit WM constraints improve the model’s alignment with human RTs
Only the authors can say what is right here.
Judge's note: Looked at Figure 2a on page 6. Maze NSC: all five noisy-model bars are below the dashed baseline. SPR NSC (critical and spillover): only λ = 0.01 is clearly above the baseline; λ = 0.001 and λ = 0.1 are below it. Provo total RT: bars sit around the baseline with no pattern. Provo first fixation: a mild rise with wide error bars. The body text is hedged ("in principle can improve", "especially true for the SPR NSC corpus") and footnote 18 states the exceptions, and the Introduction says "can help". The abstract and conclusion drop these qualifications. Wording overstatement, not a wrong number.
Second check: another model agreed
Report a problem with this finding
Wording Section 3.1 defines "τ0 is the highest achievable precision in memory encoding" and two sentences later says "τ0 is the base level of irreducible noise"; footnote 14 likewise has "there is still a base noise τ0". τ0 is a precision (inverse variance), so the noise level is 1/τ0, not τ0.
- p. 4 Section 3.1, after Eq. 2
where τ0 is the highest achievable precision in memory encoding
- p. 4 Section 3.1, after Eq. 2
τ0 is the base level of irreducible noise
- p. 6 Footnote 14
there is still a base noise τ0
- p. 4 Section 3.1, after Eq. 2
where τ0 is the highest achievable precision in
Judge's note: Read page 4 (after Eq. 2) and footnote 14 on page 6 in the page images. Eq. 1 defines τ as the inverse of the variance, and Section 4.1 sets τ0 = 10000 "equivalent to σ = 0.01". The same symbol is called both the maximal precision and the base noise, which are reciprocal quantities. The intended meaning is recoverable (the noise corresponding to τ0), so this is a wording defect.
Second check: another model agreed
Report a problem with this finding
Wording "Moderate" constraint is λ = 0.01 in one paragraph of Section 5.2 ("moderate WM constraint (λ = 0.01)"; also Limitations: "a moderate level of λ (e.g., λ = 0.01 for the SPR NSC corpus)") and λ = 0.001 in another ("Although moderate constraint (λ = 0.001) indeed encourages ... this effect is weakened for mid-to-higher constraint (λ ≥ 0.01)"); Section 4.2 calls 0 < λ ≤ 0.01 "lower-to-mid".
- p. 6 Section 5.2, 'Better alignment with human RTs with WM constraint'
moderate WM constraint (λ = 0.01)
- p. 7 Section 5.2, 'Emergent but dissociable locality bias in attention'
Although moderate constraint (λ = 0.001) indeed
- p. 7 Section 5.2, same sentence continued in right column
mid-to-higher constraint (λ ≥0.01)
- p. 10 Limitations
peaks at a moderate level of λ (e.g., λ = 0.01 for the SPR NSC corpus)
Only the authors can say what is right here.
Judge's note: Read pages 5, 6, 7, 9 and 10. λ = 0.01 is labelled "moderate" for the reading-time result and falls in "mid-to-higher" for the attention result, where "moderate" is λ = 0.001. The General discussion then says "moderate WM constraints indeed enhance the locality bias in attention weights, this effect diminishes with more stringent constraint", without a value, so the reader cannot tell which λ is meant. A second, cosmetic point in this cluster is also real: the corpus abbreviation is defined as "SPRNSC" in Section 5.1 and written "SPR NSC" in Section 5.2, the Figure 2a panel titles and the Limitations (S4).
Second check: another model agreed
Report a problem with this finding
Wording Section 6.2: "As WM constraint gets stronger, the representational space becomes increasingly concentrated with lower item-to-item distance", but in Figure 3c the mean pairwise distance is not monotone in λ: the animate within-category and overall distances rise from λ = 0.1 to λ = 1, and the inanimate and overall distances rise from λ = 0.001 to λ = 0.01.
- p. 8 Section 6.2, 'More compressed representational space with higher WM constraint'
the representational space becomes increasingly concentrated with lower item-to-item distance
- p. 7 Figure 3c, within-category (animate) and within-category (inanimate) lines
Mean pairwise distance
- p. 8 Section 6.2, first paragraph
increasingly concentrated with lower item-to-item
Judge's note: Read Figure 3c in the enlarged crop. Approximate values for λ = 0, 0.001, 0.01, 0.1, 1: animate 209, 184, 182, 148, 166; inanimate 208, 159, 166, 135, 132; overall 248, 187, 190, 154, 165. Every constrained model is below λ = 0, so the general direction is as stated, but the minimum for the animate and overall series is at λ = 0.1 and distance goes back up at λ = 1. Footnote 22 says the standard errors are small, so the reversals are not within the plotted error. No statement elsewhere qualifies "increasingly". Values are read from a plot by eye.
Second check: another model agreed
Report a problem with this finding
18 minor (typos and the like; most checked by the judge only)
Wording On page 4 the letter z denotes the attention-head output ("The output z of each attention head at query position q", Eqs. 3, 5-7, bold) and also the loss normaliser ("where z is a normalization constant given by", Eqs. 8-9, italic).
- p. 4 Section 3.1, before Eq. 3
The output z of each attention head at query position q
- p. 4 Section 3.2, Eq. 8-9
where z is a normalization constant given by
Judge's note: Checked page 4 image. The attention output is bold z and the normalisation constant is italic z, so the two are typographically distinct, but the same letter is reused within one page for unrelated quantities. Eq. 8 also indexes the precision as τ_hltk where Eq. 2 used τ_qk. A notation overload, not a contradiction.
Second check: another model agreed
Report a problem with this finding
Wording Eq. 9 "z = L H T (T − 1)/2" uses L, H and T without defining them; the text after Eq. 8 defines only the lower-case indices ("over all attention heads h and layers l"), and Eq. 8 uses calligraphic ℒ for the loss.
- p. 4 Eq. 8 and Eq. 9
z = L H T (T −1)/2
Judge's note: Checked Eq. 8 and Eq. 9 on page 4 and the continuation on page 5. L, H and T (presumably number of layers, heads and sequence length; T also appears as the upper limit of the sum over t) are never defined in the text. The loss is calligraphic ℒ and the layer count is italic L, so they are distinguishable in print. Readers can infer the meaning; a minor notation gap.
Second check: another model agreed
Report a problem with this finding
Wording Section 6.2: "at least some linguistic features (e.g., prepositions in our case) are encoded more categorically with stronger WM constraint", while in Figure 4 the temporal-vs-spatial silhouette score peaks at λ = 0.01 and is lower at λ = 0.1 and 1, and the animate-vs-inanimate score is highest at λ = 0.
- p. 8 6.2 Emergence of categorical encoding
are encoded more categorically with stronger WM
- p. 8 Fig. 4
temporal vs. spatial
Judge's note: Read Figure 4 on page 8. Temporal vs. spatial: about 0.09 (λ = 0), 0.145 (0.001), 0.295 (0.01), 0.21 (0.1), 0.21 (1). Animate vs. inanimate: about 0.26 at λ = 0, then about 0.14-0.17. All constrained models score above λ = 0 on the preposition contrast, so the sentence holds as a contrast with the unconstrained model, and the text itself limits the claim to "at least some" features and concedes that animacy shows no categorical bias. What is not said is that the preposition effect is not monotone in λ, and the abstract's "more compressed and categorical" drops the limitation. Values read from a plot by eye.
Second check: another model agreed
Report a problem with this finding
Wording Appendix A says the learning rate was "linearly warmed up over the first 3% of the maximal training steps (i.e., 500 steps)", but with "16000 training steps" 3% is 480 steps; 500 steps is 3.125%.
- p. 14 Appendix A, Training procedure
first 3% of the maximal training steps (i.e., 500 steps)
- p. 13 Appendix A, Training procedure
16000 training steps
- p. 14 Appendix A, Training procedure
training steps (i.e., 500 steps)
- p. 14 Appendix A, Training procedure
linearly warmed up over the first 3% of the maximal training steps (i.e., 500 steps)
Judge's note: Page 13 gives 16000 training steps, and the Figure 5 x-axis on page 14 ends at 16000. Page 14 gives the warm-up as quoted. 3% of 16000 is 480, and 500 is 3.125% of 16000. No other statement of the warm-up length was found in the text. Most likely one of the two figures is rounded (3% for 3.125%, or 500 for 480), so "i.e." is loose rather than a substantive error.
Second check: another model agreed
Report a problem with this finding
Typo Dataset name capitalised inconsistently: "Wikitext-103" (Section 4.1 and Appendix A) versus "wikitext-103" (Limitations).
- p. 5 Section 4.1
Wikitext-103
- p. 10 Limitations, last paragraph
wikitext-103
Judge's note: Page 5 image shows monospace "Wikitext-103"; page 10 crop shows monospace "wikitext-103"; the text layer of Appendix A (page 13) has "Wikitext-103". Purely cosmetic.
Second check: none (judge only)
Report a problem with this finding
Typo Subject-verb agreement: "models with explicit WM constraint yields higher perplexity on RT corpora" should read "yield".
- p. 6 Section 5.2, first paragraph
models with ex- plicit WM constraint yields higher perplexity
- p. 6 Section 5.2, first paragraph
models with ex- plicit WM constraint yields higher perplexity on RT corpora
Judge's note: Read in the page 6 image, Section 5.2 first paragraph. The head noun is the plural "models".
Second check: none (judge only)
Report a problem with this finding
Typo Figure 2a y-axis label reads "Δlog-Likihood"; it should be "Δlog-Likelihood".
- p. 6 Figure 2a, y-axis label
Δlog-Likihood
Judge's note: The label is part of the figure graphic and absent from the text layer. In the enlarged crop of Figure 2 the vertical axis label reads "Δlog-Likihood" (the letters "el" are missing).
Second check: none (judge only)
Report a problem with this finding
Typo Figure 3a, last row of both matrices reads "xn,1 xn,1 xn,1 … xn,V"; following rows 1 and 2 (x1,1 x1,2 x1,3; x2,1 x2,2 x2,3) it should be xn,1 xn,2 xn,3 … xn,V.
- p. 7 Figure 3a, bottom row of both matrices (animate and inanimate)
xn,1 xn,1 xn,1 … xn,V
- p. 7 Figure 3a, both matrices, last row
xn,1 xn,1 xn,1
Judge's note: Confirmed in the enlarged crop of Figure 3a: both the animate (teal) and the inanimate (orange) matrix have the column index 1 printed three times in the bottom row. The text layer agrees. An index typo in a schematic; no data affected.
Second check: none (judge only)
Report a problem with this finding
Typo "using Principle Component Analysis (PCA)" should read "Principal Component Analysis".
- p. 8 Section 6.1, second paragraph
using Principle Component Analysis (PCA)
- p. 8 Section 6.1
Principle
- p. 8 6.1 Experiment setup
Principle Component Analysis (PCA)
Judge's note: Read in the page 8 image, Section 6.1, second paragraph: "Principle Component Analysis (PCA)".
Second check: none (judge only)
Report a problem with this finding
Typo Figure 8 (PDF page 16, not page 10) titles its first and last panels "λ = 0.0" and "λ = 1.0", while the Figure 5 legend and the body text write "λ = 0" and "λ = 1".
- p. 10 Figure 8 panel titles
λ = 0.0
Judge's note: The crop for page 10 shows only body text and references; Figure 8 is on PDF page 16. There the panel titles read λ = 0.0, 0.001, 0.01, 0.1 and 1.0. The Figure 5 legend on page 14 reads λ = 0 and λ = 1, and the body text writes λ = 0. Same values, formatting only.
Second check: none (judge only)
Report a problem with this finding
Typo Proper nouns are lowercase in three reference titles: "temporal prepositions in english" (Dubinsky and Williams 1995), "The provo corpus" (Luke and Christianson 2018) and "Findings of the babylm challenge" (Warstadt et al. 2023).
- p. 11 References, Dubinsky and Williams 1995
temporal prepositions in english
- p. 12 References, Luke and Christianson 2018
The provo corpus
- p. 13 References, Warstadt et al. 2023
Findings of the babylm
Judge's note: All three read as quoted on the page images (pages 11, 12, 13). The body text writes "Provo" capitalised throughout, and the Warstadt entry's own venue line writes "BabyLM Challenge". Unprotected capitals in the bibliography source; cosmetic.
Second check: none (judge only)
Report a problem with this finding
Typo Two journal references are incomplete: "Dissociating language and thought in large language models. Trends in Cognitive Sciences." (Mahowald et al. 2024) has no volume or pages, and "Annual Review of Psychology, 77." (Frank and Goodman 2025) has a volume but no pages.
- p. 12 References, Mahowald et al. 2024
Dissociating language and thought in large language models. Trends in Cognitive Sciences.
- p. 11 References, Frank and Goodman 2025
Annual Review of Psychology, 77.
Only the authors can say what is right here.
Judge's note: Both entries read as quoted on pages 12 and 11. Other journal entries in the list carry volume, issue and pages. The Frank and Goodman article may have been in press when cited, so its missing pages may be legitimate. The missing details are not in the files and would have to come from the authors.
Second check: none (judge only)
Report a problem with this finding
Typo Three reference entries end their author lists with the malformed phrase "and 1 others": Radford et al. 2019 ("Dario Amodei, Ilya Sutskever, and 1 others. 2019."), Goldstein et al. 2022 ("Alon Cohen, and 1 others. 2022.") and Warstadt et al. 2023 ("Linzen, and 1 others. 2023.").
- p. 13 References, Radford et al. 2019
Dario Amodei, Ilya Sutskever, and 1 others. 2019.
- p. 11 References, Goldstein et al. 2022
Alon Cohen, and 1 others. 2022.
- p. 13 References, Warstadt et al. 2023
Linzen, and 1 others. 2023.
Judge's note: All three read as quoted on pages 13 and 11. "1 others" is ungrammatical and is a bibliography-style artefact of an "and others" author field; the fix is "and others" or "et al.". The reviewer's further claim that the count of one is wrong (Radford et al. having only six authors, the other two papers many more) rests on the cited works' real author lists, which are not in the files; I did not verify it and it is not needed to confirm the defect.
Second check: none (judge only)
Report a problem with this finding
Typo Hu et al. (2020) gives the venue in lowercase, "Proceedings of the 58th annual meeting of the association for computational linguistics", while other entries capitalise it ("Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics" in Arora et al. 2024).
- p. 12 References, Hu et al. 2020
Proceedings of the 58th annual meeting of the association for computational linguistics
Judge's note: Read on page 12; the Arora et al. entry on page 10 has the capitalised form. Style inconsistency within the reference list, no change of meaning.
Second check: none (judge only)
Report a problem with this finding
Typo In Vaswani et al. (2017) an author's first name is printed with a stray space: "Ł ukasz Kaiser".
- p. 13 References, Vaswani et al. 2017
Ł ukasz Kaiser
Judge's note: The gap between Ł and ukasz is visible on the page image and the crop, so it is not a text-layer artefact. It comes from an unbraced \L macro in the bibliography source.
Second check: none (judge only)
Report a problem with this finding
Typo Warstadt et al. (2023) lists an author as "Bhargavi Paranjabe"; the surname is probably "Paranjape".
- p. 13 References, Warstadt et al. 2023
Bhargavi Paranjabe
Only the authors can say what is right here.
Judge's note: Rests on outside knowledge: that the BabyLM Challenge findings paper's author is spelled Paranjape. The page prints "Paranjabe" as quoted (crop and page 13). The correct spelling cannot be checked from the files.
Second check: none (judge only)
Report a problem with this finding
Typo Appendix C misspells "separability" twice: "the separatability between the two categories" and "indicates better separatability".
- p. 14 Appendix C, last paragraph
the separatability between the two cate-
- p. 14 Appendix C, last sentence
indicates better separatability
- p. 14 Appendix C, last paragraph
Then, the separatability between the two cate-
- p. 14 Appendix C, last paragraph
ette score thus indicates better separatability.
Judge's note: Both occurrences are visible on the page 14 image, so this is not an extraction artefact.
Second check: none (judge only)
Report a problem with this finding
Typo Appendix B reads "Figure 6 and 7 show the attention scores on key tokens"; two figures with a plural verb call for "Figures 6 and 7".
- p. 14 Appendix B
Figure 6 and 7 show the attention scores on key
Judge's note: Read on page 14. The verb "show" is plural, so the singular "Figure" is a grammatical slip. Cosmetic.
Second check: none (judge only)