The Flexibility Trap: Rethinking the Value of Arbitrary Order in Diffusion Language Models
Peer-reviewed conference; ICML 2026 Outstanding Paper, 2026 · paper · AI and machine learning
Examines arbitrary-order generation in diffusion language models and finds it can limit reasoning on math and coding by letting models bypass uncertain tokens. Proposes JustGRPO, which applies standard GRPO without arbitrary order, reaching 89.1% accuracy on GSM8K.
Checked: PDF from raw.githubusercontent.com, fetched 5 Oct 2026. Page numbers are those of that PDF.
1 finding
Wording Appendix B.3 says the phenomenon "is consistent across these diverse tokens" and "the Arbitrary Order consistently results in lower entropy", but in Figure 12 (middle panel) 'because' has Arbitrary Order ≈1.07 above AR Order ≈0.90.
- p. 14 Appendix B.3, second paragraph
Conversely, the Arbitrary Order consistently results in lower entropy
- p. 16 Figure 12, middle panel, 'because' bars
Entropy comparison results on more forking tokens.
Judge's note: Checked the Figure 12 page image. In the middle panel, the blue (Arbitrary Order) bar for 'because' is clearly taller than the orange (AR) bar. Every other one of the 33 tokens shows AR higher. The universal wording therefore overstates the figure, and a qualifier such as 'all but one' would fix it.
Second check: another model agreed
Report a problem with this finding
6 minor (typos and the like; most checked by the judge only)
Substance Confidence-based HumanEval Pass@128 for LLaDA-Instruct is "67.1" in Table 4 but "Confidence (Pass@128: 66.94%)" in Figure 11(b), while AR agrees (83.0 vs 83.04%).
- p. 15 Table 4, Pass@128, Confidence-based, HumanEval
67.1
- p. 14 Figure 11(b), x-axis label of 'Confidence vs AR' panel
Confidence (Pass@128: 66.94%)
- p. 15 Table 4, Pass@128 Confidence-based HumanEval
Confidence-based 97.0 71.4 67.1 67.1
Only the authors can say what is right here.
Judge's note: Checked Figure 11(b) on p.14 (x-axis label 'Confidence (Pass@128: 66.94%)', y-axis 'AR (Pass@128: 83.04%)') and Table 4 on p.15. Appendix B gives HumanEval with LLaDA-Instruct as its setting. 66.94 rounds to 66.9, not 67.1, while the AR value matches. The MBPP cell in the same Table 4 row is also 67.1, which suggests a possible copy slip. Separate runs are not stated anywhere. The difference is small, so only the authors can say which value is right.
Second check: another model agreed
Report a problem with this finding
Wording Section 6.1 says LLaDOU and LLaDA-1.5 "report higher accuracy on HumanEval", but in Table 1 LLaDA-1.5 is above JustGRPO only at length 512 (51.9 vs 48.7) and below it at 128 (29.3 vs 37.8) and 256 (39.6 vs 49.4).
- p. 8 Section 6.1, last paragraph of column 2
which leverage an auxiliary trainable module or larger-scale private training data and report higher accuracy on Hu-
- p. 8 Table 1, LLaDA-1.5 HumanEval row
29.3 39.6 51.9
- p. 8 Section 6.1, last paragraph
and report higher accuracy on Hu-
- p. 8 Section 6.1 text; Table 1
report higher accuracy on Hu-
Judge's note: Checked Table 1 and the text on p.8. LLaDOU at 256 (59.1 vs 49.4) supports the sentence. LLaDA-1.5 does so only at 512, which is the bolded cell. The sentence is not strictly false, since each model has a bolded HumanEval cell above JustGRPO, but it is unqualified while the headline comparison uses 256. Wording point only.
Second check: another model agreed
Report a problem with this finding
Wording Appendix C states "Under Pass@128, random order is at best comparable to confidence-based sampling" and the Table 4 caption says "Random order does not improve coverage", but in Table 4 Fully Random exceeds Confidence-based on MBPP (71.8 vs 67.1) and GSM8K (97.5 vs 97.0).
- p. 15 Appendix C, 'Random order does not help coverage' paragraph
Under Pass@128, random order is at best com-parable to confidence-based sampling
- p. 15 Table 4 caption
Random order does not improve coverage
- p. 15 Table 4, Pass@128 rows, MBPP column
71.8
- p. 15 Appendix C, 'Random order does not help coverage'
random order is at best comparable to confidence-based sampling, and it still falls well short of AR order
Judge's note: Checked the Table 4 image on p.15. The Pass@128 rows are Confidence 97.0/71.4/67.1/67.1, Random 97.5/70.6/71.8/64.9 and AR 99.0/75.6/78.7/83.0. The 4.7-point MBPP gain over confidence-based sampling is not 'at best comparable'. The claim that random order still falls below AR holds in every column. The GSM8K gap to AR (1.5) sits near the ceiling, so I do not count 'well short' as a separate defect.
Second check: another model agreed
Report a problem with this finding
Wording Appendix B.3 lists 26 capitalised forking tokens ("Therefore" … "Consider", "Also", "Express", and "Write"), but Figure 12 plots 33 bars, adding lower-case 'since', 'when', 'given', 'let', 'first', 'then', 'calculate' and showing 'follows', 'because', 'but', 'or', 'consider', 'also', 'express', 'write' only in lower case.
- p. 15 Appendix B.3, list of tokens
“Consider”, “Also”, “Express”, and “Write”.
- p. 16 Figure 12 x-axis labels
Entropy comparison results on more forking tokens.
Judge's note: Counted the list on p.15: 26 tokens. Counted the Figure 12 bars on p.16: 11 per panel, 33 in all, which is 26 plus 7 lower-case duplicates of listed tokens. The list does not distinguish case, so it does not describe the plotted token set exactly. This is a minor labelling inconsistency.
Second check: another model agreed
Report a problem with this finding
Typo Figure 2 caption reads "By foregoing complex arbitrary-order adaptations", while the abstract and Sections 5/5.2 use "forgoing".
- p. 2 Figure 2 caption
By foregoing complex arbitrary-order adaptations and adopting just GRPO
Judge's note: Checked the Figure 2 caption crop on p.2. The text layer has 'forgoing' in the abstract and in Section 5 ('forgoing it enables', 'By forgoing arbitrary-order generation'). 'Foregoing' is a recognised variant spelling, but the paper is inconsistent with itself. This is cosmetic.
Second check: none (judge only)
Report a problem with this finding
Typo Model name written "LLaDA 1.5" in Section 4.1 (p.4) but "LLaDA-1.5" in Table 1, its caption and footnotes (p.8).
- p. 4 Section 4.1
and LLaDA 1.5 (Zhu et al., 2025)
- p. 8 Table 1
LLaDA-1.5‡ (Zhu et al., 2025)
Judge's note: Checked the crops for p.4 and Table 1 on p.8. The reference title also uses 'LLaDA 1.5', and the hyphenated form appears in Table 1 and the footnotes. This is a cosmetic inconsistency with no effect on meaning.
Second check: none (judge only)