LLMs Get Lost In Multi-Turn Conversation

• p. 21 Situation 2 equation vs Figure 7 panel 2

  100, if 4 ≤ i ≤ 9, 1 ≤ j ≤ 7

Judge's note: I looked at page 21 and at the aligned text layer of the grid (main.layout.txt lines 1489-1499). Rows 1-5 have 6 checks, rows 6-9 have 7, and row 10 has none. That gives 58 checks out of 100, but the panel label says 60. The equation does give 60. Situation 3's grid matches its equation (3 full rows plus 5 rows of 6, which is 60). So only panel 2 is drawn wrong; the equation and label agree with each other.

Second check: another model agreed

• p. 2 Introduction, para 5

  (1) generate overly verbose responses, leading them to (2) propose final solutions prematurely in conversation, (3) make incorrect assumptions about underspecified details, and (4) rely too heavily on previous (incorrect) answer attempts

• p. 8 Section 5.2, last para

  We identify four specific causes

Judge's note: I compared the intro list on p.2 with the Section 5.2 list on pp.8-9 and the Appendix F summary. In the body, premature answers and assumptions form one cause (§F.1), and loss-of-middle-turns (§F.3) is a separate cause. The intro omits loss-of-middle-turns, counts assumptions as a cause of its own, and adds a causal chain (verbosity leading to premature solutions) that the body does not state. The intro reads as a loose paraphrase, but its numbered list does not match the body's.

Second check: another model agreed

• p. 10 Section 8, para 2

  the simulation process ensures a new shard of information is revealed at each turn

• p. 2 Introduction

  reveals at most one shard of information per conversation turn

• p. 41 User simulator prompt

  or -1 if you did not reveal any shards

Judge's note: Checked p2, Section 3.2 text (p3–4) and the simulator prompt (p41). The method allows turns that reveal no shard, so 'ensures a new shard ... at each turn' misdescribes it. This is a simplification in the limitations section.

Second check: another model agreed

• p. 6 Section 4.2

  By running N simulations for an instruction

• p. 20 Appendix E

  Let’s imagine we are provided with ten instructions (N = 10)

• p. 3 Section 3.2

  in a conversation with N shards, the assistant gets up to N answer attempts

• p. 20 Appendix E

  ten instructions (N = 10), each FULL and SHARDED

Judge's note: Text layer lines 217, 479, 1955 and 2009 match the crops on pp.3, 6 and 20. Section 5.3 uses N for shards. Appendix F.4 and the later appendices use N=10 for simulations. Each use is defined locally, so this is an inconsistency in notation, not a factual error.

Second check: another model agreed

• p. 9 Section 6, System and Agent Builders

  a final corrective turn informs the assistant that its prior solutions were incorrect

• p. 4 Section 3.3, RECAP

  adds one recap turn that restates every shard

• p. 35 Appendix N

  an additional recapitulation turn that gathers all the previous user utterances

Only the authors can say what is right here.

Judge's note: Read Section 6 (p9), Section 3.3 (p4), G.1 (p25–26) and Appendix N (p35). No description other than Section 6 mentions telling the model its earlier answers were wrong. Appendix N also says the recap is built from verbalized user utterances, not the shards themselves. The RECAP prompt text is not printed, so only the authors can say which description is right.

Second check: another model agreed

• p. 24 Appendix F.3

  averages numbers across the 15 LLMs included in our main experiment

• p. 6 Section 4.3

  Two models (Phi-4, OLMo-2-13B) could not accommodate the summarization task’s context length and were excluded from it

Judge's note: The page 24 image shows that Figure 9 analyses Summary-task citations only. The page 6 crop says the two models could not handle the summarization context and were excluded (dashes in Table 1). At most 13 LLMs can contribute to Figure 9. I found no statement that the two models were run separately for this analysis.

Second check: another model agreed

• p. 7 Table 1 caption

  The last two columns average the performance drops from the CONCAT and SHARDED compared to the FULL in percentages

Judge's note: The Overall headers on the image are CONCAT/FULL and SHARDED/FULL, and the values (91.6, 62.5, 103.6, 100.4) are percentages retained. A drop of 103.6% would be meaningless. Only the caption wording is wrong.

Second check: another model agreed

• p. 8 Section 5.1, first paragraph

  averaging 95.1% of the F ULL counterpart

• p. 7 Table 1, Overall CONCAT/FULL column

  103.6

• p. 8 Section 5.1

  CONCAT performance averaging 95.1% of the FULL counterpart

• p. 7 Table 1, Overall CONCAT column

  91.6

Only the authors can say what is right here.

Judge's note: I read the Overall CONCAT/FULL column from the Table 1 image. Its 15 values sum to 1437.1, a mean of 95.8. The same aggregation over the SHARDED/FULL column gives 61.0, which reproduces the paper's -39%, so a simple model mean appears to be the paper's convention. The paper does not describe any other aggregation for the 95.1 figure. The gap is small, and only the authors can say which value is right.

Second check: another model agreed

• p. 8 Section 5.2, para 1

  the two most able models (GPT-4.1 and Gemini 2.5 Pro) achieve the lowest unreliability

• p. 7 Figure 5b, Full and Concat rows

  (b) Observed Model Degradations

Only the authors can say what is right here.

Judge's note: Figure 5b values from the page image and text layer. Full: A is Pro 83, then 4.1 and Flash tied at 82; U is lowest for Scout and Pro (13), then 3.3-70B and 4.1 (14). Concat: A is Flash 83, Pro 83, Phi-4 82, then 4.1 81; U is 3.3-70B 14, Scout 15, Pro 15, R1 18, 4.1 19. The general tendency holds, but the specific superlative does not hold in both settings. Severity is reduced to wording.

Second check: another model agreed

• p. 8 Section 5.2, para 1

  the two models with the lowest aptitude (Llama3.1-8B-Instruct and OLMo-2-13B) are also the most unreliable

• p. 7 Figure 5b, Concat row

  48%

Only the authors can say what is right here.

Judge's note: Figure 5b Concat U: Llama3.1-8B 50%, OLMo2 45%, Phi-4 48%. Concat A is lowest for Llama3.1-8B and OLMo2 (63 each). The claim holds in Full (U 49% and 47%, then Phi-4 39%) but not in Concat.

Second check: another model agreed

• p. 10 Section 7, Conclusion

  known remediations that work for simpler settings (agent-like concatenation or decreasing temperature) are ineffective in multi-turn settings

• p. 9 Section 6

  Both strategies improve over SHARDED but fall short of FULL or CONCAT performance

Only the authors can say what is right here.

Judge's note: Checked Table 5 on p26: SHARDED 50.4/59.1, RECAP 66.5/76.6, SNOWBALL 61.8/65.3 for 4o-mini/4o. Section 6 calls them 'not sufficient', and G.1 says they mitigate the drop by 15-20%. 'Ineffective' overstates this for the agent-like remediations. The temperature half of the claim matches G.2.

Second check: another model agreed

• p. 26 Appendix G.1

  it can mitigate the F ULL-to-S HARDED performance deterioration by 15-20%

• p. 9 Section 6

  S NOWBALL offers the more achievable improvement in practice (15–20%)

• p. 26 Table 5

  86.8 84.4 50.4 66.5 61.8

• p. 26 Table 5

  61.8

Only the authors can say what is right here.

Judge's note: I checked Table 5 on page 26. 4o-mini reads Full 86.8, Concat 84.4, Sharded 50.4, Recap 66.5, Snowball 61.8. 4o reads Full 93.0, Concat 90.9, Sharded 59.1, Recap 76.6, Snowball 65.3. The share of the gap recovered is 31% for 4o-mini and 18% for 4o. The relative gain over Sharded is 23% and 10%. The paper does not define how the 15–20% was computed, so only the authors can say which measure was meant.

Second check: another model agreed

• p. 18 Appendix C intro

  followed by a manual step that was conducted by an author of the paper

• p. 19 Appendix C, Step 4

  were all manually reviewed by two authors of the work

• p. 19 Step 4

  all manually reviewed by two authors of the work

Only the authors can say what is right here.

Judge's note: Checked p18 and p19 (and the p18 caption text). The sources give one author, two authors and 'the authors' for the manual step. These could be reconciled (one author edits, two review), but the paper does not say so.

Second check: another model agreed

• p. 20 Table 2

  Overall Success 97.8 100.0 96.0 96.0 100.0

Only the authors can say what is right here.

Judge's note: Table 2 on p20. Overall Success is one global label per conversation. With N=100 the all-task percentage must be an integer; with N=200 it must be a multiple of 0.5. 97.8 is neither. The unweighted mean of the per-task values is 98.0. This likely ties to the 100/200 discrepancy in C025. Only the authors can say how 97.8 was computed.

Second check: another model agreed

• p. 21 Figure 7, panel '2: Drop in Reliability'

  P = 60, A = 90, U = 90

• p. 21 Situation 2 equation

  100, if 1 ≤ i ≤ 3, 1 ≤ j ≤ 6

Judge's note: I counted the ticks in the crop: rows 1–5 have 6 passes, rows 6–9 have 7, row 10 has none. The equation implies 3 rows of 6 and 6 rows of 7. A and U are unaffected, but the grid's mean is 58, not the P=60 printed under it. The other panels match their equations.

Second check: another model agreed

• p. 27 Appendix G.2

  GPT-4o-mini does not see improvements in reliability as AT is decreased (in all user-temperature settings)

• p. 27 Table 7, 4o-mini UT=0.0 row

  38.5 28.0 30.5

Only the authors can say what is right here.

Judge's note: I checked Table 7 on page 27. 4o-mini reads UT=1.0: 49.8, 46.8, 51.0; UT=0.5: 31.7, 34.0, 40.5; UT=0.0: 38.5, 28.0, 30.5. The UT=0.0 drop of about 21% is as large as GPT-4o's drops (22%, 19%, 17%), which the same sentence calls minor improvements 'on the order of 15-20%'. The claim holds at UT=1.0 and UT=0.5 but not at UT=0.0. The parenthetical is ambiguous ('not in all' or 'in none'), so only the authors can say which they meant.

Second check: another model agreed

• p. 27 Appendix G.2 text

  with a drop in unreliability (U90 10 ) of 50-80% when the assistant temperature decreases

• p. 27 Table 7, 4o FULL row

  17.8 8.0 2.8

Judge's note: I checked Table 7 on page 27. The four single-turn drops from AT=1.0 to AT=0.0 are 57% (4o-mini FULL), 53% (4o-mini CONCAT), 84% (4o FULL) and 71% (4o CONCAT). One of them lies slightly outside the stated range. This is a loose summary rather than a substantive error.

Second check: another model agreed

• p. 28 Appendix G.3

  with BLEU scores being within 10% difference of each other in all settings

• p. 28 Table 8, 4o row

  35.9 38.5 40.9

• p. 28 App G.3 text

  BLEU scores being within 10% difference of each other in all settings

• p. 28 Table 8

  4o 35.9 38.5 40.9

Judge's note: I checked Table 8 on page 28. 4o-mini reads 41.7, 43.4, 42.1, which is within 10%. For 4o, SHARDED is 5.0 BLEU points (13.9%) above FULL. The claim of no degradation still holds because SHARDED is higher. Only the '10%' bound is exceeded in relative terms; it would hold if read as 10 BLEU points.

Second check: another model agreed

• p. 2 Section 2, first sentence

  were designed for single-turn tasks, and so do the evaluation

• p. 2 Section 2

  and so do the evaluation

Judge's note: Seen on the p.2 image. The verb does not agree with the plural past-tense clause before it. A fix such as "and so was their evaluation" is evident.

Second check: none (judge only)

• p. 2 Introduction, para 3

  (LLama3.1-8B-Instruct)

Judge's note: Visible on the p.2 image. Sections 4 and 5 spell it Llama3.1-8B-Instruct.

Second check: none (judge only)

• p. 5 Figure 4 caption

  based on existing datasets (Instruction Origin)

Judge's note: Checked on the p.5 image: the figure label reads "Instruction Source". The meaning is clear; the label is simply inconsistent.

Second check: none (judge only)

• p. 8 Sec 5.2

  performing degrading 50 percent points on average

Judge's note: Seen on the p.8 image. The cluster's quote "performing degrading" is a mis-transcription; the page reads "performance degrading 50 percent points". The issue is cosmetic terminology only.

Second check: none (judge only)

• p. 11 References, Belem et al. 2024

  Pouya Pezeskhpour

Only the authors can say what is right here.

Judge's note: Rests on outside knowledge: that the co-author's surname is spelled 'Pezeshkpour'. The page image shows 'Pezeskhpour', so this is not an extraction artefact.

Second check: none (judge only)

• p. 11 Reproducibility statement

  running each simulated conversations 10 independent times

Judge's note: The page image shows the text as quoted.

Second check: none (judge only)

• p. 16 Reference list, Zipf 1949

  An introduction to human eoclogy

Judge's note: The page image on p16 shows 'eoclogy' in italics, so this is not an extraction artefact.

Second check: none (judge only)

• p. 20 Appendix D

  inspection of each user and assistance utterance

Judge's note: Seen in the p20 page image.

Second check: none (judge only)

• p. 21 Appendix E, after Situation 3

  when compared to fully-specific simulations

• p. 21 Appendix E end of Situation 3

  compared to fully-specific simulations

Judge's note: Seen on p21. This is the only occurrence of 'fully-specific' in the text.

Second check: none (judge only)

• p. 24 Appendix F.4

  generating a Python function likely requires a longer than generating an SQL query

• p. 24 F.4

  likely requires a longer than generating an SQL query

Judge's note: The page 24 image shows the sentence as quoted, with no noun after 'longer'.

Second check: none (judge only)

• p. 26 G.1

  is to observe the model can shift its behavior

Judge's note: The page 26 image shows the sentence as quoted.

Second check: none (judge only)

• p. 30 Appendix J.1

  did not produce shared instructions with enough shards

• p. 30

  it was simplistic and did not produce shared instructions with enough shards

Judge's note: The crop of page 30 shows 'shared instructions'. The paper's term elsewhere is 'sharded instructions'.

Second check: none (judge only)

• p. 34 Fig 10 Turn 2/3

  Total Calories for Type E: 20z0 * 85 = 17,000 calories

• p. 34 Fig 10 Turn 3

  Andrew's mini cinnamon rollsz

Judge's note: The crops of page 34 show both strings. 200 × 85 = 17,000, so '20z0' stands for 200. These may be verbatim Llama3.1-8B output, so this is only a cosmetic point.

Second check: none (judge only)

• p. 36 Prompts heading

  O.1 SHARDING

• p. 35 Appendix O heading

  O N OBTAINING DETERMINISTIC OUTPUTS FROM LLM S

Judge's note: Page 35 shows the Appendix O heading. Page 36 shows a bare 'PROMPTS' heading followed by 'O.1 SHARDING', and the page 39 text shows 'O.2 EXPERIMENTS'. The heading should presumably be Appendix P. This is a numbering and cosmetic issue.

Second check: none (judge only)

• p. 36 Segmentation prompt

  You are a given a fully specified instruction

• p. 36 Segmentation prompt

  into a units of information

• p. 39 Answer Extraction prompt

  you must absoltutely include it

• p. 41 User simulator prompt

  favor being succint

Judge's note: I checked all four strings in the page crops and they appear as quoted. They sit inside prompt boxes that the paper presents as the prompts it actually used, so the authors may be reproducing them word for word on purpose. Correcting them would then misstate what was run. For that reason this is recorded as a cosmetic observation for the authors to decide, not as a required fix.

Second check: none (judge only)

• p. 37 Rephrasing prompt rules

  You do not need to keep the order the segments that are provided in.

Judge's note: I confirmed the sentence in crop C047-a, in the [Order of shards] rule. It is a grammatical slip inside a prompt that is printed word for word. It is cosmetic, and whether to fix it depends on whether the authors want to keep the prompt exactly as used.

Second check: none (judge only)

• p. 38 Verification prompt, Example 1 output

  {"converage": "complete"}

Judge's note: Crop C048-a shows {"converage": "complete"} as the Example 1 output. A text search finds {"coverage": "incomplete", "missing_segment": ...} in Example 2 of the same prompt (main.txt line 3487). So the same key is spelled two ways within one printed prompt. If the prompt is reproduced word for word, the misspelling was probably in the prompt that was run. The authors should say whether to correct it or keep it as used.

Second check: none (judge only)