Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

• p. 5 Section 3.2, paragraph on manual examination

  mathematically correct except two

• p. 25 Appendix D.1

  Of these 50, only one arrived at the correct answer through

• p. 5 Section 3.2, paragraph on manual examination

  ically and mathematically correct except two

• p. 5 Section 3.2, error analysis

  except two that coincidentally arrived at the correct answer

Only the authors can say what is right here.

Judge's note: Read page 5, page 25 and page 26. Both passages describe the same 50 LaMDA 137B GSM8K samples, and the main text cites D.1. D.1 says one was correct by chance and "the other 49 had correct logic and math", of which five had minor imperfections (1 underspecified, 1 unrelated statement, 2 imperfect equation, 1 inverted semantics). Table 9 shows exactly six examples, one of them marked correct by chance. So "two" (main text) conflicts with "one" (D.1), and "7 ... other 43" (Table 9 caption) conflicts with the six listed, which would leave 44. The acknowledgements mention a corrected mistake in the manual error analysis, which suggests a partial update. I found no statement that reconciles the counts. Only the authors can say which counts are right.

Second check: another model agreed · checked by hand

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 7 Section 4, Results

  outperforming the prior state of the art on StrategyQA (75.6% vs

• p. 22 Table 4, PaLM 540B row, StrategyQA CoT

  68.6 77.8

• p. 7 Section 4 Results

  outperforming the prior state of the art on StrategyQA (75.6% vs 69.4%)

• p. 7 Section 4 Results

  StrategyQA (75.6% vs 69.4%)

Judge's note: Viewed page 7 and page 22. The sentence names PaLM 540B with chain-of-thought prompting, which is exactly the Table 4 cell printing 77.8. The Figure 7 StrategyQA panel plots the 540B chain-of-thought point at about 77-78, above the 75 level, agreeing with the table. A text search finds 75.6 nowhere else in the paper and no statement of a different run or split. The Sports figure in the same sentence (95.4%) does match Table 4. The table and figure agree, so the text value is the one that is wrong.

Second check: another model agreed · checked by hand

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 18 Appendix A.2, 'Prompt engineering still matters'

  both were above standard prompting = 50.0% (see Table 7)

• p. 23 Table 7, Standard prompting, Coin

  49.0 ±2.1

• p. 18 Appendix A.2, prompt engineering paragraph

  standard prompting = 50.0% (see Table 7)

• p. 18 A.2 prompt engineering

  standard prompting = 50.0%

Judge's note: Read the 'Prompt engineering still matters' paragraph on page 18 and Table 7 on page 23. The other two numbers in the sentence match the table (99.6 for Annotator A, i.e. the chain-of-thought row, and 71.4 for Annotator C). The standard prompting Coin cell is 49.0 ±2.1, and Table 5 on page 22 gives 49.0 for LaMDA 137B on the in-domain coin flip. 50.0% is chance level, but the sentence labels it standard prompting and points at Table 7. Two tables agree on 49.0, so the text value is the one that is wrong. The difference is one point and does not change the argument.

Second check: another model agreed · checked by hand

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 23 Table 6, 'Chain of thought prompting' row, SVAMP

  36.7 ±0.4

• p. 20 Table 1, LaMDA 137B, Chain of thought, SVAMP

  37.5 (+8.0)

• p. 21 Table 2, LaMDA 137B, SVAMP

  29.5 37.5

Only the authors can say what is right here.

Judge's note: Read the Table 6 crop and pages 20, 21 and 23. Table 6 says its results are for LaMDA 137B over five seeds. Every other cell in its first two rows equals Tables 1 and 2: standard 6.5 / 29.5 / 40.1 / 43.2 and chain of thought 14.3 / 46.6 / 57.9. Only SVAMP chain of thought differs, 36.7 against 37.5. Table 1's +8.0 is consistent with 37.5. Table 6's annotator C row has 37.5 for SVAMP, which may be the source of a transcription slip in one direction or the other. No statement in the text explains a different run or analysis set for this one cell. Which value is right cannot be told from the paper.

Second check: another model agreed · checked by hand

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 27 Appendix D.2, 'Calculator error only' bullet

  went up from 14.3% to 17.3%

• p. 20 Table 1, LaMDA 137B, + ext. calc, GSM8K

  17.8

• p. 27 Appendix D.2, 'Calculator error only'

  went up from 14.3% to 17.3% when we added a Python program

Only the authors can say what is right here.

Judge's note: Page 27 reads 17.3%. In Table 1 (page 20 crop) the LaMDA 137B rows for GSM8K are Standard 6.5, Chain of thought 14.3 (+7.8), + ext. calc 17.8. The starting value 14.3 matches, so the same experiment is meant. The text layer has no other external-calculator figure for this model and no second calculator variant is described. Which of 17.3 and 17.8 is right only the authors can say.

Second check: another model agreed · checked by hand

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 26 Table 9 caption

  there were 7 salient cases where the chain of

• p. 26 Table 9 caption

  We found the other 43 to be equivalent

• p. 25 Appendix D.1

  Five had minor imperfections while maintaining coherent and

• p. 26 Table 9 caption

  there were 7 salient cases

Judge's note: Page 25, D.1: of 50, "only one" was correct by chance; "The other 49 had correct logic and math"; "Five had minor imperfections", itemised as 1 + 1 + 2 + 1. Page 26, Table 9 has exactly six rows: correct by chance, underspecified statements, unrelated statement, imperfect equation (two rows), inverted semantics. That matches D.1 and gives 6 and 44, not the caption's 7 and 43. Nothing in D.1 or the caption describes a seventh case.

Second check: another model agreed · checked by hand

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 39 Table 26, first exemplar

  is 12/30/2014. So one week from today will be 01/05/2015.

• p. 39 Table 26, first exemplar

  So one week from today will be 01/05/2015.

Judge's note: Viewed the full page image of PDF page 39. The chain of thought itself fixes today as 12/30/2014. December has 31 days, so 30 + 7 - 31 = 6, giving 01/06/2015, not 01/05/2015. Nothing on the page or in the caption explains the difference. The exemplar's reasoning and final answer are internally inconsistent as published.

Second check: another model agreed · checked by hand

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 5 Figure 4 caption

  for GSM8K, Jie et al. (2022) for SVAMP, and Lan

• p. 20 Table 1 caption

  b & e: Pi et al. (2022), c: Lan et al. (2021)

• p. 20 Table 1, Prior best row, SVAMP

  57.4

• p. 5 Figure 4 caption

  Jie et al. (2022) for SVAMP, and Lan et al. (2021) for MAWPS

Only the authors can say what is right here.

Judge's note: Read the Figure 4 caption on page 5 and the Table 1 caption and Prior best row on page 20 (57.4 superscript b for SVAMP, 75.3 superscript c for ASDiv, 88.4 superscript e for MAWPS). The two captions attribute the prior best for the same benchmarks to different papers, and nothing in the paper explains the difference. The MAWPS line in Figure 4 sits near 88, matching Table 1, so at least the MAWPS attribution in one caption is wrong. The plotted SVAMP value is handled in C011.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 5 Figure 4, SVAMP row, dashed 'Prior supervised best' line

  Prior supervised best

• p. 20 Table 1, Prior best, SVAMP

  57.4

Only the authors can say what is right here.

Judge's note: On the page 5 image the dashed line in the SVAMP row sits roughly a third of the way from the 40 tick to the 60 tick, about 47, clearly below 57.4. The GSM8K line (about 55) and the MAWPS line (about 88) match Table 1's 55 and 88.4, so the axis reading is reliable. This fits the different sources named in the two captions (Jie et al. in the figure, Pi et al. in the table, see C008). The value is read off a plot and is approximate. The text's claims hold under either value (PaLM 540B standard 69.4 and chain of thought 79.0 exceed both). Kept separate from C008 because this is a discrepancy in the plotted value, not in the attribution.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 30 Appendix E.3

  symbolic reasoning datasets were created synthetically, as described in Section 4.

• p. 8 Section heading

  Symbolic Reasoning

• p. 30 Appendix E.3

  created synthetically, as described in Section 4.

• p. 30 E.3

  created synthetically, as described in Section 4

Judge's note: Page 30 reads "Section 4". The heading on page 8 is "5 Symbolic Reasoning", and the introduction says "commonsense reasoning (Section 4), and symbolic reasoning (Section 5)". The pointer should be to Section 5.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 23 Table 7 caption

  Ablation and robustness results for four datasets in commonsense and symbolic reasoning.

• p. 23 Table 7 column headers

  Date Sports SayCan Concat Coin

• p. 23 Table 7 column headers

  SayCan

• p. 23 Table 7 caption

  results for four datasets in commonsense and symbolic reasoning

Judge's note: Read Table 7 on page 23: three commonsense columns (Date, Sports, SayCan) and two symbolic (Concat, Coin). The caption itself goes on to discuss SayCan as part of the table, so no four-dataset reading excludes it. The 'eight datasets in arithmetic, commonsense, and symbolic reasoning (Table 6 and Table 7)' sentence on page 17 was confirmed in the extracted text only; Table 6 has four columns, so the two tables hold nine. Both counts are consistent with SayCan having been added without the counts being updated, but the paper does not say so.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 27 Appendix D.2, 'Calculator error only' bullet

  as an external calculator, as shown in Table 2

• p. 21 Table 2 header

  standard CoT

• p. 21 Table 2 caption

  Table 2: Standard prompting versus chain of thought prompting on five arithmetic reasoning bench-

• p. 27 Appendix D.2

  as shown in Table 2

Judge's note: Table 2 on page 21 has "standard" and "CoT" columns for five benchmarks and no calculator rows or columns. The "+ ext. calc" rows are in Table 1 (page 20 crop). The pointer should be to Table 1.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 30 Appendix E.1

  tasks in Table 20–Table 27 in Appendix G

• p. 41 Table 28 caption

  Table 28: Few-shot exemplars for full chain of thought prompt for SayCan robot planning tasks.

• p. 30 Appendix E.1

  all tasks in Table 20–Table 27 in Appendix G

• p. 41 Table 28

  Few-shot exemplars for full chain of thought prompt for SayCan

Judge's note: Page 30 gives the range as Table 20 to Table 27. Table 28 (page 41 crop) is captioned "Few-shot exemplars for full chain of thought prompt for SayCan robot planning tasks", and Table 19 refers to "the scene as defined in Table 28". SayCan is one of the evaluated tasks, so "all tasks" is not covered by Tables 20 to 27. The range was probably not updated when SayCan was added (the version notes on the same page say it was added in V3).

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 38 Table 24, third exemplar

  only natural habitat is in the forest. So the answer is (b).

• p. 38 Table 24, third exemplar, answer choices

  (a) pretty flowers (b) hen house (c) natural habitat (d) storybook

Judge's note: Viewed the full page image of PDF page 38. As printed, (b) is hen house and (c) is natural habitat, so the answer letter contradicts the exemplar's own chain of thought. The table caption only says newlines between choices are omitted, which does not explain the mismatch. Rated S2 because the reasoning is right and only the letter label is wrong. Whether the prompt actually used in the experiments contained the same error is something only the authors can say.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 1 Abstract

  Experiments on three large language models show

• p. 4 Section 3.1, Language models

  We evaluate five large language models.

• p. 30 Appendix E.2

  For all three language models we evaluated

• p. 1 Abstract

  Experiments on three large language models

Judge's note: Read the abstract on page 1, Section 3.1 on page 4 and E.2 on page 30. Section 3.1 lists five named models, and Tables 1 and 2 report all five. The version note on page 30 says "V4 -> V5. Added Codex and UL2 results", so the abstract and E.2 (and A.2's "all three models (LaMDA, GPT-3, and PaLM)") were left at the old count. No statement in the paper defines "three" as model families. The results themselves are unaffected, so I rated it as wording rather than substantive.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 4 Section 3.2, first takeaway

  only yields performance gains when used with models of

• p. 21 Table 2, PaLM 62B and LaMDA 68B rows, GSM8K

  9.6 29.9

Only the authors can say what is right here.

Judge's note: Read Table 2 on page 21. UL2 20B: 4.1 to 4.4, 10.1 to 12.5, 16.0 to 16.9, 20.5 to 23.6, 16.6 to 19.1. PaLM 8B improves on SVAMP, ASDiv, AQuA and MAWPS (not GSM8K, 4.9 to 4.1). LaMDA 68B improves on GSM8K, SVAMP, ASDiv and MAWPS. Appendix A.1 gives a different threshold ("hurts performance for most models smaller than 10B parameters"), and the Table 2 caption says only "a model of sufficient scale". The tilde makes ~100B approximate, but "only" is contradicted at 8B and 20B. The text overstates the data.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 7 Section 4, Prompts

  For SayCan, we use six examples from the training set

• p. 41 Table 28, last exemplar

  Human: Bring me a drink.

• p. 7 Section 4 Prompts

  we use six examples from the training set

• p. 7 Section 4 Prompts

  we selected the first ten examples as exemplars

Only the authors can say what is right here.

Judge's note: Viewed page 7 and page 41. Counted the Human: entries in Table 28: chips, grapefruit drink, snacks, not a fruit, rice chips and tea, redbull, "Bring me a drink" = 7. The text does not say that one exemplar was added from outside the training set. Only the authors can say whether six or seven were used. The BIG-bench 'first ten' count raised in the same cluster is handled under C018.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 7 Section 4, Prompts

  the first ten examples as exemplars in the evaluation set as few-shot exemplars

• p. 39 Table 26 caption

  Few-shot exemplars for full chain of thought prompt for Date Understanding.

• p. 40 Table 27 caption

  Few-shot exemplars for full chain of thought prompt for Sports Understanding.

• p. 7 Section 4 Prompts

  we selected the first ten examples as exemplars

Only the authors can say what is right here.

Judge's note: Viewed pages 7, 39 and 40. Counted Q/A pairs: Table 26 has 6, Table 27 has 8. Both captions say "Few-shot exemplars for full chain of thought prompt", and nothing in the text says some of the ten were dropped. The sentence also repeats itself ("as exemplars ... as few-shot exemplars"). Cosmetic points in the same cluster are real on the page image: Table 26 has "Feburary" and both "16-year-old birthday" and "16-year old birthday" (S4).

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 7 Section 4, Results paragraph

  chain-of-thought prompting led to further gains, with improvements appear-

• p. 22 Table 4, PaLM rows, CSQA and Sports

  19.8 24.9

• p. 7 Section 4 Results

  with improvements appear-

• p. 22 Table 4, PaLM rows

  65.4 68.1

Only the authors can say what is right here.

Judge's note: Viewed page 7 and Table 4 on page 22 and recomputed the PaLM gains. CSQA: +5.1 at 8B, +2.7 at 62B, +1.8 at 540B. Sports: +20.1, +21.5, +14.9. StrategyQA: -2.1, +5.0, +9.2. Date: +0.2, +14.9, +16.3. SayCan: +5.8, +4.2, +10.9. The claim holds for three of five tasks. The same paragraph does concede "gain was minimal on CSQA", and "appearing to be" hedges, so this is an overstatement in wording, not a wrong number.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 8 Section 5, Results

  symbols for these three tasks only arises at the scale of 100B model parameters

• p. 8 Section 5, Tasks

  We use the following two toy tasks.

• p. 8 Section 5, Results paragraph

  only arises at the scale of 100B model parameters

• p. 22 Table 5, PaLM 62B row, in-domain columns

  6.8 85.0

Judge's note: Viewed page 8 and Table 5 on page 22. Section 5 defines two tasks (last letter concatenation, coin flip); the Figure 8 caption says "two symbolic reasoning tasks" and the Table 5 caption says "two symbolic manipulation tasks". No third symbolic task appears anywhere, so "three" should be "two". The second point raised is a weaker wording issue: Table 5 shows PaLM 62B with chain of thought at 85.0 (letter concatenation, in domain) and 96.8 (coin flip, in domain), and LaMDA 68B at 52.0 and 83.2, so "only arises at the scale of 100B" is loose. Section 3.2 uses "~100B" in the same approximate way, so I do not confirm that part as an error on its own.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 17 Appendix A.2, 'Different annotators'

  for all three annotators on eight

• p. 23 Table 6 and Table 7 column headers

  GSM8K SVAMP ASDiv MAWPS

• p. 17 Appendix A.2, 'Different annotators'

  of thought performed better than the baseline by a large margin for all three annotators on eight

• p. 23 Table 7, different annotator (C), SayCan

  85.0 ±1.8

Judge's note: Viewed page 17 and both tables on page 23. Table 6 columns: GSM8K, SVAMP, ASDiv, MAWPS. Table 7 columns: Date, Sports, SayCan, Concat, Coin. Annotator B and C rows are filled for all nine. The Table 7 caption says "four datasets in commonsense and symbolic reasoning" and then describes SayCan as "the exception" run on PaLM, which suggests SayCan was added later and neither count was updated (4 + 4 = 8). Nothing in the text says SayCan is left out of the count. The side point about "large margin" (annotator C: SayCan 85.0 against 80.8, Date 25.5 against 21.5) is a matter of judgement and is not confirmed as an error.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 18 Appendix A.2, 'Different language models'

  for all datasets except CSQA and StrategyQA for GPT-3 (Table 1, Table 4,

• p. 20 Table 1, LaMDA 137B, AQuA

  20.6 (-4.9)

• p. 18 Appendix A.2, 'Different language models'

  for all datasets except CSQA and StrategyQA for GPT-3

Judge's note: Read the A.2 bullet on page 18 and Table 1 on page 20. LaMDA 137B AQuA is 25.5 standard and 20.6 with chain of thought, and the table itself prints the negative difference. Table 2 repeats 25.5 / 20.6. LaMDA 137B is the largest LaMDA model, so restricting the claim to the largest model per family does not remove the exception. Table 3 also shows drops on AddSub for GPT-3 175B (83.3 to 81.3) and PaLM 540B (93.9 to 91.9), but those are MAWPS subsets and Table 3 is not cited for the claim. No qualifying statement was found elsewhere in the text. Rated S3 because the sentence overstates the tables; the table values themselves are consistent.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 19 Appendix A.3

  PaLM 540B already achieves performance of 90% or higher

• p. 21 Table 3, PaLM 540B, SingleEq

  86.5 92.3

• p. 19 Appendix A.3

  already achieves performance of 90% or higher

Only the authors can say what is right here.

Judge's note: Read A.3 on page 19 and Table 3 on page 21. PaLM 540B standard / chain of thought: SingleOp 94.1 / 94.1, SingleEq 86.5 / 92.3, AddSub 93.9 / 91.9. 'Already' and the headroom argument refer to performance before chain of thought, and on that reading SingleEq is below 90 and gains 5.8 points, which is not small next to 0.0 and -2.0 on the other two. If the 90% is meant for chain-of-thought results, all three qualify, but then the sentence does not support the headroom argument. The authors would have to say which was meant, hence author_query.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 23 Table 6 caption

  “Equation only” performs in between standard prompting

• p. 23 Table 6, GSM8K column

  5.4 ±0.2

• p. 23 Table 6 caption

  “Equation only” performs in between standard prompting and chain of thought prompting

Judge's note: Read Table 6 on page 23. Equation only against standard and chain of thought: GSM8K 5.4 vs 6.5 and 14.3 (below standard); SVAMP 35.1 between 29.5 and 36.7; ASDiv 45.9 between 40.1 and 46.6; MAWPS 50.1 between 43.2 and 57.9. The caption is unqualified. Appendix A.4 on page 19 does say that for GSM8K 'using the equation only did not improve performance substantially', which acknowledges the exception elsewhere but does not make the caption accurate, so I did not reject as explained elsewhere.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send