Self-Consistency Improves Chain of Thought Reasoning in Language Models
2022 · paper · AI and machine learning
Proposes self-consistency, a decoding strategy that samples diverse reasoning paths from a language model and picks the most consistent final answer. It improves chain-of-thought results on reasoning benchmarks, including GSM8K by 17.9% and SVAMP by 11.0%.
Checked: PDF from arxiv.org, fetched 4 Oct 2026. Page numbers are those of that PDF.
5 findings
Substance Table 3, UL2-20B, ARC-c: self-consistency is printed as "49.5 (+6.8)" over CoT-prompting "42.9", but 49.5 - 42.9 = 6.6.
- p. 5 Table 3, UL2-20B, ARC-c column, Self-consistency row
49.5 (+6.8)
- p. 5 Table 3, UL2-20B, ARC-c column, CoT-prompting row
42.9
Only the authors can say what is right here.
Judge's note: Page 5 image: the cells read 42.9 and 49.5 (+6.8). The difference of the printed values is 6.6. A gap of 0.2 cannot come from rounding one-decimal values (maximum 0.1). I checked the other gains in Table 3 and they match their subtraction. One of the three numbers is wrong, and only the authors can say which. +6.8 is also the UL2 SVAMP gain in Table 2, so a copy slip is possible. No explanation in the text.
Second check: another model agreed · checked by hand
Report a problem with this finding
Wording Section 3.1 says "we use the same prompts as in Wei et al. (2022): for all arithmetic reasoning tasks we use the same set of 8 manually written exemplars" and Table 17 is captioned "Few-shot exemplars for all arithmetic reasoning tasks, from Wei et al. (2022)", but Appendix A.3 gives AQUA-RAT (an arithmetic task in Table 2) its own 4-exemplar prompt (Table 14), "where we manually composed the example chain-of-thought in this paper".
- p. 4 Section 3.1, Language models and prompts
for all arithmetic reasoning tasks we use the same set of 8 manually written exemplars
- p. 19 Appendix A.3
two newly-introduced datasets, AQUA-RAT (Ling et al., 2017) and AI2 Reasoning Challenge (ARC) (Clark et al., 2018), where we manually composed the example chain-of-thought in this paper
- p. 21 Table 17 caption
Few-shot exemplars for all arithmetic reasoning tasks, from Wei et al. (2022).
- p. 19 Appendix A.3, first paragraph and Table 14
two newly-introduced datasets, AQUA-RAT
Judge's note: Page 4 image: the Section 3.1 sentence is as quoted, and AQUA-RAT is listed under arithmetic reasoning. Page 19 image: A.3 says the AQUA-RAT and ARC chains of thought were composed in this paper, and Table 14 has 4 multiple-choice exemplars. Page 21 image: Table 17 has 8 exemplars and the caption says "all arithmetic reasoning tasks". No sentence anywhere exempts AQuA from "all arithmetic reasoning tasks". The same applies to ARC against "the same prompts as in Wei et al.". This is a methods description that is wrong for one task, not a data error.
Second check: another model agreed
Report a problem with this finding
Wording Section 3.2: "we see +3%-6% absolute accuracy improvement over UL2-20B but +9%-23% for LaMDA-137B and GPT-3" does not match Table 2: GPT-3 code-davinci-001 gains include +6.7 (AQuA) and +8.8 (GSM8K), code-davinci-002 gains include +2.2, +3.8 and +7.6, and the UL2-20B gains reach +6.6 and +6.8.
- p. 5 Section 3.2, Arithmetic Reasoning paragraph
+3%-6% absolute accuracy improvement over UL2-20B but +9%-23% for LaMDA-
- p. 5 Table 2, GPT-3 Code-davinci-001, AQuA
25.6 (+6.7)
- p. 5 Table 2, GPT-3 Code-davinci-002, AddSub
91.6 (+2.2)
- p. 5 Section 3.2, Arithmetic Reasoning paragraph
+9%-23% for LaMDA- 137B and GPT-3
Judge's note: Page 5 image, Table 2 gains. UL2-20B: 6.6, 4.3, 4.6, 3.3, 6.8, 3.2 (range 3.2 to 6.8, so the upper bound of 6 is truncated). LaMDA-137B: 10.6, 23.9, 9.2, 9.1, 14.4, 10.6 (fits 9 to 23 only if truncated). code-davinci-001: 10.6, 23.2, 9.2, 6.7, 14.7, 8.8. code-davinci-002: 2.2, 3.8, 7.6, 12.2, 11.0, 17.9. The stated lower bound of 9% fails for both GPT-3 engines. The next sentence does treat GPT-3 separately as an already-accurate model, but the quoted sentence names GPT-3 for the 9-23 range without qualification. The direction of the claim (larger gains for larger models) still roughly holds, so this is a wording defect.
Second check: another model agreed
Report a problem with this finding
Wording Appendix A.1.4 says "we use 40 sets of prompts, or 40 prompt permutations" and Table 11 reports "Different sets of prompts (x40)", while Section 3.5 says "we manually write 3 different sets of prompts" and A.1.2 says "We manually wrote 3 different sets"; the source of 40 prompt sets is never stated.
- p. 17 A.1.4
we use 40 sets of prompts, or 40 prompt
- p. 7 Section 3.4, ensembles
we manually write 3 different sets of prompts
- p. 17 A.1.4 text
we use 40 sets of prompts, or 40 prompt permutations
- p. 18 Table 11 row 1
Different sets of prompts (x40)
Only the authors can say what is right here.
Judge's note: Viewed pages 7, 16, 17 and 18. Section 3.5 and Table 7 use an ensemble of 3 prompt sets; A.1.2 and Table 9 also show exactly 3 prompt sets. A.1.4 and Table 11 use 40 sets with no explanation of how they were obtained (more written, sampled exemplars, or something else). I searched the text for 'sets of prompts' and 'prompt set' and found no explanation. This is not a contradiction, since Table 11 is a separate experiment, but it is an unexplained methods gap that only the authors can resolve.
Second check: another model agreed
Report a problem with this finding
Wording Reproducibility Statement: "we provide the exact input prompts for all tasks in Appendix A.3", but A.3 gives no prompts for the symbolic tasks (Letter, Coinflip) and for two others says only "The prompts for CommonsenseQA and StrategyQA are the same as used in Wei et al. (2022)."
- p. 10 Reproducibility Statement
exact input prompts for all tasks in Appendix A.3
- p. 20 Appendix A.3, after Table 16
The prompts for CommonsenseQA and StrategyQA are the same as used in Wei et al. (2022).
Judge's note: Page 19 and 20 images plus the A.3 text (tables 14 to 21): prompts are given for AQUA-RAT, ARC, HotpotQA, the arithmetic tasks, ANLI, e-SNLI, RTE and BoolQ. CommonsenseQA and StrategyQA are covered only by reference to Wei et al., and a text search finds no mention of the letter-concatenation or coin-flip prompts anywhere in the appendix. The Reproducibility Statement wording was read in the text layer (extract/main.txt), not on the page 10 image. "All tasks in Appendix A.3" is therefore an overstatement.
Second check: another model agreed
Report a problem with this finding
17 minor (typos and the like; most checked by the judge only)
Typo Figure 1, second sampled reasoning path (red box): "This means she she sells the remainder for $2 * (16 - 4 - 3) = $26 per day." The word "she" is doubled.
- p. 2 Figure 1, second sampled reasoning path (red box)
This means she she sells the
- p. 2 Figure 1, sampled output 4
This means she she sells the remainder
- p. 2
This means she she sells the remainder for $2 * (16 - 4 - 3)
- p. 2
9 eggs * $2 = $18. This means she she sells the remainder
Judge's note: Read on the page 2 image: the red box reads "This means she she sells the remainder". The wrong arithmetic in that box is deliberate (it illustrates an incorrect path), but the doubled word serves no purpose and the other boxes are clean. The figure is not stated to be verbatim model output.
Second check: code check
Report a problem with this finding
Typo Table 2 source letters run a, b, c, d, e and then g ("35e / 55g"; caption "e: GPT-3 175B finetuned with 7.5k examples (Cobbe et al., 2021), g: GPT-3 175B finetuned plus an additional 175B verifier"); there is no f.
- p. 5 Table 2, Previous SoTA row, GSM8K
35e / 55g
- p. 5 Table 2 caption
g: GPT-3 175B finetuned plus an additional 175B verifier
- p. 5 Table 2 caption and Previous SoTA row
e: GPT-3 175B finetuned with 7.5k examples (Cobbe et al., 2021), g: GPT-3 175B finetuned plus an additional 175B verifier
Judge's note: Page 5 image: the Previous SoTA row uses superscripts a, b, c, d, e and g, and the caption defines a, b, c, d, e and g. The table and caption agree with each other, so nothing is mislabelled. The skipped letter is cosmetic.
Second check: none (judge only)
Report a problem with this finding
Typo Section 3.5: "The gain is relatively lower for smaller models due to certain abilities (e.g., arithmetic) only emerge when the model reaches a sufficient scale" is ungrammatical ("due to ... only emerge"); it needs "because certain abilities ... only emerge" or "only emerging".
- p. 8 Section 3.5, first paragraph
certain abilities (e.g., arithmetic) only emerge when the model reaches a sufficient scale
Judge's note: Page 8 image: the sentence reads as quoted. This is a grammar slip, and the meaning is clear.
Second check: none (judge only)
Report a problem with this finding
Typo Figure 8 labels the x-axis "#Sampled Reasoning Chains" in the six arithmetic panels and "#Sampled Reasoning Paths" in the four bottom panels; the AQuA legend is clipped to "Greedy Decode (Single-path".
- p. 17 Figure 8, top two rows (AddSub, ASDiv, AQuA, MultiArith, SVAMP, GSM8K)
#Sampled Reasoning Chains
- p. 17 Figure 8, bottom row (Commonsense QA, Strategy QA, ARC)
#Sampled Reasoning Paths
- p. 17 Figure 8 legend
Greedy Decode (Single-path
Judge's note: Viewed page 17 and both crops. Top two rows (AddSub, ASDiv, AQuA, MultiArith, SVAMP, GSM8K) read 'Chains'; the bottom row (Commonsense QA, Strategy QA, ARC Easy, ARC Challenge) reads 'Paths'. Figures 2, 3, 6 and 7 and the Figure 8 caption all use 'paths'. The AQuA legend is cut off at the right edge, losing the closing parenthesis; the GSM8K and ARC legends are complete. Same quantity, cosmetic only.
Second check: none (judge only)
Report a problem with this finding
Typo Appendix A.2: "For GPT-3 models the experiments are done though the public API"; 'though' should be 'through'.
- p. 18 Appendix A.2, first paragraph
the experiments are done though the public
- p. 18 Appendix A.2, first paragraph
the experiments are done though the public API
Judge's note: Read on the page 18 image and the C020 crop, which covers the same paragraph: the word printed is 'though'. Plain misspelling.
Second check: none (judge only)
Report a problem with this finding
Typo Appendix A.2: "until the start of the next “Q:" to parse the final answers" opens with a curly quotation mark and closes with a straight one.
- p. 18 A.2
start of the next “Q:" to parse
Judge's note: The crop shows a curly opening double quote before Q: and a straight double quote after the colon. It is in the rendered page, not an extraction effect. Cosmetic.
Second check: none (judge only)
Report a problem with this finding
Typo Table 12, CommonsenseQA example: "(a) niagra falls"; the usual spelling is 'niagara'.
- p. 18 Table 12, CommonsenseQA question
(a) niagra falls
Judge's note: The crop shows 'niagra falls' as an answer option. The question is quoted from the benchmark; whether the misspelling comes from the CommonsenseQA item cannot be checked from these files. If it does, reproducing it verbatim is legitimate, so this is an observation, not a correction.
Second check: none (judge only)
Report a problem with this finding
Typo Table 13, AQuA row, Sampled Path 1: "So the answer is (d)" has no final full stop, unlike the other cells.
- p. 19 Table 13, AQuA, Sampled Path 1
So the answer is (d)
Judge's note: The crop shows '(d)' with no period at the end of Sampled Path 1; Greedy Decode ends '(c).' and Sampled Path 2 ends '(d).'. This is model output, probably reproduced verbatim, so either form could be legitimate.
Second check: none (judge only)
Report a problem with this finding
Typo Table 14, second exemplar: "8a + 5b = 22,then find the value of a" has no space after the comma.
- p. 19 Table 14, second exemplar
8a + 5b = 22,then find the value of a
Judge's note: The crop shows '22,then' with no space. The question text is an AQUA-RAT item used as the prompt, so the spacing may be the dataset's own and may have been kept on purpose to show the exact prompt.
Second check: none (judge only)
Report a problem with this finding
Typo Table 14, third exemplar: "A person is traveling at 20 km/hr and reached his destiny in 2.5 hr"; 'destiny' is used where 'destination' is meant.
- p. 19 Table 14, third exemplar
reached his destiny in 2.5 hr
Judge's note: The crop shows 'destiny'. The question is an AQUA-RAT item quoted as the prompt; the wording is probably the dataset's and is shown as the exact prompt used. The word is wrong as printed, but a fix would change the reported prompt.
Second check: none (judge only)
Report a problem with this finding
Typo Table 15, third exemplar answer: "When water is freezed, the particles are arranged in a fixed position"; 'freezed' should be 'frozen'.
- p. 20 Table 15, third exemplar
When water is freezed
Only the authors can say what is right here.
Judge's note: The crop shows 'freezed'. Section A.3 says the authors manually composed the chain-of-thought answers for ARC, so this is the authors' own wording, not dataset text. It is a grammatical error, but the table reports the exact prompt given to the models, so the authors must decide whether to correct it or mark it as verbatim.
Second check: none (judge only)
Report a problem with this finding
Typo Table 17: weekday names are lowercase in the questions and one answer ("from monday to thursday", "On tuesday, he lost 23 golf balls. On wednesday") but capitalised in another answer ("on Tuesday", "On Wednesday").
- p. 21 Table 17, 6th and 7th exemplars
from monday to thursday
- p. 21 Table 17, 7th exemplar
On tuesday, he lost 23 golf balls
Judge's note: The crop shows lowercase monday, thursday, tuesday and wednesday in the questions and in the computers answer, and capitalised Tuesday and Wednesday in the golf-ball answer. The paper says these are the prompts from Wei et al. (2022), so the inconsistency is probably inherited and reproduced verbatim.
Second check: none (judge only)
Report a problem with this finding
Typo Table 18, second exemplar premise: "One of our member will carry out your instructions minutely." ('member' should be plural).
- p. 22 Table 18, second exemplar
One of our member will carry out your instructions minutely.
Judge's note: The crop shows 'One of our member'. This is a premise sentence taken from the NLI dataset and used as a prompt exemplar, so it is probably the dataset's wording reproduced verbatim. Ungrammatical as printed, but not evidently the authors' error.
Second check: none (judge only)
Report a problem with this finding
Typo Table 19, fourth exemplar: "does not imply their parents or anyone is anyone behind it" (garbled or duplicated 'anyone').
- p. 23 Table 19, fourth exemplar
does not imply their parents or anyone is anyone behind it
Judge's note: The crop and the page 23 image show the phrase as quoted. The Table 19 caption states 'the rationales are crowd-sourced annotations from the original dataset', so the paper itself says this text is taken from e-SNLI. Garbled as printed, but disclosed as verbatim dataset text.
Second check: none (judge only)
Report a problem with this finding
Typo Table 19, last exemplar: "A:The children must be present to see them smiling and waving." has no space after "A:", unlike the other exemplars.
- p. 23 Table 19, last exemplar
A:The children must be present
Judge's note: The crop and the page 23 image show 'A:The' with no space; the other five exemplars in the table have 'A: ' with a space. It is in the rendered page, not an extraction artefact. It may be a typesetting slip or the exact prompt string; cosmetic either way.
Second check: none (judge only)
Report a problem with this finding
Typo Table 19, fifth exemplar: ""The kids are frowning." is true? OPTIONS:" has OPTIONS: on the question line; the other exemplars put it on its own line.
- p. 23 Table 19, fifth exemplar
"The kids are frowning." is true? OPTIONS:
Judge's note: The crop and the page 23 image show 'is true? OPTIONS:' on one line although the line is short enough that a line break was simply left out; the other five exemplars start OPTIONS: on a new line. Formatting inconsistency only.
Second check: none (judge only)
Report a problem with this finding
Typo Table 20 (PDF p. 24), second RTE exemplar: "Catholic Church." is true?’" carries a stray closing single quotation mark after the question mark; it should read "is true?" as in the other exemplars.
- p. 24 Table 20, second RTE exemplar
Catholic Church." is true?’
- p. 24 Table 20, second exemplar
is the new leader of the Roman Catholic Church." is true?’
Judge's note: Viewed the crop of Table 20: the Benedict XVI exemplar's question ends with "is true?’" with a visible right single quote after the question mark. The other three RTE exemplars in the table (Weapons of Mass Destruction, Hutchinson, Herceptin) and all the NLI exemplars in the preceding tables end with "is true?" and no trailing mark, per the image and the text layer. Nothing opens a matching single quote, so it is a stray character. Cosmetic only; it may be copied verbatim from the prompt actually used, but it is still a typographic defect in the printed table.
Second check: none (judge only)