Training Compute-Optimal Large Language Models

• p. 1 Abstract

  67.5% on the MMLU benchmark

• p. 11 Table 6 and Section 4.2.2

  with an average accuracy of 67.6%

• p. 1 Abstract

  67.5% on the MMLU benchmark, greater than a 7% improvement over Gopher

• p. 11 Table 6 / Section 4.2.2

  with an average accuracy of 67.6% (improving upon Gopher by 7.6%)

Judge's note: I checked the abstract (p1) and Table 6 and Sec. 4.2.2 (p11) on the page images. Both report the average 5-shot accuracy over 57 MMLU tasks. Table 6 gives Gopher 60.0%, and 67.6 − 60.0 = 7.6, which matches the text. I found nothing in the paper that defines a different average for the abstract. The gap is only 0.1 point, so I rate it minor, but it is the same quantity printed two ways.

Second check: another model agreed

• p. 19 References, J. W. Rae et al. 2020

  K. Choromanski, V. Likhosherstov, D. Dohan, X. Song, A. Gane, T. Sarlos, et al. Compressive transformers for long-range sequence modelling.

Only the authors can say what is right here.

Judge's note: Rests on outside knowledge: that 'Compressive Transformers for Long-Range Sequence Modelling' is by Rae, Potapenko, Jayakumar, Hillier and Lillicrap (ICLR 2020), and that Choromanski, Likhosherstov, Dohan, Song, Gane and Sarlos are authors of the Performer paper. On p.19 the entry reads exactly as quoted. This cannot be checked against the files.

Second check: another model agreed

• p. 1 Introduction

  increase 5.5× while the number of training tokens should only increase 1.8×

• p. 8 Table 2, Kaplan row

  0.73

Judge's note: Table 2 (p8 image) lists Kaplan a=0.73 and b=0.27. From these, 10^0.73 ≈ 5.37 and 10^0.27 ≈ 1.86, i.e. about 5.4× and 1.9×. The printed 5.5× and 1.8× correspond to exponents of about 0.74 and 0.26. The difference is small and may come from unrounded exponents, so I treat it as an observation. The calculation uses powers, so it is not listed in the arithmetic field.

Second check: another model agreed

• p. 1 Abstract

  on 5 to 500 billion tokens

• p. 2 Introduction

  trained on 5B to over 400B tokens

Judge's note: I checked p1 and p2 on the images. 'Over 400B' is compatible with 500B, so the two statements do not contradict each other, but the paper gives the upper bound in two different forms.

Second check: another model agreed

• p. 1 Abstract

  ranging from 70 million to over 16 billion

• p. 2 Introduction

  ranging from under 70M

• p. 36 Table A9 first row

  44

• p. 1 Abstract

  ranging from 70 million to over 16 billion parameters

Judge's note: Table A9 on the p36 image lists 44, 57 and 74 million-parameter models. The Introduction (p2) correctly says 'under 70M'. The abstract's lower bound of 70 million describes the same set of models less accurately.

Second check: another model agreed

• p. 1 Abstract

  Chinchilla uniformly and significantly outperforms Gopher (280B), GPT-3 (175B), Jurassic-1 (178B), and Megatron-Turing NLG (530B)

• p. 13 Table 8 PIQA

  82.0%

• p. 30 Table A5 dm_mathematics/ubuntu_irc

  1.037

• p. 1 Abstract

  Chinchilla uniformly and significantly outperforms Gopher (280B)

Judge's note: In Table 8 (p13), PIQA is MT-NLG 82.0% vs Chinchilla 81.8%, with Gopher tied at 81.8%. In Table A5 (p30), dm_mathematics has the third column's 1.037 in bold, better than Chinchilla's 1.111. Sec. 4.2.2 (p11) also mentions four MMLU tasks where Chinchilla does not improve. 'Uniformly' is therefore stronger than the data support; this is a wording issue.

Second check: another model agreed

• p. 12 Section 4.2.5

  We find that Chinchilla outperforms both Gopher and GPT-3 on all tasks

• p. 13 Table 8, PIQA row

  81.8%

• p. 1 Abstract

  Chinchilla uniformly and significantly outperforms Gopher (280B), GPT-3 (175B), Jurassic-1 (178B), and Megatron-Turing NLG (530B)

• p. 13 Table 8

  PIQA 81.8% 81.8% 81.0% 82.0%

Judge's note: Checked the p.12 text and Table 8 on the p.13 image. The PIQA tie means the text should say "matches or outperforms".

Second check: another model agreed

• p. 8

  0.46 (0.454, 0.455)

Only the authors can say what is right here.

Judge's note: Checked Table 2 (p.8 image), Eq. (4) (p.7 image) and Eq. (10) (p.25 image). The printed point estimate does not match its own interval or the fitted α and β. This is a query for the authors.

Second check: another model agreed, code check

• p. 16 Section 5

  Chinchilla does suffer from bias and toxicity but interestingly it seems less affected than Gopher

• p. 15 Sample toxicity paragraph

  Gopher is 0.081 (0.064), compared to 0.087 (0.066) for Chinchilla

• p. 16 Section 5, second paragraph

  Chinchilla does suffer from bias and toxicity but interestingly it seems less affected than Gopher.

• p. 16 Discussion

  it seems less affected than Gopher

Only the authors can say what is right here.

Judge's note: I checked p.15. The toxicity summary statistics are all marginally higher for Chinchilla, and the text says 'the difference between the models is negligible'. Table 10 (Winogender) favours Chinchilla, so 'less affected' fits bias but not toxicity. The sentence on p.16 names both bias and toxicity. The authors should say which they meant.

Second check: another model agreed

• p. 22 Appendix A

  both the MassiveWeb and Wikipedia subsets are both used

• p. 15 Section 5, third paragraph

  the training runs for our analysis have all been trained on less than an epoch of data

• p. 22 Appendix A / Table A1

  both the MassiveWeb and Wikipedia subsets are both used for more than one epoch

Only the authors can say what is right here.

Judge's note: I checked p.15 and p.22. Chinchilla's own 1.4T-token run repeats Wikipedia 3.4 times and MassiveWeb 1.24 times. Scaling runs of up to 500B tokens would also see Wikipedia more than once. 'Less than an epoch' may refer to the corpus as a whole, but the paper does not say so. Separately, the Appendix A sentence uses 'both ... both' (S4 wording).

Second check: another model agreed

• p. 24 Equation (6)

  L(f) ≜ 𝔼[log f(x)_y]

• p. 25 Equation (8)

  L̂_D(f) ≜ Ê_D[log f(x)_y]

Only the authors can say what is right here.

Judge's note: The page images of p.24 and p.25 show no minus sign in either equation. Cross-entropy is −E[log f(x)_y]. Minimising E[log f(x)_y] as printed would drive the predicted probability of the observed token towards zero, so the argmin does not yield the Bayes classifier as stated. My reworking shows the definition is wrong by a sign. The rest of the decomposition (Eq. 9, the fitted form (10)) does not depend on this, so the stated result stands.

Second check: another model agreed

• p. 26 Table A3, Approach 3, 67 Billion row

  1.71e+24

Only the authors can say what is right here.

Judge's note: I recomputed every row on p.26. 400M, 1B, 10B, 520B, 1T and 10T agree with 6ND to within about 0.5%; the 67B and 280B rows do not. The parameter or token figure may be rounded differently (1.71e24/(6*4.1e12) ≈ 69.5B), but only the authors can say which number is intended.

Second check: another model agreed

• p. 26 Table A3, Approach 3, 280 Billion row

  3.52e+25

Only the authors can say what is right here.

Judge's note: Checked on p.26. The neighbouring rows (10B, 520B, 1T) match 6ND. The printed FLOPs would imply about 20.95T tokens (or about 292B parameters). Only the authors can say which figure is wrong.

Second check: another model agreed

• p. 34 Appendix J

  we list the model size and configuration of all models used in this study

• p. 29 Appendix G / Figure A7

  For a 417M (blue) and 1.4B model (green)

• p. 27 Appendix D.4

  We train a 4.74 billion parameter and a 2.80 billion

Judge's note: I read Table A9 on page 36. Its sizes run from 44 to 16,183 million. The nearest entries to the models in question are 425, 632/664, 2,980 and 4,516; there is no 417, 680, 2,800 or 4,740 entry, and no 70B entry. The 417M and 680M models are named on page 29, and the 4.74B and 2.80B models on page 27. A different parameter-counting convention (for example, excluding embeddings) could explain some of the gaps, but the paper does not say so. The Table A9 caption narrows the scope only to models "trained as part of this work". This is a minor completeness or wording point.

Second check: another model agreed

• p. 1 Abstract

  4× more more data

• p. 1

  ut with 70B parameters and 4× more more data. Chinchilla un

Judge's note: The duplicated 'more more' is visible on the p1 page image, so it is not an extraction artefact.

Second check: code check

• p. 1 Footer

  © 2023 DeepMind. All rights reserved

Judge's note: The p1 image shows the footer year 2023 next to the arXiv margin stamp '29 Mar 2022'. This is probably a template year, but the copyright year printed is later than the release date.

Second check: none (judge only)

• p. 1 Introduction, para 3

  increase computational budget, they suggests

• p. 1 Introduction, third paragraph

  given a 10× increase computational budget, they suggests

• p. 1 Introduction

  10× increase computational budget

Judge's note: Confirmed on the p1 image: the word 'in' is missing and 'they suggests' is a subject-verb agreement error.

Second check: none (judge only)

• p. 2 Introduction

  while being training on 4 times more

• p. 2 Introduction, last paragraph

  while being training on 4 times more tokens

• p. 2 Introduction

  through its usage for inference an

Judge's note: Confirmed on the p2 image. The separate 'inference an' typo that one reviewer also mentions is handled in C012.

Second check: none (judge only)

• p. 2 Introduction, last paragraph

  usage for inference an

• p. 2 Introduction, last paragraph

  usage for inference an fine-tuning

• p. 2 Introduction, para after Eq. 1

  through its usage for inference an

• p. 2

  The energy cost of a large language model is amortized through its usage for inference an

Judge's note: On the p2 image, 'an' sits at the end of a line and 'fine-tuning' starts the next one. The word itself is truncated, so this is not a line-break artefact.

Second check: none (judge only)

• p. 4 Estimating hyperparameters for large models

  parameters to chose when selecting

Judge's note: Confirmed on the p4 image.

Second check: none (judge only)

• p. 5 Section 3.2

  for each point7. in

Judge's note: Confirmed on the p5 image: the sentence continues in lowercase after the full stop, so the full stop should be a comma.

Second check: none (judge only)

• p. 7 Section 3.4, last line

  Our findings suggests that the current generation of

Judge's note: Seen on the p.7 page image, last line of Section 3.4.

Second check: none (judge only)

• p. 8 Table 3, 10 Billion row

  1/46

Judge's note: Checked Table 3 on the p.8 image. Other rows are consistent with rounding (6.7, 17.2), and 5.76e23/46 = 1.25e22 would not print as 1.23e22. The denominator may simply be truncated, so this is a minor formatting inconsistency.

Second check: none (judge only)

• p. 13 Table 8 caption

  a comparison between Chinchilla, Gopher, and MT-NLG 530B

Judge's note: Checked Table 8 and its caption on the p.13 image. This is minor caption incompleteness.

Second check: none (judge only)

• p. 19 References

  Research, 21(140):1–67, 2020b.

Judge's note: On p.19 the two entries have identical authors, title, journal, volume and pages. The only difference is the URL. Appendix C on p.22 cites 2020b. The duplicate should be merged.

Second check: none (judge only)

• p. 22 Table A1 caption

  In the rightmost column show the

Judge's note: Checked in the Table A1 caption on p.22.

Second check: none (judge only)

• p. 25 Appendix D.2

  Instead, be obtain

Judge's note: Checked on p.25.

Second check: none (judge only)

• p. 28 Table A4 first row

  73M

• p. 36 Table A9

  74

Judge's note: I compared Table A4 (p.28) with Table A9 (p.36). The larger configurations agree (552, 1.1B/1,143, 1.6B/1,593, 6.8B/6,796). The 1M gaps fit truncation in one table and rounding in the other, so either form may be legitimate. The two tables should still agree.

Second check: none (judge only)

• p. 29 Appendix G / Fig A6 caption

  an 680 million parameter model

Judge's note: On page 29 the phrase "an 680 million parameter model" appears in the body text of Appendix G and again in the Figure A6 caption. "680" is read "six hundred eighty", which starts with a consonant sound, so the article should be "a". This is a cosmetic slip.

Second check: none (judge only)