Scaling Laws for Neural Language Models

• p. 4 Eq. (1.3)

  ∼3.1 × 10^8 (PF-days)

• p. 20 Table 5

  = 3.1 × 10^8 PF-days

• p. 15 Figure 13 legend

  L = (Cmin/2.3 · 10^8)^−0.050

• p. 3 Figure 1 legend, compute panel

  L = (Cmin/2.3 · 10^8)^−0.050

Only the authors can say what is right here.

Judge's note: Checked Eq. (1.3) on page 4, Table 5 on page 20, the Figure 1 legend on page 3 and the Figure 13 legend on page 15. Same quantity, same exponent, same units; Section 1.2 points to Figure 1 for these laws. No second fit or explanation is given. The naive-C fit shows the same pattern (Table 5 Cc = 1.6 × 10^7, Figure 13 legend 2.0 · 10^7). Only the authors can say which constant is right.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 17 Eq. (6.7)

  (Cmin/PF-Day)^0.26

• p. 3 Section 1.1, 'Convergence is inefficient'

  D ∼ C^0.27

• p. 20 Table 6

  pD = 0.27

• p. 17 Eq. (6.7)

  (4 × 10^10 tokens) (Cmin/PF-Day)^0.26

Only the authors can say what is right here.

Judge's note: Eq. (6.7) read on the page 17 image; Table 6 on page 20; the summary on page 3. Using N = 1.3·10^9·Cmin^0.73 (Figure 14 legend, Table 6) and one PF-day = 8.64 × 10^19 operations, Eq. (6.7)'s own formula gives a prefactor of about 2.2 × 10^10 tokens and an exponent of 0.27, which agrees with Table 6 and not with the 4 × 10^10 and 0.26 printed in the equation. Cross-check: 2·Be·Se from Table 6 is also about 2.2 × 10^10. The factor 2 for critical-batch training is already in the numerator, so it does not explain the difference. The exponent difference could be rounding of an unrounded fit; the prefactor difference is a factor of two. Authors should say which is intended.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 5 Eq. 1.8 and following text

  αmin C = 1/ (1/αS + 1/αB + 1/αN)

• p. 4 Eq. 1.3

  ∼0.050

Only the authors can say what is right here.

Judge's note: Corrected description. The fitted 0.050 (Eq. 1.3, Table 5) against a predicted value is a legitimate fitted-versus-predicted difference and is not a defect. The defect is that the predicted value itself is 0.054 on page 16 (viewed) and 0.052 in Eq. (B.7) (text layer only, line 1599). With the Table 5 values αS = 0.76, αB = 0.21, αN = 0.076 the formula gives 0.052; with αN = 0.077 it is still 0.052. The authors may have used unrounded inputs, so this is a query.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 15 Figure 13 legend

  L = (Cmin/2.3 108)

• p. 4 Eq. (1.3)

  ∼3.1 × 108 (PF-days)

• p. 20 Table 5

  Cc = 1.6 × 107 PF-days

• p. 20 Table 5

  = 3.1 × 108 PF-days

Only the authors can say what is right here.

Judge's note: Figure 13 legend read on the page 15 image; the Figure 1 compute panel (page 3) also prints 2.3·10^8 with exponent −0.050. Eq. (1.3) on page 4 and Table 5 on page 20 print 3.1 × 10^8 and 1.6 × 10^7. The exponents are the same (0.050, 0.057), so these are the same fits with different scale constants. Section 6.1 and Appendix D.7 give no statement of a different fit that would explain it. Only the authors can say which constants are right.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 20 Table 5, αC row

  Cc = 1.6 × 10^7 PF-days

• p. 15 Figure 13 legend

  L = (C/2.0 · 10^7)^−0.057

• p. 20 Table 5

  Cc = 1.6 × 107 PF-days

Only the authors can say what is right here.

Judge's note: Page 20 image, Table 5: αC = 0.057, Cc = 1.6 × 10^7 PF-days. Page 15 image, Figure 13 legend, orange line: L = (C/2.0·10^7)^−0.057. Same exponent and same quantity (the fixed-batch empirical L(C) fit). I searched the text for 1.6 and 2.0 scale values and for any second fit or population and found no explanation. Only the authors can say which value is right.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 16 Eq. (6.4)

  ≈ 0.054

• p. 21 Eq. (B.7)

  αC = 1/ (1/αS + 1/αB + 1/αN) ≈ 0.052

• p. 21 Appendix B.2, below Eq. B.7

  ≈0.052

Only the authors can say what is right here.

Judge's note: Both equations read on the page images (pages 16 and 21). With αS = 0.76, αB = 0.21, αN = 0.076 (Table 5) the expression is 0.0520; with αN = 0.077 (Table 3) it is 0.0525. Neither rounds to 0.054. Appendix B says its αC is adjusted for the critical batch size, so it is the same quantity. Eq. (6.5)'s exponent 0.71 matches 0.054/0.076, whereas 0.052/0.076 is 0.68, so the 0.054 carries through to Eq. (6.5). The authors should say which inputs were used.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 14 Section 6, first sentence

  the top-right of Figure 1

• p. 10 Section 3.3

  the heavy black line on the left-hand plot in Figure 1

• p. 3 Figure 1

  Compute

• p. 10 Section 3.3

  The result appears as the heavy black line on the left-hand plot in Figure 1

Judge's note: Figure 1 on the page 3 image has panels Compute (left), Dataset Size (middle), Parameters (right) in one row. Page 14 image confirms "top-right of Figure 1"; page 10 image confirms "left-hand plot in Figure 1". The Section 6 pointer does not match the published figure.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 7 Table 1, 'Attention: Project' row, FLOPs column

  2nlayer dattn dembd

• p. 9 Sec 3.3

  (nlayer, nembd) = (36, 1280)

Judge's note: Subscript 'embd' is visible in Table 1 on page 7. Section 2.1 (page 6) defines only dmodel, dff, dattn, nlayer, nheads and nctx. For FLOPs to equal twice the parameters, as in the other rows, it must be dmodel. The related nembd on page 9 is handled in C012.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 9 Section 3.3, second paragraph

  (nlayer, nembd) = (36, 1280)

Judge's note: Visible on page 9. nembd is not defined in Section 2.1; pages 8 and Figure 5 write (nlayer, dmodel). 12 × 36 × 1280² is about 708M, matching the 708M model in Figure 4, so dmodel is meant.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 9 Figure 7, right panel, 'Per-token Test Loss' axis

  6 4 5 3 2

Judge's note: On the page 9 image the evenly spaced tick labels are 6, 4, 5, 3, 2 from top to bottom, so the labelling is not monotonic. The page image is at normal resolution only, but the digits are legible. The text layer does not preserve the order.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 10 Section 3.3, first line under Figure 8

  with the best performance on step S = C/6BS

• p. 10 Section 3.3, top of page

  with the best performance on step S =

• p. 9 Section 3.3

  C = 6NBS

Judge's note: Page 10 image shows the fraction as C over 6BS. Page 9 image shows "C = 6NBS, where B is the batch size, S is the number of parameter updates". Solving for S gives C/(6NB). No alternative definition of S found. Evident typo in the denominator; rated S2 because the slip is self-evident and no number is affected.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 12 Section 4.2, after Eq. (4.2)

  of N and D, as shown in Figure 16

• p. 11 Figure 9 caption

  The extent of overfitting depends predominantly on the ratio

• p. 12 Section 4.2, after Eq. (4.2)

  δL depends only a specific combination of N and D, as shown in Figure 16

• p. 11 Figure 9 caption

  Right: The extent of overfitting depends predominantly on the ratio

Judge's note: Page 12 image confirms the sentence. Figure 9 right panel (page 11) is titled Overfitting, plots L/L(D=∞)−1 against N^(αN/αD)/D, and its caption cites Eq. (4.3). Figure 16 (page 23) plots S_stop against Sc×[L(N,D)−L(N,∞)]^(−1/αS) on the left and train/test loss against step for 300M-parameter models on the right; neither panel shows δL against a combination of N and D. The pointer should be to Figure 9.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 18 Section 7, second paragraph

  Very recent work [RRBS19b] studies scaling with both dataset size and model size

• p. 18 Footnote 8

  After this work was completed, [RRBS19a] also appeared

• p. 29 References

  A constructive prediction of the generalization error across scales, 2019, 1909.12673

Judge's note: Read page 29: the two entries are identical apart from the 'arXiv:' prefix. Page 18 cites [RRBS19b] in the body ('Very recent work [RRBS19b] studies scaling...') and [RRBS19a] in footnote 8 ('After this work was completed, [RRBS19a] also appeared'). This is a duplicate bibliography entry, cited under two labels; the fix is to merge them.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 22 Appendix B.4, first sentence

  We can solve A.1 to find an expression

• p. 22 Appendix B.4, after Eq. (B.15)

  Using A.6 and A.9, we can eliminate L

• p. 22 Appendix B.4

  The result is shown in Figure X.

• p. 22 Appendix B.4

  (see Figure Y)

Judge's note: Page 22 image shows all four pointers as quoted. Appendix A (page 20) holds only Tables 4–6 and has no numbered equations; the appendix equations are numbered B.1–B.17. The equations meant are evidently B.1 (or B.3, from which B.15 follows directly), B.6 and B.9. The content described for Figure X and Figure Y (0.6x–2.2x model sizes at 20% more compute; fewer steps for larger models) is Figure 12 left and right on page 15, and Section 6.1 cites it as 'Figure 12 ... (see Appendix B.4)'.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 5 Section 1.2, paragraph after Eq. (1.4)

  Equation (1.1) and (1.2) together suggest

• p. 4 Eqs. (1.1), (1.2)

  αD ∼0.095

• p. 5 Section 1.2, text before Eq. (1.5)

  Equation (1.1) and (1.2) together suggest that as we increase the model size, we should increase the dataset size sublinearly according to D ∝ N^(αN/αD) ∼ N^0.74

• p. 4 Eqs. (1.1), (1.2)

  αN ∼ 0.076

Only the authors can say what is right here.

Judge's note: Read Eqs. (1.1) and (1.2) on page 4 and the sentence on page 5. Table 2 (text layer, page 11) gives αN = 0.076 and αD = 0.103, whose ratio is 0.74. Page 11 says the joint-fit parameters "differ very slightly from those obtained in Section 3", which explains where 0.74 comes from, but the page 5 sentence attributes it to Eqs. (1.1) and (1.2), whose values do not give it.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send