PaLM: Scaling Language Modeling with Pathways

• p. 12 Table 4, Lambada (EM), 0-shot Prior SOTA (superscript f = Chinchilla)

  77.7

• p. 49 Table 21, Lambada 0-shot, Chinchilla

  77.4

• p. 49 Table 21, Lambada 0-shot, Chinchilla column

  Lambada 0-shot

• p. 12 Table 4, Lambada 0-shot prior SOTA (superscript f = Chinchilla)

  77.7f

Only the authors can say what is right here.

Judge's note: Table 4 on page 12 prints 77.7 with superscript f, which the caption defines as Chinchilla (Hoffmann et al., 2022). Table 21 on page 49 prints 77.4 in the Chinchilla column for Lambada 0-shot; the text layer of Table 31 (PDF page 77) also has 77.4. The other Chinchilla values shared by Tables 4 and 21 agree (HellaSwag 80.8, Winogrande 74.9, BoolQ 83.7), so a different source is unlikely. Which value is right cannot be told from the files. PaLM's 77.9 exceeds both, so the comparison is not reversed.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 49 Table 21, HellaSWAG 0-shot, PaLM 540B

  83.6

• p. 12 Table 4, HellaSwag 0-shot / 1-shot PaLM 540B

  83.4

• p. 77 Table 31, HellaSWAG 0-shot, PaLM 540B

  83.6

• p. 49 Table 21, HellaSWAG 0-shot, PaLM 540B

  HellaSWAG 0-shot 68.7 79.7 83.6

Judge's note: Viewed all four tables. Table 4: HellaSwag 0-shot PaLM 540B 83.4, 1-shot 83.6. Table 40: 0-shot 540B 83.4, 1-shot 83.6. Tables 21 and 31 print 83.6 in the row labelled 0-shot. The 8B and 62B cells of the same row (68.7, 79.7) match Table 40's 0-shot columns, so the row is meant to be 0-shot. The text around Tables 21 and 31 gives no different setup or checkpoint. The 540B average of 70.3 is computed with the printed values; with 83.4 here and 88.0 for BoolQ (C067) it would be 70.2.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 49 Table 21, BoolQ 0-shot, PaLM 540B

  88.7

• p. 12 Table 4, BoolQ 0-shot / 1-shot PaLM 540B

  88.0

• p. 77 Table 31, BoolQ 0-shot, PaLM 540B

  88.7

• p. 49 Table 21, BoolQ 0-shot

  BoolQ 0-shot 68.3 84.8 88.7

Judge's note: Viewed all four tables. Table 4: BoolQ 0-shot PaLM 540B 88.0, 1-shot 88.7. Table 40: 0-shot 540B 88.0, 1-shot 88.7. Tables 21 and 31 print 88.7 in the row labelled 0-shot. The 8B and 62B cells (68.3, 84.8) match Table 40's 0-shot columns. No explanation in the surrounding text. With the correct 0-shot values for BoolQ and HellaSwag the 540B average would be 70.2, not the printed 70.3 (second expression).

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 14 Table 9, PaLM 540B (finetuned) row

  90.4 91.9 94.4/96.0 99.0 88.7/63.6 94.2/93.3 95.9 77.4 95.9

• p. 14 Table 9, row PaLM 540B (finetuned)

  90.4

• p. 14 Table 9, row PaLM 540B (finetuned), RTE

  95.9

Only the authors can say what is right here.

Judge's note: Values read from the high-resolution crop of Table 9. Averaging the eight tasks (paired metrics averaged first) gives 91.16 for ST-MoE-32B (printed 91.2) and 71.79 for the decoder-only LM (printed 71.8), and reproduces the Table 7 averages for ST-MoE and PaLM, but gives 90.65 for this row against a printed 90.4. A gap of 0.25 is more than rounding. The RTE entry (bold, 95.9) equals the WSC entry in the same row and exceeds the dev-set RTE of 95.7 in Table 7; an RTE near 94 would reproduce 90.4, but the files do not show which cell is wrong. No explanation of a different averaging rule was found in the text.

Second check: another model agreed · checked by hand

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 16 Section 6.2, discontinuity definition paragraph

  The logical sequence task had normalized accuracy scores of 13%, 25%, and 87%

• p. 16 Section 6.2, preceding paragraph

  the improvement from 25% for PaLM 62B to 87% for PaLM

• p. 17 Figure 5, panels english_proverbs and logical_sequence

  logical_sequence

• p. 16 Sec 6.2, same page

  the improvement from 25% for PaLM 62B to 87% for PaLM 540B

Judge's note: Compared the page 16 text with the Figure 5 panels on page 17 and the enlarged crop. The english_proverbs panel shows about 13%, 25% and 87%, and the preceding paragraph gives 25% (62B) and 87% (540B) for english_proverbs. The logical_sequence panel has its 8b point below 0%, its 62b point just under 10% and its 540b point near 90%. The worked example (37% projection, +50% discontinuity) is arithmetically right for the english_proverbs numbers, so the task named is wrong. Plot values are read by eye, but the 8b and 62b points are clearly not 13% and 25%.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 28 Table 14, 0-shot PaLM 540B column

  38.5

• p. 38 Table 19, PaLM 540B 0-shot Full Set / Clean Set columns

  40.4 43.8

• p. 38 Table 19, row En-Fr, PaLM 540B 0-shot

  38.4

• p. 38 Table 19, En-Fr row, 540B columns

  88.9% 21.4 21.6 25.5 25.0 38.2 38.4

Only the authors can say what is right here.

Judge's note: Viewed pages 28 and 38. Table 14 is the full WMT test set result; Table 19's Full Set column should repeat it. Instead five of six Table 14 values equal Table 19's Clean Set column, and De-En differs from the Full Set column by 3.4 BLEU (43.8 vs 40.4). The text on page 37 says "The results on De-En are roughly 3.0 BLEU worse on the clean set", yet Table 19 prints the clean set as higher at every scale (34.9 vs 31.7, 34.8 vs 31.5, 43.8 vs 40.4). Both points suggest the Full Set and Clean Set headers in Table 19 are swapped, but only the authors can say which. En-Fr 38.5 (Table 14) vs 38.2 / 38.4 (Table 19) remains unexplained either way. Corrected description: some reviewer notes said the Full Set column matches Table 14 for five pairs; it is the Clean Set column that does. No statement in the paper explains the difference.

Second check: another model agreed · checked by hand

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 37 Section 8, last paragraph

  The results on De-En are roughly 3.0 BLEU worse on the clean set

• p. 38 Table 19, De-En row

  31.7 34.9 31.5 34.8 40.4 43.8

• p. 38 Table 19, row De-En, PaLM 540B 0-shot (Full Set BLEU / Clean Set BLEU)

  43.8

• p. 28 Table 14, de-en 0-shot PaLM 540B

  43.8

Only the authors can say what is right here.

Judge's note: Read page 37 text and Table 19 on the page 38 image: headers are Full Set then Clean Set for each model, and the De-En row reads 31.7 34.9 31.5 34.8 40.4 43.8. Clean minus full is +3.2, +3.3, +3.4, the opposite direction to the sentence. Table 14 (page 28 image) 0-shot PaLM 540B gives de-en 43.8, fr-en 41.1, ro-en 39.9, en-ro 24.2, en-de 31.8, which match the column labelled Clean Set, supporting a header swap. En-Fr is 38.5 in Table 14 and matches neither column (38.2 / 38.4), so a swap does not reconcile everything. No explanation found in the text. Only the authors can say whether the headers or the sentence is wrong.

Second check: another model agreed · checked by hand

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 29 Section 6.5, paragraph after Figure 15

  English-German language pair observes a 13 BLEU increase and English-French language pair observes a 17 BLEU increase

• p. 38 Table 19, En-Fr and En-De rows, 62B and 540B Full Set

  25.5 25.0 38.2 38.4

• p. 38 Table 19, rows En-Fr and En-De, PaLM 62B and 540B 0-shot

  15.0

• p. 29 Evaluation on English-centric language pairs, scaling paragraph

  English-German language pair observes a 13 BLEU increase

Judge's note: Viewed pages 29 and 38. The sentence refers to 0-shot scaling from 62B to 540B (Figure 15-right). Table 19 0-shot: En-De 62B 15.0 / 14.8, 540B 31.8 / 31.8; En-Fr 62B 25.5 / 25.0, 540B 38.2 / 38.4. Either column gives about 17 for En-De and about 13 for En-Fr. Figure 15(b) shows the same: the en-de jump from 62b to 540b is the larger one. The uncertainty over Table 19's column headers (C026) does not change the assignment.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 86 Table 44, XSum (en), Finetuning PaLM 540B

  56.36

• p. 32 Table 16, XSum (en), Finetuning PaLM 540B

  21.2

• p. 87 Table 46, row XSum (en), T5 XXL

  56.36

• p. 86 Table 44, XSum (en), last column

  22.5/20.3/21.0 56.36

Judge's note: On the p86 image every other cell of Table 44 is a three-part P/R/F value with one decimal; the XSum finetuned PaLM 540B cell is a lone 56.36. Table 16 (p32 crop) gives ROUGE-2 F = 21.2 for the same cell, and the neighbouring Table 44 cells in the row are 12.2 to 23.2 in F. Table 46 (p87) has 56.36 in the XSum row under finetuned T5 XXL (BLEURT-20), so the value was pasted from the wrong table. The correct F is 21.2 per Table 16; the precision and recall are not recoverable from the paper.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 37 Table 18, Winograd, PaLM 8B 1-Shot Full Set Accuracy

  82.4

• p. 83 Table 40, Winograd row, 1-shot PaLM 8B and 62B

  85.3

• p. 37 Table 18, Winograd row, PaLM 8B 1-shot full-set accuracy

  Winograd 61.5% 82.4 -4.4 87.5 -1.8

• p. 83 Table 40, Winograd 1-shot

  Winograd 80.6 87.5 90.1 85.3 82.4 88.6 87.5

Only the authors can say what is right here.

Judge's note: Page 37 image, Table 18 Winograd row: 61.5% 82.4 -4.4 87.5 -1.8. Page 83 image, Table 40 Winograd row: 0-shot 80.6 87.5 90.1; 1-shot 85.3 (8B) 82.4 (62B 795B) 88.6 (62B 1325B) 87.5 (540B). The 540B value agrees (87.5); the 8B value does not. The other 8B rows of Table 18 (48.5, 12.6, 57.8, 71.3, 42.3, 81.4, 87.8, 41.1) match Table 40's 8B 1-shot column, apart from SQuADv2 (see C048). No different checkpoint or split is described. Either Table 18 is wrong or the 8B and 62B entries in Table 40 are transposed. I did not re-derive the Table 5 averages that one reviewer offered as support.

Second check: another model agreed · checked by hand

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 37 Table 18, SQuADv2 (F1), PaLM 8B 1-Shot Full Set Accuracy

  50.1

• p. 83 Table 40, SQuADv2 (F1) and SQuADv2 (EM), 1-shot PaLM 8B

  56.0

• p. 37 Table 18, SQuADv2 (F1), PaLM 8B full set

  SQuADv2 (F1) 14.8% 50.1 -2.5 82.9 +1.1

• p. 83 Table 40, PaLM 8B 1-shot

  SQuADv2 (F1) 56.4 75.6 80.8 56.0 ... SQuADv2 (EM) 49.6 69.9 75.5 50.1

Only the authors can say what is right here.

Judge's note: Page 37 image, Table 18: SQuADv2 (F1) 14.8% 50.1 -2.5 82.9 +1.1. Page 83 image, Table 40 1-shot: SQuADv2 (F1) 56.0 (8B) ... 82.9 (540B); SQuADv2 (EM) 50.1 (8B) ... 78.7 (540B). So the row mixes EM for 8B with F1 for 540B. Whether the 8B delta of -2.5 was computed on EM or F1 cannot be told from the paper, so the authors need to confirm the corrected value and delta.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 70 Table 27 (Datasheet), 'Any other comments?', second bullet

  The language distribution is shown in Figure 29.

• p. 7 Table 2 caption

  with the distribution given in Appendix Table 29

• p. 73 Table 29 caption

  Table 29: Language distribution of the training data

• p. 73 Table 29 caption

  Table 29: Language distribution of the training data (excluding code data).

Judge's note: Page 70 image shows 'Figure 29'. Page 73: 'Table 29: Language distribution of the training data (excluding code data).' Page 79: 'Figure 29: Examples questions where PaLM 62B had incorrect responses with one-step missing errors'. The Table 2 caption (page 7 text) points correctly to 'Appendix Table 29'. Wrong pointer type; the fix is Table 29.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 12 Table 4, Natural Questions (EM), 1-shot and Few-shot Prior SOTA

  32.5

Only the authors can say what is right here.

Judge's note: Checked on the high-resolution crop of Table 4. Both cells carry superscript a, defined in the caption as one model (GLaM 62B/64E), and the few-shot cell's shot count is (1). The TriviaQA row above is consistent (75.8 and 75.8 (1)). Either the shot count (1) or the value or source is wrong; only the authors can say which.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 12 Table 4 caption

  GLaM 62B/64E

• p. 13 Table 5, second row

  GLaM 64B/64E

Only the authors can say what is right here.

Judge's note: Read both on the page images (pages 12 and 13). Both tables take the GLaM results from Du et al. (2021), so they refer to one model. The two names are inconsistent; the files do not establish which is correct, though "62B" is also the size of a PaLM model and may have been carried over.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 12 Table 4, RTE, Few-shot Prior SOTA

  76.8 (5)

• p. 12 Table 4, ReCoRD, Few-shot Prior SOTA

  90.6 (2)

Only the authors can say what is right here.

Judge's note: Checked on the high-resolution crop: every other prior-SOTA cell has a superscript a, b, c, e or f; these two have none. Scanning the whole table on page 12, no cell uses d. The sources of the two values are therefore not identified; the authors would have to supply them.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 14 Table 7, Record column

  93.8/93.2

• p. 14 Table 7, Record column, ST-MoE-32B and PaLM rows

  95.0/95.6

Only the authors can say what is right here.

Judge's note: Rests on outside knowledge: that the ReCoRD pair is F1 and exact match, and that exact match cannot exceed F1 on the same predictions. The paper does not label the two values in this column. On the page image the three Table 7 rows are as quoted, and all three Table 9 rows are high/low. If the fact holds, no single ordering fits all three Table 7 rows, so either the order differs between rows or values are swapped. The row averages are unaffected because the pair is averaged.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 21 Figure 9, title inside the figure

  Error analysis of PaLM 64B vs. 540B on 150 GSM8K Examples

• p. 21 Figure 9 caption

  found that PaLM 62B makes reasoning errors on

• p. 21 Figure 9 caption

  PaLM 62B makes reasoning errors on 45 of them

• p. 21 Figure 9 labels

  62B made 20 errors of this type

Judge's note: Viewed page 21. The title inside the figure says 64B; the three bar labels, the right-hand annotation ("from PaLM 62B to 540B") and the caption all say 62B. No 64B model exists in the paper. Label error in a figure, so S2 rather than S4.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 24 Section 6.4, Datasets paragraph

  Table 12 summarizes the amount of code training data used: the row PaLM 540B shows the

• p. 24 Section 6.4, Datasets paragraph

  Table 12 summarizes the amount of code training data used: the row PaLM 540B shows the number of tokens used for pre-training

• p. 24 Table 11 caption

  Table 11: Amount of code tokens processed during training

• p. 24 Datasets paragraph

  Table 12 summarizes the amount of code training data used

Judge's note: Viewed pages 24 and 25. Table 11 (page 24) has the rows PaLM 540B and PaLM-Coder 540B with code token counts. Table 12 (page 25) has pass@k results and no token counts. Page 25 itself says the total code data "is shown in Table 11".

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 25 Figure 12, x-axis tick labels (all four panels)

  535b

• p. 25 Figure 12 caption

  PaLM and PaLM-Coder models scales 8B, 62B and 540B

• p. 25 Figure 12 caption

  PaLM models scales 8B, 62B and 540B

• p. 25 Figure 12, x-axis tick labels

  8b 64b 535b

Judge's note: Viewed page 25. All four panels have ticks 8b, 64b, 535b. The caption and the rest of the paper use 62B and 540B, and Figure 10 (page 22) labels the same models 62b and 540b. A text search finds 64b and 535b only in this figure, and nothing in the paper defines them (for example as non-embedding parameter counts).

Second check: another model agreed · checked by hand

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 34 Figure 17 caption

  The checkpoints are all spaced 2B tokens apart that ends with the final model checkpoint

• p. 34 Section 6.8, text

  We take 15 different evenly spaced model checkpoints

• p. 34 Section 6.8, paragraph before Figure 17

  Given that each checkpoint only differs by 2B pre-training tokens

• p. 34 Figure 17, x-axis 'Tokens Trained (Billions)'

  765

Only the authors can say what is right here.

Judge's note: Viewed Figure 17 on the page image and a 4x render. Each series has 15 points; the first sits about half-way between the left edge and the 765 tick and the last just under 777.5, with roughly five points per 5B of axis. Fifteen checkpoints 2B apart would span 28B (752B to 780B). The same page says results elsewhere use the 780B checkpoint for 8B and 540B and 795B for 62B, yet all three series share the same x-range ending near 777B. So either the axis values or the '2B' and 'ends with the final checkpoint' statements are wrong; the authors must say which. Point positions are read by eye, so the spacing is approximate, but it cannot be 2B.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 41 Section 10.1.1, paragraph after Figure 20

  according to the 2015 US BLS

• p. 41 Figure 21 caption

  based on 2016 US BLS occupation data

• p. 41 Section 10.1.1, paragraph after Figure 20

  according to the 2015 US BLS occupation statistics

• p. 41 Disaggregated Winogender paragraph

  2015 US BLS occupation statistics

Only the authors can say what is right here.

Judge's note: Both read on the page 41 image; both passages describe the same stereotypical/gotcha definition and the caption cites Rudinger et al., 2018. The paper gives no reason for two years. Which year is right depends on the cited source, which is not in the files, so the authors must say.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 79 Table 34, religion rows

  {term} practitioners are...”

• p. 43 Figure 22, y-axis labels

  One who practices Atheism is

• p. 79 Table 34, religion row

  {term} practitioners are...

• p. 79 Table 34, religion row

  {term} practitioners are...” {term} practitioners are...” All {term} practitioners are...

Judge's note: Table 34 on the p79 image shows two identical religion lines, both ending in a closing quotation mark with no opening one, then 'All {term} practitioners are...'. Figure 22 on p43 labels three templates per religion: 'All X practitioners are', 'X practitioners are' and 'One who practices X is'. The duplicated line is evidently meant to be the 'One who practices' template. The point about the Table 36 caption prompt raised in one note is judged under C094.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 47 Section 12, second paragraph

  trained Transformer (GPT) series, decoder-only models, (Radford et al., 2018; Raffel et al., 2020) set

• p. 59 Reference list

  Exploring the limits of transfer learning with a unified text-to-text transformer

• p. 47 Related Work

  (Radford et al., 2018;

• p. 47 Section 12, Related Work

  The Generative Pre-trained Transformer (GPT) series, decoder-only models, (Radford et al., 2018; Raffel et al., 2020)

Judge's note: Viewed p47 and the reference entry on p59 ("Exploring the limits of transfer learning with a unified text-to-text transformer", JMLR 2020). The very next sentence on p47 cites the same reference for "11B parameter encoder-decoder models", so the paper itself treats it as an encoder-decoder work. It cannot support a statement about the decoder-only GPT series; a second GPT reference was presumably intended. Which reference was intended is for the authors, but the miscitation is evident from the paper alone.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 75 Table 30 (Model Card), Fine-tuning Dataset

  SuperGLUE (?)

• p. 75 Table 30, Fine-tuning Dataset

  We include finetuning results on SuperGLUE (?)

• p. 75

  SuperGLUE (?),

Judge's note: Page 75 image shows a literal bold '(?)' after SuperGLUE, the mark of a citation key that did not resolve. Section 6.1 cites the benchmark as 'SuperGLUE (Wang et al., 2019a)', which is the evident fix.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 75 Table 30, Evaluation Dataset

  multilingual generation and question answering tasks (Section 6.6)

• p. 75 Table 30, Benchmark Information

  Multi-lingual Question Answering (Section 6.7)

Judge's note: Page 75 image. The Evaluation Dataset row gives only Section 6.6 for generation and question answering together. The Benchmark Information row of the same table gives GEM as Section 6.6 and multilingual QA and TyDiQA as Section 6.7, and the introduction and conclusion likewise split them ('summarization (Section 6.6), and question answering (Section 6.7)'). The pointer is incomplete for QA; it should read Sections 6.6 and 6.7.

Second check: another model agreed · checked by hand

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 80 Table 36 caption

  prompts like “People would describe the {term} person as...”

• p. 79 Table 34, race rows

  The {term} man was very...”

• p. 80 Table 36 caption

  People would describe the {term} person as...

• p. 79 Table 34, race row

  The {term} man was very...

Only the authors can say what is right here.

Judge's note: Read Table 34 (p79), the Table 36 caption (p80) and Section 10.1.2 (p42) on the page images. The text says 'For each prompt (Table 34)' and discusses race results with 'The term was' / 'The term was very' and a continuation starting 'The White man was very'. The template quoted in the Table 36 caption appears nowhere else in the paper. The captions of Tables 35 and 37 quote prompts that are in Table 34. Either Table 34 omits a race prompt or the caption quotes the wrong one; only the authors can say which.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 4 Multilingual understanding bullet

  bridge the gap with prior

• p. 32 Table 16, MLSum rows

  MLSum (de) 0.9 4.6 10.5 12.8 36.4

• p. 32 Generation quality bullet

  is larger for non-English generation

Only the authors can say what is right here.

Judge's note: Read the page 4 bullet and Table 16 with the Section 6.6 bullets on page 32. The only non-English-output summarization tasks are MLSum (de) and MLSum (es); 1-shot is about a third of prior SOTA on both. Section 6.6 itself says non-English (MLSum) finetuning does not achieve SOTA and that the few-shot versus finetuning gap is larger for non-English generation. Nothing in the paper supports the introduction's wording for few-shot results; the WikiLingua x->en tasks generate English. The authors must say what was meant.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 31 Section 6.6, Finetuning methodology

  20x smaller than during pretraining

• p. 10 Section 5, Optimization hyperparameters

  We use an Adafactor learning rate of 10−2 for the first 10,000

• p. 10 Section 5, Optimization hyperparameters

  for the first 10,000

• p. 10 Optimization hyperparameters

  Adafactor learning rate of 10−2

Only the authors can say what is right here.

Judge's note: Read page 31 (image) and the Section 5 optimisation bullet on page 10. 1e-2 / 5e-5 = 200. With 1/sqrt(k) decay the rate would reach 1e-3 (20x the finetuning rate) only at one million steps, more than the training lengths described for the 540B model (about 255k steps, giving roughly 2e-3, 40x). The paper defines no other reference rate, and Section 6.1.2 repeats the 5×10−5 value without the comparison. Either the multiplier or the reference point is wrong or unstated; only the authors can say which. The finetuning rate itself is consistent across sections, so the affected item is the comparison rather than a result.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 21 Section 6.3.1, second paragraph

  GSM8K (Cobbe et al., 2021), SVAMP (Patel et al., 2021), MAWPS (Koncel-Kedziorski et al., 2016), and AQuA

• p. 22 Section 6.3.1, first paragraph on page

  Across the 7 reasoning datasets

• p. 21 Section 6.3.1, second paragraph

  StrategyQA (Geva et al., 2021). This prompting setup uses only 8-shot exemplars

• p. 21 Section 6.3.1

  arithmetic datasets GSM8K

Judge's note: Viewed pages 21 and 22. The setup paragraph names four arithmetic and two commonsense datasets. The next page and Figure 10 (seven panels, including ASDiv) cover seven. A text search finds ASDiv only in that results sentence, the Figure 10 panel title and the Figure 10 caption; it has no dataset citation. The count of 7 matches the figure, so the dataset list is incomplete.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 30 Section 6.5.1, second bullet

  We found that in most cases, using the language names to induce translation (0-shot setting) provided stronger performance than

• p. 28 Table 14, PaLM 540B 0-shot / 1-shot / few-shot

  24.2

• p. 30 Section 6.5.1, second bullet

  We found that in most cases, using the language names to induce translation (0-shot setting) provided stronger performance than only using input-output exemplars (1-shot and few-shot settings)

• p. 28 Table 14, PaLM 540B columns

  44.0

Judge's note: Read Tables 14 and 15 from the page images. 0-shot/1-shot/few-shot: en-fr 38.5/37.5/44.0; en-de 31.8/31.8/37.4; en-ro 24.2/28.2/28.7; fr-en 41.1/37.4/42.8; de-en 43.8/43.9/47.5; ro-en 39.9/42.1/43.8; en-kk 1.8/4.2/5.1; de-fr 28.6/20.9/25.7; kk-en 18.0/20.3/20.8; fr-de 25.2/9.5/17.4. 0-shot is above 1-shot for en-fr, fr-en, de-fr and fr-de only, and above few-shot for de-fr and fr-de only; few-shot beats 0-shot on all six Table 14 pairs. The table captions themselves hedge with 'in some language pairs', which fits the data; 'in most cases' in the bullet does not. No other translation results are reported that could support the wording.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 44 Section 10.2, paragraph on Figure 23

  we find that the TPC increases with the TPP, while consistently lower than the prompt toxicity and the human baseline (except at the highest levels of prompt toxicity)

• p. 44 Figure 23, first two bins

  0.10

Only the authors can say what is right here.

Judge's note: Viewed Figure 23 on p44 and the crop. First bin (centre 0.05): 8B at 0.10, 62B/540B at 0.12, all above the bin's upper edge of 0.1. Second bin (centre 0.15): 0.14 and 0.16, about equal to the prompt toxicity. From the third bin up the claim holds. The stated exception covers only the highest prompt-toxicity levels, so the sentence overstates for the lowest bins. Wording issue, not a data error.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 68 Religions paragraph

  Table 25 shows the 20 TF-IDF tokens that co-occurred with each religion

• p. 69 Table 25

  TF-IDF tokens co-occurring most frequently with religions

• p. 68 Appendix C, Religions

  Table 25 shows the 20 TF-IDF tokens that co-occurred with each religion.

• p. 69 Table 25

  Table 25: TF-IDF tokens co-occurring most frequently with religions

Judge's note: Counted the rows of Table 25 on the page 69 image: Islam 11, Christianity 11, Hinduism 11, Buddhism 9, Judaism 11. No column has 20. The only other '20' is the per-datapoint list ('saving the 20 tokens with the highest TF-IDF values for each datapoint', page 68), which is a different quantity and does not make the table show 20 per religion. The parallel sentence for Table 26 says 'the top co-occurring TF-IDF tokens', which is the evident fix. The reviewers' row counts (10) were slightly off; the defect stands.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send