Llama 2: Open Foundation and Fine-Tuned Chat Models

• p. 6 Figure 5 y-axis label and caption

  Train PPL

Judge's note: On page 6 the y-axis reads "Train PPL" and runs from 1.4 to 2.2, while the caption and the Section 2.2 text both say training loss. The axis label disagrees with the caption.

Second check: another model agreed

• p. 14 Figure 7 caption and x-axis

  N ∈[1, . . . , 100]

Only the authors can say what is right here.

Judge's note: On page 14 the Figure 7 x-axis has ticks at 10^0 and 10^1, with the last data points at roughly 16 and 20. Figure 8, on the same page, does extend to 10^2.

Second check: another model agreed

• p. 23 Section 4.2 item 2

  the general RLHF pipeline described in Sec- tion 3.2.2

Judge's note: The section headings in the text layer are 3.2.1 Human Preference Data Collection, 3.2.2 Reward Modeling and 3.2.3 Iterative Fine-Tuning. Because the sentence also covers the safety reward model, the pointer is partly apt; 3.2 or 3.2.3 would fit the RLHF pipeline better.

Second check: another model agreed

• p. 24 Section 4.2.3

  collecting human preference data for safety similar to Section 3.2.2

Judge's note: Section 3.2.1 is "Human Preference Data Collection" and 3.2.2 is "Reward Modeling". The same sentence cites 3.2.2 again, correctly, for the reward model.

Second check: another model agreed

• p. 5 Section 2, first paragraph

  trained on 40% more total tokens

• p. 6 Table 1, Tokens column

  1.0T

Judge's note: Table 1 on page 6 lists Llama 1 at 1.0T, 1.0T, 1.4T and 1.4T, and every Llama 2 model at 2.0T. The Introduction's "increased the size of the pretraining corpus by 40%" fits the 1.4T corpus. The Section 2 wording about tokens trained on is ambiguous for the smaller models.

Second check: another model agreed

• p. 11 Table 6, Total row, Avg. # Turns

  1.6

Only the authors can say what is right here.

Judge's note: Read from the page 11 image. The token columns of the Total row come out close to (though not exactly) comparison-weighted means (590.6 vs 595.7; 113.2 vs 108.2), but the turns column does not (about 2.52 vs 1.6). Weighting by dialogues (comparisons divided by turns) would give about 1.66, so a dialogue-based weighting might explain it. The paper does not state the weighting, so this is a question for the authors.

Second check: another model agreed

• p. 12 Table 7, Avg column

  55.3

Judge's note: Checked on the page 12 crop. Every row differs from the unweighted mean. The Avg may be weighted by test-set size, but this is not stated in the caption or text.

Second check: another model agreed

• p. 30 Section 4.4, last sentence

  although their violation percentages look similar (3.88 vs 4.45)

Judge's note: On page 30, Figure 17a shows violation percentages of roughly 7–8% for both Falcon 40b-instruct and Llama-2 34b-chat. Figure 17b shows mean ratings of about 3.9 (Falcon) and 4.45 (Llama-2 34b). So 3.88 and 4.45 are the ratings, but the sentence places them after 'violation percentages look similar'.

Second check: another model agreed

• p. 33 Figure 21 caption and x-axes

  T ∈{k/10 | k ∈N : 1 ≤k ≤15}

Only the authors can say what is right here.

Judge's note: On page 33, the x-axes of both the Factual and Creative panels have points from 0.4 to 1.4 only (11 values), while the caption lists 15 values from 0.1 to 1.5. The paper gives no reason for the omitted temperatures; only the authors can say whether these were run but not plotted.

Second check: another model agreed

• p. 46 A.1 Contributions

  All authors sorted alphabetically by last name.

• p. 46 Core Contributors

  Xavier Martinet, Yuning Mao, Igor Molybog, Todor Mihaylov

• p. 46 Contributors

  Lukas Blecher, Dan Bikel, Shruti Bhosale

Judge's note: Checked on the page 46 image. Core Contributors lists Martinet before Mao, Molybog before Mihaylov, and Zhang before Zarov. Contributors lists Blecher/Bikel/Bhosale in reverse order, Kloumann before Khabsa, Korenev before Kerkez, Lu before Lee, and Schelten before Saladi. The leadership lists are in order.

Second check: another model agreed

• p. 58 A.4.2

  once we use 50% of safety data

• p. 61 Table 36, 25% row

  I can’t satisfy your request, I’m just an AI, I cannot provide content that

• p. 58 A.4.2 Qualitative Results on Safety Data Scaling

  learns to refuse to generate offensive content (Table 36) once we use 50% of safety data

• p. 61 Table 36, 25% row

  I can’t satisfy your request, I’m just an AI, I cannot provide content that is offensive or inappropriate.

Only the authors can say what is right here.

Judge's note: I read Table 36 on p.61. The 0%, 1% and 10% rows give roasts (safety RM 0.05 and 0.18; no score is printed for 0%). The 25% row begins with a refusal (safety RM 0.91), then still offers some 'lighthearted' roast examples. The 50% and 100% rows refuse without giving examples (0.94 and 0.93). The 50% threshold in the text could mean a complete refusal, but the text does not say so. The authors should be asked whether 25% or 50% is the intended onset.

Second check: another model agreed

• p. 69 A.4.8 Toxicity bullet

  Mexicans, Latinos, and women tend to be the top three demographic groups

• p. 71 Table 45, Jewish and Women columns, pretrained rows

  Jewish

Only the authors can say what is right here.

Judge's note: Read Table 45 on p71 and in the layout text. Means over the 12 pretrained rows: Mexican 30.87, Jewish 27.24, Latino 26.73, Women 24.35. Women is in a row's top three only for Llama 1 7B, Llama 1 65B and Llama 2 70B. The wording is hedged, so this is a misdescription rather than a hard error.

Second check: another model agreed

• p. 76 Table 51, MMLU-Humanities, Not Clean, 70B

  85.12

• p. 76 Table 51, MMLU-Humanities, Not Clean, 7B

  85.2

Only the authors can say what is right here.

Judge's note: Checked on the p76 image. The other matching pairs are 0.05/0.05, 2.73/2.73 and 94.5/94.5, and the HellaSwag values match exactly. Contamination is defined from the evaluation samples, which are the same for both models (same n). 85.12 does not round to 85.2. Only the authors can say which value is right.

Second check: another model agreed

• p. 1 author list

  Jian Xiang Kuan Puxin Xu Zheng Yan

• p. 46 A.1 Core Contributors

  Ross Taylor, Jacob Xu, Yuchen Zhang, Iliyan Zarov.

Only the authors can say what is right here.

Judge's note: A.1 states "All authors sorted alphabetically by last name." Puxin Xu appears only on page 1, and Jacob Xu appears only in A.1 (searched the text layer). Both are probably the same person under two names, but only the authors can say which form is intended.

Second check: none (judge only)

• p. 3 Introduction, paragraph 2

  LLaMa-1 (Touvron et al., 2023)

• p. 4 Introduction, release list

  an updated version of Llama 1

Judge's note: Page 3 shows "LLaMa-1". A search of the text layer finds this as the only occurrence of that form; page 4 and elsewhere use "Llama 1". The difference is cosmetic.

Second check: none (judge only)

• p. 6 Figure 5 y-axis and caption

  Figure 5: Training Loss for Llama 2 models. We compare the training loss

• p. 5 Section 2.2 Hyperparameters

  Figure 5 (a) shows the training loss for Llama 2

Judge's note: Page 6 shows Figure 5 as one panel without sub-labels; page 5 refers to "Figure 5 (a)". The axis-label part of this cluster is the same defect as C004 and is judged there.

Second check: none (judge only)

• p. 8 Section 2.3, paragraph after Table 3

  Llama 2 7B and 30B models outperform MPT models of the corresponding size

• p. 48 A.2.2 Reading Comprehension

  Llama 1 30B performs slightly better

• p. 48 A.2.2 Reading Comprehension

  except the QUAC 0-shot where Llama 1 30B performs slightly better

• p. 8 Section 2.3 text after Table 3

  Llama 2 7B and 30B models outperform MPT models

Judge's note: Table 1 and Table 3 list Llama 2 as 7B/13B/34B/70B and Llama 1 as 7B/13B/33B/65B. In Table 23 (page 50) the QUAC 0-shot best score is Llama 1 33B at 44.1. No 30B Llama model exists in the paper.

Second check: none (judge only)

• p. 27 Section 4.2.4

  “respectful’,’

Judge's note: The page 27 crop shows the misplaced quotes and comma in the rendered PDF, so this is not an extraction artefact.

Second check: none (judge only)

• p. 27 Table 13, right column preprompt

  a question that can ellicit unsafe behavior

• p. 27

  The following is a question that can ellicit unsafe behavior

Judge's note: The crop of page 27 shows 'ellicit' in the right-column preprompt. Elsewhere the paper spells it 'elicit' (e.g., the Table 39 templates, text lines 5060/5070).

Second check: none (judge only)

• p. 28 Section 4.3

  “red teaming,“

Judge's note: The page 28 crop shows an opening-style double quote after 'teaming,'.

Second check: none (judge only)

• p. 30 Section 4.4

  0.70to0.95

• p. 30 Section 4.4, Safety Human Evaluation, IRR sentence

  The IRR scores range from 0.70to0.95 depending

• p. 30 Safety human evaluation, IRR sentence

  The IRR scores range from 0.70to0.95

• p. 77 Table 52 model card

  Responsible Use Guide available available at

Judge's note: The page 30 crop shows '0.70to0.95' set in math mode with an italic 'to' and no spaces. The other typos bundled into this cluster also check out on the images: 'available available at' (Table 52, p77) and 'starts to be behave more conservatively' (A.4.2, p58).

Second check: none (judge only)

• p. 32 Section 5.1, Beyond Human Supervision

  This phenomena is illustrated in Figure 20

Judge's note: Confirmed in the page 32 crop.

Second check: none (judge only)

• p. 34 Figure 23 caption

  understand the tools’s applications

• p. 34 Figure 23 caption

  able to understand the tools’s applications

Judge's note: The page 34 crop shows 'tools’s'. It should be 'tools’'.

Second check: none (judge only)

• p. 34 Section 5.1, Tool Use Emergence

  we never explicitly annotate tool-use usage

Judge's note: The page 34 crop confirms the wording. 'Tool-use usage' is redundant; it is cosmetic only.

Second check: none (judge only)

• p. 42 Reference list, Nakano et al.

  Lonbrown Ouyanbrown

• p. 44 Touvron et al. reference

  Aur’elien Rodriguez

Judge's note: The p42 crop shows 'Lonbrown Ouyanbrown'. The p44 crop shows 'Aur’elien', with an apostrophe where the accent should be; A.1 spells the same person 'Aurelien Rodriguez'.

Second check: none (judge only)

• p. 48 A.2.2 Reading Comprehension

  except the QUAC 0-shot where Llama 1 30B performs slightly better

• p. 50 Table 23, Llama 1 33B QUAC 0-shot

  44.1

Judge's note: Checked Table 23 on p.50. The Llama 1 sizes listed are 7B/13B/33B/65B. The QUAC 0-shot best is Llama 1 33B at 44.1 (bold), higher than the best Llama 2 value of 42.9 (34B). On p.48 the text says '30B'. Every other Llama 1 table in the paper also uses 33B. The model meant is clear, so this is a label slip only.

Second check: none (judge only)

• p. 51 A.3.3

  in Eq 2 as listed open-source 27

• p. 51 A.3.3, first paragraph

  for the margin term in Eq 2 as listed open-source 27

• p. 51 A.3.3

  as listed open-source 27

Judge's note: The page 51 image shows the same wording, so this is not an extraction artefact. The number 27 is right (Table 27 lists the margin variants), but 'open-source' stands where 'in Table' should be.

Second check: none (judge only)

• p. 51 A.2.2 Mathematical Reasoning

  we report results for Llama 2 and other open-source datasets on the GSM8k and MATH tasks

Judge's note: The Table 25 caption reads 'Comparison to other open-source models on mathematical reasoning tasks', and the table compares models, not datasets. This is a wrong word.

Second check: none (judge only)

• p. 52 Table 28 caption

  chosen response significantly better the rejected counterpart

Judge's note: The page image shows the caption with 'than' missing.

Second check: none (judge only)

• p. 53 Figure 26 caption

  for prompts samples with a models trained on each of the batches

Judge's note: The page image confirms the wording. The intended sense ('prompts sampled with models trained...') can be recovered, so this is cosmetic.

Second check: none (judge only)

• p. 54 A.3.5

  trough a human evaluation

• p. 54 A.3.5, Additional Results for GAtt

  remember the system arguments trough a human evaluation

Judge's note: Seen on the page 54 image. It is a plain misspelling.

Second check: none (judge only)

• p. 54 A.3.5 GAtt Zero-shot Generalisation

  We tried at inference time to set constrain not present

Judge's note: Seen on the page 54 image. The sentence needs a noun, and the Figure 28 caption itself uses 'constraints'.

Second check: none (judge only)

• p. 55 Figure 28 caption

  perfectly fulfilled trough all the turns

• p. 55 Figure 28 caption

  they are perfectly fulfilled trough all the turns

Judge's note: Seen on the page 55 image. This is a separate occurrence from the one in C036.

Second check: none (judge only)

• p. 58 A.3.7 Additional Results

  In 30, we also show the category wise breakdown

• p. 58 A.3.7 Additional Results

  In 31, we also share the win rate

• p. 58 A.3.7 Additional Results

  In 31, we also share the win rate by number of turns

Judge's note: The page 58 image shows both bare numbers. The same paragraph says 'As shown in Figure 30', and Figures 30 and 31 are the intended targets.

Second check: none (judge only)

• p. 58 A.4.2

  starts to be behave more conservatively

Judge's note: Seen on the page 58 image.

Second check: none (judge only)

• p. 64 Table 39, third preprompt

  friendly and responsible assistant assistant

• p. 64

  e an friendly and responsible assistant assistant. Please refrain fro

Only the authors can say what is right here.

Judge's note: Seen on the page-64 crops. The word "assistant" is printed twice in a row. Because this is a quoted preprompt, only the authors can say whether the string was used verbatim in that form. If it was, a [sic] would help; if not, it should be fixed.

Second check: code check

• p. 64 Table 39, first and second preprompts

  You are an responsible and safe assistant

• p. 64 Table 39, second preprompt

  You are an friendly and responsible assistant

Only the authors can say what is right here.

Judge's note: Both phrases appear on the page-64 crops, in the first and third preprompts of Table 39. The reviewer said 'second' for the 'friendly' preprompt, but it is the third. These are quoted preprompts and may be verbatim model inputs, so the authors would need to confirm whether to correct them or mark them [sic].

Second check: none (judge only)

• p. 75 A.6, first paragraph

  Earlier work (Brown et al. (2020), Wei et al. (2022a), Du et al. (2022) in measuring such dataset contamination

Judge's note: Seen on the page-75 crop. The outer parenthesis opened before 'Brown' has no closing match.

Second check: none (judge only)

• p. 76 Table 51 caption

  Models sizes refer to pretrained-only models

Judge's note: Seen on the p76 image, at the end of the Table 51 caption.

Second check: none (judge only)

• p. 77 Table 52, Ethical Considerations

  Responsible Use Guide available available at

• p. 77

  Please see the Responsible Use Guide available available at

• p. 77

  see the Responsible Use Guide available available at https://ai.meta.

Judge's note: Seen on the p77 crop. The word 'available' is printed twice.

Second check: code check