GPT-4 Technical Report
2023 · paper · AI and machine learning
Reports the development of GPT-4, a large multimodal model that accepts image and text inputs and produces text. It shows human-level performance on various professional and academic exams, though less capable than humans in many real-world settings, and covers predictable scaling and safety.
Checked: PDF from arxiv.org, fetched 4 Oct 2026. Page numbers are those of that PDF.
9 findings
Substance Table 10, USABO row: "3.00% 58.17% 58.17% 28.89% N/A". The overall score equals the non-contaminated score even though the contaminated subset scores much lower, and Degradation is N/A although it can be computed.
- p. 31 Table 10, USA Biolympiad Semifinal Exam 2020 row
3.00%
- p. 31 Table 10, same row
28.89%
- p. 31 Table 10, USA Biolympiad Semifinal Exam 2020
3.00% 58.17% 58.17% 28.89% N/A
Only the authors can say what is right here.
Judge's note: Checked against the p31 image. If 3% of the exam is contaminated and scores 28.89%, the overall and non-contaminated scores cannot be equal. Other rows fit the weighted mean, for example LSAT. 3% of 150 is 4.5, so the weighting may be by points rather than by questions, but that still cannot make the two scores equal. The authors need to clarify.
Second check: another model agreed
Report a problem with this finding
Reference or label System Card Figure 8 credits the baseline to "Askell et al [101]" (bars labelled 'Askell et al. 2022'), but [101] is Bai et al., "Constitutional AI: Harmlessness from AI Feedback"; main-report Figure 7 credits the same bars to "Anthropic-LM from Bai et al. [67]".
- p. 65 System Card Figure 8 caption and x-axis labels
GPT-4 significantly outperforms both GPT-3.5 and Askell et al [101]
- p. 78 System Card reference [101]
Constitutional AI: Harmlessness from AI Feedback
- p. 11 Main text Figure 7 caption (same data)
Anthropic-LM from Bai et al. [67]
- p. 11 Main report Figure 7 caption
GPT-4 significantly outperforms both GPT-3.5 and Anthropic-LM from Bai et al. [67].
Only the authors can say what is right here.
Judge's note: Page 65 shows the caption and the x-axis labels 'Askell et al. 2022', with the legend reading 'Anthropic-LM'. The page 78 reference list gives [101] first author Y. Bai, with Askell fourth. The page 11 main Figure 7, which shows the same chart, credits Bai et al. [67]. The author name, the reference number and the main-report attribution disagree, and only the authors know which source is intended.
Second check: another model agreed
Report a problem with this finding
Reference or label Table 10 caption defines "The degradation (non-contaminated percent minus contaminated)", but the printed Degradation column equals (non-contaminated − overall GPT-4)/overall GPT-4.
- p. 31 Table 10 caption
The degradation (non-contaminated percent minus contaminated) is generally small
- p. 31 Table 10, AP English Literature (MCQ) row
-17.50%
- p. 31 Table 10 caption
The degradation (non-contaminated percent minus contaminated)
- p. 31 Table 10, AP English Literature and Composition (MCQ) row
72.73% 60.00% 75.56% -17.50%
Judge's note: Checked against the p31 image. AP Eng Lit MCQ: the caption's formula gives -15.56, the printed value is -17.50%, and the relative-to-overall formula gives -17.50. LSAT: the caption's formula gives 19.51, the printed value is 10.01%, and the relative formula gives 10.01. The data hold together under a different definition from the one in the caption, so the caption's definition is wrong.
Second check: another model agreed
Report a problem with this finding
Reference or label The System Card roadmap reads "In Section 1, we outline some of the observed safety challenges … In Section 2, we discuss our process for deployment preparation … In Section 3, we conclude". The headings, however, are 1 Introduction, 2 GPT-4 Observed Safety Challenges, 3 Deployment Preparation, 4 System Safety and 5 Conclusion and Next Steps.
- p. 43 System Card §1.1, last paragraph
In Section 1, we outline some of the observed safety challenges in the development of GPT-4. In Section 2, we discuss our process for deployment preparation
- p. 44 System Card section heading
GPT-4 Observed Safety Challenges
- p. 44 System Card section heading
2 GPT-4 Observed Safety Challenges
- p. 61 System Card section heading
3 Deployment Preparation
Judge's note: The page 43 image shows the roadmap. The text layer has "1 Introduction" on p42. The heading images show "2 GPT-4 Observed Safety Challenges" (p44), "3 Deployment Preparation" (p61) and "5 Conclusion and Next Steps" (p68). The text layer has "4 System Safety" on p66. Every pointer is one too low. The separate refusals pointer on p47 is handled in C028.
Second check: another model agreed
Report a problem with this finding
Reference or label §2.3 reads "Our work on model refusals (described in Section 2)", but the sentence is itself inside Section 2, and refusal training is described in Section 3.1, Model Mitigations.
- p. 47 System Card §2.3
Our work on model refusals (described in Section 2)
- p. 47 §2.3 Harmful Content
Our work on model refusals (described in Section 2) aimed to reduce the tendency of the model
- p. 61 §3.1 Model Mitigations
Model Mitigations
Judge's note: The p47 crop shows the sentence in §2.3. The text layer has "3.1 Model Mitigations" on p61, under "3 Deployment Preparation". This is a separate sentence from the C024 roadmap and probably has the same off-by-one cause.
Second check: another model agreed
Report a problem with this finding
Reference or label §2.6 refers to "the section above on Interactions with other systems", but that section is §2.10, which comes later (p56).
- p. 52 System Card §2.6
as highlighted in the section above on Interactions with other systems
Judge's note: The crop on p52 shows the sentence. The text layer has the §2.6 heading on p52 and §2.10 "Interactions with other systems" after it, on p56. Earlier, the topic appears only as an item in the Section 2 list of topics, not as a section, so the pointer should say "below".
Second check: another model agreed
Report a problem with this finding
Reference or label System Card §2.9 cites "there is evidence that existing models can identify power-seeking as an instrumentally useful strategy.[29]", but [29] is Perez et al., "Red Teaming Language Models with Language Models," Feb. 2022; the model-written-evaluations paper that fits the claim is [19].
- p. 55 System Card §2.9
existing models can identify power-seeking as an instrumentally useful strategy.[29]
- p. 73 System Card reference [29]
Red Teaming Language Models with Language Models
- p. 73 Reference [29]
“Red Teaming Language Models with Language Models,” Feb. 2022.
- p. 72 Reference [19]
“Discovering Language Model Behaviors with Model-Written Evaluations,” Dec. 2022.
Only the authors can say what is right here.
Judge's note: Rests on outside knowledge: that Perez et al. 2022 'Red Teaming LMs with LMs' does not study power-seeking, while 'Discovering Language Model Behaviors with Model-Written Evaluations' does. Page 55 image shows the sentence citing [29]. Page 73 image shows [29] = Red Teaming Language Models with Language Models, and page 72 shows [19] = Discovering Language Model Behaviors with Model-Written Evaluations. The titles point to a likely mix-up of the two Perez et al. references; only the authors can say which was meant.
Second check: another model agreed
Report a problem with this finding
Reference or label The System Card Figure 7 caption describes "Left:" and "Right: Moderation API trigger rates on the disallowed categories" panels, but the figure has a single panel, "Incorrect Behavior Rate on Disallowed and Sensitive Content".
- p. 63 System Card Figure 7 caption
Right: Moderation API trigger rates on the disallowed categories
- p. 63 Figure 7 plot
Incorrect Behavior Rate on Disallowed and Sensitive Content
Only the authors can say what is right here.
Judge's note: The full page 63 image shows one bar chart (Sensitive/Disallowed prompts; text-davinci-003, gpt-3.5-turbo, gpt-4). There is no Moderation API panel. The caption does not match the published figure, and only the authors can say whether the panel should be restored or the caption shortened.
Second check: another model agreed
Report a problem with this finding
Reference or label System Card §3.1 cites "evaluations such as TruthfulQA[34]", but [34] is Evans et al., "Truthful AI: Developing and governing AI that does not lie"; TruthfulQA is [32] (Lin, Hilton, Evans).
- p. 64 System Card §3.1, last paragraph
evaluations such as TruthfulQA[34]
- p. 73 System Card references [32] and [34]
Truthful AI: Developing and governing AI that does not lie
- p. 64 System Card §3.1, last paragraph
by evaluations such as TruthfulQA[34]
- p. 73 System Card references
[34] O. Evans, O. Cotton-Barratt, L. Finnveden
Judge's note: The page 64 image shows 'TruthfulQA[34]'. The page 73 reference list shows [32] = 'TruthfulQA: Measuring How Models Mimic Human Falsehoods' and [34] = 'Truthful AI...'. The reference list itself shows the citation number is wrong.
Second check: another model agreed
Report a problem with this finding
37 minor (typos and the like; most checked by the judge only)
Reference or label Reference [103] gives two years, "OpenAI, “Introducing chatgpt,” November 2022 2020."; [104] repeats its year, "June 2020 2020".
- p. 78 System Card reference [103]
“Introducing chatgpt,” November 2022 2020.
- p. 78 References [103], [104]
“Introducing chatgpt,” November 2022 2020
- p. 78 Reference [104]
“Openai api,” June 2020 2020
Judge's note: The page 78 image confirms both entries. [103] contradicts itself (November 2022 vs 2020). [104] has only a redundant duplicate year. Both look like bibliography-template artefacts; the fix is to drop the trailing year.
Second check: another model agreed
Report a problem with this finding
Wording Intro says GPT-4 'surpasses the English-language state-of-the-art in 24 of 26 languages considered', but Table 2 labels 75.2% (Flan-PaLM) as MMLU 'SOTA'. Only 19 of 26 languages in Figure 5 exceed that figure; 24 of 26 exceed the language-model baselines (≤70.7%).
- p. 1 Introduction, paragraph 3
surpasses the English-language state-of-the-art in 24 of 26 languages considered
- p. 7 Table 2, MMLU row
75.2%
- p. 8 Figure 5
GPT-3.5-English
Only the authors can say what is right here.
Judge's note: Checked p1, the Table 2 image on p7 and Figure 5 on p8. Figure 5 values above 75.2%: Italian through Icelandic, 19 languages. Above 70.1% (or the 70.7% LM SOTA): those 19 plus Bengali 73.2, Urdu 72.6, Nepali 72.2, Thai 71.8 and Punjabi 71.4, giving 24; Marathi 66.7 and Telugu 62.0 fall below. The p7 text and the Figure 5 caption compare against 'existing language models', which is consistent with 24/26. Only the intro uses 'state-of-the-art', the same label Table 2 gives to the 75.2% column. This is an ambiguity in the wording rather than a wrong number.
Second check: another model agreed
Report a problem with this finding
Wording §3.1 'but using at most 10,000x less compute than GPT-4' and §3.2 'with at most 1, 000× less compute' state the bound the wrong way round. The abstract ('no more than 1/1,000th the compute') and Figures 1–2 show the small models used at least that much less compute.
- p. 2 §3.1
but using at most 10,000x less compute than GPT-4
- p. 2 §3.2
with at most 1, 000× less compute (Figure 2)
- p. 1 Abstract
no more than 1/1,000th the compute of
- p. 3 Figure 1 x-axis
100p
Judge's note: p2: both sentences read 'at most'. p3: in Figure 1 the largest observed point lies just below 100µ (1e-4), and in Figure 2 it lies at about 6e-4, below 0.001. Both are therefore at least 10,000× and at least 1,000× smaller than GPT-4 (compute=1). The abstract on p1 states the bound correctly as 'no more than 1/1,000th'. The §3 intro's '1,000×–10,000× less compute' is also consistent with 'at least'. The intended meaning can be recovered, so this is a wording fix ('at least').
Second check: another model agreed
Report a problem with this finding
Wording Appendix E says 'We conducted contamination checking to verify the test set for GSM-8K is not included in the training set (see Appendix D)', but Table 11 in Appendix D reports '~1%' contamination for GSM-8K and gives no non-contaminated score.
- p. 29 Appendix E
to verify the test set for GSM-8K is not included in the training set (see Appendix D)
- p. 32 Table 11, GSM-8K row
~1%
Only the authors can say what is right here.
Judge's note: Checked p29 (Appendix E) and the Table 11 image on p32: the GSM-8K contamination entry is ~1%, and the non-contaminated and degradation cells are '-'. Appendix D says substring matching can give false positives, which may account for the ~1%, but the paper does not say this for GSM-8K. The wording 'verify ... is not included' therefore overstates what Table 11 shows. A query for the authors.
Second check: another model agreed
Report a problem with this finding
Wording Table 11, DROP row: Degradation is printed as "0", but the non-contaminated score is "82.8* (subsample)" and the caption says "GPT-4's score on the entire subsample was 82.5".
- p. 32 Table 11, DROP row and caption
For DROP, GPT-4’s score on the entire subsample was 82.5.
- p. 32 Table 11, DROP (F1)
82.8* (subsample)
Only the authors can say what is right here.
Judge's note: Checked against the p32 image. Using the HumanEval convention in the same table (−2.12%), DROP would come out at about +0.36%. Against the 80.9 full score it would be about +2.35%. Neither is 0. The 0 may be a rounded value meant to read as negligible, but no text explains it.
Second check: another model agreed
Report a problem with this finding
Wording The §2.10 tool list (literature search, molecule search, "A web search", "A purchase check tool (checks if a SMILES string is purchasable...)", synthesis planner) differs from the Figure 5 prompt tools (Molecule search, "Purchase: Places an order for a compound", Patent Search, Modify compound, Email, Literature Answer).
- p. 56 System Card §2.10 bullet list
A purchase check tool (checks if a SMILES21 string is purchasable against a known commercial catalog)
- p. 57 System Card Figure 5 prompt
Purchase: Places an order for a compound.
Judge's note: Page 56 shows the bullet list followed by 'We note that the example in Figure 5 is illustrative'. The page 57 Figure 5 prompt defines a different tool set: it has no web search, Purchase places an order rather than checking a catalog, and it adds Patent Search and Email. The red teamer may have used more than one configuration, so this is a consistency point, not a clear error.
Second check: another model agreed
Report a problem with this finding
Wording In System Card Figure 9 the prompt asks "should the following text be classified as N0 or N1?", yet the policy defines N0/N1/N2 and the example classification is "Classification: N2".
- p. 67 System Card Figure 9
should the following text be classified as N0 or N1?
- p. 67 System Card Figure 9, example classification
Classification: N2
Judge's note: The page 67 image confirms the question offers only N0 or N1 while the shown answer is N2. The figure may reproduce the prompt verbatim, but as an illustration it is internally inconsistent.
Second check: another model agreed
Report a problem with this finding
Wording System Card Conclusion: "GPT-4 can still be vulnerable to adversarial attacks and exploits or, “jailbreaks,” and harmful content is not the source of risk." This appears to be missing 'only'.
- p. 68 System Card §5 Conclusion
and harmful content is not the source of risk
- p. 68
and harmful content is not the source of risk.
Only the authors can say what is right here.
Judge's note: The page 68 image confirms the wording. Read literally, it says harmful content is not a source of risk, which conflicts with the next sentence about latent potential to generate harmful content. It is probably a missing word, but the authors must confirm the intended phrasing.
Second check: another model agreed
Report a problem with this finding
Typo Table 2 HellaSwag SOTA cell reads '85.6' with no % sign, unlike every other accuracy cell; only the DROP F1 row is legitimately unitless.
- p. 7 Table 2, HellaSwag SOTA column
85.6
Judge's note: The Table 2 image on p7 shows '85.6' in the SOTA column (ALUM [53]). All other accuracy entries carry '%'.
Second check: none (judge only)
Report a problem with this finding
Typo p7 text reads 'English-language performance of GPT 3.5 and existing language models'; the model is written 'GPT-3.5' elsewhere, including in the same Table 2.
- p. 7 paragraph on multilingual MMLU
English-language performance of GPT 3.5 and existing language models
Judge's note: Seen in the p7 image, in the paragraph below Table 2: 'GPT 3.5', with no hyphen.
Second check: none (judge only)
Report a problem with this finding
Typo Figure 5 axis label reads 'Ukranian' (should be 'Ukrainian').
- p. 8 Figure 5 y-axis label
Ukranian
Judge's note: Visible in the Figure 5 image on p8. The label is part of the rendered figure, so this is not an OCR artefact.
Second check: none (judge only)
Report a problem with this finding
Typo In 'OpenAI adversarial testers', 'Daniel Kokotajlo' appears twice: out of order in 'Jeremiah Currier, Daniel Kokotajlo, David Dohan' and again in 'Hendrik Kirchner, Daniel Kokotajlo'.
- p. 16 Authorship, 'OpenAI adversarial testers'
Jeremiah Currier, Daniel Kokotajlo, David Dohan
- p. 16 OpenAI adversarial testers
Daniel Kokotajlo
- p. 16 OpenAI adversarial testers
Shyamal Anadkat
Judge's note: Seen in the p16 image and crop. Footnote 11 on p17 says 'All author lists sorted alphabetically.' The first occurrence breaks the order and repeats the name. The duplicate Shyamal Anadkat entry, also raised in this cluster, is handled separately as C008.
Second check: none (judge only)
Report a problem with this finding
Typo In 'OpenAI adversarial testers', 'Shyamal Anadkat' appears twice: 'Lama Ahmad, Shyamal Anadkat' and, out of order, 'Sarah Shoker, Shyamal Anadkat'.
- p. 16 Authorship, 'OpenAI adversarial testers'
Sarah Shoker, Shyamal Anadkat
Judge's note: Seen in the p16 image. The second occurrence breaks the alphabetical order that footnote 11 states.
Second check: none (judge only)
Report a problem with this finding
Typo A contributor's surname is spelled 'Mario Salterelli' (adversarial testers, p16) and 'Mario Saltarelli' (System administration & on-call support, p17).
- p. 16 OpenAI adversarial testers
Mario Salterelli
- p. 17 System administration & on-call support
Mario Saltarelli
Judge's note: Both spellings are visible in the page images. This is an internal inconsistency; which spelling is right cannot be settled from the files, although 'Saltarelli' is likely.
Second check: none (judge only)
Report a problem with this finding
Typo The Evaluation & analysis core-contributor list puts 'Benjamin Sokolowsky Capability prediction co-lead' before 'Jack Rae', against footnote 11 'All author lists sorted alphabetically.'
- p. 16 Evaluation & analysis, Core contributors
Benjamin Sokolowsky Capability prediction co-lead
Judge's note: The p16 image shows the order ... Mishkin, Sokolowsky, Rae, Voss ... ; Rae should come before Sokolowsky. The rest of the list is alphabetical by surname.
Second check: none (judge only)
Report a problem with this finding
Typo The adversarial testers list gives 'Joost Huizinga, Janko, Tomer Kaftan', a first name only; the same person appears as 'Janko Altenschmidt' in other lists.
- p. 16 OpenAI adversarial testers
Joost Huizinga, Janko, Tomer Kaftan
Judge's note: In the p16 image 'Janko' sits between Huizinga and Kaftan, which suggests the full name was lost. 'Janko Altenschmidt' appears in System card & broader impacts (p16) and Trust & safety monitoring (p17).
Second check: none (judge only)
Report a problem with this finding
Typo Launch partners & product operations list reads 'Teddy Lee,, Yaniv Markovski', with a doubled comma.
- p. 17 Launch partners & product operations
Teddy Lee„ Yaniv Markovski
- p. 17 Launch partners & product operations
Teddy Lee,, Yaniv Markovski
Judge's note: The double comma is visible in the p17 image and crop. The text layer renders it as '„'.
Second check: none (judge only)
Report a problem with this finding
Typo The company name is written 'CaseText and Stanford CodeX' in A.1 (p24) but 'Casetext' (four times) in the Acknowledgements (p18).
- p. 24 A.1 Sourcing
CaseText and Stanford CodeX
- p. 18 Acknowledgements
Casetext and Stanford CodeX
Judge's note: Both forms are seen in the images. The paper is internally inconsistent. That 'Casetext' is the company's own styling rests on outside knowledge.
Second check: none (judge only)
Report a problem with this finding
Typo Appendix A.8 example prompt: after 'Problem 6. <PROBLEM TEXT AND ANSWER CHOICES GO HERE>' the header reads 'Explanation for Problem 4: <MODEL EXPLANATION ...>'.
- p. 27 Appendix A.8, example multiple-choice prompt
Explanation for Problem 4: <MODEL EXPLANATION
- p. 27 Appendix A.8 template
Problem 6. <PROBLEM TEXT AND ANSWER CHOICES GO HERE> Explanation for Problem 4:
Only the authors can say what is right here.
Judge's note: p27 image: the explanation headers for Problems 4 and 5 match their numbers, but the Problem 6 slot is headed 'Explanation for Problem 4'. Only the authors can say whether this is a slip in the published text or a verbatim copy of the prompt actually used.
Second check: none (judge only)
Report a problem with this finding
Typo Table 9 caption reads "(e.g. codeforces, Unified Bar Exam)", but the exam is called "Uniform Bar Exam" in the first row of the same table and in Tables 1 and 10.
- p. 30 Table 9 caption
codeforces, Unified Bar Exam
- p. 30 Table 9 first row
Uniform Bar Exam
- p. 30 Table 9 caption
(e.g. codeforces, Unified Bar Exam)
- p. 30
(e.g. codeforces, Unified Bar Exam) contain no images nor contamination
Judge's note: On the page 30 image the caption says "Unified Bar Exam" and the first row says "Uniform Bar Exam (MBE+MEE+MPT)". This is a naming slip. The lower-case "codeforces" is also minor.
Second check: none (judge only)
Report a problem with this finding
Typo Table 9 caption: "a range is reported because many student receive the same final score" should read "many students receive".
- p. 30 Table 9 caption
many student receive the same final score
- p. 30
because many student receive the same final score
Judge's note: The page 30 image shows "many student receive". This is a number-agreement slip.
Second check: none (judge only)
Report a problem with this finding
Typo Table 17 user prompt: "Below is part of the InstuctGPT paper." should read "InstructGPT".
- p. 37 Table 17, user prompt
Below is part of the InstuctGPT paper.
Judge's note: The crop image shows "InstuctGPT". The model response in the same example spells it "InstructGPT". It could be a verbatim user prompt, so this is cosmetic.
Second check: none (judge only)
Report a problem with this finding
Typo System Card footnote 5: "dates back at least as far as the 1988". The article before the bare year is superfluous.
- p. 43 System Card footnote 5
at least as far as the 1988
- p. 43 System Card footnote 5
dates back at least as far as the 1988
Judge's note: The p43 image shows the footnote text as quoted.
Second check: none (judge only)
Report a problem with this finding
Typo "We conducted internal adversarial testing GPT-4-launch on March 10, 2023." is missing a preposition ("testing of").
- p. 45 System Card §2.1.1
We conducted internal adversarial testing GPT-4-launch
- p. 45 Section 2.1.1
We conducted internal adversarial testing GPT-4-launch on March 10, 2023.
Judge's note: The crop image confirms the wording. It also shows "namely“a" with no space before the quotation mark.
Second check: none (judge only)
Report a problem with this finding
Typo "We measured close domain hallucinations" should read "closed domain", the term used in the preceding sentence and in footnote 10.
- p. 46 System Card §2.2
We measured close domain hallucinations
- p. 46 Section 2.2 Hallucinations
We measured close domain hallucinations using automatic
Judge's note: The crop image shows "close domain" right after "closed domain and open domain contexts".
Second check: none (judge only)
Report a problem with this finding
Typo Figure 3 caption: "These examples demonstrates" should read "demonstrate".
- p. 50 System Card Figure 3 caption
These examples demonstrates
Judge's note: The crop image confirms the wording.
Second check: none (judge only)
Report a problem with this finding
Typo Stray word in System Card §2.11: "Similar trends his may manifest via GPT-4 in various ways".
- p. 58 System Card §2.11
Similar trends his may manifest
- p. 58 Section 2.11 Economic Impacts
Similar trends his may manifest via GPT-4
- p. 58
Similar trends his may manifest via GPT-4 in various
Judge's note: The page 58 crop shows 'Similar trends his may manifest'. The word 'his' is extraneous.
Second check: none (judge only)
Report a problem with this finding
Typo Misspelling in System Card §2.10: "they may inadvertantly correlate their decisions" (should be 'inadvertently').
- p. 58 System Card §2.10
inadvertantly
- p. 58
they may inadvertantly correlate their decisions and create
Judge's note: The page 58 image and the text layer both read 'inadvertantly'. Elsewhere the paper spells it 'inadvertently' (pp. 4, 59).
Second check: none (judge only)
Report a problem with this finding
Typo Apostrophes are missing throughout the System Card: "Weve found" and "users intentions" (p60), "OpenAIs Charter" (p59 fn 23), "OpenAIs Terms of Use" and "classifiers dont exacerbate" (p66), "OpenAIs policies" (p70), "models tendency" (p63).
- p. 60 System Card §2.13
Weve found that GPT-4 exhibits enhanced steerability
- p. 70 Acknowledgements
OpenAIs policies
- p. 59 footnote 23
OpenAIs Charter states
Judge's note: The page images on pp. 59, 60 and 70 show the missing apostrophes. Apostrophes render correctly nearby (e.g. 'we've', 'model's' on p60), so this is not a font artefact. The p66 instances are confirmed on the full page image.
Second check: none (judge only)
Report a problem with this finding
Typo A stray baseline "11" precedes footnote marker 30: "GPT-3.5 Turbo RLHF on 61.1% of prompts.11^30".
- p. 64 System Card §3.1, first paragraph
61.1% of prompts.11
- p. 64 System Card §3.1, first paragraph
GPT-3.5 Turbo RLHF on 61.1% of prompts.11
- p. 64 §3.1, para 2
GPT-3.5 Turbo RLHF on 61.1% of prompts.1130
Judge's note: The page 64 image shows a body-size '11' directly before the superscript 30. Footnote 30 is the relevant note, so the '11' looks like a leftover footnote mark from the main report (footnote 7 there).
Second check: none (judge only)
Report a problem with this finding
Typo Singular/plural slip: "GPT-3.5 produces toxic generation 6.48% of the time", following "GPT-4 produces toxic generations 0.73% of the time".
- p. 64 top of page, RealToxicityPrompts sentence
produces toxic generation 6.48% of the time
Judge's note: The page 64 image (top line) shows 'produces toxic generation 6.48% of the time'. The preceding clause on p63 uses 'generations'.
Second check: none (judge only)
Report a problem with this finding
Typo A leftover editing note is printed in the System Card Figure 8 caption: "...outperforms both GPT-3.5 and Askell et al [101].fixes to plot legend and title".
- p. 65 System Card Figure 8 caption
Askell et al [101].fixes to plot legend and title
Judge's note: The page 65 image shows the caption ending with 'fixes to plot legend and title', joined to the citation with no space. It is a revision note, not caption text.
Second check: none (judge only)
Report a problem with this finding
Typo RBRM prompt option (E) reads "The message much not invalidate the emotional state"; 'much' should be 'must'.
- p. 80 option (E)
The message much not invalidate
Judge's note: Checked in the crop: option (E) on p.80 prints 'much not invalidate' right after 'must start' and 'must not mention'. This is a real typo, but it sits inside a reproduced prompt, which may be the verbatim text the model was given, so it is filed as an observation.
Second check: none (judge only)
Report a problem with this finding
Typo RBRM prompt option (H) reads "If you have a additional questions, please"; the article is wrong ('any additional').
- p. 80 option (H)
If you have a additional questions, please
Judge's note: Checked in the crop: p.80 option (H) prints 'a additional questions'. This is a typo inside reproduced prompt text, which may be the verbatim text that was used.
Second check: none (judge only)
Report a problem with this finding
Typo RBRM prompt option (L) reads "suicide, self-multilation, and eating disorders"; 'self-multilation' is a misspelling of 'self-mutilation'.
- p. 80 option (L)
self-multilation
Judge's note: Checked in the crop: p.80 option (L) prints 'self-multilation'. This is a misspelling inside reproduced prompt text.
Second check: none (judge only)
Report a problem with this finding
Typo RBRM prompt option (H) opens a parenthesis that is never closed: "(e.g., actually complying with the request."
- p. 80 option (H)
(e.g., actually complying with the request.
Judge's note: Checked in the crop: p.80 option (H) ends 'unrelated to the refusal (e.g., actually complying with the request.' with no closing parenthesis before option (I) begins. This is a punctuation slip inside reproduced prompt text.
Second check: none (judge only)
Report a problem with this finding
Typo Figure 11 caption reads "Results on IF evaluations across GPT3.5, GPT3.5-Turbo, GPT-4-launch", while the figure axes and the text use 'GPT-3.5' and 'GPT-3.5 Turbo'.
- p. 98 Figure 11 caption
across GPT3.5, GPT3.5-Turbo, GPT-4-launch
Judge's note: Checked p.98: the axis labels read 'GPT-3.5 RLHF' and 'GPT-3.5 Turbo RLHF'. A grep finds 'GPT3.5' nowhere else in the text; the main text uses 'GPT-3.5 Turbo'. This is an inconsistency in model-name formatting only.
Second check: none (judge only)