DeepSeek-V3 Technical Report
2024 · paper · AI and machine learning
Presents DeepSeek-V3, a mixture-of-experts language model with 671B total and 37B active parameters, pre-trained on 14.8 trillion tokens. It reportedly outperforms other open-source models, approaches leading closed models, and needed 2.788M H800 GPU hours with stable training.
Checked: PDF from arxiv.org, fetched 4 Oct 2026. Page numbers are those of that PDF.
3 findings
Substance Section 5.3.3 says "DeepSeek-V3 achieves an impressive win rate of over 86% against the baseline GPT-4-0314", but Table 7 reports 85.5 for DeepSeek-V3 on Arena-Hard.
- p. 33 Section 5.3.3, paragraph 1
DeepSeek-V3 achieves an impressive win rate of over 86% against the baseline GPT-4-0314
- p. 33 Table 7, DeepSeek-V3 row, Arena-Hard column
85.5
- p. 33 5.3.3 Open-Ended Evaluation
achieves an impressive win rate of over 86% against the baseline GPT-4-0314
- p. 33 Sec 5.3.3 text
win rate of over 86% against the baseline GPT-4-0314
Judge's note: Table 7 on p33 shows DeepSeek-V3 at 85.5 in the Arena-Hard column. The same paragraph later says it is the first open-source model to 'surpass 85%', which matches 85.5. The paper reports no other Arena-Hard figure or variant.
Second check: another model agreed
Report a problem with this finding
Reference or label Two different figures are both numbered Figure 10: "Figure 10 | Loss curves comparison between BF16 and FP8 training." (p.47) and "Figure 10 | Expert load of auxiliary-loss-free and auxiliary-loss-based models on three domains" (p.53). As a result, Appendix C's "as demonstrated in Figure 10" (p.48) is ambiguous.
- p. 47 Appendix B caption
Figure 10 | Loss curves comparison between BF16 and FP8 training.
- p. 53 Appendix C figure caption
Figure 10 | Expert load of auxiliary-loss-free and auxiliary-loss-based models on three domains
- p. 48 Appendix C text
tion across all layers, as demonstrated in Figure 10.
- p. 48 Appendix C text
as demonstrated in Figure 10.
Judge's note: I viewed pages 47, 48 and 53. Both captions read 'Figure 10' in the page images. B.1 cites Figure 10 for the loss curves, and Appendix C cites Figure 10 for the expert-load heatmaps. The expert-load figure should carry its own number (e.g. Figure 11), and the Appendix C pointer should be updated to match. The point in reviewer note 6 about panel layer ranges is handled under C019.
Second check: another model agreed
Report a problem with this finding
Wording "on AIME, MATH-500, and CNMO 2024, DeepSeek-V3 outperforms the second-best model, Qwen2.5 72B, by approximately 10% in absolute scores". Table 6 gives margins of 15.9 (AIME, where Qwen2.5 ties LLaMA-3.1 at 23.3), 10.2 (MATH-500) and 27.3 (CNMO).
- p. 32 Section 5.3.2, Code and Math Benchmarks
DeepSeek-V3 outperforms the second-best model, Qwen2.5 72B, by approximately 10% in absolute scores
- p. 31 Table 6, Math rows
43.2
- p. 32 Code and Math Benchmarks
outperforms the second-best model, Qwen2.5 72B, by approximately 10% in absolute scores
- p. 31 Table 6, Math rows
AIME 2024 (Pass@1)
Only the authors can say what is right here.
Judge's note: Read the Table 6 math rows on p31. AIME: 39.2 vs 23.3. MATH-500: 90.2 vs 80.0. CNMO: 43.2 vs 15.9. Only MATH-500 is about 10 points. The authors may have meant 'at least 10', so they need to clarify the wording.
Second check: another model agreed
Report a problem with this finding
10 minor (typos and the like; most checked by the judge only)
Wording Introduction says "DeepSeek-V3 outperforms all other open-source models, achieving 88.5 on MMLU", but Table 6 gives the open-source LLaMA-3.1 405B-Inst. 88.6 on MMLU (EM), with both values bolded.
- p. 6 Summary of Core Evaluation Results, Knowledge bullet
DeepSeek-V3 outperforms all other open-source models, achieving 88.5 on MMLU
- p. 31 Table 6, MMLU (EM) row
88.6
Only the authors can say what is right here.
Judge's note: Checked the p6 Knowledge bullet and the Table 6 MMLU row on p31. LLaMA-3.1-405B-Inst. scores 88.6 and DeepSeek-V3 scores 88.5, and both are bold. Section 5.3.2 (p32) itself says only "on par with" LLaMA-3.1-405B on MMLU. The claim does hold for MMLU-Pro and GPQA. So the overall wording overstates MMLU, where the result is at most a tie.
Second check: another model agreed
Report a problem with this finding
Wording Section 3.3 says both validation scales were trained "for approximately 1 trillion tokens", while Appendix B.1 gives "16B total parameters on 1.33T tokens" and "230B total parameters on around 0.9T tokens".
- p. 15 Section 3.3 intro
training for approximately 1 trillion tokens (see more details in Appendix B.1)
- p. 47 B.1
16B total parameters on 1.33T tokens
- p. 47 B.1
230B total parameters on around 0.9T tokens
Judge's note: Checked p15 and p47. The main-text summary is loose: 0.9T is close to 1T, but 1.33T is a third more. The wording is imprecise rather than contradictory, so this is a consistency observation.
Second check: another model agreed
Report a problem with this finding
Wording "DeepSeek-V3-Base comprehensively outperforms DeepSeek-V2-Base and Qwen2.5 72B Base" is not true on every row of Table 3: V2-Base is higher on WinoGrande, RACE-Middle, RACE-High, CMRC and CCPM, and Qwen2.5 is higher on CMMLU and RACE-Middle.
- p. 25 Section 4.4.2 text below Table 3
DeepSeek-V3-Base comprehensively outperforms DeepSeek-V2-Base and Qwen2.5 72B Base
- p. 25 Table 3, WinoGrande / RACE / CMRC / CCPM / CMMLU rows
86.3
Judge's note: Read Table 3 on p25. WinoGrande: V2 86.3 vs V3 84.9. RACE-Middle: V2 73.1, Qwen 68.1, V3 67.1. RACE-High: V2 52.6 vs V3 51.3. CMRC: V2 77.4 vs V3 76.3. CCPM: V2 93.0 vs V3 92.0. CMMLU: Qwen 89.5 vs V3 88.8. Every one of these gaps is larger than the caption's 0.3 'same level' threshold. The next paragraph admits the CMMLU exception for Qwen but not the V2 exceptions. 'Comprehensively' overstates the result; most rows do favour V3.
Second check: another model agreed
Report a problem with this finding
Wording On FRAMES, "DeepSeek-V3 closely trails GPT-4o while outperforming all other models by a significant margin", but Table 6 gives V3 73.3 against Claude-3.5-Sonnet 72.5, a lead of 0.8, while GPT-4o's 80.5 is 7.2 ahead.
- p. 32 Section 5.3.2, English Benchmarks
V3 closely trails GPT-4o while outperforming all other models by a significant margin
- p. 31 Table 6, FRAMES (Acc.) row
72.5
- p. 32 Sec 5.3.2 English Benchmarks
DeepSeek-V3 closely trails GPT-4o while outperforming all other models by a significant margin
- p. 31 Table 6, FRAMES row
FRAMES (Acc.)
Only the authors can say what is right here.
Judge's note: Read the Table 6 FRAMES row on p31 and the text on p32. The 0.8-point lead over Claude does not support 'significant margin', and the 7.2-point gap to GPT-4o is larger than that lead even though the same sentence calls it 'close'. This is a wording overstatement.
Second check: another model agreed
Report a problem with this finding
Typo The reference list has the same work twice: "B. D. Rouhani ... Microscaling data formats for deep learning. arXiv preprint arXiv:2310.10537, 2023a." (p.41) and the identical entry with "2023b." (p.42). The text cites it as both 2023a (p.4) and 2023b.
- p. 41 Reference list
Microscaling data formats for deep learning. arXiv preprint arXiv:2310.10537, 2023a.
- p. 42 Reference list
arXiv:2310.10537, 2023b.
- p. 42 References
Microscaling data formats for deep learning. arXiv preprint arXiv:2310.10537, 2023b.
- p. 4 Section 1, paragraph 3
Rouhani et al., 2023a
Judge's note: I compared the page-41 and page-42 crops. Authors, title and arXiv ID (2310.10537) are the same and only the suffix differs. The text cites 'Rouhani et al., 2023a' on p.4 and 'Rouhani et al., 2023b' in the microscaling-formats sentence. Both citations point to the same work, so the defect is cosmetic: the two entries should be merged into one reference.
Second check: none (judge only)
Report a problem with this finding
Typo In-text citation "(Fishman et al., 2024; He et al.; Sun et al., 2024)" has no year, and the reference entry for B. He et al., "Understanding and minimising outlier features in transformer training", gives no year either.
- p. 14 Section 3.3, first paragraph
(Fishman et al., 2024; He et al.; Sun et al., 2024)
- p. 38 Reference list
In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
Judge's note: Both page images show the year missing: the citation on p14 and the reference entry on p38, which names only the NeurIPS venue. The other entries carry years, so this is an incomplete reference.
Second check: none (judge only)
Report a problem with this finding
Typo Narrative citation used where a parenthetical one is needed: "co-processor like NVIDIA SHARP Graham et al. (2016)"; the same happens with "RACE Lai et al. (2017)" and "WinoGrande ... Sakaguchi et al. (2019)" in Section 4.4.1.
- p. 20 Section 3.5.1
NVIDIA SHARP Graham et al. (2016)
Judge's note: Seen on the p20 image. The text layer shows the same pattern for the RACE and WinoGrande citations, while neighbouring citations such as DROP (Dua et al., 2019) are parenthetical. This is a cosmetic citation-format lapse.
Second check: none (judge only)
Report a problem with this finding
Typo "trained on a larger corpus compromising 18T tokens": 'compromising' should be 'comprising'.
- p. 32 Section 5.3.2, Chinese Benchmarks
compromising 18T tokens
- p. 32 Chinese Benchmarks paragraph
trained on a larger corpus compromising 18T tokens
Judge's note: The word is visible on the p32 page image.
Second check: none (judge only)
Report a problem with this finding
Typo The reference list has duplicate entries: Qi et al. "Zero bubble pipeline parallelism" appears as both 2023a and 2023b with the same arXiv ID 2401.10241, and Rouhani et al. "Microscaling data formats for deep learning" appears as both 2023a and 2023b (arXiv:2310.10537). The Dettmers title is printed as "Gpt3. int8 ()".
- p. 41 Reference list
Zero bubble pipeline parallelism. arXiv preprint arXiv:2401.10241, 2023a.
- p. 41 Reference list
Zero bubble pipeline parallelism, 2023b.
- p. 41 References
Zero bubble pipeline parallelism, 2023b. URL https://arxiv.org/abs/2401.10241.
- p. 37 Reference list, Dettmers et al.
Gpt3. int8 (): 8-bit matrix multiplication
Judge's note: The p41 images show two Qi entries with identical authors, title and arXiv ID. The p41-42 text and the p42 image show two identical Rouhani entries, and the text cites both 2023a and 2023b of each. The p37 image shows the garbled Dettmers title. Rests on outside knowledge only for the Dettmers point: the true title begins 'LLM.int8()'. The duplicates themselves are visible in the files.
Second check: none (judge only)
Report a problem with this finding
Typo The Appendix C sub-captions use end-exclusive ranges without saying so. "(a) Layers 1-7" shows layers 1-6, "(d) Layers 19-25" shows layers 19-24, and "(e) Layers 25-27" shows only layers 25-26. Read as inclusive ranges, the captions overlap and name layers that the panels do not contain.
- p. 49 Sub-caption (a)
(a) Layers 1-7
- p. 50 Sub-caption (b)
(b) Layers 7-13
- p. 51 Sub-caption (c)
(c) Layers 13-19
- p. 53 Sub-caption (e)
(e) Layers 25-27
Judge's note: I viewed pages 49, 52 and 53. Panel (a) has titles for layers 1 to 6, panel (d) for 19 to 24, and panel (e) for 25 and 26 only. The captions match a half-open (Python-style) convention, but the paper never states it. Each heatmap carries its own explicit layer title, so no data are misattributed. This is a cosmetic labelling inconsistency: the captions should read 1-6, 7-12, ..., 25-26, or the convention should be stated.
Second check: none (judge only)