Mixtral of Experts
2024 · paper · AI and machine learning
Introduces Mixtral 8x7B, a sparse mixture-of-experts language model where a router selects two of eight expert blocks per token, using 13B of 47B parameters. It reportedly matches or outperforms Llama 2 70B and GPT-3.5, with an instruction-tuned version.
Checked: PDF from arxiv.org, fetched 4 Oct 2026. Page numbers are those of that PDF.
7 findings
Reference or label Page 4: "despite its significantly smaller capacity (47B tokens compared to 70B)" — 47B and 70B are parameter counts, not tokens.
- p. 4 Paragraph 'Comparison with Llama 2 70B and GPT-3.5'
capacity (47B tokens compared to 70B)
- p. 4 Comparison with Llama 2 70B and GPT-3.5
(47B tokens compared to 70B)
- p. 4 Comparison with Llama 2 70B and GPT-3.5
47B tokens compared to 70B
- p. 4
despite its significantly smaller capacity (47B tokens compared to 70B)
Judge's note: Seen in the crop and the page 4 image. The same page says 'its sparse parameter count, 47B' and the abstract says 'each token has access to 47B parameters'. The only token figure in the paper is the 32k context size. 'Tokens' is the wrong unit; it should be 'parameters'.
Second check: another model agreed
Report a problem with this finding
Reference or label Section 4: "Mixtral – Instruct reaches a score of 8.30 on MT-Bench [33] (see Table 2)" — the MT Bench score is in Table 3; Table 2 has no MT-Bench column.
- p. 6 Section 4, first paragraph
score of 8.30 on MT-Bench [33] (see Table 2)
- p. 5 Table 3, MT Bench row
8.30
- p. 5 Table 3
MT Bench
- p. 5 Table 3
MT Bench 6.86 8.32 8.30
Judge's note: Opened both tables. Table 2 (page 4) has columns Active Params, MMLU, HellaS, WinoG, PIQA, Arc-e, Arc-c, NQ, TriQA, HumanE, MBPP, Math and GSM8K, with no MT-Bench entry. Table 3 (page 5) has the row 'MT Bench (for Instruct Models)' with 6.86, 8.32 and 8.30, the last for Mixtral 8x7B. The pointer on page 6 should be to Table 3.
Second check: another model agreed
Report a problem with this finding
Reference or label Section 3.3 says "report the results in Table 5" for the BBQ/BOLD bias results, but they are in the float captioned "Figure 5: Bias Benchmarks"; Table 5 (page 8) is "Percentage of expert assignment repetitions".
- p. 6 Section 3.3, paragraph after the float
the results in Table 5
- p. 6 Float caption beside Section 3.3
Figure 5: Bias Benchmarks.
- p. 8 Table 5 caption
Table 5: Percentage of expert assignment repetitions.
- p. 6 Section 3.3, paragraph below the float
report the results in Table 5
Judge's note: Viewed page 6 and page 8. The bias results sit in the float on page 6 captioned Figure 5. Table 5 on page 8 is the expert-repetition table and Section 5 cites it correctly for that purpose. No other Table 5 exists. The pointer should read Figure 5.
Second check: another model agreed
Report a problem with this finding
Wording Abstract: "it outperforms or matches Llama 2 70B and GPT-3.5 across all evaluated benchmarks", but Tables 2 and 3 show Mixtral below Llama 2 70B on HellaSwag, WinoGrande and TriviaQA, and below GPT-3.5 on WinoGrande and MT Bench; the Introduction says "most benchmarks".
- p. 1 Abstract
or matches Llama 2 70B and GPT-3.5 across all evaluated benchmarks
- p. 5 Table 3, WinoGrande row
83.2% 81.6% 81.2%
- p. 4 Table 2, LLaMA 2 70B vs Mixtral 8x7B, WinoG column
80.4%
- p. 1 Introduction, first paragraph
Mixtral outperforms Llama 2 70B and GPT-3.5 on most benchmarks
Judge's note: Read the abstract and Introduction on page 1, Table 2 on page 4 and Table 3 on page 5 from the page images. Table 2: Llama 2 70B is bolded as best on HellaS (85.4% vs 84.4%), WinoG (80.4% vs 77.2%) and TriQA (73.0% vs 71.5%). Table 3: Mixtral is the lowest of three on WinoGrande (83.2% / 81.6% / 81.2%), below Llama 2 70B on HellaSwag (87.1% vs 86.7%) and below GPT-3.5 on MT Bench (8.32 vs 8.30). Gaps of 2 to 3.2 points are not 'matches'. The paper's own weaker wording elsewhere (Introduction 'most benchmarks', Table 2 caption 'almost all', Table 3 caption 'most metrics') shows the abstract's 'all' is an overstatement. No statement in the paper restricts the abstract's claim to a subset. The same overstatement in the Figure 2 caption is handled in C006.
Second check: another model agreed
Report a problem with this finding
Wording Introduction: "Mixtral – Instruct also demonstrates reduced biases, and a more balanced sentiment profile in benchmarks such as BBQ, and BOLD", but Section 3.3 says "we measure the base model performance on" BBQ and BOLD, and Figure 5 reports only "Mixtral 8x7B".
- p. 2 Introduction, paragraph on Mixtral 8x7B – Instruct
Mixtral – Instruct also demonstrates reduced biases, and a more balanced sentiment
- p. 6 Section 3.3
measure the base model performance on
Only the authors can say what is right here.
Judge's note: Read the Introduction on page 2 and Section 3.3 with Figure 5 on page 6. The only BBQ/BOLD results in the paper are for the base model, measured 'to identify possible flaws to be corrected by fine-tuning / preference modeling'. Section 4 on the Instruct model reports MT-Bench and the LMSys leaderboard only, with no bias results. So the Introduction's claim about Mixtral – Instruct has no supporting data in the paper. Only the authors can say whether the Introduction names the wrong model or Instruct results were omitted.
Second check: another model agreed
Report a problem with this finding
Wording Figure 2 caption: "Mixtral outperforms or matches Llama 2 70B on all benchmarks." — the figure's own bars show Llama 2 70B above Mixtral 8x7B on Comprehension, AGI Eval and BBH.
- p. 3 Figure 2 caption
Mixtral outperforms or matches Llama 2 70B on all benchmarks.
- p. 3 Figure 2, bar groups Comprehension, AGI Eval, BBH
AGI Eval
- p. 4 Figure 3 caption
except on reading comprehension benchmarks
- p. 3 Figure 2 caption
Mixtral outperforms or matches Llama 2 70B on all benchmarks
Judge's note: Viewed Figure 2 in the page 3 image and the crop. The dark green LLaMA 2 70B bar is taller than the yellow Mixtral 8x7B bar for Comprehension (about 67 vs 65.5), AGI Eval (about 53.5 vs 51.5) and BBH (about 45 vs 42); heights are read by eye but the ordering is clear in all three groups. The Figure 3 caption on page 4 itself concedes the reading-comprehension exception, the body text says 'across most metrics', and the Table 2 caption says 'almost all'. Table 2 also shows losses on HellaS, WinoG and TriQA. Kept separate from C001 because this is a different sentence (figure caption, Llama 2 70B only) contradicted by the figure itself.
Second check: another model agreed
Report a problem with this finding
Wording Figure 7 caption ("The gray dashed vertical line marks 1/8") and Figure 9 caption ("The gray dashed vertical line marks 1/8") call the reference line vertical; in both figures it is horizontal, at 0.125 on the selection-proportion axis.
- p. 7 Figure 7 caption
The gray dashed vertical line marks 1/8
- p. 12 Figure 9 caption
The gray dashed vertical line marks
Judge's note: Viewed pages 7 and 12. Every panel has a gray dashed horizontal line at about 0.125 on the y-axis and no dashed vertical line. The captions should say "horizontal". Readers are not misled about the data.
Second check: another model agreed
Report a problem with this finding
11 minor (typos and the like; most checked by the judge only)
Reference or label The same model is "LLaMA 1 33B" (Active Params 33B) in Tables 2 and 4 but "Llama 1 34B" in the body text, footnote 2 ("we report results for Llama 1 34B"), the Figure 2 legend and the Figure 3 x-axis.
- p. 4 Table 2, third row
LLaMA 1 33B
- p. 5 Table 4, first row
LLaMA 1 33B
- p. 4 Text below Figure 3
Llama 1 34B
- p. 4 Footnote 2
we report results for Llama 1 34B
Only the authors can say what is right here.
Judge's note: Checked on the page images: Table 2 (page 4) and Table 4 (page 5) both have the row 'LLaMA 1 33B' with 33B active parameters; the text under Figure 3 and footnote 2 on page 4 say 'Llama 1 34B'; the Figure 2 legend (page 3) says 'LLaMA 1 34B' and the Figure 3 axes have a 34B tick. It is clearly one model named two ways, so this is a labelling inconsistency and the results are unaffected. The inconsistency itself is internal to the paper; which label is the right one is for the authors.
Second check: another model agreed
Report a problem with this finding
Typo Section 2.1: "given n expert networks {E0, Ei, ..., En−1}" — the second element should be E1.
- p. 2 Section 2.1, before the first equation
given n expert networks {E0 , Ei , ..., En−1 }
- p. 2 Section 2.1
{E0, Ei, ..., En−1}
Judge's note: Checked in the crop and the page 2 image: the set is typeset as E_0, E_i, ..., E_{n-1}. An enumeration from 0 to n-1 needs E_1 as the second element; there is no ellipsis before E_i that would make it a generic term. The sum below runs i=0 to n-1. Typographical slip with no effect on meaning.
Second check: none (judge only)
Report a problem with this finding
Typo Section 2.1: "G(x)_i denotes the n-dimensional output of the gating network for the i-th expert" — G(x) is n-dimensional; G(x)_i is its scalar i-th component.
- p. 2 Section 2.1, after the first equation
denotes the n-dimensional output of the gating network for the i-th expert
Judge's note: Read on the page 2 image. G(x)_i is used as a scalar weight in the sum, and the later definition has logits in R^n, so G(x) is the n-dimensional vector and G(x)_i one coordinate. The sentence is loosely worded, but the intended meaning (the i-th entry of the n-dimensional gating output) is clear from context, so this is a minor wording imprecision.
Second check: none (judge only)
Report a problem with this finding
Typo Section 2: "Mixture-of-Expert layers" where the paper elsewhere writes "Mixture of Experts" / "Mixture-of-Experts".
- p. 2 Section 2
Mixture-of-Expert layers
Judge's note: Seen on the page 2 image and crop. The next heading and sentence use 'Mixture of Experts layer', and page 4 has 'sparse Mixture-of-Experts model'. A small inconsistency in the term; the singular form in a compound modifier is defensible.
Second check: none (judge only)
Report a problem with this finding
Typo Section 3: "categorized as follow:" should read "as follows".
- p. 3 Section 3, first paragraph
categorized as follow:
Judge's note: Seen on the page 3 image. Grammatical slip.
Second check: none (judge only)
Report a problem with this finding
Typo Figure 3 caption: "for Mistral (7B/8x7B) vs Llama 2 (7B/13B/70B)" — the 8x7B model is Mixtral, not Mistral.
- p. 4 Figure 3 caption
for Mistral (7B/8x7B) vs Llama 2 (7B/13B/70B)
Judge's note: Seen on the page 4 image. The points in the figure are labelled 'Mistral 7B' and 'Mixtral 8x7B', and the legend uses 'Mistral' for the orange line covering both. 'Mistral' here can be read as the family or company name parallel to 'Llama 2', so either form could be intended; it is at most a loose label.
Second check: none (judge only)
Report a problem with this finding
Typo Page 4: "the SMoEs layer" — plural abbreviation with a singular noun; the abbreviation is defined as SMoE.
- p. 4 Size and Efficiency paragraph
the SMoEs layer
Judge's note: Seen in the crop of page 4. The abstract defines 'Sparse Mixture of Experts (SMoE)'. The next sentence continues with 'They are more suitable', so either 'the SMoE layer' or 'SMoE layers' was meant. Cosmetic.
Second check: none (judge only)
Report a problem with this finding
Typo Figure 5 caption: "Compared Llama 2 70B, Mixtral presents less bias" is missing "to" ("Compared to Llama 2 70B").
- p. 6 Figure 5 caption
Compared Llama 2 70B,
- p. 6 Figure 5 caption
Compared Llama 2 70B
Judge's note: Visible in the page 6 image. The body text below uses "Compared to Llama 2".
Second check: none (judge only)
Report a problem with this finding
Typo Figure 7 caption: "A breakdown of the proportion of assignments done in each case cane be seen in Figure 9 in the Appendix"; "cane" should be "can".
- p. 7 Figure 7 caption, last sentence
cane be seen in Figure 9
- p. 7 Figure 7 caption
in each case cane be seen in Figure 9
- p. 7 Figure 7 caption
cane be seen
- p. 6 Figure 5 caption
Compared Llama 2 70B
Judge's note: Visible in the page 7 image, so not an extraction artefact. One reviewer note also mentions "Compared Llama 2 70B", which is handled in C018.
Second check: none (judge only)
Report a problem with this finding
Typo Table 5 caption: "≈ 46% for “First and second choice”", where the column header reads "First or second choice".
- p. 8 Table 5 caption
for “First
- p. 8 Table 5 column header
First or second choice
- p. 8 Table 5 caption
≈ 46% for “First
Judge's note: Viewed page 8. The caption quotes the column name with "and"; the table header and the Figure 10 panel title say "First or second choice". The formula 1-(6/8)(5/7) is the probability for the "or" event, so "and" is a slip.
Second check: none (judge only)
Report a problem with this finding
Typo Reference [29] author list: "Ed H Chi, Denny Zhou, , and Jason Wei." has a stray doubled comma (empty author slot).
- p. 10 Reference [29]
Denny Zhou, , and Jason Wei
Judge's note: Visible in the crop of page 10, so not an extraction artefact.
Second check: none (judge only)