Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
2021 · paper · AI and machine learning
Introduces the Switch Transformer, a simplified mixture-of-experts model that activates different parameters per input at constant compute cost. It reports up to 7× faster pre-training than T5 baselines, multilingual gains, and trillion-parameter models with a 4× speedup over T5-XXL.
Checked: PDF from arxiv.org, fetched 4 Oct 2026. Page numbers are those of that PDF.
10 findings
Substance Table 1 caption says Switch-Base+ increases "the model hidden-size from 768 to 896 and the number of heads from 14 to 16", but Table 9 lists Switch-Base (d_model 768) with 12 heads.
- p. 9 Table 1 caption
the number of heads from 14 to 16
- p. 23 Table 9, Switch-Base row, Num. Heads
12
- p. 9 Table 1 caption
number of heads from 14 to 16
- p. 23 Table 9, Switch-Base row
Switch-Base
Only the authors can say what is right here.
Judge's note: The page 9 image shows the caption text; the page 23 image shows Table 9 with T5-Base and Switch-Base at d_model 768, d_kv 64 and Num. Heads 12. No 14-head base variant is described anywhere. With d_kv 64, 768 corresponds to 12 heads and 896 to 14, so the intended pair may be 12 to 14 or 12 to 16; the authors must say which.
Second check: another model agreed
Report a problem with this finding
Substance "Our Switch-Base 64 expert model achieves the same performance of the T5-Base model at step 60k at step 450k, which is a 7.5x speedup in terms of step time." The step counts are reversed.
- p. 12 Section 3.1
achieves the same performance of the T5-Base model at step 60k at step 450k
Judge's note: The page 12 image shows the sentence as quoted. As printed, Switch reaches at step 450k what T5-Base reaches at step 60k, which would be a slowdown. Figure 4 (right) shows the Switch curves well above T5-Base at every step, with T5-Base ending near -1.56 at 450k, a level the Switch-Base 64e curve passes early. The intended reading is that Switch at step 60k matches T5-Base at step 450k; 450/60 gives the stated 7.5x.
Second check: another model agreed
Report a problem with this finding
Substance Section 5.6 says "we find both Switch Transformer variants to improve over the T5-XXL version's negative log perplexity by over 0.061" after 250k steps, but Table 9 gives T5-XXL -1.147, Switch-XXL -1.086 and Switch-C "-1.096".
- p. 23 Section 5.6, Sample efficiency versus T5-XXL
negative log perplexity by over 0.061
- p. 23 Table 9, Neg. Log Perp. @250k
-1.096
- p. 23 Section 5.6, 'Sample efficiency versus T5-XXL'
we find both Switch Transformer variants to improve over the T5-XXL version’s negative log perplexity by over 0.061
- p. 23 Sample efficiency versus T5-XXL
by over 0.061
Only the authors can say what is right here.
Judge's note: Read Table 9 and the paragraph on page 23. The Switch-XXL gap is exactly 0.061 and the Switch-C gap is 0.051, so the statement does not hold for both variants. Footnote 10 says the difference is a lower bound because T5-XXL trained on an easier data set, but that does not change the tabulated Switch-C gap. The sentence may have been meant for Switch-XXL only; the authors should clarify.
Second check: another model agreed
Report a problem with this finding
Reference or label Table 11 caption says "measured by the negative log perplexity, under different randomness-strategies for selecting the expert (lower is better)", but the column header is "Quality (Neg. Log Perp.) (↑)" and the bolded best value is the highest (-1.468).
- p. 30 Table 11 caption
(lower is better)
- p. 30 Table 11 header
Quality (Neg. Log Perp.) (↑)
- p. 30 Table 11 caption
under different randomness-strategies for selecting the expert (lower is better)
- p. 30 Table 11 caption
the expert (lower is better)
Judge's note: Viewed page 30. The header carries an up arrow and the bolded entry is Input jitter at -1.468, the highest of the four values (-1.471, -1.570, -1.480, -1.468). Appendix C on page 29 says input jitter was found "to empirically perform the best". The caption's "lower is better" contradicts the header, the bolding and the text. This is a direction-label error; the data are unaffected.
Second check: another model agreed
Report a problem with this finding
Wording Table 6 caption says "a large sparse model with 100x more parameters", but Table 6 lists the teacher at "3,800M" and the student at "223M" (about 17x), and the text says "≈ 1/20th of the parameters".
- p. 17 Table 6 caption
a large sparse model with 100x more parameters
- p. 17 Table 6, Parameters column
3,800M
- p. 16 Section 4.2
with only ≈ 1/20th of the parameters
- p. 16 Distillation techniques
only ≈1/20th of the parameters
Only the authors can say what is right here.
Judge's note: Read Table 6, its caption and the Section 4.2 text on pages 16-17. The table's ratio is about 17x, in line with the body's ≈1/20th but not with 100x. Even the largest teacher in Table 7 (14.7B) is only about 66x. Section 7 separately speaks of 'compression rates of 10 to 100x', so the caption may be a loose general claim; the authors should say what figure is intended for this table.
Second check: another model agreed
Report a problem with this finding
Wording Table 6 caption says "we find no improvement of T5-Base initialized with the expert weights, but trained normally without distillation", while the table row reads "Init. non-expert weights from teacher".
- p. 17 Table 6 caption
initialized with the expert weights, but trained normally without distillation
- p. 17 Table 6, last row
Init. non-expert weights from teacher
- p. 17 Table 6 caption
we find no improvement of T5-Base initialized with the expert weights
- p. 17 Table 6 caption
initialized with the expert weights, but trained normally
Judge's note: Read on the page 17 image. The baseline row and the first sentence of the same caption both say non-expert weights, and Section 4.2 (page 16) explains that only non-expert layers have matching dimensions. The caption's last sentence should read 'non-expert weights'.
Second check: another model agreed
Report a problem with this finding
Wording Switch-XXL SQuAD score is given as "SQuAD accuracy on the validation set increases to 89.7" in Section 5.6 and as "89.6 for the smaller Switch-XXL model" in Section 8.
- p. 24 Section 5.6, Reasoning fine-tuning performance
increases to 89.7 versus state-of-the-art
- p. 26 Section 8, item 2
89.6 for the smaller Switch-XXL model
- p. 24 Section 5.6, Reasoning fine-tuning performance
SQuAD accuracy on the validation set increases to 89.7
- p. 24 Reasoning fine-tuning performance
increases to 89.7 versus state-of-the-art of 91.3
Only the authors can say what is right here.
Judge's note: Read on pages 24 and 26. Same model and benchmark, two values differing by 0.1. Section 5.6 describes a partially pre-trained checkpoint with multi-task training; Section 8 gives no setting. A text search found no statement distinguishing two Switch-XXL SQuAD evaluations, so the authors should say which value is right or whether the settings differ.
Second check: another model agreed
Report a problem with this finding
Wording Figure 15 applies exploration noise as "router_logits += mtf.random_uniform(shape=router_logits.shape, minval=1-eps, maxval=1+eps)" (additive, on the logits), whereas Appendix C describes "multiplicative jitter noise on the incoming representation".
- p. 34 Figure 15
router logits += mtf.random uniform(shape=router logits.shape, minval=1−eps, maxval=1+eps)
- p. 29 Appendix C
multiplicative jitter noise on the incoming representation
- p. 34 Figure 15, router pseudo-code
router_logits += mtf.random_uniform(shape=router_logits.shape, minval=1-eps, maxval=1+eps)
Only the authors can say what is right here.
Judge's note: Viewed pages 34 and 29. The pseudo-code uses "+=" with uniform noise in [1-eps, 1+eps] applied to router_logits. Appendix C lists option 4 as "multiplicative jitter noise on the incoming representation" and says input jitter is used throughout. Noise centred on 1 fits a multiplicative factor, so the operator, and arguably the tensor it is applied to, disagree with the prose. Only the authors can say whether the code should read "*=" and where the noise is applied.
Second check: another model agreed
Report a problem with this finding
Wording Figure 15 defines "def router(inputs, capacity_factor):", but the body uses expert_capacity and Figure 16 calls "router(inputs, expert_capacity)".
- p. 34 Figure 15, function signature
def router(inputs, capacity factor):
- p. 35 Figure 16
router(inputs, expert capacity)
- p. 34 Figure 15, function signature
def router(inputs, capacity_factor):
Judge's note: Viewed pages 34 and 35. The signature names the second parameter capacity_factor. The body uses expert_capacity in mtf.less(position_in_expert, expert_capacity) and in one_hot(..., dimension=expert_capacity), and never uses capacity_factor. Figure 16 computes expert_capacity = tokens_per_core * capacity_factor / num_experts and passes expert_capacity to router. The parameter name in the signature is therefore wrong; the evident fix is to rename it expert_capacity.
Second check: another model agreed
Report a problem with this finding
Wording Figure 16 combine step "mtf.einsum([expert_outputs, combine_tensor], reduce_dims=[tokens_per_core])" reduces a dimension the stated output keeps, and the comment "# expert_outputs shape: [num_experts, num_cores, tokens_per_core, d_model]" contradicts the reshape just above it.
- p. 35 Figure 16, combine step
mtf.einsum([expert outputs, combine tensor], reduce dims=[tokens per core])
- p. 35 Figure 16, comment
# expert outputs shape: [num experts, num cores, tokens per core, d model]
Only the authors can say what is right here.
Judge's note: Viewed page 35 and reworked the shapes. The line immediately above reshapes expert_outputs to [num_experts, num_cores, expert_capacity, d_model]. combine_tensor is [num_cores, tokens_per_core, num_experts, expert_capacity]. The stated result, expert_outputs_combined, is [num_cores, tokens_per_core, d_model]. Reaching that shape requires summing over num_experts and expert_capacity. Reducing tokens_per_core, as printed, would give [num_experts, num_cores, expert_capacity, d_model] and remove a dimension the output is said to keep. The comment also gives expert_outputs a tokens_per_core dimension it does not have. The printed reduce_dims appears carried over from the dispatch einsum, where reducing tokens_per_core is correct. It is a pseudo-code slip and the described algorithm is unaffected.
Second check: another model agreed
Report a problem with this finding
23 minor (typos and the like; most checked by the judge only)
Wording Introduction cites "MoE models have had notable successes in machine translation (Shazeer et al., 2017, 2018; Lepikhin et al., 2020)", but Related Work says of Shazeer et al. (2018) that "there were no accompanying NLP results".
- p. 3 Introduction, paragraph 2
MoE models have had notable successes in machine translation (Shazeer et al., 2017, 2018; Lepikhin et al., 2020)
- p. 25 Section 6 Related Work
however, there were no accompanying NLP results
Only the authors can say what is right here.
Judge's note: Checked in the text layer only (extract/main.txt, pages 3 and 25); no image opened, as nothing here is numeric. Both sentences are present as quoted. Related Work says the Mesh TensorFlow library (Shazeer et al., 2018) reintroduced MoE layers into the Transformer without NLP results, which sits oddly with citing the same work as an MoE machine-translation success. Only the authors can say whether the citation or the wording should change.
Second check: another model agreed
Report a problem with this finding
Wording Table 1 caption says "all our models required pre-training beyond 100k steps to achieve our level threshold of -1.50", then "T5-Base did not achieve this negative log perplexity in the 100k steps the models were trained".
- p. 9 Table 1 caption
all our models required pre-training beyond 100k steps to achieve our level threshold of -1.50
- p. 9 Table 1 caption, dagger footnote
T5-Base did not achieve this negative log perplexity in the 100k steps the models were trained
- p. 8 Section 2.3
All models were trained for the same number of steps on identical hardware
Only the authors can say what is right here.
Judge's note: On the page 9 image, every quality value at 100k steps is below -1.50 (the best is -1.534), yet time-to-threshold values are reported. Training must therefore have run past 100k steps, which contradicts the dagger note's "the 100k steps the models were trained". Section 2.3 only says all models were trained for the same number of steps, without giving the number. The training length is left ambiguous; the authors should clarify.
Second check: another model agreed
Report a problem with this finding
Wording Text says expert dropout "leads to performance improvements on four smaller downstream tasks" and the Table 4 caption says it is found "to perform the best"; Table 4 shows the (d=0.1, ed=0.4) row at 85.2/19.6/83.7/73.0 against Switch-Base (d=0.1) at 84.7/19.1/83.7/73.0 and (d=0.2) at 84.4/19.2/83.9/73.2.
- p. 11 Section 2.4, paragraph after Table 4
leads to performance improvements on four smaller downstream tasks
- p. 11 Table 4, rows Switch-Base (d=0.1) and (d=0.1, ed=0.4)
85.2 19.6 83.7 73.0
- p. 11 Table 4 caption
to perform the best
- p. 11 Table 4, rows Switch-Base (d=0.2) and (d=0.1, ed=0.4)
84.4 19.2 83.9 73.2
Only the authors can say what is right here.
Judge's note: Read from the page 11 image. Against Switch-Base d=0.1, expert dropout gains on GLUE (+0.5) and CNNDM (+0.5) and ties on SQuAD and SuperGLUE. Uniform d=0.2 is higher on SQuAD (83.9) and SuperGLUE (73.2). Against T5-Base it gains on three tasks and ties CNNDM. The text overstates the table: under no comparison are there four improvements, and the row is not best in every column.
Second check: another model agreed
Report a problem with this finding
Wording "We observe that simply increasing the dropout across all layers leads to worse performance." Table 4 shows d=0.2 (84.4/19.2/83.9/73.2) higher than d=0.1 (84.7/19.1/83.7/73.0) on three of four tasks.
- p. 11 Section 2.4
simply increasing the dropout across all layers leads to worse performance
- p. 11 Table 4, Switch-Base (d=0.2)
84.4 19.2 83.9 73.2
- p. 11 Section 2.4, paragraph after Table 4
We observe that simply increasing the dropout across all layers leads to worse performance.
Judge's note: Read from the page 11 image. Going from d=0.1 to d=0.2 lowers GLUE only; CNNDM, SQuAD and SuperGLUE rise slightly. At d=0.3, GLUE, SQuAD and SuperGLUE fall while CNNDM rises to 19.6. The blanket statement holds mainly for d=0.3 and is loose for d=0.2; this is a wording overstatement.
Second check: another model agreed
Report a problem with this finding
Wording Table 7 'Compression Percent' row "82 % 90 % 95 % 97 % 99 %" (and the text's "compressing 82%") does not follow from the 223M student and the printed teacher sizes 1.1B, 2.0B, 3.8B, 7.4B and 14.7B.
- p. 18 Table 7, Compression Percent row
82 %
- p. 17 Section 4.2, Achievable compression rates
gain of the 1.1B parameter model while compressing 82%
- p. 18 Table 7, Compression Percent row
82 % 90 % 95 %
- p. 17 Section 4.2, Achievable compression rates
while compressing 82%
Only the authors can say what is right here.
Judge's note: Read Table 7 on page 18 and the text on page 17. Recomputing with the 223M student gives about 79.7%, 88.9%, 94.1%, 97.0% and 98.5%. Rounding of the 1.1B teacher size (up to 1.15B gives 80.6%) cannot reach 82%, and Table 6 prints 3,800M for the 3.8B teacher, which gives 94%. No alternative definition of compression is stated. The gaps are 1-2 points, so this is a query about how the row was computed.
Second check: another model agreed
Report a problem with this finding
Wording Switch-XXL is described as "with nearly 10x larger FLOPs per sequence" (Section 5.6) and "applies ≈10x the FLOPS per token than the Switch-C model" (Section 8), but Table 9 lists FLOPs/seq of "6.3T" for Switch-XXL and "890B" for Switch-C, about 7x.
- p. 23 Section 5.6, Training instability
with nearly 10x larger FLOPs
- p. 26 Section 8, item 2
applies ≈10x the FLOPS per token
- p. 23 Table 9, FLOPs/seq
890B
- p. 23 Section 5.6, Training instability
with nearly 10x larger FLOPs per sequence
Only the authors can say what is right here.
Judge's note: Read Table 9 and the text on page 23, and Future Work item 2 on page 26. The ratio is about 7.1. The parameter ratio in the same Section 8 sentence (≈4x) does reproduce (1571/395 ≈ 3.98). No difference in sequence length is stated that would make per-token and per-sequence ratios differ. This is a loose approximation that overstates the table's ratio.
Second check: another model agreed
Report a problem with this finding
Typo The footnote on page 7 is numbered 2 ("2. A potential source of confusion"), duplicating footnote 2 on page 1 ("2. Tensorflow code for Switch Transformer is available") and falling between footnotes 3 (page 6) and 4 (page 8).
- p. 7 Footnote attached to the definition of P_i
2. A potential source of confusion
- p. 1 Footnote 2
2. Tensorflow code for Switch Transformer is available
Judge's note: The page 7 image shows the marker "2" after "expert i," and the footnote "2. A potential source of confusion". The text layer shows footnotes 1 and 2 on page 1 and footnote 3 on page 6; the page 8 image shows footnotes 4 and 5. The page 7 footnote is misnumbered.
Second check: none (judge only)
Report a problem with this finding
Typo Duplicated word: "small enough to not to overwhelm the primary cross-entropy objective".
- p. 8 Section 2.2, end of load-balancing loss paragraph
small enough to not to overwhelm
Judge's note: Visible on the page 8 image, first paragraph.
Second check: none (judge only)
Report a problem with this finding
Typo Table 2 Speed column bolds "1390" for Switch-Base (bfloat16) but not the identical "1390" for Switch-Base (Selective precision).
- p. 10 Table 2, Speed column
1390
Judge's note: The page 10 image shows 1390 bold in the bfloat16 row and plain in the Selective precision row. The unrounded speeds may differ (the caption says "nearly equal speed"), so this is a cosmetic inconsistency in how a tie is marked.
Second check: none (judge only)
Report a problem with this finding
Typo Table 2 caption misspelling: "with Selective prevision we notice similar learning dynamics" should read "Selective precision".
- p. 10 Table 2 caption
with Selective prevision we notice
- p. 10 Table 2 caption
after a fixed step count early in training its speed performance
- p. 10 Table 2 caption
with Selective prevision we notice similar learning
- p. 10 Table 2 caption
Selective prevision
Judge's note: Visible on the page 10 image. The row label in the same table reads "Selective precision". The missing-word point raised in one reviewer note is handled under C009.
Second check: none (judge only)
Report a problem with this finding
Typo Table 2 caption is missing a word: "We measure the quality of a 32 expert model after a fixed step count early in training its speed performance."
- p. 10 Table 2 caption
after a fixed step count early in training its speed performance
Judge's note: The page 10 image shows the sentence as quoted. A conjunction such as "and" is missing before "its speed performance". This is a separate defect from the "prevision" misspelling in C008.
Second check: none (judge only)
Report a problem with this finding
Typo Table 4 CNNDM column: "19.6" is bold for T5-Base (d=0.1) and Switch-Base (d=0.1, ed=0.4) but the same 19.6 for Switch-Base (d=0.3) is not bold.
- p. 11 Table 4, CNNDM column, Switch-Base (d=0.3)
19.6
Judge's note: Confirmed on the page 11 image. No bolding rule is stated. This is a cosmetic inconsistency.
Second check: none (judge only)
Report a problem with this finding
Typo Table 4 bolds non-maximal values: SQuAD "83.7" (two rows) is bold alongside the maximum 83.9, and SuperGLUE "73.0" (two rows) is bold alongside the maximum 73.2.
- p. 11 Table 4, SQuAD and SuperGLUE columns
83.7
- p. 11 Table 4, SQuAD and SuperGLUE columns
83.7 73.0
Judge's note: Confirmed on the page 11 image. No bolding criterion is stated, and GLUE 84.7 is not bold next to 85.2, so the convention is applied inconsistently. This is cosmetic and distinct from C012, which concerns an unbolded tie.
Second check: none (judge only)
Report a problem with this finding
Typo Misspelling: "sentence similarty (MRPC, STS-B, QQP)" should read "similarity".
- p. 15 Section 4.1, Fine-tuning tasks and data sets
sentence similarty
- p. 15 Fine-tuning tasks and data sets
sentence similarty (MRPC, STS-B, QQP)
Judge's note: Checked in the text layer only (extract/main.txt, page 15: "word sense disambiguation (WIC), sentence similarty (MRPC, STS-B, QQP)"); the page image was not opened. A plain misspelling of this kind is unlikely to be an extraction artefact.
Second check: none (judge only)
Report a problem with this finding
Typo Missing word 'for' in "The Rouge-2 metric is used both the CNNDM and XSum."
- p. 15 Section 4.1, Fine-tuning metrics
metric is used both the CNNDM and XSum
- p. 15 Fine-tuning metrics
The Rouge-2 metric is used both the CNNDM and XSum.
Judge's note: Read on the page 15 image (Fine-tuning metrics paragraph). The sentence is printed as quoted; 'for' is missing.
Second check: none (judge only)
Report a problem with this finding
Typo Misspelling in Section 5.4: "to analogusly receive the tokens from each expert" should read 'analogously'.
- p. 22 Section 5.4
to analogusly receive the tokens
- p. 22
to analogusly receive the tokens from each expert
Judge's note: Read on the page 22 image; printed as quoted.
Second check: none (judge only)
Report a problem with this finding
Typo Wrong word in Section 5.5: "how the number of experts effects the downstream performance" should read 'affects'.
- p. 22 Section 5.5
the number of experts effects the downstream performance
Judge's note: Read on the page 22 image; 'effects' is used as the verb where 'affects' is meant.
Second check: none (judge only)
Report a problem with this finding
Typo Misspelling in Section 6: "certain bit patterns occuring in the model hidden-states" should read 'occurring'.
- p. 24 Section 6, second paragraph
bit patterns occuring in the
Judge's note: Read on the page 24 image; printed as quoted.
Second check: none (judge only)
Report a problem with this finding
Typo Wrong word in Section 6: "these techniques are complimentary" should read 'complementary'.
- p. 25 Section 6, last paragraph
techniques are complimentary
- p. 25 Section 6 Related Work, last paragraph
these techniques are complimentary
Judge's note: Read on the page 25 image. The intended sense is 'complementary', which the paper spells correctly in Section 5 on page 18 ('complementary scaling strategies').
Second check: none (judge only)
Report a problem with this finding
Typo Misspelling in Appendix A: "dense feedfoward network (FFN)" should read 'feedforward'.
- p. 27 Appendix A
dense feedfoward network (FFN)
Judge's note: Read on the page 27 image; printed as quoted.
Second check: none (judge only)
Report a problem with this finding
Typo Table 10's last row is labelled "Expert FF + Attention", while the row two above it and the caption use "Experts FF + Attention".
- p. 28 Table 10 rows
Expert FF + Attention
Judge's note: Viewed page 28. Row 4 reads "Experts FF + Attention" (float32) and row 5 reads "Expert FF + Attention" (bfloat16); the caption uses "Experts FF + Attention". The same table also has "Expert Attention" rows, so the labelling is loose throughout, but the inconsistency for one model across two rows is real and cosmetic.
Second check: none (judge only)
Report a problem with this finding
Typo Figure 11 caption: "route them to the expert with which has the second highest probability" contains a stray word ("with").
- p. 30 Figure 11 caption
to the expert with which has the second highest probability
Judge's note: Viewed page 30. The caption reads exactly as quoted; it is a grammatical slip with a superfluous "with".
Second check: none (judge only)
Report a problem with this finding
Typo Figure 14 comments name variables that differ from the code: "# expert_index contains the expert with the highest router probability in one-hot format." describes the argument expert_mask, and "# density_l for a single core" / "# density_l_proxy" use the letter l where the code has density_1.
- p. 33 Figure 14, comments
# expert index contains the expert with the highest router probability in one−hot format.
- p. 33 Figure 14, comments
# density l for a single core
Judge's note: Viewed page 33 and the crop. The function signature is load_balance_loss(router_probs, expert_mask) and the next comment line gives the shape of expert_mask, so the one-hot tensor being described is expert_mask; in Figure 15 expert_index is the non-one-hot index. In the crop, the later comments show the letter "l" in density_l and density_l_proxy, whereas the code and the earlier comments use the digit 1. These are comment-level typos.
Second check: none (judge only)