GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding

• p. 8 Section 3.2, first sentence

  Due to the daunting size and computation demand of tensors in Algorithm 1

• p. 8 Section 3.2 first paragraph

  size and computation demand of tensors in Algorithm 1

• p. 8 Section 3.2 first paragraph

  computation demand of tensors in Algorithm 1

Only the authors can say what is right here.

Judge's note: Page 8 image: Algorithm 1 (p.7) is 'Group-level top-2 gating with auxiliary loss'. The next sentence says the sharding of 'each tensor in the algorithm is illustrated by underscored letters in Algorithm 2', and the annotated code that follows is Algorithm 2. So the pointer most likely should read Algorithm 2. Algorithm 2 contains the gating, so Algorithm 1 is not strictly wrong; only the authors can say which was meant.

Second check: another model agreed

• p. 15 Section 4.3

  as described in Section 2.1

• p. 15 Section 4.3 strategy pillars

  as described in Section 2.2 and make use of learned routing modules to (sparsely) assign tokens to experts as described in Section 2.1

Judge's note: Page 15 image checked against the Section 2.1/2.2 text. Section 2.2 (Position-wise MoE Layer) defines both the expert FFNs (eq. 2) and GATE(.). Section 2.1 only introduces MoE replacement and top-2 gating briefly. The 2.1 pointer is loosely defensible, so this is an observation.

Second check: another model agreed

• p. 24 Model scaling paragraph

  Brown et al. showed that a non-sparse 175

Judge's note: The page 24 image confirms there is no bracketed citation; this is not an extraction artefact. The work is listed in the reference list as [6] (and again as [12]).

Second check: another model agreed

• p. 33 Figure 12, step 3

  (e.g., 0 left halo and 2 right halo for partition 2)

• p. 33 Figure 11 legend

  Right halo size for shard i: 1 + 1 * i

• p. 33 Figure 12 panel 3

  0 left halo and 2 right halo for partition 2

Only the authors can say what is right here.

Judge's note: I read Figures 11 and 12 and the text on page 33. The text uses zero-based numbering: 'Partition 1 ... needs to get 0 elements from Partition 0, and 2 elements from Partition 2'. Step 4, 'partition 3 has 4 elements in the invalid region', also fits zero-based numbering of the Figure 11 case. My own working: partition 3 covers input 11..15, so 1 element is valid and 4 are invalid, as the figure shows. The step 3 panel shows a slice with 0 left and 2 right halo, which is partition 1. Figure 12 is captioned as a 'general' halo exchange, so the authors should confirm which partition step 3 means.

Second check: another model agreed

• p. 9 Section 3.2 code listing, line 6

  Top2Gating(gating_logits)

• p. 8 Algorithm 2, line 2

  Top2Gating(gates)

Judge's note: Pages 8 and 9 images: the snippet otherwise copies Algorithm 2 line by line, and 'gating_logits' is never defined. It is a pseudocode inconsistency.

Second check: another model agreed

• p. 18 Section 4.4, 'Relaxing the Capacity Bottleneck' paragraph 2

  which can be seen by comparing 12 versus 36 layer models both with 128 experts

• p. 18 Section 4.4, 'Relaxing the Capacity Bottleneck', second paragraph

  which can be seen by comparing 12 versus 36 layer models both with 128 experts. Interestingly increasing the depth does not help as much if the capacity bottleneck is not relaxed.

Only the authors can say what is right here.

Judge's note: Page 18 places the bottleneck between 128 and 512 experts, so the 128E models are the unrelaxed case. The very next sentence says depth helps less when the bottleneck is not relaxed. In Table 3 the 128E depth gain is the smallest (+2.3, against +3.7 for 512E). The intended example was probably 512 or 2048 experts; only the authors can say.

Second check: another model agreed

• p. 19 Table 3, row (2) MoE(2048E, 12L)

  7.5

Only the authors can say what is right here.

Judge's note: Page 19 image. Footnote 14 defines core years as cores × wall-clock time. Rows (1) and (3)-(6) reproduce within rounding: 22.4, 15.4 vs 15.5 printed, 4.9, 6.1, 1.9. Row (2) needs at most 1.34 days to give 7.5, which would round to 1.3 days, not 1.4. The step rate (250k steps at 2.15 steps/s) gives about 1.35 days. Either the 1.4 or the 7.5 is off, and only the authors can say which.

Second check: another model agreed

• p. 1 Abstract

  can efficienctly

• p. 16 Figure 6 caption

  raging from high-

• p. 1 Abstract

  efficienctly

Judge's note: Both misspellings are visible in the page images: the abstract on p1 and the Figure 6 caption on p16.

Second check: none (judge only)

• p. 2 Figure 1 caption and x-axis

  Increasing the model size from 37.5B to 600B (16x)

• p. 17 Table 1, row (5)

  37B

Judge's note: Figure 1 (p2) shows 37.5B. Table 1 (p17) shows 37B, while the 12-layer model in the same column is printed to one decimal (12.5B). This looks like a rounding inconsistency for one model; the 16x ratio in the Figure 1 caption uses 37.5.

Second check: none (judge only)

• p. 2 Section 1.1, Super-linear scaling paragraph

  Straightforward scaling of the mode size

Judge's note: Visible in the p2 page image.

Second check: none (judge only)

• p. 2 Figure 1 caption

  results in computation cost increase from 6 to 22 years (3.6x)

Judge's note: Table 3 on p19 gives 22.4 and 6.1 TPU core-years for models (1) and (5). 3.6 appears to be a truncation of 3.67 rather than a rounding.

Second check: none (judge only)

• p. 4 Figure 2 caption

  is not-dependent of the number of devices

Judge's note: Visible in the p4 page image. A wording/grammar slip.

Second check: none (judge only)

• p. 5 Figure 3 panel headings

  Transfomer Encoder

Judge's note: Read from the panel headings in the p5 figure image.

Second check: none (judge only)

• p. 5 Section 2.2, paragraph after Eq. 3

  has one non-negative for each expert

Judge's note: Visible in the p5 page image.

Second check: none (judge only)

• p. 7 Section 3.1, paragraph 2

  non-zero vaules

• p. 7 Section 3.1

  at most two non-zero vaules

Judge's note: Visible in the p7 page image.

Second check: none (judge only)

• p. 7 Section 3.1, paragraph 1

  wi amd wo

• p. 7 Section 3.1

  single 3-D tensors wi amd wo

Judge's note: Visible in the p7 page image.

Second check: none (judge only)

• p. 7 Section 3.1 heading

  Positions-wise Mixture-of-Expert Layer

• p. 8 Algorithm 2 caption

  Positions-wise MoE layer

• p. 7 Section 3.1 heading

  Positions-wise Mixture-of-Expert Layer Expressed in Linear Algebra

• p. 8 Algorithm 2 caption

  Forward pass of the Positions-wise MoE layer

Judge's note: Seen on p7 (heading), p8 (Algorithm 2 caption) and p4 (Section 2.2 heading).

Second check: none (judge only)

• p. 7 Section 3.1, paragraph 2

  combine_weight[g, s, :, :]

Judge's note: The p7 page image shows combine_weights[g, s, e, c] and then combine_weight[g, s, :, :] in the same paragraph. Algorithm 2 (p8) uses combine_weights.

Second check: none (judge only)

• p. 8 Section 3.1, complexity analysis

  scaling with number the of devices D

Judge's note: Seen in the page 8 image, first line below Algorithm 2.

Second check: none (judge only)

• p. 9 end of page 9 / start of page 10

  the dispatching Einsum (Line 3) in Algorithm 2

Judge's note: The page 9 image shows the last line ending '(Line 3) in Algorithm 2'. The text layer for page 10 begins 'in Algorithm 2, which uses...', so the phrase is repeated.

Second check: none (judge only)

• p. 11 Section 3.3.2, paragraph 2

  the dimension to partitition

Judge's note: Seen in the page 11 image, Section 3.3.2, second paragraph.

Second check: none (judge only)

• p. 13 Section 3.3.3, Halo exchange

  partitinoning window-based operators

Judge's note: Seen in the page 13 image, Halo exchange paragraph.

Second check: none (judge only)

• p. 13 Section 3.3.3, Static operator configurations

  then slice out the the irrelevant parts

• p. 13

  a than needed, then slice out the the irrelevant parts. A

Judge's note: The page 13 image shows 'slice out the' at the end of one line and 'the irrelevant parts' at the start of the next.

Second check: code check

• p. 14 Section 4.1, paragraph 2

  compared to a models trained on a single language pair

Judge's note: Seen in the page 14 image, Section 4.1, paragraph 2.

Second check: none (judge only)

• p. 16 Figure 6 caption

  raging from high-

• p. 22 Table 4 caption

  kernal spatial dimensions

Judge's note: Both are seen in the images: the Figure 6 caption on page 16 and the Table 4 caption on page 22. The 'resouce' axis label is handled under C030.

Second check: none (judge only)

• p. 16 Figure 6 x-axis

  high-resouce languages

Judge's note: Seen in the page 16 image, Figure 6 x-axis.

Second check: none (judge only)

• p. 16 Figure 6 legend

  MoE(2048,36L) - 600B

Judge's note: I checked the page 16 image. The two 2048-expert legend entries drop the 'E', while the 512E and 128E entries and the table below include it. The meaning is still clear, so this is cosmetic.

Second check: none (judge only)

• p. 17 Section 4.4, 'Deeper Models...' paragraph

  three times the depth within each group:

Judge's note: The page 17 image shows the colon, and the next paragraph begins 'For each configuration shown in Fig. 6'. Either the punctuation is wrong or some content was dropped.

Second check: none (judge only)

• p. 20 Section 5.1, last paragraph

  On this other hand

• p. 20 Section 5.1

  On this other hand, weight memory

Judge's note: Seen in the crop image.

Second check: none (judge only)

• p. 22 Table 4 caption

  kernal spatial dimensions

Judge's note: Seen in the crop image.

Second check: none (judge only)

• p. 23 Figure 9 x-axis and legend

  Number of partitions (N)

Judge's note: Page 23 image. The caption says 'O(√D), where D is the number of partitions', and N is used elsewhere in the paper for something else. The meaning can be recovered from the caption, so this is a notation inconsistency only.

Second check: none (judge only)

• p. 24 Section 6, Parallelism paragraph

  each of which equiped with several accelerators

• p. 24 Parallelism paragraph

  each of which equiped with

Judge's note: Seen in the page 24 image.

Second check: none (judge only)

• p. 26 References [6] and [12]

  Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.

Judge's note: On the page 26 image, the author lists, titles and arXiv identifiers of the two entries match exactly. The same work is listed twice.

Second check: none (judge only)

• p. 26 References [6]

  [6] Tom B Brown, Benjamin Mann, Nick Ryder

• p. 26 References [12]

  [12] Tom B Brown, Benjamin Mann, Nick Ryder

Judge's note: I compared entries [6] and [12] on page 26 in the crop images. They match word for word: same authors, title, arXiv number and year. This is a copy-editing duplicate.

Second check: none (judge only)

• p. 27 Reference [22]

  Automatic differentiation in pytorch. 2017.

• p. 30 Reference [77]

  Automatic differentiation in

• p. 30 References [77]

  Automatic differentiation in pytorch. 2017.

• p. 26 References [6] and [12]

  Language models are few-shot learners

Judge's note: I compared [22] on page 27 with [77] on page 30. They match word for word. The duplicate pair [6]/[12] that some reviewers also mention is handled under C046; this cluster covers only [22]/[77].

Second check: none (judge only)