Neural Architecture Search with Reinforcement Learning

• p. 12 References

  Densely connected convolutional networks. arXiv preprint arXiv:1608.06993, 2016a.

• p. 12 References

  Densely connected convolutional networks. arXiv preprint arXiv:1608.06993, 2016b.

• p. 12 References

  Gao Huang, Zhuang Liu, and Kilian Q. Weinberger. Densely connected convolutional networks. arXiv preprint arXiv:1608.06993, 2016a.

• p. 12 References

  Gao Huang, Zhuang Liu, Kilian Q. Weinberger, and Laurens van der Maaten. Densely connected convolutional networks. arXiv preprint arXiv:1608.06993, 2016b.

Only the authors can say what is right here.

Judge's note: Page 12 image shows both entries with the same title and the same arXiv ID; only the author lists differ. The text and Table 1 cite 2016a for the DenseNet rows and 2016b for DenseNet-BC, so a reader cannot tell which source each points to. The entries are probably meant to be different arXiv versions of one paper, but no version is given. The authors should merge the entries or state the version for each.

Second check: another model agreed

• p. 1 Abstract

  achieves a test error rate of 3.65, which is 0.09 percent better

Judge's note: Page 1 abstract checked. The arithmetic gives 0.09 percentage points. Writing "percent" for percentage points is common loose usage, so this is only a wording point.

Second check: another model agreed

• p. 2 Introduction, paragraph 2

  while being 1.05x faster than the current best model

• p. 7 Table 1, DenseNet-BC row

  3.46

• p. 8 Section 4.1 Results

  1.05x as fast as the DenseNet model that achieves 3.74%

Judge's note: Page 2 has no qualifier. The abstract limits the claim to "the previous state-of-the-art model that used a similar architectural scheme". Section 4.1 (p.8) names the comparison model as the 3.74% DenseNet and explains that the 3.46% DenseNet-BC uses 1x1 convolutions. Without the qualifier the introduction overstates the comparison; the fix is to restore it.

Second check: another model agreed

• p. 4 Section 3.3

  which has N −1 content-based sigmoids

• p. 4 Section 3.3, below the sigmoid equation

  ranges from 0 to N −1

Only the authors can say what is right here.

Judge's note: Page 4 checked. If j runs from 0 to N−1 there are N indices, which does not match N−1 sigmoids. The paper says (p.4–5) that the image is used as input when a layer has no input connections, so index 0 is not clearly a separate sigmoid. This is an off-by-one ambiguity in the indexing that only the authors can resolve.

Second check: another model agreed

• p. 8 Section 4.1, third results paragraph

  the best human-invented architecture that achieves 3.74%

• p. 8 Section 4.1, same paragraph

  The DenseNet model that achieves 3.46% error rate (Huang et al., 2016b)

• p. 7 Table 1

  3.46

Judge's note: Pages 7–8 checked. Later in the same paragraph the paper says the 3.46% DenseNet-BC uses 1x1 convolutions and is "not an exact comparison", which partly explains the choice. The phrase "the best human-invented architecture" still has no qualifier.

Second check: another model agreed

• p. 8 Section 4.2, Search space

  15,000, the number of architectures that we allow our controller to evaluate

• p. 10 Figure 6 x-axis and caption

  every 400 models run

• p. 8 Section 4.2 Search space

  much larger than 15,000, the number of architectures that we allow our controller to evaluate

• p. 10 Figure 6 x-axis and caption

  random search every 400 models run

Only the authors can say what is right here.

Judge's note: Page 10 Figure 6: the last plotted points sit just above 20,000. The paper does not say how "Iteration" relates to models evaluated; it might count random-search and controller models together. Only the authors can reconcile the two numbers.

Second check: another model agreed

• p. 8 Section 4.1, second results paragraph

  As stated earlier, this is more challenging because the search space is larger.

Only the authors can say what is right here.

Judge's note: I searched the text for "stride", "larger" and "challenging". The only earlier mentions of stride are in the Figure 2 caption and the p.7 search-space sentence, and neither says that predicting strides is harder or makes the search space larger. The back-reference has nothing to point to.

Second check: another model agreed

• p. 9 Transfer Learning Results

  settings with 5-6M parameters confirm that the new cell does indeed generalize

• p. 10 Table 3, our rows

  6.57M

• p. 9 Transfer Learning Results

  The results on small settings with 5-6M parameters confirm that the new cell does indeed generalize

Only the authors can say what is right here.

Judge's note: Page 9 text and Table 3 on the page 10 image. In Table 3, Two layer LSTM and Two Layer with New Cell are both 6.57M, outside the stated 5-6M range. Only the Ha et al. rows (4.92M, 5.06M) are near that range. This is a loose approximation rather than a substantive error.

Second check: another model agreed

• p. 9 Section 4.2 Results

  the models found by Neural Architecture Search outperform other state-of-the-art models on this dataset

• p. 9 Table 2

  67.9

Judge's note: Table 2 on the page 9 image: the NAS rows are 67.9, 64.0 and 62.4, and Zilly et al. is 66.0. Only the two NAS models with shared embeddings beat 66.0, so the general claim overstates for one of the three NAS models. The authors may have meant comparable configurations.

Second check: another model agreed

• p. 10 GNMT training details

  dividing by 2 after every 100K steps until it reaches 0.1

Judge's note: Page 10 image. Halving 0.5 gives 0.25, 0.125, 0.0625, so it never equals 0.1. The sentence most plausibly means 0.1 is a floor (0.25, 0.125, then clipped to 0.1), but the wording leaves the schedule ambiguous. This is a minor clarity point.

Second check: another model agreed

• p. 2 Section 2, Related Work, auto-regressive paragraph

  predicts hyperpa- rameters one a time

Judge's note: Checked in the Related Work paragraph on the page 2 image: the word "at" is missing.

Second check: none (judge only)

• p. 4 Section 3.2, paragraph after baseline equation

  does not depend on the on the current action

• p. 9 Results

  Not only is our cell is better

• p. 8 Search space

  the search space is has ap-

• p. 2 Related work

  one a time

Judge's note: All three are visible on the page images for pp.4, 9 and 10. The cluster also lists "one a time" and "search space is has"; those are judged under C004 and C010.

Second check: none (judge only)

• p. 8 Section 4.2, Search space

  the search space is has ap-

Judge's note: Checked in the Search space paragraph on the page 8 image.

Second check: none (judge only)

• p. 8 PTB dataset paragraph

  Zaremba et al. (2014) and (Gal, 2015)

Judge's note: Checked in the PTB Dataset paragraph on the page 8 image. Cosmetic only.

Second check: none (judge only)

• p. 9 Section 4.2, Results

  Not only is our cell is better

• p. 9

  Not only is our cell is better, the model

Judge's note: Page 9 image, Results paragraph, shows the second "is" printed as quoted. It should read "Not only is our cell better".

Second check: none (judge only)

• p. 10 GNMT paragraph

  with a parameter sever and 12 workers

• p. 10 GNMT paragraph

  a parameter sever and 12 workers

Judge's note: Page 10 image, GNMT paragraph, shows "parameter sever".

Second check: none (judge only)