REALM: Retrieval-Augmented Language Model Pre-Training
2020 · paper · AI and machine learning
Augments language model pre-training with a latent knowledge retriever over a corpus such as Wikipedia, trained without supervision using masked language modeling. Fine-tuned for open-domain question answering, it outperforms previous methods by 4–16% absolute accuracy on three benchmarks.
Checked: PDF from arxiv.org, fetched 4 Oct 2026. Page numbers are those of that PDF.
4 findings
Substance Section 4.4 says "from Base to 11B, the model is 50 times larger, and gains roughly 5 points in accuracy", but Table 1 gives T5 (base) 27.0 NQ / 29.1 WQ and T5 (11b) 34.5 / 37.4, a gain of 7.5 and 8.3 points.
- p. 7 Section 4.4, right column, first paragraph
from Base to 11B, the model is 50 times larger, and gains roughly 5 points in accuracy
- p. 7 Table 1, rows T5 (base) and T5 (11b)
27.0
- p. 7 Table 1, row T5 (11b)
34.5
- p. 7 Section 4.4, second paragraph
50 times larger, and gains roughly 5 points in accuracy
Only the authors can say what is right here.
Judge's note: Read the sentence and Table 1 on page 7. The 50-times size ratio matches the table (11318m / 223m = 50.8), so the comparison is Base to 11B as stated. A gain of about 5 points matches Large to 11B (4.7 and 5.2), not Base. T5 has no CT result, and nothing else in the paper gives about 5. Footnote 4 on page 6 says the authors' own T5 numbers were replaced by those of Roberts et al., which may have left the text stale. Whether the fix is the number or the model pair is for the authors.
Second check: another model agreed
Report a problem with this finding
Reference or label Reference Roberts et al. (2020) carries an unfilled placeholder identifier: "How much knowledge can you pack into the parameters of a language model? arXiv preprint arXiv:TBD, 2020."
- p. 10 References, Roberts et al.
arXiv preprint arXiv:TBD, 2020
- p. 10 References
arXiv preprint arXiv:TBD, 2020.
Judge's note: Read on the page 10 image, right column. The entry prints "arXiv:TBD" as quoted. The placeholder is visible in the file and needs no outside fact to confirm; the entry cannot be located by its identifier as printed. The authors must supply the actual arXiv number.
Second check: another model agreed
Report a problem with this finding
Wording Abstract and introduction say REALM beats all previous methods by "4-16% absolute accuracy", but in Table 1 the margin over the best previous system on WQ (T5 (11b), 37.4) is 3.3 points (40.7) or 2.8 (40.2), and the largest margin (CT, 46.8 vs 30.1) is 16.7.
- p. 1 Abstract
methods by a significant margin (4-16% absolute
- p. 2 Section 1, last paragraph
systems by 4-16% absolute accuracy
- p. 7 Table 1, WQ column
37.4
- p. 7 Table 1, WQ column, Ours (CC-News)
40.7
Only the authors can say what is right here.
Judge's note: Read the abstract (page 1 crop), the introduction (page 2) and Table 1 (page 7). Best previous per column: NQ 34.5 (T5 11b), WQ 37.4 (T5 11b), CT 30.1 (ORQA). The lower bound of 4 holds only if T5 is left out on WQ (40.7 - 36.4 = 4.3), yet the text says "all previous" and Section 4.4 says REALM outperforms the largest T5-11B. No statement in the paper restricts the comparison set. Footnote 4 says the T5 numbers were replaced with those of Roberts et al., which may be why the range is stale. Rated as wording because the range is a loose headline summary; only the authors can say which baseline set was meant.
Second check: another model agreed
Report a problem with this finding
Wording Appendix D states "In practice, we find that RU increases steadily over the course of pre-training" and points to Figure 4, where RU for salient span masking and random span masking levels off after about 30 thousand of 200 thousand steps.
- p. 11 Appendix D, last paragraph
increases steadily over the course of pre-training
- p. 12 Figure 4
Salient span masking
- p. 11 Appendix D
we find that RU increases steadily over the course of pre-training
- p. 12 Figure 4
Retrieval Utility
Only the authors can say what is right here.
Judge's note: Read Appendix D on the page 11 image and Figure 4 on the page 12 image. Salient span masking (the REALM setting) rises from about 0.5 at step 0 to about 2.0 by 30 thousand steps, then stays between about 1.9 and 2.1 to 200 thousand. Random span masking rises to about 1.0 by 30 thousand steps and stays between about 0.95 and 1.1. Only random uniform masking keeps rising slowly, from about 0.25 to about 0.6. The Figure 4 caption says only that RU "is impacted by ... the number of pre-training steps". Nothing in the text says the sentence refers to a different run. "Increases steadily" therefore overstates the plot for two of the three settings, including the main one. Only the authors can say whether another run or a looser meaning was intended.
Second check: another model agreed
Report a problem with this finding
16 minor (typos and the like; most checked by the judge only)
Wording Table 3 row (c) says "marginalizing over top 8 retrieved documents", while Section 4.3 says "we retrieve and marginalize over 8 candidate documents, including the null document" (so 7 retrieved plus the null document).
- p. 8 Table 3, row (c)
marginalizing over top 8 retrieved documents
- p. 7 Section 4.3, Pre-training
8 candidate documents, including the null document
- p. 7 Sec. 4.3 Pre-training
marginalize over 8 candidate documents, including the null document
Judge's note: Read Table 3 on page 8, Section 4.3 on page 7 and the null-document paragraph in Section 3.4 on page 5, which says the null document is added to the top k retrieved documents. The paper does not say whether the Table 3 example used a different setting or counts the null document loosely among the retrieved ones; either is possible, so this is an unresolved wording ambiguity.
Second check: another model agreed
Report a problem with this finding
Typo The encoder distribution is written "p(y | z, x)" in Equation 1 and Section 3.2 but "p(y | x, z)" in the restatement of the marginal in Section 3.3, in "expected value of p(y | x, z)" on page 5 and in Table 3.
- p. 3 Equation 1
p(y | z, x) p(z | x)
- p. 4 Section 3.3, second paragraph
p(y | x, z) p(z | x) involves a sum-
- p. 5 Section 3.3, What does the retriever learn?
value of p(y | x, z) when randomly sampling a document
Judge's note: Checked Equation 1 (page 3 crop), Section 3.3 on page 4 (sum of p(y | x, z) p(z | x)), page 5 (both orders in adjacent sentences) and Table 3 row (b) on page 8. The two orders mean the same thing, so this is a notation consistency point only.
Second check: none (judge only)
Report a problem with this finding
Typo "As shown below, the trainer sends the index builder a snapshot of its parameters" refers to Figure 3, which is typeset above the continuation of the sentence on page 5 and is not cited by number anywhere in the text.
- p. 4 Section 3.3, Implementing asynchronous MIPS refreshes, last line of page
job, which embeds and indexes the documents. As shown
- p. 5 Top of left column, under Figure 3
below, the trainer sends the index builder a snapshot of its
Judge's note: Page 4 ends with "As shown" and page 5 opens with Figure 3 followed by "below, the trainer sends...". A search of the text finds "Figure 3" only in its caption. This is a float-placement effect and cosmetic, but the figure being uncited is real.
Second check: none (judge only)
Report a problem with this finding
Typo Grammar slips: "is viable approach" (p.6, missing "a"); "REALM outperform all previous approaches" (p.7); "The results in Table 2 suggests" (p.8); "our work share motivations" (p.8).
- p. 6 Section 4.2, Generation-based Open-QA
viable approach, but they only experimented
- p. 6 Section 4.2
is viable approach
- p. 7 Section 4.4
REALM outperform all previous
- p. 8 MIPS index refresh rate
The results in Table 2 suggests
Judge's note: All four read on the page images of pages 6, 7 and 8; none is a line-break or extraction artefact. The paper writes "REALM outperforms" later in Section 4.4.
Second check: none (judge only)
Report a problem with this finding
Typo "the retrievers for both REALM pretraining and ORQA" uses unhyphenated "pretraining" where the paper otherwise writes "pre-training".
- p. 6 Retrieval-based Open-QA
REALM pretraining and ORQA
Judge's note: Read on page 6, Section 4.2. A text search finds "pretraining" in the body only here (the other hit is the title of a cited work in the reference list). Spelling consistency only.
Second check: none (judge only)
Report a problem with this finding
Typo Table 1 caption: "The number of train/test examples are shown in paretheses below each benchmark"; "paretheses" should be "parentheses" (and "The number ... are" disagrees).
- p. 7 Table 1 caption
are shown in paretheses below each benchmark
- p. 7 Table 1 caption
shown in paretheses below
- p. 7 Table 1 caption
shown in paretheses below each benchmark
Judge's note: Read on the page 7 image; the misspelling is in the published PDF, not an extraction artefact.
Second check: none (judge only)
Report a problem with this finding
Typo Table 1 header for CT reads "(1k /1k)" with a space before the slash, unlike "(79k/4k)" and "(3k/2k)".
- p. 7 Table 1 CT column header
(1k /1k)
Judge's note: Visible on the page 7 image: the CT sub-header has a gap before the slash that the NQ and WQ sub-headers do not. Purely cosmetic.
Second check: none (judge only)
Report a problem with this finding
Typo Section 4.5: "REALM (row (c)) gives the word a much high probability" (should be "much higher") and "REALM is able to retrieve document to fill in the masked word" (should be "a document" or "documents").
- p. 8 Section 4.5, Examples of retrieved documents
gives the word a much high probability
- p. 8 Examples of retrieved documents
a much high probability
- p. 7 Section 4.4
REALM outperform all previous
- p. 8 Examples of retrieved documents
retrieve document to fill in the masked word
Judge's note: Both read on the page 8 image; the Table 3 caption on the same page writes "much higher probability". The other slips some reviewers listed here ("REALM outperform", "The results in Table 2 suggests", "our work share motivations") are also real but are covered by C006.
Second check: none (judge only)
Report a problem with this finding
Typo Section 4.5 (MIPS index refresh rate): "The results in Table 2 suggests that a stale index can hurt model training" has a plural subject with a singular verb; it should read "suggest".
- p. 8 Section 4.5, MIPS index refresh rate
The results in Table 2 suggests that
Judge's note: Read on the page 8 image, left column, MIPS index refresh rate paragraph. The text is as quoted. Plain subject-verb disagreement.
Second check: none (judge only)
Report a problem with this finding
Typo Section 5 (Scalable grounded neural memory): "From this view, our work share motivations with works such as product key memory" has a singular subject with a plural verb; it should read "shares".
- p. 8 Section 5, Scalable grounded neural memory
share motivations with works such as product key mem-
Judge's note: Read on the page 8 image, right column. The text is as quoted. Plain subject-verb disagreement.
Second check: none (judge only)
Report a problem with this finding
Typo Section 4.5 (Examples of retrieved documents): "This shows that REALM is able to retrieve document to fill in the masked word" lacks an article or plural; it should read "retrieve a document" or "retrieve documents".
- p. 8 Section 4.5, Examples of retrieved documents
able to retrieve document to fill in the masked word
Judge's note: Read on the page 8 image, left column, last paragraph. The text is as quoted. The same paragraph says "retrieve some documents", so the plural is the likely intent.
Second check: none (judge only)
Report a problem with this finding
Typo Reference Chen et al. (2017) gives the volume twice: "Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pp. 1870–1879, 2017."
- p. 9 References, Chen et al. (2017)
(Volume 1: Long Papers),
- p. 9 References, Chen et al. (2017)
volume 1, pp. 1870–1879, 2017.
Judge's note: Read on the page 9 image, left column of the reference list. The proceedings title already contains "(Volume 1: Long Papers)" and is followed by "volume 1". Cosmetic duplication from the bibliography tool; the entry can still be located.
Second check: none (judge only)
Report a problem with this finding
Typo Reference Peters et al. (2019) prints an author as "IV, R. L. L." (a generational suffix in the surname position) and gives no venue or identifier: "Peters, M. E., Neumann, M., IV, R. L. L., Schwartz, R., Joshi, V., Singh, S., and Smith, N. A. Knowledge enhanced contextual word representations, 2019."
- p. 10 References, Peters et al. (2019)
Peters, M. E., Neumann, M., IV, R. L. L., Schwartz, R.,
Only the authors can say what is right here.
Judge's note: Rests on outside knowledge: that the third author is Robert L. Logan IV, so the entry should read "Logan IV, R. L.". The page 10 image, left column, shows the text as quoted. Visible in the file without outside knowledge: "IV" sits where a surname belongs, and the entry has no venue or identifier, unlike the other entries.
Second check: none (judge only)
Report a problem with this finding
Typo Reference Shen et al. (2015) prints the last author as "Tao Shen, H.", unlike the "Surname, Initials" form of the other authors in the entry: "Shen, F., Liu, W., Zhang, S., Yang, Y., and Tao Shen, H."
- p. 10 References, Shen et al. (2015)
Shen, F., Liu, W., Zhang, S., Yang, Y., and Tao Shen,
- p. 10 References
Yang, Y., and Tao Shen, H.
Only the authors can say what is right here.
Judge's note: Rests on outside knowledge: that the last author is Heng Tao Shen, whose name would be "Shen, H. T." in this list's format. The page 10 image, right column, shows the text as quoted. From the page alone this is a format inconsistency; that "Tao" is a given name cannot be verified from the files.
Second check: none (judge only)
Report a problem with this finding
Typo Appendix C: "we replace the knowledge corpus with a more recent version of Wikipedia corpus after pre-training is done" lacks the article "the" before "Wikipedia corpus".
- p. 11 Appendix C
edge corpus with a more recent version of Wikipedia cor-
Judge's note: Read on the page 11 image, right column. The text is as quoted. The second instance the reviewer mentions, "is the prime minister of United Kingdom", is a quoted model input in monospace and is not counted as a defect.
Second check: none (judge only)
Report a problem with this finding
Typo Table 4 caption: "so the model was not about to recover the word when the knowledge corpus is outdated (2018)" should read "not able to recover".
- p. 12 Table 4 caption
so the model was not about to recover the word
- p. 12 Table 4 caption
the model was not about to recover the word
Judge's note: Read the Table 4 caption on the page 12 image. The text is as quoted. The next sentence says the same model "is able to retrieve the document in the updated corpus", and the table shows the 2018 corpus failing to produce "lawrence". "about" is a wrong-word slip for "able"; the intended meaning is still recoverable from context.
Second check: none (judge only)