Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
2020 · paper · AI and machine learning
Introduces retrieval-augmented generation models that pair a pretrained sequence-to-sequence generator with a dense index of Wikipedia accessed by a neural retriever. They set the state of the art on three open-domain question answering tasks and generate more specific, factual language.
Checked: PDF from arxiv.org, fetched 4 Oct 2026. Page numbers are those of that PDF.
4 findings
Substance Appendix G (p. 19) describes the index as "21M 728 dimensional vectors, consisting of 15.3B values", while Section 2.2 says d(z) is produced by a BERT-BASE document encoder.
- p. 19 Appendix G
21M 728 dimensional vectors, consisting of 15.3B values
- p. 3 Section 2.2
produced by a BERTBASE document encoder
Only the authors can say what is right here.
Judge's note: Rests on outside knowledge: BERT-base (as used by DPR) has a hidden size and [CLS] embedding of 768. The paper never states 768, and no projection to 728 dimensions is described. 15.3B is internally consistent with 728 (21M×728≈15.29B). With 768 dimensions the count would be about 16.1B, so both the dimension and the value count would be affected. One reviewer's claim that 21M×768=15.3B is arithmetically wrong. Only the authors can confirm the intended dimension.
Second check: another model agreed
Report a problem with this finding
Substance Sec 4.3 says "both RAG and BART were factual in a further 17% of cases", but Table 4 Factuality 'Both good' is "11.7%" (17.7% is 'Both poor').
- p. 6 Section 4.3, first paragraph
both RAG and BART were factual in a further 17% of cases
- p. 8 Table 4, Factuality column, 'Both good' row
11.7%
- p. 8 Table 4, Factuality
Both good 11.7%
- p. 6
and both RAG and BART were factual in a further 17% of cases
Judge's note: Page 6 text and Table 4 on page 8 both checked. There is a single factuality evaluation (452 pairs), so both numbers describe the same quantity. The Factuality column sums to 100.0, so the table is internally consistent; the text value of 17% matches no 'both factual' entry.
Second check: another model agreed
Report a problem with this finding
Substance Table 4 Specificity column ("16.8% 37.4% 11.8% 6.9% 20.1%") sums to 93.0%, while the five categories are exhaustive and the Factuality column sums to 100.0%.
- p. 8 Table 4, Specificity column
16.8%
- p. 8 Table 4, Specificity column
16.8% 37.4% 11.8% 6.9% 20.1%
Only the authors can say what is right here.
Judge's note: Values read from the page 8 image. No footnote or appendix text (Appendix B checked) explains a missing category or excluded responses. Removing annotators would not leave the percentages short of 100. At least one specificity value appears to be in error; only the authors can say which.
Second check: another model agreed
Report a problem with this finding
Reference or label "who train RoBERTa [35]" cites reference [35], which is "Hairong Liu ... Robust neural machine translation with joint textual and phonetic embedding" (ACL 2019), not the RoBERTa paper.
- p. 7 Section 4.4, 2-way FEVER paragraph
who train RoBERTa [35]
- p. 13 Reference [35]
Robust neural machine translation with joint textual and phonetic embedding
Judge's note: Page 13 reference list: [35] is the NMT paper. A text search for RoBERTa finds only the citing sentence, so no RoBERTa entry exists in the bibliography.
Second check: another model agreed
Report a problem with this finding
22 minor (typos and the like; most checked by the judge only)
Typo Sec 2.1: "repeating the process with the following output token, Formally, we define:": the comma before 'Formally' should be a full stop.
- p. 3 Section 2.1, RAG-Token Model
with the following output token, Formally, we define:
Judge's note: Page 3 crop shows the comma before the capitalised 'Formally'.
Second check: none (judge only)
Report a problem with this finding
Typo Sec 2.5 RAG-Token transition probability sums over "z ∈ top-k(p(·|x))" but the summand is written "pη(zi|x)pθ(yi|x, zi, y1:i−1)" with z_i rather than the bound variable z.
- p. 4 Section 2.5, RAG-Token
pη(zi|x)pθ(yi|x, zi, y1:i−1)
Judge's note: Page 4 crop confirms that the summation index is z while the summand uses z_i. The corresponding Sec 2.1 RAG-Token equation (p3 crop) uses z throughout. This is a notational inconsistency only; the meaning is clear.
Second check: none (judge only)
Report a problem with this finding
Typo Sec 3.1 dataset list: "TriviaQA (TQA) [24]. WebQuestions (WQ) [3]": a full stop appears where a comma is needed.
- p. 4 Section 3.1
TriviaQA (TQA) [24]. WebQuestions (WQ) [3]
Judge's note: Page 4 crop shows the period mid-list.
Second check: none (judge only)
Report a problem with this finding
Typo Misspellings: "quuestion A is better" (Sec 3.3, p5) and "compehension" (Appendix D, p18).
- p. 5 Section 3.3, end
quuestion A is better
- p. 18 Appendix D
compehension
- p. 5
quuestion A is better, question B is better, both are good, or neither is good
Judge's note: Both misspellings are visible in the page images (p5 Sec 3.3 last line; p18 TriviaQA Evaluation setups paragraph).
Second check: none (judge only)
Report a problem with this finding
Typo Sec 3.4: "coupled with an challenging entailment reasoning task": should be 'a challenging'.
- p. 5 Section 3.4
coupled with an challenging entailment reasoning task
Judge's note: Visible on page 5 image, Sec 3.4.
Second check: none (judge only)
Report a problem with this finding
Typo Sec 4.1: "Documents with clues about the answer but do not contain the answer verbatim" is missing a relative pronoun ('but that do not contain' or 'that ... but do not').
- p. 5 Sec 4.1
Documents with clues about the answer but do not contain the answer verbatim
Judge's note: Page 5 image, last paragraph: the text reads as quoted. Minor grammar slip.
Second check: none (judge only)
Report a problem with this finding
Typo Sec 4.3: "4 shows human evaluation results, over 452 pairs": the word 'Table' is missing before '4'.
- p. 6 Section 4.3, first paragraph
4 shows human evaluation results, over 452
- p. 6 Section 4.3
4 shows human evaluation results
Judge's note: The page 6 image shows the sentence beginning with a bare '4' after 'Q-BLEU-1.', so this is not an extraction artefact.
Second check: none (judge only)
Report a problem with this finding
Typo Inconsistent dataset labels across tables/text: "MSMARCO" (Table 2, 5), "MS-MARCO" (text, Table 3), "MSMarco" (Table 6); "FVR3" (Table 2) vs "FVR-3" (Table 6); "Jeopardy QGen" (Table 5) vs "Jeopardy-QGen" (Table 6).
- p. 8 Table 5/6 headers
Jeopardy-QGen
- p. 6 Table 2 header
MSMARCO FVR3 FVR2
Judge's note: The variants are visible in the page 6, 7 and 8 images. This is purely stylistic and the referent is unambiguous.
Second check: none (judge only)
Report a problem with this finding
Typo Table 6 MSMarco B-1: RAG-Token "49.4" is bold as best, but RAG-Token-Frozen also shows 49.4 and is not bold.
- p. 8 Table 6, MSMarco B-1 column
49.4
Judge's note: In the page 8 image, the B-1 column reads 48.4, 46.9, 49.4, 47.3, 49.4 (bold), 47.5. The tie is marked inconsistently, possibly because the unrounded values differ.
Second check: none (judge only)
Report a problem with this finding
Typo Discussion (p. 9) reads "We conducted an thorough investigation"; this should be "a thorough".
- p. 9 Section 6 Discussion
We conducted an thorough investigation
- p. 9 Section 6 Discussion, sentence 4
We conducted an thorough investigation of the learned retrieval component
Judge's note: The page-9 crop shows "an thorough investigation of the learned retrieval component" as printed. It is not an extraction artefact.
Second check: none (judge only)
Report a problem with this finding
Typo Discussion (p. 9) reads "a denoising objective similar to BART or some another objective"; this should be "some other objective" or "another objective".
- p. 9 Section 6 Discussion
similar to BART or some another objective
- p. 9 Section 6 Discussion, future-work sentence
either with a denoising objective similar to BART or some another objective
- p. 9 Discussion
or some another objective
Judge's note: The page-9 crop shows "or some another objective" split across a line break, as printed.
Second check: none (judge only)
Report a problem with this finding
Typo Related Work, Memory-based Architectures (p. 9), reads "comprised of raw text rather distributed representations"; "than" is missing after "rather".
- p. 9 Related Work, Memory-based Architectures
comprised of raw text rather distributed representations
- p. 9 Memory-based Architectures
raw text rather distributed representations
Judge's note: The page-9 crop shows "raw text rather" at the end of one line and "distributed representations" at the start of the next. "than" is missing.
Second check: none (judge only)
Report a problem with this finding
Typo Retrieve-and-Edit paragraph (p. 9) reads "including less of emphasis on lightly editing a retrieved item"; this should be "less emphasis".
- p. 9 Retrieve-and-Edit
including less of emphasis on lightly editing
Judge's note: The page-9 crop shows "less of emphasis" as printed.
Second check: none (judge only)
Report a problem with this finding
Typo Acknowledgments (p. 10) reads "EP thanks supports from the NSF Graduate Research Fellowship"; this should be "thanks support from" or "acknowledges support from".
- p. 10 Acknowledgments
EP thanks supports from the NSF Graduate Research Fellowship
Judge's note: The page-10 crop shows "EP thanks supports from the NSF Graduate Research Fellowship" as printed.
Second check: none (judge only)
Report a problem with this finding
Typo References [17] and [18] are the same work (Gu, Wang, Cho, Li, "Search engine guided neural machine translation", AAAI 2018), and the text cites the two entries separately.
- p. 11 References [17], [18]
Search engine guided neural machine translation
- p. 11 References [17],[18]
Search engine guided neural machine translation.
Judge's note: The page-11 crop shows both entries with identical authors and title, both AAAI 2018; [18] adds page numbers. In the text, [17] is cited for translation in Related Work and [18] under Retrieve-and-Edit (p. 9, "Machine Translation [18, 22]"). This is a duplicate bibliography entry.
Second check: none (judge only)
Report a problem with this finding
Typo Reference [18] repeats the venue three times: "In 32nd AAAI Conference on Artificial Intelligence, AAAI 2018, 32nd AAAI Conference on Artificial Intelligence, AAAI 2018, pages 5133–5140. AAAI press, 2018. 32nd AAAI Conference on Artificial Intelligence, AAAI 2018 ; Conference date: ..."
- p. 11 Refs [17],[18]
32nd AAAI Conference on Artificial Intelligence, AAAI 2018, 32nd
Judge's note: The page-11 crop confirms the venue string is repeated inside entry [18]. This defect is inside the entry and is separate from the duplicate-entry problem in C021; that part of this cluster is covered by C021.
Second check: none (judge only)
Report a problem with this finding
Typo Reference [61] prints the unrendered LaTeX command "F. d\textquotesingle Alché-Buc" instead of "F. d'Alché-Buc".
- p. 15 Reference [61]
F. d\textquotesingle Alché-Buc
Judge's note: The page-15 crop shows "d\textquotesingle Alché-Buc" literally in the rendered reference, so this is not an extraction artefact.
Second check: none (judge only)
Report a problem with this finding
Typo Appendix C (p. 17) reads "After submission, We have ported our code"; "We" is capitalised mid-sentence.
- p. 17 App C
After submission, We have ported our code
Judge's note: The page-17 crop shows a capital "We" after the comma.
Second check: none (judge only)
Report a problem with this finding
Typo Appendix D, TriviaQA Evaluation setups (p. 18), reads "reading compehension purposes"; this should be "comprehension".
- p. 18 Appendix D, TriviaQA Evaluation setups
reading compehension purposes
Judge's note: The page-18 crop shows "compehension" as printed.
Second check: none (judge only)
Report a problem with this finding
Typo Appendix G (p. 18) reads "406M trainable parameters from BART-large, 406M parameters, making a total of 626M"; "406M parameters" is redundantly repeated.
- p. 18 Appendix G
406M trainable parameters from BART-large, 406M parameters
- p. 18
406M trainable parameters from BART-large, 406M parameters, making a total of 626M
Judge's note: The page-18 crop confirms the repeated phrase. A second reviewer also points to "400M" on p. 3 against 406M here. That is ordinary rounding of the same model size, not a defect.
Second check: none (judge only)
Report a problem with this finding
Typo Appendix D (p. 18) reads "using the datasets splits used in DPR"; this should be "dataset splits".
- p. 18 App D
using the datasets splits used in DPR
Judge's note: The page-18 crop (TriviaQA Evaluation setups paragraph) shows "datasets splits".
Second check: none (judge only)
Report a problem with this finding
Typo Appendix G (p. 19) reads "but does consists of 21M" (should be "does consist of") and "These can be easily be stored" ("be" appears twice).
- p. 19 Appendix G, end
These can be easily be stored
- p. 19 App G
but does consists of 21M
Judge's note: The page-19 crop shows both errors as printed.
Second check: none (judge only)