BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2019 · paper · AI and machine learning
Introduces BERT, a language model pre-trained on unlabeled text using both left and right context, then fine-tuned with one added output layer. It reports new state-of-the-art results on eleven language tasks, including GLUE, MultiNLI and SQuAD.
Checked: PDF from arxiv.org, fetched 4 Oct 2026. Page numbers are those of that PDF.
3 findings
Wording "Without TriviaQA fine-tuning data, we only lose 0.1-0.4 F1": the Table 2 Dev F1 differences with and without TriviaQA are 0.2 (single, 91.1 vs 90.9) and 0.4 (ensemble, 92.2 vs 91.8); 0.1 is the single-model EM difference (84.2 vs 84.1).
- p. 7 Section 4.2, last sentence (below Table 3)
tuning data, we only lose 0.1-0.4 F1
- p. 7 Table 2, rows BERTLARGE (Single) and (Sgl.+TriviaQA)
84.1 90.9
- p. 7 Table 2, row BERTLARGE (Sgl.+TriviaQA)
84.2 91.1 85.1 91.8
- p. 7 Section 4.2, last sentence (top of left column text)
we only lose 0.1-0.4 F1
Only the authors can say what is right here.
Judge's note: Table 2 read on the p7 image and crop. The only with/without-TriviaQA comparisons available are on Dev (Test cells for the non-TriviaQA rows are "-"). F1 losses are 0.2 and 0.4; EM losses are 0.1 and 0.4, which match the printed range exactly, suggesting the range covers EM and F1 together or the metric was mislabelled. Unrounded values could in principle give 0.1 for F1, so this is a query for the authors. The reviewer note about footnote 7 (91.0 vs 90.9) is a separate point handled in C002.
Second check: another model agreed
Report a problem with this finding
Wording MRPC training-set size is "3.5k" in Table 1 but Section 5.2 says "MRPC which only has 3,600 labeled training examples".
- p. 6 Table 1, row of training-set sizes, column MRPC
3.5k
- p. 8 Section 5.2, second paragraph
for MRPC which only has 3,600 labeled train-
- p. 8 Section 5.2
MRPC which only has 3,600 labeled train- ing examples
- p. 8 Section 5.2
MRPC which only has 3,600 labeled train-
Only the authors can say what is right here.
Judge's note: Table 1 size row (p6 crop) shows 3.5k under MRPC; Section 5.2 (p8 crop) prints 3,600. No single count rounds to both 3.5k and 3,600, and nothing in the paper explains the difference. The difference is small and does not affect any result; the authors would need to say which figure is right.
Second check: another model agreed
Report a problem with this finding
Wording Section 5.2 says "larger models lead to a strict accuracy improvement across all four datasets", but Table 6, which the sentence describes, reports Dev accuracy on three datasets only (MNLI-m, MRPC, SST-2); the fourth numeric column is "LM (ppl)", a perplexity.
- p. 8 Section 5.2, second paragraph
curacy improvement across all four datasets
- p. 9 Table 6 header
LM (ppl) MNLI-m MRPC SST-2
- p. 8 Section 5.2, second paragraph
across all four datasets
- p. 9 Table 6 header
MNLI-m MRPC SST-2
Only the authors can say what is right here.
Judge's note: Read Section 5.2 on p8 and Table 6 with its caption on p9. The table has three accuracy columns; the caption defines LM (ppl) as masked LM perplexity of held-out training data, which is not an accuracy. Nothing in the text names a fourth dataset for this table. Whether the count should be three or a column was dropped is for the authors.
Second check: another model agreed
Report a problem with this finding
17 minor (typos and the like; most checked by the judge only)
Wording "BERTLARGE outperforms the authors' baseline ESIM+ELMo system by +27.1% and OpenAI GPT by 8.3%": these are absolute differences in accuracy points (86.3 - 59.2, 86.3 - 78.0), not relative percentages.
- p. 7 Section 4.4
by +27.1% and OpenAI GPT by 8.3%
- p. 7 Table 4 Test
86.3
Judge's note: Table 4 and the sentence read on the p7 image. Both differences reproduce as percentage-point gaps. Elsewhere the paper marks such gaps as "absolute" (e.g. "4.6% absolute accuracy improvement" in Section 4.1); here the qualifier is missing, so "%" could be read as relative (which would be about 45.8% and 10.6%). The numbers themselves are right as point differences; a wording ambiguity only.
Second check: another model agreed
Report a problem with this finding
Wording Appendix A.2 prints "256 sequences * 512 tokens = 128,000 tokens/batch" with an equals sign; 256 x 512 is 131,072. The rounded 128,000 is reused in A.4 and C.1.
- p. 13 Appendix A.2, last paragraph of left column
sequences * 512 tokens = 128,000 tokens/batch
- p. 13 Appendix A.2, second paragraph
256 sequences * 512 tokens = 128,000 tokens/batch
- p. 16 Appendix C.1, Question 1
128,000 words/batch * 1,000,000 steps
- p. 14 A.4
batch size of 128,000 words
Judge's note: Read on page 13: the equation is printed as an equality with no "approximately", although the same sentence uses "approximately" for the 40 epochs. 128,000 is evidently a round figure (128k); no statement in the paper defines it otherwise. The epoch claim holds with either number (about 39 to 40 epochs over 3.3 billion words), so nothing downstream changes. Minor: fix is "≈ 128,000" or 131,072. The words-versus-tokens unit point is handled under C037.
Second check: another model agreed
Report a problem with this finding
Wording The same batch size is given as "128,000 tokens/batch" in A.2 but as "a batch size of 128,000 words" in A.4 and "128,000 words/batch" in C.1; the model uses WordPiece tokens, which are not words.
- p. 14 A.4 bullet 3
BERT was trained for 1M steps with a batch size of 128,000 words
- p. 13 A.2
128,000 tokens/batch
Judge's note: Read on pages 13, 14 and 16. A.2 derives the figure from sequences times tokens, so tokens is the unit; A.4 and C.1 call it words, and A.4 sets it beside GPT's "32,000 words". Loose usage, with no effect on any result, but the unit is inconsistent. Kept separate from C032, which is the arithmetic of the product.
Second check: another model agreed
Report a problem with this finding
Typo Figure 2 caption says "the segmentation embeddings" while the figure itself is labelled "Segment Embeddings" and Section 3 says "token, segment, and position embeddings".
- p. 5 Figure 2 caption
the segmenta- tion embeddings
- p. 4 Section 3
token, segment, and position embeddings
Judge's note: Caption and figure label read on the p5 image; Section 3 text on p4. The terminology differs between caption and figure/text; cosmetic.
Second check: none (judge only)
Report a problem with this finding
Typo Misspelling in Section 4.2: "befor fine-tuning on SQuAD" should read "before".
- p. 6 Section 4.2, right column
befor fine-tuning on SQuAD
Judge's note: Seen on the p6 page image, right column: "(Joshi et al., 2017) befor fine-tuning on SQuAD." Not an extraction artefact.
Second check: none (judge only)
Report a problem with this finding
Typo Subject-verb disagreement in Section 4.4: "The only task-specific parameters introduced is a vector whose dot product with the [CLS] token representation C denotes a score for each choice".
- p. 7 Sec. 4.4
The only task-specific parameters introduced is a vec- tor
Judge's note: Read on the p7 image. Plural "parameters" with singular "is a vector"; should be "The only task-specific parameter introduced is a vector" or "parameters ... are".
Second check: none (judge only)
Report a problem with this finding
Typo Section 5.1: "(c) this it is strictly less powerful" has an extra word ("this is" or "it is"); Section 4.4: "The only task-specific parameters introduced is a vector" has a number disagreement.
- p. 8 Section 5.1, last paragraph
(c) this it is strictly less powerful
- p. 7 Section 4.4
The only task-specific parameters introduced is a vec-
Judge's note: Both read on the images (p8 right column; p7 crop). Plain copy-editing slips.
Second check: none (judge only)
Report a problem with this finding
Typo Table 5 caption's last sentence calls the model "LTR + No NSP", while the table row, the earlier caption sentence and the text call it "LTR & No NSP".
- p. 8 Table 5 caption, last sentence
domly initialized BiLSTM on top of the “LTR + No
- p. 8 Table 5 row label
LTR & No NSP
- p. 8 Table 5 caption
“LTR + No NSP” model
- p. 8 Table 5 caption
“LTR + No NSP” model during fine-tuning
Judge's note: Read the Table 5 caption and row labels on p8. The same model is named with '&' three times and '+' once. Meaning unaffected.
Second check: none (judge only)
Report a problem with this finding
Typo Appendix A.1: "we chose the 4-th token (which corresponding to hairy)" should read "which corresponds to" or "corresponding to".
- p. 12 Appendix A.1, Masked LM and the Masking Procedure
we chose the 4-th token (which corresponding to
Judge's note: Read on p12, right column. Grammar slip as quoted.
Second check: none (judge only)
Report a problem with this finding
Typo Appendix A.1: "In Section C.2, we evaluate the impact this procedure." is missing "of".
- p. 12 Appendix A.1, paragraph after the bullet list
Section C.2, we evaluate the impact this proce-
- p. 12 Appendix A.1
we evaluate the impact this procedure
- p. 12
we evaluate the impact this proce-
Judge's note: Read on p12, right column, end of the paragraph after the bullet list.
Second check: none (judge only)
Report a problem with this finding
Typo Appendix misspellings and slips: "Compared to standard langauge model training, the masked LM only make predictions" (language; makes); "To speed up pretraing" (A.2); heading "Comparison of BERT, ELMo ,and OpenAI GPT" and "Here we studies the differences" (A.4); "Table1" without a space (B.1).
- p. 12 Appendix A.1, last paragraph on the page
Compared to standard langauge model training,
- p. 12 Appendix A.1, last paragraph
Compared to standard langauge model training
- p. 14 A.4 heading and first sentence
Comparison of BERT, ELMo ,and
- p. 14 A.4 first line
Here we studies the differences
Judge's note: Each read on an image: p12 for "langauge" and "only make"; crops of p13 for "pretraing"; p14 crops for "ELMo ,and", "we studies" and "Table1". The two further items in the cluster ("impact this procedure", "which corresponding to") are real but are reported under C027 and C026.
Second check: none (judge only)
Report a problem with this finding
Typo Misspelling on PDF page 12: "Compared to standard langauge model training"; "langauge" should be "language".
- p. 12
Compared to standard langauge model training
Judge's note: Read on the crop: the word is printed "langauge" in the PDF itself, so it is not an extraction artefact.
Second check: none (judge only)
Report a problem with this finding
Typo Misspelling in Appendix A.2: "To speed up pretraing in our experiments"; should be "pretraining" (the paper elsewhere writes "pre-training").
- p. 13 Appendix A.2, last paragraph
length. To speed up pretraing in our experiments,
- p. 13 Appendix A.2
To speed up pretraing in our experiments
Judge's note: Read on the page 13 image, right column: printed "pretraing".
Second check: none (judge only)
Report a problem with this finding
Typo Subject-verb disagreement in Appendix A.4: "Here we studies the differences in recent popular representation learning models"; should be "we study".
- p. 14 Appendix A.4, first sentence
Here we studies the differences in recent popular
Judge's note: Read on the page 14 image, first sentence of A.4.
Second check: none (judge only)
Report a problem with this finding
Typo Heading of Appendix A.4 is printed "Comparison of BERT, ELMo ,and OpenAI GPT", with a space before the comma and none after; should be "ELMo, and".
- p. 14 Appendix A.4 heading
Comparison of BERT, ELMo ,and
Judge's note: Read on the page 14 image: the misplaced space is visible in the heading itself, not an extraction artefact.
Second check: none (judge only)
Report a problem with this finding
Typo Missing space in Appendix B.1: "Our GLUE results in Table1 are obtained from"; should be "Table 1".
- p. 14 Appendix B.1, first sentence
Our GLUE results in Table1 are obtained
- p. 14 Appendix B.1
results in Table1
Judge's note: Read on the page 14 image: "Table1" is set with no space although the line is widely spaced; every other table reference has a space.
Second check: none (judge only)
Report a problem with this finding
Typo Wrong word in Appendix C.2: "The right part of the paper represents the Dev set results"; should be "the right part of the table" (Table 8).
- p. 16 Appendix C.2, paragraph after Table 8
The right part of the paper represents the
- p. 16 Appendix C.2, paragraph after Table 8
The right part of the paper represents the Dev set results
- p. 16 C.2
The right part of the paper represents the Dev set results.
Judge's note: Read on the page 16 image. The preceding sentence says "The numbers in the left part of the table", and Table 8's right-hand columns are headed "Dev Set Results", so "paper" is a slip for "table".
Second check: none (judge only)