Universal Language Model Fine-tuning for Text Classification
Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2018 · paper · Computational linguistics
Proposes ULMFiT, a transfer learning method that fine-tunes a pretrained language model for text classification tasks. It reportedly reduces error by 18–24% on most of six datasets and, with 100 labeled examples, matches training from scratch on far more data.
Checked: PDF from aclanthology.org, fetched 5 Oct 2026. Page numbers are those of that PDF.
5 findings
Substance Section 2 (p. 2) reads "Features in deep neural networks in CV have been observed to transition from task-specific to general from the first to the last layer (Yosinski et al., 2014)". The direction is reversed: it should be general to task-specific.
- p. 2 Section 2, Transfer learning in CV
transition from task-specific to general from the first to the last layer
- p. 2 Related work, Transfer learning in CV
transition from task-specific to general from the first to the last layer (Yosinski et al., 2014)
Judge's note: I checked the page 2 image, and the sentence is printed as quoted, with 'specific' and 'general' in italics. The next sentence says that "for this reason, most work in CV focuses on transferring the last layers". Section 3.3 (extract line 449-450) says gradual unfreezing starts "from the last layer as this contains the least general knowledge (Yosinski et al., 2014)". Both of these assume that the first layers are general and the last layers are task-specific, so the paper's own text shows the direction in this sentence is reversed. That also matches the commonly known finding of the cited work, but the internal contradiction is enough to confirm it.
Second check: another model agreed
Report a problem with this finding
Substance Eq. 3 decay branch "1 − (t−cut)/(cut·(ratio−1))" does not make p reach 0 at t = T. With cut_frac = 0.1 and ratio = 32, p ≈ 0.71 at the end of training, whereas Figure 2 shows the LR decaying to near its minimum. The denominator consistent with the definition of p is cut·(1/cut_frac − 1).
- p. 4 Equation 3
cut·(ratio−1)
- p. 4
p = ( t/cut, if t < cut 1 − t−cut cut·(ratio−1), otherwis
Judge's note: Viewed Eq. 3 and Figure 2 on p.4. p is defined as the fraction of the decay iterations completed, and Figure 2 decays almost to the minimum by the last iteration; the printed formula contradicts both. The second reviewer note ('repeated cut') is a text-layer artefact of the fraction and not a separate defect.
Second check: another model agreed, code check
Report a problem with this finding
Substance Table 7 (p. 8) prints 'Full + discr' IMDb validation error as 4.57. That is lower than the bolded 5.00 for 'Freez + discr + stlr', yet the text says full ULMFiT "achieves the best performance on IMDB".
- p. 8 Table 7, row 'Full + discr', IMDb column
Full + discr 4.57 6.21 5.62
- p. 8 Impact of classifier fine-tuning paragraph
achieves the best performance on IMDB and TREC-6
- p. 8 Table 7, Full + discr row and bold 5.00
Full + discr 4.57
- p. 8 Impact of classifier fine-tuning, final paragraph
achieves the best performance on IMDB and
Only the authors can say what is right here.
Judge's note: I checked the page 8 image. In Table 7, Full + discr reads 4.57 / 6.21 / 5.62. The bottom row reads 5.00 / 5.69 / 5.38, with 5.00 bolded as the best IMDb value. The text says the bottom row achieves the best performance on IMDB and TREC-6. Since 4.57 < 5.00, the table cell contradicts both the bold marker and the text. The same configuration in Table 6 (LM 'Full + discr') is 5.55, but that is a different ablation. Nothing in the paper explains this row. Only the authors can say whether the cell is a misprint or the text and bolding are wrong. A minor side point: 'IMDB' here is spelled 'IMDb' everywhere else.
Second check: another model agreed
Report a problem with this finding
Reference or label The Blitzer et al. (2007) ACL reference carries the DOI "https://doi.org/10.1109/IRPS.2011.5784441", an IEEE IRPS 2011 identifier that does not match a 2007 ACL paper.
- p. 10 References, Blitzer et al. 2007
https://doi.org/10.1109/IRPS.2011.5784441
Judge's note: Viewed the reference list on p.10. The DOI's own string (IRPS, 2011) is visibly inconsistent with the entry's year and venue.
Second check: another model agreed
Report a problem with this finding
Wording Abstract and Contributions claim "reducing the error by 18-24% on the majority of datasets", but by the paper's own figures only 3 of 6 datasets (IMDb 22%, AG 23.7%, Yelp-bi 18.2%) fall in that range; TREC-6 (~7.7%), DBpedia (4.8%) and Yelp-full (2.0%) do not.
- p. 1 Abstract
reducing the error by 18- 24% on the majority of datasets
- p. 2 Contributions
with an error reduction of 18-24% on the majority of datasets
- p. 7 Results text
we reduce the error by 4.8%, 18.2%, 2.0% respectively
- p. 7 Results text
On DBpedia, Yelp-bi, and Yelp-full, we reduce the error by 4.8%, 18.2%, 2.0% respectively.
Judge's note: Checked Tables 2 and 3 (p.6) and the results text (p.6-7). Against the best prior result, three of six datasets fall in 18-24%, which is half, not a majority. Using CoVe as the IMDb baseline (43.9%) lowers the count further. Nothing in the paper defines 'majority' differently.
Second check: another model agreed
Report a problem with this finding
7 minor (typos and the like; most checked by the judge only)
Wording Abstract/Conclusion state the method "significantly outperforms the state-of-the-art on six text classification tasks", while the results say the TREC-6 improvement "is not statistically significant".
- p. 1 Abstract
Our method significantly outper- forms the state-of-the-art on six text clas- sification tasks
- p. 9 Conclusion
Our method significantly outperformed ex- isting transfer learning techniques and the state- of-the-art on six representative text classification tasks.
- p. 7 Results, TREC-6 paragraph
is not statistically significant, due to the small size of the 500-examples test set
Only the authors can say what is right here.
Judge's note: Viewed abstract (p.1), conclusion (p.9) and the TREC-6 paragraph (p.7). The paper itself says the TREC-6 gain is not statistically significant, so the blanket claim covering six tasks is internally inconsistent. 'Significantly' may be meant colloquially. No significance tests are reported for the other datasets.
Second check: another model agreed
Report a problem with this finding
Wording "pretraining on more than two orders of magnitude less data than the 7 million sentence pairs used by McCann et al." compares 103 million words (p.3) with 7 million sentence pairs, with no common unit given to support a >100× gap.
- p. 7 Results, TREC-6 paragraph
pretraining on more than two orders of magnitude less data than the 7 million sentence pairs
- p. 3 Section 3.1
28,595 prepro- cessed Wikipedia articles and 103 million words
Only the authors can say what is right here.
Judge's note: Rests on outside knowledge: the typical number of words per sentence pair in the CoVe MT data. In-paper, 103M words against 7M sentence pairs does not evidently amount to a two-orders-of-magnitude difference. The authors should state the unit of comparison.
Second check: another model agreed
Report a problem with this finding
Wording "Besides discriminative fine-tuning and triangular learning rates, we propose gradual unfreezing" refers to the paper's STLR as plain 'triangular learning rates', the name of Smith (2017)'s method.
- p. 5 Section 3.3, paragraph before Gradual unfreezing
Besides discriminative fine-tuning and triangular learning rates, we propose
- p. 5 Section 3.3
Besides discriminative fine- tuning and triangular learning rates
Judge's note: Viewed p.5. On p.4 the paper explicitly distinguishes STLR from Smith's triangular LR, so the shorthand here is ambiguous.
Second check: another model agreed
Report a problem with this finding
Wording "'Discr' consistently boosts the performance of 'Full' and 'Freez', except for the large AG": on AG, Table 7 shows Full 5.81 → Full + discr 5.62 (an improvement); only Freez worsens (5.81 → 6.04).
- p. 8 Impact of classifier fine-tuning
‘Discr’ consistently boosts the performance of ‘Full’ and ‘Freez’, except for the large AG
- p. 8 Impact of classifier fine-tuning text vs Table 7
‘Discr’ consistently boosts the performance of ‘Full’ and ‘Freez’, except for the large AG.
Judge's note: Viewed Table 7 on p.8. As worded, the AG exception applies to both methods, but it holds only for 'Freez'.
Second check: another model agreed
Report a problem with this finding
Typo Corpus name spelled "Wikitext-103" (p.3) and "WikiText-103" (p.7).
- p. 3 Section 3.1
Wikitext-103
- p. 7 Impact of pretraining
WikiText-103
Judge's note: Both spellings are visible in the crops. This is a cosmetic inconsistency.
Second check: none (judge only)
Report a problem with this finding
Typo "the small TREC dataset (Voorhees and Tice, 1999) dataset of open-domain" contains a duplicated 'dataset'.
- p. 5 Question Classification paragraph
small TREC dataset (Voorhees and Tice, 1999) dataset of open-domain
Judge's note: The duplication is visible in the crop on p.5.
Second check: none (judge only)
Report a problem with this finding
Typo In the Loshchilov and Hutter (2017) reference, the venue is printed as "Proceedings of the Internal Conference on Learning Representations 2017". It should read 'International'.
- p. 11 References, Loshchilov and Hutter 2017
Proceedings of the Internal Conference on Learning
- p. 11 References, Loshchilov and Hutter 2017
Proceedings of the Internal Conference on Learning Representations 2017
Judge's note: I checked the crop of the reference on page 11, and it is printed as quoted. Another reference in the same list (extract line 1203-1204) correctly prints 'International Conference on Learning Representations 2017'. This is a clear typo.
Second check: none (judge only)