LoRA: Low-Rank Adaptation of Large Language Models

• p. 7 Table 3, GPT-2 M (FTTop2)*

  25.19M

• p. 21 Table 13, GPT-2 Medium FTTop2

  24M

• p. 22 Table 14, FTTop2

  FTTop2 (24M)

• p. 21 Table 13

  FTTop2 24M

Only the authors can say what is right here.

Judge's note: Checked the page images. Table 3 (p7) prints 25.19M for GPT-2 M (FT^Top2)*. Table 13 (p21) prints 24M and Table 14 (p22) prints (24M), all for the same baseline and model. Elsewhere in these tables, parameter counts are truncated or rounded consistently (354.92M→354M, 11.09M→11M). By either rule, 25.19M gives 25M, not 24M. I found no explanation in the text. The value may come from Li & Liang, but only the authors can say which count is right.

Second check: another model agreed

• p. 5 Section 5, end of first paragraph

  We use NVIDIA Tesla V100 for all experiments.

• p. 4 Table 1 caption

  We use an NVIDIA Quadro RTX8000.

• p. 17 Appendix B

  on an NVIDIA Quadro RTX8000

Judge's note: Page 5 has the V100 statement. The Table 1 caption on page 4 and Appendix B (text line 1317) name the RTX8000. 'All experiments' probably means the task-performance experiments, but as worded it overstates.

Second check: another model agreed

• p. 5 Section 4.2 and footnote 5

  We also observe a 25% speedup

• p. 5 footnote 5

  the throughput is 43.1 tokens/s per V100 GPU for LoRA

• p. 5 footnote 5

  32.5 tokens/s per V100 GPU for LoRA

Only the authors can say what is right here.

Judge's note: Footnote 5 on page 5 checked. A 25% speedup matches only if it means the reduction in time per token (24.6%). As a ratio of throughputs it is 1.33x. The authors should state which definition they mean.

Second check: another model agreed

• p. 9 Section 7, first paragraph

  (up to 10,000×) without adversely affecting task performances

• p. 8 Table 4, GPT-3 (FT) and GPT-3 (LoRA) rows

  175,255.8M

Judge's note: Checked p9 Section 7 and Table 4 (p8). The 10,000× figure matches the checkpoint-size reduction for the 18M configuration (350GB to 35MB, Section 4.2) and the abstract's '10,000 times'. As an upper bound ('up to'), it is below the paper's own smallest-budget ratio. This is a wording point: the figure reads as an order of magnitude and understates rather than overstates.

Second check: another model agreed

• p. 23 Table 16, GPT-3 (PrefixLayer), MNLI(m)-392K

  89.6

• p. 24 Table 17, PrefixTune row

  lp = li = 8

• p. 23 Table 15, PrefixLayer lp = 8, li = 8, MNLI-m

  89.5

• p. 8 Table 4, GPT-3 (PreLayer) MNLI-m

  89.5

Only the authors can say what is right here.

Judge's note: Table 17 (p24) gives lp=li=8 and LR 1.00E-04 for 392K, the same as the main PreLayer run (Table 12). Table 15 reports lp=8, li=8 as 89.5, and Table 4 reports PreLayer as 89.5. Table 16 prints 89.6, the Table 15 value for lp=32, li=4. This could be a separate run, so it is a query for the authors.

Second check: another model agreed

• p. 10 Table 5 caption

  Adapting both Wq and Wv gives the best performance overall.

• p. 10 Section 7.1 text below Table 5

  while adapting both Wq and Wv yields the best result

• p. 10 Table 5, Wq,Wk,Wv,Wo column

  91.7

Only the authors can say what is right here.

Judge's note: Checked Table 5 on p10. Both 73.7 entries are bold, and only 91.7 (the four-matrix column) is bold for MultiNLI. The unqualified 'best' claim does not match the table. The authors may mean among one- or two-type options; only they can say which.

Second check: another model agreed

• p. 11 Section 7.2, Subspace similarity between different r

  obtain the right-singular unitary matrices UAr=8 and UAr=64

• p. 22 Appendix G, first paragraph

  taking columns of the left singular matrices of A and B

• p. 22 Appendix G

  obtained by taking columns of the left singular matrices of A and B

• p. 11 Section 7.2

  obtain the right-singular unitary matrices

Only the authors can say what is right here.

Judge's note: Checked p11 (Section 7.2 and footnote 7) and p22 (Appendix G). A ∈ R^{r×k}, so its left-singular matrix is r×r and could not be compared across r=8 and r=64. The main text's right-singular choice is the workable one. Appendix G's generic wording (left singular, d×i) contradicts it unless A there stands for A^T, which the paper does not say.

Second check: another model agreed

• p. 23 Table 16, GPT-3 (LoRA), MNLI(m)-392K

  91.7

• p. 24 Table 17, Adaptation-Specific, LoRA

  rq = rv = 8

• p. 23 Table 15, LoRA rq = rv = 8, MNLI-m

  91.6

Only the authors can say what is right here.

Judge's note: Table 17 (p24) gives LoRA rq=rv=8, LR 2E-4, batch 128 and 2 epochs for 392K, which matches Table 12. Table 15 (p23) reports rq=rv=8 as 91.6, and Table 4 (p8) reports the 37.7M LoRA as 91.6. Table 16 prints 91.7, the value Table 15 gives for rv=2 and for the all-four configurations. The Fine-Tune (89.5) and PrefixEmbed (88.6) entries in Table 16 match the main tables. Table 16 could be a separate run, so only the authors can say which value is right.

Second check: another model agreed

• p. 26 Table 18 caption

  Unlike on GPT-3 where r = 1 suffices for many tasks

• p. 24 Appendix H.2

  between 4 and 16 depending on the metric used, which is similar to that for GPT-3 175B

Judge's note: I checked the page 26 caption and H.2 on page 24. The caption sets the two models up as different and then calls them similar. It may mean similar in that both ranks are small relative to d_model, but the wording leaves this ambiguous.

Second check: another model agreed

• p. 15 References

  Models are Unsupervised Multitask Learners. pp. 24, b.

• p. 1 Section 1

  GPT-2 (Radford et al., b)

• p. 15 References

  Language Models are Unsupervised Multitask Learners. pp. 24, b.

Judge's note: The p15 reference list shows no year in either Radford entry, and the in-text citations (e.g. p1) read 'Radford et al., b'.

Second check: none (judge only)

• p. 2 Section 1, first paragraph

  these method often fail to match

• p. 2 Introduction, first paragraph

  these method often fail to match the fine-tuning baselines

• p. 2 Intro

  these method often fail to

Judge's note: Seen on the page 2 image, at the top of the page.

Second check: none (judge only)

• p. 15 References

  Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019.

• p. 2 Terminologies and Conventions

  Loshchilov & Hutter, 2019

• p. 19 Appendix D.3

  AdamW (Loshchilov & Hutter, 2017)

• p. 13 References

  Bert: Pre-training of deep bidirectional transformers for language understanding, 2019a.

Judge's note: The p15 reference list shows both entries with the same title. The p2 text cites 2019 and Appendix D.3/D.4 (p19) cite 2017. The BERT duplicate this cluster also mentions is handled in C027.

Second check: none (judge only)

• p. 3 Equation (2)

  log (pΦ0+∆Φ(Θ)(yt|x, y<t))

Judge's note: On the page 3 image, Eq. (1) has an uppercase P_Φ and Eq. (2) a lowercase p. The meaning is clear; this is only a notation inconsistency.

Second check: none (judge only)

• p. 5 Section 4.2

  we reduce that VRAM usage

• p. 3 Section 3, Adapter Layers

  there is no direct ways to bypass

Judge's note: Both phrases seen on the page 5 and page 3 images.

Second check: none (judge only)

• p. 4 Table 1 caption

  Infernece latency

• p. 4 Table 1 caption

  Infernece latency of a single forward pass

• p. 4

  Table 1: Infernece latency of a single forward pass in GPT-2 medium

Judge's note: Seen in the Table 1 caption on the page 4 image.

Second check: none (judge only)

• p. 4 Section 4.1

  low “instrisic dimension”

• p. 4

  have a low “instrisic dimension” and can still

Judge's note: Seen in Section 4.1 on the page 4 image. The same paragraph spells it correctly as "intrinsic rank".

Second check: none (judge only)

• p. 9 Section 6, Parameter-Efficient Adaptation

  A comtenporary extension of adapter

• p. 9 Section 6, Parameter-Efficient Adaptation

  A comtenporary extension of adapter is

• p. 4 Table 1 caption

  Infernece latency

• p. 9 Related Works

  A comtenporary extension

Judge's note: All five appear in the page images: p9 Related Works, p4 Table 1 caption and Section 4.1, p6 Section 5.1, p23 Table 16 caption. All are cosmetic.

Second check: none (judge only)

• p. 5 Section 4.2, first paragraph

  parameter-efficiency.We further study

• p. 5 Section 5.1, Fine-Tuning paragraph

  undergo gradient updates.A simple

Judge's note: Both are visible on the page 5 image, so this is not an extraction artefact.

Second check: none (judge only)

• p. 6 Table 2, RoBlarge (AdptP)† 3.0M row, MRPC

  90.2±.7

• p. 6 Table 2, RoBlarge (LoRA)† row, MRPC

  90.2±1.0

Judge's note: Checked in the crop. Elsewhere tied values are bolded on both entries (e.g. QNLI 94.8 in the † section). The bolding might have used unrounded values, which only the authors can say.

Second check: none (judge only)

• p. 6 Section 5.1, Adapter tuning

  another deign proposed in Pfeiffer et al. (2021)

• p. 6 Sec 5.1 Adapter tuning

  another deign proposed in Pfeiffer et al.

• p. 6

  This is very similar to another deign proposed in Pfeiffer et al. (2021)

Judge's note: Both are visible on the page 6 image.

Second check: none (judge only)

• p. 6 Section 5.1, Prefix-embedding tuning

  We use lp (resp. li) denote the number

Judge's note: Seen on the page 6 image.

Second check: none (judge only)

• p. 10 Section 7.2, first paragraph

  {Wq, Wk, Wv, Wc}

• p. 10

  {Wq, Wk, Wv, Wc}, and just Wq for a comparison.

Judge's note: The page 10 image shows a subscript c. Wc is not defined anywhere in the paper.

Second check: none (judge only)

• p. 13 References

  Bert: Pre-training of deep bidirectional transformers for language understanding, 2019a.

Judge's note: The p13 reference list shows two Devlin et al. entries with the same title. The text cites 2019a (Section 5.1) and 2019b (Section 6).

Second check: none (judge only)

• p. 17 Appendix B, first paragraph

  large enough to full utilize the hardware parallelism

• p. 17 Appendix B

  large enough to full utilize the

Judge's note: The p17 image shows 'full utilize'. This is a cosmetic grammar slip.

Second check: none (judge only)

• p. 18 Appendix C, WikiSQL

  The dataset is release under the BSD 3-Clause License.

Judge's note: I checked the crop of page 18. The WikiSQL paragraph reads "is release under", while the neighbouring SAMSum and GLUE paragraphs say "released under". This is a grammatical slip.

Second check: none (judge only)

• p. 21 Table 12 caption

  GPT-3 adaption methods

• p. 21 Table 13 caption

  all adaption approaches

Judge's note: I checked page 21 and the Table 13 crop, and both captions read "adaption". "Adaption" is an accepted variant spelling, so this is only an inconsistency with the paper's usual "adaptation".

Second check: none (judge only)

• p. 22 Appendix G, first sentence

  ∥U i⊤ A UB∥2 F / min{i,j}

Judge's note: The page 22 image and the text layer ("UB∥2") both show U_B without the superscript j, although the left-hand side and the later definitions use U_B^j. This is a notational slip.

Second check: none (judge only)

• p. 22 Appendix F.3, first sentence

  in the low-data regime. we randomly sample

Judge's note: I checked the page 22 image. The punctuation is as quoted.

Second check: none (judge only)

• p. 23 Table 15, PrefixEmbed lp = 32, li = 8

  0.4 M

Judge's note: The formula |Θ| = d_model × (lp + li) is in Section 5.1. The other PrefixEmbed rows are rounded, not truncated: 0.885→0.9 and 1.67→1.7. The 0.4 M entry is inconsistent with that convention.

Second check: none (judge only)

• p. 23 Table 16 caption

  LoRA performs exhibits favorable sample-efficiency

Judge's note: I checked the page 23 image. The caption is as quoted.

Second check: none (judge only)

• p. 24 Table 17, Adaptation-Specific rows

  PrefixTune

Judge's note: I checked the page 24 image. The label is inconsistent with the rest of the table, but lp = li = 8 makes clear that it means PrefixLayer.

Second check: none (judge only)