QLoRA: Efficient Finetuning of Quantized LLMs

• p. 2 Table 1, row Guanaco 13B

  916 ± 1

• p. 11 Table 7, Vicuna / GPT-4 column, row Guanaco-13B

  913

• p. 2 Table 1

  Guanaco 13B 10 GB 916 ± 1

• p. 11 Table 7, Vicuna / GPT-4 column

  Guanaco-13B 975 6 913 6 885

Only the authors can say what is right here.

Judge's note: Read both tables on the page images. Table 1 (p2): 1348, 1022, 992, 974, 966, 916, 902, 879. Table 7 Vicuna/GPT-4 column (p11): 1348, 1022, 992, 966, 974, 913, 879, 902. Seven of eight values are identical, so the two report the same tournament, and Section 5.3 points to Table 1 for these ratings. Only Guanaco 13B differs, by 3 points against a stated ±1 interval. No explanation found in the text. Which value is right cannot be told from the paper (916 is also ChatGPT's human-rater Elo in Table 7, which may hint at a copy slip, but that is speculation).

Second check: another model agreed · checked by hand

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 2 Table 1, row Guanaco 7B, Size

  6 GB

• p. 2 Introduction, text beside Table 1

  (7B parameters) requires just 5 GB of memory

• p. 10 Table 6, Guanaco 7B, Memory

  5 GB

• p. 26 Table 13, Guanaco 7B, Size

  5 GB

Judge's note: Verified on page images: Table 1 (p2) row Guanaco 7B reads 6 GB; the sentence directly below it on the same page says 5 GB; Table 6 (p10) Guanaco 7B Memory 5 GB; Table 13 (p26) Guanaco 7B Size 5 GB. The other sizes (41, 21, 26, 10 GB) are identical across Tables 1, 6 and 13, so the columns measure the same thing. Figure 6 labels the 7B model segment 5046 MB, which supports 5 GB. No statement that Table 1 uses a different measure. This cluster also carried the '26 GB Alpaca / more than 20 points' finding; that is adjudicated under C004.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 2 Introduction, paragraph beside Table 1

  outperforms a 26 GB Alpaca model by more than 20 percentage points on the Vicuna benchmark (Table 6)

• p. 10 Table 6, rows Alpaca 13B and Guanaco 7B

  69.4%

• p. 9 Section 5.3

  still scoring nearly 20 percentage points higher than Alpaca 13B

• p. 2 Table 1, row Guanaco 7B, Size

  6 GB

Only the authors can say what is right here.

Judge's note: Table 6 on the p10 image: Guanaco 7B mean 87.0%; Alpaca 65B 41 GB 70.7%, 33B 21 GB 73.6%, 13B 10 GB 69.4%, 7B 5 GB 64.4%. The only 26 GB row is Vicuna 13B (16-bit). No Alpaca row is 26 GB, and the only gap above 20 points is against the 5 GB Alpaca 7B (22.6). The introduction ('more than 20') and Section 5.3 ('nearly 20') also disagree with each other; 17.6 is closer to the latter but still loosely described. A 16-bit Alpaca 13B would be about 26 GB, but no such model is in the cited table. Secondary point from one reviewer: the Alpaca 13B Mean 69.4% is 0.85 below the average of its two columns (70.25), a larger gap than in any other row (others within about 0.3); the Mean may be computed per prompt, so that part is an observation only. The 6 GB vs 5 GB size mismatch also listed here is adjudicated under C003.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 10 Table 6

  Alpaca 13B 4-bit 10 GB 63.8% 76.7% 69.4%

• p. 10 Table 6, Alpaca 13B

  63.8% 76.7% 69.4%

• p. 2 Introduction

  outperforms a 26 GB Alpaca model by more than 20 percentage points

• p. 9 Section 5.3

  nearly 20 percentage points higher than Alpaca 13B

Only the authors can say what is right here.

Judge's note: Two related defects in this cluster, both checked on images (page 10 table, page 2 and page 9 crops). (1) Table 6 has no 26 GB Alpaca: every Alpaca row is 4-bit (5, 10, 21, 41 GB); 26 GB is the 16-bit Vicuna 13B. The gap to Alpaca 13B is 17.6 points, so 'more than 20' (intro) conflicts with the table and with 'nearly 20' (Section 5.3). Even against Alpaca 65B (70.7) or 33B (73.6) the gap is below 20; only Alpaca 7B (64.4, 5 GB) gives more than 20 (22.6). (2) The Mean column is not defined and is not the plain average of the two order columns in most rows (deviations up to 0.3), so it is evidently computed from underlying scores; Alpaca 13B deviates by 0.85, far more than any other row, so 69.4 may be a misprint. If the mean were 70.25 the gap would be 16.75. Searched the text for a definition of Mean or a 16-bit Alpaca model: none found. Only the authors can say which figures are right.

Second check: another model agreed · checked by hand

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 4 Section 2, Memory Requirement of Parameter-Efficient Finetuning

  the input gradients reduce to an average of 18 MB per sequence making them more memory intensive than all LoRA weights combined

• p. 4 same paragraph

  the LoRA parameters take up only 26 MB

• p. 4 Section 2, preceding sentence

  while the LoRA parameters take up only 26 MB

• p. 4 Section 2, Memory requirement

  reduce to an average of 18 MB per sequence making them more memory intensive than all LoRA weights combined

Only the authors can say what is right here.

Judge's note: Read on the p4 image. The setting is stated as batch size 1, so 18 MB per sequence is 18 MB in total, which is less than the 26 MB given for the LoRA parameters; the comparison as written contradicts the paper's own numbers. Nothing in the paragraph or Appendix G reconciles it (Figure 6's small adapter and input-gradient segments carry no legible labels that settle it). Either a figure or the direction of the comparison is wrong; the authors must say which.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 8 Table 4, NFloat4 + DQ row, Mean

  53.1

• p. 8 Table 4, NFloat4 + DQ row

  39.0 44.5 47.5 50.7 57.3 59.2 61.8 63.9 53.1

• p. 8 Table 4, NFloat4 + DQ row, Mean

  NFloat4 + DQ 39.0 44.5 47.5 50.7 57.3 59.2 61.8 63.9 53.1

Only the authors can say what is right here.

Judge's note: Read the row from the page 8 image and recomputed. The BFloat16 and Float4 means reproduce (53.04 -> 53.0, 52.19 -> 52.2). Rounding of the cells cannot explain it: eight values each rounded to one decimal move the mean by at most 0.05, giving at most 53.04. Either the mean or one of the cells is wrong; only the authors can say which. The effect is small (0.1) but it makes NF4+DQ look ahead of BFloat16 instead of equal. No statement in the paper says the mean is taken over anything other than the eight columns.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 10 Section 5.3, paragraph after Table 6

  Guanaco 33B and 65B models outperform all models besides GPT-4 on the Vicuna and OA benchmarks

• p. 11 Table 7, Open Assistant column

  1015

• p. 11 Table 7 caption

  Overall, Guanaco 65B and 33B tend to be preferred to ChatGPT-3.5 on the benchmarks studied.

• p. 11 Table 7, OA column

  ChatGPT-3.5 Turbo 916 7 966 5 1015 2

Judge's note: Read Table 7 from the page 11 image and the paragraph on page 10. The claim holds only for the Vicuna/GPT-4 column (1022 and 992 against 974 for the next model). The next sentence concedes that 'the larger OA benchmark favors ChatGPT', which contradicts the claim instead of explaining it. The Table 7 caption ('tend to be preferred to ChatGPT-3.5 on the benchmarks studied') is hedged and is consistent with the median ranks, so the defect is the sentence in Section 5.3.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 13 Section 6.1, Math, sentence after the factorization example

  the true factorization is 3 × 17 × 43

Judge's note: Read on the page 13 image: the prompt is 'Please factorize 1833' and the model's reply also says 1833, so the number being factorized is not in doubt. The sentence is the authors' own prose, not model output. 3*17*43 = 2193; 3*13*47 = 1833, with 13 and 47 prime.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 8 Section 5.1, Data

  corpora aggregations (FLAN v2 [12])

• p. 22 Appendix B.1, FLAN v2

  The FLAN v2 collection [39]

• p. 17 Reference [12]

  Scaling instruction-finetuned language models

• p. 6 Section 4, Experimental setup

  Flan v2 [39]

Judge's note: Read the three citing passages (pages 6, 8, 22) and the reference entries [12], [39], [60], [61]. The inconsistency is internal to the paper: it calls [39] 'the FLAN v2 collection' itself, so [12] in Section 5.1 should be [39]. [60] and [61] both give Wang et al., Super-NaturalInstructions, EMNLP 2022, with slightly different author order and title wording, so they are a duplicated entry.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 15 Section 7, Instruction Finetuning (continued at top of page)

  Super-NaturalInstructions [61, 50]

• p. 19 Reference [50]

  Multitask prompted training enables zero-shot task generalization

• p. 22 Appendix B.1, FLAN v2

  T0++ [50], Super-Natural Instructions [60]

Only the authors can say what is right here.

Judge's note: Read the citation on the page 15 image, reference [50] in the page 19 crop, and Appendix B.1 in the page 22 crop; entries [60] and [61] read in the text, both Wang et al. Super-NaturalInstructions (they look like two entries for the same work). [50] is not a Super-NaturalInstructions paper. Either [61, 60] was meant, or T0 was meant as its own list item and its name was dropped; the authors must say which.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 18 Reference [23]

  J. Henderson, S. Ruder, et al. Compacter: Efficient low-rank hypercomplex adapter layers.

Only the authors can say what is right here.

Judge's note: Rests on outside knowledge: that the Compacter paper is by R. Karimi Mahabadi, J. Henderson and S. Ruder, with Karimi Mahabadi as first author. The page image shows the entry as quoted. Internally it is also anomalous: every other entry in the list writes out ten authors before 'et al.', whereas this one stops after two, which fits a BibTeX name-parsing failure.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 20 Reference [60]

  Super-naturalinstructions:generalization via declarative instructions on 1600+ tasks. In EMNLP, 2022.

• p. 20 Reference [61]

  Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks.

• p. 20 Reference [60]

  Super-naturalinstructions:generalization via declarative

Judge's note: Viewed the page image. Both entries start with Y. Wang, S. Mishra, P. Alipoormolabashi, Y. Kordi, A. Mirzaei, name the same remaining authors in a different order, carry the same title (bar 'nlp') and the same venue and year (EMNLP 2022). The text cites the work as [61] in the main text (Section 4, 'Super-NaturalInstructions ... [61] with T5') and as [60] in the appendices (A.2 'Wang et al. [60]', B.1, B.2 '[62, 60]'). No explanation such as a preprint versus proceedings version is possible since both are given as EMNLP 2022. The missing space after the colon in [60] is visible in the image.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 24 Appendix B.4, last sentence

  We obtain similar findings for chatbot performance as discussed in .

• p. 24 B.4

  as discussed in .

• p. 24 B.4

  chatbot performance as discussed in .

Judge's note: Viewed the page image: the sentence ends 'as discussed in .' with a blank before the full stop, so it is in the PDF and not an extraction loss. The target is missing (probably the chatbot results in Section 5, but the authors must confirm which section or table was meant).

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 15 Section 8, first paragraph

  we did not establish that QLORA can match full 16-bit finetuning performance at 33B and 65B scales

• p. 16 Section 9, first paragraph

  while not degrading performance relative to a full finetuning baseline

• p. 2 Introduction

  without degrading the runtime or predictive performance compared to a 16-bit fully finetuned baseline

• p. 16 Section 9, paragraph 1

  not degrading performance relative to a full finetuning baseline

Only the authors can say what is right here.

Judge's note: Read Section 8 (page 15), Section 9 (page 16) and the abstract and Section 5 crops; the introduction and Section 4 were read in the text. Section 4 confirms the limitation: full 16-bit finetuning was compared only for RoBERTa and T5 models of 125M to 3B parameters (Table 3), and from 7B to 65B the comparison is with 16-bit LoRA only (Table 4), 'since full finetuning models at and beyond 11B parameters requires more than one server'. The headline claims are worded as results for the 33B/65B models and so overstate what was shown; the paper's own limitation paragraph says so. Wording issue, not a wrong number.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 5 Section 3, top of page

  to use all 2k bits for a k-bit datatype

Judge's note: Read on the p5 image (first line): "ensure a discrete zeropoint of 0 and to use all 2^k bits for a k-bit datatype". The superscript is clear in the image. The surrounding text speaks of 2^k values q_i of the data type (p4), so "values" or "bit combinations" is intended.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 7 Section 4, 'The first focuses on...' paragraph

  of RoBERTA and T5 models sized 125M to 3B parameters

• p. 7 Table 3 header

  T5-80M

• p. 7 Table 3, row BF16, column T5-11B

  62.0

• p. 6 Section 4 setup

  full-finetuning for models up to 3B

Only the authors can say what is right here.

Judge's note: Checked Table 3 and the paragraph on page 7 and the setup on page 6. The smallest model in Table 3 is T5-80M, and the introduction gives '80M to 65B', so '125M' is wrong for this table (it appears to be carried over from the Table 2 setup). The upper bound '3B' is partly defensible: the 'BF16 replication' row has '-' for T5-11B, so the authors' own full-finetuning runs stop at 3B, though the table still prints a BF16 score for 11B without saying where it comes from. The 125M-13B vs 125M-65B point raised here belongs to C017. Capitalisation 'RoBERTA' vs 'RoBERTa' confirmed as a cosmetic point.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 8 Table 4 caption

  while FP4 is consistently one percentage point behind both

• p. 8 Table 4, Float4 row

  63.3

• p. 8 Table 4, rows BFloat16 and Float4

  37.2

• p. 8 Table 4 caption

  FP4 is consistently one percentage point behind both

Judge's note: Read all three rows of Table 4 from the page 8 image and recomputed each column difference. The body text on page 7 says only 'by about 1 percentage point', which fits the mean; the caption's 'consistently' does not describe the cells. The side point about the NF4+DQ mean is handled in C023.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 8 Section 5.1, Training Setup

  We halve the learning rate for 33B and 65B while doubling the batch size.

• p. 23 Table 9, rows 65B All / 65B HH-RLHF / 33B OASST1 / 65B OASST1

  64

• p. 8 Training Setup

  We halve the learning rate for 33B and 65B while doubling the batch size

• p. 23 Table 9, 65B All

  65B All 64 1e-4 2500

Only the authors can say what is right here.

Judge's note: Read Table 9 from the page 23 image. The learning rate is halved as stated (2e-4 to 1e-4) for every 33B and 65B row. The batch size is doubled in only four of the eight 33B/65B rows: it is quadrupled for 65B All and HH-RLHF and unchanged for both OASST1 rows, which are the Guanaco models. Reading 'doubling' as per step (16, 32, 64) fits All and HH-RLHF but not OASST1 or 65B Longform. The sentence is a loose summary of the table.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 13 Section 6.1, Math, first example

  Lee earned $582 mowing lawns last week.

• p. 13 same example, last line

  $528 + $30 = $558

• p. 13 Section 6.1, Math, lead-in

  When Guanaco shows its work, it tends to be accurate, e.g.,

• p. 13 Section 6.1 Math

  When Guanaco shows its work, it tends to be accurate

Only the authors can say what is right here.

Judge's note: Read on the page 13 image. 33*16 = 528 and 528 + 30 = 558, so the worked answer is right and the headline answer $582 is wrong (digits transposed). It is quoted model output, so either the generation was mistranscribed or the model really gave an inconsistent first line; in the second case the example does not illustrate accuracy as presented. Only the authors can say which.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 24 Appendix B.4

  improves MMLU only marginally (0.0 - 0.5 MMLU), while the difference between datasets is up to 40x larger (1.5 - 8.0 MMLU)

• p. 24 B.4

  improves MMLU only marginally (0.0 - 0.5 MMLU)

• p. 24 Table 11, Unnatural Instructions 50000

  42.20

• p. 24 Table 11, Unnatural Instructions, 50000

  38.10 42.20 38.10

Only the authors can say what is right here.

Judge's note: Read B.4 and Table 11 on the page-24 image. I recomputed all Table 11 row and column means; they match the printed values, so the table is internally consistent. Against it, the text's 0.0-0.5 range fails at every level: single cells differ by up to 4.1 across epochs and 2.0 across sizes, epoch means within a dataset differ by up to 2.2 (Unnatural Instructions 39.30 vs 41.50), and size row means by 0.86. Between-dataset gaps in the means are about 4.3 to 10.0, not 1.5-8.0. The '40x' is only reachable as 8.0/0.2, a value the paper does not state. No definition elsewhere in the paper explains the ranges (the sentence that follows even ends with an empty cross-reference, 'as discussed in .'). The qualitative conclusion (dataset matters more than size) still holds, so wording rather than a wrong result; only the authors can say how the ranges were derived.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 25 Appendix F, last sentence

  this verifies the claim that neural network weights.

Only the authors can say what is right here.

Judge's note: Page-25 image: the paragraph ends 'hidden units. this verifies the claim that neural network weights.' The sentence has no predicate (presumably 'are mostly normally distributed') and starts in lower case. It also follows directly on the report that 7.5% of neurons test as non-normal against a 5% threshold, so the intended strength of the claim is unclear; the authors must supply the missing words.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send