Toolformer: Language Models Can Teach Themselves to Use Tools
Advances in Neural Information Processing Systems 36, 2023 · paper · AI and machine learning
Introduces Toolformer, a language model trained in a self-supervised way to decide which external tools to call, such as a calculator, search engine or translator, and how to use the results. It improves zero-shot performance across tasks without losing core language modeling ability.
Checked: PDF from arxiv.org, fetched 4 Oct 2026. Page numbers are those of that PDF.
4 findings
Substance GPT-J is described as having "6.7B parameters" (p.2) and as a "6.7B parameter GPT-J model" (p.11), but the paper's own reference is titled "GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model" (p.14).
- p. 2 Introduction
with 6.7B parameters
- p. 11 Conclusion
6.7B parameter GPT-J model
- p. 14 References, Wang and Komatsuzaki 2021
GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model
Only the authors can say what is right here.
Judge's note: Rests on outside knowledge: GPT-J has about 6B (6.05B) parameters. On the page images, p.2 reads "with 6.7B parameters" and p.11 reads "6.7B parameter GPT-J model". The Wang and Komatsuzaki 2021 entry on p.14 is titled "GPT-J-6B: A 6 Billion Parameter ...". Figure 4 also places GPT-J near 6700M on the x-axis. I found no other GPT-J variant or alternative parameter count anywhere in the paper, so the paper's text and its own reference title disagree.
Second check: another model agreed
Report a problem with this finding
Substance Table 9, WebQS k=1 row "19.3 17.1 19.9 8.5": the All score of 19.3 cannot be the mix of AC 17.1 (on 8.5% of examples) and NC 19.9 (on the rest), which gives about 19.66.
- p. 9 Table 9, WebQS, k=1 row
19.3 17.1 19.9 8.5
Only the authors can say what is right here.
Judge's note: The Table 9 caption defines AC as the subset with an API call and NC as "all remaining examples", so All should equal the weighted mean of AC and NC. Even with every value pushed to its rounding limit, the mean stays at least 19.61. The other rows reproduce within rounding: T-REx k=1 47.8, T-REx k=3 52.9, WebQS k=3 26.3. I found no statement explaining this row, and only the authors can say which value is wrong.
Second check: another model agreed
Report a problem with this finding
Reference or label The text refers to "“Fast train success” in the fourth example", but in Table 10 that example is the fifth row (score 0.92).
- p. 9 Section 5, Data Quality
“Fast train success” in the fourth example
- p. 10 Table 10, fifth row
WikiSearch(Fast train success)
- p. 10 Table 10, fifth row
Catch this fast train to success!
Judge's note: Table 10 rows on the p.10 image, in order: Flodden 5.49, WL Calendar 2.11, Nile 2.08, Venus Calculator 1.59, Fast train 0.92. The fourth row (Venus) is marked useful, so the text cannot mean that row. The text is on p.9.
Second check: another model agreed
Report a problem with this finding
Reference or label Appendix B cites "DeepSpeed’s ZeRO-3 (Rasley et al., 2020)", but the reference list has no Rasley entry.
- p. 17 Appendix B, Toolformer Training
DeepSpeed’s ZeRO-3 (Rasley et al., 2020)
- p. 17 Appendix B
DeepSpeed’s ZeRO-3 (Rasley
Judge's note: Viewed crop C023-a on p.17. A grep of main.txt finds 'Rasley' only at the citation (line 1951). The reference list goes from Radford et al. 2019 directly to Reichart and Rappoport 2007.
Second check: another model agreed
Report a problem with this finding
16 minor (typos and the like; most checked by the judge only)
Wording The Figure 2 caption indexes the per-position API call candidates as "c_i^1, c_i^2, ..., c_i^k", but Section 2 uses m for the number of calls per position ("up to m API calls c_i^1, ..., c_i^m") and k for the number of positions.
- p. 2 Figure 2 caption
API call candidates c1 i , c2 i , . . . , ck i
- p. 3 Section 2, Sampling API Calls
we then obtain up to m API calls c1 i , . . . , cm i
- p. 2 Figure 2 caption
sample a position i and corresponding API call candidates c1 i , c2 i , . . . , ck i
- p. 3 Section 2, Sampling API Calls
we then obtain up to m API calls
Judge's note: I checked the page images. On p.2 the caption superscript is k. On p.3, Section 2 defines k as the maximum number of candidate positions and m as the number of API calls per position. Appendix A (p.15) uses "top k = 5 such positions" and "m = 5 API calls". The caption should use m. This is a notation inconsistency only.
Second check: another model agreed
Report a problem with this finding
Wording Section 4.1 says "we only consider texts for the calculator tool if they contain at least three numbers", but Appendix A.1 also admits texts under criterion (ii), which needs only one of the listed sequences "followed by a number".
- p. 4 Section 4.1, Dataset Generation
we only consider texts for the calculator tool if they contain at least three numbers
- p. 15 Appendix A.1, Calculator
(ii) contain one of the sequences “=”, “equals”, “equal to”, “total of”, “average of” followed by a number
Judge's note: Both passages are on the page images: p.4 (Section 4.1) and p.15 (Appendix A.1). Criteria (i) and (iii) require three numbers; criterion (ii) does not. The main text introduces the rule "For example" and points to Appendix A, but its "only ... if" wording states a necessary condition that the appendix does not apply.
Second check: another model agreed
Report a problem with this finding
Wording Table 9, WebQS k=10 row "26.3 26.4 – 100.0": with API calls on 100% of examples and no NC subset, All (26.3) should equal AC (26.4).
- p. 9 Table 9, WebQS, k=10 row
26.3 26.4 – 100.0
- p. 9 Section 5 text
and 100% for k = 10
Only the authors can say what is right here.
Judge's note: The text on p.9 also says "100% for k = 10". The two values could only both round as printed if a few unlisted NC examples (under 0.05% of the set) existed while AC sat at its lower rounding edge, and the "–" in the NC column suggests there are none. Only the authors can say which value is right.
Second check: another model agreed
Report a problem with this finding
Wording Calculator prompt example: the Input reads "This is three times less than the 2169 goals last year." but the Output reads "This is twenty goals more than the [Calculator(723 - 20)] 703 goals last year.", so the Output is not the Input with calls inserted.
- p. 16 Appendix A.2, Calculator prompt, third example
This is three times less than the 2169 goals last year.
- p. 16 Appendix A.2, Calculator prompt, third example output
This is twenty goals more than the [Calculator(723 - 20)] 703 goals last year.
- p. 16 Appendix A.2, Calculator prompt, third example
three times less than the 2169 goals
- p. 16 same example, Input
three times less than the 2169 goals last year
Only the authors can say what is right here.
Judge's note: Viewed crops C019-a and C019-b on p.16. In every other demonstration the Output repeats the Input text with API calls added. Here the second sentence changes content. Both versions are internally consistent (723*3=2169; 723-20=703), but they do not match each other. Only the authors can say which prompt text was actually used.
Second check: another model agreed
Report a problem with this finding
Wording Calculator prompt example "[Calculator(658,893 / 11.4%)] 5,763,868" uses '%' and thousands separators, and the call does not yield the printed number (658893/0.114 ≈ 5,779,763).
- p. 16 Appendix A.2, Calculator prompt, second example
[Calculator(658,893 / 11.4%)] 5,763,868
Judge's note: Viewed crop C020-a on p.16. Appendix A.1 (text near 'only supports the operators +, −, ∗, and /. It does not return any result for syntactically invalid equations') shows that the demonstrated call is outside the calculator's stated syntax. The mismatch comes from the rounded 11.4% in the source sentence. Minor illustrative inconsistency.
Second check: another model agreed
Report a problem with this finding
Wording MT prompt Chinese example: the Input begins with the Chinese span followed by "a plane of reference...", but the Output is "[MT(南京高淳县住房和城乡建设局 城市新区 设 计)] a plane of reference...", so the original Chinese text is dropped and appears only inside the call.
- p. 16 Appendix A.2, Machine Translation prompt, third example
Output: [MT(南京高淳县住房和城乡建设局 城市新区 设 计)] a plane of reference
Only the authors can say what is right here.
Judge's note: Viewed crop C021-a on p.16. The other two MT examples keep the source text and insert the call after it. Here the Chinese text is replaced by the call, which is inconsistent with the Output = Input + call pattern.
Second check: another model agreed
Report a problem with this finding
Wording Appendix D example Calendar output "Today is Sunday, November 20, 2020" pairs a weekday and a date that do not agree.
- p. 17 Appendix D, DATESET
Today is Sunday, November 20, 2020
- p. 17 Appendix D
“Today is Sunday, November 20, 2020”
Only the authors can say what is right here.
Judge's note: Rests on outside knowledge: November 20, 2020 fell on a Friday (1 Jan 2020 was a Wednesday). The p.17 crop shows the text as quoted. This is an illustrative example in the DATESET description, not a result.
Second check: another model agreed
Report a problem with this finding
Typo Section 2 defines the kept positions as I = {i | p_i > τ_s} (strict inequality), while Appendix A paraphrases this as positions where the <API> probability "is at least 5%".
- p. 3 Section 2
I = {i | pi > τs}
- p. 15 Appendix A
probability of the <API> token is at least 5%
Judge's note: Both statements are on the page images (p.3 and p.15). The difference between > and >= is negligible for continuous probabilities, but the two wordings do not match exactly.
Second check: none (judge only)
Report a problem with this finding
Typo "to ensure a sufficiently larger number of examples" should read "sufficiently large number of examples".
- p. 5 Section 4.1, Dataset Generation
to ensure a suffi- ciently larger number of examples
- p. 5 Section 4.1, Dataset Generation
suffi- ciently larger number of examples
- p. 5 Section 4.1 Dataset Generation
ensure a suffi-ciently larger number of examples
Judge's note: The phrase appears as quoted on the p.5 page image.
Second check: none (judge only)
Report a problem with this finding
Typo "we only at most one API call per input" is missing a verb (e.g. "we only allow at most one API call per input").
- p. 6 Section 4.2, first paragraph
we only at most one API call per input
Judge's note: The p.6 page image shows the same words with no verb, so this is not an extraction artefact.
Second check: none (judge only)
Report a problem with this finding
Typo The Table 5 caption "Results for various question answering dataset." should read "datasets".
- p. 7 Table 5 caption
Results for various question answering dataset.
Judge's note: The caption reads as quoted on the p.7 image, and the table shows three datasets.
Second check: none (judge only)
Report a problem with this finding
Typo "Toolformer learns for itself to explicitly asks for the right information" should read "to explicitly ask".
- p. 10 Section 6, Language Model Pretraining
to explicitly asks for the right information
- p. 10 Section 6, Language Model Pretraining
learns for itself to explicitly asks for the right information
- p. 10 Related Work, Language Model Pretraining
Toolformer learns for itself to explicitly asks for the right information
- p. 10
Toolformer learns for itself to explicitly asks for the right information.
Judge's note: The phrase appears as quoted on the p.10 page image.
Second check: none (judge only)
Report a problem with this finding
Typo "which tools needs to be used" should read "which tools need to be used".
- p. 10 Section 6, Tool Use
which tools needs to be used
Judge's note: The phrase appears as quoted on the p.10 page image, in the Tool Use paragraph.
Second check: none (judge only)
Report a problem with this finding
Typo Reference titles lower-case model/dataset acronyms, e.g. "Htlm: Hyper-text pre-training...", "Talm: Tool augmented language models", "Webgpt: Browser-assisted...", "Ctrl: A conditional transformer...", while the body uses HTLM/TALM/WebGPT/CTRL.
- p. 11 References, Aghajanyan et al. 2021
Htlm: Hyper-text pre-training and prompting of language models.
- p. 13 References, Parisi et al. 2022
Talm: Tool augmented language models.
- p. 14 References, Zhang et al. 2022
Opt: Open pre-trained transformer language models.
Judge's note: Viewed crops: p.11 shows 'Htlm:', p.13 shows 'Talm:' and 'Webgpt:', p.12 shows 'Ctrl:'. A BibTeX casing artefact that is visible in print. Cosmetic only.
Second check: none (judge only)
Report a problem with this finding
Typo Joulin et al. 2016 reference prints the author as "Hérve Jégou" (accent on the wrong vowel).
- p. 12 References, Joulin et al. 2016
Hérve Jégou
Only the authors can say what is right here.
Judge's note: Rests on outside knowledge: the author's name is spelled Hervé Jégou. The p.12 crop shows 'Hérve Jégou' as printed.
Second check: none (judge only)
Report a problem with this finding
Typo MT prompt example misspells "(“The Supressed Man”)" (Suppressed) in both Input and Output.
- p. 16 Appendix A.2, Machine Translation prompt
(“The Supressed Man”)
Judge's note: Viewed crop C019-a/C019-b, right column of p.16. 'Supressed' is printed twice. This is a spelling error in the published prompt, though it may have been copied verbatim from the prompt actually used.
Second check: none (judge only)