Zero-Shot Text-to-Image Generation
2021 · paper · AI and machine learning
Describes a transformer that models text and image tokens as a single stream for text-to-image generation. Trained at large scale (12 billion parameters, 250 million image-text pairs), it is competitive with earlier specialised models in zero-shot evaluation.
Checked: PDF from arxiv.org, fetched 4 Oct 2026. Page numbers are those of that PDF.
6 findings
Reference or label "The ELB now becomes difficult to optimize: as qψ is a discrete distribution, and we cannot use the reparameterization gradient to maximize it.": the encoder distribution is q_φ; ψ parameterises only the transformer prior p_ψ. The 'as ..., and' construction is also ungrammatical.
- p. 3 Section 2.1, second paragraph
The ELB now becomes difficult to optimize: as qψ is a dis-
- p. 2 Section 2, definitions after Equation 1
qφ denotes the distribution over the 32 × 32 image
Judge's note: Page 3 image: the subscript in Section 2.1, second paragraph, is ψ, the same glyph as in p_ψ in the paragraph above. Page 2 defines q_φ as the dVAE encoder distribution and p_ψ as the transformer's joint distribution; no q_ψ is defined anywhere. The sentences before and after use q_φ and q_φ^τ.
Second check: another model agreed
Report a problem with this finding
Reference or label "such as when rendering text (Figure 2b) or when probed on sentences like "an illustration of a baby hedgehog in a christmas sweater walking a dog" (Figure 2c)": the panel letters are swapped. Figure 2(b) is the hedgehog and Figure 2(c) is the neon sign that renders text.
- p. 8 Section 3.3, second paragraph
such as when rendering text (Figure 2b)
- p. 8 Section 3.3, second paragraph
in a christmas sweater walking a dog” (Figure 2c)
- p. 2 Figure 2, sub-captions (b) and (c)
(b) an illustration of a baby hedgehog in a christmas sweater walking a dog
- p. 2 Figure 2, sub-caption (c)
(c) a neon sign that reads “backprop”.
Judge's note: Page 2 image: the hedgehog pictures sit above sub-caption (b) 'an illustration of a baby hedgehog in a christmas sweater walking a dog', the neon signs above (c) 'a neon sign that reads "backprop"'. Page 8 image: Section 3.3 cites 2b for text rendering and 2c for the hedgehog. The pointers to 2a (tapir) and 2d (cat sketch) are correct.
Second check: another model agreed
Report a problem with this finding
Reference or label "The likelihood for pθ is evaluated using the log-laplace distribution (see Appendix A.3 for a derivation)": Appendix A.3 is titled 'The Logit-Laplace Distribution' and says 'we call it the logit-Laplace distribution'.
- p. 3 Section 2.1, second paragraph
for pθ is evaluated using the log-laplace distribution (see
- p. 12 Appendix A.3 heading
A.3. The Logit-Laplace Distribution
- p. 13 Appendix A.3, after Equation 2
we call it the logit-Laplace distribution
Judge's note: Page 3 image shows 'log-laplace'. The page 12 crop shows the heading 'A.3. The Logit-Laplace Distribution'; the page 13 crop shows Equation 2, built from the sigmoid of a Laplace variable on (0,1), named 'logit-Laplace distribution'. The main text gives the wrong name for the distribution it points to.
Second check: another model agreed
Report a problem with this finding
Reference or label The Acknowledgements cite "the contrastive model described in Radford et al. (2019) that we used to rerank the samples from our model"; the reference-list entry for Radford et al. 2019 is "Language models are unsupervised multitask learners", and Section 2.6 cites the reranking contrastive model as "(Radford et al., 2021)". The year should be 2021.
- p. 9 Acknowledgements
trastive model described in Radford et al. (2019) that we
- p. 6 Section 2.6
model (Radford et al., 2021). Given a caption and a candi-
- p. 10 Reference list
Sutskever, I. Language models are unsupervised multitask
- p. 9 Acknowledgements
contrastive model described in Radford et al. (2019)
Judge's note: Page 9 image shows "Radford et al. (2019)" in the Acknowledgements. Page 6 crop shows Section 2.6 citing the pretrained contrastive model as Radford et al., 2021, and Section 3.2 on page 8 also cites Radford et al. (2021). The page 10 crop shows the 2019 entry is the language-model paper. The text layer shows the other Radford entry lists Kim, J. W. as second author, the person being thanked. The inconsistency is internal to the paper, so no outside knowledge is needed.
Second check: another model agreed
Report a problem with this finding
Wording Appendix C says "As described in Section 2.3, this dataset incorporates Conceptual Captions, the text-image pairs from Wikipedia, and a filtered subset of YFCC100M", but Section 2.3 says only that the dataset "does include Conceptual Captions and a filtered subset of YFCC100M" and does not mention Wikipedia.
- p. 14 Appendix C, first paragraph
the text-image pairs from Wikipedia, and a
- p. 4 Section 2.3, second paragraph
does not include MS-COCO, but does include Conceptual
- p. 14 Appendix C, first paragraph
As described in Section 2.3, this dataset incorporates Conceptual Captions, the text-image pairs from Wikipedia, and a filtered subset of YFCC100M
- p. 4 Section 2.3, second paragraph
does not include MS-COCO, but does include Conceptual Captions and a filtered subset of YFCC100M
Only the authors can say what is right here.
Judge's note: Both crops confirm the wording. A search of the text finds "Wikipedia" only in Appendix C. Section 2.3's "does include" need not be exhaustive, so the dataset composition may be consistent; the defect is that the back-reference attributes the Wikipedia component to a section that does not contain it.
Second check: another model agreed
Report a problem with this finding
Wording Appendix A.2 introduces its list with "Several quantities are decayed during training, all of which use a cosine schedule", but item 1 reads "The KL weight β is increased from 0 to 6.6 over the first 5000 updates."
- p. 12 Appendix A.2, first paragraph and item 1
quantities are decayed during training, all of which use a cosine schedule
- p. 12 Appendix A.2, item 1
The KL weight β is increased from 0 to 6.6 over the first 5000 updates.
Judge's note: Page 12 image and crop confirm both sentences. Items 2 and 3 are decays (temperature and step size are annealed downward), and the closing sentence speaks of "the decay schedules for the relaxation temperature and the step size", so only the lead-in is loose: "scheduled" or "varied" would fit all three. The meaning is recoverable.
Second check: another model agreed
Report a problem with this finding
11 minor (typos and the like; most checked by the judge only)
Typo "who showed that the DRAW Gregor et al. (2015) generative model": a textual citation sits mid-phrase where a parenthetical one, 'DRAW (Gregor et al., 2015)', is needed.
- p. 1 Introduction
the DRAW Gregor et al. (2015) generative
Judge's note: Page 1 image shows the text exactly as quoted. The same paragraph uses the parenthetical form correctly for '(Goodfellow et al., 2014)', so this is a citation-command slip, cosmetic only.
Second check: none (judge only)
Report a problem with this finding
Typo "This previously required custom approaches (Isola et al., 2017), rather emerging as a capability of a single, large generative model.": 'than' is missing ('rather than emerging').
- p. 2 Introduction, last paragraph
rather emerging as a capability of a single, large generative
- p. 2 Intro, last paragraph
rather emerging as a capability
Judge's note: Page 2 image, end of the Introduction: the sentence reads as quoted and does not parse without 'than'.
Second check: none (judge only)
Report a problem with this finding
Typo Section 3.3 quotes the caption as "the exact same cat on the top as a sketch at the bottom"; Figure 2(d) and Figure 14(a) print "the exact same cat on the top as a sketch on the bottom".
- p. 8 Section 3.3, third paragraph
at the bottom” and the top 15 × 32 part of the image token
- p. 2 Figure 2, sub-caption (d)
top as a sketch on the bottom
- p. 8 Section 3.3, third paragraph
“the exact same cat on the top as a sketch at the bottom”
- p. 2 Figure 2(d) caption
(d) the exact same cat on the top as a sketch on the bottom
Judge's note: Page 8 image shows 'at the bottom' inside quotation marks, with a pointer to Figure 2d. The page 2 image (Figure 2(d)) and page 20 image (Figure 14(a)) both show 'on the bottom'. A one-word difference in a prompt presented as verbatim; it has no effect on the meaning.
Second check: none (judge only)
Report a problem with this finding
Typo "the true validation ELB with qφ intsead of qτφ": 'intsead' should be 'instead'.
- p. 3 Section 2.1, first bullet
intsead of
- p. 3 Section 2.1, first bullet
intsead of q
- p. 3 Section 2.1, first bullet
with qφ intsead of qτ
- p. 3
intsead of qτ φ
Judge's note: Page 3 image, first bullet in Section 2.1: the misspelling is in the rendered page.
Second check: none (judge only)
Report a problem with this finding
Typo "250 million text-images pairs from the internet": should be 'text-image pairs', as in the preceding paragraph and Appendix C.
- p. 4 Section 2.3, second paragraph
250 million text-images pairs from the internet
- p. 14 Appendix C
250 million text-image pairs
Judge's note: The page 4 crop shows 'text-images pairs' in the second paragraph of Section 2.3 and 'text-image pairs' in the first. Appendix C (text layer) has '250 million text-image pairs'.
Second check: none (judge only)
Report a problem with this finding
Typo "We found that Sun et al. (2020) had independently developed similar procedure for training convolutional networks in 4-bit precision": the article 'a' is missing before 'similar procedure'.
- p. 5 Section 2.4, last paragraph
independently developed similar procedure for training con-
- p. 5 Section 2.4
had independently developed similar procedure
Judge's note: The page 5 crop shows the sentence as quoted.
Second check: none (judge only)
Report a problem with this finding
Typo "We found an 12% overlap rate for this dataset": 'an 12%' should be 'a 12%'.
- p. 7 Section 3.1, CUB paragraph
found an 12% overlap rate for this dataset
- p. 7 Section 3.1
We found an 12% overlap rate
Judge's note: Page 7 image, CUB paragraph of Section 3.1, reads 'an 12% overlap rate'. The paper gives the CUB overlap rate nowhere else, so the number itself cannot be checked; the confirmed defect is the article only.
Second check: none (judge only)
Report a problem with this finding
Typo Appendix C: "If we were to use to very tall or wide images" has a stray second "to"; it should read "to use very tall or wide images".
- p. 15 Appendix C, continuation below Listing 2
If we were to use to very tall or wide images
Judge's note: The crop of page 15 shows the sentence as quoted.
Second check: none (judge only)
Report a problem with this finding
Typo Appendix D: "Since the gradient scaling keeps the gradients close to right end of the exponent range" is missing an article; it should read "close to the right end".
- p. 16 Appendix D, guideline 3, second paragraph
Since the gradient scaling keeps the gradients close to right
Judge's note: The crop of page 16 shows the sentence as quoted, running across a line break with no "the" on either line.
Second check: none (judge only)
Report a problem with this finding
Typo Appendix E.2, step 5: "The orthogonalizalied P matrices are stored in 1-6-9 format" misspells "orthogonalized".
- p. 17 Appendix E.2, step 5
The orthogonalizalied P matrices are stored in 1-6-9 format
- p. 17 E.2 step 5
The orthogonalizalied P matrices
- p. 17 E.2 Implementation Details, step 5
The orthogonalizalied P matrices are stored
Judge's note: The crop of page 17 shows "orthogonalizalied" within one line, so it is not a hyphenation or extraction artefact.
Second check: none (judge only)
Report a problem with this finding
Typo Appendix E.2 twice prints "errors buffers" for "error buffers": step 3, "from the errors buffers and the Q matrices" (page 17), and optimization 4, "it suffices to store the sums of the errors buffers taken across all GPUs" (page 19).
- p. 17 Appendix E.2, step 3
from the errors buffers and the Q matrices
- p. 19 Appendix E.2, optimization 4, second paragraph
it suffices to store the sums of the errors
- p. 19 Optimizations 4
sums of the errors buffers taken across all GPUs
Judge's note: Both crops show "errors buffers". The text layer has "error buffer(s)" in about twenty other places, including the sentences next to each occurrence.
Second check: none (judge only)