Datasheets for Datasets
2018 · paper · AI and machine learning
Proposes that every machine learning dataset be accompanied by a datasheet documenting its motivation, composition, collection process and recommended uses. It supplies a question set and workflow for dataset creators, aiming to increase transparency and accountability.
Checked: PDF from arxiv.org, fetched 4 Oct 2026. Page numbers are those of that PDF.
2 findings
Reference or label References [6] and [7] are the same work entered twice ("Yang Trista Cao and Hal Daumé. 2020. Toward Gender-Inclusive Coreference Resolution. In Proceedings of ... (ACL). abs/1910.13913." and "Yang Trista Cao and Hal Daumé, III. 2020. Toward Gender-Inclusive Coreference Resolution. In Proceedings of ... (ACL)."); the text cites it as [6] in Section 3 and as [7] in Section 4.
- p. 12 References [6]
[6] Yang Trista Cao and Hal Daumé. 2020. Toward Gender-Inclusive Coreference Resolution.
- p. 12 References [6], last line
abs/1910.13913.
- p. 12 References [7]
[7] Yang Trista Cao and Hal Daumé, III. 2020. Toward Gender-Inclusive Coreference
- p. 4 Section 3, second paragraph
please see that of Cao and Daumé [6]
Judge's note: Read both entries on page 12: same title, year and venue; they differ only in the author suffix and the unlabelled "abs/1910.13913." at the end of [6]. Page 4 cites "Cao and Daumé [6]" and page 10 cites "[e.g., 7, 10, 23, 26]". Both entries name the ACL proceedings, so they are not presented as a preprint and a published version.
Second check: another model agreed
Report a problem with this finding
Wording Datasheet footnote 1 says "All information in this datasheet is taken from one of the following five sources" but lists four URLs.
- p. 15 Fig. 1 datasheet, footnote 1 (left column, bottom)
is taken from one of the following five
- p. 15 Fig. 1 datasheet, footnote 1
taken from one of the following five sources
- p. 15 Fig. 1 footnote 1
All information in this datasheet is taken from one of the following five sources
- p. 15
All information in this datasheet is taken from one of the following five
Only the authors can say what is right here.
Judge's note: Read on the crop of page 15. The semicolon-separated items are: the movie-review-data page; xxx.lanl.gov/pdf/cs/0409058v1; rt-polaritydata.README.1.0.txt; poldata.README.2.0.txt. The line breaks fall inside single URLs and do not create a fifth item. The Maintenance answer on page 18 names further README and diff files, so either a source was dropped from the footnote or the count is wrong; only the authors can say which.
Second check: another model agreed
Report a problem with this finding
11 minor (typos and the like; most checked by the judge only)
Reference or label The datasheet points to a section by a quoted name that does not exist: "details on the mapping from number of stars to polarity is given below in “Data Preprocessing”"; the section is headed "Preprocessing/cleaning/labeling".
- p. 15 Fig. 1 datasheet, Composition, 'What data does each instance consist of?'
is given below in “Data Preprocess-
- p. 17 Fig. 3 datasheet, section heading
Preprocessing/cleaning/labeling
- p. 15 Fig. 1, Composition
given below in “Data Preprocess-ing”
Judge's note: Read on pages 15 and 17. The datasheet's section headings are Motivation, Composition, Collection Process, Preprocessing/cleaning/labeling, Uses, Distribution and Maintenance; none is "Data Preprocessing". The other quoted pointer in the same column, “Collection Process”, matches its heading exactly. The star-to-polarity mapping is in the Preprocessing/cleaning/labeling section, so the reader can still find it. The same sentence has a subject-verb slip ("details ... is given").
Second check: another model agreed
Report a problem with this finding
Wording The example datasheet (PDF p. 15) says the star-to-polarity mapping "is given below in “Data Preprocessing”", but no section has that name; the section that holds it (PDF p. 17) is headed "Preprocessing/cleaning/labeling".
- p. 15 Composition, 'What data does each instance consist of?'
“Data Preprocess-ing”
Judge's note: Read the full page images for PDF pp. 15 and 17. The p. 15 answer under 'What data does each instance consist of?' puts “Data Preprocessing” in quotation marks as if it were a section title. The datasheet's section headings are Motivation, Composition, Collection Process, Preprocessing/cleaning/labeling, Uses, Distribution and Maintenance. The mapping from stars to polarity is in the Preprocessing/cleaning/labeling section on p. 17, so the target can be found; the quoted name simply does not match the heading. The same page also says 'See preprocessing below' without quotation marks. A minor naming inconsistency, and either form could be read as informal.
Second check: another model agreed
Report a problem with this finding
Typo Section 1.1: "For example, some the questions are appropriate for academic researchers" is missing "of" ("some of the questions").
- p. 2 Section 1.1, last paragraph
For example, some the questions are appropriate for academic
- p. 2 Section 1.1
For example, some the questions are appropriate
- p. 9 Section 3.6
How will the dataset will be distributed
Judge's note: Read on the page 2 image: the word is absent in the PDF, not lost in extraction. The duplicated "will" that one reviewer also mentions is handled in C003.
Second check: none (judge only)
Report a problem with this finding
Typo Duplicated word in the Distribution question, "How will the dataset will be distributed (e.g., tarball on website, API, GitHub)?", in Section 3.6 (page 9) and again in the example datasheet (page 17).
- p. 9 Section 3.6, second bullet
How will the dataset will be distributed
- p. 17 Fig. 3 datasheet, Distribution
How will the dataset will be distributed
- p. 9 Section 3.6 Distribution, second bullet
How will the dataset will be distributed (e.g., tarball on website,
Judge's note: Both occurrences read on the page images. The neighbouring questions ("When will the dataset be distributed?") show the intended form.
Second check: none (judge only)
Report a problem with this finding
Typo Reference [10] prints the title with a space before the colon: "QuAC : Question Answering in Context."
- p. 12 Reference [10]
QuAC : Question Answering in Context.
Judge's note: The space is visible on the page 12 image and no other title in the list has one. It may reproduce the styling of the cited paper's own title, which cannot be checked from the files, so either form could be legitimate.
Second check: none (judge only)
Report a problem with this finding
Typo Acknowledgments print the name as "Brian Mcfee"; the usual capitalisation of this surname is "McFee".
- p. 12 Acknowledgments
Brian Mcfee
Only the authors can say what is right here.
Judge's note: Rests on outside knowledge: that the person thanked spells the surname "McFee". The page 12 image shows "Brian Mcfee", directly followed by "Emily McReynolds" with the internal capital. Nothing in the files settles how this person writes the name.
Second check: none (judge only)
Report a problem with this finding
Typo Reference [24] gives the corporate author as "Doha Suppy Systems. 2017. Facial Recognition. [Online; accessed 14-March-2018]."; "Suppy" is probably a misspelling of "Supply", and the entry has no URL.
- p. 13 References [24]
Doha Suppy Systems. 2017. Facial Recognition.
- p. 13
[24] Doha Suppy Systems. 2017. Facial Recognition.
Only the authors can say what is right here.
Judge's note: Rests on outside knowledge: that the organisation is named Doha Supply Systems. The page 13 image prints "Suppy", so this is not an extraction artefact. The entry is marked as online but gives no URL, as do [11] and [21].
Second check: none (judge only)
Report a problem with this finding
Typo Reference [15] has doubled punctuation: "What Do Industry Practitioners Need?. In 2019 ACM CHI Conference".
- p. 13 Reference [15]
What Do Industry Practitioners Need?.
Judge's note: Read on the page 13 image. This is the bibliography style adding a full stop after a title that ends in a question mark; cosmetic. Reference [4] has a title ending in "?" with no added full stop, so the list is also inconsistent with itself.
Second check: none (judge only)
Report a problem with this finding
Typo Missing space in the datasheet question: "“Raw” data (e.g., unprocessed text or images)or features?"
- p. 15 Fig. 1 datasheet, right column, 'What data does each instance consist of?'
cessed text or images)or features?
- p. 15 Fig. 1, Composition, 'What data does each instance consist of?'
“Raw” data (e.g., unpro-cessed text or images)or features?
Judge's note: The enlarged crop of page 15 shows "images)or" with no space, while the other words on the line are normally spaced. The same question in Section 3.2 reads "images) or features" (text search).
Second check: none (judge only)
Report a problem with this finding
Typo Missing article in the example datasheet (PDF p. 18, export controls answer): "Unknown to authors of the datasheet."; every other occurrence reads "Unknown to the authors of the datasheet."
- p. 18 Fig. 4, Distribution, export controls answer
Unknown to authors of the datasheet.
- p. 16 Fig. 2, Collection Process
Unknown to the authors of the datasheet.
Judge's note: Read on the full page image of PDF p. 18: the answer to 'Do any export controls or other regulatory restrictions apply...' is printed without 'the'. On PDF p. 16 the same stock answer appears five times as 'Unknown to the authors of the datasheet.', and the main text quotes it in that form (section 3). Cosmetic inconsistency with an evident fix.
Second check: none (judge only)
Report a problem with this finding
Typo Duplicated word in the example datasheet's Distribution question (PDF p. 17): "How will the dataset will be distributed (e.g., tarball on website, API, GitHub)?"; should read "How will the dataset be distributed".
- p. 17 Fig. 3, Distribution section
How will the dataset will be distributed (e.g., tarball on website, API,
Judge's note: Read on the full page image of PDF p. 17, Distribution section, second question: the second 'will' is printed. The same duplication is in the question list in the main text (extract/main.txt, 'How will the dataset will be distributed'), so the datasheet copies it. This cluster concerns the p. 17 occurrence; no cluster for the main-text occurrence is among those given to me, so it is confirmed here in its own right.
Second check: none (judge only)