Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing
2012 · paper · Computer science
Introduces resilient distributed datasets, a fault-tolerant distributed memory abstraction for in-memory computation on large clusters, based on coarse-grained transformations. Implemented in Spark, it targets iterative algorithms and interactive data mining, and can express a wide class of computations.
Checked: PDF from usenix.org, fetched 4 Oct 2026. Page numbers are those of that PDF.
0 findings
None that passed a second check.
14 minor (typos and the like; most checked by the judge only)
Wording Intro, §6 bullet and Conclusion say Spark is "up to 20× faster than Hadoop" for iterative applications, but §6.1 reports "Spark 25.3× and 20.7× faster than Hadoop and HadoopBinMem" (Fig. 7/8a: 76 s vs 3 s).
- p. 2 Introduction
Spark is up to 20× faster than Hadoop for iterative applications
- p. 9 Section 6 summary bullet
Spark outperforms Hadoop by up to 20× in itera-
- p. 9 Section 6.1, Subsequent Iterations
Spark 25.3× and 20.7× faster than Hadoop and HadoopBinMem respectively
- p. 14 Conclusion
outperforms Hadoop by up to 20× in iterative applications
Only the authors can say what is right here.
Judge's note: Checked p2, p9 and p14. Fig. 7 and Fig. 8(a) at 100 machines show Hadoop 76 s, HadoopBinMem 62 s and Spark 3 s for later iterations, and the §6.1 text gives 25.3× over Hadoop. The 20× figure matches HadoopBinMem (and the 'Understanding the Speedup' paragraph applies 20× to HadoopBinMem). The headline understates rather than overstates the result, so this is minor. Only the authors can say whether 'Hadoop' in these sentences means the HadoopBinMem baseline.
Second check: another model agreed
Report a problem with this finding
Wording The persist flag for replication/checkpointing is called "persist with a RELIABLE flag" in §3.2.2 but "a REPLICATE flag to persist" in §5.4.
- p. 6 Section 3.2.2
can call persist with a RELIABLE flag to do this
- p. 8 Section 5.4
REPLICATE flag to persist
- p. 6 Section 3.2.2
persist with a RELIABLE flag
- p. 8 Section 5.4
a REPLICATE flag to persist
Only the authors can say what is right here.
Judge's note: Both names appear in the images (p6 left column; p8 §5.4). Both passages describe a way to replicate or checkpoint RDDs such as ranks, and the paper never says there are two flags. Only the authors can say which name is right.
Second check: another model agreed
Report a problem with this finding
Wording §6.1 says converting pre-parsed binary data into Java objects "took 3 seconds". The difference between the Figure 9 binary-input bars (in-mem local file 6.9 s, Spark RDD 2.9 s) is 4.0 s.
- p. 10 Section 6.1, discussion of Figure 9
the pre-parsed binary data into Java objects took 3 sec-
- p. 10 Figure 9, Binary Input bars
6.9
Only the authors can say what is right here.
Judge's note: The other overheads derived in the text match the Figure 9 bars within rounding: HDFS 15.4−13.1 and 8.4−6.9 ≈ 2 s; parsing 15.4−8.4 = 7 s. The obvious pair for deserialization gives 4.0 s. The 3 s may come from a separate measurement, but the paper does not say so. Query to the authors.
Second check: another model agreed
Report a problem with this finding
Wording Section 6.3 says the failure "increased the iteration time to 80s", but the iteration-6 failure bar in Figure 11 is labelled 81.
- p. 10 Section 6.3
increased the it- eration time to 80s
- p. 10 Figure 11, iteration 6 bar label
81
Judge's note: Checked the page 10 image. The Figure 11 bar label for iteration 6 (failure) reads 81, and the text reads "80s" without "about". The prose is probably approximate, since it also says "about 58 seconds" for bars labelled 56–59. Still, 81 s would not normally round to 80 s, so the text and figure give different values for the same measurement. It is a minor consistency point.
Second check: another model agreed
Report a problem with this finding
Typo §6.1 Subsequent Iterations: "For logistic regression, Spark 25.3× and 20.7× faster than Hadoop and HadoopBinMem" has no verb ('was' is missing). The same section also reads "Spark still achieved speedup of 1.9×".
- p. 9 Section 6.1, Subsequent Iterations
Spark 25.3× and 20.7× faster than Hadoop and HadoopBinMem
- p. 2 Introduction, paragraph summarising the evaluation
We show that Spark is up to 20× faster than Hadoop for iterative applications
- p. 9 §6 summary bullet
Spark outperforms Hadoop by up to 20× in itera-tive machine learning and graph applications
- p. 9 §6.1 Subsequent Iterations; Figure 8(a)
Spark 25.3× and 20.7× faster than Hadoop and HadoopBinMem respectively on 100 ma-chines
Judge's note: Seen on the p9 image. The 'up to 20×' versus 25.3× part of this cluster is judged under C001. The 'plan to use to' quote is C010.
Second check: none (judge only)
Report a problem with this finding
Typo Figure 1 arrow label reads "filter(_.contains(“HDFS”)))", which has one closing parenthesis too many.
- p. 3 Figure 1, second arrow label
filter(_.contains(“HDFS”)))
Judge's note: The crop shows three closing parentheses against two opening ones. The other labels in the figure are balanced.
Second check: none (judge only)
Report a problem with this finding
Typo §4 (p6): "while join leads to to wide dependencies" repeats the word 'to'.
- p. 6 Section 4, para on narrow/wide dependencies
while join leads to to wide
- p. 9 Subsequent Iterations
Spark 25.3× and 20.7× faster than
- p. 11 Section 6.6
Spark’ ability
- p. 6 Section 4
leads to to wide
Judge's note: On the p6 image both 'to's sit on the same line. The other quotes bundled into this cluster are separate defects: 'build build' is C009, 'plan to use to' is C010 and the missing 'was' is C012. 'Spark’ ability' (p11, §6.6) also shows as a typo in the text layer.
Second check: code check
Report a problem with this finding
Typo §3.2.2: "to the machine that its link lists is on" has a subject–verb number disagreement.
- p. 6 Section 3.2.2, after partitionBy code
the machine that its link lists is on
- p. 6 Section 3.2.2
its link lists is on
Judge's note: Seen on the p6 image, left column, below the partitionBy code. The preceding paragraph says 'its link list'.
Second check: none (judge only)
Report a problem with this finding
Typo Figure 5 caption: "we build build stages at wide dependencies" repeats the word 'build'.
- p. 7 Figure 5 caption
we build build stages at wide dependencies
- p. 7 Figure 5 caption
we build build stages
- p. 7
To run an action on RDD G, we build build stages at wide depe
Judge's note: The crop of the p7 caption shows 'build build' on one line.
Second check: code check
Report a problem with this finding
Typo §5.2: "We also plan to use to run higher-level query languages" is missing the object (presumably 'use it to run').
- p. 8 Section 5.2, last paragraph
We also plan to use to run higher-level query languages
- p. 8 Section 5.2
We also plan to use to
Judge's note: Seen on the p8 image at the end of §5.2.
Second check: none (judge only)
Report a problem with this finding
Typo §6.1 First Iterations: "an extra MapReduce job to convert the data to binary, it and had to write this data" has a misplaced 'it'.
- p. 9 Section 6.1, First Iterations
to binary, it and had to write this data
- p. 10 Section 6.1
incurred least 25s of overhead
- p. 9 First Iterations
binary, it and had to write
- p. 9 First Iterations paragraph
to binary, it and had to write
Judge's note: Seen on the p9 image, right column. The 'incurred least 25s' quote also listed here is covered by C014.
Second check: none (judge only)
Report a problem with this finding
Typo §6.1 Understanding the Speedup: "saw that these at incurred least 25s of overhead" has its words transposed (should be 'incurred at least').
- p. 10 Section 6.1, Understanding the Speedup, factor (1)
saw that these at incurred least 25s of overhead
- p. 10 Section 6.1, factor (1)
saw that these at
- p. 10 Understanding the Speedup
at incurred least 25s of overhead
- p. 10 6.1 Understanding the Speedup
these at incurred least 25s of overhead
Judge's note: The p10 image, left column, shows the word order as printed. This is not a line-break artefact.
Second check: none (judge only)
Report a problem with this finding
Typo The Figure 11 y-axis label is misspelled: "Iteratrion time (s)" should read "Iteration time (s)".
- p. 10 Figure 11 y-axis
Iteratrion time (s)
- p. 10
Iteratrion time (s)!
Judge's note: The page 10 image clearly shows "Iteratrion time (s)" on the Figure 11 y-axis. Figures 9 and 10 spell it "Iteration time (s)".
Second check: none (judge only)
Report a problem with this finding
Typo Section 6.6 has a malformed possessive: "To demonstrate Spark’ ability to interactively query big datasets" should read "Spark’s ability".
- p. 11 Section 6.6
To demonstrate Spark’ ability
- p. 11
To demonstrate Spark’ ability to interactively query big
Judge's note: The page 11 image shows "Spark’ ability" with no s after the apostrophe.
Second check: none (judge only)