The Demikernel Datapath OS Architecture for Microsecond-scale Datacenter Systems
ACM SIGOPS Symposium on Operating Systems Principles (SOSP), 2021 · paper · Computer science
Proposes Demikernel, a datapath operating system architecture for microsecond-scale datacenter systems using kernel-bypass devices. The authors build two prototypes and report that porting existing systems takes minimal effort and applications run across devices with nanosecond-scale overheads and no code changes.
Checked: PDF from irenezhang.net, fetched 4 Oct 2026. Page numbers are those of that PDF.
11 findings
Substance §7.3 says "Catnip imposes 125ns per UDP packet and 200ns per TCP packet", but Figure 5, whose caption gives the upper number as "total time spent in Demikernel for 4 I/O operations", shows 0.025 µs (UDP) and 0.053 µs (TCP) for Catnip, which is about 6 and 13 ns per I/O.
- p. 12 §7.3, para after Fig 6
Catnip imposes 125ns per UDP packet and 200ns per TCP packet
- p. 11 Figure 5 upper labels / caption
total time spent in Demikernel for 4 I/O operations
- p. 9 Section 6.3
application coroutine in 53ns
- p. 11 Figure 5
0.025
Only the authors can say what is right here.
Judge's note: Figure 5 on p11 shows Catmint 0.949, Catnip (UDP) 0.025 and Catnip (TCP) 0.053 µs. For Catmint, 0.949/4 ≈ 237 ns, consistent with the stated 250 ns. For Catnip the per-I/O values are about 20 times smaller than the text. §6.3 (p9) says Catnip processes a TCP packet in 53 ns, and the introduction says ≈50 ns per I/O; both fit the figure's 0.053 better than 200 ns. Either the caption's '4 I/O' definition or the text values need clarifying, and only the authors can say which.
Second check: another model agreed
Report a problem with this finding
Substance §7.6 says "Catnap has 69% lower latency than TxnStore's TCP stack", but the Figure 12 bars read about 527 µs (Linux TCP) and about 205 µs (Catnap) on average, about 61% lower. In addition, §7.1 says standard deviations are reported in Figure 12, which shows only average and p99 bars.
- p. 10 §7.1 para 2
except for § 7.6 where we report them in Figure 12
- p. 14 Figure 12
YCSB-t average and tail latencies for TxnStore
- p. 13 Section 7.6
Catnap has 69% lower latency than TxnStore’s TCP stack
- p. 13 Sec 7.6
Catnap has 69% lower latency than
Only the authors can say what is right here.
Judge's note: I measured the bars on the Figure 12 crop (200 µs ≈ 44 px). Linux TCP average ≈ 527 µs, Catnap ≈ 205 µs, Linux UDP ≈ 277 µs. The 27% claim against UDP matches the figure, but 69% does not: about 163 µs would be needed, and the Catnap bar reaches the 200 µs gridline. p99 gives about 59%, so that does not account for it either. Bar reading carries some uncertainty, so this is an author query. I also checked that Figure 12 has no error bars or SDs, despite the §7.1 statement "except for § 7.6 where we report them in Figure 12".
Second check: another model agreed
Report a problem with this finding
Substance §7.3 says "Catnip is 3.1µs faster than Shenango but 1.7µs slower than Caladan", but Figure 5 shows Catnip (TCP) 7.1+0.053, Catnip (UDP) 6.0+0.025, Shenango 7.5 and Caladan 6.6 µs.
- p. 11 §7.3 DEMILIN Latencies, para 2
Catnip is 3.1µs faster than Shenango but 1.7µs slower than Caladan.
- p. 11 Figure 5 bar labels
7.1
- p. 11 Figure 5 bar labels
7.5
- p. 11 Sec 7.3
Catnip is 3.1µs faster than Shenango but 1.7µs slower than Caladan
Only the authors can say what is right here.
Judge's note: I read the Figure 5 labels on the zoomed crop. Catnip(TCP) totals 7.153 µs, which is 0.35 µs faster than Shenango and 0.55 µs slower than Caladan. Catnip(UDP) totals 6.025 µs, which is 1.48 µs faster than Shenango and faster than Caladan. No pair of bars gives 3.1 or 1.7 µs. The text values may come from an earlier version of the figure; only the authors can say.
Second check: another model agreed
Report a problem with this finding
Substance §7.3 says "eRPC has 0.2µs lower latency than Catmint", but Figure 5 shows Catmint 5.3+0.949 ≈ 6.25 µs and eRPC 5.8 µs, a difference of about 0.45 µs.
- p. 11 §7.3 para 2
eRPC has 0.2µs lower latency than Catmint
- p. 11 Figure 5
0.949
- p. 11 Sec 7.3
eRPC has 0.2µs lower
- p. 11 Figure 5
5.3 ... 5.8
Only the authors can say what is right here.
Judge's note: I checked the Figure 5 labels on the crop (Catmint 5.3 with Demikernel portion 0.949; eRPC 5.8). The total RTT difference is 0.449 µs, and using the lower segment alone would make Catmint faster. Neither reading gives 0.2 µs.
Second check: another model agreed
Report a problem with this finding
Substance §7.4 says Catnip reduces latency "by 11µs on average and 13.7µs at the p99", but Figure 10 gives Linux p99 27.6 µs and Catnip p99 14.9 µs, a 12.7 µs difference.
- p. 13 §7.4
by 11µs on average and 13.7µs at the p99
- p. 13 Figure 10
27.6
- p. 13 Sec 7.4
11µs on average and 13.7µs at the p99
- p. 13 Figure 10
27.6 ... 14.9
Judge's note: Bar labels in Figure 10 (p13 image): Linux 24.9/27.6, io_uring 24.4/25.8, Catnip 13.9/14.9. The average difference (11.0) matches the text, so the baseline is Linux. Against io_uring the p99 difference is 10.9, which does not give 13.7 either.
Second check: another model agreed
Report a problem with this finding
Substance §7.5 says "Catnap has 75-80% lower peak throughput", but in Figure 11 (In-memory) Catnap reaches about 105K (GET) and 120K (SET) ops/s against Linux at about 170K and 165K, roughly 27–38% lower.
- p. 13 §7.5 para 1
Catnap has 75-80% lower peak throughput
- p. 13 Figure 11 in-memory bars
In-memory
- p. 13 Figure 11, In-memory panel
Redis benchmark throughput in-memory and on-disk
Only the authors can say what is right here.
Judge's note: Bar heights read from the p13 image against the 50K gridlines. A 75–80% reduction would put Catnap at about 35–42K ops/s, far below what is plotted. 'Achieves 75-80% of' may have been intended, but GET is about 62%, so only the authors can say.
Second check: another model agreed
Report a problem with this finding
Reference or label Reference [15] (Persephone) gives the venue as "Proceedings of the 26th Symposium on Operating Systems Principles (2021)". The paper's own numbering makes SOSP 2021 the 28th symposium and the 26th the 2017 meeting.
- p. 15 Reference [15]
Proceedings of the 26th Symposium on Operating Systems Principles (2021)
- p. 1 ACM Reference Format
ACM SIGOPS 28th Symposium on Operating Systems Principles
- p. 1 ACM Reference Format
ACM SIGOPS 28th Symposium on Operating Systems Principles (SOSP ’21)
Judge's note: Page 15, ref [15], reads '26th ... (2021)'. The paper itself gives the other pairings. The ACM Reference Format on page 1 names this paper's venue, SOSP '21, as the 'ACM SIGOPS 28th Symposium on Operating Systems Principles'. Several other references (at text lines 1883, 1912 and 2018) give '26th ... (2017)', and two (at lines 1832 and 1938) give '27th ACM ... (2019)'. So the paper contradicts itself, and the edition number in ref [15] should read 28th. I did not need any outside facts.
Second check: another model agreed
Report a problem with this finding
Reference or label Figure 1 caption cites "Caladan [23] and eRPC [8]'s architectures", but [8] is Chen et al., ScaleRPC; eRPC is [34] ("eRPC [34] and ScaleRPC [8]" in §8).
- p. 2 Figure 1 caption
Caladan [23] and eRPC [8]’s
- p. 2 Figure 1 caption
Arrakis [73], Caladan [23] and eRPC [8]’s
- p. 15 Reference [8]
Scalable RDMA RPC on reliable connection with efficient resource sharing
- p. 2 Figure 1 caption
eRPC [8]’s
Judge's note: Reference [8] on p15 reads 'Chen, Y., Lu, Y., and Shu, J. Scalable RDMA RPC on reliable connection with efficient resource sharing' (EuroSys 2019). The related-work text on p14 cites 'eRPC [34] and ScaleRPC [8]'. The citation in the Figure 1 caption is therefore wrong.
Second check: another model agreed
Report a problem with this finding
Reference or label The Figure 8 caption cites "replicating NetPIPE [3]", but [3] is Belay et al., "IX: A protected dataplane operating system"; the text cites NetPIPE as [68].
- p. 12 Figure 8 caption
replicating NetPIPE [3]
- p. 12 §7.3 text
NetPIPE [68]
- p. 15 Reference [3]
IX: A protected dataplane op- erating system
- p. 12 Sec 7.3
we use NetPIPE [68]
Judge's note: Reference list on p15: [3] is the IX paper and [68] is 'Network Protocol Independent Performance Evaluator'. The body text on p12 cites NetPIPE [68].
Second check: another model agreed
Report a problem with this finding
Reference or label The §7.5 persistence paragraph says "Figure 11 (left) gives the results", but the persistent-log results are in the right half of Figure 11 (labelled "Persistent Log").
- p. 13 §7.5 para 2
Figure 11 (left) gives the results
- p. 13 Section 7.5, second paragraph
Figure 11 (left) gives the results.
Judge's note: Figure 11 (p13) has an 'In-memory' panel on the left and a 'Persistent Log' panel on the right. The previous paragraph already uses 'Figure 11 (left)' for the in-memory results.
Second check: another model agreed
Report a problem with this finding
Wording Introduction states "I/O processing latencies of ≈50ns per I/O", while §7.3 reports "Catmint imposes 250ns of latency overhead per I/O, while Catnip imposes 125ns per UDP packet and 200ns per TCP packet".
- p. 2 §1 last para
I/O processing latencies of ≈50ns per I/O
- p. 12 §7.3
Catmint imposes 250ns of latency overhead per I/O
- p. 12 Section 7.3
On average, Catmint imposes 250ns of latency overhead per I/O
Only the authors can say what is right here.
Judge's note: Checked p2 (intro), p11 (Figure 5 and its caption) and p12 (§7.3 text). The only value near 50ns is in §6.3: Catnip dispatches one incoming TCP packet in 53ns. Figure 5 labels Catnip (TCP) 0.053µs, but its caption says the upper number is the total Demikernel time for 4 I/O operations. §7.3 gives 125–250ns per I/O. The headline per-I/O figure therefore does not match the per-I/O averages in the evaluation, and only the authors can say which quantity ≈50ns refers to. Figure 5's labels (Catmint 0.949, Catnip UDP 0.025, Catnip TCP 0.053 µs) also do not match the §7.3 per-packet numbers in an obvious way.
Second check: another model agreed
Report a problem with this finding
27 minor (typos and the like; most checked by the judge only)
Reference or label "perftest [84]" is cited to reference [84], "RDMA CM connection and RDMA ping-pong test. http://manpages.ubuntu.com/manpages/bionic/man1/rping.1.html", which is the rping man page.
- p. 10 §7.1 Comparison Systems
perftest [84]
- p. 17 Reference [84]
RDMA CM connection and RDMA ping-pong test
Only the authors can say what is right here.
Judge's note: Rests on outside knowledge: rping (librdmacm) is a separate tool from the perftest suite. The reference list (page 17) confirms that [84] is the rping.1 man page, while §7.1 (page 10) describes perftest as included with the RDMA SDKs and measuring send/recv latency.
Second check: another model agreed
Report a problem with this finding
Wording Introduction's "a 17-26% peak throughput overhead, compared to directly using kernel-bypass APIs" refers to single-client NetPIPE goodput at 256kB (§7.3 "17% and 26% performance penalty ... for 256kB messages"), not to the paper's 'Peak Throughput' experiment (70% of Caladan, 46% of eRPC).
- p. 2 §1 last para
a 17-26% peak throughput overhead
- p. 12 §7.3 Single Client Throughput Overheads
imposing a 17% and 26% performance penalty
- p. 13 §7.3 Peak Throughput
Catnip (UDP) achieves 70% of the peak throughput of Caladan
- p. 2 Section 1, last paragraph
and a 17-26% peak throughput overhead, compared to directly using kernel-bypass APIs
Judge's note: On p12 the 17% and 26% figures recompute from the Figure 8 values (Catnip 33.3 and 29.7 Gbps vs testpmd 40.3; Catmint 31.5 vs perftest 37.7). The intro's phrase 'compared to directly using kernel-bypass APIs' does match that NetPIPE comparison. The word 'peak' can, however, be read as pointing to the separate p13 'Peak Throughput' experiment, which compares against Caladan and eRPC and gives larger gaps. This is a wording ambiguity; the numbers themselves are consistent.
Second check: another model agreed
Report a problem with this finding
Wording §2.2: "kernel-bypass requires the I/O device's IOMMU ... which requires coordination with the CPU's IOMMU and TLB". The pairing with TLB suggests the CPU's MMU was meant.
- p. 2 §2.2 para 2
which requires coordination with the CPU’s IOMMU and TLB
Only the authors can say what is right here.
Judge's note: Read on the p2 image. The sentence first attributes the IOMMU to the I/O device and then calls for coordination with the 'CPU's IOMMU and TLB', which as written is coordination of the IOMMU with itself. CPU-side translation is the MMU/TLB. This is probably a slip, but only the authors can confirm what was meant.
Second check: another model agreed
Report a problem with this finding
Wording §7.3 says "Catpaw reduces latency by 27×", but the Figure 6a totals (WSL 93.6+11.7 µs, Catnap 71.7+21.2 µs, Catpaw 1.8+1.3 µs) give about 30–34×.
- p. 12 §7.3 DEMIWIN
Catpaw reduces latency by 27×
- p. 12 Figure 6a
93.6
- p. 12 Section 7.3, DEMIWIN
while Catpaw reduces latency by 27×
- p. 12 Sec 7.3 / Fig 6a
Catpaw reduces latency
Only the authors can say what is right here.
Judge's note: Values read from Figure 6a on the p12 image. No pair of totals or components I tried gives 27: totals give 34× (WSL) and 30× (Catnap), and lower components only give 52× and 40×. The authors should say which baseline the 27× uses.
Second check: another model agreed
Report a problem with this finding
Wording §7.3 says "Catmint trades off latency on the critical path for better throughput", while the Figure 9 caption says "Catmint and Catnip (UDP) are optimized for latency but not throughput".
- p. 12 §7.3 para after Fig 6
Catmint trades off latency on the critical path for better throughput
- p. 13 Figure 9 caption
Catmint and Catnip (UDP) are optimized for latency but not throughput
Judge's note: Both statements are on the page images (p12 text, p13 Figure 9 caption). The first may refer to the per-I/O overhead coming from background coroutines, but as written the two describe Catmint's trade-off in opposite directions. A clarifying sentence would resolve this.
Second check: another model agreed
Report a problem with this finding
Wording §7.6 says "Catnap has 69% lower latency than TxnStore’s TCP stack", but Figure 12 shows Linux (TCP) average ≈520 µs and Catnap ≈205 µs (about 60% lower; p99 ≈665 vs ≈275, about 59%).
- p. 13 §7.6
Catnap has 69% lower latency than TxnStore’s TCP stack
- p. 14 Figure 12
Linux (TCP)
Only the authors can say what is right here.
Judge's note: Figure 12 bars are unlabelled, so readings are approximate (±10 µs), but 69% would need Catnap at about 160 µs. The companion 27% figure for UDP (≈280 vs ≈205) is consistent with the plot.
Second check: another model agreed
Report a problem with this finding
Wording Section 7.6 describes the TxnStore workload key distribution as "a uniform Zipf distribution". This is ambiguous: it could mean a uniform distribution or a Zipf distribution, and the Zipf skew parameter is not given.
- p. 13
We use a uniform Zipf distribution
Only the authors can say what is right here.
Judge's note: I checked the crop of page 13, section 7.6, and it reads "We use a uniform Zipf distribution, 64 B keys, and 700 B values." No other passage in the text gives a Zipf parameter or explains the phrase. The phrase may mean Zipf with zero skew, which is equivalent to uniform. Only the authors can say which distribution was used, so I have filed this as a wording query.
Second check: another model agreed
Report a problem with this finding
Typo Stray word: "heterogenous kernel-bypass devices with while minimizing OS overheads".
- p. 2 §1 first para
bypass devices with while minimizing OS overheads
- p. 2 Section 1, first column end
kernel-bypass devices with while minimizing OS overheads
Judge's note: Visible on the p2 page image, end of the first column of §1.
Second check: none (judge only)
Report a problem with this finding
Typo Number agreement: "because different device offload different OS features" should read 'different devices offload'.
- p. 2 §2.1
because different device offload different OS
Judge's note: Visible on the p2 image, §2.1.
Second check: none (judge only)
Report a problem with this finding
Typo Missing word: "by treating the kernel-bypass hardware the datapath kernel" should read 'treating the kernel-bypass hardware as the datapath kernel'.
- p. 3 §3.3
bypass hardware the datapath kernel and accommodating
Judge's note: Visible on the p3 image, §3.3.
Second check: none (judge only)
Report a problem with this finding
Typo Enumeration lacks '(1)': "zero-copy memory coordination: a portable API with clear semantics for I/O memory buffer ownership, (2) a zero-copy, DMA-capable heap and (3) use-after-free".
- p. 4 §3.3 DMA-capable heap para
zero-copy memory coordination: a portable API with clear
- p. 4 Section 3.3, 'A DMA-capable heap' paragraph
a portable API with clear semantics for I/O memory buffer ownership, (2) a zero-copy,
- p. 4 Sec 3.3
a portable API with clear semantics for I/O memory buffer ownership, (2) a zero-copy
- p. 4 Sec 3.3
a portable API with clear semantics for I/O memory buffer ownership, (2)
Judge's note: Visible on the p4 image, §3.3: no '(1)' before the first item.
Second check: none (judge only)
Report a problem with this finding
Typo Wrong preposition: "there is no affordable way to Demikernel datapath OSes to offer write-protection" should read 'way for Demikernel datapath OSes'.
- p. 4 §3.3 UAF para
buffers because there is no affordable way to Demikernel
Judge's note: Visible on the p4 image, §3.3 UAF paragraph.
Second check: none (judge only)
Report a problem with this finding
Typo Ungrammatical: "Use-after-free protection guarantees that I/O buffers are not be freed until both the application and datapath OS free them".
- p. 5 §4.2 Memory
protection guarantees that I/O buffers are not be freed until
- p. 5 Section 4.2 Memory
I/O buffers are not be freed until
Judge's note: Visible on the p5 image, §4.2: 'are not be freed'.
Second check: none (judge only)
Report a problem with this finding
Typo Paragraph in §4.1 begins with a lowercase letter: "kernel-bypass applications require zero-copy I/O to make" (should be "Kernel-bypass").
- p. 5 Section 4.1, third paragraph
kernel-bypass applications require zero-copy I/O to make
Judge's note: I checked the crop and page 5. The paragraph does start with a lowercase "kernel-bypass".
Second check: none (judge only)
Report a problem with this finding
Typo A word is missing in §4.1: "To better this task, Demikernel offers ..." (probably "To better support this task").
- p. 5 Section 4.1
To better this task, Demik-
Judge's note: The crop shows the phrase as quoted. A verb is clearly missing.
Second check: none (judge only)
Report a problem with this finding
Typo Misspelling in the Figure 2 listing comment: "// I/O processing, notifcation and memory calls" (should be "notification").
- p. 6 Figure 2 right listing, line 1
notifcation
Judge's note: The page 6 image shows the typeset comment "notifcation" with the 'i' missing.
Second check: none (judge only)
Report a problem with this finding
Typo Ungrammatical phrase in §5.1: "support for co-routines, which we are being actively developing".
- p. 6 §5.1
which we are being actively developing
Judge's note: The crop shows the text as quoted. It should read "which are being actively developed" or "which we are actively developing".
Second check: none (judge only)
Report a problem with this finding
Typo The API call is named "int err = lseek(int qd, ...);" in Figure 2 but "seek and truncate move the read cursor" in §6.4 (code font).
- p. 9 Sec 6.4
seek and truncate move
- p. 6 Figure 2
int err = lseek(int qd, ...);
Judge's note: Figure 2 line 11 shows lseek, and §6.4 shows `seek` in typewriter font. This is a small naming inconsistency; it may be informal shorthand.
Second check: none (judge only)
Report a problem with this finding
Typo Misspelling in §5.3: "registers an entire superblock on the first invokation" (should be "invocation").
- p. 7 §5.3
perblock on the first invokation
- p. 7 Section 5.3
on the first invokation
Judge's note: Visible in the crop of page 7, right column.
Second check: none (judge only)
Report a problem with this finding
Typo Misspelling in §5.5: "which Catnip×Cattree shares betwen the networking and storage stacks" (should be "between").
- p. 8 §5.5
betwen the networking and storage stacks
Judge's note: Visible in the crop of page 8.
Second check: none (judge only)
Report a problem with this finding
Typo The interface is spelled two ways: "Catpaw is built on NDSPI v2 [63]" (§6) but "design for Catpaw atop NSDPI" (§6.2).
- p. 8 §6
Catpaw is built on NDSPI v2 [63]
- p. 9 §6.2
design for Catpaw atop NSDPI
- p. 9 Sec 6.2
atop NSDPI
- p. 9 Section 6.2
similar design for Catpaw atop NSDPI
Judge's note: Both crops confirm the two spellings. NSDPI is a transposition of the NDSPI used in §6.
Second check: none (judge only)
Report a problem with this finding
Typo Miscapitalisation in §6: "Table 2 lists the library Oses that make up each datapath OS" (should be "OSes" as elsewhere).
- p. 8 Section 6, first paragraph
Oses that make up each datapath OS
Judge's note: The page 8 crop shows "Oses" at the start of a line. Everywhere else the paper writes "OSes".
Second check: none (judge only)
Report a problem with this finding
Typo Missing words: "Catnip is able control all inputs to the TCP stack" (§6.3; should be "able to control") and "end-to-end latency is not concern for these servers" (§7.2; should be "not a concern").
- p. 9 §6.3
Catnip is able control all inputs
- p. 11 §7.2 TURN
is not concern for these servers
- p. 9 Section 6.3
Catnip is able control all inputs to the TCP stack
Judge's note: Both phrases appear as quoted in the crops of pages 9 and 11.
Second check: none (judge only)
Report a problem with this finding
Typo Misspelling in the §7.2 Redis paragraph: "This design is ineffcient" should read "inefficient".
- p. 11 §7.2 Redis
ineffcient
- p. 11 Section 7.2 Redis
ineffcient: it requires more than one system call
- p. 11 Redis paragraph
This design is ineffcient
Judge's note: The page image shows 'ineffcient', so this is in the typeset text and not a lost ligature.
Second check: none (judge only)
Report a problem with this finding
Typo Missing article in the TURN UDP Relay paragraph: "end-to-end latency is not concern for these servers" should read "is not a concern".
- p. 11 TURN UDP Relay
is not concern for these servers
Judge's note: Seen on the p11 page image, top of the left column.
Second check: none (judge only)
Report a problem with this finding
Typo Stray character in §7.4: "We use a non0-kernel-bypass Linux-based traffic generator" should read "non-kernel-bypass".
- p. 13 §7.4
use a non0-kernel-bypass Linux-based traffic generator
- p. 13 Section 7.4
non0-kernel-bypass Linux-based traffic generator
- p. 13 Sec 7.4
non0-kernel-bypass
- p. 13
We use a non0-kernel-bypass Linux-based traffic generator
Judge's note: The '0' is visible in the p13 page image.
Second check: none (judge only)
Report a problem with this finding
Typo The Figure 9 x-axis label reads "Throughput (Gpbs)" instead of "Gbps".
- p. 13 Figure 9 x-axis
Throughput (Gpbs)
Judge's note: The axis label on the p13 image reads 'Throughput (Gpbs)'; Figure 8 and the text use Gbps.
Second check: none (judge only)