Talon's data plane moves large blocks between NVMe and the network. This page explains how that path is built, why it is split into two layers, and what happened when the design was measured — including the parts that were measured and rejected.
The central constraint is that large object bytes must never enter userspace.
A 256 MiB block read into a Vec<u8> and written back out costs two copies,
heap pressure, and buffer bloat on whatever runtime is scheduling the
connection. So the data plane splits into two layers with different jobs:
Layer 1 — protocol scheduling. Accepts connections, reads and writes 16-byte frame headers, decodes small control messages, runs timers and metrics. This is where connection management lives, and it handles a few dozen bytes per request.
Layer 2 — bulk movement. Moves the payload with Linux zero-copy syscalls:
sendfile(2) from the block file into the socket on a read, splice(2) from the
socket through a pipe into a staged file on ingest. The bytes go
NVMe → page cache → kernel → socket and are never in a Rust buffer.
The layers are deliberately decoupled. Layer 2 depends on no runtime types —
send_file_range and splice_to_file take &impl AsRawFd — which is what made
the runtime experiment below possible without touching the zero-copy path at all.
sendfile and splice are blocking syscalls. They can block on socket
backpressure, on disk, or on pipe readiness. Running them on the thread that
schedules connections means one slow client stalls every connection multiplexed
onto that thread.
So they run on a blocking helper pool. This matters more, not less, on a thread-per-core runtime: blocking one worker thread of a work-stealing pool is survivable, but blocking a single-threaded ring stalls everything it owns.
Layer 1's job — many concurrent connections, small messages, high syscall rate —
is what io_uring is designed for. It submits and completes operations in
batches through shared ring buffers rather than one syscall per operation, and a
thread-per-core design removes cross-thread scheduling from the path entirely.
Talon runs the io_uring Layer 1 by default, one ring per core, and falls back
to a portable Tokio implementation on hosts that cannot use io_uring:
talon-worker # io_uring, one ring per core (default)
talon-worker --data-plane-rings 4 # exactly four rings
# Pin the Tokio path explicitly (escape hatch)
TALON_WORKER_FORCE_TOKIO_DATA_PLANE=1 talon-workerThe fallback is a runtime capability probe, not a kernel-version check.
io_uring needs both a recent enough kernel and permission to use it — the
io_uring_disabled sysctl, a seccomp filter, or a container runtime's default
profile can each block the syscall on a kernel that nominally supports it. The
worker probes the real capability before binding and logs a warning if it falls
back, so a silent performance cliff is visible in the logs.
The rest of this page is how that second implementation was designed and what measuring it revealed.
monoio has no work-stealing scheduler. It scales by running N independent
single-threaded runtimes that share nothing. Each ring thread:
- pins itself to a core, so a connection's protocol scheduling never migrates and its state stays in that core's caches;
- builds its own ring with its own blocking pool for
sendfile; - binds the listen address with
SO_REUSEPORT.
That last point is the mechanism the whole design rests on. Every ring binds the same address, and the kernel distributes incoming connections across them by hashing the TCP 4-tuple. There is no shared accept queue, no lock, and no thundering herd — a connection is assigned to a ring at accept time and stays there for its lifetime.
Measured scaling on an 8-core host, one request in flight per connection:
| rings | throughput | scaling | per-core |
|---|---|---|---|
| 1 | 13,415 rps | 1.00× | 14,271 |
| 2 | 24,447 rps | 1.82× | 12,867 |
| 4 | 50,587 rps | 3.77× | 13,072 |
| 8 | 107,993 rps | 8.05× | 13,935 |
| 16 | 118,137 rps | 8.81× | 12,204 |
Per-core throughput is flat from 1 to 16 rings, which is the number that matters: it means there is no cross-ring contention to amortise. The fall-off at 16 is SMT — this host has 8 physical cores, so the extra rings land on hyperthread siblings. The useful setting is one ring per physical core.
With requests pipelined per connection the picture changes, and it is worth seeing both. Re-measured at depth 16, 64 connections, 64 KiB ranges:
| rings | throughput | GB/s | scaling | per-core | worker CPU | kernel share |
|---|---|---|---|---|---|---|
| 1 | 51,718 rps | 3.39 | 1.00× | 51,718 | 1.00 | 88% |
| 2 | 71,907 rps | 4.71 | 1.39× | 35,953 | 1.99 | 88% |
| 4 | 138,144 rps | 9.05 | 2.67× | 34,536 | 3.98 | 88% |
| 8 | 189,379 rps | 12.41 | 3.66× | 23,672 | 5.71 | 89% |
Two things are different, and neither is a defect.
Absolute throughput is much higher — 189K rps against 108K — because at depth 1 each connection pays a round trip per request and the link idles between them.
Per-core throughput is no longer flat, and the 8-ring point uses only 5.71 of
8 cores rather than saturating them. That is not cross-ring contention appearing;
it is the ceiling moving from Talon to the kernel. A single pipelined ring
already spends 88% of its CPU in sendfile and the TCP stack, so adding rings
adds contention for one shared network stack rather than for anything Talon owns.
The system stops being CPU-bound before it runs out of cores.
Read the first table for scheduling efficiency — it isolates Talon's own per-core cost — and the second for achievable throughput. And note that both are loopback; what binds the serve path below records what survives a real network, which is considerably less.
Three plausible-sounding design decisions did not survive contact with a benchmark. They are recorded here because the negative results shaped the implementation more than the positive ones.
Thread-per-core suggests an obvious companion: shard BlockIndex per ring so
each owns its slice with no locking, using Rc<RefCell<...>> instead of
Arc<Mutex<...>>. It loses on two independent counts.
Cross-ring forwarding costs 30–67% of throughput. Connections are
distributed by TCP 4-tuple; blocks are keyed by block_id. The two hashes are
unrelated, so only ~1/N of requests arrive at the ring that owns the block —
87.5% need forwarding at 8 shards, and forwarding is not free.
Per-shard eviction budgets waste capacity. With 256 MiB blocks and 64 GiB per worker there are only ~256 block slots, or 32 per shard at 8 rings. That is far too few for hash uniformity: measured skew reached 47–59%, so some shards evict while others sit below capacity. Simulated against realistic workloads, this cost up to 5.1 percentage points of hit rate when the working set sat near capacity — and each avoidable miss is a 256 MiB refetch.
Worth separating the two ideas: sharding the eviction policy is free — hit rate was identical to global LRU across every distribution tested, including an adversarial one concentrating all hot blocks in one shard. Sharding the budget is what costs.
So worker state stays shared behind an Arc. This is sound on a thread-per-core
runtime because !Send constrains futures crossing threads, not shared state
read from within a ring.
This was expected to be the hardest part of the design, and it turned out to be the opposite.
The Tokio path cannot sendfile directly onto its socket: a Tokio TcpStream is
non-blocking, so a blocking sendfile on it would spuriously EAGAIN. It
therefore round-trips the socket out of and back into the runtime on every
single transfer — into_std, set_nonblocking(false), move to a blocking
thread, move back, set_nonblocking(true), from_std.
A ring-owned fd needs none of that. It is passed straight to the blocking pool
and the ring resumes on the same stream afterwards. The io_uring implementation
of this path is smaller than the Tokio one.
A single ring caps at roughly 13k rps per core regardless of connection count. It is the simplest thing to build and it is a throughput ceiling on a machine meant to saturate NVMe and a NIC. Thread-per-core is not an optimisation here; it is the point.
Two independent measurements exist, at different concurrency levels, and they disagree in an instructive way.
Under concurrent load, measured in-repo by scripts/dataplane_loadtest.sh
against a real worker — real WorkerRuntime, real sendfile, real block store,
64 KiB ranges, 10 s per point after a 3 s warmup:
| conns | io_uring rps | tokio rps | io_uring p50 | tokio p50 | io_uring p99 | tokio p99 |
|---|---|---|---|---|---|---|
| 1 | 5,636 | 4,837 | 161 µs | 191 µs | 259 µs | 293 µs |
| 64 | 54,755 | 55,191 | 900 µs | 1,057 µs | 5,371 µs | 2,853 µs |
| 256 | 42,417 | 56,999 | 1,243 µs | 3,835 µs | 7,423 µs | 10,283 µs |
| 1024 | 51,668 | 41,060 | 1,175 µs | 19,905 µs | 4,781 µs | 72,911 µs |
At 1024 connections io_uring delivers 26% more throughput at 17× lower p50 and 15× lower p99, on comparable CPU and 19% less memory. Throughput, tail latency, and memory improve together — there is no trade being made.
The sweep is not monotonic, which is worth stating plainly. At 64 and 256 connections Tokio sometimes wins: work-stealing balances a moderate load well, while thread-per-core rings can sit idle. The advantage becomes decisive only at high connection counts. A cache fleet fronting a compute cluster lives at the right-hand end of that table, which is why the default follows it — but a deployment that will only ever see a hundred concurrent readers should measure rather than assume.
An earlier standalone harness, using a simplified protocol outside the tree, measured +35% throughput and −26% p99 at the same connection count. The in-repo figures supersede it: same direction, larger effect, and reproducible with one command.
At concurrency 1, measured by the in-repo benchmark
(cargo bench -p talon-worker --bench dataplane_benches), the two runtimes are
statistically indistinguishable. Across seven runs, medians landed between
218 and 295 µs for both, and the sign of the difference flipped run to run. The
spread within one implementation exceeded the gap between them.
That is not a contradiction — it is the expected shape of the result.
io_uring's advantage is amortising syscalls across many in-flight operations.
With one request in flight there is nothing to amortise, and the bulk bytes
bypass the ring via sendfile on both paths anyway, leaving a 16-byte header
exchange as the only work that differs.
The two results measure different things, and only one of them measures the thing Talon is built for.
A cache fleet serves many clients at once. That is the whole premise: workers
sit between a compute fleet and an object store, and the interesting regime is
hundreds or thousands of concurrent readers. The 1024-connection measurement is
the one taken in that regime, and it favours io_uring on every axis
simultaneously — throughput, per-core efficiency, tail latency, and memory.
The concurrency-1 result is not evidence against that. It is a measurement taken with an instrument that cannot detect the effect in question: with one request in flight there are no syscalls to batch, so the mechanism that produces the win is not exercised. A null result from an instrument blind to the effect says nothing about whether the effect exists.
So io_uring is the default, and the Divan benchmark keeps its job as a
regression floor — it catches a change that makes single-request latency
materially worse, which is a real thing to guard against, and it is honest about
not being the evidence for the default.
Both measurements now live in the tree and can be re-run:
cargo bench -p talon-worker --bench dataplane_benches # latency floor
scripts/dataplane_loadtest.sh # concurrent sweepThe tables above compare the two planes against each other. They do not say what the ceiling is, or which resource reaches it first — and that turns out to decide which optimisations are worth attempting at all.
Measured on a managed Kubernetes cluster (Standard_E64ads_v5 class node, worker
capped at 8 CPU by its cgroup), 64 KiB ranges, all cache hits, 32 connections at
pipeline depth 16:
| path | rps | GB/s | Gbps |
|---|---|---|---|
| loopback | 167,278 | 11.0 | 87.7 |
| cross-node | 44,662 | 2.93 | 23.4 |
Both rows are 32 connections, so they are directly comparable to each other. The loopback peak is higher still — 189,379 rps at 64 connections across 8 rings, per the table above — which makes the cross-node share 24% of what the worker can do locally.
Two findings, and they point the same way.
On loopback, 85% of the worker's CPU is kernel time — sendfile copying
bytes and the TCP stack — against 15% in Talon's own code, sampled from
utime/stime in /proc/<pid>/stat across the measured window. Halving all
of Talon's user-space work on the serve path would move total CPU by about 7%.
Only changes that remove copies or syscalls act on the large share.
Across the network, the NIC binds first. 23.4 Gbps against a 25 GbE link is the wire, saturated. On that cluster the worker is not the constraint at all.
The consequence for anyone optimising here: a change that moves the loopback number but not the cross-node number has not made the deployed system faster. Loopback measures a component ceiling with the network taken out of the way; it is a useful instrument, not a capacity claim.
Two results already follow from this. Pipelining — keeping requests in flight
per connection rather than paying a round trip each — doubles cross-node
throughput (22,315 → 44,592 rps), because it fills a link that was idling.
Multiplexing, serving one connection's requests concurrently with out-of-order
replies, was implemented and measured and did not pay: kernel time unchanged,
user time up 12.6% for scheduling overhead, and max latency 99× worse, because
two responses' bytes must never interleave and sendfile cannot be paused, so
the write half must be held exclusively — reintroducing at the writer the
serialisation it removes at the reader. It was not merged.
BENCHMARKS.md records the full tables, the method, and the conditions under which multiplexing could still be worth revisiting.
Even on a ring, the worker is not Tokio-free, and the reason is worth stating.
block_store runs its filesystem I/O on tokio::task::spawn_blocking so a large
read or a write-plus-fsync never stalls the reactor, and parts of the miss path
use Tokio timers and synchronisation primitives. Driven from a bare ring, those
calls panic. Ring threads therefore enter a Tokio runtime handle before running.
This is coexistence by design rather than a workaround: the ring owns protocol
scheduling and hands sendfile to its own blocking pool, while Tokio's blocking
pool absorbs filesystem work that belongs on neither. It does mean two blocking
pools share pinned cores, so the per-ring pool is kept deliberately small.
The same reasoning applies at the cluster level. The control plane, coordinator admin API, management UI, etcd and Kubernetes clients, the miss loader, and the metrics endpoints all stay on Tokio — they are not the hot path, and their ecosystems are Tokio-bound. The ring is used precisely where its advantage is measurable.
# The table above: both planes, sweeping connection counts against a real worker
scripts/dataplane_loadtest.sh
CONNS=1,512,2048 SECONDS_PER=30 scripts/dataplane_loadtest.sh
# Single-request latency floor, both planes, same block over loopback TCP
cargo bench -p talon-worker --bench dataplane_benches
# Drive an already-running worker, sampling its CPU and RSS
talon-loadgen --addr 127.0.0.1:7001 --backend az --conns 1024 --server-pid "$(pgrep -x talon-worker)"
# Confirm SO_REUSEPORT: N listeners on one port
talon-worker --data-plane-rings 2 --listen 127.0.0.1:7001
ss -ltn | grep 7001See BENCHMARKS.md for the harness, the committed baselines, and why the benchmark CI job is informational rather than gating.