Skip to content

feat(transport): add io_uring frame reader for the monoio data plane - #286

Merged
beinan merged 1 commit into
milvus-io:mainfrom
beinan:feat/273-monoio-dataplane
Jul 25, 2026
Merged

feat(transport): add io_uring frame reader for the monoio data plane#286
beinan merged 1 commit into
milvus-io:mainfrom
beinan:feat/273-monoio-dataplane

Conversation

@beinan

@beinan beinan commented Jul 25, 2026

Copy link
Copy Markdown
Collaborator

First step of #285. Purely additive — adds transport::uring alongside the existing Tokio frame reader, with no caller changes, so it can land and be verified in isolation.

Why a second reader instead of an abstraction

Tokio's AsyncRead borrows a buffer for the duration of a read. io_uring is completion-based: the kernel owns the buffer while the operation is in flight, so monoio's AsyncReadRent moves it in and hands it back with the result:

// tokio
stream.read_exact(&mut buf).await?;

// monoio
let (res, buf) = stream.read_exact(buf).await;

A borrowed-buffer signature cannot be made sound over a completion ring, so the reader is rewritten rather than made generic over both. The wire format is identical — same FrameHeader, same payload bytes.

The limits are the point

read_frame's DoS protections (#111) are what make this more than a thin wrapper, and they are reimplemented verbatim with tests that fail if they regress:

protection test
per-type cap checked before allocating rejects_oversized_control_payload_before_allocating — sends a header claiming MAX_CONTROL_PAYLOAD_LEN + 1 and no payload; if the reader allocated first it would block instead of returning immediately
Ping carries no payload rejects_ping_with_payload
timeout on a stalled peer times_out_a_stalled_payload — header then silence; asserts it returns well inside the deadline
clean EOF is Eof, not an error clean_eof_at_frame_boundary
truncated payload is an error, not a short frame truncated_payload_is_an_error — would otherwise desync the protocol
stream left at the next frame boundary reads_consecutive_frames

Plus the happy paths: a normal frame, a zero-length payload (must not allocate or block on a second read), and the write helper returning its buffer for reuse.

Ten tests, all on a real io_uring runtime over loopback TCP. CI can execute them — the capability probe in #281 confirmed PROBE_RESULT=SUPPORTED on ubuntu-latest, which is why this needs no feature gate.

Dependency note

monoio is added with the sync feature, which provides the spawn_blocking the zero-copy sendfile path will need to stay off the ring. Worth knowing for the next PR: unlike tokio's, it requires an explicitly attached thread pool and panics at runtime without one (execute blocking task without thread pool attached).

Verification

  • cargo test --workspace --all-features --locked — 28 suites pass, including the 10 new ones
  • cargo clippy --workspace --all-targets --all-features -- -D warnings — clean
  • cargo fmt --all --check — clean
  • cargo doc --workspace --no-deps --all-features — clean

Next in #285

Thread-per-core accept loop behind a config flag, then porting handle_conn/handle_put/handle_delete to use this reader.

Refs #285, #273

🤖 Generated with Claude Code

First step of milvus-io#285. Adds transport::uring — the completion-based counterpart
to limits::read_frame — alongside the Tokio reader rather than replacing it.
Purely additive: no caller changes, so it lands and is tested in isolation.

The two cannot share an implementation. Tokio's AsyncRead borrows a buffer for
the duration of a read; io_uring is completion-based, so the kernel owns the
buffer while the operation is in flight and monoio's AsyncReadRent moves it in
and hands it back with the result. A borrowed-buffer signature cannot be made
sound over a completion ring.

What the wire format shares, the limits must share too. The DoS protections
from milvus-io#111 are reimplemented verbatim and covered by tests that would fail if
they regressed: the advertised length is checked against the per-message-type
cap before the payload buffer is allocated (a control frame claiming a
data-plane-sized payload is rejected without committing that memory), both
reads are timeout-bounded so a peer stalling mid-frame is dropped rather than
pinning a buffer, a clean EOF at a frame boundary is Eof rather than an error,
and a truncated payload is an I/O error rather than a short frame that would
desync the protocol.

Ten tests run on a real io_uring runtime over loopback TCP, including
consecutive frames on one connection to prove the reader leaves the stream at
the next frame boundary. CI can execute these: the capability probe added in
milvus-io#281 confirmed io_uring works on GitHub-hosted runners, which is why this
needs no feature gate.

monoio is added with the "sync" feature, which provides the spawn_blocking the
zero-copy sendfile path will need to stay off the ring. Note it requires an
explicitly attached thread pool and panics without one, unlike tokio's.

Refs milvus-io#285, milvus-io#273

Co-Authored-By: Claude <noreply@anthropic.com>
@beinan
beinan merged commit 31002e1 into milvus-io:main Jul 25, 2026
14 checks passed
beinan added a commit that referenced this pull request Jul 25, 2026
)

* feat(worker): wire the io_uring data plane behind data_plane_rings

Fourth step of #285. Connects the pieces from #286, #287, and #288 so the
data_plane_rings knob actually selects the io_uring data plane. Unset, the
worker serves on Tokio exactly as before; the default does not change.

RingConnHandler adapts handle_conn to the ring runtime's RingHandler trait and
carries the connection cap from #111. The cap is per ring rather than global,
since each ring owns an independent semaphore, so main.rs divides
MAX_DATA_PLANE_CONNECTIONS across the configured rings -- otherwise N rings
would silently admit N times the intended budget.

serve() now takes a tokio::runtime::Handle and enters it on every ring thread.
This is required, not incidental: WorkerRuntime reaches Tokio internally, with
block_store running filesystem I/O on tokio::task::spawn_blocking (#115) and
parts of the miss path using Tokio timers and sync primitives, all of which
panic on a bare ring. The split is deliberate -- the ring owns protocol
scheduling and hands sendfile to its own blocking pool, while Tokio's absorbs
filesystem work belonging to neither. Because that means two blocking pools
coexist on pinned cores, the per-ring pool is kept small (4) and the reasoning
recorded next to the constant.

serve() blocks until its ring threads exit and must not occupy a Tokio worker
thread while handing that runtime out, so main.rs drives it on spawn_blocking
and parks on the join.

A new test exercises the wiring end to end: a plain std::net client completes a
real range request against the ring runtime, which also demonstrates that
Tokio-side clients interoperate with the ring data plane unchanged -- the
protocol is plain framed TCP, so porting fuse::worker_client is an optimization
rather than a prerequisite.

Verified on the built binary: with --data-plane-rings 2 the worker opens two
listening sockets on one port via SO_REUSEPORT and logs both rings; without the
flag it opens one and takes the Tokio path. Fifteen consecutive test runs are
clean.

Refs #285, #273

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(worker): resolve a broken intra-doc link

RingHandler is not in scope in uring_conn, so the shorthand link failed
rustdoc's broken-intra-doc-links lint under -D warnings. Use the full
path.

My local cargo doc run passed because it reused a cached build; CI's clean
build caught it. Verified here with RUSTDOCFLAGS="-D warnings" and a forced
rebuild, which is what the local check should have been.

---------

Co-authored-by: Claude <noreply@anthropic.com>
@beinan
beinan deleted the feat/273-monoio-dataplane branch July 25, 2026 11:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant