feat(transport): add io_uring frame reader for the monoio data plane - #286
Merged
Conversation
First step of milvus-io#285. Adds transport::uring — the completion-based counterpart to limits::read_frame — alongside the Tokio reader rather than replacing it. Purely additive: no caller changes, so it lands and is tested in isolation. The two cannot share an implementation. Tokio's AsyncRead borrows a buffer for the duration of a read; io_uring is completion-based, so the kernel owns the buffer while the operation is in flight and monoio's AsyncReadRent moves it in and hands it back with the result. A borrowed-buffer signature cannot be made sound over a completion ring. What the wire format shares, the limits must share too. The DoS protections from milvus-io#111 are reimplemented verbatim and covered by tests that would fail if they regressed: the advertised length is checked against the per-message-type cap before the payload buffer is allocated (a control frame claiming a data-plane-sized payload is rejected without committing that memory), both reads are timeout-bounded so a peer stalling mid-frame is dropped rather than pinning a buffer, a clean EOF at a frame boundary is Eof rather than an error, and a truncated payload is an I/O error rather than a short frame that would desync the protocol. Ten tests run on a real io_uring runtime over loopback TCP, including consecutive frames on one connection to prove the reader leaves the stream at the next frame boundary. CI can execute these: the capability probe added in milvus-io#281 confirmed io_uring works on GitHub-hosted runners, which is why this needs no feature gate. monoio is added with the "sync" feature, which provides the spawn_blocking the zero-copy sendfile path will need to stay off the ring. Note it requires an explicitly attached thread pool and panics without one, unlike tokio's. Refs milvus-io#285, milvus-io#273 Co-Authored-By: Claude <noreply@anthropic.com>
beinan
added a commit
that referenced
this pull request
Jul 25, 2026
) * feat(worker): wire the io_uring data plane behind data_plane_rings Fourth step of #285. Connects the pieces from #286, #287, and #288 so the data_plane_rings knob actually selects the io_uring data plane. Unset, the worker serves on Tokio exactly as before; the default does not change. RingConnHandler adapts handle_conn to the ring runtime's RingHandler trait and carries the connection cap from #111. The cap is per ring rather than global, since each ring owns an independent semaphore, so main.rs divides MAX_DATA_PLANE_CONNECTIONS across the configured rings -- otherwise N rings would silently admit N times the intended budget. serve() now takes a tokio::runtime::Handle and enters it on every ring thread. This is required, not incidental: WorkerRuntime reaches Tokio internally, with block_store running filesystem I/O on tokio::task::spawn_blocking (#115) and parts of the miss path using Tokio timers and sync primitives, all of which panic on a bare ring. The split is deliberate -- the ring owns protocol scheduling and hands sendfile to its own blocking pool, while Tokio's absorbs filesystem work belonging to neither. Because that means two blocking pools coexist on pinned cores, the per-ring pool is kept small (4) and the reasoning recorded next to the constant. serve() blocks until its ring threads exit and must not occupy a Tokio worker thread while handing that runtime out, so main.rs drives it on spawn_blocking and parks on the join. A new test exercises the wiring end to end: a plain std::net client completes a real range request against the ring runtime, which also demonstrates that Tokio-side clients interoperate with the ring data plane unchanged -- the protocol is plain framed TCP, so porting fuse::worker_client is an optimization rather than a prerequisite. Verified on the built binary: with --data-plane-rings 2 the worker opens two listening sockets on one port via SO_REUSEPORT and logs both rings; without the flag it opens one and takes the Tokio path. Fifteen consecutive test runs are clean. Refs #285, #273 Co-Authored-By: Claude <noreply@anthropic.com> * fix(worker): resolve a broken intra-doc link RingHandler is not in scope in uring_conn, so the shorthand link failed rustdoc's broken-intra-doc-links lint under -D warnings. Use the full path. My local cargo doc run passed because it reused a cached build; CI's clean build caught it. Verified here with RUSTDOCFLAGS="-D warnings" and a forced rebuild, which is what the local check should have been. --------- Co-authored-by: Claude <noreply@anthropic.com>
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First step of #285. Purely additive — adds
transport::uringalongside the existing Tokio frame reader, with no caller changes, so it can land and be verified in isolation.Why a second reader instead of an abstraction
Tokio's
AsyncReadborrows a buffer for the duration of a read. io_uring is completion-based: the kernel owns the buffer while the operation is in flight, so monoio'sAsyncReadRentmoves it in and hands it back with the result:A borrowed-buffer signature cannot be made sound over a completion ring, so the reader is rewritten rather than made generic over both. The wire format is identical — same
FrameHeader, same payload bytes.The limits are the point
read_frame's DoS protections (#111) are what make this more than a thin wrapper, and they are reimplemented verbatim with tests that fail if they regress:rejects_oversized_control_payload_before_allocating— sends a header claimingMAX_CONTROL_PAYLOAD_LEN + 1and no payload; if the reader allocated first it would block instead of returning immediatelyrejects_ping_with_payloadtimes_out_a_stalled_payload— header then silence; asserts it returns well inside the deadlineEof, not an errorclean_eof_at_frame_boundarytruncated_payload_is_an_error— would otherwise desync the protocolreads_consecutive_framesPlus the happy paths: a normal frame, a zero-length payload (must not allocate or block on a second read), and the write helper returning its buffer for reuse.
Ten tests, all on a real io_uring runtime over loopback TCP. CI can execute them — the capability probe in #281 confirmed
PROBE_RESULT=SUPPORTEDonubuntu-latest, which is why this needs no feature gate.Dependency note
monoiois added with thesyncfeature, which provides thespawn_blockingthe zero-copysendfilepath will need to stay off the ring. Worth knowing for the next PR: unlike tokio's, it requires an explicitly attached thread pool and panics at runtime without one (execute blocking task without thread pool attached).Verification
cargo test --workspace --all-features --locked— 28 suites pass, including the 10 new onescargo clippy --workspace --all-targets --all-features -- -D warnings— cleancargo fmt --all --check— cleancargo doc --workspace --no-deps --all-features— cleanNext in #285
Thread-per-core accept loop behind a config flag, then porting
handle_conn/handle_put/handle_deleteto use this reader.Refs #285, #273
🤖 Generated with Claude Code