Skip to content

fix(fuse): fail closed on incomplete or invalid namespace listings - #375

Merged
beinan merged 1 commit into
milvus-io:mainfrom
charleshuang119:huangcharles/fix-fuse-listing-integrity
Jul 28, 2026
Merged

fix(fuse): fail closed on incomplete or invalid namespace listings#375
beinan merged 1 commit into
milvus-io:mainfrom
charleshuang119:huangcharles/fix-fuse-listing-integrity

Conversation

@charleshuang119

Copy link
Copy Markdown
Contributor

Summary

Follow-up to #369. Requiring a namespace prefix fixed the empty-root startup request, but the mount could still report success with a partial, out-of-scope, or POSIX-unrepresentable listing.

This change makes startup listing fail closed end to end:

  • Worker listing now returns an actionable error instead of partial success when it reaches 10,000 objects, 20 backend pages, or the 1 MiB control payload limit.
  • Worker runtime records its configured object-store backend and rejects mismatched list, stat, read, write, staged-write, and delete requests before they touch the backend or cache.
  • Coordinator retries explicit worker rejections and transport failures under one shared deadline. Listings receive a 25-second total budget, while each attempt preserves a bounded retry window for later workers. Aggregated diagnostics are UTF-8 safe and size bounded.
  • FUSE validates the complete response before mutating the namespace tree. It rejects entries outside the requested raw prefix, non-canonical paths, NUL or over-255-byte components, duplicates, file-directory collisions, file-as-ancestor conflicts, and non-empty trailing-slash objects.
  • FUSE configuration rejects ambiguous prefixes while preserving a meaningful trailing slash such as s3/bucket/dir/, which prevents raw prefix dir from also matching dir2.
  • The empty-visible-namespace warning now uses the final populated count, so listings containing only hidden internal objects are reported correctly.
  • The Docker example now includes the required namespace prefix, and Java/Python API docs describe bounded listing failures. The Java docs also clarify conversion from mount-relative paths to read URIs.

Correctness model

The control protocol remains schema v2 and intentionally does not add pagination in this follow-up. Because there is no continuation token on the wire, crossing any server-side listing limit is an explicit failure rather than an apparently successful truncated mount.

The coordinator uses one absolute deadline across all worker attempts. A silent worker cannot reset or consume the complete request budget: with a 25-second listing budget and two workers, the first can use up to 24 seconds while one second remains for failover. When the remaining budget is smaller, attempts share it evenly.

All listing paths are validated as one transaction before populate_from_listing runs, so a single invalid entry cannot leave a partially built mount.

Tests

  • cargo test -p talon-core -p talon-fuse --locked
  • cargo test -p talon-coordinator --locked
  • cargo test -p talon-python --locked
  • cargo clippy -p talon-core -p talon-coordinator -p talon-python --all-targets --all-features --locked -- -D warnings
  • cargo clippy -p talon-fuse --all-targets --locked -- -D warnings
  • cargo fmt --all -- --check
  • git diff --check

The full worker crate cannot compile on this macOS host because its existing splice, sendfile, CPU-affinity, and io_uring paths are Linux-only. The new worker regression tests are included for Linux CI. Java changes are Javadoc-only; Java CI will run the Maven suite.

@beinan
beinan merged commit c79a60c into milvus-io:main Jul 28, 2026
21 of 22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants