Skip to content

feat(fuse): stream large sparse writes - #348

Merged
beinan merged 1 commit into
milvus-io:mainfrom
Kazimierzsier:feat/fuse-streaming-writes
Jul 28, 2026
Merged

feat(fuse): stream large sparse writes#348
beinan merged 1 commit into
milvus-io:mainfrom
Kazimierzsier:feat/fuse-streaming-writes

Conversation

@Kazimierzsier

Copy link
Copy Markdown
Contributor

Summary

Add a bounded-memory write path for large and sparse FUSE objects without increasing the 1 GiB in-memory threshold. Dirty contents spill to a sparse temporary file, stream through the worker data plane, and upload to S3, GCS, or Azure without materializing the complete object in memory.

This PR is stacked on #346. Its diff will shrink to the streaming-write commit after the prerequisite POSIX stack merges.

Closes #347.

Changes

  • Keep objects at or below 1 GiB in the existing in-memory working copy.
  • Promote writes beyond the threshold to a sparse NamedTempFile.
  • Enforce a separate 4 GiB logical object limit so large offsets remain bounded.
  • Stream staged files from FUSE to workers with a size-aware timeout.
  • Receive large worker PUT bodies in fixed 8 MiB chunks and preserve zero ranges as sparse holes.
  • Add BackendStore::put_file and streamed HTTP request support.
  • Hash S3 payloads with a fixed 1 MiB buffer before SigV4 signing.
  • Send GCS and Azure streamed PUTs with an exact Content-Length.
  • Avoid inserting streamed multi-block objects into the single-block worker cache.
  • Cover sparse staging, worker transfer, backend request construction, and real-kernel FUSE writeback.

Large truncate remains memory-backed and retains the existing 1 GiB limit. This PR addresses large sparse writes; it does not claim large truncate support.

Test plan

Implementation commit: 7fd6bbf2725dd80ebbd33a6f0dfa6fe1018a2e6f

pjdfstest runner commit: 26f10147a214ed8dc6a59298aca4c5c3a4011263

  • cargo test -p talon-fuse --lib --features mount: 122 passed
  • cargo test -p talon-backend --lib: 77 passed
  • cargo test -p talon-worker --lib: 89 passed
  • cargo test -p talon-fuse --test mount_e2e --features mount --no-run
  • cargo check --workspace --all-targets
  • Strict clippy for core, backend, worker, and FUSE mount targets
  • cargo fmt --all -- --check
  • git diff --check
  • Exact implementation SHA passed 15/15 privileged real-kernel mount tests in falcon-phx-ca
  • open/25.t: 6/6 passed through a real kernel FUSE mount
  • open,rename,unlink,truncate,mkdir,rmdir: 5,981/5,981 passed

The open/25.t validation writes one byte at offset 2 GiB + 1, verifies the resulting size, reads the sparse hole, and reads the written byte. The FUSE layer, worker, and pjdfstest mock all use bounded-memory streaming for this path.

Checklist

  • PR title follows Conventional Commits
  • Public behavior is covered by tests
  • The 1 GiB in-memory threshold is unchanged
  • All code, tests, commit messages, and PR text are in English

@Kazimierzsier
Kazimierzsier force-pushed the feat/fuse-streaming-writes branch from 72c067a to b1a7d27 Compare July 28, 2026 00:09
@beinan
beinan merged commit bead8ed into milvus-io:main Jul 28, 2026
19 checks passed
@beinan

beinan commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator

Superseded by #351, which cherry-picks this commit onto current main with your authorship preserved and the content unchanged.

The conflict was historical rather than semantic: the lower half of this stack was squash-merged (#309#344), which rewrites history, so this PR still carried commits whose content was already upstream under different SHAs. Cherry-picking the top commit applies cleanly with no conflict markers, which is what confirmed that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fuse(posix): support large sparse writes without whole-object allocation

2 participants