feat(fuse): stream large sparse writes - #348
Merged
Merged
Conversation
Kazimierzsier
force-pushed
the
feat/fuse-streaming-writes
branch
5 times, most recently
from
July 27, 2026 18:34
013fbd2 to
72c067a
Compare
Kazimierzsier
force-pushed
the
feat/fuse-streaming-writes
branch
from
July 28, 2026 00:09
72c067a to
b1a7d27
Compare
Collaborator
|
Superseded by #351, which cherry-picks this commit onto current The conflict was historical rather than semantic: the lower half of this stack was squash-merged (#309 → #344), which rewrites history, so this PR still carried commits whose content was already upstream under different SHAs. Cherry-picking the top commit applies cleanly with no conflict markers, which is what confirmed that. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add a bounded-memory write path for large and sparse FUSE objects without increasing the 1 GiB in-memory threshold. Dirty contents spill to a sparse temporary file, stream through the worker data plane, and upload to S3, GCS, or Azure without materializing the complete object in memory.
This PR is stacked on #346. Its diff will shrink to the streaming-write commit after the prerequisite POSIX stack merges.
Closes #347.
Changes
NamedTempFile.BackendStore::put_fileand streamed HTTP request support.Content-Length.Large
truncateremains memory-backed and retains the existing 1 GiB limit. This PR addresses large sparse writes; it does not claim large truncate support.Test plan
Implementation commit:
7fd6bbf2725dd80ebbd33a6f0dfa6fe1018a2e6fpjdfstest runner commit:
26f10147a214ed8dc6a59298aca4c5c3a4011263cargo test -p talon-fuse --lib --features mount: 122 passedcargo test -p talon-backend --lib: 77 passedcargo test -p talon-worker --lib: 89 passedcargo test -p talon-fuse --test mount_e2e --features mount --no-runcargo check --workspace --all-targetscargo fmt --all -- --checkgit diff --checkfalcon-phx-caopen/25.t: 6/6 passed through a real kernel FUSE mountopen,rename,unlink,truncate,mkdir,rmdir: 5,981/5,981 passedThe
open/25.tvalidation writes one byte at offset2 GiB + 1, verifies the resulting size, reads the sparse hole, and reads the written byte. The FUSE layer, worker, and pjdfstest mock all use bounded-memory streaming for this path.Checklist