feat(worker): write-back skeleton — durable staging and flush queue - #360
Merged
Conversation
The write path is write-through today: the backend PUT is the durability point, so there is no worker-side dirty state. Write-back inverts that and needs machinery write-through never did. Add `WriteCache`: staged-write + fsync + atomic-rename durability with a sidecar commit marker written after the body, crash recovery that rebuilds the pending set and reclaims uncommitted bodies, a FIFO flush queue with per-object coalescing and an in-flight boundary, requeue-on-failure, and a checksum-verified read-your-writes lookup. Not wired into WorkerRuntime or the serve path: whether Talon promises write-back at all is an ADR-level decision (milvus-io#274 item 4), and NVMe alone is not durability — real write-back also needs replication before ack. Refs milvus-io#358 Co-Authored-By: Claude <noreply@anthropic.com>
This was referenced Jul 28, 2026
beinan
added a commit
that referenced
this pull request
Jul 28, 2026
WriteCache (#360) has a queue but nothing that drains it. Add Flusher: take_next -> backend.put -> complete on success, requeue on failure. ADR 0002 §5 rules out "retry forever", since unbounded retry turns a write failure into an unbounded capacity leak. So each object gets an attempt budget, counted per object so one poisonous object cannot exhaust the budget of everything queued behind it, with capped exponential backoff. An object that spends its budget is parked, not deleted — it is acknowledged data the origin refused, and discarding it would turn a delivery failure into silent data loss. Parked entries keep their files, stay counted in dirty_bytes, are enumerable via failed_objects, remain readable for read-your-writes, and are re-armed by retry_failed once the cause is fixed. A rewrite of the object releases the parked generation. Not reachable in production: ADR 0002 rejected write-back behind an experimental flag, so no config key constructs a Flusher and nothing on the serve path references one. Closes #364 Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #358. Part of #274 (roadmap item 4).
Why
The write path is strictly write-through today:
talon-fusebuffers a whole object in client memory and, atflush/fsync, PUTs it through the owning worker, which uploads to the origin and then caches the bytes. The backend PUT is the durability point, so there is no worker-side dirty state to lose.Write-back inverts that — acknowledge once the bytes are durable locally, upload afterwards — and needs machinery write-through never did. This PR builds that machinery as a self-contained module.
What
talon_worker::write_cache::WriteCache:Stager(staged write +fsync+ atomic rename), plus a directoryfsync. The sidecar is written after the body and is the commit marker, so an interrupt leaves either a complete entry or reclaimable garbage — never a pending entry pointing at a truncated body.seqwhere several generations survived, reclaims corrupt sidecars, and resumesseqabove the highest seen so numbers never repeat across a restart.take_nexthands an entry to the flusher it is immune to coalescing, so a hot object neither accumulates N uploads nor has its bytes pulled out from under an upload in progress.requeueretries a failed flush; if a newer generation was staged meanwhile, the failed one is discarded rather than moving the origin backwards.completeignores a staleseq, so a late completion cannot retire newer data. An unreadable staged file is dropped loudly rather than wedging the queue.staged_bytesreturns the newest staged generation, checksum-verified — corruption surfaces as an error, not as silently wrong data served in place of the origin.dirty_bytes()exposes acknowledged-but-not-yet-durable-at-the-origin bytes, the figure a write-back deployment has to alarm on.Deliberately out of scope
WorkerRuntimeor the serve path, and no config flag. Write-through remains the only reachable behavior. Whether Talon promises write-back at all is an ADR-level decision (roadmap: post-v1 direction — fail-open, cache admission, RDMA, write cache, freshness, and gaps #274 item 4, which also requires revisiting ADR 0001).Tests
16 unit tests covering staging, coalescing, the in-flight boundary, stale completions, requeue supersession, read-your-writes, checksum rejection, FIFO order, queue-wedging, and four recovery scenarios.
just ciclean.🤖 Generated with Claude Code