Skip to content

feat(worker): write-back skeleton — durable staging and flush queue - #360

Merged
beinan merged 1 commit into
milvus-io:mainfrom
beinan:write-cache
Jul 28, 2026
Merged

feat(worker): write-back skeleton — durable staging and flush queue#360
beinan merged 1 commit into
milvus-io:mainfrom
beinan:write-cache

Conversation

@beinan

@beinan beinan commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator

Closes #358. Part of #274 (roadmap item 4).

Why

The write path is strictly write-through today: talon-fuse buffers a whole object in client memory and, at flush/fsync, PUTs it through the owning worker, which uploads to the origin and then caches the bytes. The backend PUT is the durability point, so there is no worker-side dirty state to lose.

Write-back inverts that — acknowledge once the bytes are durable locally, upload afterwards — and needs machinery write-through never did. This PR builds that machinery as a self-contained module.

What

talon_worker::write_cache::WriteCache:

  • Durable staging. Body and a JSON sidecar each go through Stager (staged write + fsync + atomic rename), plus a directory fsync. The sidecar is written after the body and is the commit marker, so an interrupt leaves either a complete entry or reclaimable garbage — never a pending entry pointing at a truncated body.
  • Crash recovery. Startup rebuilds the pending set from committed sidecars, drops bodies whose sidecar never landed, keeps only the highest seq where several generations survived, reclaims corrupt sidecars, and resumes seq above the highest seen so numbers never repeat across a restart.
  • Coalescing with an in-flight boundary. Repeat writes to one object collapse to a single pending entry. Once take_next hands an entry to the flusher it is immune to coalescing, so a hot object neither accumulates N uploads nor has its bytes pulled out from under an upload in progress.
  • Failure handling. requeue retries a failed flush; if a newer generation was staged meanwhile, the failed one is discarded rather than moving the origin backwards. complete ignores a stale seq, so a late completion cannot retire newer data. An unreadable staged file is dropped loudly rather than wedging the queue.
  • Read-your-writes. staged_bytes returns the newest staged generation, checksum-verified — corruption surfaces as an error, not as silently wrong data served in place of the origin.
  • dirty_bytes() exposes acknowledged-but-not-yet-durable-at-the-origin bytes, the figure a write-back deployment has to alarm on.

Deliberately out of scope

Tests

16 unit tests covering staging, coalescing, the in-flight boundary, stale completions, requeue supersession, read-your-writes, checksum rejection, FIFO order, queue-wedging, and four recovery scenarios. just ci clean.

🤖 Generated with Claude Code

The write path is write-through today: the backend PUT is the durability
point, so there is no worker-side dirty state. Write-back inverts that and
needs machinery write-through never did.

Add `WriteCache`: staged-write + fsync + atomic-rename durability with a
sidecar commit marker written after the body, crash recovery that rebuilds
the pending set and reclaims uncommitted bodies, a FIFO flush queue with
per-object coalescing and an in-flight boundary, requeue-on-failure, and a
checksum-verified read-your-writes lookup.

Not wired into WorkerRuntime or the serve path: whether Talon promises
write-back at all is an ADR-level decision (milvus-io#274 item 4), and NVMe alone is
not durability — real write-back also needs replication before ack.

Refs milvus-io#358

Co-Authored-By: Claude <noreply@anthropic.com>
@beinan
beinan merged commit 5a389e6 into milvus-io:main Jul 28, 2026
21 checks passed
beinan added a commit that referenced this pull request Jul 28, 2026
WriteCache (#360) has a queue but nothing that drains it. Add Flusher:
take_next -> backend.put -> complete on success, requeue on failure.

ADR 0002 §5 rules out "retry forever", since unbounded retry turns a write
failure into an unbounded capacity leak. So each object gets an attempt
budget, counted per object so one poisonous object cannot exhaust the
budget of everything queued behind it, with capped exponential backoff.

An object that spends its budget is parked, not deleted — it is
acknowledged data the origin refused, and discarding it would turn a
delivery failure into silent data loss. Parked entries keep their files,
stay counted in dirty_bytes, are enumerable via failed_objects, remain
readable for read-your-writes, and are re-armed by retry_failed once the
cause is fixed. A rewrite of the object releases the parked generation.

Not reachable in production: ADR 0002 rejected write-back behind an
experimental flag, so no config key constructs a Flusher and nothing on the
serve path references one.

Closes #364

Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(worker): write-back skeleton — durable dirty staging and flush queue

1 participant