feat: add "reset wal" to empty an instance's WAL - #535
Open
tinswzy wants to merge 4 commits into
Open
Conversation
Collaborator
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: tinswzy The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
@tinswzy Thanks for your contribution. Please submit with DCO, see the contributing guide https://github.com/milvus-io/milvus/blob/master/CONTRIBUTING.md#developer-certificate-of-origin-dco. |
"Switch a stopped instance onto a different WAL" and "restore an instance from a
cold backup" both assume the WAL is already empty, and nothing in the operator's
toolbox could establish that. reset wal fills the gap for woodpecker: it deletes
every log's data across both storage tiers, then clears the instance metadata.
The log id counter is deliberately preserved. logId appears in the data paths of
both tiers, and residue cannot be ruled out in a distributed system, so a wiped
counter would let a new log write into a previous one's directory. Monotonic
logId is the invariant that makes reuse safe; ClearMetaExceptLogIdGen keeps it.
Milvus must be stopped, but the WAL service, etcd and object storage must stay
up: the log-store nodes are what delete their own local data.
Object storage roots at "{minio.rootPath}/wp", the suffix Milvus appends
unconditionally when it builds woodpecker's config. The command derives it rather
than asking for the final path, because the value an operator would paste from
milvus.yaml is wrong by exactly that suffix — and a wrong root fails silently: the
node's local reclaim is an os.RemoveAll on a path that does not exist and the
object prefix lists empty, so every tier reports success while all of the data
survives.
Verified end to end on minikube against milvus v3.0.0 with a three-node woodpecker
service deployment: 1500 rows seeded and flushed, milvus stopped, reset wal then
reset checkpoint, milvus restarted. Node-local data.log files for all 16 logs were
removed on all three nodes, the log id counter held at 32 across the clear, no pod
restarted, existing rows read back intact and new writes landed on the empty WAL.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
tinswzy
force-pushed
the
feat/clear-wal
branch
from
August 31, 2026 05:12
b0446c2 to
258d93f
Compare
|
Tick the box to add this pull request to the merge queue (same as
|
A clear that removes nothing looked exactly like a clear that had nothing to remove: the node's reclaim is an os.RemoveAll on a path that may not exist and the object prefix simply lists empty, so a wrong --minio-bucket or --minio-root-path produced a confident "deleted N log(s)" while every byte survived. woodpecker now returns per-tier counts, so the command prints what it actually did and calls out the case where it deleted metadata but found no data anywhere. That is a warning rather than a failure: an already-empty WAL looks identical, and refusing would break the idempotent re-run the command depends on. Also corrects the dry-run hint, which still told the operator to stop the WAL service. The service must stay up — its nodes are what delete their own local data. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
Picks up the verbatim object prefix, the deleting gate surviving a synchronous delete, the retryable cleanup-storage initializer, the sync_applied handshake, and the parked-log object sweep. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
A wrong --meta-prefix was the one argument that produced a confident success with nothing done. Every enumeration is scoped to the prefix, so a wrong one listed no logs, deleted nothing, re-seeded the instance-level keys somewhere harmless, and printed "wal is empty. Next: reset checkpoint". An operator following that instruction rewrites every position to earliest against a WAL that still holds all of its data — the exact corruption this command exists to prevent. Verified on a live deployment: with --meta-prefix WRONGPREFIX the run reported success while all 52 metadata keys and 92 objects survived untouched. The command now probes the version key before doing anything. ClearMeta re-seeds that key, so a re-run of a completed clear still passes and the idempotency contract holds; an instance that never started woodpecker has no key either, and refusing there is right too, since there is no WAL to clear. Metadata at the legacy top-level "woodpecker" prefix — which woodpecker itself falls back to — is reported as such rather than as missing, because the fix differs: point --meta-prefix at it instead of concluding there is no WAL. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #534
Adds
reset wal, which deletes an instance's WAL data and metadata so theinstance can be restarted onto an empty WAL. It is the missing first half of the
cold-switch flow:
reset checkpointrewrites the positions, but until now nothingcould establish the precondition both it and cold restore assume — that the WAL is
actually empty.
Only woodpecker is supported. The other WALs are external services with their own
tooling, and the command rejects them by name rather than pretending to handle them.
Usage
Dry-run by default;
--run=trueexecutes. Run it beforereset checkpoint.Milvus must be stopped, but the WAL service, etcd and object storage must stay
up. The log-store nodes are what delete their own local data — with woodpecker
scaled to zero the client cannot resolve any node and the delete fails closed.
What it does
GetAllLogs→ prints the logs it is about to removeDeleteAllLogsSync→ per log: fence the nodes, delete objects by prefix, deletemetadata last
ClearMetaExceptLogIdGen→ clears the instance's woodpecker metadataWhy the log id counter is preserved
logIdappears in the data paths of both storage tiers, and residue cannot beruled out in a distributed system — a node that is permanently down, a segment
whose metadata was already truncated so its node is not in the delete fan-out at
all, a node decommissioned between write and delete. "All historical files are
gone" is not a guarantee this system can offer, so correctness rests on an
invariant instead:
logIdnever goes backwards. Wiping the counter would restartat
logId=1and let a new log write into a previous one's directory.The
/wpsuffix, and why a wrong root is dangerousObject storage roots at
{minio.rootPath}/wp— the suffix Milvus appendsunconditionally in
pkg/streaming/walimpls/impls/wp/builder.go. The commandderives it rather than asking for the final path, because the obvious thing to
paste into
--minio-root-pathis therootPathfrom milvus.yaml, and that valueis wrong by exactly this suffix.
A wrong root does not fail loudly. The node's local reclaim is an
os.RemoveAllona path that simply does not exist, and the object prefix lists empty, so every
tier reports success while all of the data survives. This was not hypothetical —
the first E2E run hit it, reported "deleted 16 log(s)", and left all 16
data.logfiles untouched on every node.
Two things guard it now. The dry-run prints the resolved prefix
(
milvus-bucket/file/wp/{logId}/) so it can be confirmed by eye, and the runreports what each tier actually removed:
When a clear deletes metadata but finds no data anywhere, it says so:
That is a warning rather than a failure on purpose: an already-empty WAL looks
identical, and failing would break the idempotent re-run this command depends on.
Verification
go test ./states/etcd/reset/...covers WAL-type rejection, that metadata roots atthe instance rather than the meta path, storage-type handling, and the
/wpderivation.
End to end on minikube — milvus v3.0.0, chart 5.0.25, three-node woodpecker
service deployment, etcd and minio in-cluster.
Reaching all three tiers took some setup worth recording. Segments only reach
object storage after the client auditor compacts them, and milvus v3.0.0 embeds
woodpecker client v0.1.33, which calls the legacy
CompactSegmentRPC that currentservers reject (
legacy CompactSegment RPC is disabled; upgrade the client to use CompactSegmentWithExpected,segmentsCompacted=0). With a stock milvus image,object storage stays empty and that tier is never exercised at all — the first
round of this E2E passed without ever deleting a single object. Data was
subsequently written through woodpecker's own client to produce genuine
footer.blk/m_0.blkobjects in the real layout.With all three tiers populated:
data.logper node × 3,logidgen=49reset wal --run=truelogidgenstill 49deleted 0 log(s), counter unchanged — idempotentreset checkpoint --target-wal woodpeckerlogidgen49 → 65, never returned to 1The 16
data.logfiles left behind on each node are orphans from the first,misdirected run: their metadata was already deleted, so nothing can enumerate them.
They are the concrete cost of the failure mode described above.
A wrong-bucket run was also exercised: object cleanup failed and the log's metadata
was deliberately kept for retry (
keeping metadata for retry) rather thandeleted, so the log stays enumerable instead of becoming an orphan.
Dependency note
go.modcurrently points at a woodpecker pseudo-version carryingzilliztech/woodpecker#281, which adds the synchronous delete,
ClearMeta, and theper-tier counts this command reports. This is temporary, to make the change
reviewable and verifiable as a whole. Once #281 merges I will repoint this at a
released woodpecker version; please do not merge before then.
🤖 Generated with Claude Code