Add reset wal to clear a stopped instance's WAL data and metadata
Background
reset checkpoint (#523) rewinds every persisted MQ position to the target WAL's
earliest position, so a stopped instance can be brought up on a different MQ. It
assumes something the operator has no tool to guarantee: that the target WAL is
actually empty.
For a genuinely new MQ that holds, and that is the case #523 was verified against.
It does not hold when the instance stays on woodpecker — switching a woodpecker
instance onto a fresh woodpecker, restoring from a cold backup, or rebuilding an
instance in place. There the WAL still has history, and Milvus has been truncating
it all along:
// internal/streamingnode/server/wal/recovery/recovery_background_task.go
if flusherCP.MessageID.LTE(checkpoint.MessageID) {
_ = rs.truncator.Truncate(ctx, flusherCP.MessageID)
} else {
_ = rs.truncator.Truncate(ctx, checkpoint.MessageID)
}
so segment 0 / entry 0 is long gone. Woodpecker does not fail that seek — it
silently moves the read position forward to the truncation point
(log_handle.go: "Only when specifying earliest and truncated segId != -1 will the
position be moved to the truncated point"). The instance comes up replaying from
somewhere in the middle, with recovery_inconsistent_event_total climbing and no
obvious symptom.
The fix is not to make reset checkpoint smarter. It is to make "the WAL is empty"
true before it runs.
What is missing
Emptying a woodpecker WAL means deleting its logs (node-local data, objects in
object storage) and then its instance metadata. Until
zilliztech/woodpecker#279 there was no way to do that reliably — DeleteLog left
object storage untouched, its local cleanup was asynchronous with no completion
signal, and there was no instance-level metadata clear at all.
That gap is being closed in zilliztech/woodpecker#281, which adds:
DeleteAllLogsSync — returns only once every node has reclaimed its local data
and the objects are gone
ClearMetaExceptLogIdGen — clears the instance's content metadata while
preserving the log id counter
Proposal
Add reset wal to the reset family, which already carries the right contract:
commands that deliberately discard metadata that is still valid, and therefore
require the cluster to be stopped.
stop milvus + the WAL service (etcd and object storage stay up)
reset wal ← the WAL becomes genuinely empty
reset checkpoint --target-wal ... ← rewinds Milvus's positions
start milvus
Putting it in the same family is deliberate: the ordering constraint between the two
is easy to get wrong and invisible when you do — running reset checkpoint first
leaves the instance pointing at "earliest" on a WAL that still holds truncated
history, which does not fail loudly.
Scope for the first version:
- woodpecker only. pulsar and kafka are external services with their own
retention and their own tooling; clearing them is not birdwatcher's call. The flag
is validated rather than ignored, so asking for one of them says so.
- Idempotent and synchronous. Deletion re-enumerates what is left and the
metadata clear is a prefix wipe plus overwrites, so an interrupted run is resumed
by running it again. The command returns only when the WAL is empty, which is what
the next step depends on.
- Default dry-run, like the rest of the family.
The log id counter is preserved
ClearMetaExceptLogIdGen, not a full wipe. logId appears in the object-storage and
node-local data paths, so restarting the counter would let a new log reuse a
directory an old log's objects may still occupy — and residue cannot be ruled out,
because a node can be permanently down when its log is deleted, and a node holding
data for an already-truncated segment is not discoverable from metadata at all.
Keeping the counter monotonic makes that collision structurally impossible rather
than contingent on the deletion having been complete.
Where woodpecker's metadata lives
Worth stating because it is the easy thing to get wrong: woodpecker roots its
metadata at {etcd.rootPath}/{meta-prefix} — a sibling of Milvus's meta path,
not a child of it. Deriving it from basePath the way the rest of the reset family
does would aim the clear at {instance}/meta/woodpecker, a prefix that holds
nothing, and the command would report success having deleted nothing.
The object-storage side (endpoint, credentials, bucket, root path) has to come from
flags: Milvus keeps those in its own config file rather than in etcd.
Dependency note
This needs zilliztech/woodpecker#281 merged and released. The PR will initially pin a
pseudo-version at that PR's commit so the whole flow can be reviewed and verified
end to end; it will be repointed at a released version before merge.
Add
reset walto clear a stopped instance's WAL data and metadataBackground
reset checkpoint(#523) rewinds every persisted MQ position to the target WAL'searliest position, so a stopped instance can be brought up on a different MQ. It
assumes something the operator has no tool to guarantee: that the target WAL is
actually empty.
For a genuinely new MQ that holds, and that is the case #523 was verified against.
It does not hold when the instance stays on woodpecker — switching a woodpecker
instance onto a fresh woodpecker, restoring from a cold backup, or rebuilding an
instance in place. There the WAL still has history, and Milvus has been truncating
it all along:
so
segment 0 / entry 0is long gone. Woodpecker does not fail that seek — itsilently moves the read position forward to the truncation point
(
log_handle.go: "Only when specifying earliest and truncated segId != -1 will theposition be moved to the truncated point"). The instance comes up replaying from
somewhere in the middle, with
recovery_inconsistent_event_totalclimbing and noobvious symptom.
The fix is not to make
reset checkpointsmarter. It is to make "the WAL is empty"true before it runs.
What is missing
Emptying a woodpecker WAL means deleting its logs (node-local data, objects in
object storage) and then its instance metadata. Until
zilliztech/woodpecker#279 there was no way to do that reliably —
DeleteLogleftobject storage untouched, its local cleanup was asynchronous with no completion
signal, and there was no instance-level metadata clear at all.
That gap is being closed in zilliztech/woodpecker#281, which adds:
DeleteAllLogsSync— returns only once every node has reclaimed its local dataand the objects are gone
ClearMetaExceptLogIdGen— clears the instance's content metadata whilepreserving the log id counter
Proposal
Add
reset walto theresetfamily, which already carries the right contract:commands that deliberately discard metadata that is still valid, and therefore
require the cluster to be stopped.
Putting it in the same family is deliberate: the ordering constraint between the two
is easy to get wrong and invisible when you do — running
reset checkpointfirstleaves the instance pointing at "earliest" on a WAL that still holds truncated
history, which does not fail loudly.
Scope for the first version:
retention and their own tooling; clearing them is not birdwatcher's call. The flag
is validated rather than ignored, so asking for one of them says so.
metadata clear is a prefix wipe plus overwrites, so an interrupted run is resumed
by running it again. The command returns only when the WAL is empty, which is what
the next step depends on.
The log id counter is preserved
ClearMetaExceptLogIdGen, not a full wipe.logIdappears in the object-storage andnode-local data paths, so restarting the counter would let a new log reuse a
directory an old log's objects may still occupy — and residue cannot be ruled out,
because a node can be permanently down when its log is deleted, and a node holding
data for an already-truncated segment is not discoverable from metadata at all.
Keeping the counter monotonic makes that collision structurally impossible rather
than contingent on the deletion having been complete.
Where woodpecker's metadata lives
Worth stating because it is the easy thing to get wrong: woodpecker roots its
metadata at
{etcd.rootPath}/{meta-prefix}— a sibling of Milvus's meta path,not a child of it. Deriving it from
basePaththe way the rest of theresetfamilydoes would aim the clear at
{instance}/meta/woodpecker, a prefix that holdsnothing, and the command would report success having deleted nothing.
The object-storage side (endpoint, credentials, bucket, root path) has to come from
flags: Milvus keeps those in its own config file rather than in etcd.
Dependency note
This needs zilliztech/woodpecker#281 merged and released. The PR will initially pin a
pseudo-version at that PR's commit so the whole flow can be reviewed and verified
end to end; it will be repointed at a released version before merge.