startup: gRPC socket only appears after full recovery — restarts on nodes with accumulated state go dark for minutes

[iaroslav-reflection]

On restart, NewSnapshotter completes all recovery work before Serve() creates the unix socket: per-daemon GetState, restarting dead daemons (including remounting their RAFS instances), the hot-upgrade check, and the initial credential reconciliation (#790 moves that one off the startup path). Recovery time scales with the daemons/mounts/snapshots accumulated on the node, and nothing prunes those, so on a long-lived node a restart can leave the socket missing for minutes — containerd gets Unavailable for every snapshot RPC and all new container creation on the node fails until recovery finishes. systemd reports the unit active the whole time (Type=simple), so nothing signals the window. A crash during recovery re-enters it every RestartSec.

Two design questions before proposing anything concrete:

1. Is serve-after-full-recovery an invariant you want to keep (correctness of snapshot ops against half-recovered state), or would you consider serving earlier and recovering daemons in the background?
2. If the ordering must stay: would you take (a) an opt-in readiness signal (sd_notify + Type=notify) so operators can gate on and observe the recovery window, and/or (b) parallelizing the per-daemon recovery walk so the window is bounded by the slowest daemon instead of the sum?

Happy to send PRs for whichever direction you prefer.

[Fricounet]

Personally, I think 1. is important. Having the nydus-snapshotter or any of the nydus daemon enter into a correctness issue is always a nightmare to fix, and I find that the best solution is often to trash the node entirely. So I'd rather not risk more situations leading to this.

For 2. I think both a and b make sense if you want to implement them

Other opinions @imeoer @BraveY ?

[iaroslav-reflection]

Measured the restart path on a test node (fusedev dedicated, 8 daemon records, processes dead at startup): socket absent for 120s, a pod created during the window took 116s to start and produced 11 error events (9× FailedCreatePodSandBox). The entire window is recoverDaemons — strictly serial, ~15s per dead daemon, and the 15s each is a retry bug in WaitUntilSocketExisted, fixed in #792. With that fix the same node serves in 0.5s and the pod starts clean.

So the pathological case is gone; 2(a) sd_notify still seems worth it for observability, and 2(b) matters mainly for many live daemons where GetState/remount time dominates. Will send 2(a) first unless you'd rather I start elsewhere.

[BraveY]

Personally, I think 1. is important. Having the nydus-snapshotter or any of the nydus daemon enter into a correctness issue is always a nightmare to fix, and I find that the best solution is often to trash the node entirely. So I'd rather not risk more situations leading to this.

For 2. I think both a and b make sense if you want to implement them

Other opinions @imeoer @BraveY ?

Agree with @Fricounet , We need to ensure that all nydusd daemons have started normally before providing services externally. Otherwise, it may cause many unpredictable issues in production env.

[imeoer]

Thanks for raising this issue. I also agree with optimizing the daemon’s parallel processing and ensuring all daemons finish before serving snapshotter requests. after all, the upper containerd will keep retrying.