talon-fuse serves stale data after the origin changes: namespace and attrs are a mount-time snapshot

[Kazimierzsier]

Found while running end-to-end freshness tests against a real Azure (ADLS Gen2) account. Filing separately from #482, which fixes the listing bug that blocked the mount in the first place.

Symptom

With a mount live, overwrite an object at the origin. The mount keeps serving the old bytes indefinitely — well past any TTL:

PUT obj1.bin (4100 bytes) -> 201
[FAIL] obj1.bin: mount converged to new content in 30s (was 110B, now 110B)
PUT obj2.bin (1048580 bytes) -> 201
[FAIL] obj2.bin: mount converged to new content in 30s (was 110B, now 110B)
DELETE obj3.bin -> 202
[FAIL] obj3.bin: mount stopped serving deleted object in 30s

The worker is not at fault

This is worth stating precisely, because the worker's revalidation works correctly. Its logs show the ETag-keyed version being re-resolved, the stale block evicted, and the new one committed:

MISS -> backend fetch  block=/az/.../obj1.bin@0x8DEF5F6F0907F9C#0+16777216
evicted block          block=/az/.../obj1.bin@0x8DEF5F679FC3644#0+16777216
committed block        block=/az/.../obj1.bin@0x8DEF5F6F0907F9C#0+16777216 bytes=4100

Two independent controls confirm the staleness lives above the worker:

1. A second mount started against the same worker immediately sees the new content and the new sizes (4100 / 1048580), while the first mount, same host, same worker, still reports 110 B.
2. echo 3 > /proc/sys/vm/drop_caches makes the original mount return the correct new bytes at once.

Two distinct causes

a) Namespace and inode size are a mount-time snapshot. populate_namespace runs once at startup and never refreshes. The inode's size is frozen there, so a read is clamped to the old length — an object that grew 110 B → 4100 B still reads 110 B, and one that was deleted keeps being served. Objects created after mount never appear at all (asserted explicitly in the harness, currently reported as visible without remount: False).

b) ATTR_TTL is 60s and the kernel page cache is never invalidated. crates/talon-fuse/src/mount.rs:40:

/// The namespace is populated from coordinator listings and is effectively
/// immutable for a mount session (read-only v1), so a generous TTL avoids a
/// callback per stat/lookup without risking staleness.
const ATTR_TTL: Duration = Duration::from_secs(60);

The comment states the assumption plainly — immutable for a mount session. Against a live object store that assumption does not hold. Even once (a) is fixed, the kernel caches both attributes and page data, so a size change must invalidate the cached pages rather than only updating the attribute.

Why it matters

The worker already does the hard part correctly: ETag-keyed blocks plus If-Match conditional GETs, which is exactly the mechanism needed for correct invalidation. The FUSE client discards that guarantee at the last hop. For the intended workload — training jobs reading datasets that are actively rewritten — silently serving a previous version is worse than a cache miss.

Suggested direction

• Refresh the namespace listing periodically (and/or revalidate an inode's stat on lookup/getattr past a short TTL) instead of trusting the startup snapshot.
• Separate the attribute TTL from the "immutable" assumption, and invalidate cached pages when a revalidation reveals a changed ETag or size.
• Decide and document the intended semantics — close-to-open consistency is the conventional choice here and is cheap to state precisely.

Repro harness (correctness + freshness against a real account, workload identity, no secrets at rest) is ready and I can contribute it alongside the fix.