Skip to content

feat: zone-aware cache reads with same-AZ placement (ADR 0006) - #532

Merged
beinan merged 1 commit into
milvus-io:mainfrom
Thor-ChenBiao:claude/zone-aware-cache-reads
Aug 12, 2026
Merged

feat: zone-aware cache reads with same-AZ placement (ADR 0006)#532
beinan merged 1 commit into
milvus-io:mainfrom
Thor-ChenBiao:claude/zone-aware-cache-reads

Conversation

@Thor-ChenBiao

@Thor-ChenBiao Thor-ChenBiao commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

What

Zone-aware cache reads (ADR 0006): steady-state cache reads stay inside the reader's availability zone, cutting cross-AZ transfer cost. Off by default; enabling changes read placement only. Closes #531.

  • Workers self-resolve their zone (TALON_ZONE env, else the node's topology.kubernetes.io/zone label via the Kubernetes API through TALON_NODE_NAME + ServiceAccount; never blocks startup) and report it in the existing status-heartbeat labels map — no wire-format change.
  • The coordinator stores the zone inside the membership record and serves it via new schema-v5 messages MembershipQueryV2/MembershipListV2 (ZonedNodeInfo appended at the enum tail, minimum_schema-gated; conformance vectors regenerated — pre-v5 encodings are byte-identical).
  • Readers opt in with TALON_ZONE_AFFINITY=true: placement is computed over the same-zone worker subset with the unchanged hash; an empty local subset falls back to full membership (counted); V2 auto-downgrades to V1 against older coordinators with a retry cooldown. Both upgrade orders are tested, including cooldown expiry re-probing.
  • Every fetched block is classified same/cross/unknown against the reader's zone. Gateway exports talon_gateway_zone_reads_total, talon_gateway_zone_read_bytes_total, talon_gateway_zone_affinity_fallback_total (classification runs with the switch off too, so the cross-zone baseline is measurable first). The FUSE mount logs throttled fallback warnings instead.
  • Deploy templates: worker zone topologySpreadConstraints, TALON_NODE_NAME downward API, talon-node-zone-reader ClusterRole, sidecar notes. docs/operations/zone-affinity.md covers the consumer-to-gateway hop via Kubernetes Topology Aware Routing (service.kubernetes.io/topology-mode: Auto) including its silent-fallback caveat.

Design notes (deliberate calls, recorded in the ADR)

  1. Zone stays out of the canonical placement version. ADR 0001 §7.1 anticipated folding a failure-domain field into the version; a late-reported zone is a client-local concern, so readers detect it via a local zones token and rebuild their own table instead of churning every reader's cache (ADR 0006 §3).
  2. ZoneReadObserver mirrors the existing CredentialsObserver → binary-newtype pattern for library-to-binary metrics; ReadStats has no labeled dimensions and Prometheus counters need push.
  3. ADR + implementation land as one PR (maintainer preference for a single reviewable change).

Verification

  • macOS + Linux container: cargo fmt, clippy --all-targets -D warnings, full test suites across all touched crates, including the Linux-only io_uring transport lib tests (69), worker (244), fuse (71).
  • New deadlock regression test concurrent_reconcile_and_zoned_snapshot_never_wedge validated in both directions: a two-lock membership layout wedges it (clean 30 s timeout failure, no CI hang); the committed single-lock layout passes in 0.08 s.
  • Conformance suite passes; the two new v5 vectors are name-keyed and inert for the Java/Python/C client jobs.
  • Config reference and conformance vectors regenerate byte-identically; typos/link checks clean.
  • Groundwork verified live on UAT clusters earlier: topology.kubernetes.io/zone present on EKS v1.34.9 / ACK v1.34.1 / AKS v1.34.8; K8s ≥ 1.27 (Topology Aware Routing) on all target clusters.

Cross-AZ traffic is billed both ways while the origin object store is
regional, so steady-state cache reads should stay inside the reader's
zone. Off by default; enabling changes read placement only.

- workers resolve their zone (TALON_ZONE env, else the node's
  topology.kubernetes.io/zone label through the Kubernetes API via
  TALON_NODE_NAME + ServiceAccount; never blocks startup) and report it
  in the existing status-heartbeat labels map — no wire-format change
- coordinator stores zones and serves them through new schema-v5
  control messages MembershipQueryV2/MembershipListV2 (ZonedNodeInfo
  appended at the enum tail, gated by minimum_schema; conformance
  vectors regenerated)
- cache clients opt in with TALON_ZONE_AFFINITY: placement is computed
  over the same-zone worker subset, an empty local subset falls back to
  full membership (counted), and V2 queries auto-downgrade to V1
  against older coordinators with a retry cooldown
- every fetched block is classified same/cross/unknown against the
  reader's zone; gateway exports talon_gateway_zone_reads_total,
  talon_gateway_zone_read_bytes_total and
  talon_gateway_zone_affinity_fallback_total (measurable baseline
  before the switch is thrown)
- deploy templates: worker zone spread constraint, TALON_NODE_NAME
  downward API, talon-node-zone-reader RBAC, sidecar notes; ops doc
  covers the consumer-to-gateway hop via Kubernetes Topology Aware
  Routing (service.kubernetes.io/topology-mode: Auto)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Thor-ChenBiao
Thor-ChenBiao force-pushed the claude/zone-aware-cache-reads branch from bd6791c to e7ddb63 Compare August 11, 2026 18:22
@beinan
beinan merged commit 7822e9a into milvus-io:main Aug 12, 2026
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Zone-aware cache reads: keep cache traffic inside the reader's AZ

2 participants