feat: zone-aware cache reads with same-AZ placement (ADR 0006) - #532
Merged
beinan merged 1 commit intoAug 12, 2026
Merged
Conversation
Cross-AZ traffic is billed both ways while the origin object store is regional, so steady-state cache reads should stay inside the reader's zone. Off by default; enabling changes read placement only. - workers resolve their zone (TALON_ZONE env, else the node's topology.kubernetes.io/zone label through the Kubernetes API via TALON_NODE_NAME + ServiceAccount; never blocks startup) and report it in the existing status-heartbeat labels map — no wire-format change - coordinator stores zones and serves them through new schema-v5 control messages MembershipQueryV2/MembershipListV2 (ZonedNodeInfo appended at the enum tail, gated by minimum_schema; conformance vectors regenerated) - cache clients opt in with TALON_ZONE_AFFINITY: placement is computed over the same-zone worker subset, an empty local subset falls back to full membership (counted), and V2 queries auto-downgrade to V1 against older coordinators with a retry cooldown - every fetched block is classified same/cross/unknown against the reader's zone; gateway exports talon_gateway_zone_reads_total, talon_gateway_zone_read_bytes_total and talon_gateway_zone_affinity_fallback_total (measurable baseline before the switch is thrown) - deploy templates: worker zone spread constraint, TALON_NODE_NAME downward API, talon-node-zone-reader RBAC, sidecar notes; ops doc covers the consumer-to-gateway hop via Kubernetes Topology Aware Routing (service.kubernetes.io/topology-mode: Auto) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Thor-ChenBiao
force-pushed
the
claude/zone-aware-cache-reads
branch
from
August 11, 2026 18:22
bd6791c to
e7ddb63
Compare
beinan
approved these changes
Aug 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Zone-aware cache reads (ADR 0006): steady-state cache reads stay inside the reader's availability zone, cutting cross-AZ transfer cost. Off by default; enabling changes read placement only. Closes #531.
TALON_ZONEenv, else the node'stopology.kubernetes.io/zonelabel via the Kubernetes API throughTALON_NODE_NAME+ ServiceAccount; never blocks startup) and report it in the existing status-heartbeatlabelsmap — no wire-format change.MembershipQueryV2/MembershipListV2(ZonedNodeInfoappended at the enum tail,minimum_schema-gated; conformance vectors regenerated — pre-v5 encodings are byte-identical).TALON_ZONE_AFFINITY=true: placement is computed over the same-zone worker subset with the unchanged hash; an empty local subset falls back to full membership (counted); V2 auto-downgrades to V1 against older coordinators with a retry cooldown. Both upgrade orders are tested, including cooldown expiry re-probing.talon_gateway_zone_reads_total,talon_gateway_zone_read_bytes_total,talon_gateway_zone_affinity_fallback_total(classification runs with the switch off too, so the cross-zone baseline is measurable first). The FUSE mount logs throttled fallback warnings instead.topologySpreadConstraints,TALON_NODE_NAMEdownward API,talon-node-zone-readerClusterRole, sidecar notes.docs/operations/zone-affinity.mdcovers the consumer-to-gateway hop via Kubernetes Topology Aware Routing (service.kubernetes.io/topology-mode: Auto) including its silent-fallback caveat.Design notes (deliberate calls, recorded in the ADR)
ZoneReadObservermirrors the existingCredentialsObserver→ binary-newtype pattern for library-to-binary metrics;ReadStatshas no labeled dimensions and Prometheus counters need push.Verification
cargo fmt,clippy --all-targets -D warnings, full test suites across all touched crates, including the Linux-only io_uring transport lib tests (69), worker (244), fuse (71).concurrent_reconcile_and_zoned_snapshot_never_wedgevalidated in both directions: a two-lock membership layout wedges it (clean 30 s timeout failure, no CI hang); the committed single-lock layout passes in 0.08 s.topology.kubernetes.io/zonepresent on EKS v1.34.9 / ACK v1.34.1 / AKS v1.34.8; K8s ≥ 1.27 (Topology Aware Routing) on all target clusters.