feat: worker-side per-tenant rate limiting

[somfornot]

Summary

Worker-side per-tenant rate limiting for the SDK-direct data plane: when a
client declares a tenant, the worker admits or throttles the read against a
static per-tenant policy. Supersedes #550 (opened from a different fork; same
feature, now with the requested read_throughput / burst_ratio config shape).

What it does

• Tenant identity (talon-core): TenantId::Named (SDK-declared) /
  Unattributed (undeclared, e.g. FUSE). The direct path is unauthenticated, so
  a declared tenant is trusted for fairness, not as a security boundary.
• Wire (talon-transport): new MsgType::GetRangeTenant carrying a
  TenantScopedRange; the reply is an ordinary GetRange frame. Fail-closed
  on older workers (a distinct opcode, like GetCachedRange), and conformance
  vectors are unchanged so the other-language clients are unaffected.
• Policy ([rate_limits] table): enabled, a default MetricLimits, and
  per-tenant overrides. Each metric is RateLimit { rate, burst_ratio } where
  the burst is rate * burst_ratio. Metrics: read_iops (req/s) and
  read_throughput (bytes/s served to the client). origin_read_bytes is a
  defined metric for a later stage.
• Enforcement (talon-worker): a lock-free TenantRateLimiter (one GCRA
  cell per (tenant, metric), single-atomic TAT, Arc-shared across io_uring
  rings). Admission before serve in both the io_uring and Tokio handlers;
  charges read_iops + read_throughput. A throttle returns
  DataErrorCode::RateLimited.
• Client (talon-cache-client): WorkerClient::with_tenant sends
  GetRangeTenant; RateLimited maps to a non-origin-fallback
  CacheReadError (a throttle can't be evaded via origin bypass).
• Gateway: RateLimited → S3 503 SlowDown / Azure 503 ServerBusy.
• Telemetry: bounded talon_worker_rate_limited_total{metric}; no raw
  tenant labels (per-tenant drill-down is a management-API concern).

Config example

[rate_limits]
enabled = true
[rate_limits.default]
read_iops = { rate = 5000, burst_ratio = 2.0 }
read_throughput = { rate = 1073741824 }        # 1 GiB/s; burst_ratio defaults to 1.0
[rate_limits.overrides.acme]
read_iops = { rate = 20000, burst_ratio = 2.0 }

Testing

cargo test green: talon-core (71) and talon-worker (253). Unit coverage:
GCRA burst/refill/burst-ratio, limiter admission (per-tenant, overrides,
throughput), [rate_limits] TOML parse, and WorkerClient tenant-scoped vs
plain-GetRange send.

Deliberate follow-ups (noted in code)

• origin_read_bytes enforcement (needs the tenant threaded into the
  cache-miss path).
• Per-SDK tenant knobs (BlockReader / talon-client CLI / C / Python / Java).
• A socket-level worker end-to-end throttle test.
• Later stages: coordinator policy distribution → consistent-hash owner/peer
  layer → Tier-2 monitoring (docs/explanation/tenant-traffic-observability.md).

🤖 Generated with Claude Code

[beinan]

Great feature! Thank you for the contribution! the cache industry and oss community were waiting for this for years!

could you fix the ci? thanks a lot!

[beinan]

The overall structure is coherent, but the inline findings below leave rate limiting bypassable or capable of permanently rejecting valid reads. Please address these before merge. The dependency-advisories check also currently fails on RUSTSEC-2026-0258 (h2 0.4.15; patched in 0.4.16).

[beinan]

A charge larger than the burst capacity can never be admitted. For example, at 10 MiB/s with burst_ratio = 1.0, a 16 MiB read always has a 1.6s increment against a 1s tolerance. Retrying after the returned 0.6s does not help because base advances to the new now, so every retry returns the same throttle forever. Please either charge/police throughput in bounded transfer chunks (as the proposal describes) or explicitly handle costs larger than the bucket capacity instead of returning a retry time that can never succeed. Add a regression test where cost > rate * burst_ratio.

[beinan]

This representation caps every configured rate above 1,000,000,000 units/s at an effective 1,000,000,000 units/s because the per-unit interval is clamped to 1ns. That already makes the PR's 1 GiB/s example about 6.9% too restrictive, and larger byte rates diverge further. Please retain fractional precision (for example with fixed-point TAT units or by computing the cost-scaled increment before division), and cover a rate above 1e9 in tests.

[beinan]

This map grows permanently for every tenant name observed on the unauthenticated data plane. Neither handler calls TenantId::validate(), and there is no cardinality bound or expiry, so a client can continuously send unique (up to control-frame-sized) tenant names and grow worker memory without limit; even tenants whose resolved MetricLimits are empty get an entry. Please validate at decode/admission and introduce bounded lifecycle/cardinality semantics for tenant cells, with a test that exercises the bound.

[beinan]

with_tenant does not actually attribute every fetch: fetch_cached_range() still emits an unscoped GetCachedRange, and both worker cached-range handlers bypass TenantRateLimiter. A tenant can therefore avoid both read_iops and read_throughput whenever the requested version is resident. Please add a tenant-scoped cache-only operation (or fail closed for this combination) and enforce the same admission in both worker handlers.