feat: add client telemetry support

[xiaofan-luan]

Summary

• add client telemetry heartbeat, command handling, bounded metrics history, recent-error reporting, and server-driven configuration
• give every sync/async logical MilvusClient its own telemetry manager, client ID, configuration, history, command cursor, replies, and heartbeat
• keep pooled GrpcHandler/AsyncGrpcHandler instances transport-only so independent clients can share an authenticated channel without sharing telemetry state
• preserve one legacy fallback telemetry manager for direct ORM/handler connections
• keep the same logical telemetry manager across use_database, reconnect, and global-primary failover; candidate failures leave the previous handler and telemetry binding intact
• generation- and lease-token-fence transport rebinding so late reconnects and stale heartbeat responses cannot restore a retired stub or mutate current state
• report one outcome per canonical public logical operation across validation, retry, and result processing; Get, iterators, and internal attempts are not counted as Query
• preserve arbitrary non-empty legacy request IDs on the RPC wire while accepting only valid trace IDs in telemetry records
• keep at most one hour / 4096 history windows and aggregate p99 from 128 bounded per-window quantile samples rather than averaging window p99 values
• keep the command heartbeat alive after runtime disable, while an initial user opt-out starts no worker
• isolate unexpected telemetry exceptions from business RPCs and client close; saturate unsupported-RPC backoff without overflow

Related work

• Design: https://github.com/milvus-io/milvus/blob/master/docs/design-docs/design_docs/20260131-client_side_telemetry.md
• Server and Go SDK correctness follow-up: enhance: make client telemetry trustworthy -- windows, commands, config replies and sampling milvus#52597
• Go SDK conformance follow-up: fix: keep telemetry commands idempotent at cursor milvus#52804

Verification

• full unit suite: 4802 passed, 3 skipped
• focused telemetry / connection lifecycle suite: 306 passed
• deterministic register-vs-reconnect and custom-command lifecycle race tests
• Black (197 files), Ruff, and git diff --check

[sre-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: xiaofan-luan
To complete the pull request process, please assign longjiquan after the PR has been reviewed.
You can assign the PR to them by writing /assign @longjiquan in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[codecov]

Codecov Report

❌ Patch coverage is 92.38489% with 129 lines in your changes missing coverage. Please review.
✅ Project coverage is 94.16%. Comparing base (64e53d1) to head (0c846be).

Files with missing lines	Patch %	Lines
pymilvus/client/telemetry.py	96.60%	36 Missing ⚠️
pymilvus/milvus_client/async_milvus_client.py	80.32%	24 Missing ⚠️
pymilvus/client/connection_manager.py	71.01%	20 Missing ⚠️
pymilvus/client/async_grpc_handler.py	84.61%	18 Missing ⚠️
pymilvus/client/grpc_handler.py	87.78%	16 Missing ⚠️
pymilvus/client/asynch.py	90.24%	8 Missing ⚠️
pymilvus/milvus_client/milvus_client.py	91.95%	7 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3770      +/-   ##
==========================================
- Coverage   94.22%   94.16%   -0.07%     
==========================================
  Files          77       78       +1     
  Lines       16204    17809    +1605     
==========================================
+ Hits        15268    16769    +1501     
- Misses        936     1040     +104     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6f60fe4601

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[pymilvus-bot]

Requesting changes for the telemetry correctness and lifecycle issues called out inline. The most serious issue is that persistent configs are passed through the one-time command watermark, so a newly relevant config can be acknowledged and hashed without ever being applied. Iterator-internal RPCs and sync Future parsing also violate the stated one-final-outcome contract, and sync shutdown can return while its worker is still running. These were verified against the current head with focused reproductions.

[pymilvus-bot]

[P2] Clear config_hash when the server's persistent set becomes empty. After DeleteClientCommand removes the last config, the heartbeat returns no persistent commands, has_persistent is false, and this leaves the old hash in place. Every later heartbeat then disagrees with the server's empty hash and repeats the config lookup indefinitely. Commit calculate_config_hash(commands) for every accepted batch, including "" for an empty set.

[xiaofan-luan]

Thanks. We will address this ambiguity on the server side by returning an authoritative effective config hash (or an explicit empty-snapshot marker), so the client can distinguish a matching non-empty config set from an authoritative empty config set. This protocol follow-up is independent of the client rollout order. In this PR, we intentionally preserve the last accepted hash on an empty command batch to avoid hash oscillation and repeated config reapplication.