Fix HC2 and HC2-BM finite variances at leverage one

[igerber]

Summary

HC2 and unweighted, unclustered HC2-BM could report finite variance for a perfectly leveraged observation by flooring 1 - h_ii. For example, a regression on [1, D] with only one observation where D=1 could retain finite covariance despite having no residual information for that observation.

These paths now follow HC3's defensive cutoff: any effective leverage h_ii >= 1 - 1e-8 produces a UserWarning, entirely NaN covariance and requested degrees of freedom, and unavailable downstream inference. Identified coefficients, residuals, and fitted values are retained. Numerical leverage above one no longer substitutes HC1. Additional unweighted BM contrast calculations return NaNs silently, avoiding duplicate covariance diagnostics.

• Exclude zero-weight rows from the HC2/HC3 guard and covariance contributions, preserving frequency-weight parity with literal replication.
• Require the versioned fail-closed HC2 capability for Rust dispatch. Older extensions use NumPy HC2 while retaining other Rust accelerations.
• Remove LWDiD's duplicate local HC2 covariance warning while retaining its independent influence-function safeguard.
• Document the policy and resolve the corresponding TODO.

Methodology references (required if estimator / math changes)

• Method name(s): HC2/HC3 heteroskedasticity-robust covariance; Bell–McCaffrey/Satterthwaite degrees of freedom; the existing weighted/clustered CR2 boundary.
• Paper / source link(s): MacKinnon and White (1985); Imbens and Kolesár, small-sample robust inference; Pustejovsky and Tipton, CR2. Repository policy is recorded in docs/methodology/REGISTRY.md and docs/methodology/variance-conventions.md.
• Intentional convention: the 1 - 1e-8 cutoff and suppression of the entire covariance/DOF vector are disclosed defensive choices, not a claim of identical reference behavior at every numerical boundary. Probability/analytical weights retain WLS leverage; frequency weights use each replicated row's leverage. Weighted or clustered HC2-BM retains its separate CR2 convention, including all-ones probability weights, so it can return finite covariance where an otherwise identical unweighted call returns NaN. Weighting is not a remedy for undefined inference. Healthy-design formulas remain unchanged.

Validation

Tests added/updated: tests/test_linalg_hc2_bm.py, tests/test_linalg.py, tests/test_rust_backend.py, tests/test_estimators_vcov_type.py, and tests/test_lwdid.py. Coverage includes inclusive cutoff and over-one cases, all-NaN inference with retained estimates, rank reduction, zero-weight deletion/replication parity, the all-ones-pweight CR2 boundary, native capability selection and fallbacks, and LWDiD warning/influence propagation.

The final affected-suite runs used the same 28 files covering linalg, estimators/TWFE, LWDiD, weighted CR2, replicate weights, checked-in reference goldens, and documentation/tracking checks. They include all 18 pre-merge-helper-resolved suites and ran sequentially in the provisioned .venv.

Local check	Result
Release extension rebuild: maturin develop --release --features accelerate --offline with .venv active	Passed before native acceptance tests
Affected pytest suites, DIFF_DIFF_BACKEND=rust	2,161 passed, 66 skipped, 41 deselected
Same suites, DIFF_DIFF_BACKEND=python	2,004 passed, 223 skipped, 41 deselected
Focused HC2 native/legacy compatibility tests	24 passed
cargo test --manifest-path rust/Cargo.toml --features accelerate --offline	57 passed; 0 doc tests
ruff check diff_diff tests; black --check diff_diff tests; cargo fmt --manifest-path rust/Cargo.toml --check; git diff --check	Passed
Changelog fragment validation; documentation dependency/existence checks	Passed
Pre-merge scanner	Exit 0; one pre-existing heuristic finding, disposition below
mypy diff_diff	Blocked before project checking by the frozen NumPy stubs

Mypy exits 2 at numpy/__init__.pyi:737: “Type statement is only supported in Python 3.12 and greater.” The frozen local environment is Python 3.14 with NumPy 2.5.2, while the repository targets Python 3.10 for mypy. This also reproduces on unchanged pinned source using mypy --no-incremental --python-version 3.10 -c "import numpy". The frozen lint workflow pins NumPy 2.4.5 with Python 3.14/mypy 2.3.1. The local environment and project type-checking configuration remain unchanged; this is not a passing type check.

Pre-merge Check D flags the existing _compute_confidence_interval definition at diff_diff/linalg.py:5526. The function and entire suffix are byte-identical to pinned main, and existing arithmetic propagates NaNs.

Backtest / simulation / notebook evidence: deterministic regression fixtures and checked-in reference goldens ran. Legacy-extension coverage simulates the prior successful flooring algorithm; no older compiled extension was loaded. No new backtest, live R/Stata reference run, notebook execution, or full Sphinx build was performed. The whole repository suite was not run; default not slow selection excluded 41 tests per mode, and skipped paths include unavailable optional reference dependencies and Python-mode native-only tests.

Security / privacy

• Confirm no secrets/PII in this PR: Yes. Canonical content and sensitive-filename scans found no matches across the complete 15-file candidate, including the new fragment; added fixtures use synthetic data.

Changelog

• Fragment added: changelog.d/20260905-hc2-leverage-one.md (behavioral change, zero-weight handling, and LWDiD warning deduplication).

[github-actions]

Overall assessment: ✅ Looks good

Executive summary

• No unmitigated P0/P1 findings.
• HC2/HC3 and unweighted HC2-BM fail closed consistently at leverage one.
• NaN covariance propagates safely through estimator inference.
• Zero-weight and frequency-weight behavior preserves replication parity.
• Behavioral changes have Registry documentation, tests, and a changelog fragment.

Methodology

• P3 — Documented defensive convention (diff_diff/linalg.py:L3632-L3659, docs/methodology/REGISTRY.md:L90)

  • Impact: The 1 - 1e-8 cutoff and design-wide NaN covariance/DOF suppression go beyond the papers’ algebraic definitions but prevent fabricated finite inference.
  • Concrete fix: None required; explicitly documented as a Registry Note.

• P3 — Documented CR2 boundary difference (diff_diff/linalg.py:L3596-L3623, docs/methodology/REGISTRY.md:L91)

  • Impact: Weighted/clustered CR2-BM, including all-ones probability weights, may remain finite when unweighted HC2-BM fails closed.
  • Concrete fix: None required; this distinct Pustejovsky–Tipton CR2 convention is explicitly documented.

Healthy-design HC2, HC3, Bell–McCaffrey/Satterthwaite, and CR2 formulas remain unchanged.

Code Quality

No findings. The guard is centralized, inference uses existing NaN-safe paths, and LWDiD’s duplicate warning was removed without removing its influence safeguard (diff_diff/lwdid.py:L2954-L2967).

Performance

• P3 — Legacy-extension fallback (diff_diff/_backend.py:L115-L126)
  • Impact: Older Rust extensions use NumPy for HC2, sacrificing only that acceleration while preserving correctness.
  • Concrete fix: None required; rebuilding/upgrading the extension restores native dispatch.

Maintainability

No findings. The Python and Rust cutoffs are parity-tested, and the versioned native capability prevents stale kernels from silently retaining the old behavior.

Tech Debt

No findings. The corresponding HC2 flooring item is appropriately removed from TODO.md because this PR resolves it.

Security

No findings; no secrets, unsafe input handling, or security-sensitive changes identified.

Documentation/Tests

No findings. The PR adds the required changelog fragment and covers inclusive cutoff behavior, over-one leverage, NaN inference propagation, rank reduction, zero-weight/frequency-weight parity, CR2 boundaries, LWDiD behavior, and legacy Rust fallback.

Focused tests could not be executed in the review environment because NumPy and pytest are unavailable.