healthbench

Advanced RAG pipelines for medical (HealthBench, MedCaseReasoning, MetaMedQA, PubMedQA) and financial (FinanceBench, Earnings Calls) QA. LangGraph orchestration + BAML structructed generation, Milvus Hybrid search (Dense + BM25 + RRF), three-layer Metadata Enrichment, Contextual AI instruction-following reranker, and DeepEval evaluation.

Single-file, zero-dependency local LLM benchmark suite + live dashboard. Deterministic scoring (Wilson CIs, paired McNemar A/B), per-question drill-down & live system metrics — for any OpenAI-compatible server (llama.cpp/vLLM/Ollama/LM Studio/ds4). Broad + medical benchmark panel.

This repository contains code and resources related to an in-depth analysis of OpenAI's HealthBench, a benchmark designed for evaluating Large Language Models in the healthcare sector.

Open-source medical LLM safety evaluation pipeline with reproducible benchmarks and high-risk clinical failure analysis.

Accuracy is not readiness: an open-source robustness stress-test of frontier LLMs (Opus 4.8, GPT-5.5, Grok 4.3, Gemini 3.5 Flash) on medical QA. Re-implements & extends Gu et al. 'The Illusion of Readiness in Health AI'.

Auditing the clinical-evidence claims in OpenAI's HealthBench — its own gold answers & rubrics — for hallucinated, overgeneralized, overlooked & misweighted evidence. By NoBSmed.

Safety before certainty: an open-source public-data stress-test showing a structured evidence-sufficiency prompt cuts unsafe overconfident clinical answers across four frontier LLMs (Opus 4.8, GPT-5.5, Gemini 3.5 Flash, Grok 4.3), with matched circularity controls and blinded clinician judge-validation.

[EE6406]ANALYTIC & ENSEMBLE LEARNING Project:End-to-end toolkit for medical SFT dataset preparation, Gemma 3 4B IT fine-tuning (LoRA/DoRA/QLoRA), vLLM inference, and HealthBench evaluation.

A clinical decision-evidence benchmark: pull the one fact an answer rests on, and see whether the assistant's action moves with the evidence. Built on HealthBench.