KB Arena: open-source benchmark for GraphRAG vs vector vs hybrid retrieval with statistical rigor

[xmpuspus]

Is your feature request related to a problem? Please describe.

For teams evaluating llm-graph-builder against vector-only RAG or hybrid approaches, there is no clean, open benchmark today that:

1. Runs graph-based retrieval head-to-head against vector and hybrid on the same user-supplied corpus
2. Reports IR metrics (Recall@k, NDCG, MAP, bpref) with paired-bootstrap 95% CIs and Wilcoxon paired two-sided p-values — not mean-only deltas
3. Treats the Neo4j-backed knowledge graph as a first-class retrieval strategy with universal schema rather than a domain-specific demo

Without this, GraphRAG advocacy lives in blog posts with single-corpus anecdotes, and skeptical teams can't justify the operational cost of standing up Neo4j next to their vector store.

Describe the solution you'd like

Reference KB Arena (MIT, PyPI kb-arena, Zenodo concept DOI 10.5281/zenodo.20319678) as a complementary evaluation tool in the README — for example a short ## Related Projects section, or a line in the existing intro pointing users who want to benchmark extraction-graph retrieval against alternatives.

KB Arena ships nine retrieval strategies, three of which are directly relevant to llm-graph-builder users:

• knowledge_graph — extraction-driven Neo4j graph with a universal 5-node-type / 7-rel-type schema (Topic, Component, Process, Config, Constraint + DEPENDS_ON, CONTAINS, CONNECTS_TO, TRIGGERS, CONFIGURES, ALTERNATIVE_TO, EXTENDS). Source provenance (source_doc_id, source_section_id) is stamped on every entity end-to-end so chunk-level retrieval matches against section ground truth.
• hybrid — RRF-fused vector + graph retrieval with three-stage intent routing (keyword scan → Haiku LLM → regex fallback). Domain-agnostic.
• naive_vector — baseline so the graph delta is honest.

It also implements RAPTOR, PageIndex, BM25, QnA-pairs, contextual vector, and rerank-vector for the broader comparison.

Describe alternatives you've considered

• BEIR / MTEB: optimize embedding-model evaluation, not graph-vs-vector architectural comparison
• Microsoft GraphRAG benchmarking-datasets: focused on Microsoft's own GraphRAG implementation
• Ad-hoc internal benchmarks: not reproducible across teams

Additional context

• Ships an aws-compute corpus (75 questions across 5 difficulty tiers, 35 with chunk-level ground truth) as a pedagogical baseline; users plug in their own docs via CLI
• v0.8.1 released 2026-05-21, 617 tests passing
• Repo: https://github.com/xmpuspus/kb-arena
• Author here, happy to open a small README PR with a single ## Related Projects bullet if that's the format the maintainers prefer.