Feat/add agentic rag workshop - #32
Merged
Merged
Conversation
c7d2f25 docs(agent-workshop): simplify agentic RAG deck 6027cad docs(ppt): disambiguate segment numbering and refine slide layouts 4eff50f docs(ppt): add introduction slide 518a24c docs: align flow.md and readme.md with the current specs e6bdc27 fix(workflow): declare the single-strong-chunk threshold per reranker 3bc3fad docs(specs): correct the F17 diagnosis and split out the dead threshold 91c7de3 docs(specs): route model-only verification out of the golden set (D53) 6e82824 docs(specs): close the reranker half of F15 and state why compression stops 5ae0e43 feat(eval): let a fixture declare the degraded reranker path ac38ce6 docs(specs): close F12 as R28/R29 and re-scope the remaining fixtures 2b69cf6 test(eval): add the cache-hit and Memory state-change golden cases f131841 feat(eval): add a same-session prelude to the strict scenario schema 22f07d0 docs(specs): narrow F15 to the scenario-blocked cases and record two findings 56fe0c3 test(eval): add the golden cases the corpus can already support 808e2db test(eval): pin the transformation strategy each golden case executes a1e5b6e docs(specs): close F16 as R27 and narrow the remaining fixture gap c5e841d feat(eval): let golden fixtures pin transformation strategy and compression path f89aec0 docs(specs): record the spec-versus-implementation review findings 0e4be02 fix(eval): grade the forbidden stage path and compression provenance 973eeb9 fix(generation): reject unregistered answer-generator fallback reasons 250386a fix(eval): grade the exhaustive generation-context bound, not only the focused one f43a98c docs(specs): stop review-backlog ids colliding with decision ids 610ab2e docs(specs): drop stale image, tab-count and tier-cost claims 181eafc docs(specs): repair misplaced and duplicated data-model sections 5e65b8f docs(specs): align the workflow spec with the implemented transitions 7f74807 feat(retrieval): implement the lexical tiers and the tier comparison eval 00bce2a docs(specs): add the retrieval tier ladder and its cost model 0556ec4 fix(schema): report an actionable error for an unsupported pymilvus 3e31611 fix(retrieval): reject fused lane passage identity disagreement 4e441f0 perf(workflow): skip chunk vectors when serializing the terminal snapshot 919daa3 fix(workflow): include retrieval-path provenance in the candidate fingerprint 30b29a7 fix(workflow): gate evidence sufficiency on relevant chunk scores df15e2e chore(demo): repair the test-isolation and strict type gates 040e469 phase 9 review: harden StructArray retrieval 47ae335 phase 9: add StructArray document retrieval 5fa2bcc feat(eval): add maintainable metric portfolio 193a268 feat(agent-workshop): implement rag optimization methods c91ed94 chore(skills): add human-writing skill e2cc026 fix(milvus): align schema and query types 894f671 fix(eval): keep baseline portable and harden tool-scope grader b165918 phase 5: add layered evaluation gates a938263 chore: add claude md file 2a6c8d4 feat(skills): add agent-eval skill for eval-driven development a5289e1 feat(skills): port research/spec/impl skills to Claude Code format git-subtree-dir: agent git-subtree-split: c7d2f254bf81f6577a9e796bf04e9ff22e1b729a
…-agentic-rag-workshop
There was a problem hiding this comment.
Pull request overview
This PR expands the Milvus “Agentic RAG” workshop by adding retrieval-tier governance (T0/T1/T2), StructArray document/passage projection + evaluation, bounded query transformation, and provenance-preserving generation-context compression, while updating the spec set and deterministic eval harness accordingly.
Changes:
- Add executable retrieval-tier ladder (lexical baselines + hybrid default) and a comparative tier evaluation script.
- Introduce StructArray (
kb_documents) projection schema, gated activation, offline/native profile eval, and UI trace/evidence surfacing for routing vs citeable evidence. - Add bounded query transformation and optional provenance-safe context compression, and tighten eval governance (metric registry + error-analysis artifacts).
Reviewed changes
Copilot reviewed 104 out of 114 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| agent/specs/index.md | Updates spec catalog, reading order, and runtime boundary diagram to include retrieval tiers, StructArray, and compression. |
| agent/specs/90-roadmap.md | Revises milestones/exit criteria to reflect query transformation, compression, StructArray lab, and metric governance. |
| agent/specs/80-glossary.md | Adds definitions for transformation, step-back, compression, metric registry, and retrieval-tier ladder. |
| agent/specs/20-ui-demo.md | Updates UI requirements to display transformation/compression provenance and StructArray routing vs evidence. |
| agent/specs/13-llm-answer-generation.md | Extends generation contract to support provenance-safe projections and StructArray evidence constraints. |
| agent/specs/11-ingestion.md | Adds StructArray projection assembly rules, chunking-eval governance, and failure policy updates. |
| agent/specs/10d-selective-agent-memory.md | Clarifies scoring-band implications and renumbers a section header. |
| agent/specs/10c-grounded-response-cache.md | Adds ResponseCacheCandidate wrapper and tightens wording about storage immutability. |
| agent/specs/10b-conversation-memory.md | Updates cross-reference anchor to renamed data-model section. |
| agent/specs/10a-openai-text-embedding.md | Clarifies image embeddings are out-of-scope and adds migration/fingerprint-gate decision reference. |
| agent/mypy.ini | Adds strict mypy configuration scoped to demo/src. |
| agent/flow.md | Updates workflow diagram to include retrieval tier selection and prepare_generation_context. |
| agent/docs/research/milvus-3-structarray-probe.md | Adds a Phase probe memo documenting StructArray behaviors and constraints. |
| agent/demo/tests/test_transitions.py | Updates transition expectations for the new prepare_generation_context node. |
| agent/demo/tests/test_struct_array_eval.py | Adds offline StructArray profile evaluation tests and strict report validation. |
| agent/demo/tests/test_server_workflow.py | Adds embedding-space startup gate tests and StructArray activation wiring tests. |
| agent/demo/tests/test_selective_memory.py | Records query calls and adds TIMESTAMPTZ literal filter assertion. |
| agent/demo/tests/test_schema.py | Extends schema/index tests for kb_documents and enforces TIMESTAMPTZ index constraints. |
| agent/demo/tests/test_runtime_transition_parity.py | Adds parity test coverage for enabled context compression and citation stability. |
| agent/demo/tests/test_query_transform.py | Adds rule-based/OpenAI transformer tests enforcing bounded strategies and protected terms. |
| agent/demo/tests/test_milvus3_capabilities.py | Adds analyzer param constant use, TIMESTAMPTZ literal test, and boolean facet aggregation behavior. |
| agent/demo/tests/test_memory_cleanup.py | Updates expected TIMESTAMPTZ literal quoting for cleanup filters. |
| agent/demo/tests/test_ingestion_eval_memory.py | Updates eval runner usage for scenarios, registry-driven reporting, and v3 report assertions. |
| agent/demo/tests/test_generation.py | Adds projection grounding tests and validates fallback reason codes against a closed set. |
| agent/demo/tests/test_eval_governance.py | Adds strict registry + error-analysis artifact governance tests and CLI output constraints. |
| agent/demo/tests/test_chunking_experiment.py | Updates chunking experiment to v2 (3 configs), adds calibrated-grader + reviewed artifact checks. |
| agent/demo/tests/test_agentic_tools.py | Updates comparison behavior expectations for parallel tool plans and retry count. |
| agent/demo/tests/init.py | Pins deterministic offline test profile by stripping provider selectors/credentials before imports. |
| agent/demo/src/agent_workshop_demo/transitions.py | Adds PREPARE_GENERATION_CONTEXT node and transition reason EVIDENCE_READY_FOR_CONTEXT. |
| agent/demo/src/agent_workshop_demo/streamlit_app.py | Adds UI surfacing for profile/granularity/offset and document shortlist routing output. |
| agent/demo/src/agent_workshop_demo/selective_memory.py | Casts decay math to float for strict typing. |
| agent/demo/src/agent_workshop_demo/selective_memory_eval.py | Renames intermediate dict for clarity and indexes scenarios safely. |
| agent/demo/src/agent_workshop_demo/schema/evolution.py | Centralizes retrieval analyzer params constant usage. |
| agent/demo/src/agent_workshop_demo/schema/collections.py | Adds kb_documents StructArray schema + indexes; centralizes analyzer params; updates TIMESTAMPTZ scalar indexes. |
| agent/demo/src/agent_workshop_demo/schema/init.py | Re-exports KB_DOCUMENTS_COLLECTION. |
| agent/demo/src/agent_workshop_demo/retrieval.py | Adds deterministic sparse-only lane (search_sparse) and formatting cleanups. |
| agent/demo/src/agent_workshop_demo/retrieval_tier.py | Introduces retrieval tier config parsing + lexical-only retriever wrapper. |
| agent/demo/src/agent_workshop_demo/response_cache.py | Adds fusion_recipe to cached evidence and tightens validation/formatting. |
| agent/demo/src/agent_workshop_demo/reranker.py | Adds per-reranker strong_single_evidence_threshold to avoid cross-scale comparisons. |
| agent/demo/src/agent_workshop_demo/models.py | Extends SearchResult with retrieval provenance fields and adds state fields for new workflow outputs. |
| agent/demo/src/agent_workshop_demo/milvus_time.py | Switches TIMESTAMPTZ predicates to ISO '...' literal form. |
| agent/demo/src/agent_workshop_demo/langgraph_workflow.py | Wires query transformer, context compressor, retrieval tier config, and StructArray retriever activation. |
| agent/demo/src/agent_workshop_demo/generation.py | Adds provenance-safe projection handling, closed fallback-reason validation, and derived-text exclusion. |
| agent/demo/src/agent_workshop_demo/events.py | Extends allowed event keys to cover transformation/compression/StructArray provenance. |
| agent/demo/src/agent_workshop_demo/embedding.py | Generalizes cosine similarity inputs to Sequence[float] for typing flexibility. |
| agent/demo/src/agent_workshop_demo/config.py | Adds env-file skip flag and shared exhaustive-context cap constant. |
| agent/demo/src/agent_workshop_demo/classification.py | Minor variable rename for clarity in Chinese-count parsing. |
| agent/demo/scripts/run_tier_eval.py | Adds retrieval-tier comparative eval entrypoint (T0/T1/T2 arms). |
| agent/demo/scripts/run_struct_array_eval.py | Adds isolated StructArray profile comparison script (offline + optional native read-only). |
| agent/demo/scripts/run_chunking_experiment.py | Adds optional reviewed recommendation artifact wiring to chunking experiment runner. |
| agent/demo/scripts/ingest_demo.py | Adds StructArray projection build + gated activation hook after chunk ingestion. |
| agent/demo/scripts/create_indexes.py | Adds typing for dry-run index definitions (but currently omits kb_documents definitions). |
| agent/demo/scripts/cleanup_milvus.py | Updates messaging to reflect the expanded set of demo collections. |
| agent/demo/scripts/init.py | Adds package marker docstring for type resolution. |
| agent/demo/README.md | Documents StructArray workflow, query transformation, context compression, and updated eval governance. |
| agent/demo/notebooks/04_milvus_hybrid_search.ipynb | Updates notebook to demonstrate flat vs StructArray retrieval and routing vs evidence. |
| agent/demo/notebooks/03_milvus_schema_and_insert.ipynb | Updates notebook to inspect both kb_chunks and derived kb_documents schemas/projection. |
| agent/demo/eval/struct_array_cases.json | Adds strict offline StructArray evaluation fixture cases. |
| agent/demo/eval/rag_eval_review.json | Adds transcript review fixture for error-analysis attribution. |
| agent/demo/eval/questions.json | Expands deterministic question fixture with scenarios, expected transformations, and terminal statuses. |
| agent/demo/eval/golden_answers.yaml | Expands golden answers to cover new scenarios (permission denial, step-back, cache/memory, reranker fallback). |
| agent/demo/eval/chunking_configs.json | Updates chunking configs to v2 with three small/medium/large profiles. |
| agent/demo/eval/chunking_anchors.json | Updates anchors to v2 and adds abstention fixture case. |
| agent/demo/config/struct_array_projection.json | Adds strict StructArray projection manifest selecting long/multi-passage docs. |
| agent/demo/.env.example | Adds env knobs for query transformation, compression, retrieval tier, and StructArray gating. |
| agent/CLAUDE.md | Adds deterministic commands, tier eval command, and expanded architecture gotchas. |
| agent/AGENTS.md | Updates deterministic unittest discovery invocation and adds tier eval command. |
| agent/.gitignore | Ignores local Claude settings file. |
| agent/.claude/skills/spec/references/templates.md | Adds spec templates reference material. |
| agent/.claude/skills/research/SKILL.md | Adds research skill definition and memo templates. |
| agent/.claude/skills/agent-eval/SKILL.md | Adds agent-eval skill definition aligned to the repo’s eval harness. |
| agent/.claude/skills/agent-eval/references/judge-calibration.md | Adds judge rubric/calibration guidance reference. |
| agent/.agents/skills/human-writing/VERSION | Adds human-writing skill version marker. |
| agent/.agents/skills/human-writing/references/reality.md | Adds human-writing skill reference: reality fact-checking. |
| agent/.agents/skills/human-writing/references/formats.md | Adds human-writing skill reference: format adaptations. |
| agent/.agents/skills/human-writing/references/fiction.md | Adds human-writing skill reference: fiction guidance. |
| agent/.agents/skills/human-writing/LICENSE | Adds license for the human-writing skill bundle. |
| agent/.agents/skills/human-writing/dist/human-writing-lite.md | Adds distilled prompt content for the human-writing skill. |
| agent/.agents/skills/human-writing/agents/openai.yaml | Adds agent interface metadata for the human-writing skill. |
Suppressed comments (1)
agent/demo/scripts/create_indexes.py:71
create_indexes.py --dry-runprints a hard-coded index-definition dict that now omits the newkb_documentsStructArray projection indexes, so the dry-run output is incomplete/misleading relative to whatcreate_indexes()actually manages. Includekb_documentshere so operators can see the full planned index set.
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+35
to
+42
| DETERMINISTIC_PROVIDERS = { | ||
| "EMBEDDING_PROVIDER": "deterministic", | ||
| "QUERY_CLASSIFIER": "rule_based", | ||
| "RERANKER": "rule_based", | ||
| "ANSWER_GENERATOR": "deterministic", | ||
| "MEMORY_SELECTOR": "rule_based", | ||
| "QUERY_TRANSFORMER": "rule_based", | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.