LLM Evaluator with DSPy

An evaluation framework for Large Language Model (LLM) responses using DSPy. For a comprehensive explanation of the concepts and methodology, please read our Medium article.

Overview

This framework provides a standardized approach to evaluate LLM-generated responses across four key metrics:

1. Relevancy: How well the answer aligns with the question
2. Correctness: Factual accuracy compared to ground truth
3. ROUGE: Text overlap with reference responses
4. Toxicity: Detection of inappropriate content

Results are standardized into a traffic light system (🟢 Green, 🟡 Yellow, 🔴 Red) for intuitive interpretation. For detailed explanations of these metrics and the evaluation methodology, refer to our Medium article.

Scoring behavior (important)

• Per-metric thresholds: Relevancy and correctness each use METRICS_THRESHOLD_RELEVANCY and METRICS_THRESHOLD_CORRECTNESS independently (with shared lower band METRICS_THRESHOLD_LOWER). ROUGE uses METRICS_THRESHOLD_ROUGE.
• Missing metric status: If a metric fails (for example the toxicity LLM call raises), that row records an errors entry and may lack that metric's status. When more than METRICS_MISSING_EVAL_THRESHOLD of rows have any missing *_status value, rows that would otherwise be overall green are downgraded to yellow so incomplete runs do not look fully passing.
• Toxicity errors: If the toxicity evaluator raises, the exception is recorded under errors for that row (it is not silently treated as non-toxic).

Installation

Install uv (Python 3.10+ required), then:

# Clone the repository
git clone https://github.com/yourusername/dspy-llm-evaluator.git
cd dspy-llm-evaluator

# Install dependencies (creates .venv from pyproject.toml / uv.lock)
uv sync --all-groups

--all-groups includes dev dependencies (pytest). For runtime only, use uv sync.

Run commands with uv run … (uses the project environment), or activate .venv and use python as usual.

Environment Variables

Copy the provided .env.example file to create your own .env file:

cp .env.example .env

Key environment variables:

Variable	Description	Default
OPENAI_API_KEY	Your OpenAI API key	None (required)
LLM_PROVIDER	The LLM provider to use	openai
MODEL_NAME	Judge model for DSPy metrics (gpt-4o recommended; gpt-4o-mini for budget smoke runs)	gpt-4o
METRICS_THRESHOLD_LOWER	Lower band for relevancy/correctness traffic lights (red below, yellow between)	0.4
METRICS_THRESHOLD_RELEVANCY	Upper band for relevancy (green at or above)	0.7
METRICS_THRESHOLD_CORRECTNESS	Upper band for correctness (green at or above)	0.7
METRICS_THRESHOLD_ROUGE	Upper band for ROUGE (green at or above; lower band is this minus 0.1)	0.5
METRICS_MISSING_EVAL_THRESHOLD	Max fraction of rows with any missing *_status before green rows with gaps downgrade to yellow	0.05
OUTPUT_DIR	Default path for evaluation results CSV	evaluation_results.csv
LOG_LEVEL	Logging level	INFO

For supported LLM providers, see DSPy docs.

Judge model: evaluations call an LLM as a scorer, not the model that produced the answers in your CSV. Use gpt-4o in .env for results you rely on (quality, thresholds, reports). Use gpt-4o-mini only for high-volume screening or the bundled real-API smoke script (scripts/smoke_real_eval.sh), which defaults to mini to limit cost. Compare runs using the same MODEL_NAME.

Usage

Basic Usage

uv run python main.py --data path/to/evaluation_data.csv --output results.csv

Command Line Arguments

• --data: Path to the evaluation data CSV (required)
• --output: Path to save evaluation results (default: evaluation_results.csv)
• --api_key: API key for the LLM service (can also be set via environment variable)
• --metrics: Comma-separated list of metrics to use (options: relevancy,correctness,rouge,toxicity or 'all')

Input Data Format

The input CSV should contain:

• question: The question or prompt given to the LLM
• response: The LLM's response to evaluate
• reference: The reference or ground truth answer (empty cells are treated as missing reference)

Example:

question,response,reference
"Who won the FIFA World Cup in 2014?","Germany won the FIFA World Cup in 2014 by defeating Argentina 1-0 in the final.","Germany won the FIFA World Cup in 2014 by defeating Argentina 1-0 in the final."

Output Example

Evaluating responses: 100%|███████████████████████████████████████████████| 11/11 [00:00<00:00, 89.32it/s]
Evaluation complete. Results saved to sample_result.csv

Evaluation Summary:
--------------------------------------------------
🎯 Relevancy: 0.55
✅ Correctness: 0.53
📝 Rouge: 0.41
🛡 Toxicity: 0.91

Overall Status Distribution:
🟢 green: 2 (18.2%)
🟡 yellow: 2 (18.2%)
🔴 red: 7 (63.6%)

Testing

uv sync --all-groups
uv run pytest -q

Real API smoke (optional)

After a dependency upgrade, confirm DSPy and your provider still work end-to-end (uses billable API calls). Set OPENAI_API_KEY in .env (or the environment). The smoke script defaults to gpt-4o-mini and example/smoke_cases.csv (six short rows: aligned answer, wrong fact, off-topic answer, numeric overlap, polite vs hostile tone for toxicity; cheap check; your main .env can still use gpt-4o for real runs):

./scripts/smoke_real_eval.sh

Override model or output path: MODEL_NAME=gpt-4o SMOKE_OUTPUT=./smoke_out.csv ./scripts/smoke_real_eval.sh (use gpt-4o here if you want the smoke to match production scoring)

example/smoke_cases.csv rows exercise:

• Aligned factual answer (expect strong relevancy/correctness/ROUGE vs reference)
• Same question with a wrong capital (correctness vs reference)
• Same photosynthesis question with an off-topic answer (relevancy)
• Numeric answer with matching wording (ROUGE overlap)
• Same greeting question with a professional response (baseline for toxicity contrast)
• Same greeting question with a hostile response (toxicity path vs polite reference)

Utility Script

The project includes a utility script for post-processing evaluation results:

uv run python scripts/llm_eval_utils.py <command> [arguments]

Available commands:

• check-quality: Validates if results meet quality thresholds (writes GitLab metrics report; use --metrics-report to set the output path, default metrics_report.txt)
• generate-trends: Creates trend reports from historical data
• compare-models: Compares results from different models
• generate-report: Generates HTML reports
• check-deployment: Checks if results meet deployment criteria

Examples:

# Generate HTML report
uv run python scripts/llm_eval_utils.py generate-report --results evaluation_results.csv --output report.html

# Quality gate with custom metrics report path
uv run python scripts/llm_eval_utils.py check-quality --results evaluation_results.csv --min-green 70 --metrics-report artifacts/metrics_report.txt

Integration with CI/CD

This evaluator can be integrated into CI/CD pipelines to ensure consistent performance of LLM assistants. See the GitLab Integration Guide for details on:

• Setting up GitLab CI/CD pipelines for automated evaluations
• Configuring quality thresholds for pipeline success/failure
• Tracking evaluation metrics over time
• Comparing different model versions
• Generating reports and visualizations

Architecture

The application follows a modular design for extensibility and maintainability:

Core Components

1. Metrics System

   • Abstract Metric and DSPyMetric base classes
   • Individual implementations for each metric type

2. DSPy Integration

   • Leverages DSPy for consistent LLM-based evaluation
   • Custom DSPy signatures and programs for evaluation

3. Scoring System

   • TrafficLightScorer standardizes scores (green/yellow/red)
   • Configurable thresholds for evaluation strictness

4. Evaluation Pipeline

   • Orchestrates end-to-end evaluation process
   • Applies each configured metric in sequence and aggregates results

Extensibility

To add new metrics:

1. Create a new class inheriting from Metric or DSPyMetric
2. Implement the required evaluate() method
3. Register the new metric in __init__.py

For more details on the conceptual framework and methodology, please refer to our Medium article or the original "LLM Evaluator: what AI Scientist must know" article.