9router Health Monitor & Auto-Optimizer

Automated health monitoring and optimization toolkit for 9router AI model routing gateway.

What is this?

9router is an AI model routing gateway that provides load balancing and failover across multiple AI providers (OpenAI, Anthropic, local models, etc.). This toolkit helps you:

• Monitor provider health - Detect when providers go down or run out of credits
• Auto-optimize routing - Automatically reorder combo priorities when providers fail
• Clean up stale configs - Remove broken providers and outdated combo entries
• Test connectivity - Verify provider endpoints and model availability
• Track performance - Measure actual request latency per provider

Why?

Managing multiple AI providers manually is tedious:

• Providers go down without warning
• Credits run out mid-conversation
• Stale combo entries cause routing delays
• Error states persist across restarts
• No visibility into which provider is actually being used

This toolkit automates the boring parts so you can focus on building.

Features

1. Health Check Script

Test all providers in a combo and identify failures:

python scripts/test-combo-latency.py udi-combo

Output:

=== Testing combo: udi-combo ===
kiro/claude-sonnet-4.5: ✓ OK (1.23s)
nvidia/nemotron-mini: ✗ FAILED (0.45s) - 404 page not found
ccode/claude-opus-4: ✗ FAILED (30.12s) - Invalid SSE response

2. Auto-Failover

Detect failed providers and reorder combo automatically:

python scripts/auto-failover.py udi-combo

Moves failing providers to the end of the combo so working providers are tried first.

3. Provider Diagnostics

Inspect provider status, error codes, and model locks:

python scripts/diagnose-providers.py

4. Cleanup Tools

Remove stale combo entries and disable broken providers:

python scripts/cleanup-stale-entries.py udi-combo

Installation

git clone https://github.com/Ubrewokan/9router-health-monitor.git
cd 9router-health-monitor
pip install -r requirements.txt

Quick Start

1. Test your combo:

   python scripts/test-combo-latency.py your-combo-name

2. Auto-fix failures:

   python scripts/auto-failover.py your-combo-name

3. Set up monitoring (optional):

   # Add to crontab to run every 10 minutes
   */10 * * * * cd /path/to/9router-health-monitor && python scripts/auto-failover.py udi-combo

Configuration

Default 9router database location: ~/.9router/db/data.sqlite

Override with environment variable:

export ROUTER_DB=/custom/path/to/data.sqlite

Common Issues & Solutions

Slow responses / lag

Symptom: Bot takes 10-30+ seconds to respond

Diagnosis:

python scripts/test-combo-latency.py your-combo

Fix:

1. Identify which provider is timing out
2. Move it to end of combo or disable it
3. Remove stale combo entries

Provider stuck in "unavailable" status

Cause: Error state persists in database

Fix:

python scripts/clear-provider-errors.py provider-name

SSE-only providers (streaming-only)

Symptom: "Invalid SSE response for non-streaming request"

Impact: Incompatible with agent workflows (which require non-streaming for tool calls)

Fix: Disable the provider - it cannot be used with agents

Project Background

This toolkit was built while troubleshooting 9router in production with Hermes Agent. Real issues we solved:

• ccode provider: SSE-only, causing 30s timeouts on every request
• NVIDIA NIM: Free tier models returning 404 despite being in API list
• Stale combo entries: Old provider prefixes causing routing delays
• Credit exhaustion: No automatic failover when provider runs out of credits

Documentation

• 9router Database Schema
• Provider Troubleshooting Guide
• SSE-only Provider Detection
• Auto-Failover Patterns

Requirements

• Python 3.8+
• 9router installed and running
• SQLite3 (included with Python)

Contributing

Found a bug? Have a feature request? Open an issue or PR.

License

MIT

Related Projects

• 9router - AI model routing gateway
• Hermes Agent - Autonomous AI agent framework

---

Built with real production experience - Every script in this repo solved an actual problem we hit while running 9router in production.