138 lines
4.6 KiB
Markdown
138 lines
4.6 KiB
Markdown
# Model Routing Policy — 2026-07-14
|
|
|
|
## User-confirmed routing shape
|
|
|
|
Production model routing must follow this provider order:
|
|
|
|
```text
|
|
Primary → admin-ai
|
|
Fallback 1 → admin-ai
|
|
Fallback 2 → OpenRouter direct
|
|
Fallback 3 → independent direct frontier-capable provider
|
|
```
|
|
|
|
At Fallback 3, the priority is **uptime/access to Sho'Nuff and model competence**, not cost optimization. The model must be capable enough for serious ops/debugging/infrastructure work, not merely available.
|
|
|
|
## Concrete target route from this session
|
|
|
|
```text
|
|
Primary:
|
|
deepseek-v4-pro via admin-ai
|
|
|
|
Fallback 1:
|
|
deepseek-chat via admin-ai
|
|
|
|
Fallback 2:
|
|
openrouter/qwen/qwen3.6-plus via OpenRouter direct
|
|
|
|
Fallback 3:
|
|
xai/grok-4.3 via xAI direct
|
|
```
|
|
|
|
This satisfies:
|
|
|
|
- admin-ai primary control and LiteLLM logging
|
|
- same-router model fallback
|
|
- independent OpenRouter fallback
|
|
- independent direct frontier-capable fallback
|
|
- no Ollama/Gemini in normal text routing
|
|
|
|
## Fallback 3 provider pool
|
|
|
|
Germaine stated keys are available for:
|
|
|
|
- OpenAI
|
|
- Perplexity
|
|
- Groq
|
|
- OpenRouter
|
|
- DeepSeek
|
|
- xAI
|
|
- Mistral
|
|
- Fireworks AI
|
|
- Anthropic
|
|
- Gemini
|
|
- Cohere
|
|
- AI21
|
|
- Deepgram
|
|
|
|
For Fallback 3:
|
|
|
|
- Exclude OpenRouter because it is already Fallback 2.
|
|
- Exclude Deepgram from LLM routing; it is speech/audio, not a general chat brain.
|
|
- Exclude Gemini from normal text routing unless Germaine explicitly re-approves it.
|
|
- Prefer frontier-capable direct providers over cheap/fast small models.
|
|
|
|
Recommended direct-provider selection order, subject to live completion and quota tests:
|
|
|
|
1. OpenAI direct GPT-5.5/GPT-5-class, if quota/billing healthy
|
|
2. Anthropic direct Claude Opus/Sonnet-class, if usage limit healthy
|
|
3. xAI direct high-end Grok
|
|
4. Mistral direct top model, if configured and tested
|
|
5. Fireworks AI high-end hosted model, if selected and tested
|
|
6. Groq direct only if the selected model is good enough for Sho'Nuff ops
|
|
7. DeepSeek direct only if frontier providers are unavailable and uptime is at risk
|
|
8. Cohere / AI21 only after explicit quality testing
|
|
|
|
## Direct-vs-aggregator distinction
|
|
|
|
Model IDs prefixed with `openrouter/` are OpenRouter routes even if the model family is OpenAI, Anthropic, Qwen, etc. Examples:
|
|
|
|
```text
|
|
openrouter/openai/gpt-5.2-pro → OpenRouter route
|
|
openrouter/anthropic/claude-opus-4.7 → OpenRouter route
|
|
openrouter/qwen/qwen3.6-plus → OpenRouter route
|
|
```
|
|
|
|
A true direct provider fallback should use that provider's direct route, e.g. `xai/grok-4.3` for xAI direct.
|
|
|
|
## Verified candidates from this session
|
|
|
|
Working via current app1 admin-ai endpoint:
|
|
|
|
```text
|
|
deepseek-v4-pro OK
|
|
deepseek-chat OK
|
|
xai/grok-4.3 OK
|
|
xai/grok-4-1-fast OK
|
|
openrouter/qwen/qwen3.6-plus OK via OpenRouter route
|
|
openrouter/openai/gpt-5.2-pro OK via OpenRouter route
|
|
openrouter/anthropic/claude-opus-4.7 OK via OpenRouter route
|
|
```
|
|
|
|
Failing/limited during this session:
|
|
|
|
```text
|
|
gpt-5.5 direct/admin-ai OpenAI insufficient_quota
|
|
claude-opus/sonnet direct Anthropic usage limit until 2026-08-01
|
|
mistral direct names tested no healthy LiteLLM deployment configured
|
|
qwen/qwen3-32b direct name no healthy LiteLLM deployment configured
|
|
```
|
|
|
|
## Explicit exclusions from automatic routing
|
|
|
|
- Ollama / `llama3.2:3b`: too weak for Sho'Nuff operations; do not use as automatic fallback or LLM cron model.
|
|
- Gemini: removed from normal model-routing plan; do not reintroduce as normal fallback just because it tests cleanly. It may remain available for vision/manual use if approved.
|
|
- GPT/Claude: escalation/frontier candidates only when quotas allow; do not make them hidden uptime dependencies while quota-limited.
|
|
|
|
## Health check requirements
|
|
|
|
Do not trust `/v1/models` alone. A route is healthy only if a real `/v1/chat/completions` call returns usable content.
|
|
|
|
Check each production tier separately:
|
|
|
|
- admin-ai primary model completion
|
|
- admin-ai fallback model completion
|
|
- OpenRouter direct fallback completion
|
|
- selected Fallback 3 direct-provider completion
|
|
|
|
Use enough `max_tokens` for reasoning models; too-small budgets can produce blank/length-limited responses and false negatives.
|
|
|
|
GPT/Claude quota failures should be reported as escalation-model unavailable, not as total admin-ai failure.
|
|
|
|
## Background jobs
|
|
|
|
- Script-only jobs should stay `no_agent=true` where possible.
|
|
- No cron/background job should be pinned to Ollama.
|
|
- Jobs using admin-ai must use admin-ai model IDs, e.g. `deepseek-chat`, not OpenRouter model IDs like `deepseek/deepseek-chat`.
|
|
- Pinned cron jobs do not inherit the conversation fallback chain; audit and pin them explicitly.
|