ippadmin/itpp-infrastructure
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: fallback chain overhaul, two-key strategy, operational model update (Aug 6 2026)

Browse Source
This commit is contained in:
root
2026-08-06 10:59:07 -04:00
parent 8561ba98c8
commit 25e6b8cb79
2 changed files with 67 additions and 36 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+12
View File
@@ -1,5 +1,17 @@
# itpp-infrastructure — CHANGELOG # itpp-infrastructure — CHANGELOG
## 2026-08-06 — Fallback Chain Overhaul & Two-Key Strategy
- **Root cause:** Aug 5 admin-ai budget cap + 4 dead fallback legs = $45 Anthropic burn in 10 hours
- Rotated all 5 fallback provider keys (new keys for deepseek, google, xai, anthropic, openai)
- Added F5: gpt-4.1-nano via OpenAI direct (independent infrastructure)
- Fixed F3: grok-4.6 → grok-4.5 (grok-4.6 never existed — LiteLLM catalog ghost)
- Documented two-key strategy: operational keys (admin-ai only) vs fallback keys (direct, daily-capped)
- Added to operational chain: claude-haiku-4-5 (lightweight), grok-4.5 (auditor 2), deepseek-v4-flash (batch)
- Synced Anita profile with identical fallback chain + provider keys
- Admin-ai budget raised: $20 → $30/day
- Updated: model-chain.md, operational-models.md
## 2026-07-16 — Audit Remediation ## 2026-07-16 — Audit Remediation
- Created CHANGELOG.md (missing per project documentation standard) - Created CHANGELOG.md (missing per project documentation standard)
docs/model-chain.md
+55 -36
View File
@@ -1,60 +1,79 @@
# AI Model Architecture — IT Pro Partner # AI Model Architecture — IT Pro Partner
**Updated:** July 31, 2026 **Updated:** August 6, 2026
Two separate concepts that must not be conflated: Two separate concepts: **fallback chain** (survival — direct API keys) and **operational chain** (daily toolbox — admin-ai only). The two-key strategy means operational keys run through admin-ai/LiteLLM; fallback keys are direct provider API keys with daily limits.
--- ---
## Fallback Chain *(auto-failover — config.yaml)* ## Fallback Chain *(auto-failover — direct API keys)*
Survives admin-ai outage. Uses direct API keys for F1–F4. Survives admin-ai outage. All direct provider keys have daily caps. Fires in order — only when the model above is unreachable.
| Tier | Model | Provider | Notes | | Tier | Model | Provider | Key type | Daily cap |
|---|---|---|---|---|
| **Primary** | `deepseek-v4-pro` | `admin-ai` | operational | $30/mo budget |
| **F1** | `deepseek-v4-flash` | `deepseek` (direct) | fallback | $3 |
| **F2** | `gemini-3.6-flash` | `google` (direct) | fallback | $2 |
| **F3** | `grok-4.5` | `xai` (direct) | fallback | $2 |
| **F4** | `claude-sonnet-5` | `anthropic` (direct) | fallback | $5 |
| **F5** | `gpt-4.1-nano` | `openai` (direct) | fallback | $2 |
Total emergency budget: **$14/day** — down from $45 single-leg burn on Aug 5.
---
## Operational Models *(daily toolbox — admin-ai only)*
All route through admin-ai. Shared budget via `hermes-agent-v5` key.
| Role | Model | Provider | Use When |
|---|---|---|---| |---|---|---|---|
| **Primary** | `deepseek-v4-pro` | `admin-ai` | Conductor & workhorse via LiteLLM v5 key | | **Conductor** | `deepseek-v4-pro` | admin-ai (DeepSeek) | All standard work — orchestration, delegation, coding |
| **F1** | `deepseek-v4-flash` | `deepseek` (direct) | Flash demoted -- fallback only, was hallucinating | | **Workhorse** | `deepseek-v4-pro` | admin-ai (DeepSeek) | Delegated tasks, scripts, infra code |
| **F2** | `gemini-3.6-flash` | `google` (direct) | Fast, free-tier eligible | | **Batch Workhorse** | `deepseek-v4-flash` | admin-ai (DeepSeek) | Bulk scripts, log parsing, repetitive tasks |
| **F3** | `grok-4.6` | `xai` (direct) | Verified working (NOT grok-4.5) | | **Lightweight** | `claude-haiku-4-5` | admin-ai (Anthropic) | Email triage, classification, simple tasks |
| **F4** | `claude-sonnet-5` | `anthropic` (direct) | Premium survival fallback | | **Simple Workhorse** | `gpt-5.6-luna` | admin-ai (OpenAI) | Lightweight tasks under 128K context |
| **Auditor** | `gpt-5.6-luna` | admin-ai (OpenAI) | Code review, QA (primary auditor) |
**Only fires when the model above it is unreachable.** | **Auditor 2** | `xai/grok-4.5` | admin-ai (xAI) | Second-opinion code review (different provider) |
| **Critical** | `claude-sonnet-5` | admin-ai (Anthropic) | Client comms, legal docs, architecture (explicit) |
| **Professional Comms** | `gemini-3.6-flash` | admin-ai (Google) | Client emails, professional messaging |
--- ---
## Operational Models *(daily toolbox -- admin-ai only)* ## Admin-AI Virtual Keys
All route through admin-ai. Shared $20/day budget via hermes-agent-v3 key. ### hermes-agent-v5 (Main — Sho'Nuff)
- **Created:** Jul 31, 2026
- **Budget:** $30/day
- **Spend:** $20.36 (as of Aug 6)
- **Models (7):** deepseek-v4-pro, deepseek-v4-flash, gemini-3.6-flash, claude-sonnet-5, claude-haiku-4-5, gpt-5.6-luna, xai/grok-4.5
| Role | Model | When | ### Anita's Hermes Key
|---|---|---| - **Budget:** $10/day
| **Conductor** | `deepseek-v4-pro` | ALL standard work -- orchestrating, delegating, infra, coding | - **Spend:** $0.11 (as of Aug 6)
| **Workhorse** | `deepseek-v4-pro` | Same model -- large context, big outputs, infra code | - **Models (7):** deepseek-v4-pro, deepseek-v4-flash, gemini-flash-latest, claude-sonnet-5, claude-haiku-4-5, gpt-5.6-luna, xai/grok-4.5
| **Workhorse** | `deepseek-reasoner` | Multi-step analysis, debugging, planning |
| **Auditor** | `gemini-3.6-flash` | Sanity checks, every output (essentially free) |
| **Auditor** | `claude-sonnet-5` | Architecture, security, auth -- sparingly ($15/M output) |
--- ---
## Admin-AI Virtual Key
- **Key Name:** `hermes-agent-v5` (regenerated Jul 31, 2026)
- **Daily Budget Cap:** $20/day shared across all operational models
- **Expiry:** Jul 31, 2027
- **Master Key:** stored in Hudu
- **Models:** deepseek-v4-pro, deepseek-v4-flash, deepseek-reasoner, gemini-3.6-flash, claude-sonnet-5, xai/grok-4.3, xai/grok-4-1-fast, grok-4.6
## Budget Targets ## Budget Targets
| Component | Daily est. | | Component | Daily est. |
|---|---| |---|---|
| Conductor + Workhorse (ds-v4-pro) | ~$2.50 | | Conductor + Workhorse (ds-v4-pro) | ~$3.00 |
| Workhorse (ds-reasoner) | ~$0.70 | | Batch Workhorse (ds-v4-flash) | ~$0.50 |
| Auditors (gemini flash + claude sparingly) | ~$0.80 | | Lightweight (haiku-4-5) | ~$0.30 |
| **Total** | **~$4/day** (well under $20 cap) | | Simple Workhorse (luna) | ~$0.50 |
| Auditors (luna + grok-4.5) | ~$0.80 |
| Critical (sonnet-5, sparingly) | ~$2.00 |
| **Total** | **~$7/day** |
---
## Key Decisions ## Key Decisions
- **Jul 29, 2026:** DeepSeek V4 Flash demoted from conductor to F1 fallback. Was hallucinating tool calls and producing inconsistent results. V4 Pro promoted to primary conductor and workhorse. - **Jul 29, 2026:** DeepSeek V4 Flash demoted from conductor to F1 fallback. Was hallucinating tool calls. V4 Pro promoted to primary conductor.
- **Jul 30, 2026:** LiteLLM key regenerated — hermes-agent-v3 created in DB. However, the config.yaml was NOT updated with the new key (doc was written but config file not patched). Hermes silently fell back to DeepSeek direct since Jul 30 19:25 UTC. - **Jul 30, 2026:** LiteLLM key regenerated — hermes-agent-v3 created but config.yaml NOT updated. Hermes fell back to DeepSeek direct since Jul 30 19:25 UTC.
- **Jul 31, 2026:** Discovered config.yaml still had stale key (`sk-DuZ..._AMg`, dead since rotation). Generated `hermes-agent-v5` with same model list + xAI voice models. Verified chat OK. Cleaned up stale keys (v3, v4, v4b). Updated model-chain doc to reflect actual state. Daily digest cron updated with HTML email + clean prompt for Telegram delivery. - **Jul 31, 2026:** Discovered stale key in config.yaml. Generated hermes-agent-v5 with same model list. Verified chat OK. Cleaned up stale keys (v3, v4, v4b).
- **Aug 5, 2026:** Admin-ai budget cap hit (~$20). Fallback chain exhausted 4 dead legs, landed on Anthropic direct. Burned $45 in 10 hours on claude-sonnet-5 via direct key. Anthropic key capped until Sep 1.
- **Aug 6, 2026:** Root cause of Aug 5 outage: 4 fallback legs dead simultaneously (admin-ai budget, DeepSeek balance $0, grok-4.6 404, Anthropic capped). Implemented two-key strategy (operational vs fallback). Rotated all 5 fallback keys. Added F5 (gpt-4.1-nano via OpenAI). Added haiku-4-5 and grok-4.5 to operational chain. Fixed grok-4.6 → grok-4.5. Synced Anita profile identically. Budget raised to $30.