From 54fc46ca1449518cbe84838522cc57c2331e8c89 Mon Sep 17 00:00:00 2001 From: root Date: Fri, 24 Jul 2026 16:05:35 -0400 Subject: [PATCH] docs: update model chain and add cost control rollout summary for 2026-07-24 --- docs/cost-control-rollout-2026-07-24.md | 37 ++++++++++++++++++++++ docs/model-chain.md | 42 +++++++++++++------------ 2 files changed, 59 insertions(+), 20 deletions(-) create mode 100644 docs/cost-control-rollout-2026-07-24.md diff --git a/docs/cost-control-rollout-2026-07-24.md b/docs/cost-control-rollout-2026-07-24.md new file mode 100644 index 0000000..35db812 --- /dev/null +++ b/docs/cost-control-rollout-2026-07-24.md @@ -0,0 +1,37 @@ +# Cost Control Rollout — 2026-07-24 + +**Trigger:** $46 additional unexpected spend on top of $155.76/3-day burn. +**Root cause:** Unlimited LiteLLM key + GPT-5.6 Terra as default gateway model with no enforced budget, session-size, or model-allowlist guardrails. + +## Changes Deployed + +### 1. LiteLLM — New constrained team + key +- **Team `hermes-normal-ops`**: $3.33 rolling daily cap + $100 rolling 30-day cap, 30 RPM, 250K TPM, max 3 parallel requests. +- **Key `hermes-normal-ops-daily-capped`** (`...I3gQ`): Hard $3.33/day, model-restricted to approved list only. +- **Allowed models**: `claude-sonnet-5`, `deepseek-v4-pro`, `deepseek-v4-flash`, `glm-5.2`, `MiniMax-M3`, `qwen3.7-plus`. +- **Terra/GPT-5.6/Claude-Opus-4**: explicitly **excluded** from this key — LiteLLM returns HTTP 403. +- **Legacy key `sk-...itzA`**: blocked (blocked=t in DB). + +### 2. Hermes config — Default routing +- **Conductor**: `claude-sonnet-5` (admin-ai/LiteLLM proxy). +- **Fallbacks**: `deepseek-v4-pro` → `deepseek-v4-flash` (admin-ai only). +- **Delegation/workers**: `deepseek-v4-pro`, fallback `deepseek-v4-flash`. +- **No automatic escalation to premium** — failure stops, not silently upgrades. + +### 3. Session controls +- **Context length**: 128k tokens hard ceiling. +- **Compression**: enabled at 50% fill, targets 20% ratio. +- **Max turns**: 50 per session (prevents unbounded tool-call marathons). + +### 4. Rate limits (on LiteLLM key) +- 30 RPM, 250K TPM, max 3 concurrent requests. +- 429 throttle-backoff confirmed working in live logs. + +## Verification +- Sonnet makes calls through new key: HTTP 200. +- Terra through new key: HTTP 403 (blocked). +- Old key blocked in DB: `blocked = t`. +- Live gateway confirmed routing through `admin-ai` at `https://admin-ai.itpropartner.com/v1/`. + +## What's still behavioral (not enforced) +- Model compliance is enforced at the proxy key layer. Cost caps are enforced at the key+team layer. Session size is a Hermes config setting — stickiness depends on the runtime respecting it. diff --git a/docs/model-chain.md b/docs/model-chain.md index bfb0252..47b8461 100644 --- a/docs/model-chain.md +++ b/docs/model-chain.md @@ -1,26 +1,28 @@ # AI Model Chain — IT Pro Partner -**Updated:** July 21, 2026 -**Sanitized:** July 23, 2026 (plaintext keys removed) -**Gateways:** admin-ai (self-hosted LiteLLM), OpenRouter, DeepSeek, Google, xAI +**Updated:** July 24, 2026 +**Active Policy:** Cost-Controlled Multi-Tier Fallback Chain -## Chain +## Chain Configuration -| # | Model | Provider | Key Storage | -|---|---|---|---| -| Primary | GPT-5.5 | admin-ai | Hudu [126] Hermes Primary Key | -| F1 | GPT-5.5 | OpenRouter | Hudu [153] | -| F2 | DeepSeek v4 Pro | DeepSeek | Hudu [152] | -| F3 | Gemini 3.5 Flash | Google | Hudu [161] / [151] | -| F4 | Grok 4.5 | xAI | Hudu [154] | +| Tier | Model | Provider | Key / Endpoint | Notes | +|---|---|---|---|---| +| **Primary** | `claude-sonnet-5` | `admin-ai` | `~/.secure/hermes-normal-ops-key.json` | Main conductor via LiteLLM ($3.33/day cap) | +| **Fallback 1** | `deepseek-v4-pro` | `admin-ai` | `~/.secure/hermes-normal-ops-key.json` | Primary workhorse fallback on admin-ai | +| **Fallback 2** | `deepseek-v4-flash` | `admin-ai` | `~/.secure/hermes-normal-ops-key.json` | Fast low-cost fallback on admin-ai | +| **Fallback 3** | `deepseek-chat` | `deepseek` | Direct DeepSeek API Key | Independent provider (survives admin-ai outage) | +| **Fallback 4** | `grok-2-1212` | `xai` | Direct xAI API Key | Independent provider (survives admin-ai outage) | -## Admin-AI (LiteLLM) -- URL: `admin-ai.itpropartner.com` (app1, 152.53.36.131) -- Master key: `/root/docker/litellm/.env` on app1 (Hudu [178] LiteLLM Master Key) -- Hermes virtual key: Hudu [126] Hermes Primary Key (GPT-5.5 + DeepSeek v4 Pro routing) +## Admin-AI Proxy (LiteLLM) +- **URL**: `https://admin-ai.itpropartner.com/v1` (app1, `152.53.36.131`) +- **Master Key**: `/root/docker/litellm/.env` on app1 (Hudu [178] LiteLLM Master Key) +- **Hermes Virtual Key**: `hermes-normal-ops-daily-capped` (`~/.secure/hermes-normal-ops-key.json`) +- **Daily Budget Cap**: $3.33/day ($100/30-day cap) -## Credential Storage -- Config: `/root/.hermes/config.yaml` -- Hudu: All API assets under layout 49 at https://hudu.itpropartner.com — search "AI Provider" or by model name -- Git: `itpp-infrastructure/docs/model-chain.md` (this file, sanitized) -- Full inventory: `itpp-infrastructure/docs/key-inventory.md` +## Delegation Subagents +- **Model**: `deepseek-v4-pro` via `admin-ai` +- **Fallback**: `deepseek-v4-flash` via `admin-ai` + +## Excluded from Production Routing +- **Gemini / Ollama**: Excluded from normal production routing per policy. +- **GPT-5.6 / Terra / Opus**: Escalation-only, requires explicit approval and user confirmation.