ippadmin/itpp-infrastructure
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: update model chain and add cost control rollout summary for 2026-07-24

Browse Source
This commit is contained in:
root
2026-07-24 16:05:35 -04:00
parent 98bb0e4d27
commit 54fc46ca14
2 changed files with 59 additions and 20 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/cost-control-rollout-2026-07-24.md
+37
View File
@@ -0,0 +1,37 @@
# Cost Control Rollout — 2026-07-24
**Trigger:** $46 additional unexpected spend on top of $155.76/3-day burn.
**Root cause:** Unlimited LiteLLM key + GPT-5.6 Terra as default gateway model with no enforced budget, session-size, or model-allowlist guardrails.
## Changes Deployed
### 1. LiteLLM — New constrained team + key
- **Team `hermes-normal-ops`**: $3.33 rolling daily cap + $100 rolling 30-day cap, 30 RPM, 250K TPM, max 3 parallel requests.
- **Key `hermes-normal-ops-daily-capped`** (`...I3gQ`): Hard $3.33/day, model-restricted to approved list only.
- **Allowed models**: `claude-sonnet-5`, `deepseek-v4-pro`, `deepseek-v4-flash`, `glm-5.2`, `MiniMax-M3`, `qwen3.7-plus`.
- **Terra/GPT-5.6/Claude-Opus-4**: explicitly **excluded** from this key — LiteLLM returns HTTP 403.
- **Legacy key `sk-...itzA`**: blocked (blocked=t in DB).
### 2. Hermes config — Default routing
- **Conductor**: `claude-sonnet-5` (admin-ai/LiteLLM proxy).
- **Fallbacks**: `deepseek-v4-pro` → `deepseek-v4-flash` (admin-ai only).
- **Delegation/workers**: `deepseek-v4-pro`, fallback `deepseek-v4-flash`.
- **No automatic escalation to premium** — failure stops, not silently upgrades.
### 3. Session controls
- **Context length**: 128k tokens hard ceiling.
- **Compression**: enabled at 50% fill, targets 20% ratio.
- **Max turns**: 50 per session (prevents unbounded tool-call marathons).
### 4. Rate limits (on LiteLLM key)
- 30 RPM, 250K TPM, max 3 concurrent requests.
- 429 throttle-backoff confirmed working in live logs.
## Verification
- Sonnet makes calls through new key: HTTP 200.
- Terra through new key: HTTP 403 (blocked).
- Old key blocked in DB: `blocked = t`.
- Live gateway confirmed routing through `admin-ai` at `https://admin-ai.itpropartner.com/v1/`.
## What's still behavioral (not enforced)
- Model compliance is enforced at the proxy key layer. Cost caps are enforced at the key+team layer. Session size is a Hermes config setting — stickiness depends on the runtime respecting it.
docs/model-chain.md
+22 -20
View File
@@ -1,26 +1,28 @@
# AI Model Chain — IT Pro Partner # AI Model Chain — IT Pro Partner
**Updated:** July 21, 2026 **Updated:** July 24, 2026
**Sanitized:** July 23, 2026 (plaintext keys removed) **Active Policy:** Cost-Controlled Multi-Tier Fallback Chain
**Gateways:** admin-ai (self-hosted LiteLLM), OpenRouter, DeepSeek, Google, xAI
## Chain ## Chain Configuration
| # | Model | Provider | Key Storage | | Tier | Model | Provider | Key / Endpoint | Notes |
|---|---|---|---| |---|---|---|---|---|
| Primary | GPT-5.5 | admin-ai | Hudu [126] Hermes Primary Key | | **Primary** | `claude-sonnet-5` | `admin-ai` | `~/.secure/hermes-normal-ops-key.json` | Main conductor via LiteLLM ($3.33/day cap) |
| F1 | GPT-5.5 | OpenRouter | Hudu [153] | | **Fallback 1** | `deepseek-v4-pro` | `admin-ai` | `~/.secure/hermes-normal-ops-key.json` | Primary workhorse fallback on admin-ai |
| F2 | DeepSeek v4 Pro | DeepSeek | Hudu [152] | | **Fallback 2** | `deepseek-v4-flash` | `admin-ai` | `~/.secure/hermes-normal-ops-key.json` | Fast low-cost fallback on admin-ai |
| F3 | Gemini 3.5 Flash | Google | Hudu [161] / [151] | | **Fallback 3** | `deepseek-chat` | `deepseek` | Direct DeepSeek API Key | Independent provider (survives admin-ai outage) |
| F4 | Grok 4.5 | xAI | Hudu [154] | | **Fallback 4** | `grok-2-1212` | `xai` | Direct xAI API Key | Independent provider (survives admin-ai outage) |
## Admin-AI (LiteLLM) ## Admin-AI Proxy (LiteLLM)
- URL: `admin-ai.itpropartner.com` (app1, 152.53.36.131) - **URL**: `https://admin-ai.itpropartner.com/v1` (app1, `152.53.36.131`)
- Master key: `/root/docker/litellm/.env` on app1 (Hudu [178] LiteLLM Master Key) - **Master Key**: `/root/docker/litellm/.env` on app1 (Hudu [178] LiteLLM Master Key)
- Hermes virtual key: Hudu [126] Hermes Primary Key (GPT-5.5 + DeepSeek v4 Pro routing) - **Hermes Virtual Key**: `hermes-normal-ops-daily-capped` (`~/.secure/hermes-normal-ops-key.json`)
- **Daily Budget Cap**: $3.33/day ($100/30-day cap)
## Credential Storage ## Delegation Subagents
- Config: `/root/.hermes/config.yaml` - **Model**: `deepseek-v4-pro` via `admin-ai`
- Hudu: All API assets under layout 49 at https://hudu.itpropartner.com — search "AI Provider" or by model name - **Fallback**: `deepseek-v4-flash` via `admin-ai`
- Git: `itpp-infrastructure/docs/model-chain.md` (this file, sanitized)
- Full inventory: `itpp-infrastructure/docs/key-inventory.md` ## Excluded from Production Routing
- **Gemini / Ollama**: Excluded from normal production routing per policy.
- **GPT-5.6 / Terra / Opus**: Escalation-only, requires explicit approval and user confirmation.