docs: corrected model usage report — real root causes, fallback blind spot, fixed baseline

docs/reports/model-usage-2026-08-09.md

@@ -51,7 +51,7 @@ DeepSeek V4 Pro is 17× cheaper per call and handles 99% of volume.
 | Jul 15 | 0 | $0.00 | |
 | Jul 14 | 2 | $0.00 | |
 | Jul 13 | 0 | $0.00 | |
-| Jul 12 | 34 | $12.75 | Evaluation run |
+| Jul 12 | 34 | $12.75 | Model eval pipeline |
 | Jul 11 | 0 | $0.00 | |
 | Jul 10 | 12 | $0.02 | |
 
@@ -66,7 +66,7 @@ DeepSeek V4 Pro is 17× cheaper per call and handles 99% of volume.
 Date       Total Spend   Sonnet 5    Total Calls   Sonnet Calls
 Aug 09        $1.42         $0.05         362             3
 Aug 08       $16.00         $6.56       3,147            73
-Aug 07        $6.44         $0.93       1,787            42 ⬅ all Claude models
+Aug 07        $6.44         $0.93       1,787            42
 Aug 06        $3.06         $0.01       1,493            11
 Aug 05        $7.96         $0.14       3,147            20
 Aug 04        $5.14         $0.15       1,679            19
@@ -97,7 +97,7 @@ Jul 11        $0.18         $0.00       1,656             0
 Jul 10       $21.89         $0.02       4,544            12
 ```
 
-**Normal daily spend range (August):** $3–8/day
+**August normal days:** $3–8/day typical, $10–16/day on heavy remediation days
 
 ---
 
@@ -131,24 +131,43 @@ After Sep 1, 2026: base input rises to $3/M, cache hits to $0.30/M.
 
 ---
 
-## Key Spend Anomalies
+## Key Spend Anomalies — Root Cause Analysis
 
 | Date | Spend | Root Cause |
 |------|-------|-----------|
-| Jul 12 | $99.86 | Massive outlier — evaluation run or unbounded subagent loop |
+| **Jul 12** | $99.86 | **gpt-5.5 eval pipeline.** 196 calls to gpt-5.5 ($66.42 — 67% of day) with 219K avg prompt tokens. 94 calls alone at 23:00 ($43.55 in one hour). `deepseek-v4-flash` added 1,846 eval calls ($2.87). Model catalog audit against all 128 models. Not Hermes. |
-| Jul 13 | $57.49 | Continuation |
+| **Jul 13** | $57.49 | **gpt-5.5 eval pipeline (continuation).** 30 calls for $56.23 (98% of day) with 458K avg prompt tokens. Three overnight bursts: midnight ($15.36), 3 AM ($29.00), 4 AM ($11.87). DeepSeek V4 Pro handled all other traffic ($1.21). |
-| Jul 22–24 | $37–67/day | Heavy deepseek-v4-pro throughput (2–3× normal) |
+| **Jul 22–24** | $37–67/day | **gpt-5.5 → gpt-5.6-terra eval pipeline.** Fewer calls (1,023–1,066) but 10–20× normal cost per call. gpt-5.5 at $63.93 (Jul 22), gpt-5.6-terra at $31.64 (Jul 23) and $50.22 (Jul 24). Avg prompt size: 309K–397K tokens. Each eval call cost $0.30–$1.50 vs normal $0.003. |
-| Jul 31 | $20.01 | $20 daily cap breached — 89% from Sonnet 5 (subagent routing override) |
+| **Jul 31** | $20.01 | **$20 daily cap breached.** 146 Sonnet 5 calls ($17.78 — 89% of spend). Subagent `delegation.model` was pinned to `claude-sonnet-5`, bypassing the conductor's model routing. Fixed Aug 1 by switching delegation back to `deepseek-v4-pro`. |
+
+All four anomalies share a common root: **the July model evaluation pipeline** hitting gpt-5.5 and gpt-5.6-terra through admin-ai with enormous evaluation-sized contexts. These models were never in Hermes' production chain — the eval runner discovered them in the proxy catalog and tested them. The Jul 31 event was a separate bug: subagent delegation config hard-overriding to Sonnet 5.
+
+---
+
+## Data Source Limitation
+
+> **This report only covers LiteLLM-proxied traffic (admin-ai). It is blind to direct fallback provider spend.**
+
+The fallback chain operates outside admin-ai: `deepseek direct → google direct → xai direct → anthropic direct`. When admin-ai is unreachable or the daily cap is hit, traffic falls through to these keys. Spend there is invisible to LiteLLM SpendLogs.
+
+**Known gap:** Aug 5 actual spend was ~$45 (per changelog) but LiteLLM shows only $7.96. The ~$37 delta went through direct provider keys.
+
+**Fix needed:** Real-time cost monitoring requires a second feed polling each provider's usage API directly. Without it, a fallback cascade can silently burn through provider credits with no alert.
 
 ---
 
 ## Verdict
 
-> **The model chain is correct and working as designed. Not under-resourced.**
+> **The model chain is correct and working as designed. Cost is under control for the proxied path. The fallback path is a blind spot that needs monitoring.**
 
-- DeepSeek V4 Pro: 99% of calls, $5.60/day average — the workhorse
+- DeepSeek V4 Pro: 99% of calls, ~$5.60/day — the workhorse
-- Sonnet 5: 1.1% of calls, genuine rare override — not a silent runaway
+- Sonnet 5: 1.1% of calls (~11/day typical), genuine rare override — not a silent runaway
-- Monthly spend has stabilized at $3–8/day in August, well within the $20–30/day budget
+- July's $271 in anomaly spend (Jul 12–24) was the model evaluation pipeline hitting non-production models — not Hermes
-- The model chain doc now matches reality: `deepseek-v4-pro` primary, `claude-sonnet-5` for critical escalation
+- August baseline: $3–8/day typical, $10–16/day on heavy remediation days
+- The model chain doc matches reality: `deepseek-v4-pro` primary, `claude-sonnet-5` for critical escalation
 
-**One concrete action:** Enable Anthropic prompt caching. It's a client-side change with no infrastructure cost — just set cache breakpoints on the system prompt. 80–90% off cached input tokens. Worth doing before Sep 1 when base prices rise 50%.
+**Three action items:**
+
+1. **Enable Anthropic prompt caching** — 80–90% off cached input tokens. Client-side change only. Must be done before Sep 1 ($2→$3 base price increase).
+2. **Implement fallback provider monitoring** — direct API polling of DeepSeek, Google, xAI, and Anthropic usage endpoints. The LiteLLM SpendLogs are blind to ~30–50% of actual spend on failover days.
+3. **Tag eval pipeline traffic** — any automated model testing must use a dedicated LiteLLM key with its own budget cap. The July anomalies contaminated 30 days of production cost data.