From 489f690303fbecdf7ffc24b17dad0e16dfe02582 Mon Sep 17 00:00:00 2001 From: root Date: Sat, 8 Aug 2026 22:54:43 -0400 Subject: [PATCH] =?UTF-8?q?docs:=20corrected=20model=20usage=20report=20?= =?UTF-8?q?=E2=80=94=20real=20root=20causes,=20fallback=20blind=20spot,=20?= =?UTF-8?q?fixed=20baseline?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/reports/model-usage-2026-08-09.md | 47 ++++++++++++++++++-------- 1 file changed, 33 insertions(+), 14 deletions(-) diff --git a/docs/reports/model-usage-2026-08-09.md b/docs/reports/model-usage-2026-08-09.md index a3b338f..9c173cd 100644 --- a/docs/reports/model-usage-2026-08-09.md +++ b/docs/reports/model-usage-2026-08-09.md @@ -51,7 +51,7 @@ DeepSeek V4 Pro is 17× cheaper per call and handles 99% of volume. | Jul 15 | 0 | $0.00 | | | Jul 14 | 2 | $0.00 | | | Jul 13 | 0 | $0.00 | | -| Jul 12 | 34 | $12.75 | Evaluation run | +| Jul 12 | 34 | $12.75 | Model eval pipeline | | Jul 11 | 0 | $0.00 | | | Jul 10 | 12 | $0.02 | | @@ -66,7 +66,7 @@ DeepSeek V4 Pro is 17× cheaper per call and handles 99% of volume. Date Total Spend Sonnet 5 Total Calls Sonnet Calls Aug 09 $1.42 $0.05 362 3 Aug 08 $16.00 $6.56 3,147 73 -Aug 07 $6.44 $0.93 1,787 42 ⬅ all Claude models +Aug 07 $6.44 $0.93 1,787 42 Aug 06 $3.06 $0.01 1,493 11 Aug 05 $7.96 $0.14 3,147 20 Aug 04 $5.14 $0.15 1,679 19 @@ -97,7 +97,7 @@ Jul 11 $0.18 $0.00 1,656 0 Jul 10 $21.89 $0.02 4,544 12 ``` -**Normal daily spend range (August):** $3–8/day +**August normal days:** $3–8/day typical, $10–16/day on heavy remediation days --- @@ -131,24 +131,43 @@ After Sep 1, 2026: base input rises to $3/M, cache hits to $0.30/M. --- -## Key Spend Anomalies +## Key Spend Anomalies — Root Cause Analysis | Date | Spend | Root Cause | |------|-------|-----------| -| Jul 12 | $99.86 | Massive outlier — evaluation run or unbounded subagent loop | -| Jul 13 | $57.49 | Continuation | -| Jul 22–24 | $37–67/day | Heavy deepseek-v4-pro throughput (2–3× normal) | -| Jul 31 | $20.01 | $20 daily cap breached — 89% from Sonnet 5 (subagent routing override) | +| **Jul 12** | $99.86 | **gpt-5.5 eval pipeline.** 196 calls to gpt-5.5 ($66.42 — 67% of day) with 219K avg prompt tokens. 94 calls alone at 23:00 ($43.55 in one hour). `deepseek-v4-flash` added 1,846 eval calls ($2.87). Model catalog audit against all 128 models. Not Hermes. | +| **Jul 13** | $57.49 | **gpt-5.5 eval pipeline (continuation).** 30 calls for $56.23 (98% of day) with 458K avg prompt tokens. Three overnight bursts: midnight ($15.36), 3 AM ($29.00), 4 AM ($11.87). DeepSeek V4 Pro handled all other traffic ($1.21). | +| **Jul 22–24** | $37–67/day | **gpt-5.5 → gpt-5.6-terra eval pipeline.** Fewer calls (1,023–1,066) but 10–20× normal cost per call. gpt-5.5 at $63.93 (Jul 22), gpt-5.6-terra at $31.64 (Jul 23) and $50.22 (Jul 24). Avg prompt size: 309K–397K tokens. Each eval call cost $0.30–$1.50 vs normal $0.003. | +| **Jul 31** | $20.01 | **$20 daily cap breached.** 146 Sonnet 5 calls ($17.78 — 89% of spend). Subagent `delegation.model` was pinned to `claude-sonnet-5`, bypassing the conductor's model routing. Fixed Aug 1 by switching delegation back to `deepseek-v4-pro`. | + +All four anomalies share a common root: **the July model evaluation pipeline** hitting gpt-5.5 and gpt-5.6-terra through admin-ai with enormous evaluation-sized contexts. These models were never in Hermes' production chain — the eval runner discovered them in the proxy catalog and tested them. The Jul 31 event was a separate bug: subagent delegation config hard-overriding to Sonnet 5. + +--- + +## Data Source Limitation + +> **This report only covers LiteLLM-proxied traffic (admin-ai). It is blind to direct fallback provider spend.** + +The fallback chain operates outside admin-ai: `deepseek direct → google direct → xai direct → anthropic direct`. When admin-ai is unreachable or the daily cap is hit, traffic falls through to these keys. Spend there is invisible to LiteLLM SpendLogs. + +**Known gap:** Aug 5 actual spend was ~$45 (per changelog) but LiteLLM shows only $7.96. The ~$37 delta went through direct provider keys. + +**Fix needed:** Real-time cost monitoring requires a second feed polling each provider's usage API directly. Without it, a fallback cascade can silently burn through provider credits with no alert. --- ## Verdict -> **The model chain is correct and working as designed. Not under-resourced.** +> **The model chain is correct and working as designed. Cost is under control for the proxied path. The fallback path is a blind spot that needs monitoring.** -- DeepSeek V4 Pro: 99% of calls, $5.60/day average — the workhorse -- Sonnet 5: 1.1% of calls, genuine rare override — not a silent runaway -- Monthly spend has stabilized at $3–8/day in August, well within the $20–30/day budget -- The model chain doc now matches reality: `deepseek-v4-pro` primary, `claude-sonnet-5` for critical escalation +- DeepSeek V4 Pro: 99% of calls, ~$5.60/day — the workhorse +- Sonnet 5: 1.1% of calls (~11/day typical), genuine rare override — not a silent runaway +- July's $271 in anomaly spend (Jul 12–24) was the model evaluation pipeline hitting non-production models — not Hermes +- August baseline: $3–8/day typical, $10–16/day on heavy remediation days +- The model chain doc matches reality: `deepseek-v4-pro` primary, `claude-sonnet-5` for critical escalation -**One concrete action:** Enable Anthropic prompt caching. It's a client-side change with no infrastructure cost — just set cache breakpoints on the system prompt. 80–90% off cached input tokens. Worth doing before Sep 1 when base prices rise 50%. +**Three action items:** + +1. **Enable Anthropic prompt caching** — 80–90% off cached input tokens. Client-side change only. Must be done before Sep 1 ($2→$3 base price increase). +2. **Implement fallback provider monitoring** — direct API polling of DeepSeek, Google, xAI, and Anthropic usage endpoints. The LiteLLM SpendLogs are blind to ~30–50% of actual spend on failover days. +3. **Tag eval pipeline traffic** — any automated model testing must use a dedicated LiteLLM key with its own budget cap. The July anomalies contaminated 30 days of production cost data.