Initial skills documentation — 25 categories, all SKILL.md + references + scripts

2026-07-15 17:42:20 -04:00
commit 1b4fcd4427
976 changed files with 188344 additions and 0 deletions
@@ -0,0 +1,51 @@
+# LiteLLM NULL-Model Failure Entries in SpendLogs
+
+## Pattern
+
+Rows in `LiteLLM_SpendLogs` where `model`, `model_group`, AND `custom_llm_provider` are all empty/NULL, status is `failure`, spend is 0. These are requests that **failed before model routing** — LiteLLM's `user_api_key_auth()` raised an exception before it could identify a model.
+
+## Real Example (Jul 14, 2026)
+
+```
+ model | model_group | custom_llm_provider | status  | count
+-------+-------------+---------------------+---------+-------
+       |             |                     | failure |   637
+```
+
+637 out of 1,305 requests (49%) were unlabeled auth failures.
+
+## Source
+
+The LiteLLM log shows `auth_exception_handler.py:95` firing repeatedly:
+
+```
+LiteLLM Proxy:ERROR: auth_exception_handler.py:95 - litellm.proxy.proxy_server.user_api_key_auth(): Exception occured -
+```
+
+These are almost certainly **NOT Hermes traffic**. Likely sources:
+- **OpenWebUI** on the same app1 server, hitting the proxy with stale/invalid credentials
+- External services or old integrations with expired API keys
+- Automated health checks using dead keys
+
+## In Spend Reports
+
+When reporting failure rates, **exclude NULL-model entries** from the "Hermes failure rate" calculation. They distort the picture. Report them separately as "unlabeled auth failures (external)" and note they're likely not Hermes traffic.
+
+To filter them out:
+
+```sql
+SELECT ...
+FROM "LiteLLM_SpendLogs"
+WHERE "startTime" >= '...' AND "startTime" < '...'
+  AND (model != '' AND model IS NOT NULL)
+```
+
+## Verification
+
+To confirm they're auth failures (not routing/model issues), check the LiteLLM container logs:
+
+```bash
+ssh app1 "docker logs litellm --since 24h 2>&1 | grep -c 'auth_exception_handler'"
+```
+
+Consistent auth-exception counts matching the NULL-model failure count confirm the pattern.