52 lines
1.8 KiB
Markdown
52 lines
1.8 KiB
Markdown
# LiteLLM NULL-Model Failure Entries in SpendLogs
|
|
|
|
## Pattern
|
|
|
|
Rows in `LiteLLM_SpendLogs` where `model`, `model_group`, AND `custom_llm_provider` are all empty/NULL, status is `failure`, spend is 0. These are requests that **failed before model routing** — LiteLLM's `user_api_key_auth()` raised an exception before it could identify a model.
|
|
|
|
## Real Example (Jul 14, 2026)
|
|
|
|
```
|
|
model | model_group | custom_llm_provider | status | count
|
|
-------+-------------+---------------------+---------+-------
|
|
| | | failure | 637
|
|
```
|
|
|
|
637 out of 1,305 requests (49%) were unlabeled auth failures.
|
|
|
|
## Source
|
|
|
|
The LiteLLM log shows `auth_exception_handler.py:95` firing repeatedly:
|
|
|
|
```
|
|
LiteLLM Proxy:ERROR: auth_exception_handler.py:95 - litellm.proxy.proxy_server.user_api_key_auth(): Exception occured -
|
|
```
|
|
|
|
These are almost certainly **NOT Hermes traffic**. Likely sources:
|
|
- **OpenWebUI** on the same app1 server, hitting the proxy with stale/invalid credentials
|
|
- External services or old integrations with expired API keys
|
|
- Automated health checks using dead keys
|
|
|
|
## In Spend Reports
|
|
|
|
When reporting failure rates, **exclude NULL-model entries** from the "Hermes failure rate" calculation. They distort the picture. Report them separately as "unlabeled auth failures (external)" and note they're likely not Hermes traffic.
|
|
|
|
To filter them out:
|
|
|
|
```sql
|
|
SELECT ...
|
|
FROM "LiteLLM_SpendLogs"
|
|
WHERE "startTime" >= '...' AND "startTime" < '...'
|
|
AND (model != '' AND model IS NOT NULL)
|
|
```
|
|
|
|
## Verification
|
|
|
|
To confirm they're auth failures (not routing/model issues), check the LiteLLM container logs:
|
|
|
|
```bash
|
|
ssh app1 "docker logs litellm --since 24h 2>&1 | grep -c 'auth_exception_handler'"
|
|
```
|
|
|
|
Consistent auth-exception counts matching the NULL-model failure count confirm the pattern.
|