# LiteLLM NULL-Model Failure Entries in SpendLogs

## Pattern

Rows in `LiteLLM_SpendLogs` where `model`, `model_group`, AND `custom_llm_provider` are all empty/NULL, status is `failure`, spend is 0. These are requests that **failed before model routing** — LiteLLM's `user_api_key_auth()` raised an exception before it could identify a model.

## Real Example (Jul 14, 2026)

```
 model | model_group | custom_llm_provider | status  | count
-------+-------------+---------------------+---------+-------
       |             |                     | failure |   637
```

637 out of 1,305 requests (49%) were unlabeled auth failures.

## Source

The LiteLLM log shows `auth_exception_handler.py:95` firing repeatedly:

```
LiteLLM Proxy:ERROR: auth_exception_handler.py:95 - litellm.proxy.proxy_server.user_api_key_auth(): Exception occured -
```

These are almost certainly **NOT Hermes traffic**. Likely sources:
- **OpenWebUI** on the same app1 server, hitting the proxy with stale/invalid credentials
- External services or old integrations with expired API keys
- Automated health checks using dead keys

## In Spend Reports

When reporting failure rates, **exclude NULL-model entries** from the "Hermes failure rate" calculation. They distort the picture. Report them separately as "unlabeled auth failures (external)" and note they're likely not Hermes traffic.

To filter them out:

```sql
SELECT ...
FROM "LiteLLM_SpendLogs"
WHERE "startTime" >= '...' AND "startTime" < '...'
  AND (model != '' AND model IS NOT NULL)
```

## Verification

To confirm they're auth failures (not routing/model issues), check the LiteLLM container logs:

```bash
ssh app1 "docker logs litellm --since 24h 2>&1 | grep -c 'auth_exception_handler'"
```

Consistent auth-exception counts matching the NULL-model failure count confirm the pattern.