LiteLLM NULL-Model Failure Entries in SpendLogs

Pattern

Rows in LiteLLM_SpendLogs where model, model_group, AND custom_llm_provider are all empty/NULL, status is failure, spend is 0. These are requests that failed before model routing — LiteLLM's user_api_key_auth() raised an exception before it could identify a model.

Real Example (Jul 14, 2026)

 model | model_group | custom_llm_provider | status  | count
-------+-------------+---------------------+---------+-------
       |             |                     | failure |   637

637 out of 1,305 requests (49%) were unlabeled auth failures.

Source

The LiteLLM log shows auth_exception_handler.py:95 firing repeatedly:

LiteLLM Proxy:ERROR: auth_exception_handler.py:95 - litellm.proxy.proxy_server.user_api_key_auth(): Exception occured -

These are almost certainly NOT Hermes traffic. Likely sources:

OpenWebUI on the same app1 server, hitting the proxy with stale/invalid credentials
External services or old integrations with expired API keys
Automated health checks using dead keys

In Spend Reports

When reporting failure rates, exclude NULL-model entries from the "Hermes failure rate" calculation. They distort the picture. Report them separately as "unlabeled auth failures (external)" and note they're likely not Hermes traffic.

To filter them out:

SELECT ...
FROM "LiteLLM_SpendLogs"
WHERE "startTime" >= '...' AND "startTime" < '...'
  AND (model != '' AND model IS NOT NULL)

Verification

To confirm they're auth failures (not routing/model issues), check the LiteLLM container logs:

ssh app1 "docker logs litellm --since 24h 2>&1 | grep -c 'auth_exception_handler'"

Consistent auth-exception counts matching the NULL-model failure count confirm the pattern.

1.8 KiB Raw Permalink Blame History