hermes-skills/skills/devops/docker-service-deployment/references/litellm-config-pitfalls.md

LiteLLM Config Precedence Pitfalls

Master Key Config Override (Jul 2026)

The Problem

LiteLLM reads general_settings.master_key from config.yaml when started with --config /app/config.yaml. This value takes precedence over the LITELLM_MASTER_KEY environment variable. If config.yaml has a stale/stub key, the stub silently becomes the UI login password.

How It Manifests

• Admin UI login at /v2/login returns 401 Unauthorized
• Error: "Invalid credentials used to access UI. Check 'UI_USERNAME', 'UI_PASSWORD' in .env file"
• Container logs: ProxyException at login_utils.py line 316 in authenticate_user()
• The /v1/chat/completions endpoint may still work normally (API keys are separate from UI password)
• User knows the correct master key (e.g., from .env) but login still fails

Root Cause

In login_utils.py, the authenticate_user() function:

1. Calls get_ui_credentials(master_key) which reads UI_USERNAME (default "admin") and UI_PASSWORD (env var or master_key parameter)
2. The master_key parameter comes from LiteLLM's internal state, which loaded it from general_settings.master_key in config.yaml
3. Compares the login form password against this config-file-derived key using secrets.compare_digest()
4. If the comparison fails and no database user matches, throws the 401 error

Detection Workflow

# 1. Check container logs for login failures
docker logs litellm --tail 50 2>&1 | grep -A 5 'login_v2\|401 Unauthorized'

# 2. Compare config sources by hash and length (NOT raw value)
ENV_KEY=$(grep LITELLM_MASTER_KEY .env | cut -d= -f2)
CFG_KEY=$(grep 'master_key:' config.yaml | sed 's/.*master_key: //')
echo "env:  len=${#ENV_KEY} sha=$(echo -n "$ENV_KEY" | sha256sum | cut -c1-16)"
echo "cfg:  len=${#CFG_KEY} sha=$(echo -n "$CFG_KEY" | sha256sum | cut -c1-16)"

# 3. Verify what the container actually sees
docker exec litellm printenv | grep LITELLM_MASTER_KEY

# 4. If lengths/hashes differ → config.yaml has a stale key
# 5. Fix by removing master_key from config.yaml
sed -i '/^  master_key:/d' config.yaml
docker compose restart litellm

# 6. Verify fix with a direct login test
MASTER_KEY=$(grep LITELLM_MASTER_KEY .env | cut -d= -f2)
curl -s -X POST http://127.0.0.1:4000/v2/login \
  -H 'Content-Type: application/json' \
  -d "{\"username\":\"admin\",\"password\":\"$MASTER_KEY\"}"
# Expected: {"user_id":"...","key":"sk-...","user_role":"proxy_admin",...}

Why API Still Works While Login Fails

LiteLLM has two separate auth paths:

• API key auth (/v1/chat/completions): Validates keys stored in LiteLLM_VerificationToken table, independent of the master_key
• UI login (/v2/login): Compares against the in-memory master_key which came from config.yaml

This is why a config.yaml with a stale key breaks UI login but leaves API traffic unaffected.

Prevention

1. Remove master_key from config.yaml — rely on LITELLM_MASTER_KEY env var only
2. Audit dual-source configs — any service with both config file AND env var for the same setting risks this category of mismatch
3. Use hash comparison for credentials — never log or transmit raw secret values; compare by hash to detect mismatches without exposing secrets