# LiteLLM Config Precedence Pitfalls

## Master Key Config Override (Jul 2026)

### The Problem

LiteLLM reads `general_settings.master_key` from config.yaml when started with `--config /app/config.yaml`. This value takes precedence over the `LITELLM_MASTER_KEY` environment variable. If config.yaml has a stale/stub key, the stub silently becomes the UI login password.

### How It Manifests

- Admin UI login at `/v2/login` returns `401 Unauthorized`
- Error: `"Invalid credentials used to access UI. Check 'UI_USERNAME', 'UI_PASSWORD' in .env file"`
- Container logs: `ProxyException` at `login_utils.py` line 316 in `authenticate_user()`
- The `/v1/chat/completions` endpoint may still work normally (API keys are separate from UI password)
- User knows the correct master key (e.g., from `.env`) but login still fails

### Root Cause

In `login_utils.py`, the `authenticate_user()` function:

1. Calls `get_ui_credentials(master_key)` which reads `UI_USERNAME` (default "admin") and `UI_PASSWORD` (env var or `master_key` parameter)
2. The `master_key` parameter comes from LiteLLM's internal state, which loaded it from `general_settings.master_key` in config.yaml
3. Compares the login form password against this config-file-derived key using `secrets.compare_digest()`
4. If the comparison fails and no database user matches, throws the 401 error

### Detection Workflow

```bash
# 1. Check container logs for login failures
docker logs litellm --tail 50 2>&1 | grep -A 5 'login_v2\|401 Unauthorized'

# 2. Compare config sources by hash and length (NOT raw value)
ENV_KEY=$(grep LITELLM_MASTER_KEY .env | cut -d= -f2)
CFG_KEY=$(grep 'master_key:' config.yaml | sed 's/.*master_key: //')
echo "env:  len=${#ENV_KEY} sha=$(echo -n "$ENV_KEY" | sha256sum | cut -c1-16)"
echo "cfg:  len=${#CFG_KEY} sha=$(echo -n "$CFG_KEY" | sha256sum | cut -c1-16)"

# 3. Verify what the container actually sees
docker exec litellm printenv | grep LITELLM_MASTER_KEY

# 4. If lengths/hashes differ → config.yaml has a stale key
# 5. Fix by removing master_key from config.yaml
sed -i '/^  master_key:/d' config.yaml
docker compose restart litellm

# 6. Verify fix with a direct login test
MASTER_KEY=$(grep LITELLM_MASTER_KEY .env | cut -d= -f2)
curl -s -X POST http://127.0.0.1:4000/v2/login \
  -H 'Content-Type: application/json' \
  -d "{\"username\":\"admin\",\"password\":\"$MASTER_KEY\"}"
# Expected: {"user_id":"...","key":"sk-...","user_role":"proxy_admin",...}
```

### Why API Still Works While Login Fails

LiteLLM has two separate auth paths:
- **API key auth** (`/v1/chat/completions`): Validates keys stored in `LiteLLM_VerificationToken` table, independent of the master_key
- **UI login** (`/v2/login`): Compares against the in-memory `master_key` which came from config.yaml

This is why a config.yaml with a stale key breaks UI login but leaves API traffic unaffected.

### Prevention

1. **Remove `master_key` from config.yaml** — rely on `LITELLM_MASTER_KEY` env var only
2. **Audit dual-source configs** — any service with both config file AND env var for the same setting risks this category of mismatch
3. **Use hash comparison for credentials** — never log or transmit raw secret values; compare by hash to detect mismatches without exposing secrets