Initial skills documentation — 25 categories, all SKILL.md + references + scripts

2026-07-15 17:42:20 -04:00
commit 1b4fcd4427
976 changed files with 188344 additions and 0 deletions
@@ -0,0 +1,66 @@
+# LiteLLM Config Precedence Pitfalls
+
+## Master Key Config Override (Jul 2026)
+
+### The Problem
+
+LiteLLM reads `general_settings.master_key` from config.yaml when started with `--config /app/config.yaml`. This value takes precedence over the `LITELLM_MASTER_KEY` environment variable. If config.yaml has a stale/stub key, the stub silently becomes the UI login password.
+
+### How It Manifests
+
+- Admin UI login at `/v2/login` returns `401 Unauthorized`
+- Error: `"Invalid credentials used to access UI. Check 'UI_USERNAME', 'UI_PASSWORD' in .env file"`
+- Container logs: `ProxyException` at `login_utils.py` line 316 in `authenticate_user()`
+- The `/v1/chat/completions` endpoint may still work normally (API keys are separate from UI password)
+- User knows the correct master key (e.g., from `.env`) but login still fails
+
+### Root Cause
+
+In `login_utils.py`, the `authenticate_user()` function:
+
+1. Calls `get_ui_credentials(master_key)` which reads `UI_USERNAME` (default "admin") and `UI_PASSWORD` (env var or `master_key` parameter)
+2. The `master_key` parameter comes from LiteLLM's internal state, which loaded it from `general_settings.master_key` in config.yaml
+3. Compares the login form password against this config-file-derived key using `secrets.compare_digest()`
+4. If the comparison fails and no database user matches, throws the 401 error
+
+### Detection Workflow
+
+```bash
+# 1. Check container logs for login failures
+docker logs litellm --tail 50 2>&1 | grep -A 5 'login_v2\|401 Unauthorized'
+
+# 2. Compare config sources by hash and length (NOT raw value)
+ENV_KEY=$(grep LITELLM_MASTER_KEY .env | cut -d= -f2)
+CFG_KEY=$(grep 'master_key:' config.yaml | sed 's/.*master_key: //')
+echo "env:  len=${#ENV_KEY} sha=$(echo -n "$ENV_KEY" | sha256sum | cut -c1-16)"
+echo "cfg:  len=${#CFG_KEY} sha=$(echo -n "$CFG_KEY" | sha256sum | cut -c1-16)"
+
+# 3. Verify what the container actually sees
+docker exec litellm printenv | grep LITELLM_MASTER_KEY
+
+# 4. If lengths/hashes differ → config.yaml has a stale key
+# 5. Fix by removing master_key from config.yaml
+sed -i '/^  master_key:/d' config.yaml
+docker compose restart litellm
+
+# 6. Verify fix with a direct login test
+MASTER_KEY=$(grep LITELLM_MASTER_KEY .env | cut -d= -f2)
+curl -s -X POST http://127.0.0.1:4000/v2/login \
+  -H 'Content-Type: application/json' \
+  -d "{\"username\":\"admin\",\"password\":\"$MASTER_KEY\"}"
+# Expected: {"user_id":"...","key":"sk-...","user_role":"proxy_admin",...}
+```
+
+### Why API Still Works While Login Fails
+
+LiteLLM has two separate auth paths:
+- **API key auth** (`/v1/chat/completions`): Validates keys stored in `LiteLLM_VerificationToken` table, independent of the master_key
+- **UI login** (`/v2/login`): Compares against the in-memory `master_key` which came from config.yaml
+
+This is why a config.yaml with a stale key breaks UI login but leaves API traffic unaffected.
+
+### Prevention
+
+1. **Remove `master_key` from config.yaml** — rely on `LITELLM_MASTER_KEY` env var only
+2. **Audit dual-source configs** — any service with both config file AND env var for the same setting risks this category of mismatch
+3. **Use hash comparison for credentials** — never log or transmit raw secret values; compare by hash to detect mismatches without exposing secrets