126 lines
6.2 KiB
Markdown
126 lines
6.2 KiB
Markdown
# Admin-AI Authentication Troubleshooting
|
|
|
|
## Symptom: Master Key from config.yaml returns 401
|
|
|
|
LiteLLM responds with `401 Authentication Error` even though the master key in `/opt/ai/config.yaml` matches what's in the `.env` file.
|
|
|
|
### Root Cause
|
|
|
|
LiteLLM has **two places** where the master key is set:
|
|
|
|
1. `/opt/ai/config.yaml` — `general_settings.master_key`
|
|
2. `/opt/ai/.env` — `LITELLM_MASTER_KEY` (passed as Docker container env var)
|
|
|
|
**The env var takes precedence.** The docker-compose.yml passes `LITELLM_MASTER_KEY` from `.env` into the container. When LiteLLM starts, it uses the env var value as the master key and registers a hash of it in the `LiteLLM_VerificationToken` Postgres table.
|
|
|
|
If `config.yaml` and `.env` have **different** master keys, the config.yaml value is ignored. The API accepts whatever key was set via `LITELLM_MASTER_KEY` at container startup.
|
|
|
|
### How a Mismatch Happens
|
|
|
|
- The DB was initialized with one master key value (via `LITELLM_MASTER_KEY` env var at first `docker compose up`)
|
|
- Later, `config.yaml` was edited to a different key value, but `.env` was NOT updated (or vice versa)
|
|
- The config.yaml change has no effect because the env var overrides it
|
|
- The DB still only has the original key's hash registered
|
|
|
|
### Diagnosis
|
|
|
|
```bash
|
|
# SSH to the AI server
|
|
ssh -i /root/.ssh/itpp-infra root@178.156.167.181
|
|
|
|
# 1. Check what the env var actually is inside the running container
|
|
docker inspect litellm --format '{{range .Config.Env}}{{println .}}{{end}}' | grep LITELLM_MASTER_KEY
|
|
|
|
# 2. Check what config.yaml says
|
|
grep master_key /opt/ai/config.yaml
|
|
|
|
# 3. Check what .env says
|
|
grep LITELLM_MASTER_KEY /opt/ai/.env
|
|
|
|
# 4. Check the DB for registered keys — find the one with real spend (this is the working one)
|
|
PGPASSWORD=litellm docker exec -e PGPASSWORD=litellm litellm_postgres psql -U litellm -d litellm_db \
|
|
-c 'SELECT token, key_name, spend FROM "LiteLLM_VerificationToken" ORDER BY spend DESC LIMIT 5;'
|
|
```
|
|
|
|
### Fix
|
|
|
|
**Option A — Insert the key hash directly into the DB (recommended when config.yaml/.env match but DB has old key):**
|
|
|
|
```bash
|
|
# SSH to the AI server
|
|
ssh -i /root/.ssh/itpp-infra root@178.156.167.181
|
|
|
|
# Compute SHA-256 hash of the master key
|
|
HASH=$(echo -n "sk-your-master-key-here" | sha256sum | cut -d' ' -f1)
|
|
|
|
# Insert into the LiteLLM_VerificationToken table
|
|
# The token column is the SHA-256 hash, and it's the PRIMARY KEY
|
|
PGPASSWORD=litellm docker exec -e PGPASSWORD=litellm litellm_postgres psql -U litellm -d litellm_db \
|
|
-c "INSERT INTO \"LiteLLM_VerificationToken\" (token, key_name, user_id, spend, models) \
|
|
VALUES ('$HASH', 'sk-...last4', 'default_user_id', 0, '{}') \
|
|
ON CONFLICT (token) DO UPDATE SET key_name = 'sk-...last4';"
|
|
```
|
|
|
|
Verification is instant — no container restart needed. The key is available immediately because LiteLLM checks the DB on every request.
|
|
|
|
**⚠️ `docker compose restart litellm` does NOT fix this** in LiteLLM v1.84.0 with `store_model_in_db: true`. The master key from config.yaml is NOT re-registered in the DB on restart. You must insert the key hash manually as shown above. This was verified experimentally on Jul 11, 2026.
|
|
|
|
**Why restart doesn't work:** LiteLLM only seeds the master key into the DB on FIRST startup. After that, the `LiteLLM_VerificationToken` table is authoritative. Changing config.yaml or .env without a corresponding DB insert has no effect. Even restarting the container does not trigger a re-read of the config for key registration — the DB is treated as the source of truth once initialized.
|
|
|
|
**Option B — Fix the .env to match the config.yaml, nuke the DB, and restart (nuclear option, use only if you can lose existing keys):**
|
|
```bash
|
|
# WARNING: This deletes all existing keys and re-seeds from config
|
|
PGPASSWORD=litellm docker exec -e PGPASSWORD=litellm litellm_postgres psql -U litellm -d litellm_db \
|
|
-c "DELETE FROM \"LiteLLM_VerificationToken\";"
|
|
docker compose down litellm && docker compose up -d litellm
|
|
# On first startup with an empty DB, LiteLLM re-registers the config.yaml master key
|
|
```
|
|
|
|
**Option C — Generate a new virtual key via the working API (if any key works):**
|
|
```bash
|
|
curl -X POST 'https://admin-ai.itpropartner.com/key/generate' \
|
|
-H "Authorization: Bearer <working-key>" \
|
|
-H 'Content-Type: application/json' \
|
|
-d '{"models": ["deepseek-chat","deepseek-v4-pro","gemini-flash-latest","gemini-pro-latest","claude-sonnet-4-6","claude-opus-4-8"],"max_budget": 50}'
|
|
```
|
|
|
|
### Verification
|
|
|
|
```bash
|
|
curl -s -o /dev/null -w 'HTTP %{http_code}' \
|
|
https://admin-ai.itpropartner.com/v1/models \
|
|
-H "Authorization: Bearer <key>" --connect-timeout 10
|
|
# Should return HTTP 200 (not 401)
|
|
```
|
|
|
|
## Current State (July 11, 2026)
|
|
|
|
- admin-ai runs on **178.156.167.181** (Hetzner AI box — NOT app1)
|
|
- `.env` file has `LITELLM_MASTER_KEY=sk-lit...c5b1`
|
|
- `config.yaml` also has `general_settings.master_key: sk-lit...c5b1`
|
|
- These ARE the same value, BUT:
|
|
- The LiteLLM container was started 6+ days ago under a DIFFERENT `.env` value
|
|
- The DB's `LiteLLM_VerificationToken` has the OLD key hash registered (spent $232+)
|
|
- Restarting the container with `docker compose restart litellm` will re-register the current key
|
|
- **Core's config.yaml has NO `admin-ai` provider section** — only `openrouter` and `ollama-local`
|
|
- The working key hash with $232 spend is the one that was set at original container startup
|
|
|
|
## LiteLLM Config Files
|
|
|
|
| File | Path | Purpose |
|
|
|------|------|---------|
|
|
| Docker compose | `/opt/ai/docker-compose.yml` | Service definitions, env var injection |
|
|
| Env file | `/opt/ai/.env` | Actual secrets (LITELLM_MASTER_KEY, POSTGRES_PASSWORD, LITELLM_SALT_KEY) |
|
|
| App config | `/opt/ai/config.yaml` | LiteLLM runtime config (minimal in DB mode) |
|
|
| Caddyfile | `/opt/ai/Caddyfile` | TLS termination, reverse proxy to :4000 |
|
|
|
|
## Table Schema
|
|
|
|
The `LiteLLM_VerificationToken` table uses `token` (the **hashed** key value) as the primary key. The `key_name` column stores the last 4 chars of the plaintext key for identification (e.g., `sk-...bWPA`).
|
|
|
|
Key fields:
|
|
- `token` — SHA-256 hash of the actual key (PK)
|
|
- `key_name` — Last 4 chars identifier (e.g., `sk-...bWPA`)
|
|
- `spend` — Cumulative spend in USD
|
|
- `max_budget` — Budget limit (null = unlimited)
|
|
- `blocked` — Boolean, null = not blocked |