# Admin-AI Authentication Troubleshooting

## Symptom: Master Key from config.yaml returns 401

LiteLLM responds with `401 Authentication Error` even though the master key in `/opt/ai/config.yaml` matches what's in the `.env` file.

### Root Cause

LiteLLM has **two places** where the master key is set:

1. `/opt/ai/config.yaml` — `general_settings.master_key`
2. `/opt/ai/.env` — `LITELLM_MASTER_KEY` (passed as Docker container env var)

**The env var takes precedence.** The docker-compose.yml passes `LITELLM_MASTER_KEY` from `.env` into the container. When LiteLLM starts, it uses the env var value as the master key and registers a hash of it in the `LiteLLM_VerificationToken` Postgres table.

If `config.yaml` and `.env` have **different** master keys, the config.yaml value is ignored. The API accepts whatever key was set via `LITELLM_MASTER_KEY` at container startup.

### How a Mismatch Happens

- The DB was initialized with one master key value (via `LITELLM_MASTER_KEY` env var at first `docker compose up`)
- Later, `config.yaml` was edited to a different key value, but `.env` was NOT updated (or vice versa)
- The config.yaml change has no effect because the env var overrides it
- The DB still only has the original key's hash registered

### Diagnosis

```bash
# SSH to the AI server
ssh -i /root/.ssh/itpp-infra root@178.156.167.181

# 1. Check what the env var actually is inside the running container
docker inspect litellm --format '{{range .Config.Env}}{{println .}}{{end}}' | grep LITELLM_MASTER_KEY

# 2. Check what config.yaml says
grep master_key /opt/ai/config.yaml

# 3. Check what .env says
grep LITELLM_MASTER_KEY /opt/ai/.env

# 4. Check the DB for registered keys — find the one with real spend (this is the working one)
PGPASSWORD=litellm docker exec -e PGPASSWORD=litellm litellm_postgres psql -U litellm -d litellm_db \
  -c 'SELECT token, key_name, spend FROM "LiteLLM_VerificationToken" ORDER BY spend DESC LIMIT 5;'
```

### Fix

**Option A — Insert the key hash directly into the DB (recommended when config.yaml/.env match but DB has old key):**

```bash
# SSH to the AI server
ssh -i /root/.ssh/itpp-infra root@178.156.167.181

# Compute SHA-256 hash of the master key
HASH=$(echo -n "sk-your-master-key-here" | sha256sum | cut -d' ' -f1)

# Insert into the LiteLLM_VerificationToken table
# The token column is the SHA-256 hash, and it's the PRIMARY KEY
PGPASSWORD=litellm docker exec -e PGPASSWORD=litellm litellm_postgres psql -U litellm -d litellm_db \
  -c "INSERT INTO \"LiteLLM_VerificationToken\" (token, key_name, user_id, spend, models) \
      VALUES ('$HASH', 'sk-...last4', 'default_user_id', 0, '{}') \
      ON CONFLICT (token) DO UPDATE SET key_name = 'sk-...last4';"
```

Verification is instant — no container restart needed. The key is available immediately because LiteLLM checks the DB on every request.

**⚠️ `docker compose restart litellm` does NOT fix this** in LiteLLM v1.84.0 with `store_model_in_db: true`. The master key from config.yaml is NOT re-registered in the DB on restart. You must insert the key hash manually as shown above. This was verified experimentally on Jul 11, 2026.

**Why restart doesn't work:** LiteLLM only seeds the master key into the DB on FIRST startup. After that, the `LiteLLM_VerificationToken` table is authoritative. Changing config.yaml or .env without a corresponding DB insert has no effect. Even restarting the container does not trigger a re-read of the config for key registration — the DB is treated as the source of truth once initialized.

**Option B — Fix the .env to match the config.yaml, nuke the DB, and restart (nuclear option, use only if you can lose existing keys):**
```bash
# WARNING: This deletes all existing keys and re-seeds from config
PGPASSWORD=litellm docker exec -e PGPASSWORD=litellm litellm_postgres psql -U litellm -d litellm_db \
  -c "DELETE FROM \"LiteLLM_VerificationToken\";"
docker compose down litellm && docker compose up -d litellm
# On first startup with an empty DB, LiteLLM re-registers the config.yaml master key
```

**Option C — Generate a new virtual key via the working API (if any key works):**
```bash
curl -X POST 'https://admin-ai.itpropartner.com/key/generate' \
  -H "Authorization: Bearer <working-key>" \
  -H 'Content-Type: application/json' \
  -d '{"models": ["deepseek-chat","deepseek-v4-pro","gemini-flash-latest","gemini-pro-latest","claude-sonnet-4-6","claude-opus-4-8"],"max_budget": 50}'
```

### Verification

```bash
curl -s -o /dev/null -w 'HTTP %{http_code}' \
  https://admin-ai.itpropartner.com/v1/models \
  -H "Authorization: Bearer <key>" --connect-timeout 10
# Should return HTTP 200 (not 401)
```

## Current State (July 11, 2026)

- admin-ai runs on **178.156.167.181** (Hetzner AI box — NOT app1)
- `.env` file has `LITELLM_MASTER_KEY=sk-lit...c5b1`
- `config.yaml` also has `general_settings.master_key: sk-lit...c5b1`
- These ARE the same value, BUT:
  - The LiteLLM container was started 6+ days ago under a DIFFERENT `.env` value
  - The DB's `LiteLLM_VerificationToken` has the OLD key hash registered (spent $232+)
  - Restarting the container with `docker compose restart litellm` will re-register the current key
- **Core's config.yaml has NO `admin-ai` provider section** — only `openrouter` and `ollama-local`
- The working key hash with $232 spend is the one that was set at original container startup

## LiteLLM Config Files

| File | Path | Purpose |
|------|------|---------|
| Docker compose | `/opt/ai/docker-compose.yml` | Service definitions, env var injection |
| Env file | `/opt/ai/.env` | Actual secrets (LITELLM_MASTER_KEY, POSTGRES_PASSWORD, LITELLM_SALT_KEY) |
| App config | `/opt/ai/config.yaml` | LiteLLM runtime config (minimal in DB mode) |
| Caddyfile | `/opt/ai/Caddyfile` | TLS termination, reverse proxy to :4000 |

## Table Schema

The `LiteLLM_VerificationToken` table uses `token` (the **hashed** key value) as the primary key. The `key_name` column stores the last 4 chars of the plaintext key for identification (e.g., `sk-...bWPA`).

Key fields:
- `token` — SHA-256 hash of the actual key (PK)
- `key_name` — Last 4 chars identifier (e.g., `sk-...bWPA`)
- `spend` — Cumulative spend in USD
- `max_budget` — Budget limit (null = unlimited)
- `blocked` — Boolean, null = not blocked