Admin-AI Authentication Troubleshooting

Symptom: Master Key from config.yaml returns 401

LiteLLM responds with 401 Authentication Error even though the master key in /opt/ai/config.yaml matches what's in the .env file.

Root Cause

LiteLLM has two places where the master key is set:

/opt/ai/config.yaml — general_settings.master_key
/opt/ai/.env — LITELLM_MASTER_KEY (passed as Docker container env var)

The env var takes precedence. The docker-compose.yml passes LITELLM_MASTER_KEY from .env into the container. When LiteLLM starts, it uses the env var value as the master key and registers a hash of it in the LiteLLM_VerificationToken Postgres table.

If config.yaml and .env have different master keys, the config.yaml value is ignored. The API accepts whatever key was set via LITELLM_MASTER_KEY at container startup.

How a Mismatch Happens

The DB was initialized with one master key value (via LITELLM_MASTER_KEY env var at first docker compose up)
Later, config.yaml was edited to a different key value, but .env was NOT updated (or vice versa)
The config.yaml change has no effect because the env var overrides it
The DB still only has the original key's hash registered

Diagnosis

# SSH to the AI server
ssh -i /root/.ssh/itpp-infra root@178.156.167.181

# 1. Check what the env var actually is inside the running container
docker inspect litellm --format '{{range .Config.Env}}{{println .}}{{end}}' | grep LITELLM_MASTER_KEY

# 2. Check what config.yaml says
grep master_key /opt/ai/config.yaml

# 3. Check what .env says
grep LITELLM_MASTER_KEY /opt/ai/.env

# 4. Check the DB for registered keys — find the one with real spend (this is the working one)
PGPASSWORD=litellm docker exec -e PGPASSWORD=litellm litellm_postgres psql -U litellm -d litellm_db \
  -c 'SELECT token, key_name, spend FROM "LiteLLM_VerificationToken" ORDER BY spend DESC LIMIT 5;'

Fix

Option A — Insert the key hash directly into the DB (recommended when config.yaml/.env match but DB has old key):

# SSH to the AI server
ssh -i /root/.ssh/itpp-infra root@178.156.167.181

# Compute SHA-256 hash of the master key
HASH=$(echo -n "sk-your-master-key-here" | sha256sum | cut -d' ' -f1)

# Insert into the LiteLLM_VerificationToken table
# The token column is the SHA-256 hash, and it's the PRIMARY KEY
PGPASSWORD=litellm docker exec -e PGPASSWORD=litellm litellm_postgres psql -U litellm -d litellm_db \
  -c "INSERT INTO \"LiteLLM_VerificationToken\" (token, key_name, user_id, spend, models) \
      VALUES ('$HASH', 'sk-...last4', 'default_user_id', 0, '{}') \
      ON CONFLICT (token) DO UPDATE SET key_name = 'sk-...last4';"

Verification is instant — no container restart needed. The key is available immediately because LiteLLM checks the DB on every request.

⚠️ docker compose restart litellm does NOT fix this in LiteLLM v1.84.0 with store_model_in_db: true. The master key from config.yaml is NOT re-registered in the DB on restart. You must insert the key hash manually as shown above. This was verified experimentally on Jul 11, 2026.

Why restart doesn't work: LiteLLM only seeds the master key into the DB on FIRST startup. After that, the LiteLLM_VerificationToken table is authoritative. Changing config.yaml or .env without a corresponding DB insert has no effect. Even restarting the container does not trigger a re-read of the config for key registration — the DB is treated as the source of truth once initialized.

Option B — Fix the .env to match the config.yaml, nuke the DB, and restart (nuclear option, use only if you can lose existing keys):

# WARNING: This deletes all existing keys and re-seeds from config
PGPASSWORD=litellm docker exec -e PGPASSWORD=litellm litellm_postgres psql -U litellm -d litellm_db \
  -c "DELETE FROM \"LiteLLM_VerificationToken\";"
docker compose down litellm && docker compose up -d litellm
# On first startup with an empty DB, LiteLLM re-registers the config.yaml master key

Option C — Generate a new virtual key via the working API (if any key works):

curl -X POST 'https://admin-ai.itpropartner.com/key/generate' \
  -H "Authorization: Bearer <working-key>" \
  -H 'Content-Type: application/json' \
  -d '{"models": ["deepseek-chat","deepseek-v4-pro","gemini-flash-latest","gemini-pro-latest","claude-sonnet-4-6","claude-opus-4-8"],"max_budget": 50}'

Verification

curl -s -o /dev/null -w 'HTTP %{http_code}' \
  https://admin-ai.itpropartner.com/v1/models \
  -H "Authorization: Bearer <key>" --connect-timeout 10
# Should return HTTP 200 (not 401)

Current State (July 11, 2026)

admin-ai runs on 178.156.167.181 (Hetzner AI box — NOT app1)
.env file has LITELLM_MASTER_KEY=sk-lit...c5b1
config.yaml also has general_settings.master_key: sk-lit...c5b1
These ARE the same value, BUT:
- The LiteLLM container was started 6+ days ago under a DIFFERENT .env value
- The DB's LiteLLM_VerificationToken has the OLD key hash registered (spent $232+)
- Restarting the container with docker compose restart litellm will re-register the current key
Core's config.yaml has NO admin-ai provider section — only openrouter and ollama-local
The working key hash with $232 spend is the one that was set at original container startup

LiteLLM Config Files

File	Path	Purpose
Docker compose	`/opt/ai/docker-compose.yml`	Service definitions, env var injection
Env file	`/opt/ai/.env`	Actual secrets (LITELLM_MASTER_KEY, POSTGRES_PASSWORD, LITELLM_SALT_KEY)
App config	`/opt/ai/config.yaml`	LiteLLM runtime config (minimal in DB mode)
Caddyfile	`/opt/ai/Caddyfile`	TLS termination, reverse proxy to :4000

Table Schema

The LiteLLM_VerificationToken table uses token (the hashed key value) as the primary key. The key_name column stores the last 4 chars of the plaintext key for identification (e.g., sk-...bWPA).

Key fields:

token — SHA-256 hash of the actual key (PK)
key_name — Last 4 chars identifier (e.g., sk-...bWPA)
spend — Cumulative spend in USD
max_budget — Budget limit (null = unlimited)
blocked — Boolean, null = not blocked

6.2 KiB Raw Permalink Blame History