Initial skills documentation — 25 categories, all SKILL.md + references + scripts
This commit is contained in:
root
+126
@@ -0,0 +1,126 @@
|
||||
# Admin-AI Authentication Troubleshooting
|
||||
|
||||
## Symptom: Master Key from config.yaml returns 401
|
||||
|
||||
LiteLLM responds with `401 Authentication Error` even though the master key in `/opt/ai/config.yaml` matches what's in the `.env` file.
|
||||
|
||||
### Root Cause
|
||||
|
||||
LiteLLM has **two places** where the master key is set:
|
||||
|
||||
1. `/opt/ai/config.yaml` — `general_settings.master_key`
|
||||
2. `/opt/ai/.env` — `LITELLM_MASTER_KEY` (passed as Docker container env var)
|
||||
|
||||
**The env var takes precedence.** The docker-compose.yml passes `LITELLM_MASTER_KEY` from `.env` into the container. When LiteLLM starts, it uses the env var value as the master key and registers a hash of it in the `LiteLLM_VerificationToken` Postgres table.
|
||||
|
||||
If `config.yaml` and `.env` have **different** master keys, the config.yaml value is ignored. The API accepts whatever key was set via `LITELLM_MASTER_KEY` at container startup.
|
||||
|
||||
### How a Mismatch Happens
|
||||
|
||||
- The DB was initialized with one master key value (via `LITELLM_MASTER_KEY` env var at first `docker compose up`)
|
||||
- Later, `config.yaml` was edited to a different key value, but `.env` was NOT updated (or vice versa)
|
||||
- The config.yaml change has no effect because the env var overrides it
|
||||
- The DB still only has the original key's hash registered
|
||||
|
||||
### Diagnosis
|
||||
|
||||
```bash
|
||||
# SSH to the AI server
|
||||
ssh -i /root/.ssh/itpp-infra root@178.156.167.181
|
||||
|
||||
# 1. Check what the env var actually is inside the running container
|
||||
docker inspect litellm --format '{{range .Config.Env}}{{println .}}{{end}}' | grep LITELLM_MASTER_KEY
|
||||
|
||||
# 2. Check what config.yaml says
|
||||
grep master_key /opt/ai/config.yaml
|
||||
|
||||
# 3. Check what .env says
|
||||
grep LITELLM_MASTER_KEY /opt/ai/.env
|
||||
|
||||
# 4. Check the DB for registered keys — find the one with real spend (this is the working one)
|
||||
PGPASSWORD=litellm docker exec -e PGPASSWORD=litellm litellm_postgres psql -U litellm -d litellm_db \
|
||||
-c 'SELECT token, key_name, spend FROM "LiteLLM_VerificationToken" ORDER BY spend DESC LIMIT 5;'
|
||||
```
|
||||
|
||||
### Fix
|
||||
|
||||
**Option A — Insert the key hash directly into the DB (recommended when config.yaml/.env match but DB has old key):**
|
||||
|
||||
```bash
|
||||
# SSH to the AI server
|
||||
ssh -i /root/.ssh/itpp-infra root@178.156.167.181
|
||||
|
||||
# Compute SHA-256 hash of the master key
|
||||
HASH=$(echo -n "sk-your-master-key-here" | sha256sum | cut -d' ' -f1)
|
||||
|
||||
# Insert into the LiteLLM_VerificationToken table
|
||||
# The token column is the SHA-256 hash, and it's the PRIMARY KEY
|
||||
PGPASSWORD=litellm docker exec -e PGPASSWORD=litellm litellm_postgres psql -U litellm -d litellm_db \
|
||||
-c "INSERT INTO \"LiteLLM_VerificationToken\" (token, key_name, user_id, spend, models) \
|
||||
VALUES ('$HASH', 'sk-...last4', 'default_user_id', 0, '{}') \
|
||||
ON CONFLICT (token) DO UPDATE SET key_name = 'sk-...last4';"
|
||||
```
|
||||
|
||||
Verification is instant — no container restart needed. The key is available immediately because LiteLLM checks the DB on every request.
|
||||
|
||||
**⚠️ `docker compose restart litellm` does NOT fix this** in LiteLLM v1.84.0 with `store_model_in_db: true`. The master key from config.yaml is NOT re-registered in the DB on restart. You must insert the key hash manually as shown above. This was verified experimentally on Jul 11, 2026.
|
||||
|
||||
**Why restart doesn't work:** LiteLLM only seeds the master key into the DB on FIRST startup. After that, the `LiteLLM_VerificationToken` table is authoritative. Changing config.yaml or .env without a corresponding DB insert has no effect. Even restarting the container does not trigger a re-read of the config for key registration — the DB is treated as the source of truth once initialized.
|
||||
|
||||
**Option B — Fix the .env to match the config.yaml, nuke the DB, and restart (nuclear option, use only if you can lose existing keys):**
|
||||
```bash
|
||||
# WARNING: This deletes all existing keys and re-seeds from config
|
||||
PGPASSWORD=litellm docker exec -e PGPASSWORD=litellm litellm_postgres psql -U litellm -d litellm_db \
|
||||
-c "DELETE FROM \"LiteLLM_VerificationToken\";"
|
||||
docker compose down litellm && docker compose up -d litellm
|
||||
# On first startup with an empty DB, LiteLLM re-registers the config.yaml master key
|
||||
```
|
||||
|
||||
**Option C — Generate a new virtual key via the working API (if any key works):**
|
||||
```bash
|
||||
curl -X POST 'https://admin-ai.itpropartner.com/key/generate' \
|
||||
-H "Authorization: Bearer <working-key>" \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"models": ["deepseek-chat","deepseek-v4-pro","gemini-flash-latest","gemini-pro-latest","claude-sonnet-4-6","claude-opus-4-8"],"max_budget": 50}'
|
||||
```
|
||||
|
||||
### Verification
|
||||
|
||||
```bash
|
||||
curl -s -o /dev/null -w 'HTTP %{http_code}' \
|
||||
https://admin-ai.itpropartner.com/v1/models \
|
||||
-H "Authorization: Bearer <key>" --connect-timeout 10
|
||||
# Should return HTTP 200 (not 401)
|
||||
```
|
||||
|
||||
## Current State (July 11, 2026)
|
||||
|
||||
- admin-ai runs on **178.156.167.181** (Hetzner AI box — NOT app1)
|
||||
- `.env` file has `LITELLM_MASTER_KEY=sk-lit...c5b1`
|
||||
- `config.yaml` also has `general_settings.master_key: sk-lit...c5b1`
|
||||
- These ARE the same value, BUT:
|
||||
- The LiteLLM container was started 6+ days ago under a DIFFERENT `.env` value
|
||||
- The DB's `LiteLLM_VerificationToken` has the OLD key hash registered (spent $232+)
|
||||
- Restarting the container with `docker compose restart litellm` will re-register the current key
|
||||
- **Core's config.yaml has NO `admin-ai` provider section** — only `openrouter` and `ollama-local`
|
||||
- The working key hash with $232 spend is the one that was set at original container startup
|
||||
|
||||
## LiteLLM Config Files
|
||||
|
||||
| File | Path | Purpose |
|
||||
|------|------|---------|
|
||||
| Docker compose | `/opt/ai/docker-compose.yml` | Service definitions, env var injection |
|
||||
| Env file | `/opt/ai/.env` | Actual secrets (LITELLM_MASTER_KEY, POSTGRES_PASSWORD, LITELLM_SALT_KEY) |
|
||||
| App config | `/opt/ai/config.yaml` | LiteLLM runtime config (minimal in DB mode) |
|
||||
| Caddyfile | `/opt/ai/Caddyfile` | TLS termination, reverse proxy to :4000 |
|
||||
|
||||
## Table Schema
|
||||
|
||||
The `LiteLLM_VerificationToken` table uses `token` (the **hashed** key value) as the primary key. The `key_name` column stores the last 4 chars of the plaintext key for identification (e.g., `sk-...bWPA`).
|
||||
|
||||
Key fields:
|
||||
- `token` — SHA-256 hash of the actual key (PK)
|
||||
- `key_name` — Last 4 chars identifier (e.g., `sk-...bWPA`)
|
||||
- `spend` — Cumulative spend in USD
|
||||
- `max_budget` — Budget limit (null = unlimited)
|
||||
- `blocked` — Boolean, null = not blocked
|
||||
Reference in New Issue
Block a user