85 lines
3.5 KiB
Markdown
85 lines
3.5 KiB
Markdown
# LiteLLM admin-ai migration: encrypted DB rows require matching keys
|
|
|
|
## When this applies
|
|
|
|
Use this during `admin-ai.itpropartner.com` / LiteLLM host migrations when:
|
|
|
|
- `/v1/models` returns HTTP 200 and lists models, but `/v1/chat/completions` hangs, times out, or returns model/auth/provider errors.
|
|
- LiteLLM logs show `CryptoError: Decryption failed` or `Error decrypting value for key: api_key` / `organization`.
|
|
- A PostgreSQL DB dump/restore was moved to a new host.
|
|
|
|
The model catalog can look healthy even while completions are broken because the restored DB rows contain encrypted provider credentials/model parameters. The new host must use the same `LITELLM_MASTER_KEY` and `LITELLM_SALT_KEY` that encrypted those rows.
|
|
|
|
## Safe diagnosis: compare hashes only
|
|
|
|
Do **not** print secret values. Compare fingerprints between old and new hosts:
|
|
|
|
```bash
|
|
for host in 'old 178.156.167.181 /opt/ai/.env' 'new 152.53.36.131 /root/docker/litellm/.env'; do
|
|
label=$(echo "$host" | awk '{print $1}')
|
|
ip=$(echo "$host" | awk '{print $2}')
|
|
envfile=$(echo "$host" | awk '{print $3}')
|
|
echo "===== $label $ip ====="
|
|
ssh -i /root/.ssh/itpp-infra root@$ip "python3 - <<'PY'
|
|
from pathlib import Path
|
|
import hashlib, os
|
|
p = Path('$envfile')
|
|
vals = {}
|
|
for line in p.read_text().splitlines():
|
|
if '=' in line and not line.lstrip().startswith('#'):
|
|
k,v = line.split('=',1)
|
|
if k in ('LITELLM_MASTER_KEY','LITELLM_SALT_KEY','DATABASE_URL','STORE_MODEL_IN_DB'):
|
|
vals[k]=v
|
|
for k in ('DATABASE_URL','LITELLM_MASTER_KEY','LITELLM_SALT_KEY','STORE_MODEL_IN_DB'):
|
|
v = vals.get(k)
|
|
if not v:
|
|
print(f'{k}=<missing>')
|
|
elif 'KEY' in k or 'URL' in k:
|
|
print(f'{k}=sha256:{hashlib.sha256(v.encode()).hexdigest()[:16]} len:{len(v)}')
|
|
else:
|
|
print(f'{k}={v}')
|
|
PY"
|
|
done
|
|
```
|
|
|
|
Expected: `DATABASE_URL`, `LITELLM_MASTER_KEY`, and `LITELLM_SALT_KEY` fingerprints should match between the host that produced the DB and the host running the restored DB. If master/salt differ, restored encrypted credentials cannot decrypt.
|
|
|
|
## Minimal fix when old host is still available
|
|
|
|
Scope: app1/new host only. Do not touch Core Hermes config during this repair.
|
|
|
|
1. Back up new host `.env`.
|
|
2. Copy only `LITELLM_MASTER_KEY` and `LITELLM_SALT_KEY` from the old host's LiteLLM `.env` to the new host's LiteLLM `.env`.
|
|
3. Restart only the new host LiteLLM container.
|
|
4. Verify fingerprints match.
|
|
5. Verify both catalog and completions.
|
|
|
|
```bash
|
|
# On the new host after updating .env:
|
|
cd /root/docker/litellm
|
|
cp .env ".env.before-keyfix-$(date +%Y%m%d%H%M%S)"
|
|
docker compose restart litellm
|
|
```
|
|
|
|
## Required verification
|
|
|
|
Do not declare success after `/v1/models` only. Test real completions:
|
|
|
|
```bash
|
|
KEY='<admin-ai virtual key from Hermes config>'
|
|
for model in gpt-5.5 deepseek-v4-pro deepseek-chat gemini-pro-latest; do
|
|
echo "=== $model ==="
|
|
curl -sS --max-time 45 http://127.0.0.1:4000/v1/chat/completions \
|
|
-H "Authorization: Bearer $KEY" \
|
|
-H 'Content-Type: application/json' \
|
|
-d "{\"model\":\"$model\",\"messages\":[{\"role\":\"user\",\"content\":\"Reply OK only.\"}],\"max_tokens\":16}" \
|
|
| python3 -m json.tool | head -40
|
|
done
|
|
```
|
|
|
|
Success means each returns `choices[0].message.content`, not just HTTP 200.
|
|
|
|
## If old keys are unavailable
|
|
|
|
Do not brute force or hand-edit encrypted DB rows. Recreate LiteLLM credentials/models fresh on the new host via LiteLLM API/UI or a known-good model import process. Keep the old AI server running until the new host passes completion tests for the core models and at least one full day of clean usage.
|