# LiteLLM admin-ai migration: encrypted DB rows require matching keys

## When this applies

Use this during `admin-ai.itpropartner.com` / LiteLLM host migrations when:

- `/v1/models` returns HTTP 200 and lists models, but `/v1/chat/completions` hangs, times out, or returns model/auth/provider errors.
- LiteLLM logs show `CryptoError: Decryption failed` or `Error decrypting value for key: api_key` / `organization`.
- A PostgreSQL DB dump/restore was moved to a new host.

The model catalog can look healthy even while completions are broken because the restored DB rows contain encrypted provider credentials/model parameters. The new host must use the same `LITELLM_MASTER_KEY` and `LITELLM_SALT_KEY` that encrypted those rows.

## Safe diagnosis: compare hashes only

Do **not** print secret values. Compare fingerprints between old and new hosts:

```bash
for host in 'old 178.156.167.181 /opt/ai/.env' 'new 152.53.36.131 /root/docker/litellm/.env'; do
  label=$(echo "$host" | awk '{print $1}')
  ip=$(echo "$host" | awk '{print $2}')
  envfile=$(echo "$host" | awk '{print $3}')
  echo "===== $label $ip ====="
  ssh -i /root/.ssh/itpp-infra root@$ip "python3 - <<'PY'
from pathlib import Path
import hashlib, os
p = Path('$envfile')
vals = {}
for line in p.read_text().splitlines():
    if '=' in line and not line.lstrip().startswith('#'):
        k,v = line.split('=',1)
        if k in ('LITELLM_MASTER_KEY','LITELLM_SALT_KEY','DATABASE_URL','STORE_MODEL_IN_DB'):
            vals[k]=v
for k in ('DATABASE_URL','LITELLM_MASTER_KEY','LITELLM_SALT_KEY','STORE_MODEL_IN_DB'):
    v = vals.get(k)
    if not v:
        print(f'{k}=<missing>')
    elif 'KEY' in k or 'URL' in k:
        print(f'{k}=sha256:{hashlib.sha256(v.encode()).hexdigest()[:16]} len:{len(v)}')
    else:
        print(f'{k}={v}')
PY"
done
```

Expected: `DATABASE_URL`, `LITELLM_MASTER_KEY`, and `LITELLM_SALT_KEY` fingerprints should match between the host that produced the DB and the host running the restored DB. If master/salt differ, restored encrypted credentials cannot decrypt.

## Minimal fix when old host is still available

Scope: app1/new host only. Do not touch Core Hermes config during this repair.

1. Back up new host `.env`.
2. Copy only `LITELLM_MASTER_KEY` and `LITELLM_SALT_KEY` from the old host's LiteLLM `.env` to the new host's LiteLLM `.env`.
3. Restart only the new host LiteLLM container.
4. Verify fingerprints match.
5. Verify both catalog and completions.

```bash
# On the new host after updating .env:
cd /root/docker/litellm
cp .env ".env.before-keyfix-$(date +%Y%m%d%H%M%S)"
docker compose restart litellm
```

## Required verification

Do not declare success after `/v1/models` only. Test real completions:

```bash
KEY='<admin-ai virtual key from Hermes config>'
for model in gpt-5.5 deepseek-v4-pro deepseek-chat gemini-pro-latest; do
  echo "=== $model ==="
  curl -sS --max-time 45 http://127.0.0.1:4000/v1/chat/completions \
    -H "Authorization: Bearer $KEY" \
    -H 'Content-Type: application/json' \
    -d "{\"model\":\"$model\",\"messages\":[{\"role\":\"user\",\"content\":\"Reply OK only.\"}],\"max_tokens\":16}" \
    | python3 -m json.tool | head -40
done
```

Success means each returns `choices[0].message.content`, not just HTTP 200.

## If old keys are unavailable

Do not brute force or hand-edit encrypted DB rows. Recreate LiteLLM credentials/models fresh on the new host via LiteLLM API/UI or a known-good model import process. Keep the old AI server running until the new host passes completion tests for the core models and at least one full day of clean usage.