LiteLLM admin-ai migration: encrypted DB rows require matching keys

When this applies

Use this during admin-ai.itpropartner.com / LiteLLM host migrations when:

/v1/models returns HTTP 200 and lists models, but /v1/chat/completions hangs, times out, or returns model/auth/provider errors.
LiteLLM logs show CryptoError: Decryption failed or Error decrypting value for key: api_key / organization.
A PostgreSQL DB dump/restore was moved to a new host.

The model catalog can look healthy even while completions are broken because the restored DB rows contain encrypted provider credentials/model parameters. The new host must use the same LITELLM_MASTER_KEY and LITELLM_SALT_KEY that encrypted those rows.

Safe diagnosis: compare hashes only

Do not print secret values. Compare fingerprints between old and new hosts:

for host in 'old 178.156.167.181 /opt/ai/.env' 'new 152.53.36.131 /root/docker/litellm/.env'; do
  label=$(echo "$host" | awk '{print $1}')
  ip=$(echo "$host" | awk '{print $2}')
  envfile=$(echo "$host" | awk '{print $3}')
  echo "===== $label $ip ====="
  ssh -i /root/.ssh/itpp-infra root@$ip "python3 - <<'PY'
from pathlib import Path
import hashlib, os
p = Path('$envfile')
vals = {}
for line in p.read_text().splitlines():
    if '=' in line and not line.lstrip().startswith('#'):
        k,v = line.split('=',1)
        if k in ('LITELLM_MASTER_KEY','LITELLM_SALT_KEY','DATABASE_URL','STORE_MODEL_IN_DB'):
            vals[k]=v
for k in ('DATABASE_URL','LITELLM_MASTER_KEY','LITELLM_SALT_KEY','STORE_MODEL_IN_DB'):
    v = vals.get(k)
    if not v:
        print(f'{k}=<missing>')
    elif 'KEY' in k or 'URL' in k:
        print(f'{k}=sha256:{hashlib.sha256(v.encode()).hexdigest()[:16]} len:{len(v)}')
    else:
        print(f'{k}={v}')
PY"
done

Expected: DATABASE_URL, LITELLM_MASTER_KEY, and LITELLM_SALT_KEY fingerprints should match between the host that produced the DB and the host running the restored DB. If master/salt differ, restored encrypted credentials cannot decrypt.

Minimal fix when old host is still available

Scope: app1/new host only. Do not touch Core Hermes config during this repair.

Back up new host .env.
Copy only LITELLM_MASTER_KEY and LITELLM_SALT_KEY from the old host's LiteLLM .env to the new host's LiteLLM .env.
Restart only the new host LiteLLM container.
Verify fingerprints match.
Verify both catalog and completions.

# On the new host after updating .env:
cd /root/docker/litellm
cp .env ".env.before-keyfix-$(date +%Y%m%d%H%M%S)"
docker compose restart litellm

Required verification

Do not declare success after /v1/models only. Test real completions:

KEY='<admin-ai virtual key from Hermes config>'
for model in gpt-5.5 deepseek-v4-pro deepseek-chat gemini-pro-latest; do
  echo "=== $model ==="
  curl -sS --max-time 45 http://127.0.0.1:4000/v1/chat/completions \
    -H "Authorization: Bearer $KEY" \
    -H 'Content-Type: application/json' \
    -d "{\"model\":\"$model\",\"messages\":[{\"role\":\"user\",\"content\":\"Reply OK only.\"}],\"max_tokens\":16}" \
    | python3 -m json.tool | head -40
done

Success means each returns choices[0].message.content, not just HTTP 200.

If old keys are unavailable

Do not brute force or hand-edit encrypted DB rows. Recreate LiteLLM credentials/models fresh on the new host via LiteLLM API/UI or a known-good model import process. Keep the old AI server running until the new host passes completion tests for the core models and at least one full day of clean usage.

3.5 KiB Raw Permalink Blame History