LiteLLM env-key migration: encrypted DB rows after host move

When migrating LiteLLM with store_model_in_db: true, /v1/models can return a healthy-looking catalog while /chat/completions fails or hangs. If logs show:

Error decrypting value for key: api_key
Did your master_key/salt key change recently?
nacl.exceptions.CryptoError: Decryption failed

then the migrated Postgres DB contains encrypted model/provider params that the new container cannot decrypt.

Safe diagnosis

Proceed read-only first. Do not print raw keys.

Compare old and new container env fingerprints:
- LITELLM_MASTER_KEY
- LITELLM_SALT_KEY
- DATABASE_URL
- STORE_MODEL_IN_DB
Compare by sha256:<first16> len:<n> only.
Check model/credential/token table counts:
- LiteLLM_ProxyModelTable
- LiteLLM_CredentialsTable
- LiteLLM_VerificationToken
Confirm public /v1/models and local /health/readiness separately from completions.

Minimal fix when fingerprints differ

Scope the change to the new host only.

Backup the new host .env first:

cp /root/docker/litellm/.env /root/docker/litellm/.env.pre-keyfix-$(date -u +%Y%m%d-%H%M%S)
chmod 600 /root/docker/litellm/.env.pre-keyfix-*

Copy only these values from the old working host into the new host .env:
- LITELLM_MASTER_KEY
- LITELLM_SALT_KEY
Verify the new host .env fingerprints match old.
Recreate the LiteLLM container; restart is not enough. Docker container env is baked at create time:
```
cd /root/docker/litellm
docker compose --env-file .env up -d --force-recreate --no-deps litellm
```
Verify the recreated container env fingerprints, not just the file.

Verification

Run public HTTPS completion probes after readiness passes:

curl -sS https://admin-ai.example.com/v1/chat/completions \
  -H "Authorization: Bearer $KEY" \
  -H 'Content-Type: application/json' \
  -d '{"model":"deepseek-chat","messages":[{"role":"user","content":"Reply with OK only."}],"max_tokens":128}'

Verify at least:

deepseek-v4-pro
deepseek-chat
gemini-pro-latest
gemini-flash-latest

Use enough max_tokens for reasoning models. A too-small token budget can produce blank content even though the call succeeded.

Interpretation

If non-OpenAI models work but gpt-5.5 returns 429 insufficient_quota, the migration/encryption issue is fixed; the remaining problem is upstream OpenAI billing/quota.

User handling standard

For Germaine, migrations touching env/secrets/service restarts must be handled as: read-only compare first, report fingerprints and exact planned minimal change, get approval, change one scoped item, verify, then proceed. Do not bundle extra config changes or unrelated cleanup.

2.7 KiB Raw Blame History