hermes-skills/skills/devops/docker-service-deployment/references/litellm-env-key-migration.md

LiteLLM env-key migration: encrypted DB rows after host move

When migrating LiteLLM with store_model_in_db: true, /v1/models can return a healthy-looking catalog while /chat/completions fails or hangs. If logs show:

Error decrypting value for key: api_key
Did your master_key/salt key change recently?
nacl.exceptions.CryptoError: Decryption failed

then the migrated Postgres DB contains encrypted model/provider params that the new container cannot decrypt.

Safe diagnosis

Proceed read-only first. Do not print raw keys.

1. Compare old and new container env fingerprints:
   • LITELLM_MASTER_KEY
   • LITELLM_SALT_KEY
   • DATABASE_URL
   • STORE_MODEL_IN_DB
2. Compare by sha256:<first16> len:<n> only.
3. Check model/credential/token table counts:
   • LiteLLM_ProxyModelTable
   • LiteLLM_CredentialsTable
   • LiteLLM_VerificationToken
4. Confirm public /v1/models and local /health/readiness separately from completions.

Minimal fix when fingerprints differ

Scope the change to the new host only.

1. Backup the new host .env first:

   cp /root/docker/litellm/.env /root/docker/litellm/.env.pre-keyfix-$(date -u +%Y%m%d-%H%M%S)
   chmod 600 /root/docker/litellm/.env.pre-keyfix-*
   

2. Copy only these values from the old working host into the new host .env:
   • LITELLM_MASTER_KEY
   • LITELLM_SALT_KEY
3. Verify the new host .env fingerprints match old.
4. Recreate the LiteLLM container; restart is not enough. Docker container env is baked at create time:

   cd /root/docker/litellm
   docker compose --env-file .env up -d --force-recreate --no-deps litellm
   

5. Verify the recreated container env fingerprints, not just the file.

Verification

Run public HTTPS completion probes after readiness passes:

curl -sS https://admin-ai.example.com/v1/chat/completions \
  -H "Authorization: Bearer $KEY" \
  -H 'Content-Type: application/json' \
  -d '{"model":"deepseek-chat","messages":[{"role":"user","content":"Reply with OK only."}],"max_tokens":128}'

Verify at least:

• deepseek-v4-pro
• deepseek-chat
• gemini-pro-latest
• gemini-flash-latest

Use enough max_tokens for reasoning models. A too-small token budget can produce blank content even though the call succeeded.

Interpretation

If non-OpenAI models work but gpt-5.5 returns 429 insufficient_quota, the migration/encryption issue is fixed; the remaining problem is upstream OpenAI billing/quota.

User handling standard

For Germaine, migrations touching env/secrets/service restarts must be handled as: read-only compare first, report fingerprints and exact planned minimal change, get approval, change one scoped item, verify, then proceed. Do not bundle extra config changes or unrelated cleanup.