75 lines
2.7 KiB
Markdown
75 lines
2.7 KiB
Markdown
# LiteLLM env-key migration: encrypted DB rows after host move
|
|
|
|
When migrating LiteLLM with `store_model_in_db: true`, `/v1/models` can return a healthy-looking catalog while `/chat/completions` fails or hangs. If logs show:
|
|
|
|
```text
|
|
Error decrypting value for key: api_key
|
|
Did your master_key/salt key change recently?
|
|
nacl.exceptions.CryptoError: Decryption failed
|
|
```
|
|
|
|
then the migrated Postgres DB contains encrypted model/provider params that the new container cannot decrypt.
|
|
|
|
## Safe diagnosis
|
|
|
|
Proceed read-only first. Do not print raw keys.
|
|
|
|
1. Compare old and new container env fingerprints:
|
|
- `LITELLM_MASTER_KEY`
|
|
- `LITELLM_SALT_KEY`
|
|
- `DATABASE_URL`
|
|
- `STORE_MODEL_IN_DB`
|
|
2. Compare by `sha256:<first16> len:<n>` only.
|
|
3. Check model/credential/token table counts:
|
|
- `LiteLLM_ProxyModelTable`
|
|
- `LiteLLM_CredentialsTable`
|
|
- `LiteLLM_VerificationToken`
|
|
4. Confirm public `/v1/models` and local `/health/readiness` separately from completions.
|
|
|
|
## Minimal fix when fingerprints differ
|
|
|
|
Scope the change to the new host only.
|
|
|
|
1. Backup the new host `.env` first:
|
|
```bash
|
|
cp /root/docker/litellm/.env /root/docker/litellm/.env.pre-keyfix-$(date -u +%Y%m%d-%H%M%S)
|
|
chmod 600 /root/docker/litellm/.env.pre-keyfix-*
|
|
```
|
|
2. Copy only these values from the old working host into the new host `.env`:
|
|
- `LITELLM_MASTER_KEY`
|
|
- `LITELLM_SALT_KEY`
|
|
3. Verify the new host `.env` fingerprints match old.
|
|
4. **Recreate the LiteLLM container; restart is not enough.** Docker container env is baked at create time:
|
|
```bash
|
|
cd /root/docker/litellm
|
|
docker compose --env-file .env up -d --force-recreate --no-deps litellm
|
|
```
|
|
5. Verify the recreated container env fingerprints, not just the file.
|
|
|
|
## Verification
|
|
|
|
Run public HTTPS completion probes after readiness passes:
|
|
|
|
```bash
|
|
curl -sS https://admin-ai.example.com/v1/chat/completions \
|
|
-H "Authorization: Bearer $KEY" \
|
|
-H 'Content-Type: application/json' \
|
|
-d '{"model":"deepseek-chat","messages":[{"role":"user","content":"Reply with OK only."}],"max_tokens":128}'
|
|
```
|
|
|
|
Verify at least:
|
|
- `deepseek-v4-pro`
|
|
- `deepseek-chat`
|
|
- `gemini-pro-latest`
|
|
- `gemini-flash-latest`
|
|
|
|
Use enough `max_tokens` for reasoning models. A too-small token budget can produce blank `content` even though the call succeeded.
|
|
|
|
## Interpretation
|
|
|
|
If non-OpenAI models work but `gpt-5.5` returns `429 insufficient_quota`, the migration/encryption issue is fixed; the remaining problem is upstream OpenAI billing/quota.
|
|
|
|
## User handling standard
|
|
|
|
For Germaine, migrations touching env/secrets/service restarts must be handled as: read-only compare first, report fingerprints and exact planned minimal change, get approval, change one scoped item, verify, then proceed. Do not bundle extra config changes or unrelated cleanup.
|