hermes-skills/skills/devops/server-architecture-plan/references/admin-ai-migration-plan.md

Admin-AI Migration Plan: Hetzner → app1

Source: 178.156.167.181 (Hetzner CPX41, 93% disk) Target: 152.53.36.131 (app1, RS 4000, 3% disk) DNS: admin-ai.itpropartner.com

Current State

Source (Old AI Box)

• LiteLLM v1.84.0 (ghcr.io/berriai/litellm) + Postgres 16
• 39 configured models with direct API keys (Anthropic, DeepSeek, Google, OpenAI, Groq, xAI, OpenRouter)
• Open WebUI (0.9.6) — 121 MB SQLite DB
• Ollama — 8 local models (~10 GB), 7 cloud-proxy models
• Qdrant vector DB — small, optional
• Caddy reverse proxy
• 226G disk at 93% — critical threshold

Target (app1)

• Docker + Compose v5.3.1 already installed
• Ollama + Open WebUI + n8n already running
• Caddy 2.6.2 active on 80/443
• 1TB at 3% used, 32 GB RAM
• Caddy already proxies: n8n.itpropartner.com, ai.itpropartner.com

Migration Plan

Phase 0 — Pre-flight (no downtime, no DNS changes)

• Pull LiteLLM image on app1
• Set up LiteLLM + Postgres docker-compose on app1
• Export Postgres DB from old → import to app1
• Add Caddy config for admin-ai.itpropartner.com on app1
• Test admin-ai endpoint on app1 via direct IP or internal DNS — model routing failure, clean restart in progress
• Pull needed Ollama models on app1 (llama3.2:3b, qwen2.5:7b) — not needed, ollama already running on app1

Phase 1 — Cutover

• Switch Hermes to fallback chain (OpenRouter deepseek-chat)
• Update DNS: admin-ai.itpropartner.com → 152.53.36.131 (TTL 60)
• Monitor admin-ai endpoint
• Run end-to-end model test through app1
• Verify vision (gemini-pro-latest) works

Phase 2 — Restore Primary

• Switch Hermes back to admin-ai primary
• Verify all 22 cron jobs pass
• Verify Anita's profile

Phase 3 — Decommission (after 3 days stable)

• Keep old box running for rollback
• After verification, decom Hetzner AI server

Detailed Steps

Step 0: Pull LiteLLM image on app1

ssh -i /root/.ssh/itpp-infra root@152.53.36.131
docker pull ghcr.io/berriai/litellm:v1.84.0

Step 1: Create LiteLLM docker-compose on app1

Create /root/docker/litellm/docker-compose.yml and /root/docker/litellm/.env.

The docker compose needs BOTH .env (master key + salt key) AND config.yaml (database_url + enforce_database). Even in DB mode, LiteLLM requires config.yaml for routing. Without it, models are listed but completions fail silently.

Step 2: Export/import Postgres DB

On old server:

PGPASSWORD=litellm docker exec litellm_postgres pg_dump -U litellm -d litellm_db --no-owner > /tmp/litellm_db_dump.sql

SCP to app1 (route through Core if needed):

scp root@178.156.167.181:/tmp/litellm_db_dump.sql /tmp/
scp /tmp/litellm_db_dump.sql root@152.53.36.131:/tmp/

On app1, start Postgres first, then import:

cd /root/docker/litellm && docker compose up -d litellm_postgres
sleep 5
PGPASSWORD=litellm docker exec -i litellm_postgres psql -U litellm -d litellm_db < /tmp/litellm_db_dump.sql

Step 3: Start LiteLLM on app1 (CRITICAL)

Start LiteLLM only after Postgres import completes AND is healthy:

cd /root/docker/litellm && docker compose up -d litellm
sleep 60  # Prisma migrations take 30-60s on first start with restored DB
curl -s http://127.0.0.1:4000/health/liveliness  # Should return "I'm alive!"

⚠️ CRITICAL — model routing failure on first start (Jul 13, 2026): After DB import, models appear in /v1/models but ALL completions fail with "Invalid model name passed." This occurs even with identical SALT_KEY, DB dump, and Docker image. Root cause is a LiteLLM startup race condition — the model routing table takes time to populate from the DB.

Fix: Do a clean restart (docker compose down && docker compose up -d). Wait for the full startup cycle (30-60s for Prisma migrations + model index rebuild). Test completions, not just the models endpoint. A model in /v1/models that fails completions means the routing layer didn't initialize properly on first boot. Verify with:

curl -s http://127.0.0.1:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-lbhhxIk_9pQ2Vmlym7bWPA" \
  -H "Content-Type: application/json" \
  -d '{"model":"deepseek-chat","messages":[{"role":"user","content":"Hi"}],"max_tokens":5}'
# Should return actual completion text, not "Invalid model name"

⚠️ config.yaml MUST be mounted: Even in DB mode (STORE_MODEL_IN_DB=True), LiteLLM requires config.yaml with general_settings.database_url and general_settings.enforce_database: true. Mount as a volume alongside the compose file.

⚠️ Healthcheck curl pitfall: The ghcr.io/berriai/litellm image does not include curl. A CMD-SHELL curl healthcheck produces 6000+ consecutive failures and marks the container (unhealthy) permanently. Use wget -qO- or a Python-native healthcheck instead.

⚠️ Separate compose per service: The old box uses a monolithic /opt/ai/docker-compose.yml with 6 containers. The new deployment uses the Docker standard — one service per /root/docker/<name>/ directory.

Step 4: Test on app1 via loopback

curl -s http://127.0.0.1:4000/v1/models -H "Authorization: Bearer sk-lbhhxIk_9pQ2Vmlym7bWPA"
# Should return 121+ models

Step 5: Add Caddy config on app1

Caddy already configured:

admin-ai.itpropartner.com {
    reverse_proxy 127.0.0.1:4000
}

Step 6: DNS cutover

Flip DNS A record to 152.53.36.131, verify with curl to admin-ai.itpropartner.com.

Rollback

If cutover fails:

1. DNS back to 178.156.167.181
2. Switch Hermes back to admin-ai (old box still running throughout)
3. Old box continues operating — zero data loss

Provider API Keys

All model API keys are encrypted in Postgres using LITELLM_SALT_KEY. As long as the same SALT_KEY is used on app1, the keys migrate transparently via the Postgres dump. Direct provider env vars (OPENAI_API_KEY, etc.) are all empty — everything routes through Postgres-stored keys or OpenRouter.