hermes-recovery/references/admin-ai-migration-plan.md

Admin-AI Migration Plan: Hetzner → app1

Source: 178.156.167.181 (Hetzner CPX41, 93% disk) Target: 152.53.36.131 (app1, RS 4000, 3% disk) DNS: admin-ai.itpropartner.com

Current State

Source (Old AI Box)

• LiteLLM v1.84.0 (ghcr.io/berriai/litellm) + Postgres 16
• 39 configured models with direct API keys (Anthropic, DeepSeek, Google, OpenAI, Groq, xAI, OpenRouter)
• Open WebUI (0.9.6) — 121 MB SQLite DB
• Ollama — 8 local models (~10 GB), 7 cloud-proxy models
• Qdrant vector DB — small, optional
• Caddy reverse proxy
• 226G disk at 93% — critical threshold

Target (app1)

• Docker + Compose v5.3.1 already installed
• Ollama + Open WebUI + n8n already running
• Caddy 2.6.2 active on 80/443
• 1TB at 3% used, 32 GB RAM
• Caddy already proxies: n8n.itpropartner.com, ai.itpropartner.com

Migration Plan

Phase 0 — Pre-flight (do now, before you leave for Colombia)

No downtime. No DNS changes. Prep work only.

• Pull LiteLLM image on app1
• Set up LiteLLM + Postgres docker-compose on app1
• Export Postgres DB from old → import to app1
• Add Caddy config for admin-ai.itpropartner.com on app1
• Test admin-ai endpoint on app1 via direct IP or internal DNS
• Pull needed Ollama models on app1 (llama3.2:3b, qwen2.5:7b)

Phase 1 — Cutover (when you're ready)

• Switch Hermes to fallback chain (OpenRouter deepseek-chat)
• Update DNS: admin-ai.itpropartner.com → 152.53.36.131 (TTL 60)
• Monitor admin-ai endpoint
• Run end-to-end model test through app1
• Verify vision (gemini-pro-latest) works

Phase 2 — Restore Primary

• Switch Hermes back to admin-ai primary
• Verify all 22 cron jobs pass
• Verify Anita's profile

Phase 3 — Decommission (after 3 days stable)

• Keep old box running for rollback
• After verification, decom Hetzner AI server

Detailed Steps

Step 0: Pull LiteLLM image on app1

ssh -i /root/.ssh/itpp-infra root@152.53.36.131
docker pull ghcr.io/berriai/litellm:v1.84.0

Step 1: Create LiteLLM docker-compose on app1

Create /root/docker/litellm/docker-compose.yml and /root/docker/litellm/.env:

services:
  litellm:
    image: ghcr.io/berriai/litellm:v1.84.0
    container_name: litellm
    restart: always
    environment:
      - DATABASE_URL=${DATABASE_URL}
      - STORE_MODEL_IN_DB=True
      - LITELLM_MASTER_KEY=${LITELLM_MASTER_KEY}
      - LITELLM_SALT_KEY=${LITELLM_SALT_KEY}
      - OPENAI_API_KEY=${OPENAI_API_KEY:-}
      - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY:-}
      - GEMINI_API_KEY=${GEMINI_API_KEY:-}
      - PERPLEXITYAI_API_KEY=${PERPLEXITYAI_API_KEY:-}
    volumes:
      - ./config.yaml:/app/config.yaml
    command: ["--config", "/app/config.yaml", "--port", "4000"]
    ports:
      - "127.0.0.1:4000:4000"
    networks:
      - litellm-net
    depends_on:
      litellm_postgres:
        condition: service_healthy

  litellm_postgres:
    image: postgres:16
    container_name: litellm_postgres
    restart: always
    environment:
      - POSTGRES_DB=litellm_db
      - POSTGRES_USER=litellm
      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
    volumes:
      - litellm_db_data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U litellm -d litellm_db"]
      interval: 5s
      timeout: 5s
      retries: 5
    networks:
      - litellm-net

networks:
  litellm-net:
    driver: bridge

volumes:
  litellm_db_data:

.env file:

POSTGRES_PASSWORD=litellm
DATABASE_URL=postgresql://litellm:litellm@litellm_postgres:5432/litellm_db
LITELLM_MASTER_KEY=sk-lit...c5b1
LITELLM_SALT_KEY=sk-lZ0..._v3A

config.yaml:

general_settings:
  master_key: "sk-lit...c5b1"
  use_dashboard: true
  database_url: postgresql://litellm:***@litellm_postgres:5432/litellm_db
  store_model_in_db: true
  database_connection_pool_limit: 10
  database_connection_timeout: 30
  enforce_database: true

Step 2: Export/import Postgres DB

On old server:

PGPASSWORD=litellm docker exec litellm_postgres pg_dump -U litellm -d litellm_db --no-owner > /tmp/litellm_db_dump.sql

Copy to Core, then to app1:

scp root@178.156.167.181:/tmp/litellm_db_dump.sql /tmp/
scp /tmp/litellm_db_dump.sql root@152.53.36.131:/tmp/

On app1, start Postgres first, then import:

cd /root/docker/litellm && docker compose up -d litellm_postgres
sleep 5
PGPASSWORD=litellm docker exec -i litellm_postgres psql -U litellm -d litellm_db < /tmp/litellm_db_dump.sql

Step 3: Start LiteLLM on app1

cd /root/docker/litellm && docker compose up -d litellm

Step 4: Test on app1 via loopback

curl -s http://127.0.0.1:4000/v1/models -H "Authorization: Bearer sk-lit...c5b1"

Step 5: Add Caddy config on app1

Add to /etc/caddy/Caddyfile:

admin-ai.itpropartner.com {
    reverse_proxy 127.0.0.1:4000
}

Step 6: DNS cutover

# Set TTL 60 first, wait, then update
hermes config set model.fallbacks '[...openrouter fallback...]'
# Flip DNS A record to 152.53.36.131
# Verify: curl https://admin-ai.itpropartner.com/v1/models

Rollback

If cutover fails:

1. DNS back to 178.156.167.181
2. Insert master key hash into old DB:

HASH=$(echo -n "sk-lit...c5b1" | sha256sum | cut -d' ' -f1)
PGPASSWORD=litellm docker exec -e PGPASSWORD=litellm litellm_postgres psql -U litellm -d litellm_db \
  -c "INSERT INTO \"LiteLLM_VerificationToken\" (token, key_name, user_id, spend, models) \
      VALUES ('$HASH', 'sk-...c5b1', 'default_user_id', 0, '{}') \
      ON CONFLICT (token) DO UPDATE SET key_name = 'sk-...c5b1';"

3. Switch Hermes back to admin-ai

Provider API Keys (needed for new server)

The Postgres dump carries encrypted API keys (LITELLM_SALT_KEY dependent). As long as the same SALT_KEY is used on app1, the keys migrate transparently.

Direct provider keys in old .env:

• OPENAI_API_KEY: (empty — uses OpenRouter)
• ANTHROPIC_API_KEY: (empty — uses OpenRouter)
• GEMINI_API_KEY: (empty — uses OpenRouter)
• PERPLEXITYAI_API_KEY: (empty — not used)

All model API keys are stored in Postgres LiteLLM_ProxyModelTable, encrypted with SALT_KEY.