Initial resurrection kit — 81 scripts, 62 references, configs, systemd units, crons, Docker compose files, Caddy config, master README
This commit is contained in:
root
@@ -0,0 +1,216 @@
|
||||
# Admin-AI Migration Plan: Hetzner → app1
|
||||
|
||||
**Source:** 178.156.167.181 (Hetzner CPX41, 93% disk)
|
||||
**Target:** 152.53.36.131 (app1, RS 4000, 3% disk)
|
||||
**DNS:** admin-ai.itpropartner.com
|
||||
|
||||
## Current State
|
||||
|
||||
### Source (Old AI Box)
|
||||
- LiteLLM v1.84.0 (ghcr.io/berriai/litellm) + Postgres 16
|
||||
- 39 configured models with direct API keys (Anthropic, DeepSeek, Google, OpenAI, Groq, xAI, OpenRouter)
|
||||
- Open WebUI (0.9.6) — 121 MB SQLite DB
|
||||
- Ollama — 8 local models (~10 GB), 7 cloud-proxy models
|
||||
- Qdrant vector DB — small, optional
|
||||
- Caddy reverse proxy
|
||||
- 226G disk at 93% — critical threshold
|
||||
|
||||
### Target (app1)
|
||||
- Docker + Compose v5.3.1 already installed
|
||||
- Ollama + Open WebUI + n8n already running
|
||||
- Caddy 2.6.2 active on 80/443
|
||||
- 1TB at 3% used, 32 GB RAM
|
||||
- Caddy already proxies: n8n.itpropartner.com, ai.itpropartner.com
|
||||
|
||||
## Migration Plan
|
||||
|
||||
### Phase 0 — Pre-flight (do now, before you leave for Colombia)
|
||||
|
||||
No downtime. No DNS changes. Prep work only.
|
||||
|
||||
- [ ] Pull LiteLLM image on app1
|
||||
- [ ] Set up LiteLLM + Postgres docker-compose on app1
|
||||
- [ ] Export Postgres DB from old → import to app1
|
||||
- [ ] Add Caddy config for admin-ai.itpropartner.com on app1
|
||||
- [ ] Test admin-ai endpoint on app1 via direct IP or internal DNS
|
||||
- [ ] Pull needed Ollama models on app1 (llama3.2:3b, qwen2.5:7b)
|
||||
|
||||
### Phase 1 — Cutover (when you're ready)
|
||||
|
||||
- [ ] Switch Hermes to fallback chain (OpenRouter deepseek-chat)
|
||||
- [ ] Update DNS: admin-ai.itpropartner.com → 152.53.36.131 (TTL 60)
|
||||
- [ ] Monitor admin-ai endpoint
|
||||
- [ ] Run end-to-end model test through app1
|
||||
- [ ] Verify vision (gemini-pro-latest) works
|
||||
|
||||
### Phase 2 — Restore Primary
|
||||
|
||||
- [ ] Switch Hermes back to admin-ai primary
|
||||
- [ ] Verify all 22 cron jobs pass
|
||||
- [ ] Verify Anita's profile
|
||||
|
||||
### Phase 3 — Decommission (after 3 days stable)
|
||||
|
||||
- [ ] Keep old box running for rollback
|
||||
- [ ] After verification, decom Hetzner AI server
|
||||
|
||||
## Detailed Steps
|
||||
|
||||
### Step 0: Pull LiteLLM image on app1
|
||||
|
||||
```bash
|
||||
ssh -i /root/.ssh/itpp-infra root@152.53.36.131
|
||||
docker pull ghcr.io/berriai/litellm:v1.84.0
|
||||
```
|
||||
|
||||
### Step 1: Create LiteLLM docker-compose on app1
|
||||
|
||||
Create `/root/docker/litellm/docker-compose.yml` and `/root/docker/litellm/.env`:
|
||||
|
||||
```yaml
|
||||
services:
|
||||
litellm:
|
||||
image: ghcr.io/berriai/litellm:v1.84.0
|
||||
container_name: litellm
|
||||
restart: always
|
||||
environment:
|
||||
- DATABASE_URL=${DATABASE_URL}
|
||||
- STORE_MODEL_IN_DB=True
|
||||
- LITELLM_MASTER_KEY=${LITELLM_MASTER_KEY}
|
||||
- LITELLM_SALT_KEY=${LITELLM_SALT_KEY}
|
||||
- OPENAI_API_KEY=${OPENAI_API_KEY:-}
|
||||
- ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY:-}
|
||||
- GEMINI_API_KEY=${GEMINI_API_KEY:-}
|
||||
- PERPLEXITYAI_API_KEY=${PERPLEXITYAI_API_KEY:-}
|
||||
volumes:
|
||||
- ./config.yaml:/app/config.yaml
|
||||
command: ["--config", "/app/config.yaml", "--port", "4000"]
|
||||
ports:
|
||||
- "127.0.0.1:4000:4000"
|
||||
networks:
|
||||
- litellm-net
|
||||
depends_on:
|
||||
litellm_postgres:
|
||||
condition: service_healthy
|
||||
|
||||
litellm_postgres:
|
||||
image: postgres:16
|
||||
container_name: litellm_postgres
|
||||
restart: always
|
||||
environment:
|
||||
- POSTGRES_DB=litellm_db
|
||||
- POSTGRES_USER=litellm
|
||||
- POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
|
||||
volumes:
|
||||
- litellm_db_data:/var/lib/postgresql/data
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U litellm -d litellm_db"]
|
||||
interval: 5s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
networks:
|
||||
- litellm-net
|
||||
|
||||
networks:
|
||||
litellm-net:
|
||||
driver: bridge
|
||||
|
||||
volumes:
|
||||
litellm_db_data:
|
||||
```
|
||||
|
||||
`.env` file:
|
||||
```
|
||||
POSTGRES_PASSWORD=litellm
|
||||
DATABASE_URL=postgresql://litellm:litellm@litellm_postgres:5432/litellm_db
|
||||
LITELLM_MASTER_KEY=sk-lit...c5b1
|
||||
LITELLM_SALT_KEY=sk-lZ0..._v3A
|
||||
```
|
||||
|
||||
`config.yaml`:
|
||||
```yaml
|
||||
general_settings:
|
||||
master_key: "sk-lit...c5b1"
|
||||
use_dashboard: true
|
||||
database_url: postgresql://litellm:***@litellm_postgres:5432/litellm_db
|
||||
store_model_in_db: true
|
||||
database_connection_pool_limit: 10
|
||||
database_connection_timeout: 30
|
||||
enforce_database: true
|
||||
```
|
||||
|
||||
### Step 2: Export/import Postgres DB
|
||||
|
||||
On old server:
|
||||
```bash
|
||||
PGPASSWORD=litellm docker exec litellm_postgres pg_dump -U litellm -d litellm_db --no-owner > /tmp/litellm_db_dump.sql
|
||||
```
|
||||
|
||||
Copy to Core, then to app1:
|
||||
```bash
|
||||
scp root@178.156.167.181:/tmp/litellm_db_dump.sql /tmp/
|
||||
scp /tmp/litellm_db_dump.sql root@152.53.36.131:/tmp/
|
||||
```
|
||||
|
||||
On app1, start Postgres first, then import:
|
||||
```bash
|
||||
cd /root/docker/litellm && docker compose up -d litellm_postgres
|
||||
sleep 5
|
||||
PGPASSWORD=litellm docker exec -i litellm_postgres psql -U litellm -d litellm_db < /tmp/litellm_db_dump.sql
|
||||
```
|
||||
|
||||
### Step 3: Start LiteLLM on app1
|
||||
|
||||
```bash
|
||||
cd /root/docker/litellm && docker compose up -d litellm
|
||||
```
|
||||
|
||||
### Step 4: Test on app1 via loopback
|
||||
|
||||
```bash
|
||||
curl -s http://127.0.0.1:4000/v1/models -H "Authorization: Bearer sk-lit...c5b1"
|
||||
```
|
||||
|
||||
### Step 5: Add Caddy config on app1
|
||||
|
||||
Add to /etc/caddy/Caddyfile:
|
||||
```
|
||||
admin-ai.itpropartner.com {
|
||||
reverse_proxy 127.0.0.1:4000
|
||||
}
|
||||
```
|
||||
|
||||
### Step 6: DNS cutover
|
||||
|
||||
```bash
|
||||
# Set TTL 60 first, wait, then update
|
||||
hermes config set model.fallbacks '[...openrouter fallback...]'
|
||||
# Flip DNS A record to 152.53.36.131
|
||||
# Verify: curl https://admin-ai.itpropartner.com/v1/models
|
||||
```
|
||||
|
||||
## Rollback
|
||||
|
||||
If cutover fails:
|
||||
1. DNS back to 178.156.167.181
|
||||
2. Insert master key hash into old DB:
|
||||
```bash
|
||||
HASH=$(echo -n "sk-lit...c5b1" | sha256sum | cut -d' ' -f1)
|
||||
PGPASSWORD=litellm docker exec -e PGPASSWORD=litellm litellm_postgres psql -U litellm -d litellm_db \
|
||||
-c "INSERT INTO \"LiteLLM_VerificationToken\" (token, key_name, user_id, spend, models) \
|
||||
VALUES ('$HASH', 'sk-...c5b1', 'default_user_id', 0, '{}') \
|
||||
ON CONFLICT (token) DO UPDATE SET key_name = 'sk-...c5b1';"
|
||||
```
|
||||
3. Switch Hermes back to admin-ai
|
||||
|
||||
## Provider API Keys (needed for new server)
|
||||
|
||||
The Postgres dump carries encrypted API keys (LITELLM_SALT_KEY dependent). As long as the same SALT_KEY is used on app1, the keys migrate transparently.
|
||||
|
||||
Direct provider keys in old .env:
|
||||
- OPENAI_API_KEY: (empty — uses OpenRouter)
|
||||
- ANTHROPIC_API_KEY: (empty — uses OpenRouter)
|
||||
- GEMINI_API_KEY: (empty — uses OpenRouter)
|
||||
- PERPLEXITYAI_API_KEY: (empty — not used)
|
||||
|
||||
All model API keys are stored in Postgres LiteLLM_ProxyModelTable, encrypted with SALT_KEY.
|
||||
Reference in New Issue
Block a user