1437 lines
41 KiB
Markdown
1437 lines
41 KiB
Markdown
# Core Rebalance Plan — IT Pro Partner Infrastructure
|
||
|
||
**Date:** July 12, 2026
|
||
**Author:** Sho'Nuff
|
||
**Status:** APPROVED — Ready for execution
|
||
**Target:** Reposition Core as lightweight Hermes command center; move everything else to app1
|
||
|
||
---
|
||
|
||
## 1. Executive Summary
|
||
|
||
### Current State
|
||
|
||
| Server | IP | Specs | What Runs |
|
||
|--------|----|-------|-----------|
|
||
| **Core** | 152.53.192.33 | RS 2000 (8C/15G/512GB) | Hermes, Caddy (all domains), Ops Portal, OSINT API, Super Search, SearXNG, Grafana, Komodo, Twenty CRM, DocuSeal, Vaultwarden, Tailscale, WireGuard, StrongSwan, Prometheus, Redis, Ollama (fallback), 22+ cron jobs |
|
||
| **app1** | 152.53.36.131 | RS 4000 (12C/32G/1TB) | n8n + Postgres, Ollama (primary models), Open WebUI, Caddy (basic) |
|
||
| **app1-bu** | 5.161.114.8 | CPX11 (2C/4G) | Warm standby for Core |
|
||
| **old-ai** | 178.156.167.181 | CPX41 (226G) | LiteLLM + Postgres (admin-ai) — decommissioning |
|
||
|
||
### Target State: Core — Hermes Command Center (Lightweight)
|
||
|
||
| Service | Action |
|
||
|---------|--------|
|
||
| Hermes Agent | **STAYS** — the one thing that never moves |
|
||
| LiteLLM | **NEW** — migrate FROM old-ai (178.156.167.181) |
|
||
| Ollama | **NEW** — lightweight fallback model FROM app1 (llama3.2:3b or qwen2.5:7b) |
|
||
| Caddy | **MOVES to app1** — Core becomes Caddy-less, all domains route through app1 |
|
||
| Ops Portal | **MOVES to app1** |
|
||
| OSINT API | **MOVES to app1** |
|
||
| Super Search | **MOVES to app1** |
|
||
| SearXNG | **MOVES to app1** |
|
||
| Grafana | **MOVES to app1** |
|
||
| Komodo | **MOVES to app1** (or decommission — TBD) |
|
||
| Twenty CRM | **MOVES to app1** |
|
||
| DocuSeal | **MOVES to app1** |
|
||
| Vaultwarden | **MOVES to app1** |
|
||
| Prometheus | **MOVES to app1** |
|
||
| Redis | **MOVES to app1** |
|
||
| Tailscale | **STAYS** — Core needs mesh VPN access |
|
||
| WireGuard | **STAYS** — home router tunnel |
|
||
| StrongSwan | **STAYS or MOVES** (L2TP VPN — evaluate) |
|
||
| ~22 cron jobs | **Hermes triggers stay on Core** — scripts that SSH out need SSH keys on Core |
|
||
|
||
### Target State: app1 — Service/Utility Host
|
||
|
||
App1 becomes the go-to box for everything that isn't Hermes. It already runs n8n + Ollama + Caddy. Everything moving from Core lands here.
|
||
|
||
### Target State: Rename app1-bu → core-bu
|
||
|
||
Simple rename. The server's role stays the same (warm standby for Core), but the name reflects the new reality: it stands by Core, not app1.
|
||
|
||
### DNS Changes Summary
|
||
|
||
| Domain | CURRENT → | NEW → |
|
||
|--------|-----------|-------|
|
||
| ops.itpropartner.com | Core IP | app1 IP |
|
||
| core.itpropartner.com | Core IP | app1 IP |
|
||
| sign.itpropartner.com | Core IP | app1 IP |
|
||
| crm.debtrecoveryexperts.com | Core IP | app1 IP |
|
||
| portal.debtrecoveryexperts.com | Core IP | app1 IP |
|
||
| internal.debtrecoveryexperts.com | Core IP | app1 IP |
|
||
| app.itpropartner.com | Core IP | app1 IP |
|
||
| shark.itpropartner.com | Core IP | app1 IP |
|
||
| shark.iamgmb.com | Core IP | app1 IP |
|
||
| admin-ai.itpropartner.com | old-ai IP (178.156.167.181) | **Core IP** (LiteLLM on Core) |
|
||
| ai.itpropartner.com | app1 IP | app1 IP (no change) |
|
||
| n8n.itpropartner.com | app1 IP | app1 IP (no change) |
|
||
|
||
---
|
||
|
||
## 2. Pre-Flight Requirements
|
||
|
||
### 2.1 Access Verification
|
||
|
||
- [ ] SSH key `itpp-infra` works to all servers: Core, app1, app1-bu, old-ai
|
||
- [ ] Root access on all four servers
|
||
- [ ] Cloudflare API token available for DNS changes (`~/.hermes/.env` → `CLOUDFLARE_API_TOKEN`)
|
||
- [ ] Wasabi S3 credentials available (`~/.aws/credentials`)
|
||
|
||
### 2.2 Downtime Window
|
||
|
||
- **Expected downtime:** 30–60 minutes for DNS propagation
|
||
- **Services affected during cutover:** Ops Portal, OSINT API, Super Search, DocuSeal, Twenty CRM, Vaultwarden, Grafana, SearXNG
|
||
- **Hermes unaffected** — stays on Core, continues operating throughout
|
||
- **Cron jobs may miss 1-2 ticks** during the move
|
||
- **Pre-notify:** Germaine — all web UIs will be briefly unavailable
|
||
|
||
---
|
||
|
||
## 3. Phase 1 — Prep (30 minutes)
|
||
|
||
### Step 1.1 — Snapshot Everything
|
||
|
||
```bash
|
||
# On Core — take a comprehensive snapshot of current state
|
||
ssh -i ~/.ssh/itpp-infra root@152.53.192.33
|
||
|
||
# List all systemd services
|
||
systemctl list-units --type=service --state=running > /tmp/core-services.txt
|
||
|
||
# List all Docker containers
|
||
docker ps --format "table {{.Names}}\t{{.Image}}\t{{.Ports}}\t{{.Status}}" > /tmp/core-docker.txt
|
||
|
||
# Dump Caddy config
|
||
cp /etc/caddy/Caddyfile /tmp/Caddyfile-backup-$(date +%F)
|
||
|
||
# Dump all Caddy domain blocks
|
||
grep -E '^[a-z].*\.itpropartner\.com|debtrecoveryexperts\.com|iamgmb\.com' /etc/caddy/Caddyfile > /tmp/caddy-domains.txt
|
||
|
||
# List all crontabs (system + hermes)
|
||
crontab -l > /tmp/root-crontab.txt
|
||
hermes cron list > /tmp/hermes-cron-jobs.txt
|
||
|
||
# Snapshot iptables rules
|
||
iptables-save > /tmp/iptables-rules.txt
|
||
```
|
||
|
||
### Step 1.2 — Audit app1 Readiness
|
||
|
||
```bash
|
||
ssh -i ~/.ssh/itpp-infra root@152.53.36.131
|
||
|
||
# Check Caddy version
|
||
caddy version
|
||
# Expected: v2.6.2+ (already installed)
|
||
|
||
# Check disk space
|
||
df -h /
|
||
# Should show >900GB free on 1TB disk
|
||
|
||
# Check Docker availability
|
||
docker ps
|
||
docker compose version
|
||
# Expected: Docker + Compose v5.x
|
||
|
||
# Check if Caddy is binding correctly (no Tailscale conflict on app1)
|
||
# app1 doesn't run Tailscale, so no conflict expected
|
||
```
|
||
|
||
### Step 1.3 — Pre-Stage Files on app1
|
||
|
||
**Create the target directories on app1:**
|
||
|
||
```bash
|
||
ssh -i ~/.ssh/itpp-infra root@152.53.36.131
|
||
|
||
# Create standard directories
|
||
mkdir -p /root/docker/{searxng,grafana,prometheus,vaultwarden,docuseal,twenty,komodo}
|
||
mkdir -p /root/docker/super-search
|
||
mkdir -p /opt/ops-portal
|
||
mkdir -p /opt/osint-api
|
||
mkdir -p /etc/caddy # already exists
|
||
mkdir -p /root/.hermes/scripts
|
||
```
|
||
|
||
### Step 1.4 — Document Current DNS Records
|
||
|
||
```bash
|
||
# List all A records currently pointing to Core IP (152.53.192.33)
|
||
# Use Cloudflare API to enumerate zones and records
|
||
# Zones: itpropartner.com, debtrecoveryexperts.com, iamgmb.com
|
||
|
||
# For each zone, list A records:
|
||
# cloudflare-dns list-records itpropartner.com | grep -E 'type.*A' | grep '152.53.192.33'
|
||
```
|
||
|
||
**Expected DNS records pointing to Core (152.53.192.33):**
|
||
- ops.itpropartner.com → 152.53.192.33
|
||
- core.itpropartner.com → 152.53.192.33
|
||
- sign.itpropartner.com → 152.53.192.33
|
||
- app.itpropartner.com → 152.53.192.33
|
||
- shark.itpropartner.com → 152.53.192.33
|
||
- crm.debtrecoveryexperts.com → 152.53.192.33
|
||
- portal.debtrecoveryexperts.com → 152.53.192.33
|
||
- internal.debtrecoveryexperts.com → 152.53.192.33
|
||
- shark.iamgmb.com → 152.53.192.33
|
||
|
||
### Step 1.5 — Gather Env Vars and Secrets from Core
|
||
|
||
```bash
|
||
ssh -i ~/.ssh/itpp-infra root@152.53.192.33
|
||
|
||
# Ops Portal env
|
||
cat /opt/ops-portal/.env 2>/dev/null || echo "No .env found"
|
||
|
||
# OSINT API env
|
||
cat /opt/osint-api/.env 2>/dev/null || echo "No .env found"
|
||
|
||
# Super Search env
|
||
cat /root/docker/super-search/exa.env 2>/dev/null
|
||
|
||
# SearXNG settings
|
||
cat /root/docker/searxng/searxng/settings.yml 2>/dev/null
|
||
|
||
# Grafana env
|
||
cat /root/docker/grafana/.env 2>/dev/null
|
||
|
||
# Docker compose files (for reference)
|
||
find /root/docker -name 'docker-compose.yml' -exec echo "=== {} ===" \; -exec cat {} \;
|
||
```
|
||
|
||
---
|
||
|
||
## 4. Phase 2 — Move Services to app1 (2 hours)
|
||
|
||
### CRITICAL ORDER: Move Caddy LAST in this phase. All other services go first so Caddy can proxy to them once it's active on app1. If you move Caddy first, you have no reverse proxy for the remaining moves.
|
||
|
||
### Step 2.1 — Move Ops Portal
|
||
|
||
```bash
|
||
# ON CORE
|
||
ssh -i ~/.ssh/itpp-infra root@152.53.192.33
|
||
|
||
# Stop the service
|
||
systemctl stop ops-portal
|
||
|
||
# Copy the SQLite DB (do NOT copy while running)
|
||
cp /opt/ops-portal/audit.db /tmp/audit.db.bak
|
||
|
||
# Tar the entire ops-portal directory
|
||
tar czf /tmp/ops-portal.tar.gz -C /opt ops-portal/
|
||
|
||
# Copy to app1
|
||
scp -i ~/.ssh/itpp-infra /tmp/ops-portal.tar.gz root@152.53.36.131:/tmp/
|
||
```
|
||
|
||
```bash
|
||
# ON APP1
|
||
ssh -i ~/.ssh/itpp-infra root@152.53.36.131
|
||
|
||
# Extract
|
||
tar xzf /tmp/ops-portal.tar.gz -C /opt/
|
||
|
||
# Set up Python venv (if not already)
|
||
cd /opt/ops-portal
|
||
python3 -m venv venv
|
||
source venv/bin/activate
|
||
pip install -r requirements.txt
|
||
|
||
# Create systemd service file
|
||
cat > /etc/systemd/system/ops-portal.service << 'EOF'
|
||
[Unit]
|
||
Description=Ops Portal Backend
|
||
After=network.target
|
||
|
||
[Service]
|
||
Type=simple
|
||
User=root
|
||
WorkingDirectory=/opt/ops-portal
|
||
EnvironmentFile=/opt/ops-portal/.env
|
||
ExecStart=/opt/ops-portal/venv/bin/uvicorn server:app --host 127.0.0.1 --port 8090
|
||
Restart=always
|
||
RestartSec=5
|
||
|
||
[Install]
|
||
WantedBy=multi-user.target
|
||
EOF
|
||
|
||
systemctl daemon-reload
|
||
systemctl enable ops-portal
|
||
systemctl start ops-portal
|
||
|
||
# Verify
|
||
sleep 2
|
||
curl -s http://127.0.0.1:8090/ | head -20
|
||
```
|
||
|
||
### Step 2.2 — Move OSINT API
|
||
|
||
```bash
|
||
# ON CORE
|
||
ssh -i ~/.ssh/itpp-infra root@152.53.192.33
|
||
|
||
systemctl stop osint-api
|
||
tar czf /tmp/osint-api.tar.gz -C /opt osint-api/
|
||
scp -i ~/.ssh/itpp-infra /tmp/osint-api.tar.gz root@152.53.36.131:/tmp/
|
||
```
|
||
|
||
```bash
|
||
# ON APP1
|
||
ssh -i ~/.ssh/itpp-infra root@152.53.36.131
|
||
|
||
tar xzf /tmp/osint-api.tar.gz -C /opt/
|
||
|
||
cd /opt/osint-api
|
||
python3 -m venv venv
|
||
source venv/bin/activate
|
||
pip install -r requirements.txt
|
||
|
||
cat > /etc/systemd/system/osint-api.service << 'EOF'
|
||
[Unit]
|
||
Description=OSINT Search API
|
||
After=network.target
|
||
|
||
[Service]
|
||
Type=simple
|
||
User=root
|
||
WorkingDirectory=/opt/osint-api
|
||
EnvironmentFile=/opt/osint-api/.env
|
||
ExecStart=/opt/osint-api/venv/bin/uvicorn server:app --host 127.0.0.1 --port 8100
|
||
Restart=always
|
||
RestartSec=5
|
||
|
||
[Install]
|
||
WantedBy=multi-user.target
|
||
EOF
|
||
|
||
systemctl daemon-reload
|
||
systemctl enable osint-api
|
||
systemctl start osint-api
|
||
curl -s http://127.0.0.1:8100/ | head -20
|
||
```
|
||
|
||
### Step 2.3 — Move Super Search (MCP Server)
|
||
|
||
```bash
|
||
# ON CORE
|
||
ssh -i ~/.ssh/itpp-infra root@152.53.192.33
|
||
|
||
systemctl stop super-search
|
||
|
||
# Copy source files (not the venv — rebuild on app1)
|
||
tar czf /tmp/super-search.tar.gz \
|
||
-C /root/docker/super-search \
|
||
server.py ratelimit.py exa.env super-search.service
|
||
|
||
scp -i ~/.ssh/itpp-infra /tmp/super-search.tar.gz root@152.53.36.131:/tmp/
|
||
```
|
||
|
||
```bash
|
||
# ON APP1
|
||
ssh -i ~/.ssh/itpp-infra root@152.53.36.131
|
||
|
||
tar xzf /tmp/super-search.tar.gz -C /root/docker/super-search/
|
||
|
||
cd /root/docker/super-search
|
||
python3 -m venv venv
|
||
source venv/bin/activate
|
||
pip install fastmcp httpx trafilatura python-dotenv
|
||
|
||
# Create systemd service
|
||
cp super-search.service /etc/systemd/system/
|
||
# Edit the service file to confirm paths are correct for app1
|
||
# WorkingDirectory=/root/docker/super-search
|
||
# ExecStart=/root/docker/super-search/venv/bin/python server.py
|
||
|
||
systemctl daemon-reload
|
||
systemctl enable super-search
|
||
systemctl start super-search
|
||
|
||
# Verify — test the MCP endpoint
|
||
sleep 2
|
||
curl -s http://127.0.0.1:8899/mcp 2>&1 | head -5
|
||
# Should return FastMCP session endpoint or tools list
|
||
|
||
# IMPORTANT: Super Search talks to SearXNG on localhost:8888
|
||
# SearXNG must be running on app1 before Super Search can do web_search
|
||
# (SearXNG is moved in Step 2.5 below — Super Search will use Exa/Firecrawl fallbacks until then)
|
||
```
|
||
|
||
### Step 2.4 — Move Grafana
|
||
|
||
```bash
|
||
# ON CORE
|
||
ssh -i ~/.ssh/itpp-infra root@152.53.192.33
|
||
|
||
# Stop Grafana
|
||
cd /root/docker/grafana && docker compose down
|
||
|
||
# Copy the entire grafana directory
|
||
tar czf /tmp/grafana.tar.gz -C /root/docker grafana/
|
||
scp -i ~/.ssh/itpp-infra /tmp/grafana.tar.gz root@152.53.36.131:/tmp/
|
||
```
|
||
|
||
```bash
|
||
# ON APP1
|
||
ssh -i ~/.ssh/itpp-infra root@152.53.36.131
|
||
|
||
tar xzf /tmp/grafana.tar.gz -C /root/docker/
|
||
|
||
cd /root/docker/grafana
|
||
docker compose up -d
|
||
|
||
# Verify
|
||
docker ps | grep grafana
|
||
curl -s http://127.0.0.1:3002/api/health
|
||
```
|
||
|
||
### Step 2.5 — Move SearXNG (Docker)
|
||
|
||
```bash
|
||
# ON CORE
|
||
ssh -i ~/.ssh/itpp-infra root@152.53.192.33
|
||
|
||
cd /root/docker/searxng && docker compose down
|
||
|
||
tar czf /tmp/searxng.tar.gz -C /root/docker searxng/
|
||
scp -i ~/.ssh/itpp-infra /tmp/searxng.tar.gz root@152.53.36.131:/tmp/
|
||
```
|
||
|
||
```bash
|
||
# ON APP1
|
||
ssh -i ~/.ssh/itpp-infra root@152.53.36.131
|
||
|
||
tar xzf /tmp/searxng.tar.gz -C /root/docker/
|
||
|
||
cd /root/docker/searxng
|
||
docker compose up -d
|
||
|
||
# Verify
|
||
docker ps | grep searxng
|
||
curl -s http://127.0.0.1:8888/search?q=test | head -5
|
||
```
|
||
|
||
### Step 2.6 — Move Prometheus
|
||
|
||
```bash
|
||
# ON CORE
|
||
ssh -i ~/.ssh/itpp-infra root@152.53.192.33
|
||
|
||
cd /root/docker/prometheus && docker compose down
|
||
tar czf /tmp/prometheus.tar.gz -C /root/docker prometheus/
|
||
scp -i ~/.ssh/itpp-infra /tmp/prometheus.tar.gz root@152.53.36.131:/tmp/
|
||
```
|
||
|
||
```bash
|
||
# ON APP1
|
||
ssh -i ~/.ssh/itpp-infra root@152.53.36.131
|
||
|
||
tar xzf /tmp/prometheus.tar.gz -C /root/docker/
|
||
cd /root/docker/prometheus && docker compose up -d
|
||
docker ps | grep prometheus
|
||
```
|
||
|
||
### Step 2.7 — Move Vaultwarden, DocuSeal, Twenty CRM, Komodo (Docker Services)
|
||
|
||
Repeat the pattern for each:
|
||
|
||
```bash
|
||
# ON CORE (for each service)
|
||
cd /root/docker/<service> && docker compose down
|
||
tar czf /tmp/<service>.tar.gz -C /root/docker <service>/
|
||
scp -i ~/.ssh/itpp-infra /tmp/<service>.tar.gz root@152.53.36.131:/tmp/
|
||
|
||
# ON APP1
|
||
tar xzf /tmp/<service>.tar.gz -C /root/docker/
|
||
cd /root/docker/<service> && docker compose up -d
|
||
docker ps | grep <service>
|
||
```
|
||
|
||
**Service-specific notes:**
|
||
- **DocuSeal:** Verify at `http://127.0.0.1:3000`
|
||
- **Twenty CRM:** Check Postgres + Redis + Worker + Server all healthy. Verify at `http://127.0.0.1:3001`
|
||
- **Vaultwarden:** Verify at `http://127.0.0.1:8080`
|
||
- **Komodo:** Verify Mongo + Core at `http://127.0.0.1:9120`. Note: Komodo API needs `"type":"Local"` in login POST body.
|
||
|
||
### Step 2.8 — Move Caddy to app1 (CRITICAL — Do This Last)
|
||
|
||
This is the biggest step. Caddy currently handles ALL domains on Core. Moving it requires:
|
||
|
||
1. Merging Core's Caddy domains into app1's Caddy config
|
||
2. Updating all backend targets (they now point to localhost on app1)
|
||
3. A brief DNS propagation window
|
||
|
||
```bash
|
||
# ON CORE — grab the Caddyfile
|
||
ssh -i ~/.ssh/itpp-infra root@152.53.192.33
|
||
scp /etc/caddy/Caddyfile root@152.53.36.131:/tmp/Caddyfile-core
|
||
```
|
||
|
||
```bash
|
||
# ON APP1 — merge Caddy configs
|
||
ssh -i ~/.ssh/itpp-infra root@152.53.36.131
|
||
|
||
# Backup existing Caddyfile
|
||
cp /etc/caddy/Caddyfile /etc/caddy/Caddyfile.bak-$(date +%F)
|
||
|
||
# Build the new merged Caddyfile
|
||
# app1 already has: ai.itpropartner.com → :3000, n8n.itpropartner.com → :5678
|
||
# Now add from Core (all pointing to localhost since services are on app1):
|
||
|
||
cat > /etc/caddy/Caddyfile << 'CADDY_EOF'
|
||
{
|
||
# No default_bind needed — app1 doesn't run Tailscale
|
||
email email@redacted
|
||
}
|
||
|
||
# === Existing app1 domains ===
|
||
ai.itpropartner.com {
|
||
reverse_proxy 127.0.0.1:3000
|
||
}
|
||
|
||
n8n.itpropartner.com {
|
||
reverse_proxy 127.0.0.1:5678
|
||
}
|
||
|
||
# === admin-ai — stays on THIS (app1) until LiteLLM moves to Core in Phase 3 ===
|
||
# After Phase 3, this block moves to Core's Caddy
|
||
admin-ai.itpropartner.com {
|
||
reverse_proxy 127.0.0.1:4000
|
||
}
|
||
|
||
# === Migrated from Core ===
|
||
ops.itpropartner.com {
|
||
reverse_proxy 127.0.0.1:8090
|
||
handle_path /static/* {
|
||
root * /opt/ops-portal/static
|
||
file_server
|
||
}
|
||
}
|
||
|
||
core.itpropartner.com {
|
||
root * /var/www/core
|
||
file_server
|
||
}
|
||
|
||
sign.itpropartner.com {
|
||
reverse_proxy 127.0.0.1:3000
|
||
}
|
||
|
||
crm.debtrecoveryexperts.com {
|
||
reverse_proxy 127.0.0.1:3001
|
||
}
|
||
|
||
portal.debtrecoveryexperts.com {
|
||
root * /var/www/dre-portal
|
||
file_server
|
||
}
|
||
|
||
internal.debtrecoveryexperts.com {
|
||
root * /var/www/dre-internal
|
||
file_server
|
||
}
|
||
|
||
app.itpropartner.com {
|
||
root * /var/www/app
|
||
file_server
|
||
handle /api/* {
|
||
reverse_proxy 127.0.0.1:8082
|
||
}
|
||
}
|
||
|
||
shark.itpropartner.com {
|
||
root * /var/www/shark
|
||
file_server
|
||
handle /api/* {
|
||
reverse_proxy 127.0.0.1:8083
|
||
}
|
||
}
|
||
|
||
shark.iamgmb.com {
|
||
root * /var/www/shark
|
||
file_server
|
||
handle /api/* {
|
||
reverse_proxy 127.0.0.1:8083
|
||
}
|
||
}
|
||
|
||
# Vaultwarden — tailnet-only, internal IP
|
||
vault.itpropartner.com {
|
||
reverse_proxy 127.0.0.1:8080
|
||
}
|
||
CADDY_EOF
|
||
|
||
# Validate the config
|
||
caddy validate --config /etc/caddy/Caddyfile
|
||
|
||
# If validation passes, reload
|
||
systemctl reload caddy
|
||
|
||
# Check for errors
|
||
journalctl -u caddy -n 20 --no-pager
|
||
```
|
||
|
||
### Step 2.9 — Copy Static Files to app1
|
||
|
||
```bash
|
||
# ON CORE
|
||
ssh -i ~/.ssh/itpp-infra root@152.53.192.33
|
||
|
||
# Copy static web roots
|
||
for dir in /var/www/core /var/www/dre-portal /var/www/dre-internal /var/www/app /var/www/shark; do
|
||
if [ -d "$dir" ]; then
|
||
tar czf /tmp/www-$(basename $dir).tar.gz -C /var/www $(basename $dir)/
|
||
scp -i ~/.ssh/itpp-infra /tmp/www-$(basename $dir).tar.gz root@152.53.36.131:/tmp/
|
||
fi
|
||
done
|
||
```
|
||
|
||
```bash
|
||
# ON APP1
|
||
ssh -i ~/.ssh/itpp-infra root@152.53.36.131
|
||
|
||
mkdir -p /var/www
|
||
for archive in /tmp/www-*.tar.gz; do
|
||
tar xzf "$archive" -C /var/www/
|
||
done
|
||
ls -la /var/www/
|
||
```
|
||
|
||
### Step 2.10 — Set Up Cron Scripts on app1
|
||
|
||
Hermes cron jobs are defined in Hermes's state.db and execute ON Core (where Hermes runs). The scripts themselves live on Core. But some scripts interact with services now on app1 — those need to be updated OR new app1-local services need to be aware.
|
||
|
||
**What stays on Core (Hermes runs these):**
|
||
- hermes-live-sync — S3 sync of ~/.hermes (stays on Core, Core is Hermes)
|
||
- IMAP-email-triage — email processing (Core, Hermes-managed)
|
||
- daily-tech-digest — Hermes agent job
|
||
- exotic-vehicle-scout — Hermes agent job
|
||
- All Hermes-agent cron jobs that don't touch moved services
|
||
|
||
**Scripts that need path/endpoint updates (still run on Core but target app1):**
|
||
- service-health-check → now checks app1 services
|
||
- ops-data-collector → now hits ops.itpropartner.com (app1)
|
||
- vps-threshold-check → checks both Core and app1
|
||
|
||
**Scripts to copy to app1 for local execution (if needed for app1-local services):**
|
||
|
||
```bash
|
||
ssh -i ~/.ssh/itpp-infra root@152.53.36.131
|
||
|
||
# Copy backup scripts that now run against app1's Docker volumes
|
||
mkdir -p /root/.hermes/scripts
|
||
```
|
||
|
||
**IMPORTANT: Hermes cron jobs that fire `--no_agent` scripts run the script ON THE HERMES HOST (Core). The script paths in cron job definitions reference ~/.hermes/scripts/ on Core. These stay on Core — only the script CONTENTS may need updating to reference app1 instead of localhost.**
|
||
|
||
### Step 2.11 — Verify All Services on app1
|
||
|
||
```bash
|
||
ssh -i ~/.ssh/itpp-infra root@152.53.36.131
|
||
|
||
# All systemd services running
|
||
systemctl status ops-portal osint-api super-search --no-pager -l
|
||
|
||
# All Docker containers running
|
||
docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"
|
||
|
||
# Caddy proxying
|
||
curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:8090 # Ops Portal
|
||
curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:8100 # OSINT API
|
||
curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:8899/mcp # Super Search
|
||
curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:3002/api/health # Grafana
|
||
|
||
# Disk space sanity check
|
||
df -h /
|
||
```
|
||
|
||
### Step 2.12 — Update DNS Records
|
||
|
||
**This is the cutover moment. Brief downtime (~5-15 min) for DNS propagation.**
|
||
|
||
```bash
|
||
# Use Cloudflare API to update all domains from Core IP → app1 IP
|
||
|
||
# For zone itpropartner.com:
|
||
# ops.itpropartner.com → 152.53.36.131
|
||
# core.itpropartner.com → 152.53.36.131
|
||
# sign.itpropartner.com → 152.53.36.131
|
||
# app.itpropartner.com → 152.53.36.131
|
||
# shark.itpropartner.com → 152.53.36.131
|
||
|
||
# For zone debtrecoveryexperts.com:
|
||
# crm.debtrecoveryexperts.com → 152.53.36.131
|
||
# portal.debtrecoveryexperts.com → 152.53.36.131
|
||
# internal.debtrecoveryexperts.com → 152.53.36.131
|
||
|
||
# For zone iamgmb.com:
|
||
# shark.iamgmb.com → 152.53.36.131
|
||
```
|
||
|
||
**DNS verification:**
|
||
|
||
```bash
|
||
# Check propagation (may take a few minutes)
|
||
for domain in ops.itpropartner.com core.itpropartner.com sign.itpropartner.com \
|
||
crm.debtrecoveryexperts.com shark.iamgmb.com; do
|
||
echo -n "$domain → "
|
||
dig +short A "$domain"
|
||
done
|
||
```
|
||
|
||
### Step 2.13 — Post-DNS Smoke Test
|
||
|
||
```bash
|
||
# From any external box (or Core, testing against app1 IP)
|
||
curl -s -o /dev/null -w "%{http_code}" https://ops.itpropartner.com
|
||
# Expected: 200
|
||
|
||
curl -s -o /dev/null -w "%{http_code}" https://sign.itpropartner.com
|
||
# Expected: 200
|
||
|
||
curl -s -o /dev/null -w "%{http_code}" https://crm.debtrecoveryexperts.com
|
||
# Expected: 200
|
||
|
||
# Super Search MCP endpoint via Caddy (if proxied) — test direct
|
||
curl -s http://152.53.36.131:8899/mcp | head -5
|
||
```
|
||
|
||
### Step 2.14 — Decommission Old Services on Core
|
||
|
||
Once DNS has propagated and all services are verified on app1:
|
||
|
||
```bash
|
||
ssh -i ~/.ssh/itpp-infra root@152.53.192.33
|
||
|
||
# Stop and disable moved systemd services
|
||
systemctl stop ops-portal osint-api super-search
|
||
systemctl disable ops-portal osint-api super-search
|
||
|
||
# Stop and disable Caddy (Core no longer serves web traffic)
|
||
systemctl stop caddy
|
||
systemctl disable caddy
|
||
|
||
# Stop moved Docker containers
|
||
cd /root/docker/searxng && docker compose down
|
||
cd /root/docker/grafana && docker compose down
|
||
cd /root/docker/prometheus && docker compose down
|
||
cd /root/docker/vaultwarden && docker compose down
|
||
cd /root/docker/docuseal && docker compose down
|
||
cd /root/docker/twenty && docker compose down
|
||
cd /root/docker/komodo && docker compose down
|
||
|
||
# Clean up old Caddy config (keep backup)
|
||
mv /etc/caddy/Caddyfile /etc/caddy/Caddyfile.moved-to-app1
|
||
|
||
# Verify Core is clean
|
||
systemctl list-units --type=service --state=running | grep -E 'ops-portal|osint-api|super-search|caddy'
|
||
docker ps
|
||
# Should show no moved services
|
||
|
||
# Free disk space — Core should be significantly lighter now
|
||
df -h /
|
||
```
|
||
|
||
---
|
||
|
||
## 5. Phase 3 — LiteLLM Migration to Core (30 minutes)
|
||
|
||
**Context:** LiteLLM currently runs on old-ai (178.156.167.181 — Hetzner CPX41 at 93% disk). Instead of moving it to app1 (previous plan), it goes directly to Core as the new Hermes command center's LLM proxy.
|
||
|
||
### Step 3.1 — Prep LiteLLM on Core
|
||
|
||
```bash
|
||
ssh -i ~/.ssh/itpp-infra root@152.53.192.33
|
||
|
||
# Install Docker if not already on Core (should be — Core runs Docker)
|
||
docker --version
|
||
|
||
# Pull LiteLLM image
|
||
docker pull ghcr.io/berriai/litellm:v1.84.0
|
||
|
||
# Create directory structure
|
||
mkdir -p /root/docker/litellm
|
||
```
|
||
|
||
**Create `/root/docker/litellm/docker-compose.yml`:**
|
||
|
||
```yaml
|
||
services:
|
||
litellm:
|
||
image: ghcr.io/berriai/litellm:v1.84.0
|
||
container_name: litellm
|
||
restart: unless-stopped
|
||
environment:
|
||
- DATABASE_URL=${DATABASE_URL}
|
||
- STORE_MODEL_IN_DB=True
|
||
- LITELLM_MASTER_KEY=${LITELLM_MASTER_KEY}
|
||
- LITELLM_SALT_KEY=${LITELLM_SALT_KEY}
|
||
volumes:
|
||
- ./config.yaml:/app/config.yaml
|
||
command: ["--config", "/app/config.yaml", "--port", "4000"]
|
||
ports:
|
||
- "127.0.0.1:4000:4000"
|
||
networks:
|
||
- litellm-net
|
||
depends_on:
|
||
litellm_postgres:
|
||
condition: service_healthy
|
||
deploy:
|
||
resources:
|
||
limits:
|
||
cpus: '2'
|
||
memory: 2G
|
||
reservations:
|
||
cpus: '0.5'
|
||
memory: 512M
|
||
logging:
|
||
driver: json-file
|
||
options:
|
||
max-size: "10m"
|
||
max-file: "3"
|
||
healthcheck:
|
||
test: ["CMD", "curl", "-f", "http://localhost:4000/health/liveliness"]
|
||
interval: 30s
|
||
timeout: 10s
|
||
retries: 3
|
||
|
||
litellm_postgres:
|
||
image: postgres:16
|
||
container_name: litellm_postgres
|
||
restart: unless-stopped
|
||
environment:
|
||
- POSTGRES_DB=litellm_db
|
||
- POSTGRES_USER=litellm
|
||
- POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
|
||
volumes:
|
||
- litellm_db_data:/var/lib/postgresql/data
|
||
networks:
|
||
- litellm-net
|
||
deploy:
|
||
resources:
|
||
limits:
|
||
cpus: '2'
|
||
memory: 2G
|
||
reservations:
|
||
cpus: '0.25'
|
||
memory: 512M
|
||
logging:
|
||
driver: json-file
|
||
options:
|
||
max-size: "10m"
|
||
max-file: "3"
|
||
healthcheck:
|
||
test: ["CMD-SHELL", "pg_isready -U litellm -d litellm_db"]
|
||
interval: 5s
|
||
timeout: 5s
|
||
retries: 5
|
||
|
||
networks:
|
||
litellm-net:
|
||
driver: bridge
|
||
|
||
volumes:
|
||
litellm_db_data:
|
||
```
|
||
|
||
### Step 3.2 — Get Secrets from old-ai
|
||
|
||
```bash
|
||
# SSH to old-ai and grab the .env
|
||
ssh -i ~/.ssh/itpp-infra root@178.156.167.181
|
||
|
||
# Get LiteLLM env vars
|
||
cat /root/docker/litellm/.env
|
||
# Critical values needed:
|
||
# LITELLM_MASTER_KEY
|
||
# LITELLM_SALT_KEY
|
||
# POSTGRES_PASSWORD
|
||
# DATABASE_URL
|
||
|
||
# Also grab config.yaml
|
||
cat /root/docker/litellm/config.yaml
|
||
```
|
||
|
||
### Step 3.3 — Create .env and config.yaml on Core
|
||
|
||
```bash
|
||
ssh -i ~/.ssh/itpp-infra root@152.53.192.33
|
||
|
||
# Create .env (chmod 600)
|
||
cat > /root/docker/litellm/.env << 'EOF'
|
||
POSTGRES_PASSWORD=<FROM_OLD_AI>
|
||
DATABASE_URL=postgresql://litellm:<FROM_OLD_AI>@litellm_postgres:5432/litellm_db
|
||
LITELLM_MASTER_KEY=<FROM_OLD_AI>
|
||
LITELLM_SALT_KEY=<FROM_OLD_AI>
|
||
EOF
|
||
chmod 600 /root/docker/litellm/.env
|
||
```
|
||
|
||
**Create `config.yaml`:**
|
||
|
||
```yaml
|
||
general_settings:
|
||
master_key: "<FROM_OLD_AI>"
|
||
use_dashboard: true
|
||
database_url: postgresql://litellm:<FROM_OLD_AI>@litellm_postgres:5432/litellm_db
|
||
store_model_in_db: true
|
||
database_connection_pool_limit: 10
|
||
database_connection_timeout: 30
|
||
enforce_database: true
|
||
```
|
||
|
||
### Step 3.4 — Dump and Import Postgres DB
|
||
|
||
```bash
|
||
# ON OLD-AI (178.156.167.181)
|
||
ssh -i ~/.ssh/itpp-infra root@178.156.167.181
|
||
|
||
cd /root/docker/litellm
|
||
PGPASSWORD=<POSTGRES_PASSWORD> docker exec litellm_postgres \
|
||
pg_dump -U litellm -d litellm_db --no-owner > /tmp/litellm_db_dump.sql
|
||
|
||
# Copy to Core via scp (Core has SSH to old-ai)
|
||
scp -i ~/.ssh/itpp-infra /tmp/litellm_db_dump.sql root@152.53.192.33:/tmp/
|
||
```
|
||
|
||
```bash
|
||
# ON CORE
|
||
ssh -i ~/.ssh/itpp-infra root@152.53.192.33
|
||
|
||
# Start Postgres first
|
||
cd /root/docker/litellm
|
||
docker compose up -d litellm_postgres
|
||
sleep 5
|
||
|
||
# Import the dump
|
||
PGPASSWORD=<POSTGRES_PASSWORD> docker exec -i litellm_postgres \
|
||
psql -U litellm -d litellm_db < /tmp/litellm_db_dump.sql
|
||
|
||
# Verify import
|
||
PGPASSWORD=<POSTGRES_PASSWORD> docker exec litellm_postgres \
|
||
psql -U litellm -d litellm_db -c "SELECT COUNT(*) FROM \"LiteLLM_ProxyModelTable\";"
|
||
# Should show ~39 models
|
||
```
|
||
|
||
### Step 3.5 — Start LiteLLM on Core
|
||
|
||
```bash
|
||
ssh -i ~/.ssh/itpp-infra root@152.53.192.33
|
||
|
||
cd /root/docker/litellm
|
||
docker compose up -d litellm
|
||
|
||
# Wait for health
|
||
sleep 10
|
||
docker ps | grep litellm
|
||
# Should show (healthy)
|
||
|
||
# Test from loopback
|
||
curl -s http://127.0.0.1:4000/v1/models \
|
||
-H "Authorization: Bearer $(grep LITELLM_MASTER_KEY /root/docker/litellm/.env | cut -d= -f2)" | python3 -m json.tool | head -30
|
||
|
||
# Test a real completion
|
||
curl -s http://127.0.0.1:4000/v1/chat/completions \
|
||
-H "Authorization: Bearer $(grep LITELLM_MASTER_KEY /root/docker/litellm/.env | cut -d= -f2)" \
|
||
-H "Content-Type: application/json" \
|
||
-d '{"model":"openrouter/deepseek/deepseek-chat","messages":[{"role":"user","content":"Say hello in one word"}]}' | python3 -c "import sys,json; print(json.load(sys.stdin)['choices'][0]['message']['content'])"
|
||
```
|
||
|
||
### Step 3.6 — Setup Caddy on Core for admin-ai
|
||
|
||
Core needs Caddy back — but ONLY for admin-ai.itpropartner.com (LiteLLM proxy). Not for all domains.
|
||
|
||
```bash
|
||
ssh -i ~/.ssh/itpp-infra root@152.53.192.33
|
||
|
||
# Install Caddy if removed (it was systemd, may still be installed)
|
||
which caddy || apt-get install -y caddy
|
||
|
||
# Minimal Caddyfile — only admin-ai
|
||
cat > /etc/caddy/Caddyfile << 'EOF'
|
||
{
|
||
email email@redacted
|
||
}
|
||
|
||
admin-ai.itpropartner.com {
|
||
reverse_proxy 127.0.0.1:4000
|
||
}
|
||
EOF
|
||
|
||
caddy validate --config /etc/caddy/Caddyfile
|
||
systemctl enable caddy
|
||
systemctl start caddy
|
||
|
||
# Verify
|
||
curl -s https://admin-ai.itpropartner.com/v1/models -H "Authorization: Bearer <LITELLM_MASTER_KEY>" | python3 -m json.tool | head -20
|
||
```
|
||
|
||
### Step 3.7 — Update DNS for admin-ai
|
||
|
||
```bash
|
||
# Change admin-ai.itpropartner.com A record:
|
||
# FROM: 152.53.36.131 (app1)
|
||
# TO: 152.53.192.33 (Core)
|
||
# TTL: 60 (reduce first, then update)
|
||
|
||
# Wait for propagation
|
||
dig +short A admin-ai.itpropartner.com
|
||
# Should resolve to 152.53.192.33
|
||
```
|
||
|
||
### Step 3.8 — Wire Hermes to Use Local LiteLLM
|
||
|
||
```bash
|
||
ssh -i ~/.ssh/itpp-infra root@152.53.192.33
|
||
|
||
# Update Hermes config to use local LiteLLM
|
||
hermes config set providers.admin-ai.base_url "https://admin-ai.itpropartner.com/v1"
|
||
hermes config set providers.admin-ai.api_key "<LITELLM_MASTER_KEY>"
|
||
|
||
# Verify
|
||
hermes config check
|
||
hermes gateway restart
|
||
|
||
# Test model access
|
||
curl -s https://admin-ai.itpropartner.com/v1/models \
|
||
-H "Authorization: Bearer <LITELLM_MASTER_KEY>" | python3 -c "
|
||
import sys, json
|
||
models = json.load(sys.stdin).get('data', [])
|
||
print(f'Models available: {len(models)}')
|
||
for m in models[:5]:
|
||
print(f' {m[\"id\"]}')
|
||
print(' ...')
|
||
"
|
||
```
|
||
|
||
### Step 3.9 — Remove admin-ai from app1 Caddy
|
||
|
||
```bash
|
||
ssh -i ~/.ssh/itpp-infra root@152.53.36.131
|
||
|
||
# Remove the admin-ai block from app1 Caddyfile
|
||
# (It was a placeholder — now Core handles it)
|
||
# Edit /etc/caddy/Caddyfile, remove the admin-ai block
|
||
caddy validate --config /etc/caddy/Caddyfile
|
||
systemctl reload caddy
|
||
```
|
||
|
||
---
|
||
|
||
## 6. Phase 4 — Ollama Migration to Core (15 minutes)
|
||
|
||
### Step 6.1 — Pull Lightweight Model on Core
|
||
|
||
```bash
|
||
ssh -i ~/.ssh/itpp-infra root@152.53.192.33
|
||
|
||
# Install Ollama if not already on Core
|
||
which ollama || curl -fsSL https://ollama.com/install.sh | sh
|
||
|
||
# Check current status
|
||
systemctl status ollama
|
||
ollama list
|
||
|
||
# Pull lightweight fallback model
|
||
ollama pull llama3.2:3b
|
||
# OR: ollama pull qwen2.5:7b
|
||
|
||
# Verify
|
||
ollama list
|
||
curl -s http://127.0.0.1:11434/api/tags | python3 -m json.tool
|
||
```
|
||
|
||
### Step 6.2 — Update Hermes Fallback Chain
|
||
|
||
```bash
|
||
ssh -i ~/.ssh/itpp-infra root@152.53.192.33
|
||
|
||
# Update Hermes model fallbacks to chain:
|
||
# Primary: admin-ai (LiteLLM on localhost) → Fallback: Ollama (local)
|
||
hermes config set model.fallbacks '[{"provider":"admin-ai","model":"openrouter/deepseek/deepseek-chat"},{"provider":"ollama","model":"llama3.2:3b"}]'
|
||
|
||
# Verify config
|
||
hermes config check
|
||
hermes gateway restart
|
||
```
|
||
|
||
### Step 6.3 — Test Fallback Chain
|
||
|
||
```bash
|
||
ssh -i ~/.ssh/itpp-infra root@152.53.192.33
|
||
|
||
# Test primary (LiteLLM → OpenRouter → DeepSeek)
|
||
curl -s http://127.0.0.1:4000/v1/chat/completions \
|
||
-H "Authorization: Bearer <LITELLM_MASTER_KEY>" \
|
||
-H "Content-Type: application/json" \
|
||
-d '{"model":"openrouter/deepseek/deepseek-chat","messages":[{"role":"user","content":"Hi"}]}' | python3 -c "import sys,json; print(json.load(sys.stdin)['choices'][0]['message']['content'][:50])"
|
||
|
||
# Test fallback (local Ollama)
|
||
curl -s http://127.0.0.1:11434/api/generate \
|
||
-d '{"model":"llama3.2:3b","prompt":"Say hello","stream":false}' | python3 -c "import sys,json; print(json.load(sys.stdin)['response'][:50])"
|
||
```
|
||
|
||
### Step 6.4 — app1 Ollama (Keep Running)
|
||
|
||
App1's Ollama stays running — it serves Open WebUI and future AI workloads. Core only gets the lightweight fallback. Primary models stay on app1.
|
||
|
||
---
|
||
|
||
## 7. Phase 5 — Rename app1-bu → core-bu (15 minutes)
|
||
|
||
### Step 7.1 — Update Documentation References
|
||
|
||
**Files to update:**
|
||
|
||
| File | Change |
|
||
|------|--------|
|
||
| `/root/.hermes/references/master-apps-services.md` | `app1-bu` → `core-bu` throughout |
|
||
| `/root/.hermes/references/server-architecture-plan` (skill) | Update server roles table |
|
||
| `/root/.hermes/references/1-DR-Master-Plan-v2.md` | Update references |
|
||
| `/root/.hermes/references/2-Backup-Security-Standard.md` | Update references |
|
||
| `/root/.hermes/references/3-Per-Server-Runbooks.md` | Update runbook title |
|
||
| `/root/.hermes/references/4-DR-Testing-Schedule.md` | Update references |
|
||
| `/root/.hermes/references/core-rebalance-plan.md` | This document — update after execution |
|
||
| `/root/.hermes/skills/devops/server-architecture-plan/SKILL.md` | Update server roles |
|
||
| `/root/.hermes/skills/devops/hermes-backup/SKILL.md` | Update references |
|
||
| `/root/.hermes/skills/autonomous-ai-agents/hermes-migration/SKILL.md` | Update references |
|
||
|
||
### Step 7.2 — Update Warm Standby Configuration
|
||
|
||
```bash
|
||
ssh -i ~/.ssh/itpp-infra root@5.161.114.8
|
||
|
||
# Update hostname on the box itself
|
||
hostnamectl set-hostname core-bu
|
||
|
||
# Update /etc/hosts
|
||
sed -i 's/app1-bu/core-bu/g' /etc/hosts
|
||
|
||
# Verify
|
||
hostname
|
||
hostname -f
|
||
```
|
||
|
||
### Step 7.3 — Update DR Plan References
|
||
|
||
The warm standby (now core-bu) watches Core. Its role doesn't change functionally — it's still the failover for Core. The rename is cosmetic + clarity:
|
||
|
||
- Old: "app1-bu stands by app1" (confusing)
|
||
- New: "core-bu stands by Core" (clear)
|
||
|
||
---
|
||
|
||
## 8. Phase 6 — Full Verification (15 minutes)
|
||
|
||
### Step 8.1 — Hermes Health Check
|
||
|
||
```bash
|
||
ssh -i ~/.ssh/itpp-infra root@152.53.192.33
|
||
|
||
# Gateway status
|
||
hermes gateway status
|
||
|
||
# Model access
|
||
hermes config check
|
||
|
||
# Cron jobs
|
||
hermes cron list
|
||
# Should show all ~22 jobs
|
||
|
||
# Memory store
|
||
python3 -c "
|
||
import sqlite3
|
||
db = sqlite3.connect('/root/.hermes/memory_store.db')
|
||
c = db.cursor()
|
||
print('Memory tables:', [t[0] for t in c.execute(\"SELECT name FROM sqlite_master WHERE type='table'\").fetchall()])
|
||
"
|
||
```
|
||
|
||
### Step 8.2 — app1 Service Health Check
|
||
|
||
```bash
|
||
ssh -i ~/.ssh/itpp-infra root@152.53.36.131
|
||
|
||
# All systemd services
|
||
systemctl is-active ops-portal osint-api super-search caddy
|
||
|
||
# All Docker containers
|
||
docker ps --format "table {{.Names}}\t{{.Status}}"
|
||
|
||
# Caddy status
|
||
systemctl status caddy --no-pager -l | head -20
|
||
|
||
# Disk usage
|
||
df -h /
|
||
free -h
|
||
|
||
# Test all web endpoints
|
||
for endpoint in \
|
||
"https://ops.itpropartner.com" \
|
||
"https://sign.itpropartner.com" \
|
||
"https://crm.debtrecoveryexperts.com" \
|
||
"https://core.itpropartner.com" \
|
||
"https://n8n.itpropartner.com" \
|
||
"https://ai.itpropartner.com"; do
|
||
code=$(curl -s -o /dev/null -w "%{http_code}" "$endpoint")
|
||
echo "$endpoint → $code"
|
||
done
|
||
```
|
||
|
||
### Step 8.3 — LiteLLM / admin-ai Verification
|
||
|
||
```bash
|
||
# Test from external
|
||
curl -s https://admin-ai.itpropartner.com/v1/models \
|
||
-H "Authorization: Bearer <LITELLM_MASTER_KEY>" | python3 -c "
|
||
import sys, json
|
||
data = json.load(sys.stdin)
|
||
models = data.get('data', [])
|
||
print(f'admin-ai models: {len(models)}')
|
||
openrouter_models = [m['id'] for m in models if 'openrouter' in m['id']]
|
||
print(f'OpenRouter models: {len(openrouter_models)}')
|
||
"
|
||
|
||
# Test a completion through LiteLLM
|
||
curl -s https://admin-ai.itpropartner.com/v1/chat/completions \
|
||
-H "Authorization: Bearer <LITELLM_MASTER_KEY>" \
|
||
-H "Content-Type: application/json" \
|
||
-d '{"model":"openrouter/deepseek/deepseek-chat","messages":[{"role":"user","content":"Say test"}]}' | python3 -c "
|
||
import sys, json
|
||
r = json.load(sys.stdin)
|
||
print('Response:', r['choices'][0]['message']['content'][:50])
|
||
print('Model:', r.get('model','?'))
|
||
"
|
||
```
|
||
|
||
### Step 8.4 — Super Search via Hermes
|
||
|
||
```bash
|
||
# On Core (where Hermes runs), update MCP endpoint for Super Search
|
||
ssh -i ~/.ssh/itpp-infra root@152.53.192.33
|
||
|
||
# Verify Super Search MCP is reachable from Core
|
||
curl -s http://152.53.36.131:8899/mcp | head -5
|
||
|
||
# Update Hermes MCP config to point to app1
|
||
# The mcp configuration in ~/.hermes/config.yaml should reference:
|
||
# super-search:
|
||
# url: http://152.53.36.131:8899/mcp
|
||
|
||
# If using Tailscale, could also use tailnet IP for private routing:
|
||
# url: http://<app1-tailnet-ip>:8899/mcp
|
||
```
|
||
|
||
### Step 8.5 — Backup Verification
|
||
|
||
```bash
|
||
# Verify daily backup still works (runs on Core)
|
||
ssh -i ~/.ssh/itpp-infra root@152.53.192.33
|
||
|
||
source /opt/awscli-venv/bin/activate
|
||
|
||
# Check latest full backup
|
||
aws s3 ls s3://hermes-vps-backups/hermes-full-backup/ \
|
||
--endpoint-url https://s3.us-east-1.wasabisys.com/ | tail -3
|
||
|
||
# Check live sync
|
||
aws s3 ls s3://hermes-vps-backups/live/ \
|
||
--endpoint-url https://s3.us-east-1.wasabisys.com/ | head -5
|
||
|
||
# Check system configs (should now reflect Core-only services)
|
||
aws s3 ls s3://itpropartner-system-configs/ \
|
||
--endpoint-url https://s3.us-east-1.wasabisys.com/
|
||
```
|
||
|
||
### Step 8.6 — Cron Job Verification
|
||
|
||
Run a few critical cron jobs manually to verify they still work post-move:
|
||
|
||
```bash
|
||
ssh -i ~/.ssh/itpp-infra root@152.53.192.33
|
||
|
||
# Service health check (should check app1 now)
|
||
bash /root/.hermes/scripts/service-health-check.sh
|
||
|
||
# VPS threshold check
|
||
bash /root/.hermes/scripts/vps-threshold-check.sh
|
||
|
||
# Ops data collector (if it hits ops.itpropartner.com which now points to app1)
|
||
bash /root/.hermes/scripts/ops-data-collector.sh 2>&1 | head -20
|
||
```
|
||
|
||
---
|
||
|
||
## 9. Cleanup & Decommission
|
||
|
||
### Step 9.1 — Core Cleanup
|
||
|
||
```bash
|
||
ssh -i ~/.ssh/itpp-infra root@152.53.192.33
|
||
|
||
# Remove old Docker volumes for moved services
|
||
docker volume prune -f
|
||
|
||
# Remove old service directories (keep for 7 days as backup)
|
||
find /root/docker -mindepth 1 -maxdepth 1 -type d | while read dir; do
|
||
case $(basename "$dir") in
|
||
litellm|ollama) echo "KEEP: $dir" ;;
|
||
*) echo "CAN REMOVE (after 7 days): $dir" ;;
|
||
esac
|
||
done
|
||
|
||
# Remove old /opt directories
|
||
# /opt/ops-portal, /opt/osint-api — keep for 7 days then remove
|
||
|
||
# Verify disk space reclaimed
|
||
df -h /
|
||
```
|
||
|
||
### Step 9.2 — old-ai Decommission (after 3 days stable)
|
||
|
||
```bash
|
||
# After admin-ai has been stable on Core for 3+ days:
|
||
# 1. Stop all containers on old-ai (178.156.167.181)
|
||
# 2. Take final snapshot
|
||
# 3. Delete Hetzner CPX41 server
|
||
# This saves ~$30-40/mo
|
||
```
|
||
|
||
### Step 9.3 — Update master-apps-services.md
|
||
|
||
The master inventory at `/root/.hermes/references/master-apps-services.md` must be fully rewritten to reflect the new architecture. Key changes:
|
||
|
||
- Core: Hermes, LiteLLM, Ollama, Tailscale, WireGuard, StrongSwan, Redis
|
||
- app1: Caddy (all domains), Ops Portal, OSINT API, Super Search, SearXNG, Grafana, Prometheus, Vaultwarden, DocuSeal, Twenty CRM, Komodo, n8n, Open WebUI, Ollama (primary)
|
||
- core-bu (formerly app1-bu): Warm standby for Core
|
||
|
||
---
|
||
|
||
## 10. Rollback Plan
|
||
|
||
If the migration fails at any phase:
|
||
|
||
### Rollback: Services on app1 not working
|
||
|
||
1. Re-point DNS for affected domains back to Core IP (152.53.192.33)
|
||
2. Start services back on Core: `systemctl start caddy ops-portal osint-api super-search`
|
||
3. Start Docker containers on Core: `cd /root/docker/<service> && docker compose up -d`
|
||
4. Verify services respond on Core
|
||
5. Debug app1 issues separately — Core continues serving
|
||
|
||
### Rollback: LiteLLM on Core not working
|
||
|
||
1. Re-point admin-ai.itpropartner.com back to app1 IP (152.53.36.131)
|
||
2. Or: re-point to old-ai IP (178.156.167.181) if old-ai is still running
|
||
3. Update Hermes config: `hermes config set providers.admin-ai.base_url "https://admin-ai.itpropartner.com/v1"`
|
||
4. Restart Hermes gateway
|
||
5. Debug Core LiteLLM separately
|
||
|
||
### Rollback: Ollama on Core not working
|
||
|
||
1. Update Hermes fallback to remove ollama from chain
|
||
2. Or: point fallback to app1 Ollama at `http://152.53.36.131:11434`
|
||
|
||
---
|
||
|
||
## 11. Key Pitfalls
|
||
|
||
### DNS Propagation Split-Brain
|
||
- **Risk:** During the 5-15 minute DNS propagation window, some clients hit Core, some hit app1
|
||
- **Mitigation:** Core's Caddy can be left running during transition (serving stale content is better than errors). Stop Core Caddy only AFTER DNS has fully propagated
|
||
|
||
### SQLite DB Corruption
|
||
- **Risk:** Copying Ops Portal's audit.db while the service is running
|
||
- **Mitigation:** `systemctl stop ops-portal` BEFORE copying. Verified in step order above.
|
||
|
||
### Super Search → SearXNG Dependency
|
||
- **Risk:** Super Search on app1 can't reach SearXNG until SearXNG is also on app1
|
||
- **Mitigation:** Move SearXNG first (Step 2.5 before Step 2.3 restart). Super Search falls back to Exa/Firecrawl if SearXNG is unreachable.
|
||
|
||
### SSH Keys for Cron Scripts
|
||
- **Risk:** Cron scripts on Core SSH to other servers. If keys are moved, scripts break.
|
||
- **Mitigation:** SSH keys stay on Core. Scripts that SSH from Core to app1 still work because Core can SSH to app1 (`itpp-infra` key).
|
||
|
||
### Caddy Let's Encrypt Rate Limits
|
||
- **Risk:** Moving many domains at once can trigger Let's Encrypt rate limits (5 failures per domain per hour)
|
||
- **Mitigation:** Ensure DNS is pointing to app1 BEFORE Caddy tries to issue certs. Test DNS first: `dig +short <domain>` must return app1 IP.
|
||
|
||
### Hermes MCP Connection
|
||
- **Risk:** Hermes on Core can't reach Super Search if the MCP URL points to localhost
|
||
- **Mitigation:** Update MCP URL in Hermes config to `http://152.53.36.131:8899/mcp` (app1's IP)
|
||
|
||
### Ollama GPU / CPU
|
||
- **Risk:** Core (RS 2000) may not have GPU for Ollama. CPU-only inference on llama3.2:3b is slow.
|
||
- **Mitigation:** This is the FALLBACK model — only used if LiteLLM is down. Slow is acceptable for fallback. Primary inference always goes through LiteLLM → OpenRouter.
|
||
|
||
### Twenty CRM Database
|
||
- **Risk:** Twenty CRM has Postgres + Redis + Worker containers. All must move together.
|
||
- **Mitigation:** Move the entire `/root/docker/twenty/` directory. Docker compose brings up all services together.
|
||
|
||
---
|
||
|
||
## 12. Timeline Summary
|
||
|
||
| Phase | Duration | Key Actions |
|
||
|-------|----------|-------------|
|
||
| Phase 1: Prep | 30 min | Snapshot, audit, pre-stage files |
|
||
| Phase 2: Move | 2 hrs | Move all services to app1, update DNS |
|
||
| Phase 3: LiteLLM | 30 min | Migrate from old-ai to Core, wire Hermes |
|
||
| Phase 4: Ollama | 15 min | Pull lightweight model on Core |
|
||
| Phase 5: Rename | 15 min | app1-bu → core-bu, update docs |
|
||
| Phase 6: Verify | 15 min | Full smoke test |
|
||
| **Total** | **~3.5 hrs** | |
|
||
|
||
**Best execution window:** Saturday or Sunday morning (lowest traffic, cron job impact minimized)
|
||
|
||
---
|
||
|
||
## 13. Post-Migration Architecture
|
||
|
||
### Core (152.53.192.33) — The Command Center
|
||
```
|
||
Hermes Agent ← systemd, port 8081 (the brain)
|
||
Caddy (minimal) ← systemd, port 80/443 — ONLY admin-ai.itpropartner.com
|
||
LiteLLM ← Docker, port 4000 — LLM proxy with 39 models
|
||
Ollama ← systemd, port 11434 — llama3.2:3b fallback
|
||
Tailscale ← systemd — mesh VPN
|
||
WireGuard ← systemd — home router tunnel
|
||
StrongSwan ← systemd — L2TP/IPsec VPN (TBD)
|
||
Redis ← systemd — cache
|
||
Node Exporter ← systemd — metrics
|
||
Hermes Browser ← systemd — Chromium headless
|
||
```
|
||
|
||
### app1 (152.53.36.131) — The Service Host
|
||
```
|
||
Caddy ← systemd, port 80/443 — ALL public domains
|
||
Ops Portal ← systemd, port 8090 (FastAPI)
|
||
OSINT API ← systemd, port 8100 (FastAPI)
|
||
Super Search ← systemd, port 8899 (FastMCP)
|
||
SearXNG ← Docker, port 8888
|
||
Grafana ← Docker, port 3002
|
||
Prometheus ← Docker, port 9090
|
||
Vaultwarden ← Docker, port 8080
|
||
DocuSeal ← Docker, port 3000
|
||
Twenty CRM ← Docker, port 3001 + Postgres + Redis + Worker
|
||
Komodo ← Docker, port 9120 + Mongo
|
||
n8n ← Docker, port 5678 + Postgres
|
||
Open WebUI ← Docker, port 3000 (ai.itpropartner.com)
|
||
Ollama (primary) ← Docker, port 11434 (full models)
|
||
```
|
||
|
||
### core-bu (5.161.114.8) — Warm Standby
|
||
```
|
||
Hermes (dormant) ← systemd — auto-activates if Core is down
|
||
Ollama ← systemd — fallback
|
||
Tailscale ← systemd
|
||
StrongSwan ← systemd
|
||
Watchdog ← system crontab — checks Core every 5 min
|
||
```
|
||
|
||
---
|
||
|
||
**END OF PLAN**
|