Files

T

root ae056eaf83 Initial resurrection kit — 81 scripts, 62 references, configs, systemd units, crons, Docker compose files, Caddy config, master README

2026-07-15 17:45:36 -04:00

41 KiB

Raw Blame History

Core Rebalance Plan — IT Pro Partner Infrastructure

Date: July 12, 2026 Author: Sho'Nuff Status: APPROVED — Ready for execution Target: Reposition Core as lightweight Hermes command center; move everything else to app1

1. Executive Summary

Current State

Server	IP	Specs	What Runs
Core	152.53.192.33	RS 2000 (8C/15G/512GB)	Hermes, Caddy (all domains), Ops Portal, OSINT API, Super Search, SearXNG, Grafana, Komodo, Twenty CRM, DocuSeal, Vaultwarden, Tailscale, WireGuard, StrongSwan, Prometheus, Redis, Ollama (fallback), 22+ cron jobs
app1	152.53.36.131	RS 4000 (12C/32G/1TB)	n8n + Postgres, Ollama (primary models), Open WebUI, Caddy (basic)
app1-bu	5.161.114.8	CPX11 (2C/4G)	Warm standby for Core
old-ai	178.156.167.181	CPX41 (226G)	LiteLLM + Postgres (admin-ai) — decommissioning

Target State: Core — Hermes Command Center (Lightweight)

Service	Action
Hermes Agent	STAYS — the one thing that never moves
LiteLLM	NEW — migrate FROM old-ai (178.156.167.181)
Ollama	NEW — lightweight fallback model FROM app1 (llama3.2:3b or qwen2.5:7b)
Caddy	MOVES to app1 — Core becomes Caddy-less, all domains route through app1
Ops Portal	MOVES to app1
OSINT API	MOVES to app1
Super Search	MOVES to app1
SearXNG	MOVES to app1
Grafana	MOVES to app1
Komodo	MOVES to app1 (or decommission — TBD)
Twenty CRM	MOVES to app1
DocuSeal	MOVES to app1
Vaultwarden	MOVES to app1
Prometheus	MOVES to app1
Redis	MOVES to app1
Tailscale	STAYS — Core needs mesh VPN access
WireGuard	STAYS — home router tunnel
StrongSwan	STAYS or MOVES (L2TP VPN — evaluate)
~22 cron jobs	Hermes triggers stay on Core — scripts that SSH out need SSH keys on Core

Target State: app1 — Service/Utility Host

App1 becomes the go-to box for everything that isn't Hermes. It already runs n8n + Ollama + Caddy. Everything moving from Core lands here.

Target State: Rename app1-bu → core-bu

Simple rename. The server's role stays the same (warm standby for Core), but the name reflects the new reality: it stands by Core, not app1.

DNS Changes Summary

Domain	CURRENT →	NEW →
ops.itpropartner.com	Core IP	app1 IP
core.itpropartner.com	Core IP	app1 IP
sign.itpropartner.com	Core IP	app1 IP
crm.debtrecoveryexperts.com	Core IP	app1 IP
portal.debtrecoveryexperts.com	Core IP	app1 IP
internal.debtrecoveryexperts.com	Core IP	app1 IP
app.itpropartner.com	Core IP	app1 IP
shark.itpropartner.com	Core IP	app1 IP
shark.iamgmb.com	Core IP	app1 IP
admin-ai.itpropartner.com	old-ai IP (178.156.167.181)	Core IP (LiteLLM on Core)
ai.itpropartner.com	app1 IP	app1 IP (no change)
n8n.itpropartner.com	app1 IP	app1 IP (no change)

2. Pre-Flight Requirements

2.1 Access Verification

SSH key itpp-infra works to all servers: Core, app1, app1-bu, old-ai
Root access on all four servers
Cloudflare API token available for DNS changes (~/.hermes/.env → CLOUDFLARE_API_TOKEN)
Wasabi S3 credentials available (~/.aws/credentials)

2.2 Downtime Window

Expected downtime: 30–60 minutes for DNS propagation
Services affected during cutover: Ops Portal, OSINT API, Super Search, DocuSeal, Twenty CRM, Vaultwarden, Grafana, SearXNG
Hermes unaffected — stays on Core, continues operating throughout
Cron jobs may miss 1-2 ticks during the move
Pre-notify: Germaine — all web UIs will be briefly unavailable

3. Phase 1 — Prep (30 minutes)

Step 1.1 — Snapshot Everything

# On Core — take a comprehensive snapshot of current state
ssh -i ~/.ssh/itpp-infra root@152.53.192.33

# List all systemd services
systemctl list-units --type=service --state=running > /tmp/core-services.txt

# List all Docker containers
docker ps --format "table {{.Names}}\t{{.Image}}\t{{.Ports}}\t{{.Status}}" > /tmp/core-docker.txt

# Dump Caddy config
cp /etc/caddy/Caddyfile /tmp/Caddyfile-backup-$(date +%F)

# Dump all Caddy domain blocks
grep -E '^[a-z].*\.itpropartner\.com|debtrecoveryexperts\.com|iamgmb\.com' /etc/caddy/Caddyfile > /tmp/caddy-domains.txt

# List all crontabs (system + hermes)
crontab -l > /tmp/root-crontab.txt
hermes cron list > /tmp/hermes-cron-jobs.txt

# Snapshot iptables rules
iptables-save > /tmp/iptables-rules.txt

Step 1.2 — Audit app1 Readiness

ssh -i ~/.ssh/itpp-infra root@152.53.36.131

# Check Caddy version
caddy version
# Expected: v2.6.2+ (already installed)

# Check disk space
df -h /
# Should show >900GB free on 1TB disk

# Check Docker availability
docker ps
docker compose version
# Expected: Docker + Compose v5.x

# Check if Caddy is binding correctly (no Tailscale conflict on app1)
# app1 doesn't run Tailscale, so no conflict expected

Step 1.3 — Pre-Stage Files on app1

Create the target directories on app1:

ssh -i ~/.ssh/itpp-infra root@152.53.36.131

# Create standard directories
mkdir -p /root/docker/{searxng,grafana,prometheus,vaultwarden,docuseal,twenty,komodo}
mkdir -p /root/docker/super-search
mkdir -p /opt/ops-portal
mkdir -p /opt/osint-api
mkdir -p /etc/caddy  # already exists
mkdir -p /root/.hermes/scripts

Step 1.4 — Document Current DNS Records

# List all A records currently pointing to Core IP (152.53.192.33)
# Use Cloudflare API to enumerate zones and records
# Zones: itpropartner.com, debtrecoveryexperts.com, iamgmb.com

# For each zone, list A records:
# cloudflare-dns list-records itpropartner.com | grep -E 'type.*A' | grep '152.53.192.33'

Expected DNS records pointing to Core (152.53.192.33):

ops.itpropartner.com → 152.53.192.33
core.itpropartner.com → 152.53.192.33
sign.itpropartner.com → 152.53.192.33
app.itpropartner.com → 152.53.192.33
shark.itpropartner.com → 152.53.192.33
crm.debtrecoveryexperts.com → 152.53.192.33
portal.debtrecoveryexperts.com → 152.53.192.33
internal.debtrecoveryexperts.com → 152.53.192.33
shark.iamgmb.com → 152.53.192.33

Step 1.5 — Gather Env Vars and Secrets from Core

ssh -i ~/.ssh/itpp-infra root@152.53.192.33

# Ops Portal env
cat /opt/ops-portal/.env 2>/dev/null || echo "No .env found"

# OSINT API env  
cat /opt/osint-api/.env 2>/dev/null || echo "No .env found"

# Super Search env
cat /root/docker/super-search/exa.env 2>/dev/null

# SearXNG settings
cat /root/docker/searxng/searxng/settings.yml 2>/dev/null

# Grafana env
cat /root/docker/grafana/.env 2>/dev/null

# Docker compose files (for reference)
find /root/docker -name 'docker-compose.yml' -exec echo "=== {} ===" \; -exec cat {} \;

4. Phase 2 — Move Services to app1 (2 hours)

CRITICAL ORDER: Move Caddy LAST in this phase. All other services go first so Caddy can proxy to them once it's active on app1. If you move Caddy first, you have no reverse proxy for the remaining moves.

Step 2.1 — Move Ops Portal

# ON CORE
ssh -i ~/.ssh/itpp-infra root@152.53.192.33

# Stop the service
systemctl stop ops-portal

# Copy the SQLite DB (do NOT copy while running)
cp /opt/ops-portal/audit.db /tmp/audit.db.bak

# Tar the entire ops-portal directory
tar czf /tmp/ops-portal.tar.gz -C /opt ops-portal/

# Copy to app1
scp -i ~/.ssh/itpp-infra /tmp/ops-portal.tar.gz root@152.53.36.131:/tmp/

# ON APP1
ssh -i ~/.ssh/itpp-infra root@152.53.36.131

# Extract
tar xzf /tmp/ops-portal.tar.gz -C /opt/

# Set up Python venv (if not already)
cd /opt/ops-portal
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

# Create systemd service file
cat > /etc/systemd/system/ops-portal.service << 'EOF'
[Unit]
Description=Ops Portal Backend
After=network.target

[Service]
Type=simple
User=root
WorkingDirectory=/opt/ops-portal
EnvironmentFile=/opt/ops-portal/.env
ExecStart=/opt/ops-portal/venv/bin/uvicorn server:app --host 127.0.0.1 --port 8090
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable ops-portal
systemctl start ops-portal

# Verify
sleep 2
curl -s http://127.0.0.1:8090/ | head -20

Step 2.2 — Move OSINT API

# ON CORE
ssh -i ~/.ssh/itpp-infra root@152.53.192.33

systemctl stop osint-api
tar czf /tmp/osint-api.tar.gz -C /opt osint-api/
scp -i ~/.ssh/itpp-infra /tmp/osint-api.tar.gz root@152.53.36.131:/tmp/

# ON APP1
ssh -i ~/.ssh/itpp-infra root@152.53.36.131

tar xzf /tmp/osint-api.tar.gz -C /opt/

cd /opt/osint-api
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

cat > /etc/systemd/system/osint-api.service << 'EOF'
[Unit]
Description=OSINT Search API
After=network.target

[Service]
Type=simple
User=root
WorkingDirectory=/opt/osint-api
EnvironmentFile=/opt/osint-api/.env
ExecStart=/opt/osint-api/venv/bin/uvicorn server:app --host 127.0.0.1 --port 8100
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable osint-api
systemctl start osint-api
curl -s http://127.0.0.1:8100/ | head -20

Step 2.3 — Move Super Search (MCP Server)

# ON CORE
ssh -i ~/.ssh/itpp-infra root@152.53.192.33

systemctl stop super-search

# Copy source files (not the venv — rebuild on app1)
tar czf /tmp/super-search.tar.gz \
  -C /root/docker/super-search \
  server.py ratelimit.py exa.env super-search.service

scp -i ~/.ssh/itpp-infra /tmp/super-search.tar.gz root@152.53.36.131:/tmp/

# ON APP1
ssh -i ~/.ssh/itpp-infra root@152.53.36.131

tar xzf /tmp/super-search.tar.gz -C /root/docker/super-search/

cd /root/docker/super-search
python3 -m venv venv
source venv/bin/activate
pip install fastmcp httpx trafilatura python-dotenv

# Create systemd service
cp super-search.service /etc/systemd/system/
# Edit the service file to confirm paths are correct for app1
# WorkingDirectory=/root/docker/super-search
# ExecStart=/root/docker/super-search/venv/bin/python server.py

systemctl daemon-reload
systemctl enable super-search
systemctl start super-search

# Verify — test the MCP endpoint
sleep 2
curl -s http://127.0.0.1:8899/mcp 2>&1 | head -5
# Should return FastMCP session endpoint or tools list

# IMPORTANT: Super Search talks to SearXNG on localhost:8888
# SearXNG must be running on app1 before Super Search can do web_search
# (SearXNG is moved in Step 2.5 below — Super Search will use Exa/Firecrawl fallbacks until then)

Step 2.4 — Move Grafana

# ON CORE
ssh -i ~/.ssh/itpp-infra root@152.53.192.33

# Stop Grafana
cd /root/docker/grafana && docker compose down

# Copy the entire grafana directory
tar czf /tmp/grafana.tar.gz -C /root/docker grafana/
scp -i ~/.ssh/itpp-infra /tmp/grafana.tar.gz root@152.53.36.131:/tmp/

# ON APP1
ssh -i ~/.ssh/itpp-infra root@152.53.36.131

tar xzf /tmp/grafana.tar.gz -C /root/docker/

cd /root/docker/grafana
docker compose up -d

# Verify
docker ps | grep grafana
curl -s http://127.0.0.1:3002/api/health

Step 2.5 — Move SearXNG (Docker)

# ON CORE
ssh -i ~/.ssh/itpp-infra root@152.53.192.33

cd /root/docker/searxng && docker compose down

tar czf /tmp/searxng.tar.gz -C /root/docker searxng/
scp -i ~/.ssh/itpp-infra /tmp/searxng.tar.gz root@152.53.36.131:/tmp/

# ON APP1
ssh -i ~/.ssh/itpp-infra root@152.53.36.131

tar xzf /tmp/searxng.tar.gz -C /root/docker/

cd /root/docker/searxng
docker compose up -d

# Verify
docker ps | grep searxng
curl -s http://127.0.0.1:8888/search?q=test | head -5

Step 2.6 — Move Prometheus

# ON CORE
ssh -i ~/.ssh/itpp-infra root@152.53.192.33

cd /root/docker/prometheus && docker compose down
tar czf /tmp/prometheus.tar.gz -C /root/docker prometheus/
scp -i ~/.ssh/itpp-infra /tmp/prometheus.tar.gz root@152.53.36.131:/tmp/

# ON APP1
ssh -i ~/.ssh/itpp-infra root@152.53.36.131

tar xzf /tmp/prometheus.tar.gz -C /root/docker/
cd /root/docker/prometheus && docker compose up -d
docker ps | grep prometheus

Step 2.7 — Move Vaultwarden, DocuSeal, Twenty CRM, Komodo (Docker Services)

Repeat the pattern for each:

# ON CORE (for each service)
cd /root/docker/<service> && docker compose down
tar czf /tmp/<service>.tar.gz -C /root/docker <service>/
scp -i ~/.ssh/itpp-infra /tmp/<service>.tar.gz root@152.53.36.131:/tmp/

# ON APP1
tar xzf /tmp/<service>.tar.gz -C /root/docker/
cd /root/docker/<service> && docker compose up -d
docker ps | grep <service>

Service-specific notes:

DocuSeal: Verify at http://127.0.0.1:3000
Twenty CRM: Check Postgres + Redis + Worker + Server all healthy. Verify at http://127.0.0.1:3001
Vaultwarden: Verify at http://127.0.0.1:8080
Komodo: Verify Mongo + Core at http://127.0.0.1:9120. Note: Komodo API needs "type":"Local" in login POST body.

Step 2.8 — Move Caddy to app1 (CRITICAL — Do This Last)

This is the biggest step. Caddy currently handles ALL domains on Core. Moving it requires:

Merging Core's Caddy domains into app1's Caddy config
Updating all backend targets (they now point to localhost on app1)
A brief DNS propagation window

# ON CORE — grab the Caddyfile
ssh -i ~/.ssh/itpp-infra root@152.53.192.33
scp /etc/caddy/Caddyfile root@152.53.36.131:/tmp/Caddyfile-core

# ON APP1 — merge Caddy configs
ssh -i ~/.ssh/itpp-infra root@152.53.36.131

# Backup existing Caddyfile
cp /etc/caddy/Caddyfile /etc/caddy/Caddyfile.bak-$(date +%F)

# Build the new merged Caddyfile
# app1 already has: ai.itpropartner.com → :3000, n8n.itpropartner.com → :5678
# Now add from Core (all pointing to localhost since services are on app1):

cat > /etc/caddy/Caddyfile << 'CADDY_EOF'
{
    # No default_bind needed — app1 doesn't run Tailscale
    email email@redacted
}

# === Existing app1 domains ===
ai.itpropartner.com {
    reverse_proxy 127.0.0.1:3000
}

n8n.itpropartner.com {
    reverse_proxy 127.0.0.1:5678
}

# === admin-ai — stays on THIS (app1) until LiteLLM moves to Core in Phase 3 ===
# After Phase 3, this block moves to Core's Caddy
admin-ai.itpropartner.com {
    reverse_proxy 127.0.0.1:4000
}

# === Migrated from Core ===
ops.itpropartner.com {
    reverse_proxy 127.0.0.1:8090
    handle_path /static/* {
        root * /opt/ops-portal/static
        file_server
    }
}

core.itpropartner.com {
    root * /var/www/core
    file_server
}

sign.itpropartner.com {
    reverse_proxy 127.0.0.1:3000
}

crm.debtrecoveryexperts.com {
    reverse_proxy 127.0.0.1:3001
}

portal.debtrecoveryexperts.com {
    root * /var/www/dre-portal
    file_server
}

internal.debtrecoveryexperts.com {
    root * /var/www/dre-internal
    file_server
}

app.itpropartner.com {
    root * /var/www/app
    file_server
    handle /api/* {
        reverse_proxy 127.0.0.1:8082
    }
}

shark.itpropartner.com {
    root * /var/www/shark
    file_server
    handle /api/* {
        reverse_proxy 127.0.0.1:8083
    }
}

shark.iamgmb.com {
    root * /var/www/shark
    file_server
    handle /api/* {
        reverse_proxy 127.0.0.1:8083
    }
}

# Vaultwarden — tailnet-only, internal IP
vault.itpropartner.com {
    reverse_proxy 127.0.0.1:8080
}
CADDY_EOF

# Validate the config
caddy validate --config /etc/caddy/Caddyfile

# If validation passes, reload
systemctl reload caddy

# Check for errors
journalctl -u caddy -n 20 --no-pager

Step 2.9 — Copy Static Files to app1

# ON CORE
ssh -i ~/.ssh/itpp-infra root@152.53.192.33

# Copy static web roots
for dir in /var/www/core /var/www/dre-portal /var/www/dre-internal /var/www/app /var/www/shark; do
    if [ -d "$dir" ]; then
        tar czf /tmp/www-$(basename $dir).tar.gz -C /var/www $(basename $dir)/
        scp -i ~/.ssh/itpp-infra /tmp/www-$(basename $dir).tar.gz root@152.53.36.131:/tmp/
    fi
done

# ON APP1
ssh -i ~/.ssh/itpp-infra root@152.53.36.131

mkdir -p /var/www
for archive in /tmp/www-*.tar.gz; do
    tar xzf "$archive" -C /var/www/
done
ls -la /var/www/

Step 2.10 — Set Up Cron Scripts on app1

Hermes cron jobs are defined in Hermes's state.db and execute ON Core (where Hermes runs). The scripts themselves live on Core. But some scripts interact with services now on app1 — those need to be updated OR new app1-local services need to be aware.

What stays on Core (Hermes runs these):

hermes-live-sync — S3 sync of ~/.hermes (stays on Core, Core is Hermes)
IMAP-email-triage — email processing (Core, Hermes-managed)
daily-tech-digest — Hermes agent job
exotic-vehicle-scout — Hermes agent job
All Hermes-agent cron jobs that don't touch moved services

Scripts that need path/endpoint updates (still run on Core but target app1):

service-health-check → now checks app1 services
ops-data-collector → now hits ops.itpropartner.com (app1)
vps-threshold-check → checks both Core and app1

Scripts to copy to app1 for local execution (if needed for app1-local services):

ssh -i ~/.ssh/itpp-infra root@152.53.36.131

# Copy backup scripts that now run against app1's Docker volumes
mkdir -p /root/.hermes/scripts

IMPORTANT: Hermes cron jobs that fire --no_agent scripts run the script ON THE HERMES HOST (Core). The script paths in cron job definitions reference ~/.hermes/scripts/ on Core. These stay on Core — only the script CONTENTS may need updating to reference app1 instead of localhost.

Step 2.11 — Verify All Services on app1

ssh -i ~/.ssh/itpp-infra root@152.53.36.131

# All systemd services running
systemctl status ops-portal osint-api super-search --no-pager -l

# All Docker containers running
docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"

# Caddy proxying
curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:8090  # Ops Portal
curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:8100  # OSINT API
curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:8899/mcp  # Super Search
curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:3002/api/health  # Grafana

# Disk space sanity check
df -h /

Step 2.12 — Update DNS Records

This is the cutover moment. Brief downtime (~5-15 min) for DNS propagation.

# Use Cloudflare API to update all domains from Core IP → app1 IP

# For zone itpropartner.com:
#   ops.itpropartner.com → 152.53.36.131
#   core.itpropartner.com → 152.53.36.131
#   sign.itpropartner.com → 152.53.36.131
#   app.itpropartner.com → 152.53.36.131
#   shark.itpropartner.com → 152.53.36.131

# For zone debtrecoveryexperts.com:
#   crm.debtrecoveryexperts.com → 152.53.36.131
#   portal.debtrecoveryexperts.com → 152.53.36.131
#   internal.debtrecoveryexperts.com → 152.53.36.131

# For zone iamgmb.com:
#   shark.iamgmb.com → 152.53.36.131

DNS verification:

# Check propagation (may take a few minutes)
for domain in ops.itpropartner.com core.itpropartner.com sign.itpropartner.com \
             crm.debtrecoveryexperts.com shark.iamgmb.com; do
    echo -n "$domain → "
    dig +short A "$domain"
done

Step 2.13 — Post-DNS Smoke Test

# From any external box (or Core, testing against app1 IP)
curl -s -o /dev/null -w "%{http_code}" https://ops.itpropartner.com
# Expected: 200

curl -s -o /dev/null -w "%{http_code}" https://sign.itpropartner.com
# Expected: 200

curl -s -o /dev/null -w "%{http_code}" https://crm.debtrecoveryexperts.com
# Expected: 200

# Super Search MCP endpoint via Caddy (if proxied) — test direct
curl -s http://152.53.36.131:8899/mcp | head -5

Step 2.14 — Decommission Old Services on Core

Once DNS has propagated and all services are verified on app1:

ssh -i ~/.ssh/itpp-infra root@152.53.192.33

# Stop and disable moved systemd services
systemctl stop ops-portal osint-api super-search
systemctl disable ops-portal osint-api super-search

# Stop and disable Caddy (Core no longer serves web traffic)
systemctl stop caddy
systemctl disable caddy

# Stop moved Docker containers
cd /root/docker/searxng && docker compose down
cd /root/docker/grafana && docker compose down
cd /root/docker/prometheus && docker compose down
cd /root/docker/vaultwarden && docker compose down
cd /root/docker/docuseal && docker compose down
cd /root/docker/twenty && docker compose down
cd /root/docker/komodo && docker compose down

# Clean up old Caddy config (keep backup)
mv /etc/caddy/Caddyfile /etc/caddy/Caddyfile.moved-to-app1

# Verify Core is clean
systemctl list-units --type=service --state=running | grep -E 'ops-portal|osint-api|super-search|caddy'
docker ps
# Should show no moved services

# Free disk space — Core should be significantly lighter now
df -h /

5. Phase 3 — LiteLLM Migration to Core (30 minutes)

Context: LiteLLM currently runs on old-ai (178.156.167.181 — Hetzner CPX41 at 93% disk). Instead of moving it to app1 (previous plan), it goes directly to Core as the new Hermes command center's LLM proxy.

Step 3.1 — Prep LiteLLM on Core

ssh -i ~/.ssh/itpp-infra root@152.53.192.33

# Install Docker if not already on Core (should be — Core runs Docker)
docker --version

# Pull LiteLLM image
docker pull ghcr.io/berriai/litellm:v1.84.0

# Create directory structure
mkdir -p /root/docker/litellm

Create /root/docker/litellm/docker-compose.yml:

services:
  litellm:
    image: ghcr.io/berriai/litellm:v1.84.0
    container_name: litellm
    restart: unless-stopped
    environment:
      - DATABASE_URL=${DATABASE_URL}
      - STORE_MODEL_IN_DB=True
      - LITELLM_MASTER_KEY=${LITELLM_MASTER_KEY}
      - LITELLM_SALT_KEY=${LITELLM_SALT_KEY}
    volumes:
      - ./config.yaml:/app/config.yaml
    command: ["--config", "/app/config.yaml", "--port", "4000"]
    ports:
      - "127.0.0.1:4000:4000"
    networks:
      - litellm-net
    depends_on:
      litellm_postgres:
        condition: service_healthy
    deploy:
      resources:
        limits:
          cpus: '2'
          memory: 2G
        reservations:
          cpus: '0.5'
          memory: 512M
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "3"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:4000/health/liveliness"]
      interval: 30s
      timeout: 10s
      retries: 3

  litellm_postgres:
    image: postgres:16
    container_name: litellm_postgres
    restart: unless-stopped
    environment:
      - POSTGRES_DB=litellm_db
      - POSTGRES_USER=litellm
      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
    volumes:
      - litellm_db_data:/var/lib/postgresql/data
    networks:
      - litellm-net
    deploy:
      resources:
        limits:
          cpus: '2'
          memory: 2G
        reservations:
          cpus: '0.25'
          memory: 512M
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "3"
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U litellm -d litellm_db"]
      interval: 5s
      timeout: 5s
      retries: 5

networks:
  litellm-net:
    driver: bridge

volumes:
  litellm_db_data:

Step 3.2 — Get Secrets from old-ai

# SSH to old-ai and grab the .env
ssh -i ~/.ssh/itpp-infra root@178.156.167.181

# Get LiteLLM env vars
cat /root/docker/litellm/.env
# Critical values needed:
#   LITELLM_MASTER_KEY
#   LITELLM_SALT_KEY
#   POSTGRES_PASSWORD
#   DATABASE_URL

# Also grab config.yaml
cat /root/docker/litellm/config.yaml

Step 3.3 — Create .env and config.yaml on Core

ssh -i ~/.ssh/itpp-infra root@152.53.192.33

# Create .env (chmod 600)
cat > /root/docker/litellm/.env << 'EOF'
POSTGRES_PASSWORD=<FROM_OLD_AI>
DATABASE_URL=postgresql://litellm:<FROM_OLD_AI>@litellm_postgres:5432/litellm_db
LITELLM_MASTER_KEY=<FROM_OLD_AI>
LITELLM_SALT_KEY=<FROM_OLD_AI>
EOF
chmod 600 /root/docker/litellm/.env

Create config.yaml:

general_settings:
  master_key: "<FROM_OLD_AI>"
  use_dashboard: true
  database_url: postgresql://litellm:<FROM_OLD_AI>@litellm_postgres:5432/litellm_db
  store_model_in_db: true
  database_connection_pool_limit: 10
  database_connection_timeout: 30
  enforce_database: true

Step 3.4 — Dump and Import Postgres DB

# ON OLD-AI (178.156.167.181)
ssh -i ~/.ssh/itpp-infra root@178.156.167.181

cd /root/docker/litellm
PGPASSWORD=<POSTGRES_PASSWORD> docker exec litellm_postgres \
  pg_dump -U litellm -d litellm_db --no-owner > /tmp/litellm_db_dump.sql

# Copy to Core via scp (Core has SSH to old-ai)
scp -i ~/.ssh/itpp-infra /tmp/litellm_db_dump.sql root@152.53.192.33:/tmp/

# ON CORE
ssh -i ~/.ssh/itpp-infra root@152.53.192.33

# Start Postgres first
cd /root/docker/litellm
docker compose up -d litellm_postgres
sleep 5

# Import the dump
PGPASSWORD=<POSTGRES_PASSWORD> docker exec -i litellm_postgres \
  psql -U litellm -d litellm_db < /tmp/litellm_db_dump.sql

# Verify import
PGPASSWORD=<POSTGRES_PASSWORD> docker exec litellm_postgres \
  psql -U litellm -d litellm_db -c "SELECT COUNT(*) FROM \"LiteLLM_ProxyModelTable\";"
# Should show ~39 models

Step 3.5 — Start LiteLLM on Core

ssh -i ~/.ssh/itpp-infra root@152.53.192.33

cd /root/docker/litellm
docker compose up -d litellm

# Wait for health
sleep 10
docker ps | grep litellm
# Should show (healthy)

# Test from loopback
curl -s http://127.0.0.1:4000/v1/models \
  -H "Authorization: Bearer $(grep LITELLM_MASTER_KEY /root/docker/litellm/.env | cut -d= -f2)" | python3 -m json.tool | head -30

# Test a real completion
curl -s http://127.0.0.1:4000/v1/chat/completions \
  -H "Authorization: Bearer $(grep LITELLM_MASTER_KEY /root/docker/litellm/.env | cut -d= -f2)" \
  -H "Content-Type: application/json" \
  -d '{"model":"openrouter/deepseek/deepseek-chat","messages":[{"role":"user","content":"Say hello in one word"}]}' | python3 -c "import sys,json; print(json.load(sys.stdin)['choices'][0]['message']['content'])"

Step 3.6 — Setup Caddy on Core for admin-ai

Core needs Caddy back — but ONLY for admin-ai.itpropartner.com (LiteLLM proxy). Not for all domains.

ssh -i ~/.ssh/itpp-infra root@152.53.192.33

# Install Caddy if removed (it was systemd, may still be installed)
which caddy || apt-get install -y caddy

# Minimal Caddyfile — only admin-ai
cat > /etc/caddy/Caddyfile << 'EOF'
{
    email email@redacted
}

admin-ai.itpropartner.com {
    reverse_proxy 127.0.0.1:4000
}
EOF

caddy validate --config /etc/caddy/Caddyfile
systemctl enable caddy
systemctl start caddy

# Verify
curl -s https://admin-ai.itpropartner.com/v1/models -H "Authorization: Bearer <LITELLM_MASTER_KEY>" | python3 -m json.tool | head -20

Step 3.7 — Update DNS for admin-ai

# Change admin-ai.itpropartner.com A record:
# FROM: 152.53.36.131 (app1) 
# TO:   152.53.192.33 (Core)
# TTL: 60 (reduce first, then update)

# Wait for propagation
dig +short A admin-ai.itpropartner.com
# Should resolve to 152.53.192.33

Step 3.8 — Wire Hermes to Use Local LiteLLM

ssh -i ~/.ssh/itpp-infra root@152.53.192.33

# Update Hermes config to use local LiteLLM
hermes config set providers.admin-ai.base_url "https://admin-ai.itpropartner.com/v1"
hermes config set providers.admin-ai.api_key "<LITELLM_MASTER_KEY>"

# Verify
hermes config check
hermes gateway restart

# Test model access
curl -s https://admin-ai.itpropartner.com/v1/models \
  -H "Authorization: Bearer <LITELLM_MASTER_KEY>" | python3 -c "
import sys, json
models = json.load(sys.stdin).get('data', [])
print(f'Models available: {len(models)}')
for m in models[:5]:
    print(f'  {m[\"id\"]}')
print('  ...')
"

Step 3.9 — Remove admin-ai from app1 Caddy

ssh -i ~/.ssh/itpp-infra root@152.53.36.131

# Remove the admin-ai block from app1 Caddyfile
# (It was a placeholder — now Core handles it)
# Edit /etc/caddy/Caddyfile, remove the admin-ai block
caddy validate --config /etc/caddy/Caddyfile
systemctl reload caddy

6. Phase 4 — Ollama Migration to Core (15 minutes)

Step 6.1 — Pull Lightweight Model on Core

ssh -i ~/.ssh/itpp-infra root@152.53.192.33

# Install Ollama if not already on Core
which ollama || curl -fsSL https://ollama.com/install.sh | sh

# Check current status
systemctl status ollama
ollama list

# Pull lightweight fallback model
ollama pull llama3.2:3b
# OR: ollama pull qwen2.5:7b

# Verify
ollama list
curl -s http://127.0.0.1:11434/api/tags | python3 -m json.tool

Step 6.2 — Update Hermes Fallback Chain

ssh -i ~/.ssh/itpp-infra root@152.53.192.33

# Update Hermes model fallbacks to chain:
# Primary: admin-ai (LiteLLM on localhost) → Fallback: Ollama (local)
hermes config set model.fallbacks '[{"provider":"admin-ai","model":"openrouter/deepseek/deepseek-chat"},{"provider":"ollama","model":"llama3.2:3b"}]'

# Verify config
hermes config check
hermes gateway restart

Step 6.3 — Test Fallback Chain

ssh -i ~/.ssh/itpp-infra root@152.53.192.33

# Test primary (LiteLLM → OpenRouter → DeepSeek)
curl -s http://127.0.0.1:4000/v1/chat/completions \
  -H "Authorization: Bearer <LITELLM_MASTER_KEY>" \
  -H "Content-Type: application/json" \
  -d '{"model":"openrouter/deepseek/deepseek-chat","messages":[{"role":"user","content":"Hi"}]}' | python3 -c "import sys,json; print(json.load(sys.stdin)['choices'][0]['message']['content'][:50])"

# Test fallback (local Ollama)
curl -s http://127.0.0.1:11434/api/generate \
  -d '{"model":"llama3.2:3b","prompt":"Say hello","stream":false}' | python3 -c "import sys,json; print(json.load(sys.stdin)['response'][:50])"

Step 6.4 — app1 Ollama (Keep Running)

App1's Ollama stays running — it serves Open WebUI and future AI workloads. Core only gets the lightweight fallback. Primary models stay on app1.

7. Phase 5 — Rename app1-bu → core-bu (15 minutes)

Step 7.1 — Update Documentation References

Files to update:

File	Change
`/root/.hermes/references/master-apps-services.md`	`app1-bu` → `core-bu` throughout
`/root/.hermes/references/server-architecture-plan` (skill)	Update server roles table
`/root/.hermes/references/1-DR-Master-Plan-v2.md`	Update references
`/root/.hermes/references/2-Backup-Security-Standard.md`	Update references
`/root/.hermes/references/3-Per-Server-Runbooks.md`	Update runbook title
`/root/.hermes/references/4-DR-Testing-Schedule.md`	Update references
`/root/.hermes/references/core-rebalance-plan.md`	This document — update after execution
`/root/.hermes/skills/devops/server-architecture-plan/SKILL.md`	Update server roles
`/root/.hermes/skills/devops/hermes-backup/SKILL.md`	Update references
`/root/.hermes/skills/autonomous-ai-agents/hermes-migration/SKILL.md`	Update references

Step 7.2 — Update Warm Standby Configuration

ssh -i ~/.ssh/itpp-infra root@5.161.114.8

# Update hostname on the box itself
hostnamectl set-hostname core-bu

# Update /etc/hosts
sed -i 's/app1-bu/core-bu/g' /etc/hosts

# Verify
hostname
hostname -f

Step 7.3 — Update DR Plan References

The warm standby (now core-bu) watches Core. Its role doesn't change functionally — it's still the failover for Core. The rename is cosmetic + clarity:

Old: "app1-bu stands by app1" (confusing)
New: "core-bu stands by Core" (clear)

8. Phase 6 — Full Verification (15 minutes)

Step 8.1 — Hermes Health Check

ssh -i ~/.ssh/itpp-infra root@152.53.192.33

# Gateway status
hermes gateway status

# Model access
hermes config check

# Cron jobs
hermes cron list
# Should show all ~22 jobs

# Memory store
python3 -c "
import sqlite3
db = sqlite3.connect('/root/.hermes/memory_store.db')
c = db.cursor()
print('Memory tables:', [t[0] for t in c.execute(\"SELECT name FROM sqlite_master WHERE type='table'\").fetchall()])
"

Step 8.2 — app1 Service Health Check

ssh -i ~/.ssh/itpp-infra root@152.53.36.131

# All systemd services
systemctl is-active ops-portal osint-api super-search caddy

# All Docker containers
docker ps --format "table {{.Names}}\t{{.Status}}"

# Caddy status
systemctl status caddy --no-pager -l | head -20

# Disk usage
df -h /
free -h

# Test all web endpoints
for endpoint in \
  "https://ops.itpropartner.com" \
  "https://sign.itpropartner.com" \
  "https://crm.debtrecoveryexperts.com" \
  "https://core.itpropartner.com" \
  "https://n8n.itpropartner.com" \
  "https://ai.itpropartner.com"; do
    code=$(curl -s -o /dev/null -w "%{http_code}" "$endpoint")
    echo "$endpoint → $code"
done

Step 8.3 — LiteLLM / admin-ai Verification

# Test from external
curl -s https://admin-ai.itpropartner.com/v1/models \
  -H "Authorization: Bearer <LITELLM_MASTER_KEY>" | python3 -c "
import sys, json
data = json.load(sys.stdin)
models = data.get('data', [])
print(f'admin-ai models: {len(models)}')
openrouter_models = [m['id'] for m in models if 'openrouter' in m['id']]
print(f'OpenRouter models: {len(openrouter_models)}')
"

# Test a completion through LiteLLM
curl -s https://admin-ai.itpropartner.com/v1/chat/completions \
  -H "Authorization: Bearer <LITELLM_MASTER_KEY>" \
  -H "Content-Type: application/json" \
  -d '{"model":"openrouter/deepseek/deepseek-chat","messages":[{"role":"user","content":"Say test"}]}' | python3 -c "
import sys, json
r = json.load(sys.stdin)
print('Response:', r['choices'][0]['message']['content'][:50])
print('Model:', r.get('model','?'))
"

Step 8.4 — Super Search via Hermes

# On Core (where Hermes runs), update MCP endpoint for Super Search
ssh -i ~/.ssh/itpp-infra root@152.53.192.33

# Verify Super Search MCP is reachable from Core
curl -s http://152.53.36.131:8899/mcp | head -5

# Update Hermes MCP config to point to app1
# The mcp configuration in ~/.hermes/config.yaml should reference:
#   super-search:
#     url: http://152.53.36.131:8899/mcp

# If using Tailscale, could also use tailnet IP for private routing:
#   url: http://<app1-tailnet-ip>:8899/mcp

Step 8.5 — Backup Verification

# Verify daily backup still works (runs on Core)
ssh -i ~/.ssh/itpp-infra root@152.53.192.33

source /opt/awscli-venv/bin/activate

# Check latest full backup
aws s3 ls s3://hermes-vps-backups/hermes-full-backup/ \
  --endpoint-url https://s3.us-east-1.wasabisys.com/ | tail -3

# Check live sync
aws s3 ls s3://hermes-vps-backups/live/ \
  --endpoint-url https://s3.us-east-1.wasabisys.com/ | head -5

# Check system configs (should now reflect Core-only services)
aws s3 ls s3://itpropartner-system-configs/ \
  --endpoint-url https://s3.us-east-1.wasabisys.com/

Step 8.6 — Cron Job Verification

Run a few critical cron jobs manually to verify they still work post-move:

ssh -i ~/.ssh/itpp-infra root@152.53.192.33

# Service health check (should check app1 now)
bash /root/.hermes/scripts/service-health-check.sh

# VPS threshold check
bash /root/.hermes/scripts/vps-threshold-check.sh

# Ops data collector (if it hits ops.itpropartner.com which now points to app1)
bash /root/.hermes/scripts/ops-data-collector.sh 2>&1 | head -20

9. Cleanup & Decommission

Step 9.1 — Core Cleanup

ssh -i ~/.ssh/itpp-infra root@152.53.192.33

# Remove old Docker volumes for moved services
docker volume prune -f

# Remove old service directories (keep for 7 days as backup)
find /root/docker -mindepth 1 -maxdepth 1 -type d | while read dir; do
    case $(basename "$dir") in
        litellm|ollama) echo "KEEP: $dir" ;;
        *) echo "CAN REMOVE (after 7 days): $dir" ;;
    esac
done

# Remove old /opt directories
# /opt/ops-portal, /opt/osint-api — keep for 7 days then remove

# Verify disk space reclaimed
df -h /

Step 9.2 — old-ai Decommission (after 3 days stable)

# After admin-ai has been stable on Core for 3+ days:
# 1. Stop all containers on old-ai (178.156.167.181)
# 2. Take final snapshot
# 3. Delete Hetzner CPX41 server
# This saves ~$30-40/mo

Step 9.3 — Update master-apps-services.md

The master inventory at /root/.hermes/references/master-apps-services.md must be fully rewritten to reflect the new architecture. Key changes:

Core: Hermes, LiteLLM, Ollama, Tailscale, WireGuard, StrongSwan, Redis
app1: Caddy (all domains), Ops Portal, OSINT API, Super Search, SearXNG, Grafana, Prometheus, Vaultwarden, DocuSeal, Twenty CRM, Komodo, n8n, Open WebUI, Ollama (primary)
core-bu (formerly app1-bu): Warm standby for Core

10. Rollback Plan

If the migration fails at any phase:

Rollback: Services on app1 not working

Re-point DNS for affected domains back to Core IP (152.53.192.33)
Start services back on Core: systemctl start caddy ops-portal osint-api super-search
Start Docker containers on Core: cd /root/docker/<service> && docker compose up -d
Verify services respond on Core
Debug app1 issues separately — Core continues serving

Rollback: LiteLLM on Core not working

Re-point admin-ai.itpropartner.com back to app1 IP (152.53.36.131)
Or: re-point to old-ai IP (178.156.167.181) if old-ai is still running
Update Hermes config: hermes config set providers.admin-ai.base_url "https://admin-ai.itpropartner.com/v1"
Restart Hermes gateway
Debug Core LiteLLM separately

Rollback: Ollama on Core not working

Update Hermes fallback to remove ollama from chain
Or: point fallback to app1 Ollama at http://152.53.36.131:11434

11. Key Pitfalls

DNS Propagation Split-Brain

Risk: During the 5-15 minute DNS propagation window, some clients hit Core, some hit app1
Mitigation: Core's Caddy can be left running during transition (serving stale content is better than errors). Stop Core Caddy only AFTER DNS has fully propagated

SQLite DB Corruption

Risk: Copying Ops Portal's audit.db while the service is running
Mitigation: systemctl stop ops-portal BEFORE copying. Verified in step order above.

Super Search → SearXNG Dependency

Risk: Super Search on app1 can't reach SearXNG until SearXNG is also on app1
Mitigation: Move SearXNG first (Step 2.5 before Step 2.3 restart). Super Search falls back to Exa/Firecrawl if SearXNG is unreachable.

SSH Keys for Cron Scripts

Risk: Cron scripts on Core SSH to other servers. If keys are moved, scripts break.
Mitigation: SSH keys stay on Core. Scripts that SSH from Core to app1 still work because Core can SSH to app1 (itpp-infra key).

Caddy Let's Encrypt Rate Limits

Risk: Moving many domains at once can trigger Let's Encrypt rate limits (5 failures per domain per hour)
Mitigation: Ensure DNS is pointing to app1 BEFORE Caddy tries to issue certs. Test DNS first: dig +short <domain> must return app1 IP.

Hermes MCP Connection

Risk: Hermes on Core can't reach Super Search if the MCP URL points to localhost
Mitigation: Update MCP URL in Hermes config to http://152.53.36.131:8899/mcp (app1's IP)

Ollama GPU / CPU

Risk: Core (RS 2000) may not have GPU for Ollama. CPU-only inference on llama3.2:3b is slow.
Mitigation: This is the FALLBACK model — only used if LiteLLM is down. Slow is acceptable for fallback. Primary inference always goes through LiteLLM → OpenRouter.

Twenty CRM Database

Risk: Twenty CRM has Postgres + Redis + Worker containers. All must move together.
Mitigation: Move the entire /root/docker/twenty/ directory. Docker compose brings up all services together.

12. Timeline Summary

Phase	Duration	Key Actions
Phase 1: Prep	30 min	Snapshot, audit, pre-stage files
Phase 2: Move	2 hrs	Move all services to app1, update DNS
Phase 3: LiteLLM	30 min	Migrate from old-ai to Core, wire Hermes
Phase 4: Ollama	15 min	Pull lightweight model on Core
Phase 5: Rename	15 min	app1-bu → core-bu, update docs
Phase 6: Verify	15 min	Full smoke test
Total	~3.5 hrs

Best execution window: Saturday or Sunday morning (lowest traffic, cron job impact minimized)

13. Post-Migration Architecture

Core (152.53.192.33) — The Command Center

Hermes Agent        ← systemd, port 8081 (the brain)
Caddy (minimal)     ← systemd, port 80/443 — ONLY admin-ai.itpropartner.com
LiteLLM             ← Docker, port 4000 — LLM proxy with 39 models
Ollama              ← systemd, port 11434 — llama3.2:3b fallback
Tailscale           ← systemd — mesh VPN
WireGuard           ← systemd — home router tunnel
StrongSwan          ← systemd — L2TP/IPsec VPN (TBD)
Redis               ← systemd — cache
Node Exporter       ← systemd — metrics
Hermes Browser      ← systemd — Chromium headless

app1 (152.53.36.131) — The Service Host

Caddy               ← systemd, port 80/443 — ALL public domains
Ops Portal          ← systemd, port 8090 (FastAPI)
OSINT API           ← systemd, port 8100 (FastAPI)
Super Search        ← systemd, port 8899 (FastMCP)
SearXNG             ← Docker, port 8888
Grafana             ← Docker, port 3002
Prometheus          ← Docker, port 9090
Vaultwarden         ← Docker, port 8080
DocuSeal            ← Docker, port 3000
Twenty CRM          ← Docker, port 3001 + Postgres + Redis + Worker
Komodo              ← Docker, port 9120 + Mongo
n8n                 ← Docker, port 5678 + Postgres
Open WebUI          ← Docker, port 3000 (ai.itpropartner.com)
Ollama (primary)    ← Docker, port 11434 (full models)

core-bu (5.161.114.8) — Warm Standby

Hermes (dormant)    ← systemd — auto-activates if Core is down
Ollama              ← systemd — fallback
Tailscale           ← systemd
StrongSwan          ← systemd
Watchdog            ← system crontab — checks Core every 5 min

END OF PLAN

41 KiB Raw Blame History Unescape Escape