ippadmin/hermes-recovery
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
master
hermes-recovery/references/server-dr-plans.md
T

13 KiB
Raw Permalink Blame History

IT Pro Partner — Per-Server Disaster Recovery Plan

Author: Sho'Nuff / Network Services Team Date: July 9, 2026 Version: 1.0 Supersedes: hermes-dr-plan-v2.md (Hermes-specific); server-provisioning-standard-v1.md (base install)

DR Architecture Overview

Internet ──► Cloudflare ──► Caddy ──► Service (HTTP/HTTPS)
                          │
                    SSH/WG ──► Management plane (via itpp-infra key)
                          │
                    S3 (Wasabi) ──► Backup plane (all servers)

Shared dependencies:

  • SSH access: itpp-infra key (Hetzner vault or deployed per-server)
  • DNS: Cloudflare (primary) / SiteGround (legacy for itpropartner.com)
  • Backup storage: Wasabi S3 (us-east-1)
  • SMTP relay: mail.germainebrown.com:2525 (MXroute)
  • Auth/SSO: Cloudflare Access (DRE portal)
  • Monitoring: Prometheus node_exporter on each server
  • VPN: WireGuard for management access

Server 1: Core (netcup RS 2000 G12)

Hostname: core IP: 152.53.192.33 Provider: netcup (RS 2000 G12, 8C EPYC 9645, 16GB RAM, 512GB NVMe) OS: Debian 13 Status: 🟢 LIVE — Primary Hermes host

Purpose

Main operations server. Runs Hermes, Docker services, ops portal, WireGuard hub, SMTP relay, local LLM fallback.

Services Running

Service Type Port Notes
Hermes Agent systemd Telegram Primary agent - deepseek-chat via admin-ai
Caddy systemd 80/443 Reverse proxy for all domains
WireGuard (wg0) systemd 51821/udp Tunnel to home router (10.77.0.1)
MySQL tunnel autossh 33060→wphost02:3306 Apex database access
SearXNG Docker 8888 Self-hosted search
DocuSeal Docker 3000 Document signing
Vaultwarden Docker 8080/tailscale Password manager
Ollama systemd 11434 Local LLM fallback (llama3.2:3b)
Ops Portal systemd 8090 FastAPI backend
node_exporter systemd 9100 Prometheus metrics

Docker Containers

  • searxng, docuseal, vaultwarden (compass), + others under /root/docker/

Backup

  • Hermes state: Every 15 min to s3://hermes-vps-backups/live/ (state.db + config)
  • Full Hermes: Daily 5 AM to s3://hermes-vps-backups/hermes-full-backup/
  • Memory snapshots: Every 10 min to s3://hermes-vps-backups/live/memories/
  • Docker volumes: Manual (not automated)

Restore Procedure

# 1. Provision RS 2000 G12 (or better) on netcup
# 2. Install Debian 13, apply base standard (see server-provisioning-standard-v1.md)
# 3. Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
# 4. Restore from S3 backup:
aws s3 cp s3://hermes-vps-backups/hermes-full-backup/<latest> /root/restore.tar.gz \
  --endpoint-url https://s3.us-east-1.wasabisys.com
tar -xzf /root/restore.tar.gz -C /root/.hermes/
# 5. Restore Docker stacks:
cd /root/docker/searxng && docker compose up -d
cd /root/docker/docuseal && docker compose up -d
cd /root/docker/vaultwarden && docker compose up -d
# 6. Restore Caddy config from S3:
aws s3 cp s3://hermes-vps-backups/live/config/Caddyfile /etc/caddy/ \
  --endpoint-url https://s3.us-east-1.wasabisys.com
# 7. Restart services:
systemctl restart caddy
systemctl restart hermes
# 8. Verify: ops.itpropartner.com, Teleegram, WG tunnel

Failover

  • app1-bu (5.161.114.8) is warm standby. Checks Core every 5 min.
  • If Core unreachable for 2 min → app1-bu syncs S3 state + starts Hermes gateway.
  • Restore Core, then manually fail back.

Server 2: wphost02 (Hetzner CPX21)

Hostname: wphost02 IP: 5.161.62.38 Provider: Hetzner (CPX21, 3C/4G/80GB) OS: Debian 12 (RunCloud managed) Status: 🟡 LIVE — Recently reconnected after SSH firewall fix

Purpose

Apex Predators Track Experience WordPress site. RunCloud-managed. WPForms registration + notification emails.

Services Running

  • nginx-rc (RunCloud) — 80/443
  • MariaDB — 3306
  • Postfix — 25 (relay via mail.germainebrown.com:2525)
  • httpd — 81 (RunCloud internal)
  • RunCloud agent — 34210
  • node_exporter — 9100

Docker Containers

None. RunCloud manages PHP directly.

Backup

  • RunCloud backup: Daily via RunCloud panel
  • MariaDB dumps: Not automated externally
  • WordPress files: Managed by RunCloud

Restore Procedure

# 1. Provision CPX21 (or better) on Hetzner
# 2. Install Debian 12
# 3. Connect to RunCloud via their panel
# 4. RunCloud provisions nginx, MariaDB, PHP
# 5. Restore WordPress from RunCloud backup
# 6. Add itpp-infra SSH key in RunCloud → SSH Access
# 7. Whitelist Core IP (152.53.192.33) in RunCloud firewall
# 8. Restore Postfix relay config:
#    - relayhost = mail.germainebrown.com:2525
#    - SASL auth with email@redacted
# 9. Verify: apextrackexperience.com loads, forms submit

Dependencies

  • Core MySQL tunnel connects here (port 33060→3306)
  • Apex mail watchdog monitors via SSH

Server 3: ai.itpropartner.com (Hetzner CPX41)

Hostname: ai IP: 178.156.167.181 Provider: Hetzner (CPX41, 4C/16G/226GB) OS: Debian 12 Status: 🟡 LIVE — 🔴 92% disk full (18G free)

⚠️ CRITICAL: Disk at 92%. Needs cleanup before migration.

Purpose

Critical infrastructure. Runs LiteLLM proxy that ALL Hermes instances use for model inference (deepseek-chat, claude-opus). Also runs:

  • Open WebUI — AI chat interface
  • Ollama — local LLM inference
  • Qdrant — vector database
  • Caddy — reverse proxy

Services Running

  • LiteLLM — systemd service, port varies (API at admin-ai.itpropartner.com)
  • Docker containers: openwebui, litellm, litellm_postgres, caddy, qdrant, ollama
  • node_exporter — 9100

Backup

  • LiteLLM config: In /root/. hermes/.env and config.yaml (API key: sk-lbh...)
  • Docker volumes: Not backed up to S3
  • Ollama models: Would need re-pull (cache only)

Restore Procedure

# 1. Provision CPX41 (or better) on netcup as part of migration
# 2. Install Debian 13, Docker
# 3. Deploy LiteLLM Docker stack:
#    - From /opt/litellm/docker-compose.yml
#    - Restore postgres data for LiteLLM routes
# 4. Deploy Ollama:
#    - Pull model: ollama pull llama3.2:3b
# 5. Deploy Open WebUI:
#    - From /opt/openwebui/docker-compose.yml
# 6. Deploy Caddy config for admin-ai.itpropartner.com
# 7. Update DNS: admin-ai.itpropartner.com → new IP
# 8. Update Hermes config.yaml on all profiles to point to new IP
# 9. Verify: curl https://admin-ai.itpropartner.com/v1/models
# 10. Decommission old Hetzner server

🔴 Pre-Reboot Checklist

# Before ANY reboot or migration of this server:
# 1. Verify at least ONE alternative model provider works:
curl https://admin-ai.itpropartner.com/v1/chat/completions \
  -H "Authorization: Bearer sk-lbh..." \
  -d '{"model":"deepseek-chat","messages":[{"role":"user","content":"ping"}]}'
# 2. Ensure local Ollama on Core is running:
curl http://127.0.0.1:11434/api/tags
# 3. Notify all Hermes users (Germaine, Anita, Tony)

Dependencies

  • ALL Hermes instances depend on this server's LiteLLM proxy
  • If this goes down, Hermes falls back to local Ollama (llama3.2:3b on Core)
  • Tony's Hermes has no fallback — would be offline

Server 4: unms.forefrontwireless.com (Hetzner CPX21)

Hostname: unms IP: 5.161.225.131 Provider: Hetzner (CPX21, 3C/4G/80GB) OS: Debian 12 Status: 🟢 LIVE — 4d 11h uptime

Purpose

UISP/UNMS (WISP management) + UCRM (customer management). Manages tower routers and customer billing for Forefront Wireless.

Docker Containers (9)

unms-api, unms-device-ws-1, unms-device-ws-2, unms-nginx, unms-rabbitmq, unms-postgres, unms-siridb, unms-fluentd, ucrm

Backup

  • Docker volumes: postgres data, UCRM data
  • UNMS config: Exportable from UNMS UI

Restore

# 1. Provision CPX21 → Reinstall Debian 12 + Docker
# 2. Restore docker-compose stacks from backup
# 3. Restore Postgres volumes
# 4. Update DNS for unms.forefrontwireless.com
# 5. Re-register tower routers in UNMS

Server 5: unifi (Hetzner CPX21)

Hostname: unifi IP: 178.156.131.57 Provider: Hetzner (CPX21, 3C/4G/80GB) OS: Debian 12 Status: 🟢 LIVE — Recently rebuilt (6h uptime)

Purpose

UniFi Network Controller — manages Ubiquiti network equipment.

Services

  • MariaDB
  • UniFi Controller software (native, not Docker)

Backup

  • UniFi backup via controller UI (automatic backups)
  • MariaDB database backup

Restore

# 1. Provision CPX21 → Install UniFi controller
# 2. Restore UniFi backup from last export
# 3. Update DNS for unifi.itpropartner.com
# 4. Adopt APs/switches back

Server 6: hudu.itpropartner.com (Hetzner CPX21)

Hostname: hudu IP: 178.156.130.130 Provider: Hetzner (CPX21, 3C/4G/80GB) OS: Debian 12 Status: 🟢 LIVE

Purpose

Hudu IT documentation platform — internal knowledge base.

Docker Containers (5)

hudu2-app-1, hudu2-worker-1, hudu2-db-1, hudu2-redis-1, letsencrypt

Backup

  • Hudu automatic: Built-in backup to S3-compatible storage
  • Postgres DB: Daily dump

Restore

# 1. Provision CPX21 → Install Docker
# 2. Deploy Hudu Docker stack
# 3. Restore Postgres from backup
# 4. Update DNS

Server 7: fleettracker360 (Hetzner CPX11)

Hostname: fleettrack360 IP: 178.156.149.32 Provider: Hetzner (CPX11, 2C/2G/40GB) OS: Debian 12 Status: 🟢 LIVE

Purpose

Fleet tracking application (BoxPilot Logistics?).

Services

  • Apache2 (80/443)
  • MariaDB (3306)

Backup

  • Apache config + web root
  • MariaDB database dump

Server 8: docker (Hetzner CPX11)

Hostname: docker IP: 178.156.168.35 Provider: Hetzner (CPX11, 2C/2G/40GB) OS: Debian 12 Status: 🟢 LIVE — 🔴 Disk 73% used (9.9G free on 38G)

Purpose

RunCloud staging + utility containers:

  • Nginx Proxy Manager
  • RustDesk server (hbbr, hbbs)
  • Uptime Kuma
  • StrongSwan

Docker Containers (8)

runcloud_php56, runcloud_php81, runcloud_mariadb, nginx-proxy-manager, hbbr, hbbs, strongswan, uptime-kuma

Server 9: app1 / n8n (Hetzner CPX11)

Hostname: n8n IP: 87.99.144.163 Provider: Hetzner (CPX11, 2C/2G/40GB) OS: Debian 12 Status: 🟢 LIVE

Purpose

n8n workflow automation.

Docker Containers (2)

n8n, n8n-postgres

Server 10: tony-vps (Hetzner CPX21)

Hostname: tony IP: 87.99.159.142 Provider: Hetzner (CPX21, 3C/4G/80GB) OS: Debian 12 Status: 🟢 LIVE

Purpose

Tony's standalone Hermes agent. Separate from main infrastructure.

Services

  • llama-server (systemd) — Llama 3.2 1B local fallback
  • hermes-heartbeat.sh (cron — every minute)

Notes

  • Tony's independent instance — not managed from Core
  • Uses admin-ai for primary model, local Llama 1B for fallback
  • No Docker containers

Server 11: app1-bu (Hetzner CPX11 — Warm Standby)

Hostname: app1-bu.itpropartner.com IP: 5.161.114.8 Provider: Hetzner (CPX11, 2C/2G/40GB) OS: Debian 12 Status: 🟢 STANDBY — Warm, gateway dormant

Purpose

Hermes warm standby. Clones Core's state via S3 sync.

Services

  • SSH (22)
  • Tailscale
  • node_exporter (9100)
  • WireGuard (wg0 — tunnel to home router at 10.77.0.3)

Failover Procedure

# Automatic (watchdog on app1-bu):
# 1. Checks Core every 5 min (ping + Hermes health)
# 2. If Core down for 2 min → sync latest S3 state
# 3. Integrity check on state.db (PRAGMA quick_check)
# 4. Start Hermes gateway → Telegram accessible
# 5. Write ACTIVE_OWNER lock to S3

# Manual activation:
ssh -i /root/.ssh/itpp-infra root@5.161.114.8
hermes gateway run --replace
# Or if Core is fenceable:
ssh root@152.53.192.33 "systemctl stop hermes && systemctl mask hermes"

Restore

# 1. Power on via Hetzner API or console
# 2. Ensure S3 sync runs:
#    /root/.hermes/scripts/hermes-standby-sync.sh
# 3. Check integrity:
#    sqlite3 /root/.hermes/state.db "PRAGMA integrity_check;"
# 4. If good → start gateway:
#    hermes gateway run
# 5. If corrupt → restore from snapshot:
#    aws s3 cp s3://hermes-vps-backups/snapshots/<latest> /root/

DR Classification Matrix

Server Tier RTO Target RPO Target Restore Complexity
Core 🔴 Critical 5 min (failover) 15 min Medium (Docker + Hermes)
ai (admin-ai) 🔴 Critical 2 hours 24 hours High (migration)
wphost02 🟡 Important 4 hours 24 hours Low (RunCloud restore)
unms 🟡 Important 4 hours 24 hours Medium (Docker stack)
hudu 🟡 Important 4 hours 24 hours Medium (Docker stack)
unifi 🟡 Important 4 hours 24 hours Low (native install)
n8n 🟢 Standard 8 hours 24 hours Low (2 containers)
docker 🟢 Standard 8 hours 24 hours Medium (8 containers)
fleettracker360 🟢 Standard 8 hours 24 hours Low (LAMP stack)
tony-vps 🟢 Standard Coordinate 24 hours Low (minimal)
app1-bu 🔴 Standby 2 min 15 min Low (S3 sync)

Recovery Bundle Location

Each server's recovery configs are bundled at:

  • /root/.hermes/recovery-bundle-2026-07-05.md (Core — needs refresh)
  • /root/.hermes/references/server-dr-plans.md (this document)
  • /root/.hermes/references/server-provisioning-standard-v1.md (base install)

S3 backup of this document: s3://hermes-vps-backups/live/config/server-dr-plans.md

Reference in New Issue View Git Blame Copy Permalink