454 lines
13 KiB
Markdown
454 lines
13 KiB
Markdown
# IT Pro Partner — Per-Server Disaster Recovery Plan
|
|
|
|
**Author:** Sho'Nuff / Network Services Team
|
|
**Date:** July 9, 2026
|
|
**Version:** 1.0
|
|
**Supersedes:** hermes-dr-plan-v2.md (Hermes-specific); server-provisioning-standard-v1.md (base install)
|
|
|
|
---
|
|
|
|
## DR Architecture Overview
|
|
|
|
```
|
|
Internet ──► Cloudflare ──► Caddy ──► Service (HTTP/HTTPS)
|
|
│
|
|
SSH/WG ──► Management plane (via itpp-infra key)
|
|
│
|
|
S3 (Wasabi) ──► Backup plane (all servers)
|
|
```
|
|
|
|
**Shared dependencies:**
|
|
- **SSH access:** `itpp-infra` key (Hetzner vault or deployed per-server)
|
|
- **DNS:** Cloudflare (primary) / SiteGround (legacy for itpropartner.com)
|
|
- **Backup storage:** Wasabi S3 (us-east-1)
|
|
- **SMTP relay:** mail.germainebrown.com:2525 (MXroute)
|
|
- **Auth/SSO:** Cloudflare Access (DRE portal)
|
|
- **Monitoring:** Prometheus node_exporter on each server
|
|
- **VPN:** WireGuard for management access
|
|
|
|
---
|
|
|
|
## Server 1: Core (netcup RS 2000 G12)
|
|
|
|
**Hostname:** core
|
|
**IP:** 152.53.192.33
|
|
**Provider:** netcup (RS 2000 G12, 8C EPYC 9645, 16GB RAM, 512GB NVMe)
|
|
**OS:** Debian 13
|
|
**Status:** 🟢 LIVE — Primary Hermes host
|
|
|
|
### Purpose
|
|
Main operations server. Runs Hermes, Docker services, ops portal, WireGuard hub,
|
|
SMTP relay, local LLM fallback.
|
|
|
|
### Services Running
|
|
|
|
| Service | Type | Port | Notes |
|
|
|---------|------|------|-------|
|
|
| Hermes Agent | systemd | Telegram | Primary agent - deepseek-chat via admin-ai |
|
|
| Caddy | systemd | 80/443 | Reverse proxy for all domains |
|
|
| WireGuard (wg0) | systemd | 51821/udp | Tunnel to home router (10.77.0.1) |
|
|
| MySQL tunnel | autossh | 33060→wphost02:3306 | Apex database access |
|
|
| SearXNG | Docker | 8888 | Self-hosted search |
|
|
| DocuSeal | Docker | 3000 | Document signing |
|
|
| Vaultwarden | Docker | 8080/tailscale | Password manager |
|
|
| Ollama | systemd | 11434 | Local LLM fallback (llama3.2:3b) |
|
|
| Ops Portal | systemd | 8090 | FastAPI backend |
|
|
| node_exporter | systemd | 9100 | Prometheus metrics |
|
|
|
|
### Docker Containers
|
|
- searxng, docuseal, vaultwarden (compass), + others under /root/docker/
|
|
|
|
### Backup
|
|
- **Hermes state:** Every 15 min to s3://hermes-vps-backups/live/ (state.db + config)
|
|
- **Full Hermes:** Daily 5 AM to s3://hermes-vps-backups/hermes-full-backup/
|
|
- **Memory snapshots:** Every 10 min to s3://hermes-vps-backups/live/memories/
|
|
- **Docker volumes:** Manual (not automated)
|
|
|
|
### Restore Procedure
|
|
```bash
|
|
# 1. Provision RS 2000 G12 (or better) on netcup
|
|
# 2. Install Debian 13, apply base standard (see server-provisioning-standard-v1.md)
|
|
# 3. Install Hermes:
|
|
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
|
|
# 4. Restore from S3 backup:
|
|
aws s3 cp s3://hermes-vps-backups/hermes-full-backup/<latest> /root/restore.tar.gz \
|
|
--endpoint-url https://s3.us-east-1.wasabisys.com
|
|
tar -xzf /root/restore.tar.gz -C /root/.hermes/
|
|
# 5. Restore Docker stacks:
|
|
cd /root/docker/searxng && docker compose up -d
|
|
cd /root/docker/docuseal && docker compose up -d
|
|
cd /root/docker/vaultwarden && docker compose up -d
|
|
# 6. Restore Caddy config from S3:
|
|
aws s3 cp s3://hermes-vps-backups/live/config/Caddyfile /etc/caddy/ \
|
|
--endpoint-url https://s3.us-east-1.wasabisys.com
|
|
# 7. Restart services:
|
|
systemctl restart caddy
|
|
systemctl restart hermes
|
|
# 8. Verify: ops.itpropartner.com, Teleegram, WG tunnel
|
|
```
|
|
|
|
### Failover
|
|
- **app1-bu** (5.161.114.8) is warm standby. Checks Core every 5 min.
|
|
- If Core unreachable for 2 min → app1-bu syncs S3 state + starts Hermes gateway.
|
|
- Restore Core, then manually fail back.
|
|
|
|
---
|
|
|
|
## Server 2: wphost02 (Hetzner CPX21)
|
|
|
|
**Hostname:** wphost02
|
|
**IP:** 5.161.62.38
|
|
**Provider:** Hetzner (CPX21, 3C/4G/80GB)
|
|
**OS:** Debian 12 (RunCloud managed)
|
|
**Status:** 🟡 LIVE — Recently reconnected after SSH firewall fix
|
|
|
|
### Purpose
|
|
Apex Predators Track Experience WordPress site. RunCloud-managed.
|
|
WPForms registration + notification emails.
|
|
|
|
### Services Running
|
|
- nginx-rc (RunCloud) — 80/443
|
|
- MariaDB — 3306
|
|
- Postfix — 25 (relay via mail.germainebrown.com:2525)
|
|
- httpd — 81 (RunCloud internal)
|
|
- RunCloud agent — 34210
|
|
- node_exporter — 9100
|
|
|
|
### Docker Containers
|
|
**None.** RunCloud manages PHP directly.
|
|
|
|
### Backup
|
|
- **RunCloud backup:** Daily via RunCloud panel
|
|
- **MariaDB dumps:** Not automated externally
|
|
- **WordPress files:** Managed by RunCloud
|
|
|
|
### Restore Procedure
|
|
```bash
|
|
# 1. Provision CPX21 (or better) on Hetzner
|
|
# 2. Install Debian 12
|
|
# 3. Connect to RunCloud via their panel
|
|
# 4. RunCloud provisions nginx, MariaDB, PHP
|
|
# 5. Restore WordPress from RunCloud backup
|
|
# 6. Add itpp-infra SSH key in RunCloud → SSH Access
|
|
# 7. Whitelist Core IP (152.53.192.33) in RunCloud firewall
|
|
# 8. Restore Postfix relay config:
|
|
# - relayhost = mail.germainebrown.com:2525
|
|
# - SASL auth with email@redacted
|
|
# 9. Verify: apextrackexperience.com loads, forms submit
|
|
```
|
|
|
|
### Dependencies
|
|
- Core MySQL tunnel connects here (port 33060→3306)
|
|
- Apex mail watchdog monitors via SSH
|
|
|
|
---
|
|
|
|
## Server 3: ai.itpropartner.com (Hetzner CPX41)
|
|
|
|
**Hostname:** ai
|
|
**IP:** 178.156.167.181
|
|
**Provider:** Hetzner (CPX41, 4C/16G/226GB)
|
|
**OS:** Debian 12
|
|
**Status:** 🟡 LIVE — 🔴 92% disk full (18G free)
|
|
|
|
### ⚠️ CRITICAL: Disk at 92%. Needs cleanup before migration.
|
|
|
|
### Purpose
|
|
**Critical infrastructure.** Runs LiteLLM proxy that ALL Hermes instances use
|
|
for model inference (deepseek-chat, claude-opus). Also runs:
|
|
- Open WebUI — AI chat interface
|
|
- Ollama — local LLM inference
|
|
- Qdrant — vector database
|
|
- Caddy — reverse proxy
|
|
|
|
### Services Running
|
|
- LiteLLM — systemd service, port varies (API at admin-ai.itpropartner.com)
|
|
- Docker containers: openwebui, litellm, litellm_postgres, caddy, qdrant, ollama
|
|
- node_exporter — 9100
|
|
|
|
### Backup
|
|
- **LiteLLM config:** In /root/. hermes/.env and config.yaml (API key: sk-lbh...)
|
|
- **Docker volumes:** Not backed up to S3
|
|
- **Ollama models:** Would need re-pull (cache only)
|
|
|
|
### Restore Procedure
|
|
```bash
|
|
# 1. Provision CPX41 (or better) on netcup as part of migration
|
|
# 2. Install Debian 13, Docker
|
|
# 3. Deploy LiteLLM Docker stack:
|
|
# - From /opt/litellm/docker-compose.yml
|
|
# - Restore postgres data for LiteLLM routes
|
|
# 4. Deploy Ollama:
|
|
# - Pull model: ollama pull llama3.2:3b
|
|
# 5. Deploy Open WebUI:
|
|
# - From /opt/openwebui/docker-compose.yml
|
|
# 6. Deploy Caddy config for admin-ai.itpropartner.com
|
|
# 7. Update DNS: admin-ai.itpropartner.com → new IP
|
|
# 8. Update Hermes config.yaml on all profiles to point to new IP
|
|
# 9. Verify: curl https://admin-ai.itpropartner.com/v1/models
|
|
# 10. Decommission old Hetzner server
|
|
```
|
|
|
|
### 🔴 Pre-Reboot Checklist
|
|
```bash
|
|
# Before ANY reboot or migration of this server:
|
|
# 1. Verify at least ONE alternative model provider works:
|
|
curl https://admin-ai.itpropartner.com/v1/chat/completions \
|
|
-H "Authorization: Bearer sk-lbh..." \
|
|
-d '{"model":"deepseek-chat","messages":[{"role":"user","content":"ping"}]}'
|
|
# 2. Ensure local Ollama on Core is running:
|
|
curl http://127.0.0.1:11434/api/tags
|
|
# 3. Notify all Hermes users (Germaine, Anita, Tony)
|
|
```
|
|
|
|
### Dependencies
|
|
- **ALL Hermes instances** depend on this server's LiteLLM proxy
|
|
- If this goes down, Hermes falls back to local Ollama (llama3.2:3b on Core)
|
|
- **Tony's Hermes** has no fallback — would be offline
|
|
|
|
---
|
|
|
|
## Server 4: unms.forefrontwireless.com (Hetzner CPX21)
|
|
|
|
**Hostname:** unms
|
|
**IP:** 5.161.225.131
|
|
**Provider:** Hetzner (CPX21, 3C/4G/80GB)
|
|
**OS:** Debian 12
|
|
**Status:** 🟢 LIVE — 4d 11h uptime
|
|
|
|
### Purpose
|
|
UISP/UNMS (WISP management) + UCRM (customer management).
|
|
Manages tower routers and customer billing for Forefront Wireless.
|
|
|
|
### Docker Containers (9)
|
|
unms-api, unms-device-ws-1, unms-device-ws-2, unms-nginx, unms-rabbitmq,
|
|
unms-postgres, unms-siridb, unms-fluentd, ucrm
|
|
|
|
### Backup
|
|
- **Docker volumes:** postgres data, UCRM data
|
|
- **UNMS config:** Exportable from UNMS UI
|
|
|
|
### Restore
|
|
```bash
|
|
# 1. Provision CPX21 → Reinstall Debian 12 + Docker
|
|
# 2. Restore docker-compose stacks from backup
|
|
# 3. Restore Postgres volumes
|
|
# 4. Update DNS for unms.forefrontwireless.com
|
|
# 5. Re-register tower routers in UNMS
|
|
```
|
|
|
|
---
|
|
|
|
## Server 5: unifi (Hetzner CPX21)
|
|
|
|
**Hostname:** unifi
|
|
**IP:** 178.156.131.57
|
|
**Provider:** Hetzner (CPX21, 3C/4G/80GB)
|
|
**OS:** Debian 12
|
|
**Status:** 🟢 LIVE — Recently rebuilt (6h uptime)
|
|
|
|
### Purpose
|
|
UniFi Network Controller — manages Ubiquiti network equipment.
|
|
|
|
### Services
|
|
- MariaDB
|
|
- UniFi Controller software (native, not Docker)
|
|
|
|
### Backup
|
|
- UniFi backup via controller UI (automatic backups)
|
|
- MariaDB database backup
|
|
|
|
### Restore
|
|
```bash
|
|
# 1. Provision CPX21 → Install UniFi controller
|
|
# 2. Restore UniFi backup from last export
|
|
# 3. Update DNS for unifi.itpropartner.com
|
|
# 4. Adopt APs/switches back
|
|
```
|
|
|
|
---
|
|
|
|
## Server 6: hudu.itpropartner.com (Hetzner CPX21)
|
|
|
|
**Hostname:** hudu
|
|
**IP:** 178.156.130.130
|
|
**Provider:** Hetzner (CPX21, 3C/4G/80GB)
|
|
**OS:** Debian 12
|
|
**Status:** 🟢 LIVE
|
|
|
|
### Purpose
|
|
Hudu IT documentation platform — internal knowledge base.
|
|
|
|
### Docker Containers (5)
|
|
hudu2-app-1, hudu2-worker-1, hudu2-db-1, hudu2-redis-1, letsencrypt
|
|
|
|
### Backup
|
|
- **Hudu automatic:** Built-in backup to S3-compatible storage
|
|
- **Postgres DB:** Daily dump
|
|
|
|
### Restore
|
|
```bash
|
|
# 1. Provision CPX21 → Install Docker
|
|
# 2. Deploy Hudu Docker stack
|
|
# 3. Restore Postgres from backup
|
|
# 4. Update DNS
|
|
```
|
|
|
|
---
|
|
|
|
## Server 7: fleettracker360 (Hetzner CPX11)
|
|
|
|
**Hostname:** fleettrack360
|
|
**IP:** 178.156.149.32
|
|
**Provider:** Hetzner (CPX11, 2C/2G/40GB)
|
|
**OS:** Debian 12
|
|
**Status:** 🟢 LIVE
|
|
|
|
### Purpose
|
|
Fleet tracking application (BoxPilot Logistics?).
|
|
|
|
### Services
|
|
- Apache2 (80/443)
|
|
- MariaDB (3306)
|
|
|
|
### Backup
|
|
- Apache config + web root
|
|
- MariaDB database dump
|
|
|
|
---
|
|
|
|
## Server 8: docker (Hetzner CPX11)
|
|
|
|
**Hostname:** docker
|
|
**IP:** 178.156.168.35
|
|
**Provider:** Hetzner (CPX11, 2C/2G/40GB)
|
|
**OS:** Debian 12
|
|
**Status:** 🟢 LIVE — 🔴 Disk 73% used (9.9G free on 38G)
|
|
|
|
### Purpose
|
|
RunCloud staging + utility containers:
|
|
- Nginx Proxy Manager
|
|
- RustDesk server (hbbr, hbbs)
|
|
- Uptime Kuma
|
|
- StrongSwan
|
|
|
|
### Docker Containers (8)
|
|
runcloud_php56, runcloud_php81, runcloud_mariadb, nginx-proxy-manager,
|
|
hbbr, hbbs, strongswan, uptime-kuma
|
|
|
|
---
|
|
|
|
## Server 9: app1 / n8n (Hetzner CPX11)
|
|
|
|
**Hostname:** n8n
|
|
**IP:** 87.99.144.163
|
|
**Provider:** Hetzner (CPX11, 2C/2G/40GB)
|
|
**OS:** Debian 12
|
|
**Status:** 🟢 LIVE
|
|
|
|
### Purpose
|
|
n8n workflow automation.
|
|
|
|
### Docker Containers (2)
|
|
n8n, n8n-postgres
|
|
|
|
---
|
|
|
|
## Server 10: tony-vps (Hetzner CPX21)
|
|
|
|
**Hostname:** tony
|
|
**IP:** 87.99.159.142
|
|
**Provider:** Hetzner (CPX21, 3C/4G/80GB)
|
|
**OS:** Debian 12
|
|
**Status:** 🟢 LIVE
|
|
|
|
### Purpose
|
|
Tony's standalone Hermes agent. Separate from main infrastructure.
|
|
|
|
### Services
|
|
- llama-server (systemd) — Llama 3.2 1B local fallback
|
|
- hermes-heartbeat.sh (cron — every minute)
|
|
|
|
### Notes
|
|
- **Tony's independent instance** — not managed from Core
|
|
- Uses admin-ai for primary model, local Llama 1B for fallback
|
|
- No Docker containers
|
|
|
|
---
|
|
|
|
## Server 11: app1-bu (Hetzner CPX11 — Warm Standby)
|
|
|
|
**Hostname:** app1-bu.itpropartner.com
|
|
**IP:** 5.161.114.8
|
|
**Provider:** Hetzner (CPX11, 2C/2G/40GB)
|
|
**OS:** Debian 12
|
|
**Status:** 🟢 STANDBY — Warm, gateway dormant
|
|
|
|
### Purpose
|
|
**Hermes warm standby.** Clones Core's state via S3 sync.
|
|
|
|
### Services
|
|
- SSH (22)
|
|
- Tailscale
|
|
- node_exporter (9100)
|
|
- WireGuard (wg0 — tunnel to home router at 10.77.0.3)
|
|
|
|
### Failover Procedure
|
|
```bash
|
|
# Automatic (watchdog on app1-bu):
|
|
# 1. Checks Core every 5 min (ping + Hermes health)
|
|
# 2. If Core down for 2 min → sync latest S3 state
|
|
# 3. Integrity check on state.db (PRAGMA quick_check)
|
|
# 4. Start Hermes gateway → Telegram accessible
|
|
# 5. Write ACTIVE_OWNER lock to S3
|
|
|
|
# Manual activation:
|
|
ssh -i /root/.ssh/itpp-infra root@5.161.114.8
|
|
hermes gateway run --replace
|
|
# Or if Core is fenceable:
|
|
ssh root@152.53.192.33 "systemctl stop hermes && systemctl mask hermes"
|
|
```
|
|
|
|
### Restore
|
|
```bash
|
|
# 1. Power on via Hetzner API or console
|
|
# 2. Ensure S3 sync runs:
|
|
# /root/.hermes/scripts/hermes-standby-sync.sh
|
|
# 3. Check integrity:
|
|
# sqlite3 /root/.hermes/state.db "PRAGMA integrity_check;"
|
|
# 4. If good → start gateway:
|
|
# hermes gateway run
|
|
# 5. If corrupt → restore from snapshot:
|
|
# aws s3 cp s3://hermes-vps-backups/snapshots/<latest> /root/
|
|
```
|
|
|
|
---
|
|
|
|
## DR Classification Matrix
|
|
|
|
| Server | Tier | RTO Target | RPO Target | Restore Complexity |
|
|
|--------|------|-----------|-----------|-------------------|
|
|
| Core | 🔴 Critical | 5 min (failover) | 15 min | Medium (Docker + Hermes) |
|
|
| ai (admin-ai) | 🔴 Critical | 2 hours | 24 hours | High (migration) |
|
|
| wphost02 | 🟡 Important | 4 hours | 24 hours | Low (RunCloud restore) |
|
|
| unms | 🟡 Important | 4 hours | 24 hours | Medium (Docker stack) |
|
|
| hudu | 🟡 Important | 4 hours | 24 hours | Medium (Docker stack) |
|
|
| unifi | 🟡 Important | 4 hours | 24 hours | Low (native install) |
|
|
| n8n | 🟢 Standard | 8 hours | 24 hours | Low (2 containers) |
|
|
| docker | 🟢 Standard | 8 hours | 24 hours | Medium (8 containers) |
|
|
| fleettracker360 | 🟢 Standard | 8 hours | 24 hours | Low (LAMP stack) |
|
|
| tony-vps | 🟢 Standard | Coordinate | 24 hours | Low (minimal) |
|
|
| app1-bu | 🔴 Standby | 2 min | 15 min | Low (S3 sync) |
|
|
|
|
---
|
|
|
|
## Recovery Bundle Location
|
|
|
|
Each server's recovery configs are bundled at:
|
|
- `/root/.hermes/recovery-bundle-2026-07-05.md` (Core — needs refresh)
|
|
- `/root/.hermes/references/server-dr-plans.md` (this document)
|
|
- `/root/.hermes/references/server-provisioning-standard-v1.md` (base install)
|
|
|
|
**S3 backup of this document:**
|
|
`s3://hermes-vps-backups/live/config/server-dr-plans.md`
|