ippadmin/hermes-recovery
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial resurrection kit — 81 scripts, 62 references, configs, systemd units, crons, Docker compose files, Caddy config, master README

Browse Source
This commit is contained in:
root
2026-07-15 17:45:36 -04:00
commit ae056eaf83
197 changed files with 26127 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
references/server-dr-plans.md
+453
View File
@@ -0,0 +1,453 @@
# IT Pro Partner — Per-Server Disaster Recovery Plan
**Author:** Sho'Nuff / Network Services Team
**Date:** July 9, 2026
**Version:** 1.0
**Supersedes:** hermes-dr-plan-v2.md (Hermes-specific); server-provisioning-standard-v1.md (base install)
---
## DR Architecture Overview
```
Internet ──► Cloudflare ──► Caddy ──► Service (HTTP/HTTPS)
│
SSH/WG ──► Management plane (via itpp-infra key)
│
S3 (Wasabi) ──► Backup plane (all servers)
```
**Shared dependencies:**
- **SSH access:** `itpp-infra` key (Hetzner vault or deployed per-server)
- **DNS:** Cloudflare (primary) / SiteGround (legacy for itpropartner.com)
- **Backup storage:** Wasabi S3 (us-east-1)
- **SMTP relay:** mail.germainebrown.com:2525 (MXroute)
- **Auth/SSO:** Cloudflare Access (DRE portal)
- **Monitoring:** Prometheus node_exporter on each server
- **VPN:** WireGuard for management access
---
## Server 1: Core (netcup RS 2000 G12)
**Hostname:** core
**IP:** 152.53.192.33
**Provider:** netcup (RS 2000 G12, 8C EPYC 9645, 16GB RAM, 512GB NVMe)
**OS:** Debian 13
**Status:** 🟢 LIVE — Primary Hermes host
### Purpose
Main operations server. Runs Hermes, Docker services, ops portal, WireGuard hub,
SMTP relay, local LLM fallback.
### Services Running
| Service | Type | Port | Notes |
|---------|------|------|-------|
| Hermes Agent | systemd | Telegram | Primary agent - deepseek-chat via admin-ai |
| Caddy | systemd | 80/443 | Reverse proxy for all domains |
| WireGuard (wg0) | systemd | 51821/udp | Tunnel to home router (10.77.0.1) |
| MySQL tunnel | autossh | 33060→wphost02:3306 | Apex database access |
| SearXNG | Docker | 8888 | Self-hosted search |
| DocuSeal | Docker | 3000 | Document signing |
| Vaultwarden | Docker | 8080/tailscale | Password manager |
| Ollama | systemd | 11434 | Local LLM fallback (llama3.2:3b) |
| Ops Portal | systemd | 8090 | FastAPI backend |
| node_exporter | systemd | 9100 | Prometheus metrics |
### Docker Containers
- searxng, docuseal, vaultwarden (compass), + others under /root/docker/
### Backup
- **Hermes state:** Every 15 min to s3://hermes-vps-backups/live/ (state.db + config)
- **Full Hermes:** Daily 5 AM to s3://hermes-vps-backups/hermes-full-backup/
- **Memory snapshots:** Every 10 min to s3://hermes-vps-backups/live/memories/
- **Docker volumes:** Manual (not automated)
### Restore Procedure
```bash
# 1. Provision RS 2000 G12 (or better) on netcup
# 2. Install Debian 13, apply base standard (see server-provisioning-standard-v1.md)
# 3. Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
# 4. Restore from S3 backup:
aws s3 cp s3://hermes-vps-backups/hermes-full-backup/<latest> /root/restore.tar.gz \
--endpoint-url https://s3.us-east-1.wasabisys.com
tar -xzf /root/restore.tar.gz -C /root/.hermes/
# 5. Restore Docker stacks:
cd /root/docker/searxng && docker compose up -d
cd /root/docker/docuseal && docker compose up -d
cd /root/docker/vaultwarden && docker compose up -d
# 6. Restore Caddy config from S3:
aws s3 cp s3://hermes-vps-backups/live/config/Caddyfile /etc/caddy/ \
--endpoint-url https://s3.us-east-1.wasabisys.com
# 7. Restart services:
systemctl restart caddy
systemctl restart hermes
# 8. Verify: ops.itpropartner.com, Teleegram, WG tunnel
```
### Failover
- **app1-bu** (5.161.114.8) is warm standby. Checks Core every 5 min.
- If Core unreachable for 2 min → app1-bu syncs S3 state + starts Hermes gateway.
- Restore Core, then manually fail back.
---
## Server 2: wphost02 (Hetzner CPX21)
**Hostname:** wphost02
**IP:** 5.161.62.38
**Provider:** Hetzner (CPX21, 3C/4G/80GB)
**OS:** Debian 12 (RunCloud managed)
**Status:** 🟡 LIVE — Recently reconnected after SSH firewall fix
### Purpose
Apex Predators Track Experience WordPress site. RunCloud-managed.
WPForms registration + notification emails.
### Services Running
- nginx-rc (RunCloud) — 80/443
- MariaDB — 3306
- Postfix — 25 (relay via mail.germainebrown.com:2525)
- httpd — 81 (RunCloud internal)
- RunCloud agent — 34210
- node_exporter — 9100
### Docker Containers
**None.** RunCloud manages PHP directly.
### Backup
- **RunCloud backup:** Daily via RunCloud panel
- **MariaDB dumps:** Not automated externally
- **WordPress files:** Managed by RunCloud
### Restore Procedure
```bash
# 1. Provision CPX21 (or better) on Hetzner
# 2. Install Debian 12
# 3. Connect to RunCloud via their panel
# 4. RunCloud provisions nginx, MariaDB, PHP
# 5. Restore WordPress from RunCloud backup
# 6. Add itpp-infra SSH key in RunCloud → SSH Access
# 7. Whitelist Core IP (152.53.192.33) in RunCloud firewall
# 8. Restore Postfix relay config:
# - relayhost = mail.germainebrown.com:2525
# - SASL auth with email@redacted
# 9. Verify: apextrackexperience.com loads, forms submit
```
### Dependencies
- Core MySQL tunnel connects here (port 33060→3306)
- Apex mail watchdog monitors via SSH
---
## Server 3: ai.itpropartner.com (Hetzner CPX41)
**Hostname:** ai
**IP:** 178.156.167.181
**Provider:** Hetzner (CPX41, 4C/16G/226GB)
**OS:** Debian 12
**Status:** 🟡 LIVE — 🔴 92% disk full (18G free)
### ⚠️ CRITICAL: Disk at 92%. Needs cleanup before migration.
### Purpose
**Critical infrastructure.** Runs LiteLLM proxy that ALL Hermes instances use
for model inference (deepseek-chat, claude-opus). Also runs:
- Open WebUI — AI chat interface
- Ollama — local LLM inference
- Qdrant — vector database
- Caddy — reverse proxy
### Services Running
- LiteLLM — systemd service, port varies (API at admin-ai.itpropartner.com)
- Docker containers: openwebui, litellm, litellm_postgres, caddy, qdrant, ollama
- node_exporter — 9100
### Backup
- **LiteLLM config:** In /root/. hermes/.env and config.yaml (API key: sk-lbh...)
- **Docker volumes:** Not backed up to S3
- **Ollama models:** Would need re-pull (cache only)
### Restore Procedure
```bash
# 1. Provision CPX41 (or better) on netcup as part of migration
# 2. Install Debian 13, Docker
# 3. Deploy LiteLLM Docker stack:
# - From /opt/litellm/docker-compose.yml
# - Restore postgres data for LiteLLM routes
# 4. Deploy Ollama:
# - Pull model: ollama pull llama3.2:3b
# 5. Deploy Open WebUI:
# - From /opt/openwebui/docker-compose.yml
# 6. Deploy Caddy config for admin-ai.itpropartner.com
# 7. Update DNS: admin-ai.itpropartner.com → new IP
# 8. Update Hermes config.yaml on all profiles to point to new IP
# 9. Verify: curl https://admin-ai.itpropartner.com/v1/models
# 10. Decommission old Hetzner server
```
### 🔴 Pre-Reboot Checklist
```bash
# Before ANY reboot or migration of this server:
# 1. Verify at least ONE alternative model provider works:
curl https://admin-ai.itpropartner.com/v1/chat/completions \
-H "Authorization: Bearer sk-lbh..." \
-d '{"model":"deepseek-chat","messages":[{"role":"user","content":"ping"}]}'
# 2. Ensure local Ollama on Core is running:
curl http://127.0.0.1:11434/api/tags
# 3. Notify all Hermes users (Germaine, Anita, Tony)
```
### Dependencies
- **ALL Hermes instances** depend on this server's LiteLLM proxy
- If this goes down, Hermes falls back to local Ollama (llama3.2:3b on Core)
- **Tony's Hermes** has no fallback — would be offline
---
## Server 4: unms.forefrontwireless.com (Hetzner CPX21)
**Hostname:** unms
**IP:** 5.161.225.131
**Provider:** Hetzner (CPX21, 3C/4G/80GB)
**OS:** Debian 12
**Status:** 🟢 LIVE — 4d 11h uptime
### Purpose
UISP/UNMS (WISP management) + UCRM (customer management).
Manages tower routers and customer billing for Forefront Wireless.
### Docker Containers (9)
unms-api, unms-device-ws-1, unms-device-ws-2, unms-nginx, unms-rabbitmq,
unms-postgres, unms-siridb, unms-fluentd, ucrm
### Backup
- **Docker volumes:** postgres data, UCRM data
- **UNMS config:** Exportable from UNMS UI
### Restore
```bash
# 1. Provision CPX21 → Reinstall Debian 12 + Docker
# 2. Restore docker-compose stacks from backup
# 3. Restore Postgres volumes
# 4. Update DNS for unms.forefrontwireless.com
# 5. Re-register tower routers in UNMS
```
---
## Server 5: unifi (Hetzner CPX21)
**Hostname:** unifi
**IP:** 178.156.131.57
**Provider:** Hetzner (CPX21, 3C/4G/80GB)
**OS:** Debian 12
**Status:** 🟢 LIVE — Recently rebuilt (6h uptime)
### Purpose
UniFi Network Controller — manages Ubiquiti network equipment.
### Services
- MariaDB
- UniFi Controller software (native, not Docker)
### Backup
- UniFi backup via controller UI (automatic backups)
- MariaDB database backup
### Restore
```bash
# 1. Provision CPX21 → Install UniFi controller
# 2. Restore UniFi backup from last export
# 3. Update DNS for unifi.itpropartner.com
# 4. Adopt APs/switches back
```
---
## Server 6: hudu.itpropartner.com (Hetzner CPX21)
**Hostname:** hudu
**IP:** 178.156.130.130
**Provider:** Hetzner (CPX21, 3C/4G/80GB)
**OS:** Debian 12
**Status:** 🟢 LIVE
### Purpose
Hudu IT documentation platform — internal knowledge base.
### Docker Containers (5)
hudu2-app-1, hudu2-worker-1, hudu2-db-1, hudu2-redis-1, letsencrypt
### Backup
- **Hudu automatic:** Built-in backup to S3-compatible storage
- **Postgres DB:** Daily dump
### Restore
```bash
# 1. Provision CPX21 → Install Docker
# 2. Deploy Hudu Docker stack
# 3. Restore Postgres from backup
# 4. Update DNS
```
---
## Server 7: fleettracker360 (Hetzner CPX11)
**Hostname:** fleettrack360
**IP:** 178.156.149.32
**Provider:** Hetzner (CPX11, 2C/2G/40GB)
**OS:** Debian 12
**Status:** 🟢 LIVE
### Purpose
Fleet tracking application (BoxPilot Logistics?).
### Services
- Apache2 (80/443)
- MariaDB (3306)
### Backup
- Apache config + web root
- MariaDB database dump
---
## Server 8: docker (Hetzner CPX11)
**Hostname:** docker
**IP:** 178.156.168.35
**Provider:** Hetzner (CPX11, 2C/2G/40GB)
**OS:** Debian 12
**Status:** 🟢 LIVE — 🔴 Disk 73% used (9.9G free on 38G)
### Purpose
RunCloud staging + utility containers:
- Nginx Proxy Manager
- RustDesk server (hbbr, hbbs)
- Uptime Kuma
- StrongSwan
### Docker Containers (8)
runcloud_php56, runcloud_php81, runcloud_mariadb, nginx-proxy-manager,
hbbr, hbbs, strongswan, uptime-kuma
---
## Server 9: app1 / n8n (Hetzner CPX11)
**Hostname:** n8n
**IP:** 87.99.144.163
**Provider:** Hetzner (CPX11, 2C/2G/40GB)
**OS:** Debian 12
**Status:** 🟢 LIVE
### Purpose
n8n workflow automation.
### Docker Containers (2)
n8n, n8n-postgres
---
## Server 10: tony-vps (Hetzner CPX21)
**Hostname:** tony
**IP:** 87.99.159.142
**Provider:** Hetzner (CPX21, 3C/4G/80GB)
**OS:** Debian 12
**Status:** 🟢 LIVE
### Purpose
Tony's standalone Hermes agent. Separate from main infrastructure.
### Services
- llama-server (systemd) — Llama 3.2 1B local fallback
- hermes-heartbeat.sh (cron — every minute)
### Notes
- **Tony's independent instance** — not managed from Core
- Uses admin-ai for primary model, local Llama 1B for fallback
- No Docker containers
---
## Server 11: app1-bu (Hetzner CPX11 — Warm Standby)
**Hostname:** app1-bu.itpropartner.com
**IP:** 5.161.114.8
**Provider:** Hetzner (CPX11, 2C/2G/40GB)
**OS:** Debian 12
**Status:** 🟢 STANDBY — Warm, gateway dormant
### Purpose
**Hermes warm standby.** Clones Core's state via S3 sync.
### Services
- SSH (22)
- Tailscale
- node_exporter (9100)
- WireGuard (wg0 — tunnel to home router at 10.77.0.3)
### Failover Procedure
```bash
# Automatic (watchdog on app1-bu):
# 1. Checks Core every 5 min (ping + Hermes health)
# 2. If Core down for 2 min → sync latest S3 state
# 3. Integrity check on state.db (PRAGMA quick_check)
# 4. Start Hermes gateway → Telegram accessible
# 5. Write ACTIVE_OWNER lock to S3
# Manual activation:
ssh -i /root/.ssh/itpp-infra root@5.161.114.8
hermes gateway run --replace
# Or if Core is fenceable:
ssh root@152.53.192.33 "systemctl stop hermes && systemctl mask hermes"
```
### Restore
```bash
# 1. Power on via Hetzner API or console
# 2. Ensure S3 sync runs:
# /root/.hermes/scripts/hermes-standby-sync.sh
# 3. Check integrity:
# sqlite3 /root/.hermes/state.db "PRAGMA integrity_check;"
# 4. If good → start gateway:
# hermes gateway run
# 5. If corrupt → restore from snapshot:
# aws s3 cp s3://hermes-vps-backups/snapshots/<latest> /root/
```
---
## DR Classification Matrix
| Server | Tier | RTO Target | RPO Target | Restore Complexity |
|--------|------|-----------|-----------|-------------------|
| Core | 🔴 Critical | 5 min (failover) | 15 min | Medium (Docker + Hermes) |
| ai (admin-ai) | 🔴 Critical | 2 hours | 24 hours | High (migration) |
| wphost02 | 🟡 Important | 4 hours | 24 hours | Low (RunCloud restore) |
| unms | 🟡 Important | 4 hours | 24 hours | Medium (Docker stack) |
| hudu | 🟡 Important | 4 hours | 24 hours | Medium (Docker stack) |
| unifi | 🟡 Important | 4 hours | 24 hours | Low (native install) |
| n8n | 🟢 Standard | 8 hours | 24 hours | Low (2 containers) |
| docker | 🟢 Standard | 8 hours | 24 hours | Medium (8 containers) |
| fleettracker360 | 🟢 Standard | 8 hours | 24 hours | Low (LAMP stack) |
| tony-vps | 🟢 Standard | Coordinate | 24 hours | Low (minimal) |
| app1-bu | 🔴 Standby | 2 min | 15 min | Low (S3 sync) |
---
## Recovery Bundle Location
Each server's recovery configs are bundled at:
- `/root/.hermes/recovery-bundle-2026-07-05.md` (Core — needs refresh)
- `/root/.hermes/references/server-dr-plans.md` (this document)
- `/root/.hermes/references/server-provisioning-standard-v1.md` (base install)
**S3 backup of this document:**
`s3://hermes-vps-backups/live/config/server-dr-plans.md`