hermes-recovery/references/server-dr-plans-public.md

# Company Name — Disaster Recovery Plan v3 (Consolidated)

**Author:** [Lead Sysadmin] / Network Services Team, incorporating third-party DR review
**Date:** July 9, 2026
**Status:** LIVE — replaces v1.0
**Supersedes:** server-dr-plans.md v1.0, server-provisioning-standard-v1.md, hermes-dr-plan-v2.md

---

## Table of Contents
1. Architecture & Shared Dependencies
2. Server Standards
3. Backup Strategy
4. Failover / Failback
5. Monitoring & Alerting
6. Restore Runbooks
7. Testing & Validation
8. Security
9. Immediate Action Items
10. Appendix: Third-Party Review Responses

---

## 1. Architecture & Shared Dependencies

```
Internet ──► CDN/DNS Provider ──► Caddy ──► Service (HTTP/HTTPS)
                          │
              ┌───────────┼───────────┐
              │           │           │
         SSH/WG     Tailscale    Management
         (company-prefix-     (Primary Host ↔      VPN (WireGuard
          infra)     Standby Host)     to home router)
              │           │           │
              └───────────┼───────────┘
                          │
              ┌───────────┴───────────┐
              │                       │
      S3 Primary                 S3 Secondary
   (Object Storage Provider us-east-1)        (Object Storage Provider us-west-2
        versioning ON          or Backblaze B2)
        object lock ON        object lock ON
```

### Shared Dependencies — Documented

| Dependency | Primary | Backup / Fallback | Credentials Location |
|---|---|---|---|
| SSH access | company-prefix-infra key (Secondary Provider vault) | Rescue mode / console | ~/.hermes/.env + Hudu |
| DNS | CDN/DNS Provider (API token in .env) | Manual via web UI | ~/.hermes/.env |
| S3 backup (primary) | Object Storage Provider us-east-1 | Object Storage Provider us-west-2 (to be created) | ~/.aws/credentials |
| S3 backup (secondary) | Object Storage Provider us-west-2 | Backblaze B2 (evaluate) | TBD |
| SMTP relay | [MAIL-SERVER-DOMAIN]:2525 | Direct Email Provider (port 465/587) | ~/.hermes/.env |
| Auth/SSO | CDN/DNS Provider Access | Local portal auth fallback | CDN/DNS Provider dashboard |
| Monitoring | Prometheus node_exporter | Uptime Kuma (docker box) | N/A (pull model) |
| VPN management | WireGuard (wg0, port 51821) | Tailscale direct tunnel | /etc/wireguard/ |

---


### 3.1 Backup Matrix

| Data Type | Frequency | Target | Retention | Status |
|---|---|---|---|---|
| Hermes state | Every 15 min | S3 primary (live/) | 48 hours | ✅ LIVE |
| Hermes full | Daily 5 AM | S3 primary (full/) | 90 days | ✅ LIVE |
| Memory snapshots | Every 10 min | S3 primary (memories/) | 48 hours | ✅ LIVE (new) |
| Memory history | Every 10 min | S3 primary (memories/history/) | 90 days | ✅ LIVE (new) |
| Router configs | Daily 6 AM | S3 primary (mikrotik/) | 90 days | ✅ LIVE |
| System configs | Daily | S3 primary (<server>/) | 90 days | ⚠️ NOT AUTOMATED |
| Docker volumes | Daily | S3 primary (<server>/) | 90 days | ❌ NOT DONE |
| Database dumps | Every 6 hours | S3 primary (<server>/) | 90 days | ❌ NOT DONE |
| Application data | Per-app | S3 primary (<server>/) | 90 days | ⚠️ PARTIAL |
| S3 secondary | Daily sync | Object Storage Provider us-west-2 | 90 days | ❌ NOT SET UP |

### 3.3 Backup Security

- **Versioning:** ON (all 3 buckets) ✅
- **Object Lock:** NOT enabled — needs implementation
- **Delete protection:** NOT configured
- **Backup credentials:** Write-only IAM user — NOT implemented (using admin credentials)
- **Encryption at rest:** Object Storage Provider server-side (default) — no client-side encryption
- **Credentials escrow:** Partial (in .env + Hudu)

**Fixes needed:**
1. Create IAM sub-user with `s3:PutObject` only (no delete, no list)
2. Enable Object Lock with 7-day minimum retention
3. Store write-only credentials separately from admin credentials
4. Client-side encrypt backup before upload (age/gpg) for the secondary copy

---

## 4. Failover / Failback

### 4.1 Warm Standby — Corrected Timing

The DR reviewer flagged this correctly: you cannot detect a 2-minute outage with a 5-minute check interval.

**Old:** Check every 5 min, failover after 2 min unreachable  
**New:** Check every 30 seconds, failover after 4 failed checks (~2 min)

```python
# Pseudocode for corrected watchdog
CHECK_INTERVAL = 30       # seconds
FAILURE_THRESHOLD = 4      # consecutive failures
CONFIRMATION_WINDOW = 120  # seconds (2 min)
# Detection time: 4 * 30s = 120s = 2 minutes
# Sync + start time: ~45-90 seconds
# Effective RTO: ~3-3.5 minutes
# Stated RTO target: 5 minutes (ACHIEVABLE)
```

### 4.2 Health Check — Two Layers

**Layer 1 (Ping):** ICMP to Primary Host ([IP-REDACTED]) every 30s  
**Layer 2 (HTTP):** `GET /[GENERIC-HEALTH-ENDPOINT]` on Tailscale IP — checks:
- Gateway heartbeat ≤ 5s old
- SQLite `PRAGMA quick_check` returns "ok"
- Local Ollama responding

**Decision Matrix:**

| Server | Hermes | Action |
|---|---|---|
| ✅ UP | ✅ UP | Stay dormant |
| ❌ DOWN | N/A | **Auto-takeover** — unambiguous |
| ✅ UP | ❌ DOWN | **ALERT + fence** — attempt SSH fence, only takeover if Primary Host Hermes confirmed stopped |

### 4.3 Fencing Protocol

```bash
# SSH Fence (preferred — server is up)
ssh [USER-REDACTED]@[IP-REDACTED] \
  "sudo systemctl stop hermes && sudo systemctl mask hermes"
# Verify: returns "inactive"
ssh [USER-REDACTED]@[IP-REDACTED] "sudo systemctl is-active hermes"

# S3 Ownership Lock
# Object: s3://hermes-vps-backups/ACTIVE_OWNER
# Primary Host refreshes every 60s while healthy
# Standby writes itself as owner on takeover
# Primary Host reads on startup — if standby owns it, Primary Host stays dormant
```

### 4.4 Failback Protocol

**⚠️ Auto-failback is FORBIDDEN.** Human clearance required.

```bash
# FAILBACK STEPS (manual):
# 1. Confirm Primary Host is healthy: SSH + systemctl is-active hermes
# 2. Stop standby gateway: ssh Standby Host "hermes gateway stop"
# 3. Sync state from standby back to Primary Host (if needed):
#    Rsync /root/.hermes/ from standby to Primary Host
# 4. Unmask Hermes on Primary Host:
#    ssh Primary Host "sudo systemctl unmask hermes && sudo systemctl start hermes"
# 5. Verify Primary Host gateway is active (Telegram responds)
# 6. Standby returns to dormant state
# 7. Update S3 ownership lock to "core"
```

### 4.5 Provider/Account Outage Scenario

If the entire hosting provider (Primary Provider) is unavailable:
1. Standby Host (Secondary Provider) takes over as active Hermes host
2. DNS records updated to point to Standby Host IP ([IP-REDACTED])
3. Services migrate to Primary Provider replacement when available
4. No auto-failback — human decision

---

## 5. Monitoring & Alerting

### 5.1 Alert Thresholds

| Alert | Threshold | Action | Priority |
|---|---|---|---|
| Disk usage | > 80% | Notify, investigate | 🟡 P2 |
| Disk usage | > 90% | Notify, immediate cleanup | 🔴 P0 |
| Backup age | > 24h since last success | Notify | 🟡 P1 |
| Backup failure | Any failed job | Notify | 🔴 P0 |
| Hermes heartbeat | No heartbeat > 30s | Notify | 🔴 P0 |
| Hermes down + server up | Layer 2 fail, Layer 1 pass | IMMEDIATE alert | 🔴 P0 |
| TLS expiration | < 14 days | Notify | 🟡 P1 |
| TLS expiration | < 7 days | Notify + remind | 🔴 P0 |
| SSH fail spikes | > 10 failures/min | Investigate | 🟡 P1 |
| Service down | Caddy/nginx/Docker inactive | Notify, attempt restart | 🔴 P0 |
| S3 sync failure | Upload fails | Abort (memory consolidate) | 🔴 P0 |
| Standby stale | No sync > 30 min | Notify | 🟡 P1 |

### 5.2 Monitoring Stack

- **Per-server:** `prometheus-node-exporter` (all boxes)
- **Dashboard:** Prometheus → Grafana ()
- **Synthetic checks:** Uptime Kuma (on docker box, [IP-REDACTED])
- **Hermes watchdog:** `service-health-check.sh` (every 5 min, Primary Host)
- **Standby watcher:** `home-router-watchdog.sh` (every 5 min, Primary Host)
- **Apex mail:** `apex-mail-watchdog.sh` (every 5 min, Primary Host)
- **Alert delivery:** Telegram (via Hermes)

## 6. Restore Runbooks (Per Server)

| Attribute | Detail |
|---|---|
| RTO target | 5 minutes (via failover to Standby Host) |
| RPO target | 15 minutes (Hermes state), 24h (Docker data) |
| Actual RTO | ~3 minutes (corrected watchdog) |
| Actual RPO | Varies by data type (see below) |

**Restore steps:**
```bash
# Option A: Failover to standby (fastest — < 5 min)
# Standby Host detects Primary Host down → auto-takeover (Section 4.1)

# Option B: Full restore on replacement box
# Step 1 — Provision RS 2000 G12 on Primary Provider
# Step 2 — Apply base standard (Section 2.2)
# Step 3 — Install Hermes
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
# Step 4 — Restore Hermes state from latest S3:
aws s3 cp s3://hermes-vps-backups/live/state.db /root/.hermes/state.db \
  --endpoint-url https://[S3-STORAGE-DOMAIN]
aws s3 cp s3://hermes-vps-backups/live/config.yaml /root/.hermes/config.yaml \
  --endpoint-url https://[S3-STORAGE-DOMAIN]
# Step 5 — Restore .env from backup:
aws s3 cp s3://hermes-vps-backups/live/.env /root/.hermes/.env \
  --endpoint-url https://[S3-STORAGE-DOMAIN]
chmod 600 /root/.hermes/.env
# Step 6 — Restore Docker stacks:
for dir in /root/docker/*/; do
  (cd "dir" && docker compose up -d)
done
# Step 7 — Restore Caddy config:
aws s3 cp s3://hermes-vps-backups/live/config/Caddyfile /etc/caddy/Caddyfile
systemctl restart caddy
# Step 8 — Start Hermes gateway:
hermes gateway run --replace
# Step 9 — Verify:
curl -s https://[OPS-PORTAL-DOMAIN]/[GENERIC-HEALTH-ENDPOINT]
curl -s https://[DOCUMENT-SIGNING-DOMAIN]
# Check Telegram responds to /ping
```

### 6.2 [AI-SERVICE-DOMAIN] (Secondary Provider CPX41) — 🔴 Critical

| Attribute | Detail |
|---|---|
| RTO target | 2 hours |
| RPO target | 24 hours |
| Current disk | **🔴 92%** — must clean before migration |

**⚠️ Pre-reboot checklist:**
```bash
# Verify at least one alternative model provider works
curl https://[AI-ADMIN-DOMAIN]/[GENERIC-LLM-ENDPOINT] \
  -H "Authorization: Bearer (cat ~/.hermes/.env | grep ADMIN_AI | cut -d= -f2)" \
  -d '{"model":"deepseek-chat","messages":[{"role":"user","content":"ping"}]}'
# Ensure local Ollama on Primary Host is running
curl -s http://[IP-REDACTED]:11434/[GENERIC-OLLAMA-ENDPOINT] || systemctl start ollama
# Notify all Hermes users ([CEO], [Operations Manager], [Developer])
```

**Restore steps:**
```bash
# Step 1 — Provision replacement (CPX41 or Primary Provider equivalent)
# Step 2 — Apply base standard (Section 2.2)
# Step 3 — Deploy Docker stacks:
cd /opt/litellm && docker compose up -d
cd /opt/openwebui && docker compose up -d
cd /opt/qdrant && docker compose up -d
# Step 4 — Pull models:
ollama pull llama3.2:3b
# Step 5 — Configure DNS: [AI-ADMIN-DOMAIN] → new IP
# Step 6 — Update all Hermes config.yaml files with new base_url
# Step 7 — Verify model access for all profiles
```

### 6.3 Legacy Web Host (Secondary Provider CPX21, Server Management Panel) — 🟡 Important

| Attribute | Detail |
|---|---|
| RTO target | 4 hours |
| RPO target | 24 hours |
| Backup method | Server Management Panel daily backup + manual DB |

**Restore steps:**
```bash
# Step 1 — Provision CPX21 on Secondary Provider
# Step 2 — Connect to Server Management Panel panel, link server
# Step 3 — Server Management Panel provisions nginx, MariaDB, PHP
# Step 4 — Restore WordPress from Server Management Panel backup → verify apex site loads
# Step 5 — Deploy company-prefix-infra SSH key in Server Management Panel → SSH Access
# Step 6 — Whitelist Primary Host IP ([IP-REDACTED]) in Server Management Panel firewall
# Step 7 — Reconfigure Postfix relay:
postconf -e "relayhost = [MAIL-SERVER-DOMAIN]:2525"
postconf -e "smtp_sasl_auth_enable = yes"
echo "[MAIL-SERVER-DOMAIN]:2525 [EMAIL-REDACTED]:[CREDENTIAL-REDACTED]" > /etc/postfix/sasl_passwd
postmap /etc/postfix/sasl_passwd && systemctl restart postfix
# Step 8 — Flush postfix queue:
postqueue -f
```

### 6.4 UNMS (Secondary Provider CPX21) — 🟡 Important

| Attribute | Detail |
|---|---|
| RTO target | 4 hours |
| RPO target | 24 hours |
| Containers | 9 (Docker compose stack) |

**Restore steps:**
```bash
# Step 1 — Provision CPX21 → Docker
# Step 2 — Restore Postgres data volumes from S3
# Step 3 — Deploy docker-compose.yml
cd /opt/unms && docker compose up -d
# Step 4 — Verify UCRM + UNMS web login
# Step 5 — Re-register tower routers via UNMS
```

### 6.5 Hudu (Secondary Provider CPX21) — 🟡 Important

| Attribute | Detail |
|---|---|
| RTO target | 4 hours |
| RPO target | 24 hours |
| Containers | 5 (Docker compose) |
| Auto-backup | Hudu built-in S3 backup |

**Restore steps:**
```bash
# Step 1 — Provision CPX21 → Docker
# Step 2 — Deploy Hudu Docker stack
# Step 3 — Restore Postgres from last backup
# Step 4 — Verify web login + recent documents accessible
```

### 6.6 UniFi (Secondary Provider CPX21) — 🟡 Important

| Attribute | Detail |
|---|---|
| RTO target | 4 hours |
| RPO target | 24 hours |
| Type | Native install (not Docker) |

**Restore steps:**
```bash
# Step 1 — Provision CPX21
# Step 2 — Install UniFi Controller (native)
# Step 3 — Restore from UniFi backup file
# Step 4 — Adopt APs and switches
```

### 6.7 n8n (Secondary Provider CPX11) — 🟢 Standard

| Attribute | Detail |
|---|---|
| RTO target | 8 hours |
| Containers | 2 (n8n + postgres) |

**Restore:**
```bash
# Provision → Docker → restore postgres volume
cd /opt/n8n && docker compose up -d
```

### 6.8 docker box (Secondary Provider CPX11) — 🟢 Standard

| Attribute | Detail |
|---|---|
| RTO target | 8 hours |
| Containers | 8 (NPM, RustDesk, Uptime Kuma, Server Management Panel) |

**Restore:**
```bash
# Provision → Docker → restore compose stacks + volumes
```

### 6.9 fleettracker360 (Secondary Provider CPX11) — 🟢 Standard

| Attribute | Detail |
|---|---|
| RTO target | 8 hours |
| Type | Apache + MariaDB (LAMP) |

**Restore:**
```bash
# Provision → Apache + MariaDB → restore web root + DB
```

### 6.10 developer-vps (Secondary Provider CPX21) — 🟢 Standard

| Attribute | Detail |
|---|---|
| RTO target | Coordinate with [Developer] |
| Type | Independent Hermes + local LLM |

**⚠️ SHADOW IT — needs documentation.**

**Current known config:**
- Hermes gateway with admin-ai model access
- Llama 3.2 1B local fallback (systemd: llama-server)
- Cron: hermes-heartbeat.sh every minute
- Minimal services, no Docker

**Actions needed:**
1. Document [Developer]'s full Hermes config.yaml and .env
2. Backup to S3 under s3://hermes-vps-backups/developer-vps/
3. Add [Developer]'s config to DR plan once documented

### 6.11 Standby Host (Secondary Provider CPX21) — 🔴 Standby

| Attribute | Detail |
|---|---|
| RTO target | 2 minutes |
| RPO target | 15 minutes |
| Type | Warm standby — gateway dormant |

**Activation:**
```bash
# Automatic: watchdog detects Primary Host down → S3 sync → start gateway
# Manual:
ssh [USER-REDACTED]@[IP-REDACTED]
sudo -i
hermes gateway run --replace
```

---

## 7. Testing & Validation

### 7.1 Restore Testing Schedule

| Test | Frequency | Who |
|---|---|---|
| Verify backup job completion (all servers) | Daily | Automated alert |
| Check backup age per server | Daily | Automated |
| Restore one database dump to test environment | Monthly | Network Team |
| Restore one full application stack | Quarterly | Network Team |
| Full DR simulation — Primary Host failover to Standby Host | Quarterly | Network Team |
| S3 credential restore/access test | Quarterly | Network Team |
| Document restore test results (date, tester, duration, findings) | Per test | Network Team |

### 7.2 Validation Criteria

Each restore test must produce:
- Date + tester name
- Server/application tested
- Backup artifact used (path + timestamp)
- Restore duration (start → fully operational)
- Data validation: check a known record exists (specific row, file, or document)
- Issues encountered
- Remediation ticket (if needed)

### 7.3 Quarterly DR Simulation

Scenario: "Primary Host is completely gone. Primary Provider account inaccessible. Standby Host has been active for 4 hours. Now Primary Provider is back — fail back."

Steps to validate:
1. Simulate Primary Provider provider outage (block Primary Host IP)
2. Verify Standby Host takes over within 5 minutes
3. Send a Telegram message through standby
4. Verify all cron jobs run on standby
5. Simulate Primary Provider recovery
6. Execute manual failback per Section 4.4
7. Verify no data loss, no split-brain

---

## 8. Security

### 8.1 Current Security Posture

| Control | Status | Notes |
|---|---|---|
| SSH keys only | ✅ Done | Password auth disabled |
| Root SSH disabled | ⚠️ Partial | Some servers still allow root |
| UFW host firewall | ✅ All | Standard on all boxes |
| fail2ban | ✅ All | Standard on all boxes |
| S3 versioning | ✅ All buckets | Enabled |
| S3 Object Lock | ❌ Not done | Needs implementation |
| S3 write-only IAM | ❌ Not done | Currently using admin creds |
| Backup encryption | ❌ Not done | Server-side only; no client-side |
| Secrets escrow | 🟡 Partial | .env + Hudu — not all secrets documented |
| TLS everywhere | ✅ All public endpoints | Caddy auto-certs |

### 8.3 Break-Glass Access

If company-prefix-infra key is lost or revoked:
1. **Secondary Provider servers:** Use Secondary Provider console → Rescue mode → mount filesystem → redeploy key
2. **Primary Provider servers:** Use SCP web console → VNC → log in with root password → redeploy key
3. **Root passwords:** Stored in ~/.hermes/.env (chmod 600) and Hudu

---