hermes-skills/skills/devops/disaster-recovery-audit/references/server-dr-plans-v3.md

# IT Pro Partner — Disaster Recovery Plan v3 (Consolidated)

**Author:** Sho'Nuff / Network Services Team, incorporating third-party DR review
**Date:** July 10, 2026
**Status:** LIVE — replaces v1.0
**Supersedes:** server-dr-plans.md v1.0, server-provisioning-standard-v1.md, hermes-dr-plan-v2.md

---

## Table of Contents
1. Architecture & Shared Dependencies
2. Server Standards
3. Backup Strategy
4. Failover / Failback
5. Monitoring & Alerting
6. Restore Runbooks
7. Testing & Validation
8. Security
9. Immediate Action Items
10. Appendix: Third-Party Review Responses

---

## 1. Architecture & Shared Dependencies

```
Internet -- Cloudflare -- Caddy -- Service (HTTP/HTTPS)
                          |
              +-----------+-----------+
              |           |           |
         SSH/WG     Tailscale    Management
         (itpp-     (Core <->     VPN (WireGuard
          infra)     app1-bu)     to home router)
              |           |           |
              +-----------+-----------+
                          |
              +-----------+-----------+
              |                       |
      S3 Primary                 S3 Secondary
   (Wasabi us-east-1)        (Wasabi us-west-2
        versioning ON          or Backblaze B2)
        object lock ON        object lock ON
```

### Shared Dependencies

| Dependency | Primary | Backup / Fallback | Credentials Location |
|---|---|---|---|
| SSH access | itpp-infra key (Hetzner vault) | Rescue mode / console | ~/.hermes/.env + Hudu |
| DNS | Cloudflare (API token in .env) | Manual via web UI | ~/.hermes/.env |
| S3 backup (primary) | Wasabi us-east-1 | Wasabi us-west-2 (to be created) | ~/.aws/credentials |
| S3 backup (secondary) | Wasabi us-west-2 | Backblaze B2 (evaluate) | TBD |
| SMTP relay | mail.germainebrown.com:2525 | Direct MXroute (port 465/587) | ~/.hermes/.env |
| Auth/SSO | Cloudflare Access | Local portal auth fallback | Cloudflare dashboard |
| Monitoring | Prometheus node_exporter | Uptime Kuma (docker box) | N/A (pull model) |
| VPN management | WireGuard (wg0, port 51821) | Tailscale direct tunnel | /etc/wireguard/ |

---

## 2. Server Standards

### 2.1 Tier Definitions

| Tier | Spec | OS | Use | Cost |
|---|---|---|---|---|
| **Standard** | RS 4000 G12 (12C/32G/1TB) | Debian 13 | app1, app2, app3 | ~$44/mo |
| **Light** | RS 2000 G12 (8C/16G/512GB) | Debian 13 | Core | ~$24/mo |
| **Standby** | CPX21 (4C/8G/80GB) | Debian 13 | app1-bu | ~$14/mo |
| **Legacy** | Variable Hetzner | Debian 12 | Existing (migrate on rebuild) | ~$8-44/mo |

### 2.2 Base Install -- Verified Checklist

```bash
# S1: hostname + timezone
hostnamectl set-hostname <name>
timedatectl set-timezone America/New_York

# S2: system update
apt update && apt upgrade -y

# S3: security baseline
apt install -y fail2ban ufw
ufw default deny incoming; ufw default allow outgoing
ufw allow ssh; ufw --force enable

# S4: monitoring
apt install -y prometheus-node-exporter

# S5: standard user + key
adduser ippadmin && usermod -aG sudo ippadmin
echo "ssh-ed25519 AAA... itpp-infra" >> /home/ippadmin/.ssh/authorized_keys

# S6: harden SSH
sed -i 's/^#PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config
sed -i 's/^PermitRootLogin yes/PermitRootLogin prohibit-password/' /etc/ssh/sshd_config
systemctl restart sshd

# S7: Docker (if needed)
curl -fsSL https://get.docker.com | bash
apt install -y docker-compose-plugin
```

### 2.3 Application Data Paths

Every server must document:
- What data lives where (volumes, bind mounts, databases, uploads, .env files)
- What can be rebuilt vs what must be restored
- Secrets location
- Caddy/nginx configs
- Cron jobs
- External API dependencies

---

## 3. Backup Strategy

### 3.1 Backup Matrix

| Data Type | Frequency | Target | Retention | Status |
|---|---|---|---|---|
| Hermes state | Every 15 min | S3 primary (live/) | 48 hours | ✅ LIVE |
| Hermes full | Daily 5 AM | S3 primary (full/) | 90 days | ✅ LIVE |
| Memory snapshots | Every 10 min | S3 primary (memories/) | 48 hours | ✅ LIVE |
| Memory history | Every 10 min | S3 primary (memories/history/) | 90 days | ✅ LIVE |
| Router configs | Daily 6 AM | S3 primary (mikrotik/) | 90 days | ✅ LIVE |
| System configs | Daily | S3 primary (<server>/) | 90 days | 🔲 NOT AUTOMATED |
| Docker volumes | Daily | S3 primary (<server>/) | 90 days | ❌ NOT DONE |
| Database dumps | Every 6 hours | S3 primary (<server>/) | 90 days | ❌ NOT DONE |
| Application data | Per-app | S3 primary (<server>/) | 90 days | 🔲 PARTIAL |
| S3 secondary | Daily sync | Wasabi us-west-2 | 90 days | ❌ NOT SET UP |

### 3.2 Backup Security

- **Versioning:** ON (all 3 buckets)
- **Object Lock:** NOT enabled
- **Delete protection:** NOT configured
- **Backup credentials:** Admin-level (can delete) -- NOT write-only
- **Encryption:** Server-side only (Wasabi default)

---

## 4. Failover / Failback

### 4.1 Warm Standby Timing (Corrected)

**Old:** Check every 5 min, failover after 2 min unreachable (impossible -- can't detect 2-min outage at 5-min intervals)

**New:** Check every 30 seconds, failover after 4 failed checks (~2 min detection) + 45-90s sync = ~3-3.5 min actual, 5 min committed RTO

### 4.2 Health Check -- Two Layers

**Layer 1 (Ping):** ICMP to Core (152.53.192.33) every 30s
**Layer 2 (HTTP):** GET /healthz on Tailscale IP -- checks Gateway heartbeat, SQLite PRAGMA, local Ollama

### 4.3 Split-Brain Prevention

S3 active-lock.json mechanism:

```json
{"active_node": "core", "heartbeat": "<timestamp>", "lock_holder": "core"}
```

Core writes heartbeat every 30s while healthy. Failover only proceeds after:
1. Core unreachable for 4 consecutive checks (app1-bu local)
2. External monitor independently confirms Core down
3. active-lock.json heartbeat stale (>90s)
4. app1-bu writes itself as lock_holder
5. Gateway starts
6. Operator alerted

If Core can be reached for fencing, app1-bu SSHes in and stops/masks Hermes before taking over.

### 4.4 Failback Protocol

**No auto-failback.** Sequence:
1. Confirm app1-bu is active node
2. Keep Core stopped -- no auto-start
3. Restore Core OS + stack
4. Sync latest state from app1-bu/S3 into Core
5. Validate sync timestamp
6. Start Core in passive mode
7. Stop app1-bu gateway
8. Update active-lock.json: "active_node": "core"
9. Start Core gateway
10. 5 consecutive health checks pass
11. Release standby lock
12. Monitor 30 min before closing

**Rollback:** If Core fails health checks mid-failback, revert immediately -- restore app1-bu as active, keep Core in maintenance.

### 4.5 Provider/Account Outage

If entire netcup account is unavailable:
1. app1-bu (Hetzner) takes over
2. DNS records pointed to app1-bu IP
3. No auto-failback -- human decision

---

## 5. Per-Server RTO/RPO Targets

| System | RTO | RPO | Notes |
|---|---|---|---|
| Core (Hermes) | 5 min | 15 min | Via warm standby (app1-bu) |
| app1-bu (standby) | N/A | <=15 min | Recovery path for Core |
| app1/2/3 (Standard tier) | 60 min | 24h | Rebuild from S3 + docker compose |
| wphost02 (WordPress) | 2h | 24h | DB restore from dump |
| fleettracker360 | 2h | 24h | DB restore from dump |
| Legacy Hetzner hosts | 4h | 24h | Via rescue mode or S3 |

---

## 6. Immediate Action Items

### P0 -- This Week

| # | Action | Details |
|---|---|---|
| 1 | Clean AI server disk | 92% full -- purge old Docker images/logs |
| 2 | Create S3 write-only IAM user | PutObject only, no delete/list |
| 3 | Enable S3 Object Lock | 7-day minimum retention |
| 4 | Deploy Docker volume backup | All Docker hosts |
| 5 | Deploy DB dump cron | wphost02 + fleettracker360 (MariaDB) |
| 6 | Deploy 30s watchdog | Currently documented, need deployment |

### P1 -- Next 2 Weeks

| # | Action |
|---|---|
| 7 | Document Tony's Hermes config + backup to S3 |
| 8 | Provision secondary S3 bucket (Wasabi us-west-2) |
| 9 | Extend snapshot retention from 48h to 30 days |
| 10 | Add backup success/failure monitoring to ops portal |

### P2 -- Next 30 Days

| # | Action |
|---|---|
| 11 | Full DR simulation for Core failover |
| 12 | Monthly restore testing schedule begins |
| 13 | Deploy Prometheus + Grafana stack |
| 14 | Add disk threshold alerts for all servers |
| 15 | Begin netcup migration |

---

## 7. Restore Runbooks (Summary)

Full per-server restore runbooks with exact commands are in the Per-Server Runbooks document (companion doc 3 of the v3 DR stack). Each covers:
- Purpose + dependencies
- DNS records, firewall ports, backup path
- Required secrets
- Step-by-step restore commands
- DB restore with pre-import dump verification
- Docker volume restore
- Health checks
- Known failure modes

Hosts covered: Core, app1-bu, app1/2/3, wphost02, fleettracker360, Legacy Hetzner hosts.

---

## 8. Third-Party Review Responses

### Reviewer 1 Findings -- Addressed

| Finding | Addressed In |
|---|---|
| Failover timing mismatch (5min check did not equal 2min RTO) | Section 4.1 -- Corrected to 30s checks |
| Backup compliance gap (standard says daily but not happening) | Section 3.1 -- Gap closure plan |
| AI server 92% disk critical | Section 6 -- P0 action item #1 |
| Memory consolidation data loss risk | Separate docs per priority-tagging proposal |
| Tony's Hermes undocumented | Section 6 -- P1 action item #7 |

### Reviewer 2 Findings -- Addressed

| Finding | Addressed In |
|---|---|
| S3 single point of failure | Section 1 + Section 3.1 -- Two-region plan |
| Backup frequency vs RPO mismatch | Section 5 -- RPO per data type |
| Restore procedures not detailed | Section 7 -- Full runbooks reference |
| Failback under-defined | Section 4.4 -- Protocol |
| No backup restore testing | Separate testing schedule doc |
| Retention too short (48h snapshots) | Section 6 -- P1 action item #9 |
| Backup security not described | Section 3.2 |
| Application backup details vague | Section 2.3 |
| Secrets recovery not documented | Section 1 (shared deps table) |
| Provider/account outage missing | Section 4.5 |

---

*This plan is ACTIVE. Review quarterly or after any infrastructure change.*