hermes-recovery/references/server-dr-plans-v3.md

# IT Pro Partner — Disaster Recovery Plan v3 (Consolidated)

**Author:** Sho'Nuff / Network Services Team, incorporating third-party DR review
**Date:** July 9, 2026
**Status:** LIVE — replaces v1.0
**Supersedes:** server-dr-plans.md v1.0, server-provisioning-standard-v1.md, hermes-dr-plan-v2.md

---

## Table of Contents
1. Architecture & Shared Dependencies
2. Server Standards
3. Backup Strategy
4. Failover / Failback
5. Monitoring & Alerting
6. Restore Runbooks
7. Testing & Validation
8. Security
9. Immediate Action Items
10. Appendix: Third-Party Review Responses

---

## 1. Architecture & Shared Dependencies

```
Internet ──► Cloudflare ──► Caddy ──► Service (HTTP/HTTPS)
                          │
              ┌───────────┼───────────┐
              │           │           │
         SSH/WG     Tailscale    Management
         (itpp-     (Core ↔      VPN (WireGuard
          infra)     app1-bu)     to home router)
              │           │           │
              └───────────┼───────────┘
                          │
              ┌───────────┴───────────┐
              │                       │
      S3 Primary                 S3 Secondary
   (Wasabi us-east-1)        (Wasabi us-west-2
        versioning ON          or Backblaze B2)
        object lock ON        object lock ON
```

### Shared Dependencies — Documented

| Dependency | Primary | Backup / Fallback | Credentials Location |
|---|---|---|---|
| SSH access | itpp-infra key (Hetzner vault) | Rescue mode / console | ~/.hermes/.env + Hudu |
| DNS | Cloudflare (API token in .env) | Manual via web UI | ~/.hermes/.env |
| S3 backup (primary) | Wasabi us-east-1 | Wasabi us-west-2 (to be created) | ~/.aws/credentials |
| S3 backup (secondary) | Wasabi us-west-2 | Backblaze B2 (evaluate) | TBD |
| SMTP relay | mail.germainebrown.com:2525 | Direct MXroute (port 465/587) | ~/.hermes/.env |
| Auth/SSO | Cloudflare Access | Local portal auth fallback | Cloudflare dashboard |
| Monitoring | Prometheus node_exporter | Uptime Kuma (docker box) | N/A (pull model) |
| VPN management | WireGuard (wg0, port 51821) | Tailscale direct tunnel | /etc/wireguard/ |

---

## 2. Server Standards

### 2.1 Tier Definitions

| Tier | Spec | OS | Use | Cost |
|---|---|---|---|---|
| **Standard** | RS 4000 G12 (12C/32G/1TB) | Debian 13 | app1, app2, app3 | ~$44/mo |
| **Light** | RS 2000 G12 (8C/16G/512GB) | Debian 13 | Core | ~$24/mo |
| **Standby** | CPX21 (4C/8G/80GB) | Debian 13 | app1-bu | ~$14/mo |
| **Legacy** | Variable Hetzner | Debian 12 | Existing (migrate on rebuild) | ~$8-44/mo |

### 2.2 Base Install — Verified Checklist

```bash
# S1: hostname + timezone
hostnamectl set-hostname <name>
timedatectl set-timezone America/New_York

# S2: system update
apt update && apt upgrade -y

# S3: security baseline
apt install -y fail2ban ufw
ufw default deny incoming; ufw default allow outgoing
ufw allow ssh; ufw --force enable

# S4: monitoring
apt install -y prometheus-node-exporter

# S5: standard user + key
adduser ippadmin && usermod -aG sudo ippadmin
echo "ssh-ed25519 AAA... itpp-infra" >> /home/ippadmin/.ssh/authorized_keys

# S6: harden SSH
sed -i 's/^#PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config
sed -i 's/^PermitRootLogin yes/PermitRootLogin prohibit-password/' /etc/ssh/sshd_config
systemctl restart sshd

# S7: Docker (if needed)
curl -fsSL https://get.docker.com | bash
apt install -y docker-compose-plugin
```

### 2.3 Application Data Paths — Per Server

Every server must document:
- What data lives where (volumes, bind mounts, databases, uploads, .env files)
- What can be rebuilt vs what must be restored
- Secrets location
- Caddy/nginx configs
- Cron jobs
- External API dependencies

(See Section 6 — Restore Runbooks for per-server details)

---

## 3. Backup Strategy

### 3.1 Backup Matrix

| Data Type | Frequency | Target | Retention | Status |
|---|---|---|---|---|
| Hermes state | Every 15 min | S3 primary (live/) | 48 hours | ✅ LIVE |
| Hermes full | Daily 5 AM | S3 primary (full/) | 90 days | ✅ LIVE |
| Memory snapshots | Every 10 min | S3 primary (memories/) | 48 hours | ✅ LIVE (new) |
| Memory history | Every 10 min | S3 primary (memories/history/) | 90 days | ✅ LIVE (new) |
| Router configs | Daily 6 AM | S3 primary (mikrotik/) | 90 days | ✅ LIVE |
| System configs | Daily | S3 primary (<server>/) | 90 days | ⚠️ NOT AUTOMATED |
| Docker volumes | Daily | S3 primary (<server>/) | 90 days | ❌ NOT DONE |
| Database dumps | Every 6 hours | S3 primary (<server>/) | 90 days | ❌ NOT DONE |
| Application data | Per-app | S3 primary (<server>/) | 90 days | ⚠️ PARTIAL |
| S3 secondary | Daily sync | Wasabi us-west-2 | 90 days | ❌ NOT SET UP |

### 3.2 Immediate Backup Gaps to Close

| Priority | Gap | Fix |
|---|---|---|
| 🔴 P0 | Docker volumes not backed up | Deploy volume backup script per Docker host |
| 🔴 P0 | Database dumps not automated | Cron job per MariaDB/Postgres instance |
| 🟡 P1 | No secondary S3 target | Provision Wasabi us-west-2 bucket + replication |
| 🟡 P1 | App data recovery undocumented | Per-app data path inventory (Section 2.3) |
| 🟢 P2 | 48h snapshot retention too short | Extend to 30 days via S3 lifecycle policy |
| 🟢 P2 | System config backup not automated | Add `/etc/` snapshot to daily backup cron |

### 3.3 Backup Security

- **Versioning:** ON (all 3 buckets) ✅
- **Object Lock:** NOT enabled — needs implementation
- **Delete protection:** NOT configured
- **Backup credentials:** Write-only IAM user — NOT implemented (using admin credentials)
- **Encryption at rest:** Wasabi server-side (default) — no client-side encryption
- **Credentials escrow:** Partial (in .env + Hudu)

**Fixes needed:**
1. Create IAM sub-user with `s3:PutObject` only (no delete, no list)
2. Enable Object Lock with 7-day minimum retention
3. Store write-only credentials separately from admin credentials
4. Client-side encrypt backup before upload (age/gpg) for the secondary copy

---

## 4. Failover / Failback

### 4.1 Warm Standby — Corrected Timing

The DR reviewer flagged this correctly: you cannot detect a 2-minute outage with a 5-minute check interval.

**Old:** Check every 5 min, failover after 2 min unreachable  
**New:** Check every 30 seconds, failover after 4 failed checks (~2 min)

```python
# Pseudocode for corrected watchdog
CHECK_INTERVAL = 30       # seconds
FAILURE_THRESHOLD = 4      # consecutive failures
CONFIRMATION_WINDOW = 120  # seconds (2 min)
# Detection time: 4 * 30s = 120s = 2 minutes
# Sync + start time: ~45-90 seconds
# Effective RTO: ~3-3.5 minutes
# Stated RTO target: 5 minutes (ACHIEVABLE)
```

### 4.2 Health Check — Two Layers

**Layer 1 (Ping):** ICMP to Core (152.53.192.33) every 30s  
**Layer 2 (HTTP):** `GET /healthz` on Tailscale IP — checks:
- Gateway heartbeat ≤ 5s old
- SQLite `PRAGMA quick_check` returns "ok"
- Local Ollama responding

**Decision Matrix:**

| Server | Hermes | Action |
|---|---|---|
| ✅ UP | ✅ UP | Stay dormant |
| ❌ DOWN | N/A | **Auto-takeover** — unambiguous |
| ✅ UP | ❌ DOWN | **ALERT + fence** — attempt SSH fence, only takeover if Core Hermes confirmed stopped |

### 4.3 Fencing Protocol

```bash
# SSH Fence (preferred — server is up)
ssh ippadmin@152.53.192.33 \
  "sudo systemctl stop hermes && sudo systemctl mask hermes"
# Verify: returns "inactive"
ssh ippadmin@152.53.192.33 "sudo systemctl is-active hermes"

# S3 Ownership Lock
# Object: s3://hermes-vps-backups/ACTIVE_OWNER
# Core refreshes every 60s while healthy
# Standby writes itself as owner on takeover
# Core reads on startup — if standby owns it, Core stays dormant
```

### 4.4 Failback Protocol

**⚠️ Auto-failback is FORBIDDEN.** Human clearance required.

```bash
# FAILBACK STEPS (manual):
# 1. Confirm Core is healthy: SSH + systemctl is-active hermes
# 2. Stop standby gateway: ssh app1-bu "hermes gateway stop"
# 3. Sync state from standby back to Core (if needed):
#    Rsync /root/.hermes/ from standby to Core
# 4. Unmask Hermes on Core:
#    ssh Core "sudo systemctl unmask hermes && sudo systemctl start hermes"
# 5. Verify Core gateway is active (Telegram responds)
# 6. Standby returns to dormant state
# 7. Update S3 ownership lock to "core"
```

### 4.5 Provider/Account Outage Scenario

If the entire hosting provider (netcup) is unavailable:
1. app1-bu (Hetzner) takes over as active Hermes host
2. DNS records updated to point to app1-bu IP (5.161.114.8)
3. Services migrate to netcup replacement when available
4. No auto-failback — human decision

---

## 5. Monitoring & Alerting

### 5.1 Alert Thresholds

| Alert | Threshold | Action | Priority |
|---|---|---|---|
| Disk usage | > 80% | Notify, investigate | 🟡 P2 |
| Disk usage | > 90% | Notify, immediate cleanup | 🔴 P0 |
| Backup age | > 24h since last success | Notify | 🟡 P1 |
| Backup failure | Any failed job | Notify | 🔴 P0 |
| Hermes heartbeat | No heartbeat > 30s | Notify | 🔴 P0 |
| Hermes down + server up | Layer 2 fail, Layer 1 pass | IMMEDIATE alert | 🔴 P0 |
| TLS expiration | < 14 days | Notify | 🟡 P1 |
| TLS expiration | < 7 days | Notify + remind | 🔴 P0 |
| SSH fail spikes | > 10 failures/min | Investigate | 🟡 P1 |
| Service down | Caddy/nginx/Docker inactive | Notify, attempt restart | 🔴 P0 |
| S3 sync failure | Upload fails | Abort (memory consolidate) | 🔴 P0 |
| Standby stale | No sync > 30 min | Notify | 🟡 P1 |

### 5.2 Monitoring Stack

- **Per-server:** `prometheus-node-exporter` (all boxes)
- **Dashboard:** Prometheus → Grafana (planned — on app1 or dedicated)
- **Synthetic checks:** Uptime Kuma (on docker box, 178.156.168.35)
- **Hermes watchdog:** `service-health-check.sh` (every 5 min, Core)
- **Standby watcher:** `home-router-watchdog.sh` (every 5 min, Core)
- **Apex mail:** `apex-mail-watchdog.sh` (every 5 min, Core)
- **Alert delivery:** Telegram (via Hermes)

### 5.3 Alerting Gaps to Close

| Gap | Fix |
|---|---|
| No Prometheus/Grafana stack | Deploy on app1 after migration |
| No backup success/failure dashboard | Add to ops portal |
| No disk threshold alerts for all servers | Add to ops-data-collector.py |
| No TLS expiry monitoring | Add cert check to service-health-check.sh |
| No centralized alert routing | Use Hermes for all alerts |

---

## 6. Restore Runbooks (Per Server)

### 6.1 Core (netcup RS 2000) — 🔴 Critical

| Attribute | Detail |
|---|---|
| RTO target | 5 minutes (via failover to app1-bu) |
| RPO target | 15 minutes (Hermes state), 24h (Docker data) |
| Actual RTO | ~3 minutes (corrected watchdog) |
| Actual RPO | Varies by data type (see below) |

**Restore steps:**
```bash
# Option A: Failover to standby (fastest — < 5 min)
# app1-bu detects Core down → auto-takeover (Section 4.1)

# Option B: Full restore on replacement box
# Step 1 — Provision RS 2000 G12 on netcup
# Step 2 — Apply base standard (Section 2.2)
# Step 3 — Install Hermes
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
# Step 4 — Restore Hermes state from latest S3:
aws s3 cp s3://hermes-vps-backups/live/state.db /root/.hermes/state.db \
  --endpoint-url https://s3.us-east-1.wasabisys.com
aws s3 cp s3://hermes-vps-backups/live/config.yaml /root/.hermes/config.yaml \
  --endpoint-url https://s3.us-east-1.wasabisys.com
# Step 5 — Restore .env from backup:
aws s3 cp s3://hermes-vps-backups/live/.env /root/.hermes/.env \
  --endpoint-url https://s3.us-east-1.wasabisys.com
chmod 600 /root/.hermes/.env
# Step 6 — Restore Docker stacks:
for dir in /root/docker/*/; do
  (cd "$dir" && docker compose up -d)
done
# Step 7 — Restore Caddy config:
aws s3 cp s3://hermes-vps-backups/live/config/Caddyfile /etc/caddy/Caddyfile
systemctl restart caddy
# Step 8 — Start Hermes gateway:
hermes gateway run --replace
# Step 9 — Verify:
curl -s https://ops.itpropartner.com/api/health
curl -s https://sign.itpropartner.com
# Check Telegram responds to /ping
```

### 6.2 ai.itpropartner.com (Hetzner CPX41) — 🔴 Critical

| Attribute | Detail |
|---|---|
| RTO target | 2 hours |
| RPO target | 24 hours |
| Current disk | **🔴 92%** — must clean before migration |

**⚠️ Pre-reboot checklist:**
```bash
# Verify at least one alternative model provider works
curl https://admin-ai.itpropartner.com/v1/chat/completions \
  -H "Authorization: Bearer $(cat ~/.hermes/.env | grep ADMIN_AI | cut -d= -f2)" \
  -d '{"model":"deepseek-chat","messages":[{"role":"user","content":"ping"}]}'
# Ensure local Ollama on Core is running
curl -s http://127.0.0.1:11434/api/tags || systemctl start ollama
# Notify all Hermes users (Germaine, Anita, Tony)
```

**Restore steps:**
```bash
# Step 1 — Provision replacement (CPX41 or netcup equivalent)
# Step 2 — Apply base standard (Section 2.2)
# Step 3 — Deploy Docker stacks:
cd /opt/litellm && docker compose up -d
cd /opt/openwebui && docker compose up -d
cd /opt/qdrant && docker compose up -d
# Step 4 — Pull models:
ollama pull llama3.2:3b
# Step 5 — Configure DNS: admin-ai.itpropartner.com → new IP
# Step 6 — Update all Hermes config.yaml files with new base_url
# Step 7 — Verify model access for all profiles
```

### 6.3 wphost02 (Hetzner CPX21, RunCloud) — 🟡 Important

| Attribute | Detail |
|---|---|
| RTO target | 4 hours |
| RPO target | 24 hours |
| Backup method | RunCloud daily backup + manual DB |

**Restore steps:**
```bash
# Step 1 — Provision CPX21 on Hetzner
# Step 2 — Connect to RunCloud panel, link server
# Step 3 — RunCloud provisions nginx, MariaDB, PHP
# Step 4 — Restore WordPress from RunCloud backup → verify apex site loads
# Step 5 — Deploy itpp-infra SSH key in RunCloud → SSH Access
# Step 6 — Whitelist Core IP (152.53.192.33) in RunCloud firewall
# Step 7 — Reconfigure Postfix relay:
postconf -e "relayhost = mail.germainebrown.com:2525"
postconf -e "smtp_sasl_auth_enable = yes"
echo "mail.germainebrown.com:2525 email@redacted:Catches.bullets1985" > /etc/postfix/sasl_passwd
postmap /etc/postfix/sasl_passwd && systemctl restart postfix
# Step 8 — Flush postfix queue:
postqueue -f
```

### 6.4 UNMS (Hetzner CPX21) — 🟡 Important

| Attribute | Detail |
|---|---|
| RTO target | 4 hours |
| RPO target | 24 hours |
| Containers | 9 (Docker compose stack) |

**Restore steps:**
```bash
# Step 1 — Provision CPX21 → Docker
# Step 2 — Restore Postgres data volumes from S3
# Step 3 — Deploy docker-compose.yml
cd /opt/unms && docker compose up -d
# Step 4 — Verify UCRM + UNMS web login
# Step 5 — Re-register tower routers via UNMS
```

### 6.5 Hudu (Hetzner CPX21) — 🟡 Important

| Attribute | Detail |
|---|---|
| RTO target | 4 hours |
| RPO target | 24 hours |
| Containers | 5 (Docker compose) |
| Auto-backup | Hudu built-in S3 backup |

**Restore steps:**
```bash
# Step 1 — Provision CPX21 → Docker
# Step 2 — Deploy Hudu Docker stack
# Step 3 — Restore Postgres from last backup
# Step 4 — Verify web login + recent documents accessible
```

### 6.6 UniFi (Hetzner CPX21) — 🟡 Important

| Attribute | Detail |
|---|---|
| RTO target | 4 hours |
| RPO target | 24 hours |
| Type | Native install (not Docker) |

**Restore steps:**
```bash
# Step 1 — Provision CPX21
# Step 2 — Install UniFi Controller (native)
# Step 3 — Restore from UniFi backup file
# Step 4 — Adopt APs and switches
```

### 6.7 n8n (Hetzner CPX11) — 🟢 Standard

| Attribute | Detail |
|---|---|
| RTO target | 8 hours |
| Containers | 2 (n8n + postgres) |

**Restore:**
```bash
# Provision → Docker → restore postgres volume
cd /opt/n8n && docker compose up -d
```

### 6.8 docker box (Hetzner CPX11) — 🟢 Standard

| Attribute | Detail |
|---|---|
| RTO target | 8 hours |
| Containers | 8 (NPM, RustDesk, Uptime Kuma, RunCloud) |

**Restore:**
```bash
# Provision → Docker → restore compose stacks + volumes
```

### 6.9 fleettracker360 (Hetzner CPX11) — 🟢 Standard

| Attribute | Detail |
|---|---|
| RTO target | 8 hours |
| Type | Apache + MariaDB (LAMP) |

**Restore:**
```bash
# Provision → Apache + MariaDB → restore web root + DB
```

### 6.10 tony-vps (Hetzner CPX21) — 🟢 Standard

| Attribute | Detail |
|---|---|
| RTO target | Coordinate with Tony |
| Type | Independent Hermes + local LLM |

**⚠️ SHADOW IT — needs documentation.**

**Current known config:**
- Hermes gateway with admin-ai model access
- Llama 3.2 1B local fallback (systemd: llama-server)
- Cron: hermes-heartbeat.sh every minute
- Minimal services, no Docker

**Actions needed:**
1. Document Tony's full Hermes config.yaml and .env
2. Backup to S3 under s3://hermes-vps-backups/tony-vps/
3. Add Tony's config to DR plan once documented

### 6.11 app1-bu (Hetzner CPX21) — 🔴 Standby

| Attribute | Detail |
|---|---|
| RTO target | 2 minutes |
| RPO target | 15 minutes |
| Type | Warm standby — gateway dormant |

**Activation:**
```bash
# Automatic: watchdog detects Core down → S3 sync → start gateway
# Manual:
ssh ippadmin@5.161.114.8
sudo -i
hermes gateway run --replace
```

---

## 7. Testing & Validation

### 7.1 Restore Testing Schedule

| Test | Frequency | Who |
|---|---|---|
| Verify backup job completion (all servers) | Daily | Automated alert |
| Check backup age per server | Daily | Automated |
| Restore one database dump to test environment | Monthly | Network Team |
| Restore one full application stack | Quarterly | Network Team |
| Full DR simulation — Core failover to app1-bu | Quarterly | Network Team |
| S3 credential restore/access test | Quarterly | Network Team |
| Document restore test results (date, tester, duration, findings) | Per test | Network Team |

### 7.2 Validation Criteria

Each restore test must produce:
- Date + tester name
- Server/application tested
- Backup artifact used (path + timestamp)
- Restore duration (start → fully operational)
- Data validation: check a known record exists (specific row, file, or document)
- Issues encountered
- Remediation ticket (if needed)

### 7.3 Quarterly DR Simulation

Scenario: "Core is completely gone. Netcup account inaccessible. app1-bu has been active for 4 hours. Now netcup is back — fail back."

Steps to validate:
1. Simulate netcup provider outage (block Core IP)
2. Verify app1-bu takes over within 5 minutes
3. Send a Telegram message through standby
4. Verify all cron jobs run on standby
5. Simulate netcup recovery
6. Execute manual failback per Section 4.4
7. Verify no data loss, no split-brain

---

## 8. Security

### 8.1 Current Security Posture

| Control | Status | Notes |
|---|---|---|
| SSH keys only | ✅ Done | Password auth disabled |
| Root SSH disabled | ⚠️ Partial | Some servers still allow root |
| UFW host firewall | ✅ All | Standard on all boxes |
| fail2ban | ✅ All | Standard on all boxes |
| S3 versioning | ✅ All buckets | Enabled |
| S3 Object Lock | ❌ Not done | Needs implementation |
| S3 write-only IAM | ❌ Not done | Currently using admin creds |
| Backup encryption | ❌ Not done | Server-side only; no client-side |
| Secrets escrow | 🟡 Partial | .env + Hudu — not all secrets documented |
| TLS everywhere | ✅ All public endpoints | Caddy auto-certs |

### 8.2 Security Gaps to Close

| Priority | Gap | Fix |
|---|---|---|
| 🔴 P0 | S3 uses admin credentials (can delete) | Create IAM sub-user with s3:PutObject only |
| 🔴 P0 | No Object Lock on backups | Enable 7-day minimum retention |
| 🟡 P1 | No client-side backup encryption | Add age/gpg before upload for secondary copy |
| 🟡 P1 | Secrets not fully documented | Audit .env, create secrets inventory in Hudu |
| 🟡 P1 | Some servers allow root SSH | Verify and disable across all boxes |
| 🟢 P2 | No vulnerability scanning | Add lynis weekly scan (already scheduled) |

### 8.3 Break-Glass Access

If itpp-infra key is lost or revoked:
1. **Hetzner servers:** Use Hetzner console → Rescue mode → mount filesystem → redeploy key
2. **netcup servers:** Use SCP web console → VNC → log in with root password → redeploy key
3. **Root passwords:** Stored in ~/.hermes/.env (chmod 600) and Hudu

---

## 9. Immediate Action Items

### 🔴 P0 — This Week

| # | Action | Owner | Details |
|---|---|---|---|
| 1 | Clean up AI server disk | Sho'Nuff | 92% full — purge old Docker images, logs, temp data |
| 2 | Create S3 write-only IAM user | Sho'Nuff | Separate creds from admin — no delete, no list |
| 3 | Enable S3 Object Lock | Sho'Nuff | 7-day minimum on all backup buckets |
| 4 | Deploy volume backup script | Sho'Nuff | All Docker hosts → S3 per-server buckets |
| 5 | Deploy database dump cron | Sho'Nuff | MariaDB on wphost02 + fleettracker360, every 6h |

### 🟡 P1 — Next 2 Weeks

| # | Action | Owner |
|---|---|---|
| 6 | Fix warm standby watchdog interval (30s not 5min) | Sho'Nuff |
| 7 | Document Tony's Hermes config + backup to S3 | Coordinate with Tony |
| 8 | Provision secondary S3 bucket (Wasabi us-west-2) | Sho'Nuff |
| 9 | Extend snapshot retention from 48h to 30 days | Sho'Nuff |
| 10 | Add backup success/failure monitoring to ops portal | Sho'Nuff |
| 11 | Implement priority tagging (Option A) for memory consolidation | Sho'Nuff |

### 🟢 P2 — Next 30 Days

| # | Action | Owner |
|---|---|---|
| 12 | Full DR simulation for Core failover | Network Team |
| 13 | Monthly restore testing schedule begins | Network Team |
| 14 | Deploy Prometheus + Grafana stack | Sho'Nuff |
| 15 | Add disk threshold alerts for all servers | Sho'Nuff |
| 16 | Begin netcup migration | Sho'Nuff |

---

## 10. Appendix: Third-Party Review Responses

### 10.1 Reviewer 1 Findings — Addressed

| Finding | Addressed In |
|---|---|
| Failover timing mismatch (5min check ≠ 2min RTO) | Section 4.1 — Corrected to 30s check |
| Backup compliance gap (standard says daily, not happening) | Section 3.2 — Gap closure plan |
| AI server 92% disk critical | Section 9 — P0 action item #1 |
| Memory consolidation data loss risk | Consolidation proposal (separate doc) |
| Tony's Hermes undocumented | Section 6.10 + P1 action item #7 |

### 10.2 Reviewer 2 Findings — Addressed

| Finding | Addressed In |
|---|---|
| S3 single point of failure | Section 3.3 — Two-region plan |
| Backup frequency vs RPO mismatch | Section 3.1 — Matrix with status |
| Restore procedures not detailed | Section 6 — Full runbooks per server |
| Failback under-defined | Section 4.4 — Protocol with no-auto-failback rule |
| No backup restore testing | Section 7 — Schedule |
| 48h retention too short | P1 item #9 — Extend to 30 days |
| Backup security not described | Section 8.2 |
| Monitoring not fully defined | Section 5 |
| Application backup details vague | Section 2.3 + Section 6 |
| Secrets recovery not documented | Section 2.2 (base install creds) + Section 8.3 |
| Provider/account outage missing | Section 4.5 |

---

*This plan supersedes all previous DR documentation. Review quarterly or after any infrastructure change.*