Initial DR plan — v3 with per-server RTO/RPO, recovery manual, issue log

dr-issue-log.md

@@ -0,0 +1,82 @@
+# DR Issue Log — Hermes Infrastructure
+
+Tracked issues found during DR audits. Each entry: date, problem, root cause, fix applied, verification.
+
+---
+
+## 2026-07-08 — Initial Full DR Audit
+
+### DR-001: Caddyfile not included in backup scripts
+**Problem:** /etc/caddy/Caddyfile was not copied by hermes-backup.sh or hermes-live-sync.sh. If Core server fails, the full Caddy reverse proxy config would need to be rebuilt from scratch.
+**Root cause:** Backup scripts were written to cover hermes config and user directories but omitted system-level config files.
+**Fix:** Added /etc/caddy/Caddyfile to hermes-backup.sh and hermes-live-sync.sh (with DR FIX: comments and dates).
+**Status:** ✅ Fixed & verified 2026-07-08
+**Verified by:** bash -n syntax check + subagent confirmed all paths in script
+
+### DR-002: Systemd service files backed up to wrong path
+**Problem:** hermes-backup.sh collected systemd services from ~/.config/systemd/user/ instead of /etc/systemd/system/. The real service files (/etc/systemd/system/hermes-agent.service, shark-game.service, etc.) were not backed up.
+**Root cause:** Backup script path pointed to user-level systemd vs system-level.
+**Fix:** Corrected path in hermes-backup.sh to /etc/systemd/system/*.service. Also updated the embedded restore.sh heredoc.
+**Status:** ✅ Fixed & verified 2026-07-08
+**Verified by:** bash -n syntax check + subagent confirmed paths in script
+
+### DR-003: migration-creds.txt not in backup scope
+**Problem:** /root/.hermes/migration-creds.txt existed but wasn't referenced by any backup script.
+**Root cause:** Added after backup script was written, never included in scope.
+**Fix:** Added to hermes-backup.sh file list with chmod 600 restore.
+**Status:** ✅ Fixed & verified 2026-07-08
+**Verified by:** bash -n syntax check + subagent confirmed file referenced
+
+### DR-004: dre-temp-passwords.txt exposed at 644 permissions
+**Problem:** /root/.hermes/references/dre-temp-passwords.txt was readable by all users (644) instead of owner-only (600).
+**Root cause:** Script created the file without explicit permission setting.
+**Fix:** chmod 600
+**Status:** ✅ Fixed 2026-07-08
+**Verified by:** ls -la confirms 600
+
+### DR-005: migration-creds.txt at 644 permissions
+**Problem:** Same issue as DR-004 — migration-creds.txt was 644.
+**Root cause:** Written without explicit permission setting.
+**Fix:** chmod 600
+**Status:** ✅ Fixed 2026-07-08
+**Verified by:** ls -la confirms 600
+
+### DR-006: Full backup had no cron job (last ran Jul 5)
+**Problem:** hermes-full-backup.tar.gz last uploaded to S3 on Jul 5. The daily 5AM cron had no scheduled job — the backup script existed but nothing was calling it.
+**Root cause:** Cron job was never created for the full backup script. The hermes-live-sync covered configs every 15 min but full archive was orphaned.
+**Fix:** Created cron job `hermes-full-backup` at `0 5 * * *` calling `run-hermes-backup.sh` wrapper.
+**Status:** ✅ Fixed & verified 2026-07-08
+**Verified by:** Cron job created and scheduled for next 5 AM ET
+
+### DR-007: home-router-daily-backup cron error
+**Problem:** Cron job errored at 06:00 today. SSH export on router produced a stuck .in_progress file that never completed.
+**Root cause:** Cron was using home-router-backup.sh (old WireGuard tunnel script) instead of the proper run-wisp-backup.sh pipeline. Stuck export files blocked SCP.
+**Fix:** Changed cron to use run-wisp-backup.sh. Cleaned stuck .in_progress files from router. Missing packages (paramiko, xl2tpd, strongSwan) installed.
+**Status:** ✅ Fixed & verified 2026-07-08
+**Verified by:** Backup ran end-to-end, config uploaded to S3
+
+### DR-008: MikroTik CCR backup stale (last Jul 5)
+**Problem:** mikrotik-ccr-backups S3 bucket had no uploads since Jul 5.
+**Root cause (two causes):** (1) wisp-backup.py failed because paramiko wasn't installed. (2) tower IP in config.yaml was 192.168.88.1 (LAN) but SSH is restricted to WireGuard tunnel network 10.77.0.0/24.
+**Fix:** Installed paramiko v5.0.0. Updated tower IP to 10.77.0.2 in wisp-backup/config.yaml. Installed missing VPN stack (xl2tpd, strongSwan).
+**Status:** ✅ Fixed & verified 2026-07-08
+**Verified by:** Backup ran end-to-end, config uploaded to S3
+
+### DR-009: app1-bu warm vs cold docs mismatch
+**Problem:** DR plan doc said "offline, boots on demand" but server is running (3 days uptime).
+**Root cause:** DR plan doc was outdated — actual design is warm standby (always on, Hermes dormant).
+**Fix:** Updated DR plan to reflect warm standby design. Server stays running.
+**Status:** ✅ Resolved — not a bug, docs were wrong
+
+### DR-010: Failover timing change (per Germaine's direction)
+**Change:** Check interval from every 10 min → every 5 min. Constant check window from 3 min → 2 min (30s × 4 cycles).
+**Rationale:** Faster detection, shorter failover.
+**Status:** ✅ Fixed & verified 2026-07-08
+**Verified by:** Cron changed to */5. Watchdog timing updated on app1-bu.
+
+### DR-011: AWS CLI missing on app1-bu standby
+**Problem:** /opt/awscli-venv was missing on app1-bu. Failover could not pull fresh state from S3.
+**Root cause:** python3-venv package not installed on standby server.
+**Fix:** Installed python3-venv, created venv, installed awscli. Created hermes-standby-sync.sh script with */10 cron.
+**Status:** ✅ Fixed & verified 2026-07-08
+**Verified by:** First sync ran end-to-end. Config.yaml timestamp went from Jul 5 → Jul 8 17:10.