83 lines
5.3 KiB
Markdown
83 lines
5.3 KiB
Markdown
# DR Issue Log — Hermes Infrastructure
|
||
|
||
Tracked issues found during DR audits. Each entry: date, problem, root cause, fix applied, verification.
|
||
|
||
---
|
||
|
||
## 2026-07-08 — Initial Full DR Audit
|
||
|
||
### DR-001: Caddyfile not included in backup scripts
|
||
**Problem:** /etc/caddy/Caddyfile was not copied by hermes-backup.sh or hermes-live-sync.sh. If Core server fails, the full Caddy reverse proxy config would need to be rebuilt from scratch.
|
||
**Root cause:** Backup scripts were written to cover hermes config and user directories but omitted system-level config files.
|
||
**Fix:** Added /etc/caddy/Caddyfile to hermes-backup.sh and hermes-live-sync.sh (with DR FIX: comments and dates).
|
||
**Status:** ✅ Fixed & verified 2026-07-08
|
||
**Verified by:** bash -n syntax check + subagent confirmed all paths in script
|
||
|
||
### DR-002: Systemd service files backed up to wrong path
|
||
**Problem:** hermes-backup.sh collected systemd services from ~/.config/systemd/user/ instead of /etc/systemd/system/. The real service files (/etc/systemd/system/hermes-agent.service, shark-game.service, etc.) were not backed up.
|
||
**Root cause:** Backup script path pointed to user-level systemd vs system-level.
|
||
**Fix:** Corrected path in hermes-backup.sh to /etc/systemd/system/*.service. Also updated the embedded restore.sh heredoc.
|
||
**Status:** ✅ Fixed & verified 2026-07-08
|
||
**Verified by:** bash -n syntax check + subagent confirmed paths in script
|
||
|
||
### DR-003: migration-creds.txt not in backup scope
|
||
**Problem:** /root/.hermes/migration-creds.txt existed but wasn't referenced by any backup script.
|
||
**Root cause:** Added after backup script was written, never included in scope.
|
||
**Fix:** Added to hermes-backup.sh file list with chmod 600 restore.
|
||
**Status:** ✅ Fixed & verified 2026-07-08
|
||
**Verified by:** bash -n syntax check + subagent confirmed file referenced
|
||
|
||
### DR-004: dre-temp-passwords.txt exposed at 644 permissions
|
||
**Problem:** /root/.hermes/references/dre-temp-passwords.txt was readable by all users (644) instead of owner-only (600).
|
||
**Root cause:** Script created the file without explicit permission setting.
|
||
**Fix:** chmod 600
|
||
**Status:** ✅ Fixed 2026-07-08
|
||
**Verified by:** ls -la confirms 600
|
||
|
||
### DR-005: migration-creds.txt at 644 permissions
|
||
**Problem:** Same issue as DR-004 — migration-creds.txt was 644.
|
||
**Root cause:** Written without explicit permission setting.
|
||
**Fix:** chmod 600
|
||
**Status:** ✅ Fixed 2026-07-08
|
||
**Verified by:** ls -la confirms 600
|
||
|
||
### DR-006: Full backup had no cron job (last ran Jul 5)
|
||
**Problem:** hermes-full-backup.tar.gz last uploaded to S3 on Jul 5. The daily 5AM cron had no scheduled job — the backup script existed but nothing was calling it.
|
||
**Root cause:** Cron job was never created for the full backup script. The hermes-live-sync covered configs every 15 min but full archive was orphaned.
|
||
**Fix:** Created cron job `hermes-full-backup` at `0 5 * * *` calling `run-hermes-backup.sh` wrapper.
|
||
**Status:** ✅ Fixed & verified 2026-07-08
|
||
**Verified by:** Cron job created and scheduled for next 5 AM ET
|
||
|
||
### DR-007: home-router-daily-backup cron error
|
||
**Problem:** Cron job errored at 06:00 today. SSH export on router produced a stuck .in_progress file that never completed.
|
||
**Root cause:** Cron was using home-router-backup.sh (old WireGuard tunnel script) instead of the proper run-wisp-backup.sh pipeline. Stuck export files blocked SCP.
|
||
**Fix:** Changed cron to use run-wisp-backup.sh. Cleaned stuck .in_progress files from router. Missing packages (paramiko, xl2tpd, strongSwan) installed.
|
||
**Status:** ✅ Fixed & verified 2026-07-08
|
||
**Verified by:** Backup ran end-to-end, config uploaded to S3
|
||
|
||
### DR-008: MikroTik CCR backup stale (last Jul 5)
|
||
**Problem:** mikrotik-ccr-backups S3 bucket had no uploads since Jul 5.
|
||
**Root cause (two causes):** (1) wisp-backup.py failed because paramiko wasn't installed. (2) tower IP in config.yaml was 192.168.88.1 (LAN) but SSH is restricted to WireGuard tunnel network 10.77.0.0/24.
|
||
**Fix:** Installed paramiko v5.0.0. Updated tower IP to 10.77.0.2 in wisp-backup/config.yaml. Installed missing VPN stack (xl2tpd, strongSwan).
|
||
**Status:** ✅ Fixed & verified 2026-07-08
|
||
**Verified by:** Backup ran end-to-end, config uploaded to S3
|
||
|
||
### DR-009: app1-bu warm vs cold docs mismatch
|
||
**Problem:** DR plan doc said "offline, boots on demand" but server is running (3 days uptime).
|
||
**Root cause:** DR plan doc was outdated — actual design is warm standby (always on, Hermes dormant).
|
||
**Fix:** Updated DR plan to reflect warm standby design. Server stays running.
|
||
**Status:** ✅ Resolved — not a bug, docs were wrong
|
||
|
||
### DR-010: Failover timing change (per Germaine's direction)
|
||
**Change:** Check interval from every 10 min → every 5 min. Constant check window from 3 min → 2 min (30s × 4 cycles).
|
||
**Rationale:** Faster detection, shorter failover.
|
||
**Status:** ✅ Fixed & verified 2026-07-08
|
||
**Verified by:** Cron changed to */5. Watchdog timing updated on app1-bu.
|
||
|
||
### DR-011: AWS CLI missing on app1-bu standby
|
||
**Problem:** /opt/awscli-venv was missing on app1-bu. Failover could not pull fresh state from S3.
|
||
**Root cause:** python3-venv package not installed on standby server.
|
||
**Fix:** Installed python3-venv, created venv, installed awscli. Created hermes-standby-sync.sh script with */10 cron.
|
||
**Status:** ✅ Fixed & verified 2026-07-08
|
||
**Verified by:** First sync ran end-to-end. Config.yaml timestamp went from Jul 5 → Jul 8 17:10.
|