5.3 KiB
DR Issue Log — Hermes Infrastructure
Tracked issues found during DR audits. Each entry: date, problem, root cause, fix applied, verification.
2026-07-08 — Initial Full DR Audit
DR-001: Caddyfile not included in backup scripts
Problem: /etc/caddy/Caddyfile was not copied by hermes-backup.sh or hermes-live-sync.sh. If Core server fails, the full Caddy reverse proxy config would need to be rebuilt from scratch. Root cause: Backup scripts were written to cover hermes config and user directories but omitted system-level config files. Fix: Added /etc/caddy/Caddyfile to hermes-backup.sh and hermes-live-sync.sh (with DR FIX: comments and dates). Status: ✅ Fixed & verified 2026-07-08 Verified by: bash -n syntax check + subagent confirmed all paths in script
DR-002: Systemd service files backed up to wrong path
Problem: hermes-backup.sh collected systemd services from ~/.config/systemd/user/ instead of /etc/systemd/system/. The real service files (/etc/systemd/system/hermes-agent.service, shark-game.service, etc.) were not backed up. Root cause: Backup script path pointed to user-level systemd vs system-level. Fix: Corrected path in hermes-backup.sh to /etc/systemd/system/*.service. Also updated the embedded restore.sh heredoc. Status: ✅ Fixed & verified 2026-07-08 Verified by: bash -n syntax check + subagent confirmed paths in script
DR-003: migration-creds.txt not in backup scope
Problem: /root/.hermes/migration-creds.txt existed but wasn't referenced by any backup script. Root cause: Added after backup script was written, never included in scope. Fix: Added to hermes-backup.sh file list with chmod 600 restore. Status: ✅ Fixed & verified 2026-07-08 Verified by: bash -n syntax check + subagent confirmed file referenced
DR-004: dre-temp-passwords.txt exposed at 644 permissions
Problem: /root/.hermes/references/dre-temp-passwords.txt was readable by all users (644) instead of owner-only (600). Root cause: Script created the file without explicit permission setting. Fix: chmod 600 Status: ✅ Fixed 2026-07-08 Verified by: ls -la confirms 600
DR-005: migration-creds.txt at 644 permissions
Problem: Same issue as DR-004 — migration-creds.txt was 644. Root cause: Written without explicit permission setting. Fix: chmod 600 Status: ✅ Fixed 2026-07-08 Verified by: ls -la confirms 600
DR-006: Full backup had no cron job (last ran Jul 5)
Problem: hermes-full-backup.tar.gz last uploaded to S3 on Jul 5. The daily 5AM cron had no scheduled job — the backup script existed but nothing was calling it.
Root cause: Cron job was never created for the full backup script. The hermes-live-sync covered configs every 15 min but full archive was orphaned.
Fix: Created cron job hermes-full-backup at 0 5 * * * calling run-hermes-backup.sh wrapper.
Status: ✅ Fixed & verified 2026-07-08
Verified by: Cron job created and scheduled for next 5 AM ET
DR-007: home-router-daily-backup cron error
Problem: Cron job errored at 06:00 today. SSH export on router produced a stuck .in_progress file that never completed. Root cause: Cron was using home-router-backup.sh (old WireGuard tunnel script) instead of the proper run-wisp-backup.sh pipeline. Stuck export files blocked SCP. Fix: Changed cron to use run-wisp-backup.sh. Cleaned stuck .in_progress files from router. Missing packages (paramiko, xl2tpd, strongSwan) installed. Status: ✅ Fixed & verified 2026-07-08 Verified by: Backup ran end-to-end, config uploaded to S3
DR-008: MikroTik CCR backup stale (last Jul 5)
Problem: mikrotik-ccr-backups S3 bucket had no uploads since Jul 5. Root cause (two causes): (1) wisp-backup.py failed because paramiko wasn't installed. (2) tower IP in config.yaml was 192.168.88.1 (LAN) but SSH is restricted to WireGuard tunnel network 10.77.0.0/24. Fix: Installed paramiko v5.0.0. Updated tower IP to 10.77.0.2 in wisp-backup/config.yaml. Installed missing VPN stack (xl2tpd, strongSwan). Status: ✅ Fixed & verified 2026-07-08 Verified by: Backup ran end-to-end, config uploaded to S3
DR-009: app1-bu warm vs cold docs mismatch
Problem: DR plan doc said "offline, boots on demand" but server is running (3 days uptime). Root cause: DR plan doc was outdated — actual design is warm standby (always on, Hermes dormant). Fix: Updated DR plan to reflect warm standby design. Server stays running. Status: ✅ Resolved — not a bug, docs were wrong
DR-010: Failover timing change (per Germaine's direction)
Change: Check interval from every 10 min → every 5 min. Constant check window from 3 min → 2 min (30s × 4 cycles). Rationale: Faster detection, shorter failover. Status: ✅ Fixed & verified 2026-07-08 Verified by: Cron changed to */5. Watchdog timing updated on app1-bu.
DR-011: AWS CLI missing on app1-bu standby
Problem: /opt/awscli-venv was missing on app1-bu. Failover could not pull fresh state from S3. Root cause: python3-venv package not installed on standby server. Fix: Installed python3-venv, created venv, installed awscli. Created hermes-standby-sync.sh script with */10 cron. Status: ✅ Fixed & verified 2026-07-08 Verified by: First sync ran end-to-end. Config.yaml timestamp went from Jul 5 → Jul 8 17:10.