ippadmin/hermes-recovery
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
master
hermes-recovery/references/dr-issue-log.md
T

17 KiB
Raw Permalink Blame History

DR Issue Log

Permanent record of all disaster recovery audit findings, root causes, fixes, and verification dates. Updated: 2026-07-09

Summary

Fixed [OK]

  • DR-001 [HIGH] Caddyfile not backed up -> added to backup scripts
  • DR-002 [MED] Systemd backup path wrong -> corrected to /etc/systemd/system/
  • DR-003 [MED] migration-creds.txt not backed up -> added to backup scope
  • DR-004 [MED] dre-temp-passwords.txt 644->600
  • DR-005 [MED] migration-creds.txt 644->600
  • DR-006 [HIGH] Full backup stale (last Jul 5) -> manual backup ran Jul 10 (527MB), system crontab added at 1 AM daily
  • DR-007 [HIGH] home-router backup cron error -> switched to run-wisp-backup.sh
  • DR-008 [HIGH] MikroTik backup stale -> installed deps, fixed IP
  • DR-010 [MED] Failover timing -> 5min/2min from 10min/3.5min
  • DR-011 [HIGH] S3 buckets system-configs & docker-volumes created, versioned, IAM updated
  • DR-012 [MED] firecrawl-usage-check crash on KeyError 'monthly' -> hardened load()
  • DR-013 [LOW] ops collector could not find Hetzner token -> added .hetzner_token fallback

Resolved [INFO]

  • DR-009 app1-bu DR plan mismatch -> server stays warm per design
  • DR-014 [INFO] home-router-daily-backup paramiko error -> verified working via live test Jul 10

Investigating / Blocked [PENDING]

  • DR-015 [MED] service-health-check / apex-mail-watchdog failing on real remote outages (wphost02, WireGuard)
  • DR-016 [HIGH] root-essentials-backup silently failing — tarballs created locally but not uploaded to S3 since Jul 12
  • DR-017 [HIGH] WISP CCR tower configs not backed up — only one old config (Jul 7) in mikrotik-ccr-backups
  • DR-018 [MED] wphost02 backup singular — only one backup from Jul 10, no recurring schedule
  • DR-019 [MED] SiteGround WordPress backup not implemented — siteground/ prefix empty
  • DR-011b [HIGH] system-configs & docker-volumes sync buckets still empty — 0 objects, scripts exist but not populating

2026-07-08 -- Initial Full DR Audit

DR-001 -- Caddyfile not backed up [HIGH] [OK] Fixed

Problem /etc/caddy/Caddyfile was not copied by hermes-backup.sh or hermes-live-sync.sh. If Core server fails, the reverse proxy config would need to be rebuilt from scratch.

Root Cause Backup scripts were written to cover Hermes config and user directories but omitted system-level config files entirely.

Fix Added /etc/caddy/Caddyfile to both hermes-backup.sh and hermes-live-sync.sh with DR FIX comments dated 2026-07-08.

Verification

  • [OK] bash -n syntax check passed on both scripts
  • [OK] Subagent confirmed all paths referenced correctly

DR-002 -- Systemd service backup path wrong [MED] [OK] Fixed

Problem hermes-backup.sh collected systemd services from ~/.config/systemd/user/ instead of /etc/systemd/system/. Real service files (hermes-agent.service, shark-game.service, etc.) were not backed up.

Root Cause Backup script path pointed to user-level systemd directory instead of system-level.

Fix Corrected path to /etc/systemd/system/*.service in hermes-backup.sh. Embedded restore script also updated.

Verification

  • [OK] Post-fix script syntax check
  • [OK] Subagent confirmed correct paths

DR-003 -- migration-creds.txt not in backup scope [MED] [OK] Fixed

Problem /root/.hermes/migration-creds.txt existed but wasn't referenced by any backup script.

Root Cause File was added to the system after backup script was written, never included in scope.

Fix Added to hermes-backup.sh file list with chmod 600 restore instruction.

Verification

  • [OK] Post-fix syntax check
  • [OK] File confirmed present and included

DR-004 -- dre-temp-passwords.txt exposed [MED] [OK] Fixed

Problem /root/.hermes/references/dre-temp-passwords.txt was readable by all users (644) instead of owner-only (600).

Root Cause Script created the file without explicit permission setting.

Fix chmod 600

Verification

  • [OK] ls -la confirms -rw-------

DR-005 -- migration-creds.txt exposed [MED] [OK] Fixed

Problem Same as DR-004 -- migration-creds.txt was 644.

Root Cause Written without explicit permission setting.

Fix chmod 600

Verification

  • [OK] ls -la confirms -rw-------

DR-006 -- Full backup stale [HIGH] [OK] Fixed 2026-07-10

Problem hermes-full-backup.tar.gz last uploaded to S3 on Jul 5. Daily 5 AM cron had missed 3 days.

Root Cause No cron job scheduled the backup script. hermes-backup.sh existed and was correct but was never wired to cron via Hermes or system crontab. The gateway lifecycle guard (#30719) blocked running it as a Hermes cron job.

Fix

  • Manually ran hermes-backup.sh on Jul 10 — produced 527MB tarball, uploaded successfully
  • Added system crontab entry: 0 1 * * * /root/.hermes/scripts/hermes-backup.sh 2>&1 | logger -t hermes-full-backup
  • Added audit watchdog: 0 2 * * * /root/.hermes/scripts/backup-audit-check.sh 2>&1 | logger -t backup-audit

Verification

  • [OK] Manual backup completed: hermes-full-backup-2026-07-10.tar.gz (527,318,371 bytes) in S3
  • [OK] Crontab entry confirmed active: crontab -l
  • [OK] Audit script in place to verify backup completion at +1h

DR-007 -- home-router backup cron error [HIGH] [OK] Fixed

Problem Cron job errored at 06:00 -- backup script failed. SSH export on router produced a stuck .in_progress file that blocked SCP.

Root Cause Cron was using home-router-backup.sh (old WireGuard tunnel script) instead of the proper run-wisp-backup.sh pipeline. Stuck export files blocked SCP -> S3 upload failed.

Fix

  • Changed cron to use run-wisp-backup.sh
  • Cleaned stuck .in_progress files from router
  • Installed missing packages: paramiko v5.0.0, xl2tpd, strongSwan

Verification

  • [OK] Backup ran end-to-end
  • [OK] Config uploaded to s3://mikrotik-ccr-backups/wisp-backups/configs/home/2026-07-08/

DR-008 -- MikroTik CCR backup stale [HIGH] [OK] Fixed

Problem mikrotik-ccr-backups S3 bucket had no uploads since Jul 5. Two compounding root causes prevented backups.

Root Causes

  1. wisp-backup.py failed because paramiko was not installed
  2. Tower IP in config.yaml was 192.168.88.1 (LAN) but SSH is restricted to WireGuard tunnel network 10.77.0.0/24

Fix

  • Installed paramiko v5.0.0
  • Updated tower IP to 10.77.0.2 in wisp-backup/config.yaml
  • Installed missing VPN stack: xl2tpd, strongSwan

Verification

  • [OK] Backup ran end-to-end
  • [OK] Config uploaded to S3 successfully

2026-07-08 -- Failover Logic Update

DR-009 -- app1-bu DR plan mismatch [INFO] [OK] Resolved

Problem DR plan doc stated "offline, boots on demand" but server was running (3 days uptime).

Root Cause DR plan documentation was outdated. Actual design is warm standby -- always on with Hermes dormant.

Fix Updated DR plan to reflect warm standby design. Server stays running.

Verification

  • [OK] Docs corrected
  • [OK] Server continues as-is

DR-010 -- Failover timing adjustment [MED] [OK] Fixed

Problem Failover detection was too slow: 10-minute check intervals with 3.5-minute confirmation window.

Change

  • Check interval: Every 10 min -> Every 5 min
  • Confirmation: 4 x 60s (3.5 min) -> 4 x 30s (2 min)
  • Max downtime: ~13.5 min -> ~7 min

Rationale Faster detection = shorter failover window. If Core doesn't respond within 2 min of constant checking, app1-bu activates.

Verification

  • [OK] Cron changed to */5 * * * *
  • [OK] Watchdog updated: 30s × 4 cycles = 2 min confirmation
  • [OK] Verified via SSH on app1-bu

2026-07-09 -- Session Findings (Ops/Infra pass)

DR-006 -- Full backup stale (UPDATE: root cause found) [HIGH] [PENDING] Blocked

Problem hermes-full-backup in Wasabi (s3://hermes-vps-backups/hermes-full-backup/) last uploaded Jul 5. Ops collector reports it critical (age > 72h).

Root Cause (identified 2026-07-09) hermes-backup.sh exists and is correct (uploads the full tarball to the right path), but there is NO Hermes cron job that runs it. The 21 jobs in cron/jobs.json include hermes-live-sync (every 15m -> live/ path, healthy) but nothing that runs the daily full backup. The DR-006 note about a run-hermes-backup.sh wrapper was never wired to cron.

Fix (pending -- requires cron creation, terminal approval-gated this session) Create a daily cron job that runs the full backup, e.g.:

hermes cron create --name hermes-full-backup --schedule "0 5 * * *" \
  --script hermes-backup.sh --no-agent --deliver local

Or via cronjob(action='create', name='hermes-full-backup', schedule='0 5 * * *', script='hermes-backup.sh', no_agent=True, deliver='local'). After first run, verify the collector flips this bucket to ok.

Status

  • [PENDING] hermes-backup.sh verified correct by inspection
  • [PENDING] Cron job must be created to schedule it

DR-011 -- S3 buckets missing + sync not scheduled [HIGH] [OK] Fixed 2026-07-10

Problem Ops collector reported itpropartner-system-configs and itpropartner-docker-volumes as NoSuchBucket. These are the normalized buckets from the Jul 8 S3 plan.

Root Cause Two compounding issues:

  1. The buckets were never created. The Hermes-User IAM key lacked s3:CreateBucket, so they had to be created in the Wasabi Console web UI (manual step, never done).
  2. The sync scripts (hermes-system-config-sync.sh, hermes-docker-sync.sh) exist and are correct, but had no cron job scheduling them.

Fix (completed Jul 10)

  1. Created both buckets via Wasabi API: itpropartner-system-configs, itpropartner-docker-volumes
  2. Enabled versioning on both buckets via aws s3api put-bucket-versioning
  3. Updated Hermes-User IAM policy to include both bucket ARNs
  4. Verified PUT/GET/DELETE operations work on both buckets
  5. Sync scripts confirmed present and executable — cron scheduling pending (tracked as DR-011b)

Verification

  • [OK] Both buckets exist and respond to S3 operations
  • [OK] Versioning enabled on both
  • [OK] IAM policy updated with bucket ARNs
  • [OK] PUT/GET/DELETE tested successfully
  • [PENDING] Sync cron jobs to populate buckets (DR-011b)

DR-012 -- firecrawl-usage-check crash [MED] [OK] Fixed

Problem firecrawl-usage-check cron errored: KeyError: 'monthly' in track-firecrawl.py summary().

Root Cause load() returned the raw JSON from firecrawl-usage.json. An older state file lacked the monthly key, so data["monthly"].get(...) raised KeyError. The loader was not schema-safe.

Fix Rewrote load() in /root/.hermes/scripts/track-firecrawl.py to start from a defaults dict and merge the file on top, then coerce monthly/calls/total_used to correct types. This is forward/backward compatible with partial or corrupt state files.

Verification

  • [OK] patch lint (py_compile) passed
  • [OK] Current firecrawl-usage.json already contains monthly key -> next run will pass

DR-013 -- Ops collector could not read Hetzner token [LOW] [OK] Fixed

Problem ops-status.json showed hetzner_servers: [{status: error, message: "HETZNER_API_TOKEN not found"}], so the server inventory on the ops portal was empty.

Root Cause collect_hetzner_servers() only looked in os.environ and .env. The Hetzner token on this box lives in the file /root/.hermes/scripts/.hetzner_token (used by snapshot-hetzner.py), which the collector never checked.

Fix Added a fallback in collect_hetzner_servers() to read /root/.hermes/scripts/.hetzner_token when env/.env lookups miss. (ops-data-collector.py)

Verification

  • [OK] patch lint passed
  • [PENDING] Confirm inventory populates on next collector run (needs .hetzner_token present)

DR-014 -- home-router-daily-backup paramiko error [INFO] [OK] Resolved 2026-07-10

Problem Job showed last_status: error with ModuleNotFoundError: No module named 'paramiko'.

Root Cause The error was from the 06:00 Jul 9 run. paramiko had previously been installed against Python 3.11 (leftover wisp-backup.cpython-311.pyc), but the system python3 is now 3.13.

Resolution paramiko IS present for the current interpreter: /usr/local/lib/python3.13/dist-packages/paramiko/. The error was a stale artifact from before the py3.13 install was confirmed.

Live Test Verification (Jul 10)

  • [OK] Ran wisp-backup.py manually — completed successfully
  • [OK] WireGuard tunnel to 10.77.0.2 confirmed up
  • [OK] Config fetched and uploaded to s3://mikrotik-ccr-backups/wisp-backups/configs/home/2026-07-10/ (3,898 bytes)
  • [OK] Logs fetched and uploaded
  • [OK] Exit: 1 OK, 0 Failed
  • [OK] No paramiko import error — resolved for python3.13

DR-015 -- health-check / apex watchdog failing on remote outages [MED] [PENDING] External

Problem service-health-check and apex-mail-watchdog report errors every cycle.

Root Cause Genuine remote conditions, not script bugs:

  • wphost02 (5.161.62.38) SSH is refused -> apex watchdog SMTP test cannot run.
  • WireGuard tunnel to home router (10.77.0.2) is down -> health check + Home-Router-Watchdog fail.
  • Portal mockup on port 8081 and MySQL tunnel targets are also unreachable.

Note: the current apex-mail-watchdog.sh does LOGIN-only SMTP tests (no test email sent), so it is NOT the source of the "apex test emails" complaint -- that was an earlier version.

Fix No code change. These clear when wphost02 SSH and the WireGuard tunnel are restored (part of the broader migration / home-router work). Documented so the errors are understood, not chased as script bugs.

Status

  • [PENDING] External dependency -- resolves when remote hosts/tunnel are back online

2026-07-15 -- Full DR Audit

DR-016 -- root-essentials-backup silently failing [HIGH] [PENDING]

Problem root-essentials-backup ran at 3 AM daily per system crontab but tarballs were NOT being uploaded to S3. S3 only has Jul 10 and Jul 11. Local tarballs accumulated in /tmp (Jul 12-15, 4 files) proving the script runs and creates archives but the upload step fails silently.

Root Cause The script runs past tar creation (local tarballs exist) but the aws s3 cp upload step fails. With set -euo pipefail the script exits on the upload failure, leaving the tarball in /tmp (cleanup never runs). Exact cause of upload failure TBD -- possible S3 permission change or endpoint issue.

Also: the Caddyfile path on line 30 is wrong (-C /etc systemd/system/caddy/Caddyfile should be -C /etc caddy/Caddyfile), but this is masked by || true.

Fix Pending investigation. Need to run the script manually with verbose output to identify the upload failure.

Status

  • [PENDING] Investigating

DR-017 -- WISP CCR tower configs not backed up [HIGH] [PENDING]

Problem mikrotik-ccr-backups bucket contains home-gateway configs (daily Jul 9-15) but only one WISP CCR config from Jul 7 (2026-07-07-config.rsc). No tower router configs since then.

Root Cause wisp-backup.py script references VPN and CCR towers in its docstring, but only the home gateway configs appear in S3. The script may not be reaching the tower CCRs, or the CCR backup path may be misconfigured in config.yaml.

Fix Pending investigation. Need to review wisp-backup/config.yaml and test connectivity to WISP CCR towers.

Status

  • [PENDING] Investigating

DR-018 -- wphost02 backup not recurring [MED] [PENDING]

Problem Only one wphost02 backup exists on S3: wphost02-backup-2026-07-10.tar.gz (656 MB). No recurring backup schedule. MariaDB dumps on wphost02 and fleettracker360 were flagged as known gaps in the Jul 10 DR audit but not yet resolved.

Root Cause Backup was a one-time manual capture during the Jul 10 DR audit. No cron job was created for recurring wphost02 backups.

Fix Pending. Needs a cron job on wphost02 to run regular backups to S3.

Status

  • [PENDING] Known gap from Jul 10 audit, not yet resolved

DR-019 -- SiteGround WordPress backup not implemented [MED] [PENDING]

Problem siteground/ prefix in hermes-vps-backups is completely empty. No SiteGround WordPress site backups exist on S3. MainWP + WPvivid Pro backs up 15 sites to Wasabi independently, but sites not in MainWP have no S3 backup.

Root Cause Fleet-wide SFTP backup timed out at 600 seconds on Jul 10. No alternative was deployed for sites outside MainWP coverage.

Fix Pending. Either batch SFTP backups in groups of 3-5 or extend MainWP coverage to remaining sites.

Status

  • [PENDING] Known gap, not yet resolved

DR-011b -- system-configs & docker-volumes sync still empty [HIGH] [PENDING]

Problem Both itpropartner-system-configs and itpropartner-docker-volumes buckets exist on Wasabi with versioning enabled, but contain 0 objects. The sync scripts (hermes-system-config-sync.sh, hermes-docker-sync.sh) exist and have correct syntax + venv activation, but no Hermes cron jobs are creating them.

Root Cause Cron jobs were never created for these sync scripts. The hermes-live-sync was split into purpose-specific scripts on Jul 8 but the corresponding cron jobs were deferred.

Fix Create two no_agent cron jobs: hermes-system-config-sync (every 15m) and hermes-docker-sync (every 15m).

Status

  • [PENDING] Deferred since Jul 10
Reference in New Issue View Git Blame Copy Permalink