# DR Issue Log

> Permanent record of all disaster recovery audit findings, root causes, fixes, and verification dates.
> Updated: 2026-07-09

---

## Summary

**Fixed** [OK]
- **DR-001** [HIGH] Caddyfile not backed up -> added to backup scripts
- **DR-002** [MED] Systemd backup path wrong -> corrected to /etc/systemd/system/
- **DR-003** [MED] migration-creds.txt not backed up -> added to backup scope
- **DR-004** [MED] dre-temp-passwords.txt 644->600
- **DR-005** [MED] migration-creds.txt 644->600
- **DR-006** [HIGH] Full backup stale (last Jul 5) -> manual backup ran Jul 10 (527MB), system crontab added at 1 AM daily
- **DR-007** [HIGH] home-router backup cron error -> switched to run-wisp-backup.sh
- **DR-008** [HIGH] MikroTik backup stale -> installed deps, fixed IP
- **DR-010** [MED] Failover timing -> 5min/2min from 10min/3.5min
- **DR-011** [HIGH] S3 buckets system-configs & docker-volumes created, versioned, IAM updated
- **DR-012** [MED] firecrawl-usage-check crash on KeyError 'monthly' -> hardened load()
- **DR-013** [LOW] ops collector could not find Hetzner token -> added .hetzner_token fallback

**Resolved** [INFO]
- **DR-009** app1-bu DR plan mismatch -> server stays warm per design
- **DR-014** [INFO] home-router-daily-backup paramiko error -> verified working via live test Jul 10

**Investigating / Blocked** [PENDING]
- **DR-015** [MED] service-health-check / apex-mail-watchdog failing on real remote outages (wphost02, WireGuard)
- **DR-016** [HIGH] root-essentials-backup silently failing — tarballs created locally but not uploaded to S3 since Jul 12
- **DR-017** [HIGH] WISP CCR tower configs not backed up — only one old config (Jul 7) in mikrotik-ccr-backups
- **DR-018** [MED] wphost02 backup singular — only one backup from Jul 10, no recurring schedule
- **DR-019** [MED] SiteGround WordPress backup not implemented — siteground/ prefix empty
- **DR-011b** [HIGH] system-configs & docker-volumes sync buckets still empty — 0 objects, scripts exist but not populating

---

## 2026-07-08 -- Initial Full DR Audit

### DR-001 -- Caddyfile not backed up  `[HIGH]  [OK] Fixed`

**Problem**
`/etc/caddy/Caddyfile` was not copied by `hermes-backup.sh` or `hermes-live-sync.sh`. If Core server fails, the reverse proxy config would need to be rebuilt from scratch.

**Root Cause**
Backup scripts were written to cover Hermes config and user directories but omitted system-level config files entirely.

**Fix**
Added `/etc/caddy/Caddyfile` to both `hermes-backup.sh` and `hermes-live-sync.sh` with DR FIX comments dated 2026-07-08.

**Verification**
- [OK] `bash -n` syntax check passed on both scripts
- [OK] Subagent confirmed all paths referenced correctly

---

### DR-002 -- Systemd service backup path wrong  `[MED]  [OK] Fixed`

**Problem**
`hermes-backup.sh` collected systemd services from `~/.config/systemd/user/` instead of `/etc/systemd/system/`. Real service files (`hermes-agent.service`, `shark-game.service`, etc.) were not backed up.

**Root Cause**
Backup script path pointed to user-level systemd directory instead of system-level.

**Fix**
Corrected path to `/etc/systemd/system/*.service` in `hermes-backup.sh`. Embedded restore script also updated.

**Verification**
- [OK] Post-fix script syntax check
- [OK] Subagent confirmed correct paths

---

### DR-003 -- migration-creds.txt not in backup scope  `[MED]  [OK] Fixed`

**Problem**
`/root/.hermes/migration-creds.txt` existed but wasn't referenced by any backup script.

**Root Cause**
File was added to the system after backup script was written, never included in scope.

**Fix**
Added to `hermes-backup.sh` file list with `chmod 600` restore instruction.

**Verification**
- [OK] Post-fix syntax check
- [OK] File confirmed present and included

---

### DR-004 -- dre-temp-passwords.txt exposed  `[MED]  [OK] Fixed`

**Problem**
`/root/.hermes/references/dre-temp-passwords.txt` was readable by all users (644) instead of owner-only (600).

**Root Cause**
Script created the file without explicit permission setting.

**Fix**
`chmod 600`

**Verification**
- [OK] `ls -la` confirms `-rw-------`

---

### DR-005 -- migration-creds.txt exposed  `[MED]  [OK] Fixed`

**Problem**
Same as DR-004 -- `migration-creds.txt` was 644.

**Root Cause**
Written without explicit permission setting.

**Fix**
`chmod 600`

**Verification**
- [OK] `ls -la` confirms `-rw-------`

---

### DR-006 -- Full backup stale  `[HIGH]  [OK] Fixed 2026-07-10`

**Problem**
`hermes-full-backup.tar.gz` last uploaded to S3 on Jul 5. Daily 5 AM cron had missed 3 days.

**Root Cause**
No cron job scheduled the backup script. `hermes-backup.sh` existed and was correct but was never wired to cron via Hermes or system crontab. The gateway lifecycle guard (#30719) blocked running it as a Hermes cron job.

**Fix**
- Manually ran `hermes-backup.sh` on Jul 10 — produced 527MB tarball, uploaded successfully
- Added system crontab entry: `0 1 * * * /root/.hermes/scripts/hermes-backup.sh 2>&1 | logger -t hermes-full-backup`
- Added audit watchdog: `0 2 * * * /root/.hermes/scripts/backup-audit-check.sh 2>&1 | logger -t backup-audit`

**Verification**
- [OK] Manual backup completed: `hermes-full-backup-2026-07-10.tar.gz` (527,318,371 bytes) in S3
- [OK] Crontab entry confirmed active: `crontab -l`
- [OK] Audit script in place to verify backup completion at +1h

---

### DR-007 -- home-router backup cron error  `[HIGH]  [OK] Fixed`

**Problem**
Cron job errored at 06:00 -- backup script failed. SSH export on router produced a stuck `.in_progress` file that blocked SCP.

**Root Cause**
Cron was using `home-router-backup.sh` (old WireGuard tunnel script) instead of the proper `run-wisp-backup.sh` pipeline. Stuck export files blocked SCP -> S3 upload failed.

**Fix**
- Changed cron to use `run-wisp-backup.sh`
- Cleaned stuck `.in_progress` files from router
- Installed missing packages: `paramiko v5.0.0`, `xl2tpd`, `strongSwan`

**Verification**
- [OK] Backup ran end-to-end
- [OK] Config uploaded to `s3://mikrotik-ccr-backups/wisp-backups/configs/home/2026-07-08/`

---

### DR-008 -- MikroTik CCR backup stale  `[HIGH]  [OK] Fixed`

**Problem**
`mikrotik-ccr-backups` S3 bucket had no uploads since Jul 5. Two compounding root causes prevented backups.

**Root Causes**
1. `wisp-backup.py` failed because `paramiko` was not installed
2. Tower IP in `config.yaml` was `192.168.88.1` (LAN) but SSH is restricted to WireGuard tunnel network `10.77.0.0/24`

**Fix**
- Installed `paramiko v5.0.0`
- Updated tower IP to `10.77.0.2` in `wisp-backup/config.yaml`
- Installed missing VPN stack: `xl2tpd`, `strongSwan`

**Verification**
- [OK] Backup ran end-to-end
- [OK] Config uploaded to S3 successfully

---

## 2026-07-08 -- Failover Logic Update

### DR-009 -- app1-bu DR plan mismatch  `[INFO]  [OK] Resolved`

**Problem**
DR plan doc stated "offline, boots on demand" but server was running (3 days uptime).

**Root Cause**
DR plan documentation was outdated. Actual design is **warm standby** -- always on with Hermes dormant.

**Fix**
Updated DR plan to reflect warm standby design. Server stays running.

**Verification**
- [OK] Docs corrected
- [OK] Server continues as-is

---

### DR-010 -- Failover timing adjustment  `[MED]  [OK] Fixed`

**Problem**
Failover detection was too slow: 10-minute check intervals with 3.5-minute confirmation window.

**Change**
- **Check interval:** Every 10 min -> **Every 5 min**
- **Confirmation:** 4 x 60s (3.5 min) -> **4 x 30s (2 min)**
- **Max downtime:** ~13.5 min -> **~7 min**

**Rationale**
Faster detection = shorter failover window. If Core doesn't respond within 2 min of constant checking, app1-bu activates.

**Verification**
- [OK] Cron changed to `*/5 * * * *`
- [OK] Watchdog updated: 30s × 4 cycles = 2 min confirmation
- [OK] Verified via SSH on app1-bu

---

## 2026-07-09 -- Session Findings (Ops/Infra pass)

### DR-006 -- Full backup stale (UPDATE: root cause found)  `[HIGH]  [PENDING] Blocked`

**Problem**
`hermes-full-backup` in Wasabi (`s3://hermes-vps-backups/hermes-full-backup/`) last uploaded
Jul 5. Ops collector reports it `critical` (age > 72h).

**Root Cause (identified 2026-07-09)**
`hermes-backup.sh` exists and is correct (uploads the full tarball to the right path), but
there is NO Hermes cron job that runs it. The 21 jobs in `cron/jobs.json` include
`hermes-live-sync` (every 15m -> `live/` path, healthy) but nothing that runs the daily full
backup. The DR-006 note about a `run-hermes-backup.sh` wrapper was never wired to cron.

**Fix (pending -- requires cron creation, terminal approval-gated this session)**
Create a daily cron job that runs the full backup, e.g.:
```
hermes cron create --name hermes-full-backup --schedule "0 5 * * *" \
  --script hermes-backup.sh --no-agent --deliver local
```
Or via cronjob(action='create', name='hermes-full-backup', schedule='0 5 * * *',
script='hermes-backup.sh', no_agent=True, deliver='local'). After first run, verify the
collector flips this bucket to `ok`.

**Status**
- [PENDING] `hermes-backup.sh` verified correct by inspection
- [PENDING] Cron job must be created to schedule it

---

### DR-011 -- S3 buckets missing + sync not scheduled  `[HIGH]  [OK] Fixed 2026-07-10`

**Problem**
Ops collector reported `itpropartner-system-configs` and `itpropartner-docker-volumes` as `NoSuchBucket`. These are the normalized buckets from the Jul 8 S3 plan.

**Root Cause**
Two compounding issues:
1. The buckets were never created. The Hermes-User IAM key lacked `s3:CreateBucket`, so they had to be created in the Wasabi Console web UI (manual step, never done).
2. The sync scripts (`hermes-system-config-sync.sh`, `hermes-docker-sync.sh`) exist and are correct, but had no cron job scheduling them.

**Fix (completed Jul 10)**
1. Created both buckets via Wasabi API: `itpropartner-system-configs`, `itpropartner-docker-volumes`
2. Enabled versioning on both buckets via `aws s3api put-bucket-versioning`
3. Updated Hermes-User IAM policy to include both bucket ARNs
4. Verified PUT/GET/DELETE operations work on both buckets
5. Sync scripts confirmed present and executable — cron scheduling pending (tracked as DR-011b)

**Verification**
- [OK] Both buckets exist and respond to S3 operations
- [OK] Versioning enabled on both
- [OK] IAM policy updated with bucket ARNs
- [OK] PUT/GET/DELETE tested successfully
- [PENDING] Sync cron jobs to populate buckets (DR-011b)

---

### DR-012 -- firecrawl-usage-check crash  `[MED]  [OK] Fixed`

**Problem**
`firecrawl-usage-check` cron errored: `KeyError: 'monthly'` in `track-firecrawl.py summary()`.

**Root Cause**
`load()` returned the raw JSON from `firecrawl-usage.json`. An older state file lacked the
`monthly` key, so `data["monthly"].get(...)` raised KeyError. The loader was not schema-safe.

**Fix**
Rewrote `load()` in `/root/.hermes/scripts/track-firecrawl.py` to start from a defaults dict
and merge the file on top, then coerce `monthly`/`calls`/`total_used` to correct types. This
is forward/backward compatible with partial or corrupt state files.

**Verification**
- [OK] `patch` lint (py_compile) passed
- [OK] Current `firecrawl-usage.json` already contains `monthly` key -> next run will pass

---

### DR-013 -- Ops collector could not read Hetzner token  `[LOW]  [OK] Fixed`

**Problem**
`ops-status.json` showed `hetzner_servers: [{status: error, message: "HETZNER_API_TOKEN not found"}]`,
so the server inventory on the ops portal was empty.

**Root Cause**
`collect_hetzner_servers()` only looked in `os.environ` and `.env`. The Hetzner token on this
box lives in the file `/root/.hermes/scripts/.hetzner_token` (used by `snapshot-hetzner.py`),
which the collector never checked.

**Fix**
Added a fallback in `collect_hetzner_servers()` to read `/root/.hermes/scripts/.hetzner_token`
when env/.env lookups miss. (`ops-data-collector.py`)

**Verification**
- [OK] `patch` lint passed
- [PENDING] Confirm inventory populates on next collector run (needs .hetzner_token present)

---

### DR-014 -- home-router-daily-backup paramiko error  `[INFO]  [OK] Resolved 2026-07-10`

**Problem**
Job showed `last_status: error` with `ModuleNotFoundError: No module named 'paramiko'`.

**Root Cause**
The error was from the 06:00 Jul 9 run. paramiko had previously been installed against Python 3.11 (leftover `wisp-backup.cpython-311.pyc`), but the system `python3` is now 3.13.

**Resolution**
paramiko IS present for the current interpreter: `/usr/local/lib/python3.13/dist-packages/paramiko/`. The error was a stale artifact from before the py3.13 install was confirmed.

**Live Test Verification (Jul 10)**
- [OK] Ran `wisp-backup.py` manually — completed successfully
- [OK] WireGuard tunnel to 10.77.0.2 confirmed up
- [OK] Config fetched and uploaded to `s3://mikrotik-ccr-backups/wisp-backups/configs/home/2026-07-10/` (3,898 bytes)
- [OK] Logs fetched and uploaded
- [OK] Exit: 1 OK, 0 Failed
- [OK] No paramiko import error — resolved for python3.13

---

### DR-015 -- health-check / apex watchdog failing on remote outages  `[MED]  [PENDING] External`

**Problem**
`service-health-check` and `apex-mail-watchdog` report errors every cycle.

**Root Cause**
Genuine remote conditions, not script bugs:
- `wphost02` (5.161.62.38) SSH is refused -> apex watchdog SMTP test cannot run.
- WireGuard tunnel to home router (10.77.0.2) is down -> health check + Home-Router-Watchdog fail.
- Portal mockup on port 8081 and MySQL tunnel targets are also unreachable.

Note: the current `apex-mail-watchdog.sh` does LOGIN-only SMTP tests (no test email sent), so
it is NOT the source of the "apex test emails" complaint -- that was an earlier version.

**Fix**
No code change. These clear when wphost02 SSH and the WireGuard tunnel are restored (part of
the broader migration / home-router work). Documented so the errors are understood, not chased
as script bugs.

**Status**
- [PENDING] External dependency -- resolves when remote hosts/tunnel are back online

---

## 2026-07-15 -- Full DR Audit

### DR-016 -- root-essentials-backup silently failing `[HIGH] [PENDING]`

**Problem**
root-essentials-backup ran at 3 AM daily per system crontab but tarballs were NOT being uploaded to S3. S3 only has Jul 10 and Jul 11. Local tarballs accumulated in /tmp (Jul 12-15, 4 files) proving the script runs and creates archives but the upload step fails silently.

**Root Cause**
The script runs past tar creation (local tarballs exist) but the aws s3 cp upload step fails. With set -euo pipefail the script exits on the upload failure, leaving the tarball in /tmp (cleanup never runs). Exact cause of upload failure TBD -- possible S3 permission change or endpoint issue.

Also: the Caddyfile path on line 30 is wrong (`-C /etc systemd/system/caddy/Caddyfile` should be `-C /etc caddy/Caddyfile`), but this is masked by `|| true`.

**Fix**
Pending investigation. Need to run the script manually with verbose output to identify the upload failure.

**Status**
- [PENDING] Investigating

---

### DR-017 -- WISP CCR tower configs not backed up `[HIGH] [PENDING]`

**Problem**
mikrotik-ccr-backups bucket contains home-gateway configs (daily Jul 9-15) but only one WISP CCR config from Jul 7 (`2026-07-07-config.rsc`). No tower router configs since then.

**Root Cause**
wisp-backup.py script references VPN and CCR towers in its docstring, but only the home gateway configs appear in S3. The script may not be reaching the tower CCRs, or the CCR backup path may be misconfigured in config.yaml.

**Fix**
Pending investigation. Need to review wisp-backup/config.yaml and test connectivity to WISP CCR towers.

**Status**
- [PENDING] Investigating

---

### DR-018 -- wphost02 backup not recurring `[MED] [PENDING]`

**Problem**
Only one wphost02 backup exists on S3: wphost02-backup-2026-07-10.tar.gz (656 MB). No recurring backup schedule. MariaDB dumps on wphost02 and fleettracker360 were flagged as known gaps in the Jul 10 DR audit but not yet resolved.

**Root Cause**
Backup was a one-time manual capture during the Jul 10 DR audit. No cron job was created for recurring wphost02 backups.

**Fix**
Pending. Needs a cron job on wphost02 to run regular backups to S3.

**Status**
- [PENDING] Known gap from Jul 10 audit, not yet resolved

---

### DR-019 -- SiteGround WordPress backup not implemented `[MED] [PENDING]`

**Problem**
siteground/ prefix in hermes-vps-backups is completely empty. No SiteGround WordPress site backups exist on S3. MainWP + WPvivid Pro backs up 15 sites to Wasabi independently, but sites not in MainWP have no S3 backup.

**Root Cause**
Fleet-wide SFTP backup timed out at 600 seconds on Jul 10. No alternative was deployed for sites outside MainWP coverage.

**Fix**
Pending. Either batch SFTP backups in groups of 3-5 or extend MainWP coverage to remaining sites.

**Status**
- [PENDING] Known gap, not yet resolved

---

### DR-011b -- system-configs & docker-volumes sync still empty `[HIGH] [PENDING]`

**Problem**
Both itpropartner-system-configs and itpropartner-docker-volumes buckets exist on Wasabi with versioning enabled, but contain 0 objects. The sync scripts (hermes-system-config-sync.sh, hermes-docker-sync.sh) exist and have correct syntax + venv activation, but no Hermes cron jobs are creating them.

**Root Cause**
Cron jobs were never created for these sync scripts. The hermes-live-sync was split into purpose-specific scripts on Jul 8 but the corresponding cron jobs were deferred.

**Fix**
Create two no_agent cron jobs: hermes-system-config-sync (every 15m) and hermes-docker-sync (every 15m).

**Status**
- [PENDING] Deferred since Jul 10