From bbce89e25ba0b63e3842d77305266bc86b062422 Mon Sep 17 00:00:00 2001 From: root Date: Tue, 15 Sep 2026 13:59:23 -0400 Subject: [PATCH] standby: fix self-matching primary guard in sync + watchdog (use unit state + MainPID); drop 45 orphan output dirs; drop dead .bashrc sync entry --- CHANGELOG.md | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 98b78e9..ae4aa2d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -29,6 +29,35 @@ Result: a full cycle now completes in **13 seconds** with a 9-line log and **zer # itpp-infrastructure — CHANGELOG +## 2026-09-15 (fourth) — the standby's "am I primary?" guard was matching the wrong thing + +**Found while verifying the cron/output cleanup:** at 17:50:01Z the armed standby (core-bu) logged `This box is primary (gateway running), sync skipped to avoid clobbering live config` - while its gateway unit was `inactive/disabled` and MainPID was 0. The guard was: + +```bash +if pgrep -f "hermes gateway" >/dev/null 2>&1; then ... exit 0; fi +``` + +`pgrep -f` matches any process whose **command line contains the string**, not the gateway. It matched transient probes, log lines and other scripts' own pgrep calls (measured: 3 matches while nothing was running). Consequences: + +- **Sync:** the state refresh silently skipped itself. A skipped sync is indistinguishable from a successful one from the outside, so the standby would quietly drift stale. Both crons fire on 5-minute boundaries, so the sync and the watchdog routinely collide. +- **Watchdog:** the same pattern guards `STANDBY_RUNNING` at `hermes-standby-watchdog.sh:159`. A false positive there is worse - it exits without ever probing the live box, so a real outage produces no failover at all. + +**Fix (both scripts):** ask systemd, which cannot be forged by a command line. + +```bash +gw="$(XDG_RUNTIME_DIR=/run/user/0 systemctl --user is-active hermes-gateway 2>/dev/null || true)" +pid="$(XDG_RUNTIME_DIR=/run/user/0 systemctl --user show -p MainPID --value hermes-gateway 2>/dev/null || echo 0)" +if [ "$gw" = "active" ] || { [ "$pid" != "0" ] && [ -n "$pid" ]; }; then ... fi +``` + +Verified: with a decoy process alive carrying the old phrase, the guard now reads `unit=inactive mainpid=0 -> not primary`, the sync runs to completion (`Sync started` ... `Sync complete`), and a full watchdog run probes Core, reports dormant, and does not take over. Unit definition confirmed sound: `ExecStart=/usr/local/lib/hermes-agent/venv/bin/python -m hermes_cli.main gateway run`, so service state is authoritative. + +**Also in this pass** + +- Removed 45 orphaned `cron/output/` dirs on Core (job ids absent from `jobs.json`; newest output 2026-09-03): 133 dirs -> 89, matching the 89 live jobs; 36 MB -> 34 MB. All 8 live state files intact and md5-verified. +- Dropped `.bashrc` from the standby sync's config list: it lives in `$HOME`, never in `$HOME/.hermes`, so `live/.bashrc` cannot exist and every cycle logged a misleading `fatal error ... Key "live/.bashrc" does not exist` that could mask a real error. The sync log is now 9 clean lines. +- Backups: `hermes-standby-sync.sh.bak3-20260915.2`, `hermes-standby-watchdog.sh.bak3-20260915.2` on core-bu. + ## 2026-09-15 (third) — Core's stale job output: root cause, cleanup, allow-list fix **Root cause.** `hermes-live-sync.sh` mirrors `~/.hermes/` to `s3://hermes-vps-backups/live/` every 11 minutes and had **no `cron/output` exclusion**, while `aws s3 sync` only ever adds and updates, never deletes. So every cron job-run output Core has ever produced is still in S3 (July 5 onward), long after Core's own 50-runs-per-job retention pruned it locally. Measured 2026-09-15: **270,262 objects / 624 MB** under `live/cron/output/` against 16,718 files on Core. The cost was never the storage, it was the listing: every restore and every standby sync had to enumerate 270k objects, which is what pushed the standby sync past its own 10-minute interval and made two run concurrently.