ippadmin/itpp-infrastructure
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

standby: fix self-matching primary guard in sync + watchdog (use unit state + MainPID); drop 45 orphan output dirs; drop dead .bashrc sync entry

Browse Source
This commit is contained in:
root
2026-09-15 13:59:23 -04:00
parent 9453bbe901
commit bbce89e25b
1 changed files with 29 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+29
View File
@@ -29,6 +29,35 @@ Result: a full cycle now completes in **13 seconds** with a 9-line log and **zer
# itpp-infrastructure — CHANGELOG
## 2026-09-15 (fourth) — the standby's "am I primary?" guard was matching the wrong thing
**Found while verifying the cron/output cleanup:** at 17:50:01Z the armed standby (core-bu) logged `This box is primary (gateway running), sync skipped to avoid clobbering live config` - while its gateway unit was `inactive/disabled` and MainPID was 0. The guard was:
```bash
if pgrep -f "hermes gateway" >/dev/null 2>&1; then ... exit 0; fi
```
`pgrep -f` matches any process whose **command line contains the string**, not the gateway. It matched transient probes, log lines and other scripts' own pgrep calls (measured: 3 matches while nothing was running). Consequences:
- **Sync:** the state refresh silently skipped itself. A skipped sync is indistinguishable from a successful one from the outside, so the standby would quietly drift stale. Both crons fire on 5-minute boundaries, so the sync and the watchdog routinely collide.
- **Watchdog:** the same pattern guards `STANDBY_RUNNING` at `hermes-standby-watchdog.sh:159`. A false positive there is worse - it exits without ever probing the live box, so a real outage produces no failover at all.
**Fix (both scripts):** ask systemd, which cannot be forged by a command line.
```bash
gw="$(XDG_RUNTIME_DIR=/run/user/0 systemctl --user is-active hermes-gateway 2>/dev/null || true)"
pid="$(XDG_RUNTIME_DIR=/run/user/0 systemctl --user show -p MainPID --value hermes-gateway 2>/dev/null || echo 0)"
if [ "$gw" = "active" ] || { [ "$pid" != "0" ] && [ -n "$pid" ]; }; then ... fi
```
Verified: with a decoy process alive carrying the old phrase, the guard now reads `unit=inactive mainpid=0 -> not primary`, the sync runs to completion (`Sync started` ... `Sync complete`), and a full watchdog run probes Core, reports dormant, and does not take over. Unit definition confirmed sound: `ExecStart=/usr/local/lib/hermes-agent/venv/bin/python -m hermes_cli.main gateway run`, so service state is authoritative.
**Also in this pass**
- Removed 45 orphaned `cron/output/<jobid>` dirs on Core (job ids absent from `jobs.json`; newest output 2026-09-03): 133 dirs -> 89, matching the 89 live jobs; 36 MB -> 34 MB. All 8 live state files intact and md5-verified.
- Dropped `.bashrc` from the standby sync's config list: it lives in `$HOME`, never in `$HOME/.hermes`, so `live/.bashrc` cannot exist and every cycle logged a misleading `fatal error ... Key "live/.bashrc" does not exist` that could mask a real error. The sync log is now 9 clean lines.
- Backups: `hermes-standby-sync.sh.bak3-20260915.2`, `hermes-standby-watchdog.sh.bak3-20260915.2` on core-bu.
## 2026-09-15 (third) — Core's stale job output: root cause, cleanup, allow-list fix
**Root cause.** `hermes-live-sync.sh` mirrors `~/.hermes/` to `s3://hermes-vps-backups/live/` every 11 minutes and had **no `cron/output` exclusion**, while `aws s3 sync` only ever adds and updates, never deletes. So every cron job-run output Core has ever produced is still in S3 (July 5 onward), long after Core's own 50-runs-per-job retention pruned it locally. Measured 2026-09-15: **270,262 objects / 624 MB** under `live/cron/output/` against 16,718 files on Core. The cost was never the storage, it was the listing: every restore and every standby sync had to enumerate 270k objects, which is what pushed the standby sync past its own 10-minute interval and made two run concurrently.