From 58de741d94eb57531966e7882e740947fb006afa Mon Sep 17 00:00:00 2001 From: root Date: Tue, 15 Sep 2026 14:01:13 -0400 Subject: [PATCH] standby watchdog: failback stops the unit and verifies it, instead of a pattern that matched nothing (silent split-brain risk) --- CHANGELOG.md | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index ae4aa2d..b7a8b40 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -29,6 +29,34 @@ Result: a full cycle now completes in **13 seconds** with a 9-line log and **zer # itpp-infrastructure — CHANGELOG +## 2026-09-15 (fifth) — failback could not actually stand the standby down + +Found in the same sweep as the guard fix, in the watchdog's failback branch (Core healthy again after a takeover, so this box must go dormant): + +```bash +pkill__ -f "hermes gateway" 2>/dev/null || true +echo "OK $(date +%s)" > "$HEALTH_STATE" +log "=== FAILBACK complete: $STANDBY_NAME dormant, Core primary ===" +``` + +The real service command line is `/usr/local/lib/hermes-agent/venv/bin/python -m hermes_cli.main hermes-gateway run`, which does **not** contain the phrase `hermes gateway`. The pattern matched nothing, so the branch logged a successful stand-down, wrote a healthy state, and left the service running - **two gateways on one bot token**, the exact split-brain this design exists to prevent. It was never exercised because core-bu has never taken over. + +**Fix:** stand down by unit, then require proof before announcing anything. + +```bash +XDG_RUNTIME_DIR=/run/user/0 systemctl --user stop hermes-gateway +sleep 3 +FB_STATE=; FB_PID=; FB_PROC= +if || || ; then + ; + if still found -> log "FAILBACK FAILED", alert, exit 1 +fi +``` + +The process check uses the exact signature and is deliberately biased toward false positives: a wrong "still running" produces a loud alert, whereas the old wrong "all clear" produced silent split-brain. + +**Verified:** a decoy process carrying the real signature is detected as STILL SERVING (would alert and exit 1, not claim success); with no decoy the same logic reads dormant; the watchdog still runs clean end-to-end (exit 0, health state refreshed, unit inactive, still armed). Backup: `hermes-standby-watchdog.sh.bak4-20260915` on core-bu. + ## 2026-09-15 (fourth) — the standby's "am I primary?" guard was matching the wrong thing **Found while verifying the cron/output cleanup:** at 17:50:01Z the armed standby (core-bu) logged `This box is primary (gateway running), sync skipped to avoid clobbering live config` - while its gateway unit was `inactive/disabled` and MainPID was 0. The guard was: