ippadmin/itpp-infrastructure
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

standby watchdog: failback stops the unit and verifies it, instead of a pattern that matched nothing (silent split-brain risk)

Browse Source
This commit is contained in:
root
2026-09-15 14:01:13 -04:00
parent bbce89e25b
commit 58de741d94
1 changed files with 28 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+28
View File
@@ -29,6 +29,34 @@ Result: a full cycle now completes in **13 seconds** with a 9-line log and **zer
# itpp-infrastructure — CHANGELOG # itpp-infrastructure — CHANGELOG
## 2026-09-15 (fifth) — failback could not actually stand the standby down
Found in the same sweep as the guard fix, in the watchdog's failback branch (Core healthy again after a takeover, so this box must go dormant):
```bash
pkill__ -f "hermes gateway" 2>/dev/null || true
echo "OK $(date +%s)" > "$HEALTH_STATE"
log "=== FAILBACK complete: $STANDBY_NAME dormant, Core primary ==="
```
The real service command line is `/usr/local/lib/hermes-agent/venv/bin/python -m hermes_cli.main hermes-gateway run`, which does **not** contain the phrase `hermes gateway`. The pattern matched nothing, so the branch logged a successful stand-down, wrote a healthy state, and left the service running - **two gateways on one bot token**, the exact split-brain this design exists to prevent. It was never exercised because core-bu has never taken over.
**Fix:** stand down by unit, then require proof before announcing anything.
```bash
XDG_RUNTIME_DIR=/run/user/0 systemctl --user stop hermes-gateway
sleep 3
FB_STATE=<is-active>; FB_PID=<MainPID>; FB_PROC=<pgrep the real signature>
if <state active> || <pid != 0> || <proc found>; then
<kill the real signature>; <re-check>
if still found -> log "FAILBACK FAILED", alert, exit 1
fi
```
The process check uses the exact signature and is deliberately biased toward false positives: a wrong "still running" produces a loud alert, whereas the old wrong "all clear" produced silent split-brain.
**Verified:** a decoy process carrying the real signature is detected as STILL SERVING (would alert and exit 1, not claim success); with no decoy the same logic reads dormant; the watchdog still runs clean end-to-end (exit 0, health state refreshed, unit inactive, still armed). Backup: `hermes-standby-watchdog.sh.bak4-20260915` on core-bu.
## 2026-09-15 (fourth) — the standby's "am I primary?" guard was matching the wrong thing ## 2026-09-15 (fourth) — the standby's "am I primary?" guard was matching the wrong thing
**Found while verifying the cron/output cleanup:** at 17:50:01Z the armed standby (core-bu) logged `This box is primary (gateway running), sync skipped to avoid clobbering live config` - while its gateway unit was `inactive/disabled` and MainPID was 0. The guard was: **Found while verifying the cron/output cleanup:** at 17:50:01Z the armed standby (core-bu) logged `This box is primary (gateway running), sync skipped to avoid clobbering live config` - while its gateway unit was `inactive/disabled` and MainPID was 0. The guard was: