core-bu armed, app1-bu disarmed (single-armed rule); app1-bu has no boot restore unit

2026-09-15 13:06:49 -04:00
parent a6a32f6914
commit 128c7b34b6
2 changed files with 23 additions and 2 deletions
@@ -1,6 +1,27 @@
+## 2026-09-15 (later) - core-bu ARMED, app1-bu disarmed
+
+Single-armed rule enforced in the safe order: app1-bu disarmed FIRST, then core-bu armed, so there was never a window with two armed standbys.
+
+- **app1-bu disarmed:** its two standby crons (`*/5` watchdog, `*/10` sync) commented with marker `#DISARMED-2026-09-15 (armed node is now core-bu; re-arm: remove this marker)`. Its unrelated 04:37 S3-restorability cron left intact.
+- **core-bu armed:** `/root/.hermes/standby.disarmed` removed (disarm gate gone). Re-disarm by recreating that file; a copy is at `/root/standby.disarmed.rearm-backup`.
+
+Verified after the swap, not assumed:
+
+- core-bu real non-dry-run watchdog cycle probed Core: `HEALTH OK unit=active pid=100998 msgs=14262`, then stayed dormant. Zero gateway processes, user unit inactive/disabled.
+- A real armed sync ran with the gate open and the standby scripts survived untouched (watchdog sha256 prefix unchanged, `63bea8b7557f634f`), so the clobber guard holds under live conditions, not just by reasoning.
+- app1-bu: both standby crons commented, non-standby crons intact, no gateway process, unit inactive/disabled.
+
+**Finding on app1-bu (recorded while disarming):** it has NO boot-time restore unit. `systemctl is-enabled hermes-standby.service` returns `not-found`, so a reboot of app1-bu would never have restored or failed over. Its two crons were the entire failover mechanism. Re-arming app1-bu requires installing that unit first.
+
+**Trade-off accepted:** both DR nodes (Core and core-bu) are now netcup. Provider diversity is gone until the US replacement lands. app1-bu remains the only non-netcup node but is no longer armed.
+
+**Operational note:** with core-bu armed, a Core Hermes outage lasting 2 consecutive 5-minute cycles (about 10 minutes) fences Core and triggers takeover. Planned gateway work expected to exceed about 10 minutes will cause a failover, then an automatic failback when Core returns. Re-disarm core-bu (`touch /root/.hermes/standby.disarmed`) before any long maintenance window.
+
+---
+
 # itpp-infrastructure — CHANGELOG

-## 2026-09-15 — core-bu Warm Standby BUILT and proven (DISARMED; arming is a posture decision)
+## 2026-09-15 — core-bu Warm Standby BUILT and proven (armed later the same day, see the entry above)

 - **core-bu (`159.195.204.203`, netcup Nuremberg) is now a working Hermes warm standby.** Every step was executed on the box and verified by reading the result back.
 - **Hermes parity, proven not assumed:** `/usr/local/lib/hermes-agent` mirrored from Core. core-bu reports `v0.21.0 (2026.8.31) · local 8dbf07e9 (+1 carried commit)`, Python `3.11.15`, method `git` — identical to Core. A real one-shot agent turn returned `STANDBY-SMOKE-OK`, so the box can serve, not just install.
@@ -483,7 +483,7 @@ modified anywhere. No production process was started, stopped, or restarted.

 ## 9. Build receipt — executed 2026-09-15 (Sho'Nuff)

-**Status: BUILT + PROVEN + DISARMED (armed-ready).** Every step below was executed on
+**Status: BUILT + PROVEN + ARMED (2026-09-15). See the CHANGELOG entry "2026-09-15 (later) - core-bu ARMED, app1-bu disarmed" for the swap and its verification.** Every step below was executed on
 `core-bu` and verified by reading the result back, not by assuming the step worked.

 ### 9.1 What was actually done