From 128c7b34b694a91d5e37915d85ead2936a35142e Mon Sep 17 00:00:00 2001 From: root Date: Tue, 15 Sep 2026 13:06:49 -0400 Subject: [PATCH] core-bu armed, app1-bu disarmed (single-armed rule); app1-bu has no boot restore unit --- CHANGELOG.md | 23 ++++++++++++++++++- .../core-bu-standby-package-2026-09-15.md | 2 +- 2 files changed, 23 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a0d040a..44d46d3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,27 @@ +## 2026-09-15 (later) - core-bu ARMED, app1-bu disarmed + +Single-armed rule enforced in the safe order: app1-bu disarmed FIRST, then core-bu armed, so there was never a window with two armed standbys. + +- **app1-bu disarmed:** its two standby crons (`*/5` watchdog, `*/10` sync) commented with marker `#DISARMED-2026-09-15 (armed node is now core-bu; re-arm: remove this marker)`. Its unrelated 04:37 S3-restorability cron left intact. +- **core-bu armed:** `/root/.hermes/standby.disarmed` removed (disarm gate gone). Re-disarm by recreating that file; a copy is at `/root/standby.disarmed.rearm-backup`. + +Verified after the swap, not assumed: + +- core-bu real non-dry-run watchdog cycle probed Core: `HEALTH OK unit=active pid=100998 msgs=14262`, then stayed dormant. Zero gateway processes, user unit inactive/disabled. +- A real armed sync ran with the gate open and the standby scripts survived untouched (watchdog sha256 prefix unchanged, `63bea8b7557f634f`), so the clobber guard holds under live conditions, not just by reasoning. +- app1-bu: both standby crons commented, non-standby crons intact, no gateway process, unit inactive/disabled. + +**Finding on app1-bu (recorded while disarming):** it has NO boot-time restore unit. `systemctl is-enabled hermes-standby.service` returns `not-found`, so a reboot of app1-bu would never have restored or failed over. Its two crons were the entire failover mechanism. Re-arming app1-bu requires installing that unit first. + +**Trade-off accepted:** both DR nodes (Core and core-bu) are now netcup. Provider diversity is gone until the US replacement lands. app1-bu remains the only non-netcup node but is no longer armed. + +**Operational note:** with core-bu armed, a Core Hermes outage lasting 2 consecutive 5-minute cycles (about 10 minutes) fences Core and triggers takeover. Planned gateway work expected to exceed about 10 minutes will cause a failover, then an automatic failback when Core returns. Re-disarm core-bu (`touch /root/.hermes/standby.disarmed`) before any long maintenance window. + +--- + # itpp-infrastructure — CHANGELOG -## 2026-09-15 — core-bu Warm Standby BUILT and proven (DISARMED; arming is a posture decision) +## 2026-09-15 — core-bu Warm Standby BUILT and proven (armed later the same day, see the entry above) - **core-bu (`159.195.204.203`, netcup Nuremberg) is now a working Hermes warm standby.** Every step was executed on the box and verified by reading the result back. - **Hermes parity, proven not assumed:** `/usr/local/lib/hermes-agent` mirrored from Core. core-bu reports `v0.21.0 (2026.8.31) · local 8dbf07e9 (+1 carried commit)`, Python `3.11.15`, method `git` — identical to Core. A real one-shot agent turn returned `STANDBY-SMOKE-OK`, so the box can serve, not just install. diff --git a/docs/infrastructure/core-bu-standby-package-2026-09-15.md b/docs/infrastructure/core-bu-standby-package-2026-09-15.md index 4110602..46e995c 100644 --- a/docs/infrastructure/core-bu-standby-package-2026-09-15.md +++ b/docs/infrastructure/core-bu-standby-package-2026-09-15.md @@ -483,7 +483,7 @@ modified anywhere. No production process was started, stopped, or restarted. ## 9. Build receipt — executed 2026-09-15 (Sho'Nuff) -**Status: BUILT + PROVEN + DISARMED (armed-ready).** Every step below was executed on +**Status: BUILT + PROVEN + ARMED (2026-09-15). See the CHANGELOG entry "2026-09-15 (later) - core-bu ARMED, app1-bu disarmed" for the swap and its verification.** Every step below was executed on `core-bu` and verified by reading the result back, not by assuming the step worked. ### 9.1 What was actually done