ippadmin/itpp-infrastructure
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

core-bu armed, app1-bu disarmed (single-armed rule); app1-bu has no boot restore unit

Browse Source
This commit is contained in:
root
2026-09-15 13:06:49 -04:00
parent a6a32f6914
commit 128c7b34b6
2 changed files with 23 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+22 -1
View File
@@ -1,6 +1,27 @@
## 2026-09-15 (later) - core-bu ARMED, app1-bu disarmed
Single-armed rule enforced in the safe order: app1-bu disarmed FIRST, then core-bu armed, so there was never a window with two armed standbys.
- **app1-bu disarmed:** its two standby crons (`*/5` watchdog, `*/10` sync) commented with marker `#DISARMED-2026-09-15 (armed node is now core-bu; re-arm: remove this marker)`. Its unrelated 04:37 S3-restorability cron left intact.
- **core-bu armed:** `/root/.hermes/standby.disarmed` removed (disarm gate gone). Re-disarm by recreating that file; a copy is at `/root/standby.disarmed.rearm-backup`.
Verified after the swap, not assumed:
- core-bu real non-dry-run watchdog cycle probed Core: `HEALTH OK unit=active pid=100998 msgs=14262`, then stayed dormant. Zero gateway processes, user unit inactive/disabled.
- A real armed sync ran with the gate open and the standby scripts survived untouched (watchdog sha256 prefix unchanged, `63bea8b7557f634f`), so the clobber guard holds under live conditions, not just by reasoning.
- app1-bu: both standby crons commented, non-standby crons intact, no gateway process, unit inactive/disabled.
**Finding on app1-bu (recorded while disarming):** it has NO boot-time restore unit. `systemctl is-enabled hermes-standby.service` returns `not-found`, so a reboot of app1-bu would never have restored or failed over. Its two crons were the entire failover mechanism. Re-arming app1-bu requires installing that unit first.
**Trade-off accepted:** both DR nodes (Core and core-bu) are now netcup. Provider diversity is gone until the US replacement lands. app1-bu remains the only non-netcup node but is no longer armed.
**Operational note:** with core-bu armed, a Core Hermes outage lasting 2 consecutive 5-minute cycles (about 10 minutes) fences Core and triggers takeover. Planned gateway work expected to exceed about 10 minutes will cause a failover, then an automatic failback when Core returns. Re-disarm core-bu (`touch /root/.hermes/standby.disarmed`) before any long maintenance window.
---
# itpp-infrastructure — CHANGELOG
## 2026-09-15 — core-bu Warm Standby BUILT and proven (DISARMED; arming is a posture decision)
## 2026-09-15 — core-bu Warm Standby BUILT and proven (armed later the same day, see the entry above)
- **core-bu (`159.195.204.203`, netcup Nuremberg) is now a working Hermes warm standby.** Every step was executed on the box and verified by reading the result back.
- **Hermes parity, proven not assumed:** `/usr/local/lib/hermes-agent` mirrored from Core. core-bu reports `v0.21.0 (2026.8.31) · local 8dbf07e9 (+1 carried commit)`, Python `3.11.15`, method `git` — identical to Core. A real one-shot agent turn returned `STANDBY-SMOKE-OK`, so the box can serve, not just install.