ippadmin/itpp-infrastructure
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

core-bu warm standby built + proven (6 defects fixed); standby package build receipt (2026-09-15)

Browse Source
This commit is contained in:
root
2026-09-15 12:57:46 -04:00
parent b9b70f0f48
commit a6a32f6914
2 changed files with 95 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+24 -1
View File
@@ -1,6 +1,29 @@
# itpp-infrastructure — CHANGELOG
## 2026-09-15 — app4 + core-bu Provisioned (Nuremberg)
## 2026-09-15 — core-bu Warm Standby BUILT and proven (DISARMED; arming is a posture decision)
- **core-bu (`159.195.204.203`, netcup Nuremberg) is now a working Hermes warm standby.** Every step was executed on the box and verified by reading the result back.
- **Hermes parity, proven not assumed:** `/usr/local/lib/hermes-agent` mirrored from Core. core-bu reports `v0.21.0 (2026.8.31) · local 8dbf07e9 (+1 carried commit)`, Python `3.11.15`, method `git` — identical to Core. A real one-shot agent turn returned `STANDBY-SMOKE-OK`, so the box can serve, not just install.
- **State parity verified against the live box:** `state.db` **140,201,984 bytes on both**, `quick_check=ok`; `sessions` 151/151; cron `jobs.json` **89 jobs / 86 enabled on both**; 99 references; memories incl. `MEMORY.md`. The standby store sits one 11-minute checkpoint behind (14,222 vs 14,262 messages) by design — the failover path re-syncs before it serves.
- **Probe path proven:** core-bu -> Core over Tailscale (`100.71.155.7`) returns `core` / `active` non-interactively. core-bu joined the tailnet as `100.113.119.108` with `RunSSH=false`, because Tailscale SSH intercepts non-interactive key auth and would have broken the probe.
- **All three failover action paths exercised under DRYRUN (gate lifted for the test, restored after):**
- health-bad: `BAD <t> 1 <t>` -> `BAD <t> 2 <t+2>` -> `[dryrun] decision reached: would fence live box and take over (nothing done)`;
- host-down: `[dryrun] decision reached: host-down path would sync from S3 and start the standby (nothing done; reason: Live host not responding to pings)`;
- failback: `[dryrun] failback would notify and stop this box's gateway (nothing done)`, with a decoy process proving DRYRUN does not kill and the real kill command does.
- **Alert channels verified:** Telegram `getMe` -> `ok=True shonuff_is_a_bot`; SMTP login OK on port **2525**.
- **Six defects found and fixed during the build:**
1. **No outbound `itpp-infra` private key on core-bu** — the health probe could not reach Core at all. The provisioning record conflates the *inbound* `authorized_keys` entry with an outbound key. Installed; fingerprint `SHA256:Jxh0bbT9dUV3q1DYYB3hHyhy/1TDj7Q8U4xrVmB38uQ` (matches Core).
2. **`DRYRUN=1` was not a global no-op** (three separate action paths). The host-down path fell through to the S3 sync and the gateway start; the failback path notified and ran its kill. A "safe" dry-run against an unreachable Core would have started a **second poller on the same bot token**. All paths now guarded.
3. **The failover's final step could not have worked.** `hermes gateway start` requires an installed unit and exits 1 without one. Unit installed with `--no-start-now --no-start-on-login` (present, `disabled`, `inactive`, 0 processes).
4. **SMTP port was wrong for netcup.** Shipped `SMTP_PORT=587`; netcup blocks outbound 25/465/587 and 587 **times out** from core-bu, so email alerts would have failed silently forever. Now `2525`. app1-bu is on Hetzner and works on both, so it was left alone.
5. **The failover sync would clobber the standby's own scripts.** Core's `~/.hermes/scripts/` is inside the synced tree, so `live/scripts/` holds **app1-bu's** copies (4,429-byte watchdog vs core-bu's host-specific 19,852-byte one). A wholesale failover sync would have overwritten `PROBE_HOST` / `PROBE_SSH_KEY` / the disarm gate at the worst moment. Added `--exclude "scripts/hermes-standby-*"`.
6. **`app1-bu` — the previously armed standby — had no `/root/.config/systemd/user/` at all** while its watchdog also calls `hermes gateway start`. **Its failover could not complete its final step either.** Unit installed there too; its email path independently verified good on both ports.
- **Disarm gate verified three ways:** watchdog logs the notice once per day then exits 0; sync logs `DISARMED, sync skipped`; boot restore exits before starting anything.
- **Not armed, deliberately.** Arming core-bu requires disarming app1-bu first (the single-armed rule is enforced in code by `DISARM_FILE`), and it **trades provider diversity away**: Core and core-bu are both netcup, while app1-bu is the only non-netcup box. That is a posture decision for Germaine, not a build step. Stage C (live failover + failback drill, which fences Core's real gateway) also remains, and needs a scheduled window.
- **Bloat note:** the wholesale pull brought 2.9 GB of Core's `docker/` and 1.1 GB of `data/` that core-bu does not run; left in place (472 GB free) but the failover sync should probably exclude them.
- Full build receipt and evidence: `docs/infrastructure/core-bu-standby-package-2026-09-15.md` section 9.
2026-09-15 — app4 + core-bu Provisioned (Nuremberg)
- **Renamed by Germaine (recorded at rename time per the changelog mandate):** the two netcup Nuremberg boxes ordered 2026-09-14 are now **app4** and **core-bu**.
- **core-bu** = `v2202609377162521279.megasrv.de`, `159.195.204.203/22`, IPv6 `2a0a:4cc0:c2:b3e0:9409:42ff:fe3b:5397` — netcup **RS 2000 G12** (8 vCPU / 15 GB / 503 GB), exact twin of Core. Role: **Core's warm standby**.