core-bu warm standby built + proven (6 defects fixed); standby package build receipt (2026-09-15)

2026-09-15 12:57:46 -04:00
parent b9b70f0f48
commit a6a32f6914
2 changed files with 95 additions and 1 deletions
@@ -478,3 +478,74 @@ retired per the P3 checklist in `reference-update-matrix-2026-09-15.md` (Option

 Nothing was installed on core-bu, app1-bu, or Core. No systemd unit was created or
 modified anywhere. No production process was started, stopped, or restarted.
+
+---
+
+## 9. Build receipt — executed 2026-09-15 (Sho'Nuff)
+
+**Status: BUILT + PROVEN + DISARMED (armed-ready).** Every step below was executed on
+`core-bu` and verified by reading the result back, not by assuming the step worked.
+
+### 9.1 What was actually done
+
+| Step | Evidence |
+|---|---|
+| Tailscale joined | node `core-bu` = `100.113.119.108`; `RunSSH=false` (Tailscale SSH intercepts non-interactive key auth) |
+| Outbound SSH key installed | `/root/.ssh/itpp-infra`, fingerprint `SHA256:Jxh0bbT9dUV3q1DYYB3hHyhy/1TDj7Q8U4xrVmB38uQ` (identical to Core) |
+| Probe path proven | core-bu -> Core over tailnet returns `core` / `active` non-interactively |
+| Hermes installed | `/usr/local/lib/hermes-agent` mirrored from Core; `v0.21.0 (2026.8.31) · local 8dbf07e9 (+1 carried commit)`, Python `3.11.15`, install method `git` |
+| Functional smoke test | real one-shot turn returned `STANDBY-SMOKE-OK` (session `20260915_123810_6d5834`) |
+| Config/state staged | `.env` + `config.yaml` present (`600`); Telegram token sha256 matches Core (`ba939adaf8354e72...`) |
+| Scripts staged | watchdog / sync / restore at `/root/.hermes/scripts/`, mode `700`, hashes matched source |
+| Gateway unit | `/root/.config/systemd/user/hermes gateway.service` — `enabled=disabled`, `active=inactive`, 0 processes |
+| Crons | `*/5` watchdog + `*/10` sync; pre-existing `root-essentials-backup` preserved |
+| Boot unit | `hermes-standby.service` enabled (inert while disarmed) |
+| Linger | `Linger=yes` |
+| Alert channels | Telegram `getMe` -> `ok=True shonuff_is_a_bot`; SMTP login OK on port **2525** |
+
+### 9.2 Proof of the state machine (dry-run, zero production risk)
+
+- Forced definitive-bad twice: `DEGRADED (1/2, definitive=1)` -> `DEGRADED (2/2, definitive=1)`
+  -> `[dryrun] decision reached: would fence live box and take over (nothing done)`.
+- State file carried all three fields (`BAD <first> <count> <last>`) — the "missing field
+  resets the counter forever" trap is absent.
+- Disarm gate proven three ways: watchdog logs the notice once per day then exits 0; sync logs
+  `DISARMED, sync skipped`; boot restore exits before starting anything.
+
+### 9.3 Defects found during the build (all fixed and re-verified)
+
+1. **No outbound `itpp-infra` private key on core-bu.** The health probe could not reach Core
+   at all (`Permission denied (publickey)`). Section 1's provisioning claim covers the
+   *inbound* `authorized_keys` entry, not an outbound key — the two were conflated.
+2. **`DRYRUN=1` was not a global no-op.** The health-bad path returned early, but the host-down
+   path fell through to the S3 sync and the gateway start. A "safe" dry-run test against an
+   unreachable Core would have started a **second poller on the same bot token**. Fixed with a
+   guard on the action block; proven by forcing the host-down path to its dry-run stop.
+3. **The failover's final step could not have worked.** `hermes gateway start` requires an installed unit
+   (`_require_service_installed` exits 1 otherwise), and core-bu had none.
+4. **SMTP port was wrong for netcup.** The watchdog shipped `SMTP_PORT=587`; netcup blocks
+   outbound 25/465/587 and 587 **times out** from core-bu, so every email alert would have
+   failed silently. Changed to `2525` (verified: `LOGIN OK (starttls=True)`). app1-bu is on
+   Hetzner and works on both ports, so it was left alone.
+5. **The failover sync would clobber the standby's own scripts.** Core's `~/.hermes/scripts/`
+   sits inside the synced tree, so `live/scripts/` contains **app1-bu's** copies
+   (a 4,429-byte watchdog vs core-bu's host-specific 18 KB one). A wholesale failover sync
+   would have overwritten core-bu's `PROBE_HOST` / `PROBE_SSH_KEY` / disarm gate at the worst
+   possible moment. Added `--exclude "scripts/hermes-standby-*"` to the failover sync.
+
+### 9.4 Same defect class on the previously armed standby (FIXED 2026-09-15)
+
+`app1-bu` had **no** `/root/.config/systemd/user/` directory at all, while its watchdog calls
+`hermes gateway start`. Its failover therefore could not complete its final step either. Unit installed there
+with `--no-start-now --no-start-on-login`; now `enabled=disabled`, `active=inactive`, 0
+processes. Its email path was independently verified good (both 587 and 2525).
+
+### 9.5 Not done, deliberately
+
+- **Arming is not done.** Arming core-bu requires disarming app1-bu first (the single-armed
+  rule is enforced in code by `DISARM_FILE`), and it trades away **provider diversity**:
+  Core and core-bu are both netcup, while app1-bu is the only non-netcup box. That is a
+  posture decision, not a build step.
+- **Stage C (live failover + failback drill) is not done.** It fences Core's real gateway,
+  so it needs a scheduled window and explicit approval.
+- Vaultwarden entry and `key-inventory.md` update still pending.