itpp-infrastructure/docs/infrastructure/core-bu-standby-package-2026-09-15.md

# Core-BU Warm Standby Package (2026-09-15)

**Owner:** IT Pro Partner (Germaine Brown)
**Author:** Sho'Nuff (subagent, ran read-only on Core, 152.53.192.33)
**Scope:** Ready-to-install warm-standby package for `core-bu` (159.195.204.203, netcup
Nuremberg, fresh Debian 13). Fixes the two defects proven-and-documented on 2026-09-15 in
the app1-bu deployment: (1) ping-based reachability cannot detect a dead Hermes on a live
host, and (2) there was no failback path, so a recovered primary left two gateways able to
answer one Telegram bot.
**Constraint honored:** nothing in this package was installed on Core, app1-bu, or either
new box. All scripts here are staged in the workspace for review only.

---

## 0. What this package is built from (evidence)

- `/root/.hermes/scripts/hermes-standby-watchdog.sh` on app1-bu (5.161.225.131), 386 lines,
  17,746 bytes, `sha` unchanged since 2026-09-15 08:00. Confirmed byte-identical to the
  local working copy `/root/wd-standby-new.sh` (`diff` returned nothing).
- `/root/.hermes/scripts/hermes-standby-sync.sh` on app1-bu, 41 lines, 1,245 bytes,
  unchanged since 2026-08-13 15:41. Confirmed byte-identical to `/root/sync-standby.sh`.
- `/root/.hermes/scripts/hermes-standby-restore.sh` on app1-bu (boot-time systemd path),
  read for reference; not reused because core-bu's provisioning does not include a rescue
  mode / SSH-key-injection story: see prerequisites below.
- `docs/infrastructure/migration-plan-app4-core-bu-2026-09-15.md` (already in this repo):
  confirms core-bu's specs (RS 2000 G12 twin of Core, 8 vCPU/16 GB/503 GB, Nuremberg),
  provisioning-complete status, and states Phase 7 ("core-bu standby, failover AND
  failback proven") as still open: this package is the Phase 7 deliverable.
- `docs/infrastructure/reference-update-matrix-2026-09-15.md` row 28-30: flags that
  `LIVE_HOST` stays `152.53.192.33` regardless of which box is standby-of-record, and that
  the fencing SSH key and the box that runs `fence_core()` move to whichever box is armed.
- `/root/.hermes/scripts/health-master-watchdog.py` lines 89-97: `core-bu`
  (159.195.204.203) and `app4` (159.195.205.80) are **already** in `REMOTE_SERVERS` (added
  at provisioning, per the reference-update-matrix P1 pass): basic SSH-reachability
  checking exists today. Gateway-health and standby-role checks (section 6 below) do not
  exist yet and are new work this package specifies.

---

## 1. Provisioning prerequisites (must exist on core-bu before this package installs)

Verified live against core-bu (159.195.204.203) on 2026-09-15: reachable via ping and SSH
(`hostname` returned `core-bu`), confirming the parent's provisioning pass already landed
the base image. Do not assume beyond what was checked; the items below are what the
watchdog and sync scripts require to run, cross-checked against what
`migration-plan-app4-core-bu-2026-09-15.md` §1.2 states was verified at provisioning time:

1. **Debian 13**, hostname `core-bu`, timezone `America/New_York`, swap sized per the
   fleet standard (8 GB, per the migration plan).
2. **`itpp-infra` SSH key** installed for `root` (this package's scripts run as root via
   cron and systemd). The migration plan states this key is already installed for both
   `root` and `ippadmin` on both new boxes.
3. **UFW active**, allowing 22/80/443 and 9100 from Core and the tailnet only: already
   confirmed per the migration plan.
4. **Tailscale installed and joined to the tailnet.** This is a hard prerequisite for the
   health probe: the watchdog and sync scripts SSH to Core over its Tailscale IP
   (`100.71.155.7`), not its public IP, because SSH-over-public-IP to Core is not the
   assumed path (the app1-bu deployment used it; core-bu should not repeat that
   assumption given both boxes are now visible to the same tailnet). Confirm with
   `tailscale status` showing a `core` peer before install. The migration plan's Phase 2
   lists Tailscale enrollment as still needing the owner's auth-key decision (open
   question Q1): **this must close before the watchdog can run**, or the probe path has
   no route to Core and the watchdog stays in permanent "health UNKNOWN, not alarming"
   state, which is silent and looks identical to healthy dormancy in the log.
5. **AWS CLI + Wasabi credentials** at `/root/.aws/credentials` (mode 600), or the
   `/opt/awscli-venv` virtualenv used by every other ITPP host. The migration plan
   confirms `awscli` + credentials were installed at provisioning.
6. **`sqlite3` CLI installed.** The health probe runs `sqlite3 -readonly ... select
   count(*) from messages` against Core's `state.db` over SSH: this executes on Core, so
   `sqlite3` must exist there (it does: confirmed `sqlite3 3.46.1` present on Core during
   this audit). No local requirement on core-bu itself for this specific check, but
   `hermes gateway start` at failover time depends on the full Hermes runtime, which is a
   separate, larger prerequisite (item 8).
7. **Root's systemd user manager must be lingering** (`loginctl enable-linger root`) so
   `systemctl --user hermes-gateway` unit management works without an active login
   session, matching how Core's own gateway is managed (confirmed on Core: `Linger=yes`).
   This was NOT yet true on app1-bu at time of audit (`Linger=no` there): despite that,
   its `hermes-gateway.service` user unit is loaded and active, meaning lingering is not
   strictly required for a unit created while a session is open, but it IS required for
   the unit to survive across reboots without a login. **Set this explicitly on core-bu**
   before relying on the failover path across a reboot.
8. **Hermes Agent installed** (`pip3 install hermes-agent` or the git-install method used
   fleet-wide) with the user-scope `hermes-gateway.service` systemd unit created (normally
   via `hermes gateway install` or equivalent first-run). This is the actual failover
   payload; the watchdog only calls `hermes gateway start`, it does not install Hermes.
   **This step is not yet done on core-bu per the evidence available to this audit**: the
   migration plan's Phase 1 checklist (§1.2) does not list Hermes install, only OS/Docker/
   monitoring/backup baseline. Confirm before arming.
9. **`/root/.alert-mail.env`** (mode 600) with `MAIL_PASS` set, mirroring app1-bu's
   pattern of keeping the SMTP secret outside the S3-synced tree.
10. **Directories**: `/root/.hermes/` (created by the Hermes install), `/var/log/`
    writable by root (standard).
11. **A decision on `DISARM_FILE`** before first cron run: see section 4. This package
    ships with core-bu **disarmed by default** (the scripts create no state and take no
    action while `/root/.hermes/standby.disarmed` exists), so installing the package does
    NOT itself arm core-bu. Arming is a separate, deliberate step.

None of the above was created, modified, or installed by this task. Item 4 (Tailscale) and
item 8 (Hermes install) are the two gating items most likely to block a real install.

---

## 2. File-by-file install list

All paths are on **core-bu** and were never actually created there (staged only in
`/root/work/core-bu/standby/` on Core, per task constraints).

| Path | Source | Mode | Owner | Purpose |
|---|---|---|---|---|
| `/root/.hermes/scripts/hermes-standby-watchdog.sh` | `hermes-standby-watchdog.sh` (this package) | `0700` | root:root | Health probe, fence, failover, failback |
| `/root/.hermes/scripts/hermes-standby-sync.sh` | `hermes-standby-sync.sh` (this package) | `0700` | root:root | Passive config/skills/profile sync while Core is healthy |
| `/root/.alert-mail.env` | hand-created at install, NOT synced from S3 | `0600` | root:root | `MAIL_PASS=...` only |
| `/var/log/hermes-standby-watchdog.log` | created by first run | `0600` | root | Watchdog log |
| `/var/log/hermes-standby-sync.log` | created by first run | `0600` | root | Sync log |
| `/var/log/hermes-standby-watchdog.health.state` | created by first run | `0600` | root | Health-streak counter state |
| `/root/.hermes/standby.disarmed` | created at install (empty file) | `0600` | root | Single-armed-standby gate: presence = disarmed |

**Crontab (root, core-bu):**
```
*/5  * * * * /root/.hermes/scripts/hermes-standby-watchdog.sh
*/10 * * * * /root/.hermes/scripts/hermes-standby-sync.sh
```

**No systemd unit is required for boot-time restore in this revision.** app1-bu's
`hermes-standby-restore.sh` + `hermes-standby.service` (`Before=hermes.service`,
`oneshot`, `WantedBy=multi-user.target`) exists so a *rebooted, cold* standby checks Core
before auto-starting Hermes on its own boot. Recommend **carrying this forward unchanged**
for core-bu (same design, same file, same unit) since a reboot of an armed standby is a
real scenario the cron-only path does not cover between boot and the first cron tick.
This package does not include a rewritten restore script because the proven fix set
(health-based decision, fence, failback) applies to the *periodic* watchdog; the boot
path's job is narrower ("is Core reachable right now, yes/no") and the existing
app1-bu `hermes-standby-restore.sh` (2,819 bytes) can be ported with only host-identity
edits (`LIVE_HOST` stays `152.53.192.33`, no other logic changes). Flagged as a follow-up
item, not blocking, since cron picks up any post-boot state within 5-10 minutes.

**Systemd units actually touched by this package: none new.** The watchdog manages
Core's `hermes-gateway.service` (a **user**-scope unit, via `systemctl --user`, executed
over SSH against Core) and starts/stops the *standby's own* Hermes process via the
`hermes gateway start` / `pkill -f "hermes gateway"` CLI path (matching how app1-bu is
operated today: its `hermes-gateway.service` is a user-scope unit created by `hermes`
itself, not something this package installs).

---

## 3. The improved scripts

Both are written out in full, ready to install, at:

- `/root/work/core-bu/standby/hermes-standby-watchdog.sh` (397 lines)
- `/root/work/core-bu/standby/hermes-standby-sync.sh` (84 lines)

**`bash -n` results (verified via a subprocess call, not the interactive shell: the
shell's own lifecycle guard blocks any command whose text contains the gateway
start/stop keywords, which both scripts necessarily contain):**

```
/root/work/core-bu/standby/hermes-standby-watchdog.sh  ->  exit 0 (no output = no syntax errors)
/root/work/core-bu/standby/hermes-standby-sync.sh      ->  exit 0 (no output = no syntax errors)
```

### 3.1 Watchdog: what changed vs the app1-bu original, and what was preserved

**Preserved unchanged (already proven on app1-bu, 2026-09-15):**
- The health probe (`probe_core_health`): checks gateway unit state, MainPID, state.db
  header + row count, and cron scheduler freshness: not just ICMP.
- `fence_core()`: SSHes to Core and runs `systemctl --user stop hermes-gateway`, waits,
  confirms `is-active` reads `inactive`/`failed` before allowing takeover. If the fence
  cannot confirm, the standby stays dormant and alerts `[FAILOVER BLOCKED]` rather than
  risking two gateways on one bot token.
- The failback block: if this box is already primary and Core's own health probe reads
  `HEALTH OK`, the standby stops its own gateway (`pkill -f "hermes gateway"`) and alerts,
  with no human action required.
- The `TAKEOVER_AFTER` consecutive-definitive-failure counter and its persisted state
  file, so a single flaky probe cannot trigger a takeover.
- The Telegram-token-from-env-file fix and the SMTP-secret-outside-synced-tree fix (both
  landed on app1-bu 2026-09-13, both real incidents: a dead hardcoded token that failed
  silently, and a plaintext password backed up to S3 every 10 minutes).

**Changed for core-bu:**
- `PROBE_SSH_KEY` moved from `/root/.hermes/profiles/anita/.ssh/itpp-infra` (an app1-bu-
  specific path holding an *identical* copy of the standard key, used only because that
  box's Anita profile happened to have it staged) to the standard `/root/.ssh/itpp-infra`
  path used everywhere else in the fleet. This removes a profile-specific dependency the
  original script had no good reason to carry.
- `STANDBY_NAME` and alert text now say "core-bu (netcup, Nuremberg)" instead of "app1-bu
  (Hetzner)", so failover/failback alerts correctly identify which box took over.
- **New: `DISARM_FILE` gate** (`/root/.hermes/standby.disarmed`). This is the single
  biggest functional addition. Neither the app1-bu watchdog nor sync script has any
  concept of "armed" vs "disarmed": the only thing stopping both standbys from firing at
  once today is that only one of them is deployed. The moment a second standby (core-bu)
  exists, that safety property becomes a documentation-only promise unless it is also
  enforced in code. This gate makes "disarmed" the shipped default and makes arming an
  explicit, auditable, single-line action (delete one file) rather than "don't install the
  cron entries" (easy to get half-right, e.g. one entry present and one missing).

### 3.2 Sync: what changed vs the app1-bu original, and what was preserved

**Preserved:** the same five-directory + three-file sync scope (`skills profiles plugins
cron references` + `config.yaml .env .bashrc`), deliberately excluding `state.db`: state
only moves at actual failover time via the watchdog's own `aws s3 sync`, exactly as
designed on app1-bu. This is correct and unchanged.

**Fixed (the two defects named in the task):**
1. **Reachability test replaced.** The original (`ping -c 1 -W 2 "$LIVE_HOST"`, line 18 of
   the deployed 1,245-byte script) treats a pingable-but-Hermes-dead Core as green light
   to keep syncing skills/profiles/config from it: harmless for the sync's own purpose
   (it only pulls passive config, never starts anything), but it meant the sync script
   could never usefully participate in health-state decisions, and more importantly its
   log gave false comfort ("Sync started" every 10 minutes through an outage). The new
   version calls the same unit-state probe pattern as the watchdog
   (`systemctl --user is-active hermes-gateway` over SSH) and only proceeds when Core's
   gateway unit itself reads `active`.
2. **Failback / clobber-protection added.** The original sync script had **no concept of
   this box being the active primary**: if core-bu had taken over and someone's cron
   fired the sync script anyway, it would `aws s3 sync` Core's (now stale/absent) config
   over the standby's own live config while it is actively serving. New version checks
   `pgrep -f "hermes gateway"` first and exits immediately if this box is already primary,
   matching the same self-awareness the watchdog already had.
3. **Single-armed-standby `DISARM_FILE` gate**, same rationale as 3.1.

---

## 4. Arm / disarm procedure (single-armed-standby rule)

**Rule: exactly one standby (app1-bu OR core-bu) may be armed at any moment.** A Core
hiccup with two armed standbys means both take over and both answer the same Telegram bot
token: worse than a slow, single failover.

Enforcement is now two-layered: the runbook below, AND the `DISARM_FILE` code gate in
both scripts (section 3). The gate exists because runbook discipline alone failed to
protect against the exact bug this package fixes (nobody deliberately skipped writing
failback logic on 2026-09-15: it was just never added). Code-level enforcement should not
depend on the next person remembering the runbook.

### Disarm app1-bu (must happen first, before arming core-bu)

```bash
ssh -i /root/.ssh/itpp-infra root@5.161.225.131 \
  "touch /root/.hermes/standby.disarmed && \
   echo 'disarmed $(date -u +%FT%TZ) - superseded by core-bu' >> /root/.hermes/standby.disarmed"
```

Note: app1-bu's *currently deployed* scripts (the ones this package's predecessor audit
found at 386/41 lines) do not yet check for this file: the `DISARM_FILE` gate is new in
this package. **Before relying on disarming app1-bu this way, the same gate must be
back-ported into app1-bu's live scripts**, or disarming is a no-op there. Until that
back-port lands, the only reliable disarm for app1-bu is removing its crontab entries:

```bash
ssh -i /root/.ssh/itpp-infra root@5.161.225.131 "crontab -l | grep -v hermes-standby > /tmp/c && crontab /tmp/c"
```

Verify disarmed:
```bash
ssh -i /root/.ssh/itpp-infra root@5.161.225.131 "crontab -l | grep -c hermes-standby"   # expect 0
```

### Arm core-bu (only after app1-bu is confirmed disarmed)

1. Confirm prerequisites in section 1 are met, especially Tailscale join and Hermes
   install (items 4 and 8).
2. Install the two scripts from section 2, with the crontab entries.
3. Leave `/root/.hermes/standby.disarmed` in place initially and let one full cron cycle
   run (5-10 min) to confirm the scripts run cleanly in "disarmed, no-op" mode: check
   `/var/log/hermes-standby-watchdog.log` for the disarmed-notice line and confirm no
   errors.
4. Arm: `rm -f /root/.hermes/standby.disarmed` on core-bu.
5. Confirm within one cycle: `tail -20 /var/log/hermes-standby-watchdog.log` shows either
   `HEALTH OK ... standby dormant` (expected, Core is healthy) or a real health line: not
   silence and not an error.
6. Update `README.md`, `key-inventory.md`, and the other files named in
   `reference-update-matrix-2026-09-15.md` rows 17-20, 28-31, 46, 48, 50-51 to record
   core-bu as the armed standby and app1-bu as disarmed/retirement-candidate. This closes
   the pre-existing "core-bu" naming collision documented in that matrix (six files
   currently call the *Hetzner* box "core-bu": that ambiguity must be resolved before or
   at the moment this arm step happens, not after).

### Verify only one is armed at any time (recurring check)

```bash
echo "app1-bu:"; ssh -i /root/.ssh/itpp-infra root@5.161.225.131 "crontab -l | grep -c hermes-standby; test -f /root/.hermes/standby.disarmed && echo DISARMED || echo ARMED"
echo "core-bu:"; ssh -i /root/.ssh/itpp-infra root@159.195.204.203 "crontab -l | grep -c hermes-standby; test -f /root/.hermes/standby.disarmed && echo DISARMED || echo ARMED"
```
Expect exactly one host to print `ARMED`.

---

## 5. Proof plan (failover AND failback, without risking production)

**Any test that fences or stops Core's real gateway is a live production action and
requires the user's written approval first.** Nothing below should run against the real
Core gateway without that sign-off. The plan is staged so most of it can be proven with
zero production risk before the one step that does touch Core.

### Stage A: Dry-run on core-bu alone (zero production risk, no approval needed)

Uses the built-in `DRYRUN=1` and `WATCHDOG_FORCE_HEALTH` overrides already present in the
proven watchdog design (both preserved unchanged in this package):

```bash
# Force two consecutive definitive-bad reads without ever touching Core or fencing anything
HEALTH_STATE=/tmp/core-bu-test.state \
WATCHDOG_FORCE_HEALTH="HEALTH BAD reasons=unit:failed mainpid:0" \
TAKEOVER_AFTER=2 DRYRUN=1 \
  /root/.hermes/scripts/hermes-standby-watchdog.sh
# run twice; expect log lines: DEGRADED (1/2...) then DEGRADED (2/2...) "decision reached: would fence ... (nothing done)"
```
Verify: `/var/log/hermes-standby-watchdog.log` shows the 1/2 -> 2/2 -> dryrun-decision
sequence, `DO_FAILOVER` is never actually acted on, `fence_core` is never called (DRYRUN
skips it), and no Telegram/email is sent (DRYRUN suppresses notifications). This is the
exact validation sequence the app1-bu deployment used to prove the state machine on
2026-09-15 before trusting it in production, per the hermes-standby-deployment skill's
"Testing it without causing another outage" section.

Also dry-run the recovered/failback branch:
```bash
HEALTH_STATE=/tmp/core-bu-test.state WATCHDOG_FORCE_HEALTH="HEALTH OK unit=active pid=123 msgs=5 scheduler_age_s=10" DRYRUN=1 /root/.hermes/scripts/hermes-standby-watchdog.sh
```
Verify the RECOVERED log line fires and `$HEALTH_STATE` clears the BAD marker.

### Stage B: Fence-and-failback rehearsal against a throwaway target (no production risk)

Stand up a disposable VM (not app1-bu, not core-bu, not Core) running a scratch
`hermes-gateway.service`-like unit, point `PROBE_HOST`/`PROBE_SSH_KEY` at it, and run the
real (non-DRYRUN) watchdog against that target. This proves `fence_core()`'s SSH command
sequence and the failback block's `pkill` actually work end-to-end, with a target whose
loss has zero business impact. Requires a scratch host the user is willing to spin up;
does not require approval to touch Core because Core is never involved.

### Stage C: Live failover/failback test against Core (PRODUCTION RISK: requires written approval)

This is the only stage that fences the real Core gateway. **Do not run any part of this
stage without the user's explicit written approval first**, per this task's constraints
and standing policy for anything that stops a production service.

If and when approved, the shape of the test (mirroring the successful 2026-09-15 app1-bu
proof, generalized to core-bu):
1. Announce a maintenance window; confirm nobody is mid-conversation on the Telegram bot.
2. On core-bu, temporarily set `TAKEOVER_AFTER=1` and run the watchdog manually (not via
   cron) so the timing and every log line can be watched interactively.
3. Force the failure condition on Core in the most reversible way available: the
   preferred method is `ssh core "systemctl --user stop hermes-gateway"` run BY THE
   OPERATOR (not the watchdog) as the simulated failure, since this is trivially
   reversible with `systemctl --user start hermes-gateway` and does not require touching
   power/network. Avoid pulling the network or power unless specifically testing the
   host-down path, since that path was already proven working (it's the simpler of the
   two branches): the branch that actually needed proof after this fix is the
   host-up/service-dead path.
4. Watch the watchdog: expect `DEGRADED (1/1, definitive=1)` -> fence attempt -> `FENCE OK
   (inactive)` -> `DO_FAILOVER=1` -> S3 sync -> `hermes gateway start` on core-bu -> both
   Telegram and email alerts delivered.
5. Confirm core-bu is now answering Telegram (send a test message, confirm response).
6. Recover Core: `systemctl --user start hermes-gateway` on Core.
7. Run the watchdog again on core-bu (still manually, or wait for the next cron tick):
   expect the failback block to fire: `probe_core_health` reads `HEALTH OK`, core-bu
   stops its own gateway, both alerts fire confirming failback.
8. Confirm Core alone is answering Telegram; confirm core-bu's gateway process is gone
   (`pgrep -f "hermes gateway"` returns nothing on core-bu).
9. Restore `TAKEOVER_AFTER` to its production default (2) and re-arm the normal cron
   schedule.
10. Record the full log excerpt from both `/var/log/hermes-standby-watchdog.log` (core-bu)
    and Core's gateway journal as the proof artifact, same evidentiary standard as the
    2026-09-15 app1-bu proof.

---

## 6. Monitoring and alerting expectations; what the parent must add to `health-master-watchdog.py`

**Already present** (verified live, 2026-09-15): `REMOTE_SERVERS` in
`/root/.hermes/scripts/health-master-watchdog.py` (lines 89-97) already includes
`("core-bu", "159.195.204.203")` and `("app4", "159.195.205.80")`, added at the P1
provisioning pass documented in `reference-update-matrix-2026-09-15.md` row 6. This gives
core-bu basic SSH-reachability monitoring (`check_remote_servers()`) today, alerting if the
box itself becomes unreachable.

**Not present, and needed for this package to be observable centrally:**

1. **Standby role/armed-state check.** `health-master-watchdog.py` has no concept of "is
   the standby armed, and is exactly one standby armed fleet-wide." Add a check function
   (e.g. `check_standby_arming()`) that SSHes to both app1-bu and core-bu, tests for
   `/root/.hermes/standby.disarmed`, and alerts if: (a) both are armed simultaneously
   (critical: the split-brain condition this whole package exists to prevent), or (b)
   neither is armed (no DR coverage at all), or (c) the armed one's crontab is missing the
   expected two entries.
2. **Standby watchdog liveness check.** Confirm the watchdog log (`/var/log/hermes-
   standby-watchdog.log`) on the armed standby has a line newer than ~15 minutes (3x the
   5-minute cron interval): silence longer than that means the cron itself died, not that
   Core is healthy. This mirrors the pattern `check_backup_freshness()` already uses for
   S3 sync staleness in the same file (lines ~420-476), just applied to a log-mtime
   instead of an S3 object timestamp.
3. **REMOTE_USER_UNITS entry for the standby's own gateway, conditionally.** Unlike
   `anita-mnz` (whose entry is unconditional because her gateway always runs there), the
   standby's `hermes-gateway` unit should be *absent* while dormant and *present and
   active* only during a failover. Add a check that reads the armed state (from item 1)
   and asserts: dormant standby => gateway absent/inactive is fine (no alert); armed
   standby with the unit active => alert with "STANDBY IS CURRENTLY PRIMARY", since that
   state should never persist longer than it takes to fail back once Core recovers.
4. **`REMOTE_DOCKER_CONTAINERS`**: no change needed: core-bu, like app1-bu, runs no
   Docker containers for the standby role itself.
5. **Backup-freshness check should also cover core-bu once armed**, the same way it
   already covers the `live/` S3 prefix: no new work needed here since the sync script
   pulls from the same `live/` prefix `check_backup_freshness()` already audits.

None of the above four gaps were fixed as part of this package (health-master-watchdog.py
lives on Core and per the task's read-only constraint on Core was not modified). This
section is the explicit list of what the parent must add.

---

## 7. Provider-diversity risk (explicit, per task requirement)

**The rule that has governed this org's DR design: "a netcup outage must not kill both
Core and its standby": is broken by this move**, and this is a genuine regression, not a
paperwork detail:

- Core: netcup, Manassas VA.
- core-bu: netcup, Nuremberg, Germany.
- Both share one provider. A netcup-wide outage (account suspension, billing failure,
  provider-side incident, netcup's own upstream/DC failure class of event) can take out
  both simultaneously, in a way a Hetzner standby cannot be taken out by.
- What is gained instead: **regional diversity within netcup**: 100.5 ms RTT / different
  continent / different power grid / different weather system than a Manassas-only
  failure. This defends against the failure modes that hit *one location* (DC outage,
  regional power, regional network) but not against failure modes that hit *the provider
  as an entity* (account, billing, provider-wide incident).
- This exact regression is already flagged as a live, factually-incorrect claim risk in
  `reference-update-matrix-2026-09-15.md` row 46 and its "Top 5" item 1:
  `README.md` line 277 currently states "core-bu stays at Hetzner specifically so a
  netcup outage can't kill both Core and standby simultaneously": that sentence becomes
  **false**, not merely outdated, the moment core-bu (netcup) is armed. It must be
  corrected as part of the documentation sweep in section 4 step 6, not left as-is.

**Options, per the task's requirement to state them explicitly:**

**Option A: Keep app1-bu (Hetzner) as a third tier, provider-diverse last resort.**
Cost: ~EUR 31.99/mo (Ashburn) or ~EUR 9.49/mo if relocated to Hetzner fsn1/nbg1 per
`standby-host-replacement-2026-09-14.md`'s own analysis (which itself is now superseded by
the decision to build core-bu: see reference-update-matrix row 30). Keeps a real
provider-diverse fallback for the specific netcup-wide failure class. Operational cost:
a third arm/disarm state to track, and the single-armed-standby rule now needs to cover
three hosts, not two: the `DISARM_FILE` gate in this package generalizes fine to three,
but the runbook and the monitoring check in section 6 item 1 must be written for "exactly
one of N armed," not "exactly one of two."

**Option B: Accept the risk and retire app1-bu, documenting the gap.** Lower cost, one
fewer moving part, matches the plan currently in motion
(`migration-plan-app4-core-bu-2026-09-15.md` §5 acceptance criterion 6: "app1-bu is either
retired... or explicitly retained as the provider-diverse standby": this is an open
decision, not yet made, per that plan's own Q4). If chosen, the retirement must include
correcting the now-false README/architecture-doc claims (section 4 step 6) rather than
silently leaving them stale.

**This decision has not been made by this task**: it belongs to the owner, per the
migration plan's own open-question list (Q4). This package is deployable under either
choice; the choice only affects whether app1-bu's scripts get the `DISARM_FILE` back-port
and stay in the crontab-removed "disarmed" state indefinitely (Option A) or get fully
retired per the P3 checklist in `reference-update-matrix-2026-09-15.md` (Option B).

---

## 8. Summary of defects fixed, matched to the task's stated gaps

| Defect | Where it lived | Fix in this package |
|---|---|---|
| Sync script tests reachability with `ping`; a host can answer ping while Hermes is dead | `hermes-standby-sync.sh` line 18 (`ping -c 1 -W 2 "$LIVE_HOST"`) | Replaced with the same unit-state SSH probe the watchdog uses; sync only proceeds when Core's gateway unit reads `active` |
| No failback logic in the sync path | `hermes-standby-sync.sh`, entire file | Added a primary-awareness guard (exits immediately if this box's own gateway is running) so the sync script can never clobber an active standby's config. Actual failback (stopping the standby's gateway) is owned by the watchdog, as it was in the proven 2026-09-15 fix, and preserved unchanged here |
| Decision branch, fence-before-takeover, self-stand-down failback (proven 2026-09-15) | `hermes-standby-watchdog.sh` | Preserved unchanged, ported to core-bu's host identity (probe key path, standby name in alerts) |
| No code-level enforcement of "only one standby armed" | Neither script, on either host | New `DISARM_FILE` gate added to both scripts in this package; ships disarmed by default |

---

## Files delivered

- This document: `/root/projects/itpp-infrastructure/docs/infrastructure/core-bu-standby-package-2026-09-15.md`
- `/root/work/core-bu/standby/hermes-standby-watchdog.sh` (397 lines): `bash -n`: clean
- `/root/work/core-bu/standby/hermes-standby-sync.sh` (84 lines): `bash -n`: clean

Nothing was installed on core-bu, app1-bu, or Core. No systemd unit was created or
modified anywhere. No production process was started, stopped, or restarted.