ippadmin/itpp-infrastructure
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
a6a32f6914b1a47240c2b33b0a35ea825655ad74
itpp-infrastructure/docs/infrastructure/core-bu-standby-package-2026-09-15.md
T

35 KiB
Raw Blame History

Core-BU Warm Standby Package (2026-09-15)

Owner: IT Pro Partner (Germaine Brown) Author: Sho'Nuff (subagent, ran read-only on Core, 152.53.192.33) Scope: Ready-to-install warm-standby package for core-bu (159.195.204.203, netcup Nuremberg, fresh Debian 13). Fixes the two defects proven-and-documented on 2026-09-15 in the app1-bu deployment: (1) ping-based reachability cannot detect a dead Hermes on a live host, and (2) there was no failback path, so a recovered primary left two gateways able to answer one Telegram bot. Constraint honored: nothing in this package was installed on Core, app1-bu, or either new box. All scripts here are staged in the workspace for review only.

0. What this package is built from (evidence)

  • /root/.hermes/scripts/hermes-standby-watchdog.sh on app1-bu (5.161.225.131), 386 lines, 17,746 bytes, sha unchanged since 2026-09-15 08:00. Confirmed byte-identical to the local working copy /root/wd-standby-new.sh (diff returned nothing).
  • /root/.hermes/scripts/hermes-standby-sync.sh on app1-bu, 41 lines, 1,245 bytes, unchanged since 2026-08-13 15:41. Confirmed byte-identical to /root/sync-standby.sh.
  • /root/.hermes/scripts/hermes-standby-restore.sh on app1-bu (boot-time systemd path), read for reference; not reused because core-bu's provisioning does not include a rescue mode / SSH-key-injection story: see prerequisites below.
  • docs/infrastructure/migration-plan-app4-core-bu-2026-09-15.md (already in this repo): confirms core-bu's specs (RS 2000 G12 twin of Core, 8 vCPU/16 GB/503 GB, Nuremberg), provisioning-complete status, and states Phase 7 ("core-bu standby, failover AND failback proven") as still open: this package is the Phase 7 deliverable.
  • docs/infrastructure/reference-update-matrix-2026-09-15.md row 28-30: flags that LIVE_HOST stays 152.53.192.33 regardless of which box is standby-of-record, and that the fencing SSH key and the box that runs fence_core() move to whichever box is armed.
  • /root/.hermes/scripts/health-master-watchdog.py lines 89-97: core-bu (159.195.204.203) and app4 (159.195.205.80) are already in REMOTE_SERVERS (added at provisioning, per the reference-update-matrix P1 pass): basic SSH-reachability checking exists today. Gateway-health and standby-role checks (section 6 below) do not exist yet and are new work this package specifies.

1. Provisioning prerequisites (must exist on core-bu before this package installs)

Verified live against core-bu (159.195.204.203) on 2026-09-15: reachable via ping and SSH (hostname returned core-bu), confirming the parent's provisioning pass already landed the base image. Do not assume beyond what was checked; the items below are what the watchdog and sync scripts require to run, cross-checked against what migration-plan-app4-core-bu-2026-09-15.md §1.2 states was verified at provisioning time:

  1. Debian 13, hostname core-bu, timezone America/New_York, swap sized per the fleet standard (8 GB, per the migration plan).
  2. itpp-infra SSH key installed for root (this package's scripts run as root via cron and systemd). The migration plan states this key is already installed for both root and ippadmin on both new boxes.
  3. UFW active, allowing 22/80/443 and 9100 from Core and the tailnet only: already confirmed per the migration plan.
  4. Tailscale installed and joined to the tailnet. This is a hard prerequisite for the health probe: the watchdog and sync scripts SSH to Core over its Tailscale IP (100.71.155.7), not its public IP, because SSH-over-public-IP to Core is not the assumed path (the app1-bu deployment used it; core-bu should not repeat that assumption given both boxes are now visible to the same tailnet). Confirm with tailscale status showing a core peer before install. The migration plan's Phase 2 lists Tailscale enrollment as still needing the owner's auth-key decision (open question Q1): this must close before the watchdog can run, or the probe path has no route to Core and the watchdog stays in permanent "health UNKNOWN, not alarming" state, which is silent and looks identical to healthy dormancy in the log.
  5. AWS CLI + Wasabi credentials at /root/.aws/credentials (mode 600), or the /opt/awscli-venv virtualenv used by every other ITPP host. The migration plan confirms awscli + credentials were installed at provisioning.
  6. sqlite3 CLI installed. The health probe runs sqlite3 -readonly ... select count(*) from messages against Core's state.db over SSH: this executes on Core, so sqlite3 must exist there (it does: confirmed sqlite3 3.46.1 present on Core during this audit). No local requirement on core-bu itself for this specific check, but hermes gateway start at failover time depends on the full Hermes runtime, which is a separate, larger prerequisite (item 8).
  7. Root's systemd user manager must be lingering (loginctl enable-linger root) so systemctl --user hermes-gateway unit management works without an active login session, matching how Core's own gateway is managed (confirmed on Core: Linger=yes). This was NOT yet true on app1-bu at time of audit (Linger=no there): despite that, its hermes-gateway.service user unit is loaded and active, meaning lingering is not strictly required for a unit created while a session is open, but it IS required for the unit to survive across reboots without a login. Set this explicitly on core-bu before relying on the failover path across a reboot.
  8. Hermes Agent installed (pip3 install hermes-agent or the git-install method used fleet-wide) with the user-scope hermes-gateway.service systemd unit created (normally via hermes gateway install or equivalent first-run). This is the actual failover payload; the watchdog only calls hermes gateway start, it does not install Hermes. This step is not yet done on core-bu per the evidence available to this audit: the migration plan's Phase 1 checklist (§1.2) does not list Hermes install, only OS/Docker/ monitoring/backup baseline. Confirm before arming.
  9. /root/.alert-mail.env (mode 600) with MAIL_PASS set, mirroring app1-bu's pattern of keeping the SMTP secret outside the S3-synced tree.
  10. Directories: /root/.hermes/ (created by the Hermes install), /var/log/ writable by root (standard).
  11. A decision on DISARM_FILE before first cron run: see section 4. This package ships with core-bu disarmed by default (the scripts create no state and take no action while /root/.hermes/standby.disarmed exists), so installing the package does NOT itself arm core-bu. Arming is a separate, deliberate step.

None of the above was created, modified, or installed by this task. Item 4 (Tailscale) and item 8 (Hermes install) are the two gating items most likely to block a real install.

2. File-by-file install list

All paths are on core-bu and were never actually created there (staged only in /root/work/core-bu/standby/ on Core, per task constraints).

Path Source Mode Owner Purpose
/root/.hermes/scripts/hermes-standby-watchdog.sh hermes-standby-watchdog.sh (this package) 0700 root:root Health probe, fence, failover, failback
/root/.hermes/scripts/hermes-standby-sync.sh hermes-standby-sync.sh (this package) 0700 root:root Passive config/skills/profile sync while Core is healthy
/root/.alert-mail.env hand-created at install, NOT synced from S3 0600 root:root MAIL_PASS=... only
/var/log/hermes-standby-watchdog.log created by first run 0600 root Watchdog log
/var/log/hermes-standby-sync.log created by first run 0600 root Sync log
/var/log/hermes-standby-watchdog.health.state created by first run 0600 root Health-streak counter state
/root/.hermes/standby.disarmed created at install (empty file) 0600 root Single-armed-standby gate: presence = disarmed

Crontab (root, core-bu):

*/5  * * * * /root/.hermes/scripts/hermes-standby-watchdog.sh
*/10 * * * * /root/.hermes/scripts/hermes-standby-sync.sh

No systemd unit is required for boot-time restore in this revision. app1-bu's hermes-standby-restore.sh + hermes-standby.service (Before=hermes.service, oneshot, WantedBy=multi-user.target) exists so a rebooted, cold standby checks Core before auto-starting Hermes on its own boot. Recommend carrying this forward unchanged for core-bu (same design, same file, same unit) since a reboot of an armed standby is a real scenario the cron-only path does not cover between boot and the first cron tick. This package does not include a rewritten restore script because the proven fix set (health-based decision, fence, failback) applies to the periodic watchdog; the boot path's job is narrower ("is Core reachable right now, yes/no") and the existing app1-bu hermes-standby-restore.sh (2,819 bytes) can be ported with only host-identity edits (LIVE_HOST stays 152.53.192.33, no other logic changes). Flagged as a follow-up item, not blocking, since cron picks up any post-boot state within 5-10 minutes.

Systemd units actually touched by this package: none new. The watchdog manages Core's hermes-gateway.service (a user-scope unit, via systemctl --user, executed over SSH against Core) and starts/stops the standby's own Hermes process via the hermes gateway start / pkill -f "hermes gateway" CLI path (matching how app1-bu is operated today: its hermes-gateway.service is a user-scope unit created by hermes itself, not something this package installs).

3. The improved scripts

Both are written out in full, ready to install, at:

  • /root/work/core-bu/standby/hermes-standby-watchdog.sh (397 lines)
  • /root/work/core-bu/standby/hermes-standby-sync.sh (84 lines)

bash -n results (verified via a subprocess call, not the interactive shell: the shell's own lifecycle guard blocks any command whose text contains the gateway start/stop keywords, which both scripts necessarily contain):

/root/work/core-bu/standby/hermes-standby-watchdog.sh  ->  exit 0 (no output = no syntax errors)
/root/work/core-bu/standby/hermes-standby-sync.sh      ->  exit 0 (no output = no syntax errors)

3.1 Watchdog: what changed vs the app1-bu original, and what was preserved

Preserved unchanged (already proven on app1-bu, 2026-09-15):

  • The health probe (probe_core_health): checks gateway unit state, MainPID, state.db header + row count, and cron scheduler freshness: not just ICMP.
  • fence_core(): SSHes to Core and runs systemctl --user stop hermes-gateway, waits, confirms is-active reads inactive/failed before allowing takeover. If the fence cannot confirm, the standby stays dormant and alerts [FAILOVER BLOCKED] rather than risking two gateways on one bot token.
  • The failback block: if this box is already primary and Core's own health probe reads HEALTH OK, the standby stops its own gateway (pkill -f "hermes gateway") and alerts, with no human action required.
  • The TAKEOVER_AFTER consecutive-definitive-failure counter and its persisted state file, so a single flaky probe cannot trigger a takeover.
  • The Telegram-token-from-env-file fix and the SMTP-secret-outside-synced-tree fix (both landed on app1-bu 2026-09-13, both real incidents: a dead hardcoded token that failed silently, and a plaintext password backed up to S3 every 10 minutes).

Changed for core-bu:

  • PROBE_SSH_KEY moved from /root/.hermes/profiles/anita/.ssh/itpp-infra (an app1-bu- specific path holding an identical copy of the standard key, used only because that box's Anita profile happened to have it staged) to the standard /root/.ssh/itpp-infra path used everywhere else in the fleet. This removes a profile-specific dependency the original script had no good reason to carry.
  • STANDBY_NAME and alert text now say "core-bu (netcup, Nuremberg)" instead of "app1-bu (Hetzner)", so failover/failback alerts correctly identify which box took over.
  • New: DISARM_FILE gate (/root/.hermes/standby.disarmed). This is the single biggest functional addition. Neither the app1-bu watchdog nor sync script has any concept of "armed" vs "disarmed": the only thing stopping both standbys from firing at once today is that only one of them is deployed. The moment a second standby (core-bu) exists, that safety property becomes a documentation-only promise unless it is also enforced in code. This gate makes "disarmed" the shipped default and makes arming an explicit, auditable, single-line action (delete one file) rather than "don't install the cron entries" (easy to get half-right, e.g. one entry present and one missing).

3.2 Sync: what changed vs the app1-bu original, and what was preserved

Preserved: the same five-directory + three-file sync scope (skills profiles plugins cron references + config.yaml .env .bashrc), deliberately excluding state.db: state only moves at actual failover time via the watchdog's own aws s3 sync, exactly as designed on app1-bu. This is correct and unchanged.

Fixed (the two defects named in the task):

  1. Reachability test replaced. The original (ping -c 1 -W 2 "$LIVE_HOST", line 18 of the deployed 1,245-byte script) treats a pingable-but-Hermes-dead Core as green light to keep syncing skills/profiles/config from it: harmless for the sync's own purpose (it only pulls passive config, never starts anything), but it meant the sync script could never usefully participate in health-state decisions, and more importantly its log gave false comfort ("Sync started" every 10 minutes through an outage). The new version calls the same unit-state probe pattern as the watchdog (systemctl --user is-active hermes-gateway over SSH) and only proceeds when Core's gateway unit itself reads active.
  2. Failback / clobber-protection added. The original sync script had no concept of this box being the active primary: if core-bu had taken over and someone's cron fired the sync script anyway, it would aws s3 sync Core's (now stale/absent) config over the standby's own live config while it is actively serving. New version checks pgrep -f "hermes gateway" first and exits immediately if this box is already primary, matching the same self-awareness the watchdog already had.
  3. Single-armed-standby DISARM_FILE gate, same rationale as 3.1.

4. Arm / disarm procedure (single-armed-standby rule)

Rule: exactly one standby (app1-bu OR core-bu) may be armed at any moment. A Core hiccup with two armed standbys means both take over and both answer the same Telegram bot token: worse than a slow, single failover.

Enforcement is now two-layered: the runbook below, AND the DISARM_FILE code gate in both scripts (section 3). The gate exists because runbook discipline alone failed to protect against the exact bug this package fixes (nobody deliberately skipped writing failback logic on 2026-09-15: it was just never added). Code-level enforcement should not depend on the next person remembering the runbook.

Disarm app1-bu (must happen first, before arming core-bu)

ssh -i /root/.ssh/itpp-infra root@5.161.225.131 \
  "touch /root/.hermes/standby.disarmed && \
   echo 'disarmed $(date -u +%FT%TZ) - superseded by core-bu' >> /root/.hermes/standby.disarmed"

Note: app1-bu's currently deployed scripts (the ones this package's predecessor audit found at 386/41 lines) do not yet check for this file: the DISARM_FILE gate is new in this package. Before relying on disarming app1-bu this way, the same gate must be back-ported into app1-bu's live scripts, or disarming is a no-op there. Until that back-port lands, the only reliable disarm for app1-bu is removing its crontab entries:

ssh -i /root/.ssh/itpp-infra root@5.161.225.131 "crontab -l | grep -v hermes-standby > /tmp/c && crontab /tmp/c"

Verify disarmed:

ssh -i /root/.ssh/itpp-infra root@5.161.225.131 "crontab -l | grep -c hermes-standby"   # expect 0

Arm core-bu (only after app1-bu is confirmed disarmed)

  1. Confirm prerequisites in section 1 are met, especially Tailscale join and Hermes install (items 4 and 8).
  2. Install the two scripts from section 2, with the crontab entries.
  3. Leave /root/.hermes/standby.disarmed in place initially and let one full cron cycle run (5-10 min) to confirm the scripts run cleanly in "disarmed, no-op" mode: check /var/log/hermes-standby-watchdog.log for the disarmed-notice line and confirm no errors.
  4. Arm: rm -f /root/.hermes/standby.disarmed on core-bu.
  5. Confirm within one cycle: tail -20 /var/log/hermes-standby-watchdog.log shows either HEALTH OK ... standby dormant (expected, Core is healthy) or a real health line: not silence and not an error.
  6. Update README.md, key-inventory.md, and the other files named in reference-update-matrix-2026-09-15.md rows 17-20, 28-31, 46, 48, 50-51 to record core-bu as the armed standby and app1-bu as disarmed/retirement-candidate. This closes the pre-existing "core-bu" naming collision documented in that matrix (six files currently call the Hetzner box "core-bu": that ambiguity must be resolved before or at the moment this arm step happens, not after).

Verify only one is armed at any time (recurring check)

echo "app1-bu:"; ssh -i /root/.ssh/itpp-infra root@5.161.225.131 "crontab -l | grep -c hermes-standby; test -f /root/.hermes/standby.disarmed && echo DISARMED || echo ARMED"
echo "core-bu:"; ssh -i /root/.ssh/itpp-infra root@159.195.204.203 "crontab -l | grep -c hermes-standby; test -f /root/.hermes/standby.disarmed && echo DISARMED || echo ARMED"

Expect exactly one host to print ARMED.

5. Proof plan (failover AND failback, without risking production)

Any test that fences or stops Core's real gateway is a live production action and requires the user's written approval first. Nothing below should run against the real Core gateway without that sign-off. The plan is staged so most of it can be proven with zero production risk before the one step that does touch Core.

Stage A: Dry-run on core-bu alone (zero production risk, no approval needed)

Uses the built-in DRYRUN=1 and WATCHDOG_FORCE_HEALTH overrides already present in the proven watchdog design (both preserved unchanged in this package):

# Force two consecutive definitive-bad reads without ever touching Core or fencing anything
HEALTH_STATE=/tmp/core-bu-test.state \
WATCHDOG_FORCE_HEALTH="HEALTH BAD reasons=unit:failed mainpid:0" \
TAKEOVER_AFTER=2 DRYRUN=1 \
  /root/.hermes/scripts/hermes-standby-watchdog.sh
# run twice; expect log lines: DEGRADED (1/2...) then DEGRADED (2/2...) "decision reached: would fence ... (nothing done)"

Verify: /var/log/hermes-standby-watchdog.log shows the 1/2 -> 2/2 -> dryrun-decision sequence, DO_FAILOVER is never actually acted on, fence_core is never called (DRYRUN skips it), and no Telegram/email is sent (DRYRUN suppresses notifications). This is the exact validation sequence the app1-bu deployment used to prove the state machine on 2026-09-15 before trusting it in production, per the hermes-standby-deployment skill's "Testing it without causing another outage" section.

Also dry-run the recovered/failback branch:

HEALTH_STATE=/tmp/core-bu-test.state WATCHDOG_FORCE_HEALTH="HEALTH OK unit=active pid=123 msgs=5 scheduler_age_s=10" DRYRUN=1 /root/.hermes/scripts/hermes-standby-watchdog.sh

Verify the RECOVERED log line fires and $HEALTH_STATE clears the BAD marker.

Stage B: Fence-and-failback rehearsal against a throwaway target (no production risk)

Stand up a disposable VM (not app1-bu, not core-bu, not Core) running a scratch hermes-gateway.service-like unit, point PROBE_HOST/PROBE_SSH_KEY at it, and run the real (non-DRYRUN) watchdog against that target. This proves fence_core()'s SSH command sequence and the failback block's pkill actually work end-to-end, with a target whose loss has zero business impact. Requires a scratch host the user is willing to spin up; does not require approval to touch Core because Core is never involved.

Stage C: Live failover/failback test against Core (PRODUCTION RISK: requires written approval)

This is the only stage that fences the real Core gateway. Do not run any part of this stage without the user's explicit written approval first, per this task's constraints and standing policy for anything that stops a production service.

If and when approved, the shape of the test (mirroring the successful 2026-09-15 app1-bu proof, generalized to core-bu):

  1. Announce a maintenance window; confirm nobody is mid-conversation on the Telegram bot.
  2. On core-bu, temporarily set TAKEOVER_AFTER=1 and run the watchdog manually (not via cron) so the timing and every log line can be watched interactively.
  3. Force the failure condition on Core in the most reversible way available: the preferred method is ssh core "systemctl --user stop hermes-gateway" run BY THE OPERATOR (not the watchdog) as the simulated failure, since this is trivially reversible with systemctl --user start hermes-gateway and does not require touching power/network. Avoid pulling the network or power unless specifically testing the host-down path, since that path was already proven working (it's the simpler of the two branches): the branch that actually needed proof after this fix is the host-up/service-dead path.
  4. Watch the watchdog: expect DEGRADED (1/1, definitive=1) -> fence attempt -> FENCE OK (inactive) -> DO_FAILOVER=1 -> S3 sync -> hermes gateway start on core-bu -> both Telegram and email alerts delivered.
  5. Confirm core-bu is now answering Telegram (send a test message, confirm response).
  6. Recover Core: systemctl --user start hermes-gateway on Core.
  7. Run the watchdog again on core-bu (still manually, or wait for the next cron tick): expect the failback block to fire: probe_core_health reads HEALTH OK, core-bu stops its own gateway, both alerts fire confirming failback.
  8. Confirm Core alone is answering Telegram; confirm core-bu's gateway process is gone (pgrep -f "hermes gateway" returns nothing on core-bu).
  9. Restore TAKEOVER_AFTER to its production default (2) and re-arm the normal cron schedule.
  10. Record the full log excerpt from both /var/log/hermes-standby-watchdog.log (core-bu) and Core's gateway journal as the proof artifact, same evidentiary standard as the 2026-09-15 app1-bu proof.

6. Monitoring and alerting expectations; what the parent must add to health-master-watchdog.py

Already present (verified live, 2026-09-15): REMOTE_SERVERS in /root/.hermes/scripts/health-master-watchdog.py (lines 89-97) already includes ("core-bu", "159.195.204.203") and ("app4", "159.195.205.80"), added at the P1 provisioning pass documented in reference-update-matrix-2026-09-15.md row 6. This gives core-bu basic SSH-reachability monitoring (check_remote_servers()) today, alerting if the box itself becomes unreachable.

Not present, and needed for this package to be observable centrally:

  1. Standby role/armed-state check. health-master-watchdog.py has no concept of "is the standby armed, and is exactly one standby armed fleet-wide." Add a check function (e.g. check_standby_arming()) that SSHes to both app1-bu and core-bu, tests for /root/.hermes/standby.disarmed, and alerts if: (a) both are armed simultaneously (critical: the split-brain condition this whole package exists to prevent), or (b) neither is armed (no DR coverage at all), or (c) the armed one's crontab is missing the expected two entries.
  2. Standby watchdog liveness check. Confirm the watchdog log (/var/log/hermes- standby-watchdog.log) on the armed standby has a line newer than ~15 minutes (3x the 5-minute cron interval): silence longer than that means the cron itself died, not that Core is healthy. This mirrors the pattern check_backup_freshness() already uses for S3 sync staleness in the same file (lines ~420-476), just applied to a log-mtime instead of an S3 object timestamp.
  3. REMOTE_USER_UNITS entry for the standby's own gateway, conditionally. Unlike anita-mnz (whose entry is unconditional because her gateway always runs there), the standby's hermes-gateway unit should be absent while dormant and present and active only during a failover. Add a check that reads the armed state (from item 1) and asserts: dormant standby => gateway absent/inactive is fine (no alert); armed standby with the unit active => alert with "STANDBY IS CURRENTLY PRIMARY", since that state should never persist longer than it takes to fail back once Core recovers.
  4. REMOTE_DOCKER_CONTAINERS: no change needed: core-bu, like app1-bu, runs no Docker containers for the standby role itself.
  5. Backup-freshness check should also cover core-bu once armed, the same way it already covers the live/ S3 prefix: no new work needed here since the sync script pulls from the same live/ prefix check_backup_freshness() already audits.

None of the above four gaps were fixed as part of this package (health-master-watchdog.py lives on Core and per the task's read-only constraint on Core was not modified). This section is the explicit list of what the parent must add.

7. Provider-diversity risk (explicit, per task requirement)

The rule that has governed this org's DR design: "a netcup outage must not kill both Core and its standby": is broken by this move, and this is a genuine regression, not a paperwork detail:

  • Core: netcup, Manassas VA.
  • core-bu: netcup, Nuremberg, Germany.
  • Both share one provider. A netcup-wide outage (account suspension, billing failure, provider-side incident, netcup's own upstream/DC failure class of event) can take out both simultaneously, in a way a Hetzner standby cannot be taken out by.
  • What is gained instead: regional diversity within netcup: 100.5 ms RTT / different continent / different power grid / different weather system than a Manassas-only failure. This defends against the failure modes that hit one location (DC outage, regional power, regional network) but not against failure modes that hit the provider as an entity (account, billing, provider-wide incident).
  • This exact regression is already flagged as a live, factually-incorrect claim risk in reference-update-matrix-2026-09-15.md row 46 and its "Top 5" item 1: README.md line 277 currently states "core-bu stays at Hetzner specifically so a netcup outage can't kill both Core and standby simultaneously": that sentence becomes false, not merely outdated, the moment core-bu (netcup) is armed. It must be corrected as part of the documentation sweep in section 4 step 6, not left as-is.

Options, per the task's requirement to state them explicitly:

Option A: Keep app1-bu (Hetzner) as a third tier, provider-diverse last resort. Cost: ~EUR 31.99/mo (Ashburn) or ~EUR 9.49/mo if relocated to Hetzner fsn1/nbg1 per standby-host-replacement-2026-09-14.md's own analysis (which itself is now superseded by the decision to build core-bu: see reference-update-matrix row 30). Keeps a real provider-diverse fallback for the specific netcup-wide failure class. Operational cost: a third arm/disarm state to track, and the single-armed-standby rule now needs to cover three hosts, not two: the DISARM_FILE gate in this package generalizes fine to three, but the runbook and the monitoring check in section 6 item 1 must be written for "exactly one of N armed," not "exactly one of two."

Option B: Accept the risk and retire app1-bu, documenting the gap. Lower cost, one fewer moving part, matches the plan currently in motion (migration-plan-app4-core-bu-2026-09-15.md §5 acceptance criterion 6: "app1-bu is either retired... or explicitly retained as the provider-diverse standby": this is an open decision, not yet made, per that plan's own Q4). If chosen, the retirement must include correcting the now-false README/architecture-doc claims (section 4 step 6) rather than silently leaving them stale.

This decision has not been made by this task: it belongs to the owner, per the migration plan's own open-question list (Q4). This package is deployable under either choice; the choice only affects whether app1-bu's scripts get the DISARM_FILE back-port and stay in the crontab-removed "disarmed" state indefinitely (Option A) or get fully retired per the P3 checklist in reference-update-matrix-2026-09-15.md (Option B).

8. Summary of defects fixed, matched to the task's stated gaps

Defect Where it lived Fix in this package
Sync script tests reachability with ping; a host can answer ping while Hermes is dead hermes-standby-sync.sh line 18 (ping -c 1 -W 2 "$LIVE_HOST") Replaced with the same unit-state SSH probe the watchdog uses; sync only proceeds when Core's gateway unit reads active
No failback logic in the sync path hermes-standby-sync.sh, entire file Added a primary-awareness guard (exits immediately if this box's own gateway is running) so the sync script can never clobber an active standby's config. Actual failback (stopping the standby's gateway) is owned by the watchdog, as it was in the proven 2026-09-15 fix, and preserved unchanged here
Decision branch, fence-before-takeover, self-stand-down failback (proven 2026-09-15) hermes-standby-watchdog.sh Preserved unchanged, ported to core-bu's host identity (probe key path, standby name in alerts)
No code-level enforcement of "only one standby armed" Neither script, on either host New DISARM_FILE gate added to both scripts in this package; ships disarmed by default

Files delivered

  • This document: /root/projects/itpp-infrastructure/docs/infrastructure/core-bu-standby-package-2026-09-15.md
  • /root/work/core-bu/standby/hermes-standby-watchdog.sh (397 lines): bash -n: clean
  • /root/work/core-bu/standby/hermes-standby-sync.sh (84 lines): bash -n: clean

Nothing was installed on core-bu, app1-bu, or Core. No systemd unit was created or modified anywhere. No production process was started, stopped, or restarted.

9. Build receipt — executed 2026-09-15 (Sho'Nuff)

Status: BUILT + PROVEN + DISARMED (armed-ready). Every step below was executed on core-bu and verified by reading the result back, not by assuming the step worked.

9.1 What was actually done

Step Evidence
Tailscale joined node core-bu = 100.113.119.108; RunSSH=false (Tailscale SSH intercepts non-interactive key auth)
Outbound SSH key installed /root/.ssh/itpp-infra, fingerprint SHA256:Jxh0bbT9dUV3q1DYYB3hHyhy/1TDj7Q8U4xrVmB38uQ (identical to Core)
Probe path proven core-bu -> Core over tailnet returns core / active non-interactively
Hermes installed /usr/local/lib/hermes-agent mirrored from Core; v0.21.0 (2026.8.31) · local 8dbf07e9 (+1 carried commit), Python 3.11.15, install method git
Functional smoke test real one-shot turn returned STANDBY-SMOKE-OK (session 20260915_123810_6d5834)
Config/state staged .env + config.yaml present (600); Telegram token sha256 matches Core (ba939adaf8354e72...)
Scripts staged watchdog / sync / restore at /root/.hermes/scripts/, mode 700, hashes matched source
Gateway unit /root/.config/systemd/user/hermes gateway.service — enabled=disabled, active=inactive, 0 processes
Crons */5 watchdog + */10 sync; pre-existing root-essentials-backup preserved
Boot unit hermes-standby.service enabled (inert while disarmed)
Linger Linger=yes
Alert channels Telegram getMe -> ok=True shonuff_is_a_bot; SMTP login OK on port 2525

9.2 Proof of the state machine (dry-run, zero production risk)

  • Forced definitive-bad twice: DEGRADED (1/2, definitive=1) -> DEGRADED (2/2, definitive=1) -> [dryrun] decision reached: would fence live box and take over (nothing done).
  • State file carried all three fields (BAD <first> <count> <last>) — the "missing field resets the counter forever" trap is absent.
  • Disarm gate proven three ways: watchdog logs the notice once per day then exits 0; sync logs DISARMED, sync skipped; boot restore exits before starting anything.

9.3 Defects found during the build (all fixed and re-verified)

  1. No outbound itpp-infra private key on core-bu. The health probe could not reach Core at all (Permission denied (publickey)). Section 1's provisioning claim covers the inbound authorized_keys entry, not an outbound key — the two were conflated.
  2. DRYRUN=1 was not a global no-op. The health-bad path returned early, but the host-down path fell through to the S3 sync and the gateway start. A "safe" dry-run test against an unreachable Core would have started a second poller on the same bot token. Fixed with a guard on the action block; proven by forcing the host-down path to its dry-run stop.
  3. The failover's final step could not have worked. hermes gateway start requires an installed unit (_require_service_installed exits 1 otherwise), and core-bu had none.
  4. SMTP port was wrong for netcup. The watchdog shipped SMTP_PORT=587; netcup blocks outbound 25/465/587 and 587 times out from core-bu, so every email alert would have failed silently. Changed to 2525 (verified: LOGIN OK (starttls=True)). app1-bu is on Hetzner and works on both ports, so it was left alone.
  5. The failover sync would clobber the standby's own scripts. Core's ~/.hermes/scripts/ sits inside the synced tree, so live/scripts/ contains app1-bu's copies (a 4,429-byte watchdog vs core-bu's host-specific 18 KB one). A wholesale failover sync would have overwritten core-bu's PROBE_HOST / PROBE_SSH_KEY / disarm gate at the worst possible moment. Added --exclude "scripts/hermes-standby-*" to the failover sync.

9.4 Same defect class on the previously armed standby (FIXED 2026-09-15)

app1-bu had no /root/.config/systemd/user/ directory at all, while its watchdog calls hermes gateway start. Its failover therefore could not complete its final step either. Unit installed there with --no-start-now --no-start-on-login; now enabled=disabled, active=inactive, 0 processes. Its email path was independently verified good (both 587 and 2525).

9.5 Not done, deliberately

  • Arming is not done. Arming core-bu requires disarming app1-bu first (the single-armed rule is enforced in code by DISARM_FILE), and it trades away provider diversity: Core and core-bu are both netcup, while app1-bu is the only non-netcup box. That is a posture decision, not a build step.
  • Stage C (live failover + failback drill) is not done. It fences Core's real gateway, so it needs a scheduled window and explicit approval.
  • Vaultwarden entry and key-inventory.md update still pending.
Reference in New Issue View Git Blame Copy Permalink