Add standby architecture doc + failover sequence

2026-07-15 18:24:23 -04:00
parent 4ca8be4b6a
commit 7aafe5fe61
1 changed files with 76 additions and 0 deletions
@@ -0,0 +1,76 @@
+# Hermes Warm Standby — Failover Architecture
+
+**Server:** app1-bu.itpropartner.com (5.161.225.131)
+**Type:** Hetzner CPX21 (3 vCPU, 4 GB, 80 GB)
+**Status:** Warm standby — dormant, auto-activates on Core failure
+
+## How It Works
+
+```
+Every 5 min:
+  Watchdog pings Core (152.53.192.33)
+    ├── Ping succeeds → exit (nothing to do)
+    └── Ping fails → confirmation loop
+           ├── Check every 30s × 4 attempts (~2 min)
+           ├── Any ping succeeds → false alarm, exit
+           └── All 4 fail → Core confirmed DOWN
+                  ├── 1. Sync latest state from S3
+                  ├── 2. Start Hermes gateway
+                  ├── 3. Telegram alert to Germaine
+                  └── 4. Email alert to g@germainebrown.com
+
+Every 10 min:
+  Sync script pulls from s3://hermes-vps-backups/live/ → ~/.hermes/
+    ├── If Core is alive AND standby gateway is running:
+    │     Stop gateway (Core is back, auto-shutdown standby)
+    └── Otherwise: sync latest state
+```
+
+## Files
+
+| File | Purpose |
+|---|---|
+| `/root/.hermes/scripts/hermes-standby-sync.sh` | Pulls state from S3 every 10 min |
+| `/root/.hermes/scripts/hermes-standby-watchdog.sh` | Health checks + failover trigger |
+
+## Crontab
+
+```
+*/5  * * * * /root/.hermes/scripts/hermes-standby-watchdog.sh
+*/10 * * * * /root/.hermes/scripts/hermes-standby-sync.sh
+```
+
+## Credentials
+
+| File | Purpose |
+|---|---|
+| `/root/.aws/credentials` | Wasabi S3 access (read-only) |
+| `/opt/awscli-venv/` | AWS CLI in venv (cron-safe) |
+
+## RTO/RPO
+
+| Metric | Target | Actual |
+|---|---|---|
+| RTO (Recovery Time) | 5 min | ~3 min (2 min detection + 1 min sync/start) |
+| RPO (Recovery Point) | 15 min | S3 sync interval |
+| Worst-case detection | 7 min | 5 min check + 2 min confirmation |
+
+## Provider Diversity
+
+Core is on netcup (Germany). Standby is on Hetzner Ashburn (US). Both providers must fail simultaneously for total outage — extremely unlikely.
+
+## Recovery (Failback)
+
+When Core comes back online:
+1. Stop the standby gateway: `systemctl stop hermes-gateway`
+2. Core syncs from its own state
+3. Standby resumes watchdog + sync mode
+4. No automatic failback — manual only to prevent split-brain
+
+## Notes
+
+- Standby config is synced from S3 — no manual config needed
+- Gateway is disabled from auto-start (`systemctl disable hermes-gateway`)
+- Standby only activates after 4 consecutive ping failures (120 seconds)
+- False alarms (transient network blips) are caught by the confirmation loop
+- If Core is alive, sync script auto-stops the standby gateway — no split-brain