diff --git a/standby-architecture.md b/standby-architecture.md new file mode 100644 index 0000000..9411cc1 --- /dev/null +++ b/standby-architecture.md @@ -0,0 +1,76 @@ +# Hermes Warm Standby — Failover Architecture + +**Server:** app1-bu.itpropartner.com (5.161.225.131) +**Type:** Hetzner CPX21 (3 vCPU, 4 GB, 80 GB) +**Status:** Warm standby — dormant, auto-activates on Core failure + +## How It Works + +``` +Every 5 min: + Watchdog pings Core (152.53.192.33) + ├── Ping succeeds → exit (nothing to do) + └── Ping fails → confirmation loop + ├── Check every 30s × 4 attempts (~2 min) + ├── Any ping succeeds → false alarm, exit + └── All 4 fail → Core confirmed DOWN + ├── 1. Sync latest state from S3 + ├── 2. Start Hermes gateway + ├── 3. Telegram alert to Germaine + └── 4. Email alert to g@germainebrown.com + +Every 10 min: + Sync script pulls from s3://hermes-vps-backups/live/ → ~/.hermes/ + ├── If Core is alive AND standby gateway is running: + │ Stop gateway (Core is back, auto-shutdown standby) + └── Otherwise: sync latest state +``` + +## Files + +| File | Purpose | +|---|---| +| `/root/.hermes/scripts/hermes-standby-sync.sh` | Pulls state from S3 every 10 min | +| `/root/.hermes/scripts/hermes-standby-watchdog.sh` | Health checks + failover trigger | + +## Crontab + +``` +*/5 * * * * /root/.hermes/scripts/hermes-standby-watchdog.sh +*/10 * * * * /root/.hermes/scripts/hermes-standby-sync.sh +``` + +## Credentials + +| File | Purpose | +|---|---| +| `/root/.aws/credentials` | Wasabi S3 access (read-only) | +| `/opt/awscli-venv/` | AWS CLI in venv (cron-safe) | + +## RTO/RPO + +| Metric | Target | Actual | +|---|---|---| +| RTO (Recovery Time) | 5 min | ~3 min (2 min detection + 1 min sync/start) | +| RPO (Recovery Point) | 15 min | S3 sync interval | +| Worst-case detection | 7 min | 5 min check + 2 min confirmation | + +## Provider Diversity + +Core is on netcup (Germany). Standby is on Hetzner Ashburn (US). Both providers must fail simultaneously for total outage — extremely unlikely. + +## Recovery (Failback) + +When Core comes back online: +1. Stop the standby gateway: `systemctl stop hermes-gateway` +2. Core syncs from its own state +3. Standby resumes watchdog + sync mode +4. No automatic failback — manual only to prevent split-brain + +## Notes + +- Standby config is synced from S3 — no manual config needed +- Gateway is disabled from auto-start (`systemctl disable hermes-gateway`) +- Standby only activates after 4 consecutive ping failures (120 seconds) +- False alarms (transient network blips) are caught by the confirmation loop +- If Core is alive, sync script auto-stops the standby gateway — no split-brain