Hermes Warm Standby — Failover Architecture

Server: app1-bu.itpropartner.com (5.161.225.131) Type: Hetzner CPX21 (3 vCPU, 4 GB, 80 GB) Status: Warm standby — dormant, auto-activates on Core failure

How It Works

Every 5 min:
  Watchdog pings Core (152.53.192.33)
    ├── Ping succeeds → exit (nothing to do)
    └── Ping fails → confirmation loop
           ├── Check every 30s × 4 attempts (~2 min)
           ├── Any ping succeeds → false alarm, exit
           └── All 4 fail → Core confirmed DOWN
                  ├── 1. Sync latest state from S3
                  ├── 2. Start Hermes gateway
                  ├── 3. Telegram alert to Germaine
                  └── 4. Email alert to g@germainebrown.com

Every 10 min:
  Sync script pulls from s3://hermes-vps-backups/live/ → ~/.hermes/
    ├── If Core is alive AND standby gateway is running:
    │     Stop gateway (Core is back, auto-shutdown standby)
    └── Otherwise: sync latest state

Files

File	Purpose
`/root/.hermes/scripts/hermes-standby-sync.sh`	Pulls state from S3 every 10 min
`/root/.hermes/scripts/hermes-standby-watchdog.sh`	Health checks + failover trigger

Crontab

*/5  * * * * /root/.hermes/scripts/hermes-standby-watchdog.sh
*/10 * * * * /root/.hermes/scripts/hermes-standby-sync.sh

Credentials

File	Purpose
`/root/.aws/credentials`	Wasabi S3 access (read-only)
`/opt/awscli-venv/`	AWS CLI in venv (cron-safe)

RTO/RPO

Metric	Target	Actual
RTO (Recovery Time)	5 min	~3 min (2 min detection + 1 min sync/start)
RPO (Recovery Point)	15 min	S3 sync interval
Worst-case detection	7 min	5 min check + 2 min confirmation

Provider Diversity

Core is on netcup (Germany). Standby is on Hetzner Ashburn (US). Both providers must fail simultaneously for total outage — extremely unlikely.

Recovery (Failback)

When Core comes back online:

Stop the standby gateway: systemctl stop hermes-gateway
Core syncs from its own state
Standby resumes watchdog + sync mode
No automatic failback — manual only to prevent split-brain

Notes

Standby config is synced from S3 — no manual config needed
Gateway is disabled from auto-start (systemctl disable hermes-gateway)
Standby only activates after 4 consecutive ping failures (120 seconds)
False alarms (transient network blips) are caught by the confirmation loop
If Core is alive, sync script auto-stops the standby gateway — no split-brain

2.6 KiB Raw Permalink Blame History Unescape Escape