77 lines
2.6 KiB
Markdown
77 lines
2.6 KiB
Markdown
# Hermes Warm Standby — Failover Architecture
|
||
|
||
**Server:** app1-bu.itpropartner.com (5.161.225.131)
|
||
**Type:** Hetzner CPX21 (3 vCPU, 4 GB, 80 GB)
|
||
**Status:** Warm standby — dormant, auto-activates on Core failure
|
||
|
||
## How It Works
|
||
|
||
```
|
||
Every 5 min:
|
||
Watchdog pings Core (152.53.192.33)
|
||
├── Ping succeeds → exit (nothing to do)
|
||
└── Ping fails → confirmation loop
|
||
├── Check every 30s × 4 attempts (~2 min)
|
||
├── Any ping succeeds → false alarm, exit
|
||
└── All 4 fail → Core confirmed DOWN
|
||
├── 1. Sync latest state from S3
|
||
├── 2. Start Hermes gateway
|
||
├── 3. Telegram alert to Germaine
|
||
└── 4. Email alert to g@germainebrown.com
|
||
|
||
Every 10 min:
|
||
Sync script pulls from s3://hermes-vps-backups/live/ → ~/.hermes/
|
||
├── If Core is alive AND standby gateway is running:
|
||
│ Stop gateway (Core is back, auto-shutdown standby)
|
||
└── Otherwise: sync latest state
|
||
```
|
||
|
||
## Files
|
||
|
||
| File | Purpose |
|
||
|---|---|
|
||
| `/root/.hermes/scripts/hermes-standby-sync.sh` | Pulls state from S3 every 10 min |
|
||
| `/root/.hermes/scripts/hermes-standby-watchdog.sh` | Health checks + failover trigger |
|
||
|
||
## Crontab
|
||
|
||
```
|
||
*/5 * * * * /root/.hermes/scripts/hermes-standby-watchdog.sh
|
||
*/10 * * * * /root/.hermes/scripts/hermes-standby-sync.sh
|
||
```
|
||
|
||
## Credentials
|
||
|
||
| File | Purpose |
|
||
|---|---|
|
||
| `/root/.aws/credentials` | Wasabi S3 access (read-only) |
|
||
| `/opt/awscli-venv/` | AWS CLI in venv (cron-safe) |
|
||
|
||
## RTO/RPO
|
||
|
||
| Metric | Target | Actual |
|
||
|---|---|---|
|
||
| RTO (Recovery Time) | 5 min | ~3 min (2 min detection + 1 min sync/start) |
|
||
| RPO (Recovery Point) | 15 min | S3 sync interval |
|
||
| Worst-case detection | 7 min | 5 min check + 2 min confirmation |
|
||
|
||
## Provider Diversity
|
||
|
||
Core is on netcup (Germany). Standby is on Hetzner Ashburn (US). Both providers must fail simultaneously for total outage — extremely unlikely.
|
||
|
||
## Recovery (Failback)
|
||
|
||
When Core comes back online:
|
||
1. Stop the standby gateway: `systemctl stop hermes-gateway`
|
||
2. Core syncs from its own state
|
||
3. Standby resumes watchdog + sync mode
|
||
4. No automatic failback — manual only to prevent split-brain
|
||
|
||
## Notes
|
||
|
||
- Standby config is synced from S3 — no manual config needed
|
||
- Gateway is disabled from auto-start (`systemctl disable hermes-gateway`)
|
||
- Standby only activates after 4 consecutive ping failures (120 seconds)
|
||
- False alarms (transient network blips) are caught by the confirmation loop
|
||
- If Core is alive, sync script auto-stops the standby gateway — no split-brain
|