# Hermes Warm Standby — Failover Architecture

**Server:** app1-bu.itpropartner.com (5.161.225.131)
**Type:** Hetzner CPX21 (3 vCPU, 4 GB, 80 GB)
**Status:** Warm standby — dormant, auto-activates on Core failure

## How It Works

```
Every 5 min:
  Watchdog pings Core (152.53.192.33)
    ├── Ping succeeds → exit (nothing to do)
    └── Ping fails → confirmation loop
           ├── Check every 30s × 4 attempts (~2 min)
           ├── Any ping succeeds → false alarm, exit
           └── All 4 fail → Core confirmed DOWN
                  ├── 1. Sync latest state from S3
                  ├── 2. Start Hermes gateway
                  ├── 3. Telegram alert to Germaine
                  └── 4. Email alert to g@germainebrown.com

Every 10 min:
  Sync script pulls from s3://hermes-vps-backups/live/ → ~/.hermes/
    ├── If Core is alive AND standby gateway is running:
    │     Stop gateway (Core is back, auto-shutdown standby)
    └── Otherwise: sync latest state
```

## Files

| File | Purpose |
|---|---|
| `/root/.hermes/scripts/hermes-standby-sync.sh` | Pulls state from S3 every 10 min |
| `/root/.hermes/scripts/hermes-standby-watchdog.sh` | Health checks + failover trigger |

## Crontab

```
*/5  * * * * /root/.hermes/scripts/hermes-standby-watchdog.sh
*/10 * * * * /root/.hermes/scripts/hermes-standby-sync.sh
```

## Credentials

| File | Purpose |
|---|---|
| `/root/.aws/credentials` | Wasabi S3 access (read-only) |
| `/opt/awscli-venv/` | AWS CLI in venv (cron-safe) |

## RTO/RPO

| Metric | Target | Actual |
|---|---|---|
| RTO (Recovery Time) | 5 min | ~3 min (2 min detection + 1 min sync/start) |
| RPO (Recovery Point) | 15 min | S3 sync interval |
| Worst-case detection | 7 min | 5 min check + 2 min confirmation |

## Provider Diversity

Core is on netcup (Germany). Standby is on Hetzner Ashburn (US). Both providers must fail simultaneously for total outage — extremely unlikely.

## Recovery (Failback)

When Core comes back online:
1. Stop the standby gateway: `systemctl stop hermes-gateway`
2. Core syncs from its own state
3. Standby resumes watchdog + sync mode
4. No automatic failback — manual only to prevent split-brain

## Notes

- Standby config is synced from S3 — no manual config needed
- Gateway is disabled from auto-start (`systemctl disable hermes-gateway`)
- Standby only activates after 4 consecutive ping failures (120 seconds)
- False alarms (transient network blips) are caught by the confirmation loop
- If Core is alive, sync script auto-stops the standby gateway — no split-brain