# Disaster Recovery Plan — IT Pro Partner

**Author:** ShoNuff (Hermes AI Agent)
**Date:** July 4, 2026
**Keep this document in:** Password manager (Bitwarden/Hudu), printed copy, and anywhere reachable without server access.

## Architecture

| Box | Provider | Location | Specs | Monthly | Runs |
|-----|----------|----------|-------|---------|------|
| app1 | Netcup | Manassas, VA | RS 2000 G12 (8c, 16GB, 512GB) | ~$27 | Portal, Hermes, VPN |
| app2 | Netcup | Manassas, VA | RS 4000 G12 (12c, 32GB, 1TB) | ~$43 | AI, Docker apps, WordPress |
| app1-bu | Hetzner | EU | CPX11 (2c, 2GB, 40GB) | ~$5 | Standby Hermes only — auto-failover |

## Failover architecture (app1 → app1-bu)

```
app1 (netcup, live)              app1-bu (Hetzner, standby)
┌─────────────────────┐         ┌──────────────────────────┐
│  Hermes (running)   │   ping  │  Hermes (off)            │
│  S3 sync every 15m  │◄───────►│  Cron watchdog every 10m│
│  Telegram connected  │ 152.53  │  Checks live box health  │
└─────────────────────┘         └──────────────────────────┘
         │                                │
         ▼                                ▼
  s3://hermes-vps-backups/live/ (15-min checkpoint — state.db, config, skills, cron, profiles)
```

**If app1 dies:**
- app1-bu watchdog detects 3 failed pings over ~60s
- Syncs latest state from `s3://hermes-vps-backups/live/` (RPO <15 min)
- Starts Hermes gateway on app1-bu
- Telegram reconnects automatically (same bot token)
- Total failover: ~2 min

## Critical Credentials (store externally, NOT on servers)

- Netcup CCP login → password manager
- Wasabi IAM keys → password manager
- Hetzner API token → password manager
- Hermes setup token → password manager
- Cloudflare API token → Cloudflare dashboard
- wisp_rsa SSH key → password manager backup
- S3 buckets: hermes-vps-backups, mikrotik-ccr-backups, itpropartner-backups (Wasabi us-east-1)

## Recovery Scenarios

### Hermes dies (app1) — auto-failover ~2 min (manual: 30 min)
**Auto:** app1-bu detects outage, syncs S3, starts Hermes. No action needed.

**Manual fallback if auto-failover didn't trigger:**
1. Verify from app1-bu: `hermes cron list` or `cat /var/log/hermes-standby-watchdog.log`
2. If Hermes stopped and didn't fail over: `hermes gateway start`
3. Or reboot app1-bu — boot restore script runs S3 sync and starts Hermes

### Emergency failover (manual) — 2 min
1. SSH into app1-bu: `ssh -i ~/.ssh/wisp_rsa root@5.161.114.8`
2. Sync latest state: `aws s3 sync s3://hermes-vps-backups/live/ ~/.hermes/ --endpoint-url https://s3.us-east-1.wasabisys.com --exclude "*.lock" --exclude "node/*" --exclude "bin/*" --exclude "cache/*" --exclude "sandboxes/*"`
3. Start Hermes: `hermes gateway start`

### AI/Apps server dies (app2) — 2-4 hr recovery
1. Order new RS 2000/4000 in Manassas
2. Install Docker + base software
3. Restore each app from `s3://hermes-vps-backups/apps/<app-name>/latest/`
4. Restart each Docker compose stack
5. Update Cloudflare DNS per subdomain

### Total loss — 4-6 hr recovery
1. Provision two new Netcup servers
2. Recover Hermes first: `aws s3 sync s3://hermes-vps-backups/live/ ~/.hermes/ --endpoint-url https://s3.us-east-1.wasabisys.com --exclude "*.lock"`
3. Recover app2 second
4. Restore VPN, router backup jobs
5. Point all Cloudflare DNS records

### Data recovery (any scenario)
- 15-min checkpoint in `s3://hermes-vps-backups/live/` for Hermes state
- Router configs: 365-day retention in `mikrotik-ccr-backups`
- S3 versioning on all buckets prevents accidental overwrite/deletion
- Lifecycle policy: 365-day retention, then expire

## DR scripts on S3

All in `s3://hermes-vps-backups/standby/`:
- `hermes-standby-restore.sh` — Boot-time restore (for Hetzner rescue mode deployment)
- `hermes-standby-watchdog.sh` — 10-min cron health check, triggers failover
- `hermes-standby.service` — systemd unit for boot-time restore
- `DR-PLAN.md` — This document
- `recovery-bundle-2026-07-05.md` — Full data dump (configs, credentials, conversation history)