hermes-skills/skills/devops/infrastructure-automation/references/disaster-recovery-plan.md

Disaster Recovery Plan — IT Pro Partner

Author: ShoNuff (Hermes AI Agent) Date: July 4, 2026 Keep this document in: Password manager (Bitwarden/Hudu), printed copy, and anywhere reachable without server access.

Architecture

Box	Provider	Location	Specs	Monthly	Runs
app1	Netcup	Manassas, VA	RS 2000 G12 (8c, 16GB, 512GB)	~$27	Portal, Hermes, VPN
app2	Netcup	Manassas, VA	RS 4000 G12 (12c, 32GB, 1TB)	~$43	AI, Docker apps, WordPress
app1-bu	Hetzner	EU	CPX11 (2c, 2GB, 40GB)	~$5	Standby Hermes only — auto-failover

Failover architecture (app1 → app1-bu)

app1 (netcup, live)              app1-bu (Hetzner, standby)
┌─────────────────────┐         ┌──────────────────────────┐
│  Hermes (running)   │   ping  │  Hermes (off)            │
│  S3 sync every 15m  │◄───────►│  Cron watchdog every 10m│
│  Telegram connected  │ 152.53  │  Checks live box health  │
└─────────────────────┘         └──────────────────────────┘
         │                                │
         ▼                                ▼
  s3://hermes-vps-backups/live/ (15-min checkpoint — state.db, config, skills, cron, profiles)

If app1 dies:

• app1-bu watchdog detects 3 failed pings over ~60s
• Syncs latest state from s3://hermes-vps-backups/live/ (RPO <15 min)
• Starts Hermes gateway on app1-bu
• Telegram reconnects automatically (same bot token)
• Total failover: ~2 min

Critical Credentials (store externally, NOT on servers)

• Netcup CCP login → password manager
• Wasabi IAM keys → password manager
• Hetzner API token → password manager
• Hermes setup token → password manager
• Cloudflare API token → Cloudflare dashboard
• wisp_rsa SSH key → password manager backup
• S3 buckets: hermes-vps-backups, mikrotik-ccr-backups, itpropartner-backups (Wasabi us-east-1)

Recovery Scenarios

Hermes dies (app1) — auto-failover ~2 min (manual: 30 min)

Auto: app1-bu detects outage, syncs S3, starts Hermes. No action needed.

Manual fallback if auto-failover didn't trigger:

1. Verify from app1-bu: hermes cron list or cat /var/log/hermes-standby-watchdog.log
2. If Hermes stopped and didn't fail over: hermes gateway start
3. Or reboot app1-bu — boot restore script runs S3 sync and starts Hermes

Emergency failover (manual) — 2 min

1. SSH into app1-bu: ssh -i ~/.ssh/wisp_rsa root@5.161.114.8
2. Sync latest state: aws s3 sync s3://hermes-vps-backups/live/ ~/.hermes/ --endpoint-url https://s3.us-east-1.wasabisys.com --exclude "*.lock" --exclude "node/*" --exclude "bin/*" --exclude "cache/*" --exclude "sandboxes/*"
3. Start Hermes: hermes gateway start

AI/Apps server dies (app2) — 2-4 hr recovery

1. Order new RS 2000/4000 in Manassas
2. Install Docker + base software
3. Restore each app from s3://hermes-vps-backups/apps/<app-name>/latest/
4. Restart each Docker compose stack
5. Update Cloudflare DNS per subdomain

Total loss — 4-6 hr recovery

1. Provision two new Netcup servers
2. Recover Hermes first: aws s3 sync s3://hermes-vps-backups/live/ ~/.hermes/ --endpoint-url https://s3.us-east-1.wasabisys.com --exclude "*.lock"
3. Recover app2 second
4. Restore VPN, router backup jobs
5. Point all Cloudflare DNS records

Data recovery (any scenario)

• 15-min checkpoint in s3://hermes-vps-backups/live/ for Hermes state
• Router configs: 365-day retention in mikrotik-ccr-backups
• S3 versioning on all buckets prevents accidental overwrite/deletion
• Lifecycle policy: 365-day retention, then expire

DR scripts on S3

All in s3://hermes-vps-backups/standby/:

• hermes-standby-restore.sh — Boot-time restore (for Hetzner rescue mode deployment)
• hermes-standby-watchdog.sh — 10-min cron health check, triggers failover
• hermes-standby.service — systemd unit for boot-time restore
• DR-PLAN.md — This document
• recovery-bundle-2026-07-05.md — Full data dump (configs, credentials, conversation history)