Initial skills documentation — 25 categories, all SKILL.md + references + scripts

2026-07-15 17:42:20 -04:00
commit 1b4fcd4427
976 changed files with 188344 additions and 0 deletions
@@ -0,0 +1,88 @@
+# Disaster Recovery Plan — IT Pro Partner
+
+**Author:** ShoNuff (Hermes AI Agent)
+**Date:** July 4, 2026
+**Keep this document in:** Password manager (Bitwarden/Hudu), printed copy, and anywhere reachable without server access.
+
+## Architecture
+
+| Box | Provider | Location | Specs | Monthly | Runs |
+|-----|----------|----------|-------|---------|------|
+| app1 | Netcup | Manassas, VA | RS 2000 G12 (8c, 16GB, 512GB) | ~$27 | Portal, Hermes, VPN |
+| app2 | Netcup | Manassas, VA | RS 4000 G12 (12c, 32GB, 1TB) | ~$43 | AI, Docker apps, WordPress |
+| app1-bu | Hetzner | EU | CPX11 (2c, 2GB, 40GB) | ~$5 | Standby Hermes only — auto-failover |
+
+## Failover architecture (app1 → app1-bu)
+
+```
+app1 (netcup, live)              app1-bu (Hetzner, standby)
+┌─────────────────────┐         ┌──────────────────────────┐
+│  Hermes (running)   │   ping  │  Hermes (off)            │
+│  S3 sync every 15m  │◄───────►│  Cron watchdog every 10m│
+│  Telegram connected  │ 152.53  │  Checks live box health  │
+└─────────────────────┘         └──────────────────────────┘
+         │                                │
+         ▼                                ▼
+  s3://hermes-vps-backups/live/ (15-min checkpoint — state.db, config, skills, cron, profiles)
+```
+
+**If app1 dies:**
+- app1-bu watchdog detects 3 failed pings over ~60s
+- Syncs latest state from `s3://hermes-vps-backups/live/` (RPO <15 min)
+- Starts Hermes gateway on app1-bu
+- Telegram reconnects automatically (same bot token)
+- Total failover: ~2 min
+
+## Critical Credentials (store externally, NOT on servers)
+
+- Netcup CCP login → password manager
+- Wasabi IAM keys → password manager
+- Hetzner API token → password manager
+- Hermes setup token → password manager
+- Cloudflare API token → Cloudflare dashboard
+- wisp_rsa SSH key → password manager backup
+- S3 buckets: hermes-vps-backups, mikrotik-ccr-backups, itpropartner-backups (Wasabi us-east-1)
+
+## Recovery Scenarios
+
+### Hermes dies (app1) — auto-failover ~2 min (manual: 30 min)
+**Auto:** app1-bu detects outage, syncs S3, starts Hermes. No action needed.
+
+**Manual fallback if auto-failover didn't trigger:**
+1. Verify from app1-bu: `hermes cron list` or `cat /var/log/hermes-standby-watchdog.log`
+2. If Hermes stopped and didn't fail over: `hermes gateway start`
+3. Or reboot app1-bu — boot restore script runs S3 sync and starts Hermes
+
+### Emergency failover (manual) — 2 min
+1. SSH into app1-bu: `ssh -i ~/.ssh/wisp_rsa root@5.161.114.8`
+2. Sync latest state: `aws s3 sync s3://hermes-vps-backups/live/ ~/.hermes/ --endpoint-url https://s3.us-east-1.wasabisys.com --exclude "*.lock" --exclude "node/*" --exclude "bin/*" --exclude "cache/*" --exclude "sandboxes/*"`
+3. Start Hermes: `hermes gateway start`
+
+### AI/Apps server dies (app2) — 2-4 hr recovery
+1. Order new RS 2000/4000 in Manassas
+2. Install Docker + base software
+3. Restore each app from `s3://hermes-vps-backups/apps/<app-name>/latest/`
+4. Restart each Docker compose stack
+5. Update Cloudflare DNS per subdomain
+
+### Total loss — 4-6 hr recovery
+1. Provision two new Netcup servers
+2. Recover Hermes first: `aws s3 sync s3://hermes-vps-backups/live/ ~/.hermes/ --endpoint-url https://s3.us-east-1.wasabisys.com --exclude "*.lock"`
+3. Recover app2 second
+4. Restore VPN, router backup jobs
+5. Point all Cloudflare DNS records
+
+### Data recovery (any scenario)
+- 15-min checkpoint in `s3://hermes-vps-backups/live/` for Hermes state
+- Router configs: 365-day retention in `mikrotik-ccr-backups`
+- S3 versioning on all buckets prevents accidental overwrite/deletion
+- Lifecycle policy: 365-day retention, then expire
+
+## DR scripts on S3
+
+All in `s3://hermes-vps-backups/standby/`:
+- `hermes-standby-restore.sh` — Boot-time restore (for Hetzner rescue mode deployment)
+- `hermes-standby-watchdog.sh` — 10-min cron health check, triggers failover
+- `hermes-standby.service` — systemd unit for boot-time restore
+- `DR-PLAN.md` — This document
+- `recovery-bundle-2026-07-05.md` — Full data dump (configs, credentials, conversation history)