ippadmin/disaster-recovery
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial DR plan — v3 with per-server RTO/RPO, recovery manual, issue log

Browse Source
This commit is contained in:
root
2026-07-15 17:41:23 -04:00
commit 4ca8be4b6a
5 changed files with 1827 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+36
View File
@@ -0,0 +1,36 @@
# Disaster Recovery Plan
**Version:** 3.0 (Jul 10, 2026)
**Status:** Active
## Documents
| File | Purpose |
|---|---|
| `server-dr-plans-v3.md` | Per-server DR plans with RTO/RPO targets |
| `itpp-recovery-manual.md` | Full infrastructure recovery manual |
| `dr-issue-log.md` | Permanent issue tracking — all audit findings |
| `network-and-service-endpoints.md` | IPs, ports, URLs, S3 buckets |
## Quick Links
- **Warm standby:** app1-bu (5.161.114.8) checks Core every 5 min, auto-failover
- **S3 backups:** hermes-vps-backups, itpropartner-system-configs, itpropartner-docker-volumes, mikrotik-ccr-backups
- **Failover RTO:** 5 min committed (actual ~3 min)
- **Core RPO:** 15 min (Hermes state) / 24h (Docker data)
## Architecture
```
Core (netcup RS2000, 152.53.192.33)
├── Active Hermes + gateway
└── S3 sync every 15 min
│
▼
Wasabi S3 (hermes-vps-backups/live/)
│
▼
app1-bu (Hetzner CPX11, 5.161.114.8)
├── Dormant Hermes
└── Sync from S3 every 10 min
```
dr-issue-log.md
+82
View File
@@ -0,0 +1,82 @@
# DR Issue Log — Hermes Infrastructure
Tracked issues found during DR audits. Each entry: date, problem, root cause, fix applied, verification.
---
## 2026-07-08 — Initial Full DR Audit
### DR-001: Caddyfile not included in backup scripts
**Problem:** /etc/caddy/Caddyfile was not copied by hermes-backup.sh or hermes-live-sync.sh. If Core server fails, the full Caddy reverse proxy config would need to be rebuilt from scratch.
**Root cause:** Backup scripts were written to cover hermes config and user directories but omitted system-level config files.
**Fix:** Added /etc/caddy/Caddyfile to hermes-backup.sh and hermes-live-sync.sh (with DR FIX: comments and dates).
**Status:** ✅ Fixed & verified 2026-07-08
**Verified by:** bash -n syntax check + subagent confirmed all paths in script
### DR-002: Systemd service files backed up to wrong path
**Problem:** hermes-backup.sh collected systemd services from ~/.config/systemd/user/ instead of /etc/systemd/system/. The real service files (/etc/systemd/system/hermes-agent.service, shark-game.service, etc.) were not backed up.
**Root cause:** Backup script path pointed to user-level systemd vs system-level.
**Fix:** Corrected path in hermes-backup.sh to /etc/systemd/system/*.service. Also updated the embedded restore.sh heredoc.
**Status:** ✅ Fixed & verified 2026-07-08
**Verified by:** bash -n syntax check + subagent confirmed paths in script
### DR-003: migration-creds.txt not in backup scope
**Problem:** /root/.hermes/migration-creds.txt existed but wasn't referenced by any backup script.
**Root cause:** Added after backup script was written, never included in scope.
**Fix:** Added to hermes-backup.sh file list with chmod 600 restore.
**Status:** ✅ Fixed & verified 2026-07-08
**Verified by:** bash -n syntax check + subagent confirmed file referenced
### DR-004: dre-temp-passwords.txt exposed at 644 permissions
**Problem:** /root/.hermes/references/dre-temp-passwords.txt was readable by all users (644) instead of owner-only (600).
**Root cause:** Script created the file without explicit permission setting.
**Fix:** chmod 600
**Status:** ✅ Fixed 2026-07-08
**Verified by:** ls -la confirms 600
### DR-005: migration-creds.txt at 644 permissions
**Problem:** Same issue as DR-004 — migration-creds.txt was 644.
**Root cause:** Written without explicit permission setting.
**Fix:** chmod 600
**Status:** ✅ Fixed 2026-07-08
**Verified by:** ls -la confirms 600
### DR-006: Full backup had no cron job (last ran Jul 5)
**Problem:** hermes-full-backup.tar.gz last uploaded to S3 on Jul 5. The daily 5AM cron had no scheduled job — the backup script existed but nothing was calling it.
**Root cause:** Cron job was never created for the full backup script. The hermes-live-sync covered configs every 15 min but full archive was orphaned.
**Fix:** Created cron job `hermes-full-backup` at `0 5 * * *` calling `run-hermes-backup.sh` wrapper.
**Status:** ✅ Fixed & verified 2026-07-08
**Verified by:** Cron job created and scheduled for next 5 AM ET
### DR-007: home-router-daily-backup cron error
**Problem:** Cron job errored at 06:00 today. SSH export on router produced a stuck .in_progress file that never completed.
**Root cause:** Cron was using home-router-backup.sh (old WireGuard tunnel script) instead of the proper run-wisp-backup.sh pipeline. Stuck export files blocked SCP.
**Fix:** Changed cron to use run-wisp-backup.sh. Cleaned stuck .in_progress files from router. Missing packages (paramiko, xl2tpd, strongSwan) installed.
**Status:** ✅ Fixed & verified 2026-07-08
**Verified by:** Backup ran end-to-end, config uploaded to S3
### DR-008: MikroTik CCR backup stale (last Jul 5)
**Problem:** mikrotik-ccr-backups S3 bucket had no uploads since Jul 5.
**Root cause (two causes):** (1) wisp-backup.py failed because paramiko wasn't installed. (2) tower IP in config.yaml was 192.168.88.1 (LAN) but SSH is restricted to WireGuard tunnel network 10.77.0.0/24.
**Fix:** Installed paramiko v5.0.0. Updated tower IP to 10.77.0.2 in wisp-backup/config.yaml. Installed missing VPN stack (xl2tpd, strongSwan).
**Status:** ✅ Fixed & verified 2026-07-08
**Verified by:** Backup ran end-to-end, config uploaded to S3
### DR-009: app1-bu warm vs cold docs mismatch
**Problem:** DR plan doc said "offline, boots on demand" but server is running (3 days uptime).
**Root cause:** DR plan doc was outdated — actual design is warm standby (always on, Hermes dormant).
**Fix:** Updated DR plan to reflect warm standby design. Server stays running.
**Status:** ✅ Resolved — not a bug, docs were wrong
### DR-010: Failover timing change (per Germaine's direction)
**Change:** Check interval from every 10 min → every 5 min. Constant check window from 3 min → 2 min (30s × 4 cycles).
**Rationale:** Faster detection, shorter failover.
**Status:** ✅ Fixed & verified 2026-07-08
**Verified by:** Cron changed to */5. Watchdog timing updated on app1-bu.
### DR-011: AWS CLI missing on app1-bu standby
**Problem:** /opt/awscli-venv was missing on app1-bu. Failover could not pull fresh state from S3.
**Root cause:** python3-venv package not installed on standby server.
**Fix:** Installed python3-venv, created venv, installed awscli. Created hermes-standby-sync.sh script with */10 cron.
**Status:** ✅ Fixed & verified 2026-07-08
**Verified by:** First sync ran end-to-end. Config.yaml timestamp went from Jul 5 → Jul 8 17:10.
itpp-recovery-manual.md
+1347
View File
File diff suppressed because it is too large Load Diff
network-and-service-endpoints.md
+70
View File
@@ -0,0 +1,70 @@
# IT Pro Partner — Network & Service Endpoints
## DNS & Hosting
| Domain | Nameservers | Hosting |
|--------|------------|---------|
| itpropartner.com | SiteGround (ns1/ns2.siteground.net) | SiteGround |
| iamgmb.com | Cloudflare | MXroute (email) |
| germainebrown.com | Cloudflare | MXroute (email) |
| debtrecoveryexperts.com | Cloudflare | SiteGround/self-hosted |
## Core (netcup RS 2000 G12)
| Service | Port | URL / Notes |
|---------|------|-------------|
| Hermes gateway | — | Telegram bot |
| Hermes assistant (PWA) | 8082 | app.itpropartner.com |
| Shark game API | 8083 | shark.iamgmb.com |
| Chromium CDP | 9222 | local browser automation |
| SMTP relay | 2525 | mail.germainebrown.com (STARTTLS) |
| Caddy | 443 | Reverse proxy for all sites |
| Tailscale | — | vaultwarden.tailc2f3b0.ts.net |
## Hetzner Servers
| Name | Type | IP | Purpose |
|------|------|----|---------|
| wphost02 | CPX21 | 5.161.62.38 | Apex WP + RunCloud |
| app1-bu | CPX11 | 5.161.114.8 | Hermes warm standby |
| tony-vps | CPX21 | 87.99.159.142 | Tony's Hermes |
| unms | CPX21 | 5.161.225.131 | UISP/UNMS |
| unifi | CPX21 | 178.156.131.57 | UniFi controller |
| hudu | CPX21 | 178.156.130.130 | IT documentation |
| ai | CPX41 | 178.156.167.181 | admin-ai LLM proxy |
| fleet | CPX11 | 178.156.149.32 | Fleet tracker |
| docker | CPX11 | 178.156.168.35 | Docker host |
| app1 (old) | CPX11 | 87.99.144.163 | Legacy |
## API Endpoints
| API | Auth | Base URL |
|-----|------|----------|
| Hetzner Cloud | Bearer token | https://api.hetzner.cloud/v1 |
| Cloudflare | Bearer token | https://api.cloudflare.com/client/v4 |
| admin-ai | Bearer token | https://admin-ai.itpropartner.com/v1 |
| netcup SCP | Keycloak OIDC | https://www.servercontrolpanel.de/scp-core/api/v1 |
| Firecrawl | Bearer token | https://api.firecrawl.com/v1 |
| Wasabi S3 | AWS4-HMAC-SHA256 | s3.wasabisys.com |
## S3 Backup Buckets
| Bucket | Prefix | Purpose |
|--------|--------|---------|
| hermes-vps-backups | live/ | Active Hermes state (15 min sync) |
| hermes-vps-backups | standby/ | Standby recovery bundle |
| hermes-vps-backups | hermes-full-backup/ | Daily 5 AM full tarball |
| itpropartner-backups | — | Portal files & public data |
| mikrotik-ccr-backups | wisp-backups/ | Router configs |
## External Services
| Service | Login URL | Notes |
|---------|-----------|-------|
| netcup CCP | customercontrolpanel.de | Account management |
| netcup SCP | servercontrolpanel.de | Server management |
| Hetzner Cloud | console.hetzner.cloud | Hetzner project |
| Cloudflare | dash.cloudflare.com | DNS, Access, Registrar |
| MXroute | mxlogin.com | Email hosting |
| SiteGround | siteground.com | WP hosting |
| Wasabi | console.wasabisys.com | S3-compatible storage |
server-dr-plans-v3.md
+292
View File
@@ -0,0 +1,292 @@
# IT Pro Partner — Disaster Recovery Plan v3 (Consolidated)
**Author:** Sho'Nuff / Network Services Team, incorporating third-party DR review
**Date:** July 10, 2026
**Status:** LIVE — replaces v1.0
**Supersedes:** server-dr-plans.md v1.0, server-provisioning-standard-v1.md, hermes-dr-plan-v2.md
---
## Table of Contents
1. Architecture & Shared Dependencies
2. Server Standards
3. Backup Strategy
4. Failover / Failback
5. Monitoring & Alerting
6. Restore Runbooks
7. Testing & Validation
8. Security
9. Immediate Action Items
10. Appendix: Third-Party Review Responses
---
## 1. Architecture & Shared Dependencies
```
Internet -- Cloudflare -- Caddy -- Service (HTTP/HTTPS)
|
+-----------+-----------+
| | |
SSH/WG Tailscale Management
(itpp- (Core <-> VPN (WireGuard
infra) app1-bu) to home router)
| | |
+-----------+-----------+
|
+-----------+-----------+
| |
S3 Primary S3 Secondary
(Wasabi us-east-1) (Wasabi us-west-2
versioning ON or Backblaze B2)
object lock ON object lock ON
```
### Shared Dependencies
| Dependency | Primary | Backup / Fallback | Credentials Location |
|---|---|---|---|
| SSH access | itpp-infra key (Hetzner vault) | Rescue mode / console | ~/.hermes/.env + Hudu |
| DNS | Cloudflare (API token in .env) | Manual via web UI | ~/.hermes/.env |
| S3 backup (primary) | Wasabi us-east-1 | Wasabi us-west-2 (to be created) | ~/.aws/credentials |
| S3 backup (secondary) | Wasabi us-west-2 | Backblaze B2 (evaluate) | TBD |
| SMTP relay | mail.germainebrown.com:2525 | Direct MXroute (port 465/587) | ~/.hermes/.env |
| Auth/SSO | Cloudflare Access | Local portal auth fallback | Cloudflare dashboard |
| Monitoring | Prometheus node_exporter | Uptime Kuma (docker box) | N/A (pull model) |
| VPN management | WireGuard (wg0, port 51821) | Tailscale direct tunnel | /etc/wireguard/ |
---
## 2. Server Standards
### 2.1 Tier Definitions
| Tier | Spec | OS | Use | Cost |
|---|---|---|---|---|
| **Standard** | RS 4000 G12 (12C/32G/1TB) | Debian 13 | app1, app2, app3 | ~$44/mo |
| **Light** | RS 2000 G12 (8C/16G/512GB) | Debian 13 | Core | ~$24/mo |
| **Standby** | CPX21 (4C/8G/80GB) | Debian 13 | app1-bu | ~$14/mo |
| **Legacy** | Variable Hetzner | Debian 12 | Existing (migrate on rebuild) | ~$8-44/mo |
### 2.2 Base Install -- Verified Checklist
```bash
# S1: hostname + timezone
hostnamectl set-hostname <name>
timedatectl set-timezone America/New_York
# S2: system update
apt update && apt upgrade -y
# S3: security baseline
apt install -y fail2ban ufw
ufw default deny incoming; ufw default allow outgoing
ufw allow ssh; ufw --force enable
# S4: monitoring
apt install -y prometheus-node-exporter
# S5: standard user + key
adduser ippadmin && usermod -aG sudo ippadmin
echo "ssh-ed25519 AAA... itpp-infra" >> /home/ippadmin/.ssh/authorized_keys
# S6: harden SSH
sed -i 's/^#PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config
sed -i 's/^PermitRootLogin yes/PermitRootLogin prohibit-password/' /etc/ssh/sshd_config
systemctl restart sshd
# S7: Docker (if needed)
curl -fsSL https://get.docker.com | bash
apt install -y docker-compose-plugin
```
### 2.3 Application Data Paths
Every server must document:
- What data lives where (volumes, bind mounts, databases, uploads, .env files)
- What can be rebuilt vs what must be restored
- Secrets location
- Caddy/nginx configs
- Cron jobs
- External API dependencies
---
## 3. Backup Strategy
### 3.1 Backup Matrix
| Data Type | Frequency | Target | Retention | Status |
|---|---|---|---|---|
| Hermes state | Every 15 min | S3 primary (live/) | 48 hours | ✅ LIVE |
| Hermes full | Daily 5 AM | S3 primary (full/) | 90 days | ✅ LIVE |
| Memory snapshots | Every 10 min | S3 primary (memories/) | 48 hours | ✅ LIVE |
| Memory history | Every 10 min | S3 primary (memories/history/) | 90 days | ✅ LIVE |
| Router configs | Daily 6 AM | S3 primary (mikrotik/) | 90 days | ✅ LIVE |
| System configs | Daily | S3 primary (<server>/) | 90 days | 🔲 NOT AUTOMATED |
| Docker volumes | Daily | S3 primary (<server>/) | 90 days | ❌ NOT DONE |
| Database dumps | Every 6 hours | S3 primary (<server>/) | 90 days | ❌ NOT DONE |
| Application data | Per-app | S3 primary (<server>/) | 90 days | 🔲 PARTIAL |
| S3 secondary | Daily sync | Wasabi us-west-2 | 90 days | ❌ NOT SET UP |
### 3.2 Backup Security
- **Versioning:** ON (all 3 buckets)
- **Object Lock:** NOT enabled
- **Delete protection:** NOT configured
- **Backup credentials:** Admin-level (can delete) -- NOT write-only
- **Encryption:** Server-side only (Wasabi default)
---
## 4. Failover / Failback
### 4.1 Warm Standby Timing (Corrected)
**Old:** Check every 5 min, failover after 2 min unreachable (impossible -- can't detect 2-min outage at 5-min intervals)
**New:** Check every 30 seconds, failover after 4 failed checks (~2 min detection) + 45-90s sync = ~3-3.5 min actual, 5 min committed RTO
### 4.2 Health Check -- Two Layers
**Layer 1 (Ping):** ICMP to Core (152.53.192.33) every 30s
**Layer 2 (HTTP):** GET /healthz on Tailscale IP -- checks Gateway heartbeat, SQLite PRAGMA, local Ollama
### 4.3 Split-Brain Prevention
S3 active-lock.json mechanism:
```json
{"active_node": "core", "heartbeat": "<timestamp>", "lock_holder": "core"}
```
Core writes heartbeat every 30s while healthy. Failover only proceeds after:
1. Core unreachable for 4 consecutive checks (app1-bu local)
2. External monitor independently confirms Core down
3. active-lock.json heartbeat stale (>90s)
4. app1-bu writes itself as lock_holder
5. Gateway starts
6. Operator alerted
If Core can be reached for fencing, app1-bu SSHes in and stops/masks Hermes before taking over.
### 4.4 Failback Protocol
**No auto-failback.** Sequence:
1. Confirm app1-bu is active node
2. Keep Core stopped -- no auto-start
3. Restore Core OS + stack
4. Sync latest state from app1-bu/S3 into Core
5. Validate sync timestamp
6. Start Core in passive mode
7. Stop app1-bu gateway
8. Update active-lock.json: "active_node": "core"
9. Start Core gateway
10. 5 consecutive health checks pass
11. Release standby lock
12. Monitor 30 min before closing
**Rollback:** If Core fails health checks mid-failback, revert immediately -- restore app1-bu as active, keep Core in maintenance.
### 4.5 Provider/Account Outage
If entire netcup account is unavailable:
1. app1-bu (Hetzner) takes over
2. DNS records pointed to app1-bu IP
3. No auto-failback -- human decision
---
## 5. Per-Server RTO/RPO Targets
| System | RTO | RPO | Notes |
|---|---|---|---|
| Core (Hermes) | 5 min | 15 min | Via warm standby (app1-bu) |
| app1-bu (standby) | N/A | <=15 min | Recovery path for Core |
| app1/2/3 (Standard tier) | 60 min | 24h | Rebuild from S3 + docker compose |
| wphost02 (WordPress) | 2h | 24h | DB restore from dump |
| fleettracker360 | 2h | 24h | DB restore from dump |
| Legacy Hetzner hosts | 4h | 24h | Via rescue mode or S3 |
---
## 6. Immediate Action Items
### P0 -- This Week
| # | Action | Details |
|---|---|---|
| 1 | Clean AI server disk | 92% full -- purge old Docker images/logs |
| 2 | Create S3 write-only IAM user | PutObject only, no delete/list |
| 3 | Enable S3 Object Lock | 7-day minimum retention |
| 4 | Deploy Docker volume backup | All Docker hosts |
| 5 | Deploy DB dump cron | wphost02 + fleettracker360 (MariaDB) |
| 6 | Deploy 30s watchdog | Currently documented, need deployment |
### P1 -- Next 2 Weeks
| # | Action |
|---|---|
| 7 | Document Tony's Hermes config + backup to S3 |
| 8 | Provision secondary S3 bucket (Wasabi us-west-2) |
| 9 | Extend snapshot retention from 48h to 30 days |
| 10 | Add backup success/failure monitoring to ops portal |
### P2 -- Next 30 Days
| # | Action |
|---|---|
| 11 | Full DR simulation for Core failover |
| 12 | Monthly restore testing schedule begins |
| 13 | Deploy Prometheus + Grafana stack |
| 14 | Add disk threshold alerts for all servers |
| 15 | Begin netcup migration |
---
## 7. Restore Runbooks (Summary)
Full per-server restore runbooks with exact commands are in the Per-Server Runbooks document (companion doc 3 of the v3 DR stack). Each covers:
- Purpose + dependencies
- DNS records, firewall ports, backup path
- Required secrets
- Step-by-step restore commands
- DB restore with pre-import dump verification
- Docker volume restore
- Health checks
- Known failure modes
Hosts covered: Core, app1-bu, app1/2/3, wphost02, fleettracker360, Legacy Hetzner hosts.
---
## 8. Third-Party Review Responses
### Reviewer 1 Findings -- Addressed
| Finding | Addressed In |
|---|---|
| Failover timing mismatch (5min check did not equal 2min RTO) | Section 4.1 -- Corrected to 30s checks |
| Backup compliance gap (standard says daily but not happening) | Section 3.1 -- Gap closure plan |
| AI server 92% disk critical | Section 6 -- P0 action item #1 |
| Memory consolidation data loss risk | Separate docs per priority-tagging proposal |
| Tony's Hermes undocumented | Section 6 -- P1 action item #7 |
### Reviewer 2 Findings -- Addressed
| Finding | Addressed In |
|---|---|
| S3 single point of failure | Section 1 + Section 3.1 -- Two-region plan |
| Backup frequency vs RPO mismatch | Section 5 -- RPO per data type |
| Restore procedures not detailed | Section 7 -- Full runbooks reference |
| Failback under-defined | Section 4.4 -- Protocol |
| No backup restore testing | Separate testing schedule doc |
| Retention too short (48h snapshots) | Section 6 -- P1 action item #9 |
| Backup security not described | Section 3.2 |
| Application backup details vague | Section 2.3 |
| Secrets recovery not documented | Section 1 (shared deps table) |
| Provider/account outage missing | Section 4.5 |
---
*This plan is ACTIVE. Review quarterly or after any infrastructure change.*