Initial skills documentation — 25 categories, all SKILL.md + references + scripts
This commit is contained in:
root
@@ -0,0 +1,292 @@
|
||||
# IT Pro Partner — Disaster Recovery Plan v3 (Consolidated)
|
||||
|
||||
**Author:** Sho'Nuff / Network Services Team, incorporating third-party DR review
|
||||
**Date:** July 10, 2026
|
||||
**Status:** LIVE — replaces v1.0
|
||||
**Supersedes:** server-dr-plans.md v1.0, server-provisioning-standard-v1.md, hermes-dr-plan-v2.md
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
1. Architecture & Shared Dependencies
|
||||
2. Server Standards
|
||||
3. Backup Strategy
|
||||
4. Failover / Failback
|
||||
5. Monitoring & Alerting
|
||||
6. Restore Runbooks
|
||||
7. Testing & Validation
|
||||
8. Security
|
||||
9. Immediate Action Items
|
||||
10. Appendix: Third-Party Review Responses
|
||||
|
||||
---
|
||||
|
||||
## 1. Architecture & Shared Dependencies
|
||||
|
||||
```
|
||||
Internet -- Cloudflare -- Caddy -- Service (HTTP/HTTPS)
|
||||
|
|
||||
+-----------+-----------+
|
||||
| | |
|
||||
SSH/WG Tailscale Management
|
||||
(itpp- (Core <-> VPN (WireGuard
|
||||
infra) app1-bu) to home router)
|
||||
| | |
|
||||
+-----------+-----------+
|
||||
|
|
||||
+-----------+-----------+
|
||||
| |
|
||||
S3 Primary S3 Secondary
|
||||
(Wasabi us-east-1) (Wasabi us-west-2
|
||||
versioning ON or Backblaze B2)
|
||||
object lock ON object lock ON
|
||||
```
|
||||
|
||||
### Shared Dependencies
|
||||
|
||||
| Dependency | Primary | Backup / Fallback | Credentials Location |
|
||||
|---|---|---|---|
|
||||
| SSH access | itpp-infra key (Hetzner vault) | Rescue mode / console | ~/.hermes/.env + Hudu |
|
||||
| DNS | Cloudflare (API token in .env) | Manual via web UI | ~/.hermes/.env |
|
||||
| S3 backup (primary) | Wasabi us-east-1 | Wasabi us-west-2 (to be created) | ~/.aws/credentials |
|
||||
| S3 backup (secondary) | Wasabi us-west-2 | Backblaze B2 (evaluate) | TBD |
|
||||
| SMTP relay | mail.germainebrown.com:2525 | Direct MXroute (port 465/587) | ~/.hermes/.env |
|
||||
| Auth/SSO | Cloudflare Access | Local portal auth fallback | Cloudflare dashboard |
|
||||
| Monitoring | Prometheus node_exporter | Uptime Kuma (docker box) | N/A (pull model) |
|
||||
| VPN management | WireGuard (wg0, port 51821) | Tailscale direct tunnel | /etc/wireguard/ |
|
||||
|
||||
---
|
||||
|
||||
## 2. Server Standards
|
||||
|
||||
### 2.1 Tier Definitions
|
||||
|
||||
| Tier | Spec | OS | Use | Cost |
|
||||
|---|---|---|---|---|
|
||||
| **Standard** | RS 4000 G12 (12C/32G/1TB) | Debian 13 | app1, app2, app3 | ~$44/mo |
|
||||
| **Light** | RS 2000 G12 (8C/16G/512GB) | Debian 13 | Core | ~$24/mo |
|
||||
| **Standby** | CPX21 (4C/8G/80GB) | Debian 13 | app1-bu | ~$14/mo |
|
||||
| **Legacy** | Variable Hetzner | Debian 12 | Existing (migrate on rebuild) | ~$8-44/mo |
|
||||
|
||||
### 2.2 Base Install -- Verified Checklist
|
||||
|
||||
```bash
|
||||
# S1: hostname + timezone
|
||||
hostnamectl set-hostname <name>
|
||||
timedatectl set-timezone America/New_York
|
||||
|
||||
# S2: system update
|
||||
apt update && apt upgrade -y
|
||||
|
||||
# S3: security baseline
|
||||
apt install -y fail2ban ufw
|
||||
ufw default deny incoming; ufw default allow outgoing
|
||||
ufw allow ssh; ufw --force enable
|
||||
|
||||
# S4: monitoring
|
||||
apt install -y prometheus-node-exporter
|
||||
|
||||
# S5: standard user + key
|
||||
adduser ippadmin && usermod -aG sudo ippadmin
|
||||
echo "ssh-ed25519 AAA... itpp-infra" >> /home/ippadmin/.ssh/authorized_keys
|
||||
|
||||
# S6: harden SSH
|
||||
sed -i 's/^#PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config
|
||||
sed -i 's/^PermitRootLogin yes/PermitRootLogin prohibit-password/' /etc/ssh/sshd_config
|
||||
systemctl restart sshd
|
||||
|
||||
# S7: Docker (if needed)
|
||||
curl -fsSL https://get.docker.com | bash
|
||||
apt install -y docker-compose-plugin
|
||||
```
|
||||
|
||||
### 2.3 Application Data Paths
|
||||
|
||||
Every server must document:
|
||||
- What data lives where (volumes, bind mounts, databases, uploads, .env files)
|
||||
- What can be rebuilt vs what must be restored
|
||||
- Secrets location
|
||||
- Caddy/nginx configs
|
||||
- Cron jobs
|
||||
- External API dependencies
|
||||
|
||||
---
|
||||
|
||||
## 3. Backup Strategy
|
||||
|
||||
### 3.1 Backup Matrix
|
||||
|
||||
| Data Type | Frequency | Target | Retention | Status |
|
||||
|---|---|---|---|---|
|
||||
| Hermes state | Every 15 min | S3 primary (live/) | 48 hours | ✅ LIVE |
|
||||
| Hermes full | Daily 5 AM | S3 primary (full/) | 90 days | ✅ LIVE |
|
||||
| Memory snapshots | Every 10 min | S3 primary (memories/) | 48 hours | ✅ LIVE |
|
||||
| Memory history | Every 10 min | S3 primary (memories/history/) | 90 days | ✅ LIVE |
|
||||
| Router configs | Daily 6 AM | S3 primary (mikrotik/) | 90 days | ✅ LIVE |
|
||||
| System configs | Daily | S3 primary (<server>/) | 90 days | 🔲 NOT AUTOMATED |
|
||||
| Docker volumes | Daily | S3 primary (<server>/) | 90 days | ❌ NOT DONE |
|
||||
| Database dumps | Every 6 hours | S3 primary (<server>/) | 90 days | ❌ NOT DONE |
|
||||
| Application data | Per-app | S3 primary (<server>/) | 90 days | 🔲 PARTIAL |
|
||||
| S3 secondary | Daily sync | Wasabi us-west-2 | 90 days | ❌ NOT SET UP |
|
||||
|
||||
### 3.2 Backup Security
|
||||
|
||||
- **Versioning:** ON (all 3 buckets)
|
||||
- **Object Lock:** NOT enabled
|
||||
- **Delete protection:** NOT configured
|
||||
- **Backup credentials:** Admin-level (can delete) -- NOT write-only
|
||||
- **Encryption:** Server-side only (Wasabi default)
|
||||
|
||||
---
|
||||
|
||||
## 4. Failover / Failback
|
||||
|
||||
### 4.1 Warm Standby Timing (Corrected)
|
||||
|
||||
**Old:** Check every 5 min, failover after 2 min unreachable (impossible -- can't detect 2-min outage at 5-min intervals)
|
||||
|
||||
**New:** Check every 30 seconds, failover after 4 failed checks (~2 min detection) + 45-90s sync = ~3-3.5 min actual, 5 min committed RTO
|
||||
|
||||
### 4.2 Health Check -- Two Layers
|
||||
|
||||
**Layer 1 (Ping):** ICMP to Core (152.53.192.33) every 30s
|
||||
**Layer 2 (HTTP):** GET /healthz on Tailscale IP -- checks Gateway heartbeat, SQLite PRAGMA, local Ollama
|
||||
|
||||
### 4.3 Split-Brain Prevention
|
||||
|
||||
S3 active-lock.json mechanism:
|
||||
|
||||
```json
|
||||
{"active_node": "core", "heartbeat": "<timestamp>", "lock_holder": "core"}
|
||||
```
|
||||
|
||||
Core writes heartbeat every 30s while healthy. Failover only proceeds after:
|
||||
1. Core unreachable for 4 consecutive checks (app1-bu local)
|
||||
2. External monitor independently confirms Core down
|
||||
3. active-lock.json heartbeat stale (>90s)
|
||||
4. app1-bu writes itself as lock_holder
|
||||
5. Gateway starts
|
||||
6. Operator alerted
|
||||
|
||||
If Core can be reached for fencing, app1-bu SSHes in and stops/masks Hermes before taking over.
|
||||
|
||||
### 4.4 Failback Protocol
|
||||
|
||||
**No auto-failback.** Sequence:
|
||||
1. Confirm app1-bu is active node
|
||||
2. Keep Core stopped -- no auto-start
|
||||
3. Restore Core OS + stack
|
||||
4. Sync latest state from app1-bu/S3 into Core
|
||||
5. Validate sync timestamp
|
||||
6. Start Core in passive mode
|
||||
7. Stop app1-bu gateway
|
||||
8. Update active-lock.json: "active_node": "core"
|
||||
9. Start Core gateway
|
||||
10. 5 consecutive health checks pass
|
||||
11. Release standby lock
|
||||
12. Monitor 30 min before closing
|
||||
|
||||
**Rollback:** If Core fails health checks mid-failback, revert immediately -- restore app1-bu as active, keep Core in maintenance.
|
||||
|
||||
### 4.5 Provider/Account Outage
|
||||
|
||||
If entire netcup account is unavailable:
|
||||
1. app1-bu (Hetzner) takes over
|
||||
2. DNS records pointed to app1-bu IP
|
||||
3. No auto-failback -- human decision
|
||||
|
||||
---
|
||||
|
||||
## 5. Per-Server RTO/RPO Targets
|
||||
|
||||
| System | RTO | RPO | Notes |
|
||||
|---|---|---|---|
|
||||
| Core (Hermes) | 5 min | 15 min | Via warm standby (app1-bu) |
|
||||
| app1-bu (standby) | N/A | <=15 min | Recovery path for Core |
|
||||
| app1/2/3 (Standard tier) | 60 min | 24h | Rebuild from S3 + docker compose |
|
||||
| wphost02 (WordPress) | 2h | 24h | DB restore from dump |
|
||||
| fleettracker360 | 2h | 24h | DB restore from dump |
|
||||
| Legacy Hetzner hosts | 4h | 24h | Via rescue mode or S3 |
|
||||
|
||||
---
|
||||
|
||||
## 6. Immediate Action Items
|
||||
|
||||
### P0 -- This Week
|
||||
|
||||
| # | Action | Details |
|
||||
|---|---|---|
|
||||
| 1 | Clean AI server disk | 92% full -- purge old Docker images/logs |
|
||||
| 2 | Create S3 write-only IAM user | PutObject only, no delete/list |
|
||||
| 3 | Enable S3 Object Lock | 7-day minimum retention |
|
||||
| 4 | Deploy Docker volume backup | All Docker hosts |
|
||||
| 5 | Deploy DB dump cron | wphost02 + fleettracker360 (MariaDB) |
|
||||
| 6 | Deploy 30s watchdog | Currently documented, need deployment |
|
||||
|
||||
### P1 -- Next 2 Weeks
|
||||
|
||||
| # | Action |
|
||||
|---|---|
|
||||
| 7 | Document Tony's Hermes config + backup to S3 |
|
||||
| 8 | Provision secondary S3 bucket (Wasabi us-west-2) |
|
||||
| 9 | Extend snapshot retention from 48h to 30 days |
|
||||
| 10 | Add backup success/failure monitoring to ops portal |
|
||||
|
||||
### P2 -- Next 30 Days
|
||||
|
||||
| # | Action |
|
||||
|---|---|
|
||||
| 11 | Full DR simulation for Core failover |
|
||||
| 12 | Monthly restore testing schedule begins |
|
||||
| 13 | Deploy Prometheus + Grafana stack |
|
||||
| 14 | Add disk threshold alerts for all servers |
|
||||
| 15 | Begin netcup migration |
|
||||
|
||||
---
|
||||
|
||||
## 7. Restore Runbooks (Summary)
|
||||
|
||||
Full per-server restore runbooks with exact commands are in the Per-Server Runbooks document (companion doc 3 of the v3 DR stack). Each covers:
|
||||
- Purpose + dependencies
|
||||
- DNS records, firewall ports, backup path
|
||||
- Required secrets
|
||||
- Step-by-step restore commands
|
||||
- DB restore with pre-import dump verification
|
||||
- Docker volume restore
|
||||
- Health checks
|
||||
- Known failure modes
|
||||
|
||||
Hosts covered: Core, app1-bu, app1/2/3, wphost02, fleettracker360, Legacy Hetzner hosts.
|
||||
|
||||
---
|
||||
|
||||
## 8. Third-Party Review Responses
|
||||
|
||||
### Reviewer 1 Findings -- Addressed
|
||||
|
||||
| Finding | Addressed In |
|
||||
|---|---|
|
||||
| Failover timing mismatch (5min check did not equal 2min RTO) | Section 4.1 -- Corrected to 30s checks |
|
||||
| Backup compliance gap (standard says daily but not happening) | Section 3.1 -- Gap closure plan |
|
||||
| AI server 92% disk critical | Section 6 -- P0 action item #1 |
|
||||
| Memory consolidation data loss risk | Separate docs per priority-tagging proposal |
|
||||
| Tony's Hermes undocumented | Section 6 -- P1 action item #7 |
|
||||
|
||||
### Reviewer 2 Findings -- Addressed
|
||||
|
||||
| Finding | Addressed In |
|
||||
|---|---|
|
||||
| S3 single point of failure | Section 1 + Section 3.1 -- Two-region plan |
|
||||
| Backup frequency vs RPO mismatch | Section 5 -- RPO per data type |
|
||||
| Restore procedures not detailed | Section 7 -- Full runbooks reference |
|
||||
| Failback under-defined | Section 4.4 -- Protocol |
|
||||
| No backup restore testing | Separate testing schedule doc |
|
||||
| Retention too short (48h snapshots) | Section 6 -- P1 action item #9 |
|
||||
| Backup security not described | Section 3.2 |
|
||||
| Application backup details vague | Section 2.3 |
|
||||
| Secrets recovery not documented | Section 1 (shared deps table) |
|
||||
| Provider/account outage missing | Section 4.5 |
|
||||
|
||||
---
|
||||
|
||||
*This plan is ACTIVE. Review quarterly or after any infrastructure change.*
|
||||
Reference in New Issue
Block a user