Initial resurrection kit — 81 scripts, 62 references, configs, systemd units, crons, Docker compose files, Caddy config, master README

2026-07-15 17:45:36 -04:00
commit ae056eaf83
197 changed files with 26127 additions and 0 deletions
@@ -0,0 +1,98 @@
+# IT Pro Partner — Per-Server Disaster Recovery Plan (REDACTED)
+
+**Author:** Sho'Nuff / Network Services Team
+**Date:** July 9, 2026
+**Version:** 1.0 — Redacted for 3rd Party Review
+
+---
+
+## Architecture Overview
+
+```
+Internet ──► CDN ──► Reverse Proxy ──► Service (HTTP/HTTPS)
+                          │
+                    SSH/WG ──► Management plane (key-based auth)
+                          │
+                    S3-Compatible Storage ──► Backup plane (all servers)
+```
+
+**Shared dependencies:**
+- **SSH access:** Ed25519 key, deployed to all servers
+- **DNS:** Cloudflare (primary) / legacy provider (one domain)
+- **Backup storage:** S3-compatible (Wasabi, us-east-1)
+- **SMTP relay:** Authenticated relay (STARTTLS, port 2525)
+- **Auth/SSO:** Cloudflare Access (internal portals)
+- **Monitoring:** Prometheus node_exporter per server
+
+---
+
+## Server Inventory Summary
+
+**11 servers total:**
+- 1 Primary operations server (netcup, dedicated EPYC)
+- 1 AI/LLM inference server (Hetzner)
+- 1 WordPress/RunCloud server (Hetzner)
+- 3 Application servers (Hetzner — UISP/UNMS, Hudu, UniFi)
+- 3 Utility servers (Hetzner — n8n, Docker staging, fleet tracking)
+- 1 Standalone Hermes server (Hetzner — Tony)
+- 1 Warm standby server (Hetzner)
+
+---
+
+## DR Classification
+
+| Server | Tier | RTO Target | RPO Target | Restore Complexity |
+|--------|------|-----------|-----------|-------------------|
+| Primary Ops | Critical | 5 min (failover) | 15 min | Medium |
+| AI/LLM | Critical | 2 hours | 24 hours | High |
+| WordPress | Important | 4 hours | 24 hours | Low |
+| UISP/UNMS | Important | 4 hours | 24 hours | Medium |
+| IT Documentation | Important | 4 hours | 24 hours | Medium |
+| Network Controller | Important | 4 hours | 24 hours | Low |
+| Automation (n8n) | Standard | 8 hours | 24 hours | Low |
+| Docker Utility | Standard | 8 hours | 24 hours | Medium |
+| Fleet Tracking | Standard | 8 hours | 24 hours | Low |
+| Standalone Agent | Standard | Coordinate | 24 hours | Low |
+| Warm Standby | Standby | 2 min | 15 min | Low |
+
+---
+
+## Backup Strategy
+
+| Data | Frequency | Location | Retention |
+|------|-----------|----------|-----------|
+| System config | Daily | S3-compatible storage | 90 days |
+| Docker volumes | Daily | S3-compatible storage | 90 days |
+| Database dumps | Daily | S3-compatible storage | 90 days |
+| Application data | Per-application | S3-compatible storage | 90 days |
+| Agent state | Every 15 min | S3-compatible storage | 48 hours |
+
+---
+
+## Restore Procedure (Generic)
+
+```bash
+# 1. Provision replacement server (same tier or better)
+# 2. Install Debian (current stable)
+# 3. Deploy SSH key
+# 4. Install Docker (if applicable)
+# 5. Restore from backup storage:
+aws s3 cp s3://backup-bucket/<server>/latest/ /opt/restore/ --recursive
+# 6. Restore Docker stacks
+cd /opt/<service> && docker compose up -d
+# 7. Verify health endpoints return 200
+```
+
+---
+
+## Key Recommendations from Audit
+
+1. **AI server disk critical (92% full):** Needs cleanup before any migration
+2. **Docker volumes not backed up to S3:** All Docker hosts need volume backup automation
+3. **Database dumps not automated:** MariaDB on wphost02, fleettracker360 need scheduled dumps
+4. **Tony's Hermes:** Independent instance — document its config separately
+5. **Migration plan:** Consolidate Hetzner services to netcup per the server standard
+
+---
+
+*Full version with IPs, credentials references, and detailed restore commands is stored internally and available on request.*