Files
hermes-recovery/references/server-dr-plans-redacted.md
T

3.4 KiB

IT Pro Partner — Per-Server Disaster Recovery Plan (REDACTED)

Author: Sho'Nuff / Network Services Team Date: July 9, 2026 Version: 1.0 — Redacted for 3rd Party Review


Architecture Overview

Internet ──► CDN ──► Reverse Proxy ──► Service (HTTP/HTTPS)
                          │
                    SSH/WG ──► Management plane (key-based auth)
                          │
                    S3-Compatible Storage ──► Backup plane (all servers)

Shared dependencies:

  • SSH access: Ed25519 key, deployed to all servers
  • DNS: Cloudflare (primary) / legacy provider (one domain)
  • Backup storage: S3-compatible (Wasabi, us-east-1)
  • SMTP relay: Authenticated relay (STARTTLS, port 2525)
  • Auth/SSO: Cloudflare Access (internal portals)
  • Monitoring: Prometheus node_exporter per server

Server Inventory Summary

11 servers total:

  • 1 Primary operations server (netcup, dedicated EPYC)
  • 1 AI/LLM inference server (Hetzner)
  • 1 WordPress/RunCloud server (Hetzner)
  • 3 Application servers (Hetzner — UISP/UNMS, Hudu, UniFi)
  • 3 Utility servers (Hetzner — n8n, Docker staging, fleet tracking)
  • 1 Standalone Hermes server (Hetzner — Tony)
  • 1 Warm standby server (Hetzner)

DR Classification

Server Tier RTO Target RPO Target Restore Complexity
Primary Ops Critical 5 min (failover) 15 min Medium
AI/LLM Critical 2 hours 24 hours High
WordPress Important 4 hours 24 hours Low
UISP/UNMS Important 4 hours 24 hours Medium
IT Documentation Important 4 hours 24 hours Medium
Network Controller Important 4 hours 24 hours Low
Automation (n8n) Standard 8 hours 24 hours Low
Docker Utility Standard 8 hours 24 hours Medium
Fleet Tracking Standard 8 hours 24 hours Low
Standalone Agent Standard Coordinate 24 hours Low
Warm Standby Standby 2 min 15 min Low

Backup Strategy

Data Frequency Location Retention
System config Daily S3-compatible storage 90 days
Docker volumes Daily S3-compatible storage 90 days
Database dumps Daily S3-compatible storage 90 days
Application data Per-application S3-compatible storage 90 days
Agent state Every 15 min S3-compatible storage 48 hours

Restore Procedure (Generic)

# 1. Provision replacement server (same tier or better)
# 2. Install Debian (current stable)
# 3. Deploy SSH key
# 4. Install Docker (if applicable)
# 5. Restore from backup storage:
aws s3 cp s3://backup-bucket/<server>/latest/ /opt/restore/ --recursive
# 6. Restore Docker stacks
cd /opt/<service> && docker compose up -d
# 7. Verify health endpoints return 200

Key Recommendations from Audit

  1. AI server disk critical (92% full): Needs cleanup before any migration
  2. Docker volumes not backed up to S3: All Docker hosts need volume backup automation
  3. Database dumps not automated: MariaDB on wphost02, fleettracker360 need scheduled dumps
  4. Tony's Hermes: Independent instance — document its config separately
  5. Migration plan: Consolidate Hetzner services to netcup per the server standard

Full version with IPs, credentials references, and detailed restore commands is stored internally and available on request.