ippadmin/hermes-skills
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
master
hermes-skills/skills/devops/disaster-recovery-audit/references/server-dr-plans-v3.md
T

9.8 KiB
Raw Permalink Blame History

IT Pro Partner — Disaster Recovery Plan v3 (Consolidated)

Author: Sho'Nuff / Network Services Team, incorporating third-party DR review Date: July 10, 2026 Status: LIVE — replaces v1.0 Supersedes: server-dr-plans.md v1.0, server-provisioning-standard-v1.md, hermes-dr-plan-v2.md

Table of Contents

  1. Architecture & Shared Dependencies
  2. Server Standards
  3. Backup Strategy
  4. Failover / Failback
  5. Monitoring & Alerting
  6. Restore Runbooks
  7. Testing & Validation
  8. Security
  9. Immediate Action Items
  10. Appendix: Third-Party Review Responses

1. Architecture & Shared Dependencies

Internet -- Cloudflare -- Caddy -- Service (HTTP/HTTPS)
                          |
              +-----------+-----------+
              |           |           |
         SSH/WG     Tailscale    Management
         (itpp-     (Core <->     VPN (WireGuard
          infra)     app1-bu)     to home router)
              |           |           |
              +-----------+-----------+
                          |
              +-----------+-----------+
              |                       |
      S3 Primary                 S3 Secondary
   (Wasabi us-east-1)        (Wasabi us-west-2
        versioning ON          or Backblaze B2)
        object lock ON        object lock ON

Shared Dependencies

Dependency Primary Backup / Fallback Credentials Location
SSH access itpp-infra key (Hetzner vault) Rescue mode / console ~/.hermes/.env + Hudu
DNS Cloudflare (API token in .env) Manual via web UI ~/.hermes/.env
S3 backup (primary) Wasabi us-east-1 Wasabi us-west-2 (to be created) ~/.aws/credentials
S3 backup (secondary) Wasabi us-west-2 Backblaze B2 (evaluate) TBD
SMTP relay mail.germainebrown.com:2525 Direct MXroute (port 465/587) ~/.hermes/.env
Auth/SSO Cloudflare Access Local portal auth fallback Cloudflare dashboard
Monitoring Prometheus node_exporter Uptime Kuma (docker box) N/A (pull model)
VPN management WireGuard (wg0, port 51821) Tailscale direct tunnel /etc/wireguard/

2. Server Standards

2.1 Tier Definitions

Tier Spec OS Use Cost
Standard RS 4000 G12 (12C/32G/1TB) Debian 13 app1, app2, app3 ~$44/mo
Light RS 2000 G12 (8C/16G/512GB) Debian 13 Core ~$24/mo
Standby CPX21 (4C/8G/80GB) Debian 13 app1-bu ~$14/mo
Legacy Variable Hetzner Debian 12 Existing (migrate on rebuild) ~$8-44/mo

2.2 Base Install -- Verified Checklist

# S1: hostname + timezone
hostnamectl set-hostname <name>
timedatectl set-timezone America/New_York

# S2: system update
apt update && apt upgrade -y

# S3: security baseline
apt install -y fail2ban ufw
ufw default deny incoming; ufw default allow outgoing
ufw allow ssh; ufw --force enable

# S4: monitoring
apt install -y prometheus-node-exporter

# S5: standard user + key
adduser ippadmin && usermod -aG sudo ippadmin
echo "ssh-ed25519 AAA... itpp-infra" >> /home/ippadmin/.ssh/authorized_keys

# S6: harden SSH
sed -i 's/^#PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config
sed -i 's/^PermitRootLogin yes/PermitRootLogin prohibit-password/' /etc/ssh/sshd_config
systemctl restart sshd

# S7: Docker (if needed)
curl -fsSL https://get.docker.com | bash
apt install -y docker-compose-plugin

2.3 Application Data Paths

Every server must document:

  • What data lives where (volumes, bind mounts, databases, uploads, .env files)
  • What can be rebuilt vs what must be restored
  • Secrets location
  • Caddy/nginx configs
  • Cron jobs
  • External API dependencies

3. Backup Strategy

3.1 Backup Matrix

Data Type Frequency Target Retention Status
Hermes state Every 15 min S3 primary (live/) 48 hours ✅ LIVE
Hermes full Daily 5 AM S3 primary (full/) 90 days ✅ LIVE
Memory snapshots Every 10 min S3 primary (memories/) 48 hours ✅ LIVE
Memory history Every 10 min S3 primary (memories/history/) 90 days ✅ LIVE
Router configs Daily 6 AM S3 primary (mikrotik/) 90 days ✅ LIVE
System configs Daily S3 primary (/) 90 days 🔲 NOT AUTOMATED
Docker volumes Daily S3 primary (/) 90 days ❌ NOT DONE
Database dumps Every 6 hours S3 primary (/) 90 days ❌ NOT DONE
Application data Per-app S3 primary (/) 90 days 🔲 PARTIAL
S3 secondary Daily sync Wasabi us-west-2 90 days ❌ NOT SET UP

3.2 Backup Security

  • Versioning: ON (all 3 buckets)
  • Object Lock: NOT enabled
  • Delete protection: NOT configured
  • Backup credentials: Admin-level (can delete) -- NOT write-only
  • Encryption: Server-side only (Wasabi default)

4. Failover / Failback

4.1 Warm Standby Timing (Corrected)

Old: Check every 5 min, failover after 2 min unreachable (impossible -- can't detect 2-min outage at 5-min intervals)

New: Check every 30 seconds, failover after 4 failed checks (~2 min detection) + 45-90s sync = ~3-3.5 min actual, 5 min committed RTO

4.2 Health Check -- Two Layers

Layer 1 (Ping): ICMP to Core (152.53.192.33) every 30s Layer 2 (HTTP): GET /healthz on Tailscale IP -- checks Gateway heartbeat, SQLite PRAGMA, local Ollama

4.3 Split-Brain Prevention

S3 active-lock.json mechanism:

{"active_node": "core", "heartbeat": "<timestamp>", "lock_holder": "core"}

Core writes heartbeat every 30s while healthy. Failover only proceeds after:

  1. Core unreachable for 4 consecutive checks (app1-bu local)
  2. External monitor independently confirms Core down
  3. active-lock.json heartbeat stale (>90s)
  4. app1-bu writes itself as lock_holder
  5. Gateway starts
  6. Operator alerted

If Core can be reached for fencing, app1-bu SSHes in and stops/masks Hermes before taking over.

4.4 Failback Protocol

No auto-failback. Sequence:

  1. Confirm app1-bu is active node
  2. Keep Core stopped -- no auto-start
  3. Restore Core OS + stack
  4. Sync latest state from app1-bu/S3 into Core
  5. Validate sync timestamp
  6. Start Core in passive mode
  7. Stop app1-bu gateway
  8. Update active-lock.json: "active_node": "core"
  9. Start Core gateway
  10. 5 consecutive health checks pass
  11. Release standby lock
  12. Monitor 30 min before closing

Rollback: If Core fails health checks mid-failback, revert immediately -- restore app1-bu as active, keep Core in maintenance.

4.5 Provider/Account Outage

If entire netcup account is unavailable:

  1. app1-bu (Hetzner) takes over
  2. DNS records pointed to app1-bu IP
  3. No auto-failback -- human decision

5. Per-Server RTO/RPO Targets

System RTO RPO Notes
Core (Hermes) 5 min 15 min Via warm standby (app1-bu)
app1-bu (standby) N/A <=15 min Recovery path for Core
app1/2/3 (Standard tier) 60 min 24h Rebuild from S3 + docker compose
wphost02 (WordPress) 2h 24h DB restore from dump
fleettracker360 2h 24h DB restore from dump
Legacy Hetzner hosts 4h 24h Via rescue mode or S3

6. Immediate Action Items

P0 -- This Week

# Action Details
1 Clean AI server disk 92% full -- purge old Docker images/logs
2 Create S3 write-only IAM user PutObject only, no delete/list
3 Enable S3 Object Lock 7-day minimum retention
4 Deploy Docker volume backup All Docker hosts
5 Deploy DB dump cron wphost02 + fleettracker360 (MariaDB)
6 Deploy 30s watchdog Currently documented, need deployment

P1 -- Next 2 Weeks

# Action
7 Document Tony's Hermes config + backup to S3
8 Provision secondary S3 bucket (Wasabi us-west-2)
9 Extend snapshot retention from 48h to 30 days
10 Add backup success/failure monitoring to ops portal

P2 -- Next 30 Days

# Action
11 Full DR simulation for Core failover
12 Monthly restore testing schedule begins
13 Deploy Prometheus + Grafana stack
14 Add disk threshold alerts for all servers
15 Begin netcup migration

7. Restore Runbooks (Summary)

Full per-server restore runbooks with exact commands are in the Per-Server Runbooks document (companion doc 3 of the v3 DR stack). Each covers:

  • Purpose + dependencies
  • DNS records, firewall ports, backup path
  • Required secrets
  • Step-by-step restore commands
  • DB restore with pre-import dump verification
  • Docker volume restore
  • Health checks
  • Known failure modes

Hosts covered: Core, app1-bu, app1/2/3, wphost02, fleettracker360, Legacy Hetzner hosts.

8. Third-Party Review Responses

Reviewer 1 Findings -- Addressed

Finding Addressed In
Failover timing mismatch (5min check did not equal 2min RTO) Section 4.1 -- Corrected to 30s checks
Backup compliance gap (standard says daily but not happening) Section 3.1 -- Gap closure plan
AI server 92% disk critical Section 6 -- P0 action item #1
Memory consolidation data loss risk Separate docs per priority-tagging proposal
Tony's Hermes undocumented Section 6 -- P1 action item #7

Reviewer 2 Findings -- Addressed

Finding Addressed In
S3 single point of failure Section 1 + Section 3.1 -- Two-region plan
Backup frequency vs RPO mismatch Section 5 -- RPO per data type
Restore procedures not detailed Section 7 -- Full runbooks reference
Failback under-defined Section 4.4 -- Protocol
No backup restore testing Separate testing schedule doc
Retention too short (48h snapshots) Section 6 -- P1 action item #9
Backup security not described Section 3.2
Application backup details vague Section 2.3
Secrets recovery not documented Section 1 (shared deps table)
Provider/account outage missing Section 4.5

This plan is ACTIVE. Review quarterly or after any infrastructure change.

Reference in New Issue View Git Blame Copy Permalink