ippadmin/hermes-skills
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
hermes-skills/skills/devops/disaster-recovery-audit/references/server-dr-plans-v3.md
T

9.9 KiB
Raw Permalink Blame History

IT Pro Partner — Disaster Recovery Plan v3 (Consolidated)

Author: Sho'Nuff / Network Services Team, incorporating third-party DR review Date: July 10, 2026 Status: LIVE — replaces v1.0 Supersedes: server-dr-plans.md v1.0, server-provisioning-standard-v1.md, hermes-dr-plan-v2.md

Table of Contents

  1. Architecture & Shared Dependencies
  2. Server Standards
  3. Backup Strategy
  4. Failover / Failback
  5. Monitoring & Alerting
  6. Restore Runbooks
  7. Testing & Validation
  8. Security
  9. Immediate Action Items
  10. Appendix: Third-Party Review Responses

1. Architecture & Shared Dependencies

Internet -- Cloudflare -- Caddy -- Service (HTTP/HTTPS)
                          |
              +-----------+-----------+
              |           |           |
         SSH/WG     Tailscale    Management
         (itpp-     (Core <->     VPN (WireGuard
          infra)     app1-bu)     to home router)
              |           |           |
              +-----------+-----------+
                          |
              +-----------+-----------+
              |                       |
      S3 Primary                 S3 Secondary
   (Wasabi us-east-1)        (Wasabi us-west-2
        versioning ON          or Backblaze B2)
        object lock ON        object lock ON

Shared Dependencies

Dependency Primary Backup / Fallback Credentials Location
SSH access itpp-infra key (Hetzner vault) Rescue mode / console ~/.hermes/.env + Hudu
DNS Cloudflare (API token in .env) Manual via web UI ~/.hermes/.env
S3 backup (primary) Wasabi us-east-1 Wasabi us-west-2 (to be created) ~/.aws/credentials
S3 backup (secondary) Wasabi us-west-2 Backblaze B2 (evaluate) TBD
SMTP relay mail.germainebrown.com:2525 Direct MXroute (port 465/587) ~/.hermes/.env
Auth/SSO Cloudflare Access Local portal auth fallback Cloudflare dashboard
Monitoring Prometheus node_exporter Uptime Kuma (docker box) N/A (pull model)
VPN management WireGuard (wg0, port 51821) Tailscale direct tunnel /etc/wireguard/

2. Server Standards

2.1 Tier Definitions

Tier Spec OS Use Cost
Standard RS 4000 G12 (12C/32G/1TB) Debian 13 app1, app2, app3 ~$44/mo
Light RS 2000 G12 (8C/16G/512GB) Debian 13 Core ~$24/mo
Standby CPX21 (4C/8G/80GB) Debian 13 app1-bu ~$14/mo
Legacy Variable Hetzner Debian 12 Existing (migrate on rebuild) ~$8-44/mo

2.2 Base Install -- Verified Checklist

# S1: hostname + timezone
hostnamectl set-hostname <name>
timedatectl set-timezone America/New_York

# S2: system update
apt update && apt upgrade -y

# S3: security baseline
apt install -y fail2ban ufw
ufw default deny incoming; ufw default allow outgoing
ufw allow ssh; ufw --force enable

# S4: monitoring
apt install -y prometheus-node-exporter

# S5: standard user + key
adduser ippadmin && usermod -aG sudo ippadmin
echo "ssh-ed25519 AAA... itpp-infra" >> /home/ippadmin/.ssh/authorized_keys

# S6: harden SSH
sed -i 's/^#PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config
sed -i 's/^PermitRootLogin yes/PermitRootLogin prohibit-password/' /etc/ssh/sshd_config
systemctl restart sshd

# S7: Docker (if needed)
curl -fsSL https://get.docker.com | bash
apt install -y docker-compose-plugin

2.3 Application Data Paths

Every server must document:

  • What data lives where (volumes, bind mounts, databases, uploads, .env files)
  • What can be rebuilt vs what must be restored
  • Secrets location
  • Caddy/nginx configs
  • Cron jobs
  • External API dependencies

3. Backup Strategy

3.1 Backup Matrix

Data Type Frequency Target Retention Status
Hermes state Every 15 min S3 primary (live/) 48 hours ✅ LIVE
Hermes full Daily 5 AM S3 primary (full/) 90 days ✅ LIVE
Memory snapshots Every 10 min S3 primary (memories/) 48 hours ✅ LIVE
Memory history Every 10 min S3 primary (memories/history/) 90 days ✅ LIVE
Router configs Daily 6 AM S3 primary (mikrotik/) 90 days ✅ LIVE
System configs Daily S3 primary (/) 90 days 🔲 NOT AUTOMATED
Docker volumes Daily S3 primary (/) 90 days ❌ NOT DONE
Database dumps Every 6 hours S3 primary (/) 90 days ❌ NOT DONE
Application data Per-app S3 primary (/) 90 days 🔲 PARTIAL
S3 secondary Daily sync Wasabi us-west-2 90 days ❌ NOT SET UP

3.2 Backup Security

  • Versioning: ON (all 3 buckets)
  • Object Lock: NOT enabled
  • Delete protection: NOT configured
  • Backup credentials: Admin-level (can delete) -- NOT write-only
  • Encryption: Server-side only (Wasabi default)

4. Failover / Failback

4.1 Warm Standby Timing (Corrected)

Old: Check every 5 min, failover after 2 min unreachable (impossible -- can't detect 2-min outage at 5-min intervals)

New: Check every 30 seconds, failover after 4 failed checks (~2 min detection) + 45-90s sync = ~3-3.5 min actual, 5 min committed RTO

4.2 Health Check -- Two Layers

Layer 1 (Ping): ICMP to Core (152.53.192.33) every 30s Layer 2 (HTTP): GET /healthz on Tailscale IP -- checks Gateway heartbeat, SQLite PRAGMA, local Ollama

4.3 Split-Brain Prevention

S3 active-lock.json mechanism:

{"active_node": "core", "heartbeat": "<timestamp>", "lock_holder": "core"}

Core writes heartbeat every 30s while healthy. Failover only proceeds after:

  1. Core unreachable for 4 consecutive checks (app1-bu local)
  2. External monitor independently confirms Core down
  3. active-lock.json heartbeat stale (>90s)
  4. app1-bu writes itself as lock_holder
  5. Gateway starts
  6. Operator alerted

If Core can be reached for fencing, app1-bu SSHes in and stops/masks Hermes before taking over.

4.4 Failback Protocol

No auto-failback. Sequence:

  1. Confirm app1-bu is active node
  2. Keep Core stopped -- no auto-start
  3. Restore Core OS + stack
  4. Sync latest state from app1-bu/S3 into Core
  5. Validate sync timestamp
  6. Start Core in passive mode
  7. Stop app1-bu gateway
  8. Update active-lock.json: "active_node": "core"
  9. Start Core gateway
  10. 5 consecutive health checks pass
  11. Release standby lock
  12. Monitor 30 min before closing

Rollback: If Core fails health checks mid-failback, revert immediately -- restore app1-bu as active, keep Core in maintenance.

4.5 Provider/Account Outage

If entire netcup account is unavailable:

  1. app1-bu (Hetzner) takes over
  2. DNS records pointed to app1-bu IP
  3. No auto-failback -- human decision

5. Per-Server RTO/RPO Targets

System RTO RPO Notes
Core (Hermes) 5 min 15 min Via warm standby (app1-bu)
app1-bu (standby) N/A <=15 min Recovery path for Core
app1/2/3 (Standard tier) 60 min 24h Rebuild from S3 + docker compose
fleettracker360 2h 24h DB restore from dump
Legacy Hetzner hosts 4h 24h Via rescue mode or S3

6. Immediate Action Items

P0 -- This Week

# Action Details
1 Clean AI server disk 92% full -- purge old Docker images/logs
2 Create S3 write-only IAM user PutObject only, no delete/list
3 Enable S3 Object Lock 7-day minimum retention
4 Deploy Docker volume backup All Docker hosts
5 Deploy DB dump cron fleettracker360 (MariaDB)
6 Deploy 30s watchdog Currently documented, need deployment

P1 -- Next 2 Weeks

# Action
7 Document Tony's Hermes config + backup to S3
8 Provision secondary S3 bucket (Wasabi us-west-2)
9 Extend snapshot retention from 48h to 30 days
10 Add backup success/failure monitoring to ops portal

P2 -- Next 30 Days

# Action
11 Full DR simulation for Core failover
12 Monthly restore testing schedule begins
13 Deploy Prometheus + Grafana stack
14 Add disk threshold alerts for all servers
15 Begin netcup migration

7. Restore Runbooks (Summary)

Full per-server restore runbooks with exact commands are in the Per-Server Runbooks document (companion doc 3 of the v3 DR stack). Each covers:

  • Purpose + dependencies
  • DNS records, firewall ports, backup path
  • Required secrets
  • Step-by-step restore commands
  • DB restore with pre-import dump verification
  • Docker volume restore
  • Health checks
  • Known failure modes

Hosts covered: Core, app1-bu, app1/2/3, fleettracker360, Legacy Hetzner hosts.

wphost02 (Hetzner CPX21) — DECOMMISSIONED (2026-08-28), deleted from Hetzner account. Removed from active recovery scope.

8. Third-Party Review Responses

Reviewer 1 Findings -- Addressed

Finding Addressed In
Failover timing mismatch (5min check did not equal 2min RTO) Section 4.1 -- Corrected to 30s checks
Backup compliance gap (standard says daily but not happening) Section 3.1 -- Gap closure plan
AI server 92% disk critical Section 6 -- P0 action item #1
Memory consolidation data loss risk Separate docs per priority-tagging proposal
Tony's Hermes undocumented Section 6 -- P1 action item #7

Reviewer 2 Findings -- Addressed

Finding Addressed In
S3 single point of failure Section 1 + Section 3.1 -- Two-region plan
Backup frequency vs RPO mismatch Section 5 -- RPO per data type
Restore procedures not detailed Section 7 -- Full runbooks reference
Failback under-defined Section 4.4 -- Protocol
No backup restore testing Separate testing schedule doc
Retention too short (48h snapshots) Section 6 -- P1 action item #9
Backup security not described Section 3.2
Application backup details vague Section 2.3
Secrets recovery not documented Section 1 (shared deps table)
Provider/account outage missing Section 4.5

This plan is ACTIVE. Review quarterly or after any infrastructure change.

Reference in New Issue View Git Blame Copy Permalink