ippadmin/hermes-recovery
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
master
hermes-recovery/references/server-dr-plans-public.md
T

18 KiB
Raw Permalink Blame History

Company Name — Disaster Recovery Plan v3 (Consolidated)

Author: [Lead Sysadmin] / Network Services Team, incorporating third-party DR review Date: July 9, 2026 Status: LIVE — replaces v1.0 Supersedes: server-dr-plans.md v1.0, server-provisioning-standard-v1.md, hermes-dr-plan-v2.md

Table of Contents

  1. Architecture & Shared Dependencies
  2. Server Standards
  3. Backup Strategy
  4. Failover / Failback
  5. Monitoring & Alerting
  6. Restore Runbooks
  7. Testing & Validation
  8. Security
  9. Immediate Action Items
  10. Appendix: Third-Party Review Responses

1. Architecture & Shared Dependencies

Internet ──► CDN/DNS Provider ──► Caddy ──► Service (HTTP/HTTPS)
                          │
              ┌───────────┼───────────┐
              │           │           │
         SSH/WG     Tailscale    Management
         (company-prefix-     (Primary Host ↔      VPN (WireGuard
          infra)     Standby Host)     to home router)
              │           │           │
              └───────────┼───────────┘
                          │
              ┌───────────┴───────────┐
              │                       │
      S3 Primary                 S3 Secondary
   (Object Storage Provider us-east-1)        (Object Storage Provider us-west-2
        versioning ON          or Backblaze B2)
        object lock ON        object lock ON

Shared Dependencies — Documented

Dependency Primary Backup / Fallback Credentials Location
SSH access company-prefix-infra key (Secondary Provider vault) Rescue mode / console ~/.hermes/.env + Hudu
DNS CDN/DNS Provider (API token in .env) Manual via web UI ~/.hermes/.env
S3 backup (primary) Object Storage Provider us-east-1 Object Storage Provider us-west-2 (to be created) ~/.aws/credentials
S3 backup (secondary) Object Storage Provider us-west-2 Backblaze B2 (evaluate) TBD
SMTP relay [MAIL-SERVER-DOMAIN]:2525 Direct Email Provider (port 465/587) ~/.hermes/.env
Auth/SSO CDN/DNS Provider Access Local portal auth fallback CDN/DNS Provider dashboard
Monitoring Prometheus node_exporter Uptime Kuma (docker box) N/A (pull model)
VPN management WireGuard (wg0, port 51821) Tailscale direct tunnel /etc/wireguard/

3.1 Backup Matrix

Data Type Frequency Target Retention Status
Hermes state Every 15 min S3 primary (live/) 48 hours ✅ LIVE
Hermes full Daily 5 AM S3 primary (full/) 90 days ✅ LIVE
Memory snapshots Every 10 min S3 primary (memories/) 48 hours ✅ LIVE (new)
Memory history Every 10 min S3 primary (memories/history/) 90 days ✅ LIVE (new)
Router configs Daily 6 AM S3 primary (mikrotik/) 90 days ✅ LIVE
System configs Daily S3 primary (/) 90 days ⚠️ NOT AUTOMATED
Docker volumes Daily S3 primary (/) 90 days ❌ NOT DONE
Database dumps Every 6 hours S3 primary (/) 90 days ❌ NOT DONE
Application data Per-app S3 primary (/) 90 days ⚠️ PARTIAL
S3 secondary Daily sync Object Storage Provider us-west-2 90 days ❌ NOT SET UP

3.3 Backup Security

  • Versioning: ON (all 3 buckets) ✅
  • Object Lock: NOT enabled — needs implementation
  • Delete protection: NOT configured
  • Backup credentials: Write-only IAM user — NOT implemented (using admin credentials)
  • Encryption at rest: Object Storage Provider server-side (default) — no client-side encryption
  • Credentials escrow: Partial (in .env + Hudu)

Fixes needed:

  1. Create IAM sub-user with s3:PutObject only (no delete, no list)
  2. Enable Object Lock with 7-day minimum retention
  3. Store write-only credentials separately from admin credentials
  4. Client-side encrypt backup before upload (age/gpg) for the secondary copy

4. Failover / Failback

4.1 Warm Standby — Corrected Timing

The DR reviewer flagged this correctly: you cannot detect a 2-minute outage with a 5-minute check interval.

Old: Check every 5 min, failover after 2 min unreachable
New: Check every 30 seconds, failover after 4 failed checks (~2 min)

# Pseudocode for corrected watchdog
CHECK_INTERVAL = 30       # seconds
FAILURE_THRESHOLD = 4      # consecutive failures
CONFIRMATION_WINDOW = 120  # seconds (2 min)
# Detection time: 4 * 30s = 120s = 2 minutes
# Sync + start time: ~45-90 seconds
# Effective RTO: ~3-3.5 minutes
# Stated RTO target: 5 minutes (ACHIEVABLE)

4.2 Health Check — Two Layers

Layer 1 (Ping): ICMP to Primary Host ([IP-REDACTED]) every 30s
Layer 2 (HTTP): GET /[GENERIC-HEALTH-ENDPOINT] on Tailscale IP — checks:

  • Gateway heartbeat ≤ 5s old
  • SQLite PRAGMA quick_check returns "ok"
  • Local Ollama responding

Decision Matrix:

Server Hermes Action
✅ UP ✅ UP Stay dormant
❌ DOWN N/A Auto-takeover — unambiguous
✅ UP ❌ DOWN ALERT + fence — attempt SSH fence, only takeover if Primary Host Hermes confirmed stopped

4.3 Fencing Protocol

# SSH Fence (preferred — server is up)
ssh [USER-REDACTED]@[IP-REDACTED] \
  "sudo systemctl stop hermes && sudo systemctl mask hermes"
# Verify: returns "inactive"
ssh [USER-REDACTED]@[IP-REDACTED] "sudo systemctl is-active hermes"

# S3 Ownership Lock
# Object: s3://hermes-vps-backups/ACTIVE_OWNER
# Primary Host refreshes every 60s while healthy
# Standby writes itself as owner on takeover
# Primary Host reads on startup — if standby owns it, Primary Host stays dormant

4.4 Failback Protocol

⚠️ Auto-failback is FORBIDDEN. Human clearance required.

# FAILBACK STEPS (manual):
# 1. Confirm Primary Host is healthy: SSH + systemctl is-active hermes
# 2. Stop standby gateway: ssh Standby Host "hermes gateway stop"
# 3. Sync state from standby back to Primary Host (if needed):
#    Rsync /root/.hermes/ from standby to Primary Host
# 4. Unmask Hermes on Primary Host:
#    ssh Primary Host "sudo systemctl unmask hermes && sudo systemctl start hermes"
# 5. Verify Primary Host gateway is active (Telegram responds)
# 6. Standby returns to dormant state
# 7. Update S3 ownership lock to "core"

4.5 Provider/Account Outage Scenario

If the entire hosting provider (Primary Provider) is unavailable:

  1. Standby Host (Secondary Provider) takes over as active Hermes host
  2. DNS records updated to point to Standby Host IP ([IP-REDACTED])
  3. Services migrate to Primary Provider replacement when available
  4. No auto-failback — human decision

5. Monitoring & Alerting

5.1 Alert Thresholds

Alert Threshold Action Priority
Disk usage > 80% Notify, investigate 🟡 P2
Disk usage > 90% Notify, immediate cleanup 🔴 P0
Backup age > 24h since last success Notify 🟡 P1
Backup failure Any failed job Notify 🔴 P0
Hermes heartbeat No heartbeat > 30s Notify 🔴 P0
Hermes down + server up Layer 2 fail, Layer 1 pass IMMEDIATE alert 🔴 P0
TLS expiration < 14 days Notify 🟡 P1
TLS expiration < 7 days Notify + remind 🔴 P0
SSH fail spikes > 10 failures/min Investigate 🟡 P1
Service down Caddy/nginx/Docker inactive Notify, attempt restart 🔴 P0
S3 sync failure Upload fails Abort (memory consolidate) 🔴 P0
Standby stale No sync > 30 min Notify 🟡 P1

5.2 Monitoring Stack

  • Per-server: prometheus-node-exporter (all boxes)
  • Dashboard: Prometheus → Grafana ()
  • Synthetic checks: Uptime Kuma (on docker box, [IP-REDACTED])
  • Hermes watchdog: service-health-check.sh (every 5 min, Primary Host)
  • Standby watcher: home-router-watchdog.sh (every 5 min, Primary Host)
  • Apex mail: apex-mail-watchdog.sh (every 5 min, Primary Host)
  • Alert delivery: Telegram (via Hermes)

6. Restore Runbooks (Per Server)

Attribute Detail
RTO target 5 minutes (via failover to Standby Host)
RPO target 15 minutes (Hermes state), 24h (Docker data)
Actual RTO ~3 minutes (corrected watchdog)
Actual RPO Varies by data type (see below)

Restore steps:

# Option A: Failover to standby (fastest — < 5 min)
# Standby Host detects Primary Host down → auto-takeover (Section 4.1)

# Option B: Full restore on replacement box
# Step 1 — Provision RS 2000 G12 on Primary Provider
# Step 2 — Apply base standard (Section 2.2)
# Step 3 — Install Hermes
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
# Step 4 — Restore Hermes state from latest S3:
aws s3 cp s3://hermes-vps-backups/live/state.db /root/.hermes/state.db \
  --endpoint-url https://[S3-STORAGE-DOMAIN]
aws s3 cp s3://hermes-vps-backups/live/config.yaml /root/.hermes/config.yaml \
  --endpoint-url https://[S3-STORAGE-DOMAIN]
# Step 5 — Restore .env from backup:
aws s3 cp s3://hermes-vps-backups/live/.env /root/.hermes/.env \
  --endpoint-url https://[S3-STORAGE-DOMAIN]
chmod 600 /root/.hermes/.env
# Step 6 — Restore Docker stacks:
for dir in /root/docker/*/; do
  (cd "dir" && docker compose up -d)
done
# Step 7 — Restore Caddy config:
aws s3 cp s3://hermes-vps-backups/live/config/Caddyfile /etc/caddy/Caddyfile
systemctl restart caddy
# Step 8 — Start Hermes gateway:
hermes gateway run --replace
# Step 9 — Verify:
curl -s https://[OPS-PORTAL-DOMAIN]/[GENERIC-HEALTH-ENDPOINT]
curl -s https://[DOCUMENT-SIGNING-DOMAIN]
# Check Telegram responds to /ping

6.2 [AI-SERVICE-DOMAIN] (Secondary Provider CPX41) — 🔴 Critical

Attribute Detail
RTO target 2 hours
RPO target 24 hours
Current disk 🔴 92% — must clean before migration

⚠️ Pre-reboot checklist:

# Verify at least one alternative model provider works
curl https://[AI-ADMIN-DOMAIN]/[GENERIC-LLM-ENDPOINT] \
  -H "Authorization: Bearer (cat ~/.hermes/.env | grep ADMIN_AI | cut -d= -f2)" \
  -d '{"model":"deepseek-chat","messages":[{"role":"user","content":"ping"}]}'
# Ensure local Ollama on Primary Host is running
curl -s http://[IP-REDACTED]:11434/[GENERIC-OLLAMA-ENDPOINT] || systemctl start ollama
# Notify all Hermes users ([CEO], [Operations Manager], [Developer])

Restore steps:

# Step 1 — Provision replacement (CPX41 or Primary Provider equivalent)
# Step 2 — Apply base standard (Section 2.2)
# Step 3 — Deploy Docker stacks:
cd /opt/litellm && docker compose up -d
cd /opt/openwebui && docker compose up -d
cd /opt/qdrant && docker compose up -d
# Step 4 — Pull models:
ollama pull llama3.2:3b
# Step 5 — Configure DNS: [AI-ADMIN-DOMAIN] → new IP
# Step 6 — Update all Hermes config.yaml files with new base_url
# Step 7 — Verify model access for all profiles

6.3 Legacy Web Host (Secondary Provider CPX21, Server Management Panel) — 🟡 Important

Attribute Detail
RTO target 4 hours
RPO target 24 hours
Backup method Server Management Panel daily backup + manual DB

Restore steps:

# Step 1 — Provision CPX21 on Secondary Provider
# Step 2 — Connect to Server Management Panel panel, link server
# Step 3 — Server Management Panel provisions nginx, MariaDB, PHP
# Step 4 — Restore WordPress from Server Management Panel backup → verify apex site loads
# Step 5 — Deploy company-prefix-infra SSH key in Server Management Panel → SSH Access
# Step 6 — Whitelist Primary Host IP ([IP-REDACTED]) in Server Management Panel firewall
# Step 7 — Reconfigure Postfix relay:
postconf -e "relayhost = [MAIL-SERVER-DOMAIN]:2525"
postconf -e "smtp_sasl_auth_enable = yes"
echo "[MAIL-SERVER-DOMAIN]:2525 [EMAIL-REDACTED]:[CREDENTIAL-REDACTED]" > /etc/postfix/sasl_passwd
postmap /etc/postfix/sasl_passwd && systemctl restart postfix
# Step 8 — Flush postfix queue:
postqueue -f

6.4 UNMS (Secondary Provider CPX21) — 🟡 Important

Attribute Detail
RTO target 4 hours
RPO target 24 hours
Containers 9 (Docker compose stack)

Restore steps:

# Step 1 — Provision CPX21 → Docker
# Step 2 — Restore Postgres data volumes from S3
# Step 3 — Deploy docker-compose.yml
cd /opt/unms && docker compose up -d
# Step 4 — Verify UCRM + UNMS web login
# Step 5 — Re-register tower routers via UNMS

6.5 Hudu (Secondary Provider CPX21) — 🟡 Important

Attribute Detail
RTO target 4 hours
RPO target 24 hours
Containers 5 (Docker compose)
Auto-backup Hudu built-in S3 backup

Restore steps:

# Step 1 — Provision CPX21 → Docker
# Step 2 — Deploy Hudu Docker stack
# Step 3 — Restore Postgres from last backup
# Step 4 — Verify web login + recent documents accessible

6.6 UniFi (Secondary Provider CPX21) — 🟡 Important

Attribute Detail
RTO target 4 hours
RPO target 24 hours
Type Native install (not Docker)

Restore steps:

# Step 1 — Provision CPX21
# Step 2 — Install UniFi Controller (native)
# Step 3 — Restore from UniFi backup file
# Step 4 — Adopt APs and switches

6.7 n8n (Secondary Provider CPX11) — 🟢 Standard

Attribute Detail
RTO target 8 hours
Containers 2 (n8n + postgres)

Restore:

# Provision → Docker → restore postgres volume
cd /opt/n8n && docker compose up -d

6.8 docker box (Secondary Provider CPX11) — 🟢 Standard

Attribute Detail
RTO target 8 hours
Containers 8 (NPM, RustDesk, Uptime Kuma, Server Management Panel)

Restore:

# Provision → Docker → restore compose stacks + volumes

6.9 fleettracker360 (Secondary Provider CPX11) — 🟢 Standard

Attribute Detail
RTO target 8 hours
Type Apache + MariaDB (LAMP)

Restore:

# Provision → Apache + MariaDB → restore web root + DB

6.10 developer-vps (Secondary Provider CPX21) — 🟢 Standard

Attribute Detail
RTO target Coordinate with [Developer]
Type Independent Hermes + local LLM

⚠️ SHADOW IT — needs documentation.

Current known config:

  • Hermes gateway with admin-ai model access
  • Llama 3.2 1B local fallback (systemd: llama-server)
  • Cron: hermes-heartbeat.sh every minute
  • Minimal services, no Docker

Actions needed:

  1. Document [Developer]'s full Hermes config.yaml and .env
  2. Backup to S3 under s3://hermes-vps-backups/developer-vps/
  3. Add [Developer]'s config to DR plan once documented

6.11 Standby Host (Secondary Provider CPX21) — 🔴 Standby

Attribute Detail
RTO target 2 minutes
RPO target 15 minutes
Type Warm standby — gateway dormant

Activation:

# Automatic: watchdog detects Primary Host down → S3 sync → start gateway
# Manual:
ssh [USER-REDACTED]@[IP-REDACTED]
sudo -i
hermes gateway run --replace

7. Testing & Validation

7.1 Restore Testing Schedule

Test Frequency Who
Verify backup job completion (all servers) Daily Automated alert
Check backup age per server Daily Automated
Restore one database dump to test environment Monthly Network Team
Restore one full application stack Quarterly Network Team
Full DR simulation — Primary Host failover to Standby Host Quarterly Network Team
S3 credential restore/access test Quarterly Network Team
Document restore test results (date, tester, duration, findings) Per test Network Team

7.2 Validation Criteria

Each restore test must produce:

  • Date + tester name
  • Server/application tested
  • Backup artifact used (path + timestamp)
  • Restore duration (start → fully operational)
  • Data validation: check a known record exists (specific row, file, or document)
  • Issues encountered
  • Remediation ticket (if needed)

7.3 Quarterly DR Simulation

Scenario: "Primary Host is completely gone. Primary Provider account inaccessible. Standby Host has been active for 4 hours. Now Primary Provider is back — fail back."

Steps to validate:

  1. Simulate Primary Provider provider outage (block Primary Host IP)
  2. Verify Standby Host takes over within 5 minutes
  3. Send a Telegram message through standby
  4. Verify all cron jobs run on standby
  5. Simulate Primary Provider recovery
  6. Execute manual failback per Section 4.4
  7. Verify no data loss, no split-brain

8. Security

8.1 Current Security Posture

Control Status Notes
SSH keys only ✅ Done Password auth disabled
Root SSH disabled ⚠️ Partial Some servers still allow root
UFW host firewall ✅ All Standard on all boxes
fail2ban ✅ All Standard on all boxes
S3 versioning ✅ All buckets Enabled
S3 Object Lock ❌ Not done Needs implementation
S3 write-only IAM ❌ Not done Currently using admin creds
Backup encryption ❌ Not done Server-side only; no client-side
Secrets escrow 🟡 Partial .env + Hudu — not all secrets documented
TLS everywhere ✅ All public endpoints Caddy auto-certs

8.3 Break-Glass Access

If company-prefix-infra key is lost or revoked:

  1. Secondary Provider servers: Use Secondary Provider console → Rescue mode → mount filesystem → redeploy key
  2. Primary Provider servers: Use SCP web console → VNC → log in with root password → redeploy key
  3. Root passwords: Stored in ~/.hermes/.env (chmod 600) and Hudu
Reference in New Issue View Git Blame Copy Permalink