Files

T

root ae056eaf83 Initial resurrection kit — 81 scripts, 62 references, configs, systemd units, crons, Docker compose files, Caddy config, master README

2026-07-15 17:45:36 -04:00

23 KiB

Raw Blame History

IT Pro Partner — Disaster Recovery Plan v3 (Consolidated)

Author: Sho'Nuff / Network Services Team, incorporating third-party DR review Date: July 9, 2026 Status: LIVE — replaces v1.0 Supersedes: server-dr-plans.md v1.0, server-provisioning-standard-v1.md, hermes-dr-plan-v2.md

Architecture & Shared Dependencies
Server Standards
Backup Strategy
Failover / Failback
Monitoring & Alerting
Restore Runbooks
Testing & Validation
Security
Immediate Action Items
Appendix: Third-Party Review Responses

1. Architecture & Shared Dependencies

Internet ──► Cloudflare ──► Caddy ──► Service (HTTP/HTTPS)
                          │
              ┌───────────┼───────────┐
              │           │           │
         SSH/WG     Tailscale    Management
         (itpp-     (Core ↔      VPN (WireGuard
          infra)     app1-bu)     to home router)
              │           │           │
              └───────────┼───────────┘
                          │
              ┌───────────┴───────────┐
              │                       │
      S3 Primary                 S3 Secondary
   (Wasabi us-east-1)        (Wasabi us-west-2
        versioning ON          or Backblaze B2)
        object lock ON        object lock ON

Shared Dependencies — Documented

Dependency	Primary	Backup / Fallback	Credentials Location
SSH access	itpp-infra key (Hetzner vault)	Rescue mode / console	~/.hermes/.env + Hudu
DNS	Cloudflare (API token in .env)	Manual via web UI	~/.hermes/.env
S3 backup (primary)	Wasabi us-east-1	Wasabi us-west-2 (to be created)	~/.aws/credentials
S3 backup (secondary)	Wasabi us-west-2	Backblaze B2 (evaluate)	TBD
SMTP relay	mail.germainebrown.com:2525	Direct MXroute (port 465/587)	~/.hermes/.env
Auth/SSO	Cloudflare Access	Local portal auth fallback	Cloudflare dashboard
Monitoring	Prometheus node_exporter	Uptime Kuma (docker box)	N/A (pull model)
VPN management	WireGuard (wg0, port 51821)	Tailscale direct tunnel	/etc/wireguard/

2. Server Standards

2.1 Tier Definitions

Tier	Spec	OS	Use	Cost
Standard	RS 4000 G12 (12C/32G/1TB)	Debian 13	app1, app2, app3	~$44/mo
Light	RS 2000 G12 (8C/16G/512GB)	Debian 13	Core	~$24/mo
Standby	CPX21 (4C/8G/80GB)	Debian 13	app1-bu	~$14/mo
Legacy	Variable Hetzner	Debian 12	Existing (migrate on rebuild)	~$8-44/mo

2.2 Base Install — Verified Checklist

# S1: hostname + timezone
hostnamectl set-hostname <name>
timedatectl set-timezone America/New_York

# S2: system update
apt update && apt upgrade -y

# S3: security baseline
apt install -y fail2ban ufw
ufw default deny incoming; ufw default allow outgoing
ufw allow ssh; ufw --force enable

# S4: monitoring
apt install -y prometheus-node-exporter

# S5: standard user + key
adduser ippadmin && usermod -aG sudo ippadmin
echo "ssh-ed25519 AAA... itpp-infra" >> /home/ippadmin/.ssh/authorized_keys

# S6: harden SSH
sed -i 's/^#PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config
sed -i 's/^PermitRootLogin yes/PermitRootLogin prohibit-password/' /etc/ssh/sshd_config
systemctl restart sshd

# S7: Docker (if needed)
curl -fsSL https://get.docker.com | bash
apt install -y docker-compose-plugin

2.3 Application Data Paths — Per Server

Every server must document:

What data lives where (volumes, bind mounts, databases, uploads, .env files)
What can be rebuilt vs what must be restored
Secrets location
Caddy/nginx configs
Cron jobs
External API dependencies

(See Section 6 — Restore Runbooks for per-server details)

3. Backup Strategy

3.1 Backup Matrix

Data Type	Frequency	Target	Retention	Status
Hermes state	Every 15 min	S3 primary (live/)	48 hours	✅ LIVE
Hermes full	Daily 5 AM	S3 primary (full/)	90 days	✅ LIVE
Memory snapshots	Every 10 min	S3 primary (memories/)	48 hours	✅ LIVE (new)
Memory history	Every 10 min	S3 primary (memories/history/)	90 days	✅ LIVE (new)
Router configs	Daily 6 AM	S3 primary (mikrotik/)	90 days	✅ LIVE
System configs	Daily	S3 primary (/)	90 days	⚠️ NOT AUTOMATED
Docker volumes	Daily	S3 primary (/)	90 days	❌ NOT DONE
Database dumps	Every 6 hours	S3 primary (/)	90 days	❌ NOT DONE
Application data	Per-app	S3 primary (/)	90 days	⚠️ PARTIAL
S3 secondary	Daily sync	Wasabi us-west-2	90 days	❌ NOT SET UP

3.2 Immediate Backup Gaps to Close

Priority	Gap	Fix
🔴 P0	Docker volumes not backed up	Deploy volume backup script per Docker host
🔴 P0	Database dumps not automated	Cron job per MariaDB/Postgres instance
🟡 P1	No secondary S3 target	Provision Wasabi us-west-2 bucket + replication
🟡 P1	App data recovery undocumented	Per-app data path inventory (Section 2.3)
🟢 P2	48h snapshot retention too short	Extend to 30 days via S3 lifecycle policy
🟢 P2	System config backup not automated	Add `/etc/` snapshot to daily backup cron

3.3 Backup Security

Versioning: ON (all 3 buckets) ✅
Object Lock: NOT enabled — needs implementation
Delete protection: NOT configured
Backup credentials: Write-only IAM user — NOT implemented (using admin credentials)
Encryption at rest: Wasabi server-side (default) — no client-side encryption
Credentials escrow: Partial (in .env + Hudu)

Fixes needed:

Create IAM sub-user with s3:PutObject only (no delete, no list)
Enable Object Lock with 7-day minimum retention
Store write-only credentials separately from admin credentials
Client-side encrypt backup before upload (age/gpg) for the secondary copy

4. Failover / Failback

4.1 Warm Standby — Corrected Timing

The DR reviewer flagged this correctly: you cannot detect a 2-minute outage with a 5-minute check interval.

Old: Check every 5 min, failover after 2 min unreachable
New: Check every 30 seconds, failover after 4 failed checks (~2 min)

# Pseudocode for corrected watchdog
CHECK_INTERVAL = 30       # seconds
FAILURE_THRESHOLD = 4      # consecutive failures
CONFIRMATION_WINDOW = 120  # seconds (2 min)
# Detection time: 4 * 30s = 120s = 2 minutes
# Sync + start time: ~45-90 seconds
# Effective RTO: ~3-3.5 minutes
# Stated RTO target: 5 minutes (ACHIEVABLE)

4.2 Health Check — Two Layers

Layer 1 (Ping): ICMP to Core (152.53.192.33) every 30s
Layer 2 (HTTP): GET /healthz on Tailscale IP — checks:

Gateway heartbeat ≤ 5s old
SQLite PRAGMA quick_check returns "ok"
Local Ollama responding

Decision Matrix:

Server	Hermes	Action
✅ UP	✅ UP	Stay dormant
❌ DOWN	N/A	Auto-takeover — unambiguous
✅ UP	❌ DOWN	ALERT + fence — attempt SSH fence, only takeover if Core Hermes confirmed stopped

4.3 Fencing Protocol

# SSH Fence (preferred — server is up)
ssh ippadmin@152.53.192.33 \
  "sudo systemctl stop hermes && sudo systemctl mask hermes"
# Verify: returns "inactive"
ssh ippadmin@152.53.192.33 "sudo systemctl is-active hermes"

# S3 Ownership Lock
# Object: s3://hermes-vps-backups/ACTIVE_OWNER
# Core refreshes every 60s while healthy
# Standby writes itself as owner on takeover
# Core reads on startup — if standby owns it, Core stays dormant

4.4 Failback Protocol

⚠️ Auto-failback is FORBIDDEN. Human clearance required.

# FAILBACK STEPS (manual):
# 1. Confirm Core is healthy: SSH + systemctl is-active hermes
# 2. Stop standby gateway: ssh app1-bu "hermes gateway stop"
# 3. Sync state from standby back to Core (if needed):
#    Rsync /root/.hermes/ from standby to Core
# 4. Unmask Hermes on Core:
#    ssh Core "sudo systemctl unmask hermes && sudo systemctl start hermes"
# 5. Verify Core gateway is active (Telegram responds)
# 6. Standby returns to dormant state
# 7. Update S3 ownership lock to "core"

4.5 Provider/Account Outage Scenario

If the entire hosting provider (netcup) is unavailable:

app1-bu (Hetzner) takes over as active Hermes host
DNS records updated to point to app1-bu IP (5.161.114.8)
Services migrate to netcup replacement when available
No auto-failback — human decision

5. Monitoring & Alerting

5.1 Alert Thresholds

Alert	Threshold	Action	Priority
Disk usage	> 80%	Notify, investigate	🟡 P2
Disk usage	> 90%	Notify, immediate cleanup	🔴 P0
Backup age	> 24h since last success	Notify	🟡 P1
Backup failure	Any failed job	Notify	🔴 P0
Hermes heartbeat	No heartbeat > 30s	Notify	🔴 P0
Hermes down + server up	Layer 2 fail, Layer 1 pass	IMMEDIATE alert	🔴 P0
TLS expiration	< 14 days	Notify	🟡 P1
TLS expiration	< 7 days	Notify + remind	🔴 P0
SSH fail spikes	> 10 failures/min	Investigate	🟡 P1
Service down	Caddy/nginx/Docker inactive	Notify, attempt restart	🔴 P0
S3 sync failure	Upload fails	Abort (memory consolidate)	🔴 P0
Standby stale	No sync > 30 min	Notify	🟡 P1

5.2 Monitoring Stack

Per-server: prometheus-node-exporter (all boxes)
Dashboard: Prometheus → Grafana (planned — on app1 or dedicated)
Synthetic checks: Uptime Kuma (on docker box, 178.156.168.35)
Hermes watchdog: service-health-check.sh (every 5 min, Core)
Standby watcher: home-router-watchdog.sh (every 5 min, Core)
Apex mail: apex-mail-watchdog.sh (every 5 min, Core)
Alert delivery: Telegram (via Hermes)

5.3 Alerting Gaps to Close

Gap	Fix
No Prometheus/Grafana stack	Deploy on app1 after migration
No backup success/failure dashboard	Add to ops portal
No disk threshold alerts for all servers	Add to ops-data-collector.py
No TLS expiry monitoring	Add cert check to service-health-check.sh
No centralized alert routing	Use Hermes for all alerts

6. Restore Runbooks (Per Server)

6.1 Core (netcup RS 2000) — 🔴 Critical

Attribute	Detail
RTO target	5 minutes (via failover to app1-bu)
RPO target	15 minutes (Hermes state), 24h (Docker data)
Actual RTO	~3 minutes (corrected watchdog)
Actual RPO	Varies by data type (see below)

Restore steps:

# Option A: Failover to standby (fastest — < 5 min)
# app1-bu detects Core down → auto-takeover (Section 4.1)

# Option B: Full restore on replacement box
# Step 1 — Provision RS 2000 G12 on netcup
# Step 2 — Apply base standard (Section 2.2)
# Step 3 — Install Hermes
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
# Step 4 — Restore Hermes state from latest S3:
aws s3 cp s3://hermes-vps-backups/live/state.db /root/.hermes/state.db \
  --endpoint-url https://s3.us-east-1.wasabisys.com
aws s3 cp s3://hermes-vps-backups/live/config.yaml /root/.hermes/config.yaml \
  --endpoint-url https://s3.us-east-1.wasabisys.com
# Step 5 — Restore .env from backup:
aws s3 cp s3://hermes-vps-backups/live/.env /root/.hermes/.env \
  --endpoint-url https://s3.us-east-1.wasabisys.com
chmod 600 /root/.hermes/.env
# Step 6 — Restore Docker stacks:
for dir in /root/docker/*/; do
  (cd "$dir" && docker compose up -d)
done
# Step 7 — Restore Caddy config:
aws s3 cp s3://hermes-vps-backups/live/config/Caddyfile /etc/caddy/Caddyfile
systemctl restart caddy
# Step 8 — Start Hermes gateway:
hermes gateway run --replace
# Step 9 — Verify:
curl -s https://ops.itpropartner.com/api/health
curl -s https://sign.itpropartner.com
# Check Telegram responds to /ping

6.2 ai.itpropartner.com (Hetzner CPX41) — 🔴 Critical

Attribute	Detail
RTO target	2 hours
RPO target	24 hours
Current disk	🔴 92% — must clean before migration

⚠️ Pre-reboot checklist:

# Verify at least one alternative model provider works
curl https://admin-ai.itpropartner.com/v1/chat/completions \
  -H "Authorization: Bearer $(cat ~/.hermes/.env | grep ADMIN_AI | cut -d= -f2)" \
  -d '{"model":"deepseek-chat","messages":[{"role":"user","content":"ping"}]}'
# Ensure local Ollama on Core is running
curl -s http://127.0.0.1:11434/api/tags || systemctl start ollama
# Notify all Hermes users (Germaine, Anita, Tony)

Restore steps:

# Step 1 — Provision replacement (CPX41 or netcup equivalent)
# Step 2 — Apply base standard (Section 2.2)
# Step 3 — Deploy Docker stacks:
cd /opt/litellm && docker compose up -d
cd /opt/openwebui && docker compose up -d
cd /opt/qdrant && docker compose up -d
# Step 4 — Pull models:
ollama pull llama3.2:3b
# Step 5 — Configure DNS: admin-ai.itpropartner.com → new IP
# Step 6 — Update all Hermes config.yaml files with new base_url
# Step 7 — Verify model access for all profiles

6.3 wphost02 (Hetzner CPX21, RunCloud) — 🟡 Important

Attribute	Detail
RTO target	4 hours
RPO target	24 hours
Backup method	RunCloud daily backup + manual DB

Restore steps:

# Step 1 — Provision CPX21 on Hetzner
# Step 2 — Connect to RunCloud panel, link server
# Step 3 — RunCloud provisions nginx, MariaDB, PHP
# Step 4 — Restore WordPress from RunCloud backup → verify apex site loads
# Step 5 — Deploy itpp-infra SSH key in RunCloud → SSH Access
# Step 6 — Whitelist Core IP (152.53.192.33) in RunCloud firewall
# Step 7 — Reconfigure Postfix relay:
postconf -e "relayhost = mail.germainebrown.com:2525"
postconf -e "smtp_sasl_auth_enable = yes"
echo "mail.germainebrown.com:2525 email@redacted:Catches.bullets1985" > /etc/postfix/sasl_passwd
postmap /etc/postfix/sasl_passwd && systemctl restart postfix
# Step 8 — Flush postfix queue:
postqueue -f

6.4 UNMS (Hetzner CPX21) — 🟡 Important

Attribute	Detail
RTO target	4 hours
RPO target	24 hours
Containers	9 (Docker compose stack)

Restore steps:

# Step 1 — Provision CPX21 → Docker
# Step 2 — Restore Postgres data volumes from S3
# Step 3 — Deploy docker-compose.yml
cd /opt/unms && docker compose up -d
# Step 4 — Verify UCRM + UNMS web login
# Step 5 — Re-register tower routers via UNMS

6.5 Hudu (Hetzner CPX21) — 🟡 Important

Attribute	Detail
RTO target	4 hours
RPO target	24 hours
Containers	5 (Docker compose)
Auto-backup	Hudu built-in S3 backup

Restore steps:

# Step 1 — Provision CPX21 → Docker
# Step 2 — Deploy Hudu Docker stack
# Step 3 — Restore Postgres from last backup
# Step 4 — Verify web login + recent documents accessible

6.6 UniFi (Hetzner CPX21) — 🟡 Important

Attribute	Detail
RTO target	4 hours
RPO target	24 hours
Type	Native install (not Docker)

Restore steps:

# Step 1 — Provision CPX21
# Step 2 — Install UniFi Controller (native)
# Step 3 — Restore from UniFi backup file
# Step 4 — Adopt APs and switches

6.7 n8n (Hetzner CPX11) — 🟢 Standard

Attribute	Detail
RTO target	8 hours
Containers	2 (n8n + postgres)

Restore:

# Provision → Docker → restore postgres volume
cd /opt/n8n && docker compose up -d

6.8 docker box (Hetzner CPX11) — 🟢 Standard

Attribute	Detail
RTO target	8 hours
Containers	8 (NPM, RustDesk, Uptime Kuma, RunCloud)

Restore:

# Provision → Docker → restore compose stacks + volumes

6.9 fleettracker360 (Hetzner CPX11) — 🟢 Standard

Attribute	Detail
RTO target	8 hours
Type	Apache + MariaDB (LAMP)

Restore:

# Provision → Apache + MariaDB → restore web root + DB

6.10 tony-vps (Hetzner CPX21) — 🟢 Standard

Attribute	Detail
RTO target	Coordinate with Tony
Type	Independent Hermes + local LLM

⚠️ SHADOW IT — needs documentation.

Current known config:

Hermes gateway with admin-ai model access
Llama 3.2 1B local fallback (systemd: llama-server)
Cron: hermes-heartbeat.sh every minute
Minimal services, no Docker

Actions needed:

Document Tony's full Hermes config.yaml and .env
Backup to S3 under s3://hermes-vps-backups/tony-vps/
Add Tony's config to DR plan once documented

6.11 app1-bu (Hetzner CPX21) — 🔴 Standby

Attribute	Detail
RTO target	2 minutes
RPO target	15 minutes
Type	Warm standby — gateway dormant

Activation:

# Automatic: watchdog detects Core down → S3 sync → start gateway
# Manual:
ssh ippadmin@5.161.114.8
sudo -i
hermes gateway run --replace

7. Testing & Validation

7.1 Restore Testing Schedule

Test	Frequency	Who
Verify backup job completion (all servers)	Daily	Automated alert
Check backup age per server	Daily	Automated
Restore one database dump to test environment	Monthly	Network Team
Restore one full application stack	Quarterly	Network Team
Full DR simulation — Core failover to app1-bu	Quarterly	Network Team
S3 credential restore/access test	Quarterly	Network Team
Document restore test results (date, tester, duration, findings)	Per test	Network Team

7.2 Validation Criteria

Each restore test must produce:

Date + tester name
Server/application tested
Backup artifact used (path + timestamp)
Restore duration (start → fully operational)
Data validation: check a known record exists (specific row, file, or document)
Issues encountered
Remediation ticket (if needed)

7.3 Quarterly DR Simulation

Scenario: "Core is completely gone. Netcup account inaccessible. app1-bu has been active for 4 hours. Now netcup is back — fail back."

Steps to validate:

Simulate netcup provider outage (block Core IP)
Verify app1-bu takes over within 5 minutes
Send a Telegram message through standby
Verify all cron jobs run on standby
Simulate netcup recovery
Execute manual failback per Section 4.4
Verify no data loss, no split-brain

8. Security

8.1 Current Security Posture

Control	Status	Notes
SSH keys only	✅ Done	Password auth disabled
Root SSH disabled	⚠️ Partial	Some servers still allow root
UFW host firewall	✅ All	Standard on all boxes
fail2ban	✅ All	Standard on all boxes
S3 versioning	✅ All buckets	Enabled
S3 Object Lock	❌ Not done	Needs implementation
S3 write-only IAM	❌ Not done	Currently using admin creds
Backup encryption	❌ Not done	Server-side only; no client-side
Secrets escrow	🟡 Partial	.env + Hudu — not all secrets documented
TLS everywhere	✅ All public endpoints	Caddy auto-certs

8.2 Security Gaps to Close

Priority	Gap	Fix
🔴 P0	S3 uses admin credentials (can delete)	Create IAM sub-user with s3:PutObject only
🔴 P0	No Object Lock on backups	Enable 7-day minimum retention
🟡 P1	No client-side backup encryption	Add age/gpg before upload for secondary copy
🟡 P1	Secrets not fully documented	Audit .env, create secrets inventory in Hudu
🟡 P1	Some servers allow root SSH	Verify and disable across all boxes
🟢 P2	No vulnerability scanning	Add lynis weekly scan (already scheduled)

8.3 Break-Glass Access

If itpp-infra key is lost or revoked:

Hetzner servers: Use Hetzner console → Rescue mode → mount filesystem → redeploy key
netcup servers: Use SCP web console → VNC → log in with root password → redeploy key
Root passwords: Stored in ~/.hermes/.env (chmod 600) and Hudu

9. Immediate Action Items

🔴 P0 — This Week

#	Action	Owner	Details
1	Clean up AI server disk	Sho'Nuff	92% full — purge old Docker images, logs, temp data
2	Create S3 write-only IAM user	Sho'Nuff	Separate creds from admin — no delete, no list
3	Enable S3 Object Lock	Sho'Nuff	7-day minimum on all backup buckets
4	Deploy volume backup script	Sho'Nuff	All Docker hosts → S3 per-server buckets
5	Deploy database dump cron	Sho'Nuff	MariaDB on wphost02 + fleettracker360, every 6h

🟡 P1 — Next 2 Weeks

#	Action	Owner
6	Fix warm standby watchdog interval (30s not 5min)	Sho'Nuff
7	Document Tony's Hermes config + backup to S3	Coordinate with Tony
8	Provision secondary S3 bucket (Wasabi us-west-2)	Sho'Nuff
9	Extend snapshot retention from 48h to 30 days	Sho'Nuff
10	Add backup success/failure monitoring to ops portal	Sho'Nuff
11	Implement priority tagging (Option A) for memory consolidation	Sho'Nuff

🟢 P2 — Next 30 Days

#	Action	Owner
12	Full DR simulation for Core failover	Network Team
13	Monthly restore testing schedule begins	Network Team
14	Deploy Prometheus + Grafana stack	Sho'Nuff
15	Add disk threshold alerts for all servers	Sho'Nuff
16	Begin netcup migration	Sho'Nuff

10. Appendix: Third-Party Review Responses

10.1 Reviewer 1 Findings — Addressed

Finding	Addressed In
Failover timing mismatch (5min check ≠ 2min RTO)	Section 4.1 — Corrected to 30s check
Backup compliance gap (standard says daily, not happening)	Section 3.2 — Gap closure plan
AI server 92% disk critical	Section 9 — P0 action item #1
Memory consolidation data loss risk	Consolidation proposal (separate doc)
Tony's Hermes undocumented	Section 6.10 + P1 action item #7

10.2 Reviewer 2 Findings — Addressed

Finding	Addressed In
S3 single point of failure	Section 3.3 — Two-region plan
Backup frequency vs RPO mismatch	Section 3.1 — Matrix with status
Restore procedures not detailed	Section 6 — Full runbooks per server
Failback under-defined	Section 4.4 — Protocol with no-auto-failback rule
No backup restore testing	Section 7 — Schedule
48h retention too short	P1 item #9 — Extend to 30 days
Backup security not described	Section 8.2
Monitoring not fully defined	Section 5
Application backup details vague	Section 2.3 + Section 6
Secrets recovery not documented	Section 2.2 (base install creds) + Section 8.3
Provider/account outage missing	Section 4.5

This plan supersedes all previous DR documentation. Review quarterly or after any infrastructure change.

23 KiB Raw Blame History

IT Pro Partner — Disaster Recovery Plan v3 (Consolidated)

Table of Contents

1. Architecture & Shared Dependencies

Shared Dependencies — Documented

2. Server Standards

2.1 Tier Definitions

2.2 Base Install — Verified Checklist

2.3 Application Data Paths — Per Server

3. Backup Strategy

3.1 Backup Matrix

3.2 Immediate Backup Gaps to Close

3.3 Backup Security

4. Failover / Failback

4.1 Warm Standby — Corrected Timing

4.2 Health Check — Two Layers

4.3 Fencing Protocol

4.4 Failback Protocol

4.5 Provider/Account Outage Scenario

5. Monitoring & Alerting

5.1 Alert Thresholds

5.2 Monitoring Stack

5.3 Alerting Gaps to Close

6. Restore Runbooks (Per Server)

6.1 Core (netcup RS 2000) — 🔴 Critical

6.2 ai.itpropartner.com (Hetzner CPX41) — 🔴 Critical

6.3 wphost02 (Hetzner CPX21, RunCloud) — 🟡 Important

6.4 UNMS (Hetzner CPX21) — 🟡 Important

6.5 Hudu (Hetzner CPX21) — 🟡 Important

6.6 UniFi (Hetzner CPX21) — 🟡 Important

6.7 n8n (Hetzner CPX11) — 🟢 Standard

6.8 docker box (Hetzner CPX11) — 🟢 Standard

6.9 fleettracker360 (Hetzner CPX11) — 🟢 Standard

6.10 tony-vps (Hetzner CPX21) — 🟢 Standard

6.11 app1-bu (Hetzner CPX21) — 🔴 Standby

7. Testing & Validation

7.1 Restore Testing Schedule

7.2 Validation Criteria

7.3 Quarterly DR Simulation

8. Security

8.1 Current Security Posture

8.2 Security Gaps to Close

8.3 Break-Glass Access

9. Immediate Action Items

🔴 P0 — This Week

🟡 P1 — Next 2 Weeks

🟢 P2 — Next 30 Days

10. Appendix: Third-Party Review Responses

10.1 Reviewer 1 Findings — Addressed

10.2 Reviewer 2 Findings — Addressed

23 KiB

Raw Blame History