docs: update DR plans and recovery docs

This commit is contained in:
root
2026-08-08 13:07:31 -04:00
parent 1b05867dd1
commit 25f11bcdd4
4 changed files with 19 additions and 13 deletions
+6
View File
@@ -0,0 +1,6 @@
# disaster-recovery — CHANGELOG
## 2026-07-16 — Audit Remediation
- Created CHANGELOG.md (missing per project documentation standard)
- Project directory: `/root/projects/disaster-recovery`
+2 -2
View File
@@ -14,7 +14,7 @@
## Quick Links ## Quick Links
- **Warm standby:** app1-bu (5.161.114.8) checks Core every 5 min, auto-failover - **Warm standby:** app1-bu (5.161.225.131) checks Core every 5 min, auto-failover
- **S3 backups:** hermes-vps-backups, itpropartner-system-configs, itpropartner-docker-volumes, mikrotik-ccr-backups - **S3 backups:** hermes-vps-backups, itpropartner-system-configs, itpropartner-docker-volumes, mikrotik-ccr-backups
- **Failover RTO:** 5 min committed (actual ~3 min) - **Failover RTO:** 5 min committed (actual ~3 min)
- **Core RPO:** 15 min (Hermes state) / 24h (Docker data) - **Core RPO:** 15 min (Hermes state) / 24h (Docker data)
@@ -30,7 +30,7 @@ Core (netcup RS2000, 152.53.192.33)
Wasabi S3 (hermes-vps-backups/live/) Wasabi S3 (hermes-vps-backups/live/)
app1-bu (Hetzner CPX11, 5.161.114.8) app1-bu (Hetzner CPX11, 5.161.225.131)
├── Dormant Hermes ├── Dormant Hermes
└── Sync from S3 every 10 min └── Sync from S3 every 10 min
``` ```
+10 -10
View File
@@ -12,7 +12,7 @@
1. [Architecture Overview](#1-architecture-overview) 1. [Architecture Overview](#1-architecture-overview)
2. [Core Server (netcup — 152.53.192.33)](#2-core-server-netcup--1525319233) 2. [Core Server (netcup — 152.53.192.33)](#2-core-server-netcup--1525319233)
3. [Hermes Agent (AI Assistant)](#3-hermes-agent-ai-assistant) 3. [Hermes Agent (AI Assistant)](#3-hermes-agent-ai-assistant)
4. [app1-bu Standby (Hetzner — 5.161.114.8)](#4-app1-bu-standby-hetzner--51611148) 4. [app1-bu Standby (Hetzner — 5.161.225.131)](#4-app1-bu-standby-hetzner--51611148)
5. [Docker Services on Core](#5-docker-services-on-core) 5. [Docker Services on Core](#5-docker-services-on-core)
- [5.1 Twenty CRM (crm.debtrecoveryexperts.com)](#51-twenty-crm) - [5.1 Twenty CRM (crm.debtrecoveryexperts.com)](#51-twenty-crm)
- [5.2 Vaultwarden Password Manager](#52-vaultwarden-password-manager) - [5.2 Vaultwarden Password Manager](#52-vaultwarden-password-manager)
@@ -43,7 +43,7 @@
| # | Server | IP | Provider | Specs | OS | What It Runs | | # | Server | IP | Provider | Specs | OS | What It Runs |
|---|--------|----|----------|-------|----|-------------| |---|--------|----|----------|-------|----|-------------|
| **1** | **Core** (Primary) | 152.53.192.33 | netcup RS 2000 | 8C/15G/512G | Debian 13 | Hermes, Caddy, Docker (Twenty, Vaultwarden, Docuseal, SearXNG), Shark Game, Ollama, VPN | | **1** | **Core** (Primary) | 152.53.192.33 | netcup RS 2000 | 8C/15G/512G | Debian 13 | Hermes, Caddy, Docker (Twenty, Vaultwarden, Docuseal, SearXNG), Shark Game, Ollama, VPN |
| **2** | **app1-bu** (Standby) | 5.161.114.8 | Hetzner CPX11 | 2C/2G/40G | Ubuntu 26.04 | Hermes standby, StrongSwan, L2TP | | **2** | **app1-bu** (Standby) | 5.161.225.131 | Hetzner CPX11 | 2C/2G/40G | Ubuntu 26.04 | Hermes standby, StrongSwan, L2TP |
| **3** | **ai.itpropartner.com** | 178.156.167.181 | Hetzner CPX41 | 8C/16G/240G | Ubuntu 24.04 | LiteLLM, Open WebUI, Ollama, Qdrant | | **3** | **ai.itpropartner.com** | 178.156.167.181 | Hetzner CPX41 | 8C/16G/240G | Ubuntu 24.04 | LiteLLM, Open WebUI, Ollama, Qdrant |
| **4** | **hudu.itpropartner.com** | 178.156.130.130 | Hetzner CPX21 | 3C/4G/80G | Ubuntu 24.04 | Hudu Docs, SWAG, MariaDB | | **4** | **hudu.itpropartner.com** | 178.156.130.130 | Hetzner CPX21 | 3C/4G/80G | Ubuntu 24.04 | Hudu Docs, SWAG, MariaDB |
| **5** | **unifi** | 178.156.131.57 | Hetzner CPX21 | 3C/4G/80G | Ubuntu 24.04 | UniFi Controller, MongoDB, MariaDB | | **5** | **unifi** | 178.156.131.57 | Hetzner CPX21 | 3C/4G/80G | Ubuntu 24.04 | UniFi Controller, MongoDB, MariaDB |
@@ -266,7 +266,7 @@ systemctl start hermes
--- ---
## 4. app1-bu Standby (Hetzner — 5.161.114.8) ## 4. app1-bu Standby (Hetzner — 5.161.225.131)
**What it is:** A warm standby server on Hetzner. If the Core server (netcup) dies, this server takes over as Hermes. **What it is:** A warm standby server on Hetzner. If the Core server (netcup) dies, this server takes over as Hermes.
@@ -274,7 +274,7 @@ systemctl start hermes
| Detail | Value | | Detail | Value |
|--------|-------| |--------|-------|
| **IP** | 5.161.114.8 | | **IP** | 5.161.225.131 |
| **Hostname** | app1-bu.itpropartner.com | | **Hostname** | app1-bu.itpropartner.com |
| **Plan** | CPX11 (2C/2G/40G) | | **Plan** | CPX11 (2C/2G/40G) |
| **OS** | Ubuntu 26.04 | | **OS** | Ubuntu 26.04 |
@@ -283,7 +283,7 @@ systemctl start hermes
### SSH Access ### SSH Access
```bash ```bash
ssh root@5.161.114.8 -i /root/.ssh/itpp-infra ssh root@5.161.225.131 -i /root/.ssh/itpp-infra
``` ```
### What Runs on Standby (Normally Dormant) ### What Runs on Standby (Normally Dormant)
@@ -312,7 +312,7 @@ The watchdog script runs every 5 minutes via cron (`*/5 * * * * /root/.hermes/sc
If you need to manually switch to standby: If you need to manually switch to standby:
```bash ```bash
ssh root@5.161.114.8 -i /root/.ssh/itpp-infra 'systemctl start hermes-gateway' ssh root@5.161.225.131 -i /root/.ssh/itpp-infra 'systemctl start hermes-gateway'
``` ```
### How to Demote (Switch Back to Core) ### How to Demote (Switch Back to Core)
@@ -320,7 +320,7 @@ ssh root@5.161.114.8 -i /root/.ssh/itpp-infra 'systemctl start hermes-gateway'
After Core is restored: After Core is restored:
```bash ```bash
ssh root@5.161.114.8 -i /root/.ssh/itpp-infra 'systemctl stop hermes-gateway' ssh root@5.161.225.131 -i /root/.ssh/itpp-infra 'systemctl stop hermes-gateway'
``` ```
Then make sure Hermes is running on Core: Then make sure Hermes is running on Core:
@@ -1182,12 +1182,12 @@ bash /root/.hermes/scripts/run-wisp-backup.sh
2. **If automatic failover doesn't trigger**, do it manually: 2. **If automatic failover doesn't trigger**, do it manually:
```bash ```bash
ssh root@5.161.114.8 -i /root/.ssh/itpp-infra 'systemctl start hermes-gateway' ssh root@5.161.225.131 -i /root/.ssh/itpp-infra 'systemctl start hermes-gateway'
``` ```
3. **Verify Hermes is running on standby:** 3. **Verify Hermes is running on standby:**
```bash ```bash
ssh root@5.161.114.8 -i /root/.ssh/itpp-infra 'systemctl status hermes-gateway' ssh root@5.161.225.131 -i /root/.ssh/itpp-infra 'systemctl status hermes-gateway'
``` ```
4. **Order a new netcup server:** 4. **Order a new netcup server:**
@@ -1320,7 +1320,7 @@ TZ=America/New_York
**Hermes internal cron jobs** (managed by Hermes itself): **Hermes internal cron jobs** (managed by Hermes itself):
- Various ticker-based tasks defined in `/root/.hermes/cron/jobs.json` - Various ticker-based tasks defined in `/root/.hermes/cron/jobs.json`
### On app1-bu (5.161.114.8) ### On app1-bu (5.161.225.131)
``` ```
*/5 * * * * /root/.hermes/scripts/hermes-standby-watchdog.sh */5 * * * * /root/.hermes/scripts/hermes-standby-watchdog.sh
+1 -1
View File
@@ -26,7 +26,7 @@
| Name | Type | IP | Purpose | | Name | Type | IP | Purpose |
|------|------|----|---------| |------|------|----|---------|
| wphost02 | CPX21 | 5.161.62.38 | Apex WP + RunCloud | | wphost02 | CPX21 | 5.161.62.38 | Apex WP + RunCloud |
| app1-bu | CPX11 | 5.161.114.8 | Hermes warm standby | | app1-bu | CPX11 | 5.161.225.131 | Hermes warm standby |
| tony-vps | CPX21 | 87.99.159.142 | Tony's Hermes | | tony-vps | CPX21 | 87.99.159.142 | Tony's Hermes |
| unms | CPX21 | 5.161.225.131 | UISP/UNMS | | unms | CPX21 | 5.161.225.131 | UISP/UNMS |
| unifi | CPX21 | 178.156.131.57 | UniFi controller | | unifi | CPX21 | 178.156.131.57 | UniFi controller |