ippadmin/disaster-recovery
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: update DR plans and recovery docs

Browse Source
This commit is contained in:
root
2026-08-08 13:07:31 -04:00
parent 1b05867dd1
commit 25f11bcdd4
4 changed files with 19 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+6
View File
@@ -0,0 +1,6 @@
# disaster-recovery — CHANGELOG
## 2026-07-16 — Audit Remediation
- Created CHANGELOG.md (missing per project documentation standard)
- Project directory: `/root/projects/disaster-recovery`
README.md
+2 -2
View File
@@ -14,7 +14,7 @@
## Quick Links
- **Warm standby:** app1-bu (5.161.114.8) checks Core every 5 min, auto-failover
- **Warm standby:** app1-bu (5.161.225.131) checks Core every 5 min, auto-failover
- **S3 backups:** hermes-vps-backups, itpropartner-system-configs, itpropartner-docker-volumes, mikrotik-ccr-backups
- **Failover RTO:** 5 min committed (actual ~3 min)
- **Core RPO:** 15 min (Hermes state) / 24h (Docker data)
@@ -30,7 +30,7 @@ Core (netcup RS2000, 152.53.192.33)
Wasabi S3 (hermes-vps-backups/live/)
│
▼
app1-bu (Hetzner CPX11, 5.161.114.8)
app1-bu (Hetzner CPX11, 5.161.225.131)
├── Dormant Hermes
└── Sync from S3 every 10 min
```
itpp-recovery-manual.md
+10 -10
View File
@@ -12,7 +12,7 @@
1. [Architecture Overview](#1-architecture-overview)
2. [Core Server (netcup — 152.53.192.33)](#2-core-server-netcup--1525319233)
3. [Hermes Agent (AI Assistant)](#3-hermes-agent-ai-assistant)
4. [app1-bu Standby (Hetzner — 5.161.114.8)](#4-app1-bu-standby-hetzner--51611148)
4. [app1-bu Standby (Hetzner — 5.161.225.131)](#4-app1-bu-standby-hetzner--51611148)
5. [Docker Services on Core](#5-docker-services-on-core)
- [5.1 Twenty CRM (crm.debtrecoveryexperts.com)](#51-twenty-crm)
- [5.2 Vaultwarden Password Manager](#52-vaultwarden-password-manager)
@@ -43,7 +43,7 @@
| # | Server | IP | Provider | Specs | OS | What It Runs |
|---|--------|----|----------|-------|----|-------------|
| **1** | **Core** (Primary) | 152.53.192.33 | netcup RS 2000 | 8C/15G/512G | Debian 13 | Hermes, Caddy, Docker (Twenty, Vaultwarden, Docuseal, SearXNG), Shark Game, Ollama, VPN |
| **2** | **app1-bu** (Standby) | 5.161.114.8 | Hetzner CPX11 | 2C/2G/40G | Ubuntu 26.04 | Hermes standby, StrongSwan, L2TP |
| **2** | **app1-bu** (Standby) | 5.161.225.131 | Hetzner CPX11 | 2C/2G/40G | Ubuntu 26.04 | Hermes standby, StrongSwan, L2TP |
| **3** | **ai.itpropartner.com** | 178.156.167.181 | Hetzner CPX41 | 8C/16G/240G | Ubuntu 24.04 | LiteLLM, Open WebUI, Ollama, Qdrant |
| **4** | **hudu.itpropartner.com** | 178.156.130.130 | Hetzner CPX21 | 3C/4G/80G | Ubuntu 24.04 | Hudu Docs, SWAG, MariaDB |
| **5** | **unifi** | 178.156.131.57 | Hetzner CPX21 | 3C/4G/80G | Ubuntu 24.04 | UniFi Controller, MongoDB, MariaDB |
@@ -266,7 +266,7 @@ systemctl start hermes
---
## 4. app1-bu Standby (Hetzner — 5.161.114.8)
## 4. app1-bu Standby (Hetzner — 5.161.225.131)
**What it is:** A warm standby server on Hetzner. If the Core server (netcup) dies, this server takes over as Hermes.
@@ -274,7 +274,7 @@ systemctl start hermes
| Detail | Value |
|--------|-------|
| **IP** | 5.161.114.8 |
| **IP** | 5.161.225.131 |
| **Hostname** | app1-bu.itpropartner.com |
| **Plan** | CPX11 (2C/2G/40G) |
| **OS** | Ubuntu 26.04 |
@@ -283,7 +283,7 @@ systemctl start hermes
### SSH Access
```bash
ssh root@5.161.114.8 -i /root/.ssh/itpp-infra
ssh root@5.161.225.131 -i /root/.ssh/itpp-infra
```
### What Runs on Standby (Normally Dormant)
@@ -312,7 +312,7 @@ The watchdog script runs every 5 minutes via cron (`*/5 * * * * /root/.hermes/sc
If you need to manually switch to standby:
```bash
ssh root@5.161.114.8 -i /root/.ssh/itpp-infra 'systemctl start hermes-gateway'
ssh root@5.161.225.131 -i /root/.ssh/itpp-infra 'systemctl start hermes-gateway'
```
### How to Demote (Switch Back to Core)
@@ -320,7 +320,7 @@ ssh root@5.161.114.8 -i /root/.ssh/itpp-infra 'systemctl start hermes-gateway'
After Core is restored:
```bash
ssh root@5.161.114.8 -i /root/.ssh/itpp-infra 'systemctl stop hermes-gateway'
ssh root@5.161.225.131 -i /root/.ssh/itpp-infra 'systemctl stop hermes-gateway'
```
Then make sure Hermes is running on Core:
@@ -1182,12 +1182,12 @@ bash /root/.hermes/scripts/run-wisp-backup.sh
2. **If automatic failover doesn't trigger**, do it manually:
```bash
ssh root@5.161.114.8 -i /root/.ssh/itpp-infra 'systemctl start hermes-gateway'
ssh root@5.161.225.131 -i /root/.ssh/itpp-infra 'systemctl start hermes-gateway'
```
3. **Verify Hermes is running on standby:**
```bash
ssh root@5.161.114.8 -i /root/.ssh/itpp-infra 'systemctl status hermes-gateway'
ssh root@5.161.225.131 -i /root/.ssh/itpp-infra 'systemctl status hermes-gateway'
```
4. **Order a new netcup server:**
@@ -1320,7 +1320,7 @@ TZ=America/New_York
**Hermes internal cron jobs** (managed by Hermes itself):
- Various ticker-based tasks defined in `/root/.hermes/cron/jobs.json`
### On app1-bu (5.161.114.8)
### On app1-bu (5.161.225.131)
```
*/5 * * * * /root/.hermes/scripts/hermes-standby-watchdog.sh
network-and-service-endpoints.md
+1 -1
View File
@@ -26,7 +26,7 @@
| Name | Type | IP | Purpose |
|------|------|----|---------|
| wphost02 | CPX21 | 5.161.62.38 | Apex WP + RunCloud |
| app1-bu | CPX11 | 5.161.114.8 | Hermes warm standby |
| app1-bu | CPX11 | 5.161.225.131 | Hermes warm standby |
| tony-vps | CPX21 | 87.99.159.142 | Tony's Hermes |
| unms | CPX21 | 5.161.225.131 | UISP/UNMS |
| unifi | CPX21 | 178.156.131.57 | UniFi controller |