docs: update DR plans and recovery docs

This commit is contained in:
root
2026-08-08 13:07:31 -04:00
parent 1b05867dd1
commit 25f11bcdd4
4 changed files with 19 additions and 13 deletions
+10 -10
View File
@@ -12,7 +12,7 @@
1. [Architecture Overview](#1-architecture-overview)
2. [Core Server (netcup — 152.53.192.33)](#2-core-server-netcup--1525319233)
3. [Hermes Agent (AI Assistant)](#3-hermes-agent-ai-assistant)
4. [app1-bu Standby (Hetzner — 5.161.114.8)](#4-app1-bu-standby-hetzner--51611148)
4. [app1-bu Standby (Hetzner — 5.161.225.131)](#4-app1-bu-standby-hetzner--51611148)
5. [Docker Services on Core](#5-docker-services-on-core)
- [5.1 Twenty CRM (crm.debtrecoveryexperts.com)](#51-twenty-crm)
- [5.2 Vaultwarden Password Manager](#52-vaultwarden-password-manager)
@@ -43,7 +43,7 @@
| # | Server | IP | Provider | Specs | OS | What It Runs |
|---|--------|----|----------|-------|----|-------------|
| **1** | **Core** (Primary) | 152.53.192.33 | netcup RS 2000 | 8C/15G/512G | Debian 13 | Hermes, Caddy, Docker (Twenty, Vaultwarden, Docuseal, SearXNG), Shark Game, Ollama, VPN |
| **2** | **app1-bu** (Standby) | 5.161.114.8 | Hetzner CPX11 | 2C/2G/40G | Ubuntu 26.04 | Hermes standby, StrongSwan, L2TP |
| **2** | **app1-bu** (Standby) | 5.161.225.131 | Hetzner CPX11 | 2C/2G/40G | Ubuntu 26.04 | Hermes standby, StrongSwan, L2TP |
| **3** | **ai.itpropartner.com** | 178.156.167.181 | Hetzner CPX41 | 8C/16G/240G | Ubuntu 24.04 | LiteLLM, Open WebUI, Ollama, Qdrant |
| **4** | **hudu.itpropartner.com** | 178.156.130.130 | Hetzner CPX21 | 3C/4G/80G | Ubuntu 24.04 | Hudu Docs, SWAG, MariaDB |
| **5** | **unifi** | 178.156.131.57 | Hetzner CPX21 | 3C/4G/80G | Ubuntu 24.04 | UniFi Controller, MongoDB, MariaDB |
@@ -266,7 +266,7 @@ systemctl start hermes
---
## 4. app1-bu Standby (Hetzner — 5.161.114.8)
## 4. app1-bu Standby (Hetzner — 5.161.225.131)
**What it is:** A warm standby server on Hetzner. If the Core server (netcup) dies, this server takes over as Hermes.
@@ -274,7 +274,7 @@ systemctl start hermes
| Detail | Value |
|--------|-------|
| **IP** | 5.161.114.8 |
| **IP** | 5.161.225.131 |
| **Hostname** | app1-bu.itpropartner.com |
| **Plan** | CPX11 (2C/2G/40G) |
| **OS** | Ubuntu 26.04 |
@@ -283,7 +283,7 @@ systemctl start hermes
### SSH Access
```bash
ssh root@5.161.114.8 -i /root/.ssh/itpp-infra
ssh root@5.161.225.131 -i /root/.ssh/itpp-infra
```
### What Runs on Standby (Normally Dormant)
@@ -312,7 +312,7 @@ The watchdog script runs every 5 minutes via cron (`*/5 * * * * /root/.hermes/sc
If you need to manually switch to standby:
```bash
ssh root@5.161.114.8 -i /root/.ssh/itpp-infra 'systemctl start hermes-gateway'
ssh root@5.161.225.131 -i /root/.ssh/itpp-infra 'systemctl start hermes-gateway'
```
### How to Demote (Switch Back to Core)
@@ -320,7 +320,7 @@ ssh root@5.161.114.8 -i /root/.ssh/itpp-infra 'systemctl start hermes-gateway'
After Core is restored:
```bash
ssh root@5.161.114.8 -i /root/.ssh/itpp-infra 'systemctl stop hermes-gateway'
ssh root@5.161.225.131 -i /root/.ssh/itpp-infra 'systemctl stop hermes-gateway'
```
Then make sure Hermes is running on Core:
@@ -1182,12 +1182,12 @@ bash /root/.hermes/scripts/run-wisp-backup.sh
2. **If automatic failover doesn't trigger**, do it manually:
```bash
ssh root@5.161.114.8 -i /root/.ssh/itpp-infra 'systemctl start hermes-gateway'
ssh root@5.161.225.131 -i /root/.ssh/itpp-infra 'systemctl start hermes-gateway'
```
3. **Verify Hermes is running on standby:**
```bash
ssh root@5.161.114.8 -i /root/.ssh/itpp-infra 'systemctl status hermes-gateway'
ssh root@5.161.225.131 -i /root/.ssh/itpp-infra 'systemctl status hermes-gateway'
```
4. **Order a new netcup server:**
@@ -1320,7 +1320,7 @@ TZ=America/New_York
**Hermes internal cron jobs** (managed by Hermes itself):
- Various ticker-based tasks defined in `/root/.hermes/cron/jobs.json`
### On app1-bu (5.161.114.8)
### On app1-bu (5.161.225.131)
```
*/5 * * * * /root/.hermes/scripts/hermes-standby-watchdog.sh