diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..946e435 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,6 @@ +# disaster-recovery — CHANGELOG + +## 2026-07-16 — Audit Remediation + +- Created CHANGELOG.md (missing per project documentation standard) +- Project directory: `/root/projects/disaster-recovery` diff --git a/README.md b/README.md index aff35cf..f3a400c 100644 --- a/README.md +++ b/README.md @@ -14,7 +14,7 @@ ## Quick Links -- **Warm standby:** app1-bu (5.161.114.8) checks Core every 5 min, auto-failover +- **Warm standby:** app1-bu (5.161.225.131) checks Core every 5 min, auto-failover - **S3 backups:** hermes-vps-backups, itpropartner-system-configs, itpropartner-docker-volumes, mikrotik-ccr-backups - **Failover RTO:** 5 min committed (actual ~3 min) - **Core RPO:** 15 min (Hermes state) / 24h (Docker data) @@ -30,7 +30,7 @@ Core (netcup RS2000, 152.53.192.33) Wasabi S3 (hermes-vps-backups/live/) │ ▼ -app1-bu (Hetzner CPX11, 5.161.114.8) +app1-bu (Hetzner CPX11, 5.161.225.131) ├── Dormant Hermes └── Sync from S3 every 10 min ``` diff --git a/itpp-recovery-manual.md b/itpp-recovery-manual.md index bb9af0f..a2ab5ff 100644 --- a/itpp-recovery-manual.md +++ b/itpp-recovery-manual.md @@ -12,7 +12,7 @@ 1. [Architecture Overview](#1-architecture-overview) 2. [Core Server (netcup — 152.53.192.33)](#2-core-server-netcup--1525319233) 3. [Hermes Agent (AI Assistant)](#3-hermes-agent-ai-assistant) -4. [app1-bu Standby (Hetzner — 5.161.114.8)](#4-app1-bu-standby-hetzner--51611148) +4. [app1-bu Standby (Hetzner — 5.161.225.131)](#4-app1-bu-standby-hetzner--51611148) 5. [Docker Services on Core](#5-docker-services-on-core) - [5.1 Twenty CRM (crm.debtrecoveryexperts.com)](#51-twenty-crm) - [5.2 Vaultwarden Password Manager](#52-vaultwarden-password-manager) @@ -43,7 +43,7 @@ | # | Server | IP | Provider | Specs | OS | What It Runs | |---|--------|----|----------|-------|----|-------------| | **1** | **Core** (Primary) | 152.53.192.33 | netcup RS 2000 | 8C/15G/512G | Debian 13 | Hermes, Caddy, Docker (Twenty, Vaultwarden, Docuseal, SearXNG), Shark Game, Ollama, VPN | -| **2** | **app1-bu** (Standby) | 5.161.114.8 | Hetzner CPX11 | 2C/2G/40G | Ubuntu 26.04 | Hermes standby, StrongSwan, L2TP | +| **2** | **app1-bu** (Standby) | 5.161.225.131 | Hetzner CPX11 | 2C/2G/40G | Ubuntu 26.04 | Hermes standby, StrongSwan, L2TP | | **3** | **ai.itpropartner.com** | 178.156.167.181 | Hetzner CPX41 | 8C/16G/240G | Ubuntu 24.04 | LiteLLM, Open WebUI, Ollama, Qdrant | | **4** | **hudu.itpropartner.com** | 178.156.130.130 | Hetzner CPX21 | 3C/4G/80G | Ubuntu 24.04 | Hudu Docs, SWAG, MariaDB | | **5** | **unifi** | 178.156.131.57 | Hetzner CPX21 | 3C/4G/80G | Ubuntu 24.04 | UniFi Controller, MongoDB, MariaDB | @@ -266,7 +266,7 @@ systemctl start hermes --- -## 4. app1-bu Standby (Hetzner — 5.161.114.8) +## 4. app1-bu Standby (Hetzner — 5.161.225.131) **What it is:** A warm standby server on Hetzner. If the Core server (netcup) dies, this server takes over as Hermes. @@ -274,7 +274,7 @@ systemctl start hermes | Detail | Value | |--------|-------| -| **IP** | 5.161.114.8 | +| **IP** | 5.161.225.131 | | **Hostname** | app1-bu.itpropartner.com | | **Plan** | CPX11 (2C/2G/40G) | | **OS** | Ubuntu 26.04 | @@ -283,7 +283,7 @@ systemctl start hermes ### SSH Access ```bash -ssh root@5.161.114.8 -i /root/.ssh/itpp-infra +ssh root@5.161.225.131 -i /root/.ssh/itpp-infra ``` ### What Runs on Standby (Normally Dormant) @@ -312,7 +312,7 @@ The watchdog script runs every 5 minutes via cron (`*/5 * * * * /root/.hermes/sc If you need to manually switch to standby: ```bash -ssh root@5.161.114.8 -i /root/.ssh/itpp-infra 'systemctl start hermes-gateway' +ssh root@5.161.225.131 -i /root/.ssh/itpp-infra 'systemctl start hermes-gateway' ``` ### How to Demote (Switch Back to Core) @@ -320,7 +320,7 @@ ssh root@5.161.114.8 -i /root/.ssh/itpp-infra 'systemctl start hermes-gateway' After Core is restored: ```bash -ssh root@5.161.114.8 -i /root/.ssh/itpp-infra 'systemctl stop hermes-gateway' +ssh root@5.161.225.131 -i /root/.ssh/itpp-infra 'systemctl stop hermes-gateway' ``` Then make sure Hermes is running on Core: @@ -1182,12 +1182,12 @@ bash /root/.hermes/scripts/run-wisp-backup.sh 2. **If automatic failover doesn't trigger**, do it manually: ```bash - ssh root@5.161.114.8 -i /root/.ssh/itpp-infra 'systemctl start hermes-gateway' + ssh root@5.161.225.131 -i /root/.ssh/itpp-infra 'systemctl start hermes-gateway' ``` 3. **Verify Hermes is running on standby:** ```bash - ssh root@5.161.114.8 -i /root/.ssh/itpp-infra 'systemctl status hermes-gateway' + ssh root@5.161.225.131 -i /root/.ssh/itpp-infra 'systemctl status hermes-gateway' ``` 4. **Order a new netcup server:** @@ -1320,7 +1320,7 @@ TZ=America/New_York **Hermes internal cron jobs** (managed by Hermes itself): - Various ticker-based tasks defined in `/root/.hermes/cron/jobs.json` -### On app1-bu (5.161.114.8) +### On app1-bu (5.161.225.131) ``` */5 * * * * /root/.hermes/scripts/hermes-standby-watchdog.sh diff --git a/network-and-service-endpoints.md b/network-and-service-endpoints.md index bbbee4d..9da025a 100644 --- a/network-and-service-endpoints.md +++ b/network-and-service-endpoints.md @@ -26,7 +26,7 @@ | Name | Type | IP | Purpose | |------|------|----|---------| | wphost02 | CPX21 | 5.161.62.38 | Apex WP + RunCloud | -| app1-bu | CPX11 | 5.161.114.8 | Hermes warm standby | +| app1-bu | CPX11 | 5.161.225.131 | Hermes warm standby | | tony-vps | CPX21 | 87.99.159.142 | Tony's Hermes | | unms | CPX21 | 5.161.225.131 | UISP/UNMS | | unifi | CPX21 | 178.156.131.57 | UniFi controller |