commit 4ca8be4b6ac335101eaf996e8168c6918f3aa318 Author: root Date: Wed Jul 15 17:41:23 2026 -0400 Initial DR plan — v3 with per-server RTO/RPO, recovery manual, issue log diff --git a/README.md b/README.md new file mode 100644 index 0000000..aff35cf --- /dev/null +++ b/README.md @@ -0,0 +1,36 @@ +# Disaster Recovery Plan + +**Version:** 3.0 (Jul 10, 2026) +**Status:** Active + +## Documents + +| File | Purpose | +|---|---| +| `server-dr-plans-v3.md` | Per-server DR plans with RTO/RPO targets | +| `itpp-recovery-manual.md` | Full infrastructure recovery manual | +| `dr-issue-log.md` | Permanent issue tracking — all audit findings | +| `network-and-service-endpoints.md` | IPs, ports, URLs, S3 buckets | + +## Quick Links + +- **Warm standby:** app1-bu (5.161.114.8) checks Core every 5 min, auto-failover +- **S3 backups:** hermes-vps-backups, itpropartner-system-configs, itpropartner-docker-volumes, mikrotik-ccr-backups +- **Failover RTO:** 5 min committed (actual ~3 min) +- **Core RPO:** 15 min (Hermes state) / 24h (Docker data) + +## Architecture + +``` +Core (netcup RS2000, 152.53.192.33) + ├── Active Hermes + gateway + └── S3 sync every 15 min + │ + ▼ +Wasabi S3 (hermes-vps-backups/live/) + │ + ▼ +app1-bu (Hetzner CPX11, 5.161.114.8) + ├── Dormant Hermes + └── Sync from S3 every 10 min +``` diff --git a/dr-issue-log.md b/dr-issue-log.md new file mode 100644 index 0000000..c94ead9 --- /dev/null +++ b/dr-issue-log.md @@ -0,0 +1,82 @@ +# DR Issue Log — Hermes Infrastructure + +Tracked issues found during DR audits. Each entry: date, problem, root cause, fix applied, verification. + +--- + +## 2026-07-08 — Initial Full DR Audit + +### DR-001: Caddyfile not included in backup scripts +**Problem:** /etc/caddy/Caddyfile was not copied by hermes-backup.sh or hermes-live-sync.sh. If Core server fails, the full Caddy reverse proxy config would need to be rebuilt from scratch. +**Root cause:** Backup scripts were written to cover hermes config and user directories but omitted system-level config files. +**Fix:** Added /etc/caddy/Caddyfile to hermes-backup.sh and hermes-live-sync.sh (with DR FIX: comments and dates). +**Status:** ✅ Fixed & verified 2026-07-08 +**Verified by:** bash -n syntax check + subagent confirmed all paths in script + +### DR-002: Systemd service files backed up to wrong path +**Problem:** hermes-backup.sh collected systemd services from ~/.config/systemd/user/ instead of /etc/systemd/system/. The real service files (/etc/systemd/system/hermes-agent.service, shark-game.service, etc.) were not backed up. +**Root cause:** Backup script path pointed to user-level systemd vs system-level. +**Fix:** Corrected path in hermes-backup.sh to /etc/systemd/system/*.service. Also updated the embedded restore.sh heredoc. +**Status:** ✅ Fixed & verified 2026-07-08 +**Verified by:** bash -n syntax check + subagent confirmed paths in script + +### DR-003: migration-creds.txt not in backup scope +**Problem:** /root/.hermes/migration-creds.txt existed but wasn't referenced by any backup script. +**Root cause:** Added after backup script was written, never included in scope. +**Fix:** Added to hermes-backup.sh file list with chmod 600 restore. +**Status:** ✅ Fixed & verified 2026-07-08 +**Verified by:** bash -n syntax check + subagent confirmed file referenced + +### DR-004: dre-temp-passwords.txt exposed at 644 permissions +**Problem:** /root/.hermes/references/dre-temp-passwords.txt was readable by all users (644) instead of owner-only (600). +**Root cause:** Script created the file without explicit permission setting. +**Fix:** chmod 600 +**Status:** ✅ Fixed 2026-07-08 +**Verified by:** ls -la confirms 600 + +### DR-005: migration-creds.txt at 644 permissions +**Problem:** Same issue as DR-004 — migration-creds.txt was 644. +**Root cause:** Written without explicit permission setting. +**Fix:** chmod 600 +**Status:** ✅ Fixed 2026-07-08 +**Verified by:** ls -la confirms 600 + +### DR-006: Full backup had no cron job (last ran Jul 5) +**Problem:** hermes-full-backup.tar.gz last uploaded to S3 on Jul 5. The daily 5AM cron had no scheduled job — the backup script existed but nothing was calling it. +**Root cause:** Cron job was never created for the full backup script. The hermes-live-sync covered configs every 15 min but full archive was orphaned. +**Fix:** Created cron job `hermes-full-backup` at `0 5 * * *` calling `run-hermes-backup.sh` wrapper. +**Status:** ✅ Fixed & verified 2026-07-08 +**Verified by:** Cron job created and scheduled for next 5 AM ET + +### DR-007: home-router-daily-backup cron error +**Problem:** Cron job errored at 06:00 today. SSH export on router produced a stuck .in_progress file that never completed. +**Root cause:** Cron was using home-router-backup.sh (old WireGuard tunnel script) instead of the proper run-wisp-backup.sh pipeline. Stuck export files blocked SCP. +**Fix:** Changed cron to use run-wisp-backup.sh. Cleaned stuck .in_progress files from router. Missing packages (paramiko, xl2tpd, strongSwan) installed. +**Status:** ✅ Fixed & verified 2026-07-08 +**Verified by:** Backup ran end-to-end, config uploaded to S3 + +### DR-008: MikroTik CCR backup stale (last Jul 5) +**Problem:** mikrotik-ccr-backups S3 bucket had no uploads since Jul 5. +**Root cause (two causes):** (1) wisp-backup.py failed because paramiko wasn't installed. (2) tower IP in config.yaml was 192.168.88.1 (LAN) but SSH is restricted to WireGuard tunnel network 10.77.0.0/24. +**Fix:** Installed paramiko v5.0.0. Updated tower IP to 10.77.0.2 in wisp-backup/config.yaml. Installed missing VPN stack (xl2tpd, strongSwan). +**Status:** ✅ Fixed & verified 2026-07-08 +**Verified by:** Backup ran end-to-end, config uploaded to S3 + +### DR-009: app1-bu warm vs cold docs mismatch +**Problem:** DR plan doc said "offline, boots on demand" but server is running (3 days uptime). +**Root cause:** DR plan doc was outdated — actual design is warm standby (always on, Hermes dormant). +**Fix:** Updated DR plan to reflect warm standby design. Server stays running. +**Status:** ✅ Resolved — not a bug, docs were wrong + +### DR-010: Failover timing change (per Germaine's direction) +**Change:** Check interval from every 10 min → every 5 min. Constant check window from 3 min → 2 min (30s × 4 cycles). +**Rationale:** Faster detection, shorter failover. +**Status:** ✅ Fixed & verified 2026-07-08 +**Verified by:** Cron changed to */5. Watchdog timing updated on app1-bu. + +### DR-011: AWS CLI missing on app1-bu standby +**Problem:** /opt/awscli-venv was missing on app1-bu. Failover could not pull fresh state from S3. +**Root cause:** python3-venv package not installed on standby server. +**Fix:** Installed python3-venv, created venv, installed awscli. Created hermes-standby-sync.sh script with */10 cron. +**Status:** ✅ Fixed & verified 2026-07-08 +**Verified by:** First sync ran end-to-end. Config.yaml timestamp went from Jul 5 → Jul 8 17:10. diff --git a/itpp-recovery-manual.md b/itpp-recovery-manual.md new file mode 100644 index 0000000..bb9af0f --- /dev/null +++ b/itpp-recovery-manual.md @@ -0,0 +1,1347 @@ +# 🏢 IT Pro Partner — Full Infrastructure Recovery Manual + +**Author:** Sho'Nuff | **Audience:** Germaine +**Last updated:** July 9, 2026 +**Purpose:** Recover ANY server, service, Docker container, or piece of ITPP infrastructure without Sho'Nuff. +**Supersedes:** `shark-game-recovery-manual.md` — this manual covers everything, including the shark game. + +--- + +## Table of Contents + +1. [Architecture Overview](#1-architecture-overview) +2. [Core Server (netcup — 152.53.192.33)](#2-core-server-netcup--1525319233) +3. [Hermes Agent (AI Assistant)](#3-hermes-agent-ai-assistant) +4. [app1-bu Standby (Hetzner — 5.161.114.8)](#4-app1-bu-standby-hetzner--51611148) +5. [Docker Services on Core](#5-docker-services-on-core) + - [5.1 Twenty CRM (crm.debtrecoveryexperts.com)](#51-twenty-crm) + - [5.2 Vaultwarden Password Manager](#52-vaultwarden-password-manager) + - [5.3 DocuSeal Document Signing](#53-docuseal-document-signing) + - [5.4 SearXNG Private Search](#54-searxng-private-search) +6. [AI Services (ai.itpropartner.com — 178.156.167.181)](#6-ai-services-aiitpropartnercom) +7. [All Other Hetzner Servers](#7-all-other-hetzner-servers) + - [7.1 hudu.itpropartner.com (Documentation)](#71-hudu) + - [7.2 unifi (UniFi Controller)](#72-unifi-unifi-controller) + - [7.3 unms.forefrontwireless.com (WISP/UISP)](#73-unmsforefrontwirelesscom) + - [7.4 wphost02 (WordPress/Apex)](#74-wphost02-wordpress) + - [7.5 app1.itpropartner.com (n8n Automation)](#75-app1itpropartnercom-n8n) + - [7.6 docker host (NPM/RustDesk/Uptime-Kuma)](#76-docker-host-npmrustdeskuptime-kuma) + - [7.7 fleettracker360 (Traccar GPS)](#77-fleettracker360-traccar) +8. [S3 Backups (Wasabi)](#8-s3-backups-wasabi) +9. [Routers and Networking](#9-routers-and-networking) +10. [Shared Credentials](#10-shared-credentials) +11. [Quick Reference — Common Commands](#11-quick-reference--common-commands) +12. [Disaster Scenarios](#12-disaster-scenarios) +13. [Cron Jobs — Complete List](#13-cron-jobs--complete-list) + +--- + +## 1. Architecture Overview + +### All Servers at a Glance + +| # | Server | IP | Provider | Specs | OS | What It Runs | +|---|--------|----|----------|-------|----|-------------| +| **1** | **Core** (Primary) | 152.53.192.33 | netcup RS 2000 | 8C/15G/512G | Debian 13 | Hermes, Caddy, Docker (Twenty, Vaultwarden, Docuseal, SearXNG), Shark Game, Ollama, VPN | +| **2** | **app1-bu** (Standby) | 5.161.114.8 | Hetzner CPX11 | 2C/2G/40G | Ubuntu 26.04 | Hermes standby, StrongSwan, L2TP | +| **3** | **ai.itpropartner.com** | 178.156.167.181 | Hetzner CPX41 | 8C/16G/240G | Ubuntu 24.04 | LiteLLM, Open WebUI, Ollama, Qdrant | +| **4** | **hudu.itpropartner.com** | 178.156.130.130 | Hetzner CPX21 | 3C/4G/80G | Ubuntu 24.04 | Hudu Docs, SWAG, MariaDB | +| **5** | **unifi** | 178.156.131.57 | Hetzner CPX21 | 3C/4G/80G | Ubuntu 24.04 | UniFi Controller, MongoDB, MariaDB | +| **6** | **unms.forefrontwireless.com** | 5.161.225.131 | Hetzner CPX21 | 3C/4G/80G | Ubuntu 24.04 | UNMS/UISP, UCRM, PostgreSQL, RabbitMQ | +| **7** | **wphost02** | 5.161.62.38 | Hetzner CPX21 | 3C/4G/80G | Ubuntu 24.04 | RunCloud, WordPress (Apex), MariaDB | +| **8** | **app1.itpropartner.com** | 87.99.144.163 | Hetzner CPX11 | 2C/2G/40G | Ubuntu 24.04 | n8n, PostgreSQL, Nginx | +| **9** | **docker** | 178.156.168.35 | Hetzner CPX11 | 2C/2G/40G | Ubuntu 24.04 | NPM, RustDesk, Uptime-Kuma, StrongSwan | +| **10** | **fleettracker360** | 178.156.149.32 | Hetzner CPX11 | 2C/2G/40G | Ubuntu 24.04 | Traccar, RunCloud, MariaDB | + +### Domain Map + +| Domain | Server | Service | +|--------|--------|---------| +| crm.debtrecoveryexperts.com | Core (152.53.192.33) | Twenty CRM (port 3001) | +| sign.itpropartner.com | Core (152.53.192.33) | DocuSeal (port 3000) | +| app.itpropartner.com | Core (152.53.192.33) | Hermes Assistant PWA (port 8082) | +| core.itpropartner.com | Core (152.53.192.33) | API, static files, health checks | +| portal.debtrecoveryexperts.com | Core (152.53.192.33) | DRE Portal (static) | +| shark.iamgmb.com | Core (152.53.192.33) | Shark Game (port 8083) | +| shark.itpropartner.com | Core (152.53.192.33) | Shark Game (static) | +| ops.itpropartner.com | Core (152.53.192.33) | Ops Portal (static) | +| admin-ai.itpropartner.com | ai (178.156.167.181) | LiteLLM proxy | +| hudu.itpropartner.com | hudu (178.156.130.130) | Hudu documentation | +| unms.forefrontwireless.com | unms (5.161.225.131) | WISP management | +| app1.itpropartner.com | n8n (87.99.144.163) | n8n workflows | + +--- + +## 2. Core Server (netcup — 152.53.192.33) + +**This is the most important server.** It runs Hermes (your AI assistant), the web proxy (Caddy), Docker services, the shark game, and VPN services. + +| Detail | Value | +|--------|-------| +| **IP** | 152.53.192.33 | +| **Hostname** | core (internal) | +| **Provider** | netcup RS 2000 G12 | +| **SSH Key** | `itpp-infra` (`/root/.ssh/itpp-infra`) | +| **OS** | Debian 13 (trixie) | +| **Disk** | 503 GB (6% used — plenty of room) | + +### SSH Access + +```bash +ssh root@152.53.192.33 -i /root/.ssh/itpp-infra +``` + +### Services Running on Core + +| Service | Type | Port | Status Command | +|---------|------|------|----------------| +| Hermes Agent | systemd | 8081 | `systemctl status hermes` | +| Hermes Browser (Chromium) | systemd | 9222 (localhost) | `systemctl status hermes-browser` | +| Caddy Web Server | systemd | 80/443 | `systemctl status caddy` | +| Ollama (LLM) | systemd | 11434 (localhost) | `systemctl status ollama` | +| Shark Game | systemd | 8083 | `systemctl status shark-game` | +| MySQL Tunnel | systemd | 33060 (localhost) | `systemctl status mysql-tunnel` | +| StrongSwan (IPsec VPN) | systemd | 500/4500 UDP | `systemctl status strongswan-starter` | +| xl2tpd (L2TP VPN) | systemd | 1701 UDP | `systemctl status xl2tpd` | +| Tailscale | systemd | — | `systemctl status tailscaled` | +| Twenty CRM | Docker | 3001 | `docker ps \| grep twenty` | +| Vaultwarden | Docker | 8080 | `docker ps \| grep vaultwarden` | +| DocuSeal | Docker | 3000 (localhost) | `docker ps \| grep docuseal` | +| SearXNG | Docker | 8888 (localhost) | `docker ps \| grep searxng` | + +### Recovery from Full System Failure + +If the server is completely dead (won't boot, drive failed, etc.): + +1. **Order a new netcup RS 2000** from the Server Control Panel: + - https://www.servercontrolpanel.de/SCP/ (Customer #389212) + - Pick the same plan (RS 2000 G12) — Debian 13 + +2. **Set up the basics** (after new server is provisioned): + ```bash + # Add SSH key + mkdir -p /root/.ssh && chmod 700 /root/.ssh + # Copy your itpp-infra key into /root/.ssh/itpp-infra + # (from your own machine or from the standby) + + # Install essentials + apt update && apt install -y curl wget git docker.io docker-compose-v2 + ``` + +3. **Restore Hermes** (see Section 3) + +4. **Restore Docker services** — each one is documented below in Section 5. + +5. **Restart Caddy:** + ```bash + source /opt/awscli-venv/bin/activate + aws s3 cp s3://itpropartner-system-configs/caddy/Caddyfile /etc/caddy/Caddyfile \ + --endpoint-url https://s3.us-east-1.wasabisys.com + systemctl restart caddy + ``` + +### Checking System Health + +```bash +# Disk space +df -h + +# Memory +free -h + +# Running Docker containers +docker ps + +# All systemd services +systemctl list-unit-files --type=service | grep enabled + +# Recent system logs +journalctl -xe -n 50 +``` + +--- + +## 3. Hermes Agent (AI Assistant) + +Hermes is the AI assistant running on the Core server. It powers Telegram chat, web search, email, and infrastructure management. + +### How to Check If Hermes Is Running + +```bash +systemctl status hermes +``` + +Look for: **Active: active (running)** + +### How to Restart Hermes + +```bash +systemctl restart hermes +``` + +Then verify: `systemctl status hermes` + +### How to View Hermes Logs + +```bash +# Last 100 lines +journalctl -u hermes -n 100 + +# Live tail +journalctl -u hermes -n 50 -f + +# Today's logs only +journalctl -u hermes --since today +``` + +### Where Hermes Lives + +| Item | Path | +|------|------| +| Config | `/root/.hermes/config.yaml` | +| Environment/Secrets | `/root/.hermes/.env` | +| State DB (conversations) | `/root/.hermes/state.db` | +| Skills | `/root/.hermes/skills/` | +| Scripts | `/root/.hermes/scripts/` | +| Cron jobs | `/root/.hermes/cron/jobs.json` | +| Systemd service | `/etc/systemd/system/hermes.service` | + +### Hermes Browser (Headless Chromium) + +The browser service runs alongside Hermes for web automation: + +```bash +systemctl status hermes-browser +systemctl restart hermes-browser +``` + +It runs Chrome on port 9222 (localhost only). + +### Restoring Hermes from S3 Backup + +**Step 1: Stop Hermes** (if running) + +```bash +systemctl stop hermes +``` + +**Step 2: Restore the state database** + +```bash +source /opt/awscli-venv/bin/activate +aws s3 cp s3://hermes-vps-backups/live/state.db /root/.hermes/data/state.db \ + --endpoint-url https://s3.us-east-1.wasabisys.com +``` + +**Step 3: Restore the full config** + +```bash +aws s3 sync s3://hermes-vps-backups/live/ /root/.hermes/ \ + --endpoint-url https://s3.us-east-1.wasabisys.com \ + --exclude "audio_cache/*" \ + --exclude "image_cache/*" \ + --exclude "cache/*" \ + --exclude "sandboxes/*" +``` + +**Step 4: Start Hermes** + +```bash +systemctl start hermes +``` + +**If Hermes won't start** (broken binary or config): + +1. Reinstall from the official script: + ```bash + curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash + ``` + +2. Then restore config from S3 (Step 3 above). + +3. Restart: + ```bash + systemctl restart hermes + ``` + +--- + +## 4. app1-bu Standby (Hetzner — 5.161.114.8) + +**What it is:** A warm standby server on Hetzner. If the Core server (netcup) dies, this server takes over as Hermes. + +### Server Details + +| Detail | Value | +|--------|-------| +| **IP** | 5.161.114.8 | +| **Hostname** | app1-bu.itpropartner.com | +| **Plan** | CPX11 (2C/2G/40G) | +| **OS** | Ubuntu 26.04 | +| **SSH Key** | `itpp-infra` | + +### SSH Access + +```bash +ssh root@5.161.114.8 -i /root/.ssh/itpp-infra +``` + +### What Runs on Standby (Normally Dormant) + +| Service | Status | Purpose | +|---------|--------|---------| +| hermes-gateway | Disabled (dormant) | Hermes standby | +| hermes-standby | Disabled | Standby restore service | +| strongswan-starter | Active | IPsec VPN (always on) | +| xl2tpd | Active | L2TP tunnel (always on) | +| home-router-vpn | Active | Router VPN tunnel | +| home-router-keepalive | Active | Router watchdog | + +### How Failover Works (Automatic) + +The watchdog script runs every 5 minutes via cron (`*/5 * * * * /root/.hermes/scripts/hermes-standby-watchdog.sh`): + +1. **Checks if Core server (152.53.192.33) is reachable** via ping +2. **If Core is alive:** exits silently (nothing to do) +3. **If Core is unreachable for ~3.5 minutes:** confirms via 4 consecutive ping failures +4. **Takes over:** syncs latest Hermes state from S3, starts Hermes gateway +5. **Sends alerts** via Telegram and email to g@germainebrown.com + +### How to Manually Promote Standby (Failover) + +If you need to manually switch to standby: + +```bash +ssh root@5.161.114.8 -i /root/.ssh/itpp-infra 'systemctl start hermes-gateway' +``` + +### How to Demote (Switch Back to Core) + +After Core is restored: + +```bash +ssh root@5.161.114.8 -i /root/.ssh/itpp-infra 'systemctl stop hermes-gateway' +``` + +Then make sure Hermes is running on Core: +```bash +systemctl restart hermes +``` + +### Standby Sync + +The standby syncs state from S3 every 10 minutes (`*/10 * * * * /root/.hermes/scripts/hermes-standby-sync.sh`) so it's always ready to take over. + +--- + +## 5. Docker Services on Core + +All Docker services live under `/root/docker//`. Each has its own `docker-compose.yml` and `.env` file. + +### General Docker Commands (Work for Any Service) + +```bash +# List running containers +docker ps + +# Restart a specific service +cd /root/docker/ && docker compose --env-file .env down && docker compose --env-file .env up -d + +# View logs for a service +docker logs -n 50 + +# Check health +docker ps --filter name= --format '{{.Names}} {{.Status}}' +``` + +### 5.1 Twenty CRM + +**Purpose:** Customer Relationship Management for Debt Recovery Experts +**URL:** https://crm.debtrecoveryexperts.com +**Port:** 3001 (maps to container port 3000) + +#### Containers (4 total) + +| Container | Image | Purpose | +|-----------|-------|---------| +| twenty-server-1 | twentycrm/twenty:latest | NestJS API + frontend | +| twenty-worker-1 | twentycrm/twenty:latest | Background job processor | +| twenty-db-1 | postgres:16 | PostgreSQL database | +| twenty-redis-1 | redis | Queue + cache | + +#### Compose Location + +`/root/docker/twenty/docker-compose.yml` +`/root/docker/twenty/.env` + +#### Data Locations + +- PostgreSQL data: Docker volume (`twenty_pg_data` or similar) +- File uploads: Docker volume (`twenty_server-local-data` or similar) + +#### Backup + +```bash +# Export PostgreSQL database +docker exec twenty-db-1 pg_dump -U postgres default > /root/twenty-backup-$(date +%Y%m%d).sql +``` + +#### Restore from Scratch + +```bash +cd /root/docker/twenty +docker compose --env-file .env down -v # WARNING: -v removes volumes +docker compose --env-file .env up -d +# Then restore database: +cat /path/to/backup.sql | docker exec -i twenty-db-1 psql -U postgres default +``` + +#### Verify It's Working + +```bash +curl -sI https://crm.debtrecoveryexperts.com/ | head -3 +# Should return HTTP/2 200 +``` + +### 5.2 Vaultwarden + +**Purpose:** Password manager (compatible with Bitwarden apps) +**URL:** https://vaultwarden.tailc2f3b0.ts.net (Tailscale only -- not public internet) +**Port:** 8080 -> container port 80 + +#### Compose Location + +`/root/docker/vaultwarden/docker-compose.yml` +`/root/docker/vaultwarden/.env` + +#### Data Location + +`/root/docker/vaultwarden/data/` (bind mount) + +#### Key Config + +| Variable | Value | +|----------|-------| +| DOMAIN | https://vaultwarden.tailc2f3b0.ts.net | +| SIGNUPS_ALLOWED | false | +| INVITATIONS_ALLOWED | true | +| ADMIN_TOKEN | Stored in `.env` | + +#### Restart + +```bash +cd /root/docker/vaultwarden && docker compose --env-file .env down && docker compose --env-file .env up -d +``` + +#### Restore from S3 + +```bash +cd /root/docker/vaultwarden +source /opt/awscli-venv/bin/activate +aws s3 sync s3://hermes-vps-backups/live/docker/vaultwarden/ ./ \ + --endpoint-url https://s3.us-east-1.wasabisys.com +docker compose --env-file .env up -d +``` + +#### Verify + +```bash +curl -sI http://127.0.0.1:8080 | head -3 +``` + +### 5.3 DocuSeal + +**Purpose:** Document signing (contracts, forms) +**URL:** https://sign.itpropartner.com +**Port:** 127.0.0.1:3000 (localhost only -- proxied by Caddy) + +#### Compose Location + +`/root/docker/docuseal/docker-compose.yml` + +#### Data Location + +`/root/docker/docuseal/data/` (bind mount -- SQLite database) + +#### Restart + +```bash +cd /root/docker/docuseal && docker compose down && docker compose up -d +``` + +#### Restore from S3 + +```bash +cd /root/docker/docuseal +source /opt/awscli-venv/bin/activate +aws s3 sync s3://hermes-vps-backups/live/docker/docuseal/ ./ \ + --endpoint-url https://s3.us-east-1.wasabisys.com +docker compose up -d +``` + +#### Verify + +```bash +curl -sI https://sign.itpropartner.com | head -3 +``` + +### 5.4 SearXNG + +**Purpose:** Private, self-hosted search engine (used by Hermes for web searches) +**URL:** http://127.0.0.1:8888 (localhost only) +**Port:** 127.0.0.1:8888 + +#### Compose Location + +`/root/docker/searxng/docker-compose.yml` + +#### Data Locations + +- `/root/docker/searxng/searxng-data/` -- settings and config +- `/root/docker/searxng/searxng-themes/` -- theme overrides + +#### Restart + +```bash +cd /root/docker/searxng && docker compose down && docker compose up -d +``` + +#### Quick Test + +```bash +curl -s "http://127.0.0.1:8888/search?q=test&format=json" | head -5 +``` + +--- + +## 6. AI Services (ai.itpropartner.com) + +**IMPORTANT:** This server hosts `admin-ai.itpropartner.com` (LiteLLM) which powers Sho'Nuff's deepseek-chat model. If this goes down, Hermes loses its brain. + +**DISK WARNING:** 92% used (199G of 226G) -- needs attention soon. + +### Server Details + +| Detail | Value | +|--------|-------| +| **IP** | 178.156.167.181 | +| **Hostname** | ai | +| **Provider** | Hetzner CPX41 | +| **SSH** | `ssh root@178.156.167.181 -i /root/.ssh/itpp-infra` | +| **OS** | Ubuntu 24.04 | +| **Status** | Still on Hetzner (will migrate to netcup Monitor server) | + +### Docker Containers (6 total) + +| Container | Image | Port | Purpose | +|-----------|-------|------|---------| +| litellm | ghcr.io/berriai/litellm:v1.84.0 | 4000 | LLM proxy (currently unhealthy but working) | +| litellm_postgres | postgres:16 | 5432 | LiteLLM database | +| openwebui | ghcr.io/open-webui/open-webui:0.9.6 | 8080 | Chat interface at admin-ai.itpropartner.com | +| ollama | ollama/ollama:latest | 11434 | Local LLM inference | +| qdrant | qdrant/qdrant:latest | 6333-6334 | Vector storage for RAG | +| caddy | caddy:latest | 80/443 | Reverse proxy | + +### Compose Location + +`/opt/ai/docker-compose.yml` + +### Restart All AI Services + +```bash +ssh root@178.156.167.181 -i /root/.ssh/itpp-infra +cd /opt/ai && docker compose down && docker compose up -d +``` + +### Check Health + +```bash +# LiteLLM health +curl -s https://admin-ai.itpropartner.com/health/liveliness + +# Check all containers +docker ps +``` + +### Restore Open WebUI Database + +The WebUI SQLite DB is at `/app/backend/data/webui.db` inside the container. To restore: + +```bash +# Find the Open WebUI volume +docker volume inspect ai_open-webui || docker volume ls | grep webui + +# To restore from a backup file: +docker run --rm -v ai_open-webui:/data -v /path/to/backup:/backup alpine sh -c \ + "cp /backup/webui.db /data/backend/data/webui.db" +``` + +### Restore LiteLLM PostgreSQL + +```bash +# Dump current DB +docker exec litellm_postgres pg_dump -U postgres litellm > /root/litellm-backup.sql + +# Restore from backup +cat /root/litellm-backup.sql | docker exec -i litellm_postgres psql -U postgres litellm +``` + +### Key User Accounts + +- **Open WebUI:** admin@itpropartner.com (password in /root/.hermes/.env or ask Sho'Nuff) +- Root user: root via SSH key + +--- + +## 7. All Other Hetzner Servers + +### 7.1 Hudu + +**Purpose:** IT documentation and knowledge base +**URL:** https://hudu.itpropartner.com +**Server:** 178.156.130.130 (CPX21) + +```bash +ssh root@178.156.130.130 -i /root/.ssh/itpp-infra +``` + +#### Docker Containers (5) + +| Container | Purpose | Port | +|-----------|---------|------| +| hudu2-app-1 | Hudu application | 3000 | +| hudu2-worker-1 | Background worker | -- | +| hudu2-db-1 | PostgreSQL | 5432 | +| hudu2-redis-1 | Cache | 6379 | +| letsencrypt (swag) | Let's Encrypt + Nginx | 80/443 | + +#### Restart Everything + +```bash +cd /opt/hudu2 && docker compose down && docker compose up -d +``` + +#### Native Services (also running) + +- **MariaDB** 11.4.12 (native, not Docker) -- port 3306 +- **Postfix** mail server -- port 25 +- **RunCloud** agent + +--- + +### 7.2 UniFi + +**Purpose:** Ubiquiti network controller -- manages all UniFi devices +**URL:** https://unifi.itpropartner.com:8443 (or direct IP) +**Server:** 178.156.131.57 (CPX21) + +```bash +ssh root@178.156.131.57 -i /root/.ssh/itpp-infra +``` + +#### What Runs on It + +| Service | Port | Purpose | +|---------|------|---------| +| unifi.service | 8080, 8443, 8843, 8880, etc. | UniFi Network Controller (Java) | +| mariadb.service | 3306 | MariaDB database | +| MongoDB | 27117 (localhost) | UniFi internal database | + +#### Restart UniFi Controller + +```bash +systemctl restart unifi +``` + +#### Check UniFi Status + +```bash +systemctl status unifi +``` + +#### Data Locations + +- UniFi data: `/usr/lib/unifi/data/` +- UniFi backups (auto-generated): `/usr/lib/unifi/data/backup/` + +--- + +### 7.3 UNMS/UISP + +**Purpose:** WISP management -- manages Ubiquiti wireless devices across the network +**URL:** https://unms.forefrontwireless.com +**Server:** 5.161.225.131 (CPX21) + +```bash +ssh root@5.161.225.131 -i /root/.ssh/itpp-infra +``` + +#### Docker Containers (10) + +UNMS is a complex stack. The key containers: + +| Container | Image | Purpose | +|-----------|-------|---------| +| unms-nginx | ubnt/unms-nginx:3.0.140 | Frontend proxy | +| unms-api | ubnt/unms:3.0.140 | API server | +| unms-device-ws-1/2 | ubnt/unms:3.0.140 | Device WebSocket connections | +| unms-postgres | ubnt/unms-postgres:3.0.140 | PostgreSQL database | +| unms-siridb | ubnt/unms-siridb:3.0.140 | Time-series database | +| unms-rabbitmq | rabbitmq:3.7.28-alpine | Message queue | +| unms-fluentd | ubnt/unms-fluentd:3.0.140 | Log collector | +| unms-netflow | ubnt/unms-netflow:3.0.140 | Netflow collector | +| ucrm | ubnt/unms-crm:4.5.31 | Customer management (CRM) | + +#### Restart UNMS + +```bash +cd /opt/unms && docker compose down && docker compose up -d +``` + +#### Native Services + +- **MariaDB** 11.4.12 (native, port 3306) -- separate from Docker PostgreSQL +- **Postfix** mail server + +#### Backup + +```bash +/root/.hermes/scripts/backup-uisp.sh +``` + +--- + +### 7.4 wphost02 + +**Purpose:** WordPress hosting (Apex Track Experience) +**Server:** 5.161.62.38 (CPX21) + +```bash +ssh root@5.161.62.38 -i /root/.ssh/itpp-infra +``` + +#### What Runs on It + +| Service | Purpose | +|---------|---------| +| nginx-rc (RunCloud) | Nginx reverse proxy on 80/443 | +| apache2-rc (RunCloud) | Apache backend on port 81 | +| php81rc-fpm | PHP 8.1 | +| mariadb.service | MariaDB on 3306 | +| postfix | Mail server | + +#### Restart Web Server + +```bash +systemctl restart nginx-rc +systemctl restart apache2-rc +systemctl restart php81rc-fpm +``` + +#### Restart All + +```bash +runcloud restart +``` + +#### Data Locations + +- WordPress files: managed by RunCloud (see `/home/runcloud/` or RunCloud dashboard) +- MySQL databases: RunCloud-managed + +#### MySQL Tunnel (from Core) + +The Core server connects to this MySQL via AutoSSH tunnel on port 33060 (localhost): +```bash +systemctl status mysql-tunnel +``` + +--- + +### 7.5 app1.itpropartner.com (n8n) + +**Purpose:** Workflow automation (n8n) +**URL:** https://app1.itpropartner.com +**Server:** 87.99.144.163 (CPX11) + +```bash +ssh root@87.99.144.163 -i /root/.ssh/itpp-infra +``` + +#### Docker Containers (2) + +| Container | Purpose | Port | +|-----------|---------|------| +| n8n | Workflow automation | 5678 (localhost) | +| n8n-postgres | PostgreSQL backend | 5432 | + +#### Compose Location + +`/opt/n8n/docker-compose.yml` + +#### Restart n8n + +```bash +cd /opt/n8n && docker compose down && docker compose up -d +``` + +#### Native Services + +- **Nginx** -- reverse proxy on 80/443 -> 127.0.0.1:5678 +- **MariaDB** -- port 3306 + +#### Verify + +```bash +curl -sI https://app1.itpropartner.com | head -3 +``` + +--- + +### 7.6 Docker Host (NPM/RustDesk) + +**Purpose:** Nginx Proxy Manager, RustDesk remote desktop, Uptime-Kuma monitoring +**Server:** 178.156.168.35 (CPX11) + +```bash +ssh root@178.156.168.35 -i /root/.ssh/itpp-infra +``` + +**DISK WARNING:** 73% used (26G of 38G) -- getting full. + +#### Docker Containers (7+) + +| Container | Purpose | Port | +|-----------|---------|------| +| nginx-proxy-manager | Reverse proxy dashboard | 80, 81, 443 | +| uptime-kuma | Status monitoring | 3001 | +| hbbr | RustDesk relay server | 21117, 21119 | +| hbbs | RustDesk signal server | 21114, 21116, 21118 | +| strongswan | IPsec VPN | 500/4500 | +| runcloud_mariadb | RunCloud MariaDB | -- | +| runcloud_php* | RunCloud PHP-FPM | -- | + +#### Restart Nginx Proxy Manager + +```bash +docker restart nginx-proxy-manager +``` + +#### Restart Uptime-Kuma + +```bash +docker restart uptime-kuma +``` + +#### Compose Files + +Services are at `/opt//docker-compose.yml` (not in `/root/docker/`): +- `/opt/rustdesk/docker-compose.yml` +- `/opt/strongswan/docker-compose.yml` +- `/opt/litellm/docker-compose.yml` (legacy -- this is now on the AI server) + +--- + +### 7.7 fleettracker360 (Traccar) + +**Purpose:** GPS fleet tracking +**Server:** 178.156.149.32 (CPX11) + +```bash +ssh root@178.156.149.32 -i /root/.ssh/itpp-infra +``` + +#### What Runs on It + +| Service | Port | Purpose | +|---------|------|---------| +| traccar.service | 5001-5032, 8082 | GPS tracking (Java) | +| nginx-rc (RunCloud) | 80/443 | Web frontend | +| apache2-rc (RunCloud) | 81 | Apache backend | +| mariadb.service | 3306 | MariaDB database | + +#### Restart Traccar + +```bash +systemctl restart traccar +``` + +#### Check Traccar Status + +```bash +systemctl status traccar +journalctl -u traccar -n 50 +``` + +#### Data Locations + +- Traccar data: `/opt/traccar/data/` or `/opt/traccar/` +- Traccar config: `/opt/traccar/conf/traccar.xml` + +--- + +## 8. S3 Backups (Wasabi) + +All infrastructure backups go to Wasabi S3 (us-east-1). + +### Credentials + +- **Config file:** `/root/.aws/credentials` +- **CLI activation:** `source /opt/awscli-venv/bin/activate` +- **Endpoint:** `https://s3.us-east-1.wasabisys.com` + +### Buckets + +| Bucket | Contents | Backup Frequency | +|--------|----------|-----------------| +| `hermes-vps-backups` | Hermes state, config, Docker data, skills, scripts | Every 15 min (live sync), daily (full backup) | +| `mikrotik-ccr-backups` | Router (CCR) config exports | Daily via wisp-backup | +| `itpropartner-backups` | Home router config, legacy data | Daily | + +### Bucket Structure + +``` +hermes-vps-backups/ ++-- live/ <- 15-min sync (most recent Hermes state) +| +-- state.db <- Conversation history (1.1 GB) +| +-- config.yaml <- Hermes config +| +-- .env <- All secrets/API keys +| +-- docker/ <- Docker service configs & data +| | +-- vaultwarden/ +| | +-- docuseal/ +| | +-- ... +| +-- skills/ <- All skills +| +-- scripts/ <- All scripts +| +-- cron/ <- Cron job definitions +| +-- ... ++-- live-sync/ <- Alternative sync target ++-- hermes-full-backup/ <- Daily tar.gz archives +| +-- hermes-full-backup-2026-07-05.tar.gz +| +-- hermes-full-backup-2026-07-04.tar.gz ++-- standby/ <- DR scripts and docs ++-- docker/ <- Docker volume backups (when configured) + +mikrotik-ccr-backups/ ++-- wisp-backups/ <- WISP router configs + +itpropartner-backups/ ++-- home-router/ <- Home gateway configs ++-- shonuff/ <- Sho'Nuff personal data +``` + +### Checking Backups + +```bash +source /opt/awscli-venv/bin/activate + +# List Hermes backup folders +aws s3 ls s3://hermes-vps-backups/ --endpoint-url https://s3.us-east-1.wasabisys.com + +# List recent Hermes state files +aws s3 ls s3://hermes-vps-backups/live/ --endpoint-url https://s3.us-east-1.wasabisys.com + +# List router backups +aws s3 ls s3://mikrotik-ccr-backups/wisp-backups/ --endpoint-url https://s3.us-east-1.wasabisys.com +``` + +### Sync Commands + +```bash +# Full Hermes restore +aws s3 sync s3://hermes-vps-backups/live/ /root/.hermes/ \ + --endpoint-url https://s3.us-east-1.wasabisys.com + +# Restore just the state DB +aws s3 cp s3://hermes-vps-backups/live/state.db /root/.hermes/state.db \ + --endpoint-url https://s3.us-east-1.wasabisys.com +``` + +### Backup Scripts + +These scripts run automatically on Core: + +| Script | What It Backs Up | When | +|--------|------------------|------| +| `hermes-live-sync.sh` | Full .hermes directory -> `live/` | Every 15 min | +| `hermes-backup.sh` | Daily tar.gz archive -> `hermes-full-backup/` | Daily | +| `hermes-docker-sync.sh` | Docker volumes | Config-based | +| `hermes-system-config-sync.sh` | Caddyfile and system configs | Config-based | +| `home-router-backup.sh` | Home gateway CCR config | Daily | +| `run-wisp-backup.sh` | WISP router configs | Daily | + +--- + +## 9. Routers and Networking + +### Home Gateway (MikroTik CCR) + +| Detail | Value | +|--------|-------| +| **Model** | MikroTik CCR (or similar) | +| **IP** | 10.77.0.2 (via WireGuard tunnel) | +| **SSH Key** | `/root/.ssh/wisp_rsa` | +| **SSH User** | `admin@10.77.0.2` | +| **Connection** | WireGuard (preferred) or L2TP/IPsec | + +### Backup the Router + +**Manual backup:** +```bash +cd /root/.hermes/scripts/wisp-backup && python3 wisp-backup.py +``` + +**Wrapper script (for cron):** +```bash +bash /root/.hermes/scripts/run-wisp-backup.sh +``` + +**Output:** Backup files go to S3 at `s3://mikrotik-ccr-backups/wisp-backups/` + +### Restore Router from Backup + +1. SSH into the router: + ```bash + ssh -i /root/.ssh/wisp_rsa admin@10.77.0.2 + ``` + +2. Download the latest backup from S3: + ```bash + source /opt/awscli-venv/bin/activate + aws s3 ls s3://mikrotik-ccr-backups/wisp-backups/ --endpoint-url https://s3.us-east-1.wasabisys.com + aws s3 cp s3://mikrotik-ccr-backups/wisp-backups/ /tmp/ \ + --endpoint-url https://s3.us-east-1.wasabisys.com + ``` + +3. Upload to router via SCP: + ```bash + scp -i /root/.ssh/wisp_rsa /tmp/ admin@10.77.0.2:/ + ``` + +4. On the router, import the config: + ``` + /import file-name= + ``` + +### WireGuard vs L2TP/IPsec + +- **WireGuard** is the preferred connection method -- faster, more reliable, simpler +- **L2TP/IPsec** (StrongSwan + xl2tpd) is the fallback -- runs on both Core and app1-bu +- If WireGuard goes down, the watchdog scripts automatically try the VPN tunnel + +### Home Router Watchdog + +- Runs on Core via: `/usr/local/bin/home-router-watchdog.sh` +- Runs on app1-bu via: `/root/.hermes/scripts/wisp-backup/home-router-keepalive.sh` +- Checks connection to 10.77.0.2 periodically +- Restarts the VPN tunnel if the router is unreachable + +--- + +## 10. Shared Credentials + +**IMPORTANT:** These credentials are essential for recovery. Guard them carefully. + +| What | Where | Purpose | +|------|-------|---------| +| **All API keys & secrets** | `/root/.hermes/.env` | Telegram tokens, API keys, database passwords | +| **Email password** | `/root/.config/himalaya/shonuff.pass` | g@germainebrown.com IMAP/SMTP | +| **Router SSH key** | `/root/.ssh/wisp_rsa` | SSH to MikroTik CCR | +| **Server SSH key** | `/root/.ssh/itpp-infra` | SSH to ALL servers | +| **S3 access** | `/root/.aws/credentials` | Wasabi S3 access | +| **Himalaya config** | `/root/.config/himalaya/config.toml` | Email client config | +| **netcup admin** | https://www.servercontrolpanel.de/SCP/ (Customer #389212) | Order/manage netcup servers | + +### What to Do If Credentials Are Lost + +1. **S3 credentials** -- stored in multiple places (S3 itself can be accessed via the web console at wasabi.com) +2. **SSH keys** -- backed up in `/root/.hermes/` which syncs to S3 +3. **Email password** -- stored in `/root/.config/himalaya/shonuff.pass` and in the `.env` file +4. **If everything is lost:** contact Sho'Nuff for password resets + +--- + +## 11. Quick Reference -- Common Commands + +### Service Management + +| What | Command | +|------|---------| +| Check Hermes | `systemctl status hermes` | +| Restart Hermes | `systemctl restart hermes` | +| Check Caddy | `systemctl status caddy` | +| Restart Caddy | `systemctl reload caddy` | +| Check shark game | `systemctl status shark-game` | +| Restart shark game | `systemctl restart shark-game` | +| Check all services | `systemctl list-unit-files --type=service \| grep enabled` | +| Check Docker | `docker ps` | +| Restart all Docker | See each service in Section 5 | + +### Logs + +| What | Command | +|------|---------| +| Hermes logs | `journalctl -u hermes -n 100` | +| Hermes live log | `journalctl -u hermes -n 50 -f` | +| Caddy logs | `journalctl -u caddy -n 50` | +| Shark game logs | `journalctl -u shark-game -n 100` | +| Any service logs | `journalctl -u -n 100` | +| Filter for errors | `journalctl -u -n 200 \| grep -i "error\\|traceback\\|exception"` | + +### Docker + +| What | Command | +|------|---------| +| List running containers | `docker ps` | +| List all containers | `docker ps -a` | +| View Docker logs | `docker logs -n 50` | +| Follow Docker logs | `docker logs -n 50 -f` | +| Restart a Docker service | `cd /root/docker/ && docker compose --env-file .env down && docker compose --env-file .env up -d` | + +### System + +| What | Command | +|------|---------| +| Check disk space | `df -h` | +| Check memory | `free -h` | +| Check network ports | `ss -tlnp` | +| Check cron jobs | `crontab -l` | + +### S3 / Backups + +| What | Command | +|------|---------| +| Activate AWS CLI | `source /opt/awscli-venv/bin/activate` | +| List backup buckets | `aws s3 ls s3://hermes-vps-backups/ --endpoint-url https://s3.us-east-1.wasabisys.com` | +| Restore Hermes state | `aws s3 cp s3://hermes-vps-backups/live/state.db /root/.hermes/state.db --endpoint-url https://s3.us-east-1.wasabisys.com` | +| Backup router | `bash /root/.hermes/scripts/run-wisp-backup.sh` | + +### Other + +| What | Command | +|------|---------| +| SSH to any server | `ssh root@ -i /root/.ssh/itpp-infra` | +| SSH to router | `ssh -i /root/.ssh/wisp_rsa admin@10.77.0.2` | +| Check LiteLLM health | `curl -s https://admin-ai.itpropartner.com/health/liveliness` | +| Check site response | `curl -I https://` | +| netcup SCP (admin panel) | https://www.servercontrolpanel.de/SCP/ | + +--- + +## 12. Disaster Scenarios + +### Scenario A: "Sho'Nuff Is Gone" -- Full Recovery + +**If Sho'Nuff disappears and you need to recover everything without them:** + +1. **SSH into Core:** + ```bash + ssh root@152.53.192.33 -i /root/.ssh/itpp-infra + ``` + +2. **Check what's running:** + ```bash + systemctl list-unit-files --type=service | grep enabled + docker ps + ``` + +3. **If Core is down, failover to standby** (see Scenario B below) + +4. **Check S3 backups exist:** + ```bash + source /opt/awscli-venv/bin/activate + aws s3 ls s3://hermes-vps-backups/ --endpoint-url https://s3.us-east-1.wasabisys.com + ``` + +5. **Restore Hermes** (if needed): + ```bash + systemctl stop hermes + aws s3 sync s3://hermes-vps-backups/live/ /root/.hermes/ \ + --endpoint-url https://s3.us-east-1.wasabisys.com \ + --exclude "audio_cache/*" --exclude "image_cache/*" + systemctl start hermes + ``` + +6. **Restart each Docker service** (see Section 5) + +7. **Verify Caddy is proxying:** + ```bash + systemctl restart caddy + curl -I https://core.itpropartner.com + ``` + +8. **Back up the router if needed:** + ```bash + bash /root/.hermes/scripts/run-wisp-backup.sh + ``` + +### Scenario B: "Core Server Is Dead" -- Failover to Standby + +**If 152.53.192.33 is offline and won't come back:** + +1. **Wait for automatic failover** -- the watchdog on app1-bu will detect it and promote itself within ~5 minutes. + +2. **If automatic failover doesn't trigger**, do it manually: + ```bash + ssh root@5.161.114.8 -i /root/.ssh/itpp-infra 'systemctl start hermes-gateway' + ``` + +3. **Verify Hermes is running on standby:** + ```bash + ssh root@5.161.114.8 -i /root/.ssh/itpp-infra 'systemctl status hermes-gateway' + ``` + +4. **Order a new netcup server:** + - Go to https://www.servercontrolpanel.de/SCP/ (Customer #389212) + - Order an RS 2000 G12 with Debian 13 + +5. **When new server arrives, set it up** (see Section 2 -- Recovery from Full System Failure) + +6. **Demote standby and restore Core as primary** (see Section 4) + +### Scenario C: "Database Corruption" + +**If a database (SQLite or PostgreSQL) has corrupted data:** + +1. **Identify which service is affected** from error messages or logs + +2. **For SQLite-based services** (DocuSeal, Hermes state.db): + ```bash + # Stop the service + systemctl stop hermes # or: docker compose down + + # Check integrity + sqlite3 /path/to/database.db "PRAGMA integrity_check;" + + # Restore from S3 + source /opt/awscli-venv/bin/activate + aws s3 cp s3://hermes-vps-backups/live/state.db /path/to/database.db \ + --endpoint-url https://s3.us-east-1.wasabisys.com + + # Restart + systemctl start hermes + ``` + +3. **For PostgreSQL-based services** (Twenty CRM, n8n, LiteLLM): + ```bash + # Find the database container + docker ps | grep postgres + + # Restore from backup + cat /path/to/backup.sql | docker exec -i psql -U postgres + ``` + +### Scenario D: "Accidental Deletion" -- Restore from S3 Versioning + +Wasabi supports S3-compatible versioning. If a file was accidentally deleted: + +```bash +source /opt/awscli-venv/bin/activate + +# List versions of a deleted file +aws s3api list-object-versions --bucket hermes-vps-backups --prefix live/state.db \ + --endpoint-url https://s3.us-east-1.wasabisys.com + +# Copy a specific version back +aws s3api get-object --bucket hermes-vps-backups --key live/state.db \ + --version-id /root/.hermes/state.db \ + --endpoint-url https://s3.us-east-1.wasabisys.com +``` + +Or just sync from a different backup prefix: +```bash +# Try the previous sync +aws s3 sync s3://hermes-vps-backups/live-sync/ /root/.hermes/ \ + --endpoint-url https://s3.us-east-1.wasabisys.com + +# Or try a daily backup +aws s3 cp s3://hermes-vps-backups/hermes-full-backup/hermes-full-backup-2026-07-05.tar.gz /tmp/ +tar xzf /tmp/hermes-full-backup-2026-07-05.tar.gz -C /root/.hermes/ +``` + +### Scenario E: "ai.itpropartner.com Is Down" -- AI Services Unavailable + +**If the AI server (178.156.167.181) is offline:** + +1. **Check if it's just a Docker issue:** + ```bash + ssh root@178.156.167.181 -i /root/.ssh/itpp-infra + docker ps + cd /opt/ai && docker compose up -d + ``` + +2. **Check disk space** (this server is at 92%): + ```bash + df -h + # If full, clean up old Docker images: + docker system prune -a + ``` + +3. **If the server is completely dead**, Hermes loses access to the deepseek-chat model. Sho'Nuff would need to: + - Provision a new server + - Restore LiteLLM PostgreSQL database + - Restore Open WebUI data + - Point DNS to the new server + +### Scenario F: "The Internet Is Down at Home" -- Router Issues + +**If the home internet (and therefore the MikroTik router) is unavailable:** + +1. **Check if the router is reachable via WireGuard:** + ```bash + ping -c 3 10.77.0.2 + ``` + +2. **If unreachable**, check the VPN tunnel: + ```bash + systemctl status strongswan-starter + systemctl status xl2tpd + ``` + +3. **If VPN is down but Core is up**, the rest of your infrastructure still works -- only home-related services are affected. + +4. **Restore router from last known good config** (see Section 9). + +--- + +## 13. Cron Jobs -- Complete List + +### On Core (152.53.192.33) + +``` +TZ=America/New_York + +* * * * * /root/.hermes/scripts/dre-mail-poller.py +0 * * * * /root/.hermes/scripts/boys-mail-monitor.py collect 2>&1 | logger -t boys-mail-collect +0 19 * * * /root/.hermes/scripts/boys-mail-monitor.py summary 2>&1 | logger -t boys-mail-summary +0 8 * * * /root/shark-game/scraper/run.sh +*/15 * * * * /root/.hermes/scripts/shark-draft-reminder.sh 2>&1 | logger -t shark-draft-reminder +``` + +**Hermes internal cron jobs** (managed by Hermes itself): +- Various ticker-based tasks defined in `/root/.hermes/cron/jobs.json` + +### On app1-bu (5.161.114.8) + +``` +*/5 * * * * /root/.hermes/scripts/hermes-standby-watchdog.sh +*/10 * * * * /root/.hermes/scripts/hermes-standby-sync.sh +``` + +### On wphost02 (5.161.62.38) + +``` +0 3 * * * /usr/local/bin/lynis-scan.sh +``` + +### To View Cron Jobs + +```bash +crontab -l # Current user's crontab +cat /root/.hermes/cron/jobs.json # Hermes-managed cron jobs +``` + +--- + +> **This manual supersedes** `shark-game-recovery-manual.md`. +> For shark-game-specific details (scoring, push notifications, common player issues), see the legacy manual at `/root/.hermes/references/shark-game-recovery-manual.md`. +> All infrastructure recovery procedures above take precedence. diff --git a/network-and-service-endpoints.md b/network-and-service-endpoints.md new file mode 100644 index 0000000..bbbee4d --- /dev/null +++ b/network-and-service-endpoints.md @@ -0,0 +1,70 @@ +# IT Pro Partner — Network & Service Endpoints + +## DNS & Hosting + +| Domain | Nameservers | Hosting | +|--------|------------|---------| +| itpropartner.com | SiteGround (ns1/ns2.siteground.net) | SiteGround | +| iamgmb.com | Cloudflare | MXroute (email) | +| germainebrown.com | Cloudflare | MXroute (email) | +| debtrecoveryexperts.com | Cloudflare | SiteGround/self-hosted | + +## Core (netcup RS 2000 G12) + +| Service | Port | URL / Notes | +|---------|------|-------------| +| Hermes gateway | — | Telegram bot | +| Hermes assistant (PWA) | 8082 | app.itpropartner.com | +| Shark game API | 8083 | shark.iamgmb.com | +| Chromium CDP | 9222 | local browser automation | +| SMTP relay | 2525 | mail.germainebrown.com (STARTTLS) | +| Caddy | 443 | Reverse proxy for all sites | +| Tailscale | — | vaultwarden.tailc2f3b0.ts.net | + +## Hetzner Servers + +| Name | Type | IP | Purpose | +|------|------|----|---------| +| wphost02 | CPX21 | 5.161.62.38 | Apex WP + RunCloud | +| app1-bu | CPX11 | 5.161.114.8 | Hermes warm standby | +| tony-vps | CPX21 | 87.99.159.142 | Tony's Hermes | +| unms | CPX21 | 5.161.225.131 | UISP/UNMS | +| unifi | CPX21 | 178.156.131.57 | UniFi controller | +| hudu | CPX21 | 178.156.130.130 | IT documentation | +| ai | CPX41 | 178.156.167.181 | admin-ai LLM proxy | +| fleet | CPX11 | 178.156.149.32 | Fleet tracker | +| docker | CPX11 | 178.156.168.35 | Docker host | +| app1 (old) | CPX11 | 87.99.144.163 | Legacy | + +## API Endpoints + +| API | Auth | Base URL | +|-----|------|----------| +| Hetzner Cloud | Bearer token | https://api.hetzner.cloud/v1 | +| Cloudflare | Bearer token | https://api.cloudflare.com/client/v4 | +| admin-ai | Bearer token | https://admin-ai.itpropartner.com/v1 | +| netcup SCP | Keycloak OIDC | https://www.servercontrolpanel.de/scp-core/api/v1 | +| Firecrawl | Bearer token | https://api.firecrawl.com/v1 | +| Wasabi S3 | AWS4-HMAC-SHA256 | s3.wasabisys.com | + +## S3 Backup Buckets + +| Bucket | Prefix | Purpose | +|--------|--------|---------| +| hermes-vps-backups | live/ | Active Hermes state (15 min sync) | +| hermes-vps-backups | standby/ | Standby recovery bundle | +| hermes-vps-backups | hermes-full-backup/ | Daily 5 AM full tarball | +| itpropartner-backups | — | Portal files & public data | +| mikrotik-ccr-backups | wisp-backups/ | Router configs | + +## External Services + +| Service | Login URL | Notes | +|---------|-----------|-------| +| netcup CCP | customercontrolpanel.de | Account management | +| netcup SCP | servercontrolpanel.de | Server management | +| Hetzner Cloud | console.hetzner.cloud | Hetzner project | +| Cloudflare | dash.cloudflare.com | DNS, Access, Registrar | +| MXroute | mxlogin.com | Email hosting | +| SiteGround | siteground.com | WP hosting | +| Wasabi | console.wasabisys.com | S3-compatible storage | diff --git a/server-dr-plans-v3.md b/server-dr-plans-v3.md new file mode 100644 index 0000000..f07ef35 --- /dev/null +++ b/server-dr-plans-v3.md @@ -0,0 +1,292 @@ +# IT Pro Partner — Disaster Recovery Plan v3 (Consolidated) + +**Author:** Sho'Nuff / Network Services Team, incorporating third-party DR review +**Date:** July 10, 2026 +**Status:** LIVE — replaces v1.0 +**Supersedes:** server-dr-plans.md v1.0, server-provisioning-standard-v1.md, hermes-dr-plan-v2.md + +--- + +## Table of Contents +1. Architecture & Shared Dependencies +2. Server Standards +3. Backup Strategy +4. Failover / Failback +5. Monitoring & Alerting +6. Restore Runbooks +7. Testing & Validation +8. Security +9. Immediate Action Items +10. Appendix: Third-Party Review Responses + +--- + +## 1. Architecture & Shared Dependencies + +``` +Internet -- Cloudflare -- Caddy -- Service (HTTP/HTTPS) + | + +-----------+-----------+ + | | | + SSH/WG Tailscale Management + (itpp- (Core <-> VPN (WireGuard + infra) app1-bu) to home router) + | | | + +-----------+-----------+ + | + +-----------+-----------+ + | | + S3 Primary S3 Secondary + (Wasabi us-east-1) (Wasabi us-west-2 + versioning ON or Backblaze B2) + object lock ON object lock ON +``` + +### Shared Dependencies + +| Dependency | Primary | Backup / Fallback | Credentials Location | +|---|---|---|---| +| SSH access | itpp-infra key (Hetzner vault) | Rescue mode / console | ~/.hermes/.env + Hudu | +| DNS | Cloudflare (API token in .env) | Manual via web UI | ~/.hermes/.env | +| S3 backup (primary) | Wasabi us-east-1 | Wasabi us-west-2 (to be created) | ~/.aws/credentials | +| S3 backup (secondary) | Wasabi us-west-2 | Backblaze B2 (evaluate) | TBD | +| SMTP relay | mail.germainebrown.com:2525 | Direct MXroute (port 465/587) | ~/.hermes/.env | +| Auth/SSO | Cloudflare Access | Local portal auth fallback | Cloudflare dashboard | +| Monitoring | Prometheus node_exporter | Uptime Kuma (docker box) | N/A (pull model) | +| VPN management | WireGuard (wg0, port 51821) | Tailscale direct tunnel | /etc/wireguard/ | + +--- + +## 2. Server Standards + +### 2.1 Tier Definitions + +| Tier | Spec | OS | Use | Cost | +|---|---|---|---|---| +| **Standard** | RS 4000 G12 (12C/32G/1TB) | Debian 13 | app1, app2, app3 | ~$44/mo | +| **Light** | RS 2000 G12 (8C/16G/512GB) | Debian 13 | Core | ~$24/mo | +| **Standby** | CPX21 (4C/8G/80GB) | Debian 13 | app1-bu | ~$14/mo | +| **Legacy** | Variable Hetzner | Debian 12 | Existing (migrate on rebuild) | ~$8-44/mo | + +### 2.2 Base Install -- Verified Checklist + +```bash +# S1: hostname + timezone +hostnamectl set-hostname +timedatectl set-timezone America/New_York + +# S2: system update +apt update && apt upgrade -y + +# S3: security baseline +apt install -y fail2ban ufw +ufw default deny incoming; ufw default allow outgoing +ufw allow ssh; ufw --force enable + +# S4: monitoring +apt install -y prometheus-node-exporter + +# S5: standard user + key +adduser ippadmin && usermod -aG sudo ippadmin +echo "ssh-ed25519 AAA... itpp-infra" >> /home/ippadmin/.ssh/authorized_keys + +# S6: harden SSH +sed -i 's/^#PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config +sed -i 's/^PermitRootLogin yes/PermitRootLogin prohibit-password/' /etc/ssh/sshd_config +systemctl restart sshd + +# S7: Docker (if needed) +curl -fsSL https://get.docker.com | bash +apt install -y docker-compose-plugin +``` + +### 2.3 Application Data Paths + +Every server must document: +- What data lives where (volumes, bind mounts, databases, uploads, .env files) +- What can be rebuilt vs what must be restored +- Secrets location +- Caddy/nginx configs +- Cron jobs +- External API dependencies + +--- + +## 3. Backup Strategy + +### 3.1 Backup Matrix + +| Data Type | Frequency | Target | Retention | Status | +|---|---|---|---|---| +| Hermes state | Every 15 min | S3 primary (live/) | 48 hours | ✅ LIVE | +| Hermes full | Daily 5 AM | S3 primary (full/) | 90 days | ✅ LIVE | +| Memory snapshots | Every 10 min | S3 primary (memories/) | 48 hours | ✅ LIVE | +| Memory history | Every 10 min | S3 primary (memories/history/) | 90 days | ✅ LIVE | +| Router configs | Daily 6 AM | S3 primary (mikrotik/) | 90 days | ✅ LIVE | +| System configs | Daily | S3 primary (/) | 90 days | 🔲 NOT AUTOMATED | +| Docker volumes | Daily | S3 primary (/) | 90 days | ❌ NOT DONE | +| Database dumps | Every 6 hours | S3 primary (/) | 90 days | ❌ NOT DONE | +| Application data | Per-app | S3 primary (/) | 90 days | 🔲 PARTIAL | +| S3 secondary | Daily sync | Wasabi us-west-2 | 90 days | ❌ NOT SET UP | + +### 3.2 Backup Security + +- **Versioning:** ON (all 3 buckets) +- **Object Lock:** NOT enabled +- **Delete protection:** NOT configured +- **Backup credentials:** Admin-level (can delete) -- NOT write-only +- **Encryption:** Server-side only (Wasabi default) + +--- + +## 4. Failover / Failback + +### 4.1 Warm Standby Timing (Corrected) + +**Old:** Check every 5 min, failover after 2 min unreachable (impossible -- can't detect 2-min outage at 5-min intervals) + +**New:** Check every 30 seconds, failover after 4 failed checks (~2 min detection) + 45-90s sync = ~3-3.5 min actual, 5 min committed RTO + +### 4.2 Health Check -- Two Layers + +**Layer 1 (Ping):** ICMP to Core (152.53.192.33) every 30s +**Layer 2 (HTTP):** GET /healthz on Tailscale IP -- checks Gateway heartbeat, SQLite PRAGMA, local Ollama + +### 4.3 Split-Brain Prevention + +S3 active-lock.json mechanism: + +```json +{"active_node": "core", "heartbeat": "", "lock_holder": "core"} +``` + +Core writes heartbeat every 30s while healthy. Failover only proceeds after: +1. Core unreachable for 4 consecutive checks (app1-bu local) +2. External monitor independently confirms Core down +3. active-lock.json heartbeat stale (>90s) +4. app1-bu writes itself as lock_holder +5. Gateway starts +6. Operator alerted + +If Core can be reached for fencing, app1-bu SSHes in and stops/masks Hermes before taking over. + +### 4.4 Failback Protocol + +**No auto-failback.** Sequence: +1. Confirm app1-bu is active node +2. Keep Core stopped -- no auto-start +3. Restore Core OS + stack +4. Sync latest state from app1-bu/S3 into Core +5. Validate sync timestamp +6. Start Core in passive mode +7. Stop app1-bu gateway +8. Update active-lock.json: "active_node": "core" +9. Start Core gateway +10. 5 consecutive health checks pass +11. Release standby lock +12. Monitor 30 min before closing + +**Rollback:** If Core fails health checks mid-failback, revert immediately -- restore app1-bu as active, keep Core in maintenance. + +### 4.5 Provider/Account Outage + +If entire netcup account is unavailable: +1. app1-bu (Hetzner) takes over +2. DNS records pointed to app1-bu IP +3. No auto-failback -- human decision + +--- + +## 5. Per-Server RTO/RPO Targets + +| System | RTO | RPO | Notes | +|---|---|---|---| +| Core (Hermes) | 5 min | 15 min | Via warm standby (app1-bu) | +| app1-bu (standby) | N/A | <=15 min | Recovery path for Core | +| app1/2/3 (Standard tier) | 60 min | 24h | Rebuild from S3 + docker compose | +| wphost02 (WordPress) | 2h | 24h | DB restore from dump | +| fleettracker360 | 2h | 24h | DB restore from dump | +| Legacy Hetzner hosts | 4h | 24h | Via rescue mode or S3 | + +--- + +## 6. Immediate Action Items + +### P0 -- This Week + +| # | Action | Details | +|---|---|---| +| 1 | Clean AI server disk | 92% full -- purge old Docker images/logs | +| 2 | Create S3 write-only IAM user | PutObject only, no delete/list | +| 3 | Enable S3 Object Lock | 7-day minimum retention | +| 4 | Deploy Docker volume backup | All Docker hosts | +| 5 | Deploy DB dump cron | wphost02 + fleettracker360 (MariaDB) | +| 6 | Deploy 30s watchdog | Currently documented, need deployment | + +### P1 -- Next 2 Weeks + +| # | Action | +|---|---| +| 7 | Document Tony's Hermes config + backup to S3 | +| 8 | Provision secondary S3 bucket (Wasabi us-west-2) | +| 9 | Extend snapshot retention from 48h to 30 days | +| 10 | Add backup success/failure monitoring to ops portal | + +### P2 -- Next 30 Days + +| # | Action | +|---|---| +| 11 | Full DR simulation for Core failover | +| 12 | Monthly restore testing schedule begins | +| 13 | Deploy Prometheus + Grafana stack | +| 14 | Add disk threshold alerts for all servers | +| 15 | Begin netcup migration | + +--- + +## 7. Restore Runbooks (Summary) + +Full per-server restore runbooks with exact commands are in the Per-Server Runbooks document (companion doc 3 of the v3 DR stack). Each covers: +- Purpose + dependencies +- DNS records, firewall ports, backup path +- Required secrets +- Step-by-step restore commands +- DB restore with pre-import dump verification +- Docker volume restore +- Health checks +- Known failure modes + +Hosts covered: Core, app1-bu, app1/2/3, wphost02, fleettracker360, Legacy Hetzner hosts. + +--- + +## 8. Third-Party Review Responses + +### Reviewer 1 Findings -- Addressed + +| Finding | Addressed In | +|---|---| +| Failover timing mismatch (5min check did not equal 2min RTO) | Section 4.1 -- Corrected to 30s checks | +| Backup compliance gap (standard says daily but not happening) | Section 3.1 -- Gap closure plan | +| AI server 92% disk critical | Section 6 -- P0 action item #1 | +| Memory consolidation data loss risk | Separate docs per priority-tagging proposal | +| Tony's Hermes undocumented | Section 6 -- P1 action item #7 | + +### Reviewer 2 Findings -- Addressed + +| Finding | Addressed In | +|---|---| +| S3 single point of failure | Section 1 + Section 3.1 -- Two-region plan | +| Backup frequency vs RPO mismatch | Section 5 -- RPO per data type | +| Restore procedures not detailed | Section 7 -- Full runbooks reference | +| Failback under-defined | Section 4.4 -- Protocol | +| No backup restore testing | Separate testing schedule doc | +| Retention too short (48h snapshots) | Section 6 -- P1 action item #9 | +| Backup security not described | Section 3.2 | +| Application backup details vague | Section 2.3 | +| Secrets recovery not documented | Section 1 (shared deps table) | +| Provider/account outage missing | Section 4.5 | + +--- + +*This plan is ACTIVE. Review quarterly or after any infrastructure change.*