ippadmin/disaster-recovery
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
master
disaster-recovery/itpp-recovery-manual.md
T

38 KiB
Raw Permalink Blame History

🏢 IT Pro Partner — Full Infrastructure Recovery Manual

Author: Sho'Nuff | Audience: Germaine
Last updated: July 9, 2026
Purpose: Recover ANY server, service, Docker container, or piece of ITPP infrastructure without Sho'Nuff.
Supersedes: shark-game-recovery-manual.md — this manual covers everything, including the shark game.

Table of Contents

  1. Architecture Overview
  2. Core Server (netcup — 152.53.192.33)
  3. Hermes Agent (AI Assistant)
  4. app1-bu Standby (Hetzner — 5.161.114.8)
  5. Docker Services on Core
  6. AI Services (ai.itpropartner.com — 178.156.167.181)
  7. All Other Hetzner Servers
  8. S3 Backups (Wasabi)
  9. Routers and Networking
  10. Shared Credentials
  11. Quick Reference — Common Commands
  12. Disaster Scenarios
  13. Cron Jobs — Complete List

1. Architecture Overview

All Servers at a Glance

# Server IP Provider Specs OS What It Runs
1 Core (Primary) 152.53.192.33 netcup RS 2000 8C/15G/512G Debian 13 Hermes, Caddy, Docker (Twenty, Vaultwarden, Docuseal, SearXNG), Shark Game, Ollama, VPN
2 app1-bu (Standby) 5.161.114.8 Hetzner CPX11 2C/2G/40G Ubuntu 26.04 Hermes standby, StrongSwan, L2TP
3 ai.itpropartner.com 178.156.167.181 Hetzner CPX41 8C/16G/240G Ubuntu 24.04 LiteLLM, Open WebUI, Ollama, Qdrant
4 hudu.itpropartner.com 178.156.130.130 Hetzner CPX21 3C/4G/80G Ubuntu 24.04 Hudu Docs, SWAG, MariaDB
5 unifi 178.156.131.57 Hetzner CPX21 3C/4G/80G Ubuntu 24.04 UniFi Controller, MongoDB, MariaDB
6 unms.forefrontwireless.com 5.161.225.131 Hetzner CPX21 3C/4G/80G Ubuntu 24.04 UNMS/UISP, UCRM, PostgreSQL, RabbitMQ
7 wphost02 5.161.62.38 Hetzner CPX21 3C/4G/80G Ubuntu 24.04 RunCloud, WordPress (Apex), MariaDB
8 app1.itpropartner.com 87.99.144.163 Hetzner CPX11 2C/2G/40G Ubuntu 24.04 n8n, PostgreSQL, Nginx
9 docker 178.156.168.35 Hetzner CPX11 2C/2G/40G Ubuntu 24.04 NPM, RustDesk, Uptime-Kuma, StrongSwan
10 fleettracker360 178.156.149.32 Hetzner CPX11 2C/2G/40G Ubuntu 24.04 Traccar, RunCloud, MariaDB

Domain Map

Domain Server Service
crm.debtrecoveryexperts.com Core (152.53.192.33) Twenty CRM (port 3001)
sign.itpropartner.com Core (152.53.192.33) DocuSeal (port 3000)
app.itpropartner.com Core (152.53.192.33) Hermes Assistant PWA (port 8082)
core.itpropartner.com Core (152.53.192.33) API, static files, health checks
portal.debtrecoveryexperts.com Core (152.53.192.33) DRE Portal (static)
shark.iamgmb.com Core (152.53.192.33) Shark Game (port 8083)
shark.itpropartner.com Core (152.53.192.33) Shark Game (static)
ops.itpropartner.com Core (152.53.192.33) Ops Portal (static)
admin-ai.itpropartner.com ai (178.156.167.181) LiteLLM proxy
hudu.itpropartner.com hudu (178.156.130.130) Hudu documentation
unms.forefrontwireless.com unms (5.161.225.131) WISP management
app1.itpropartner.com n8n (87.99.144.163) n8n workflows

2. Core Server (netcup — 152.53.192.33)

This is the most important server. It runs Hermes (your AI assistant), the web proxy (Caddy), Docker services, the shark game, and VPN services.

Detail Value
IP 152.53.192.33
Hostname core (internal)
Provider netcup RS 2000 G12
SSH Key itpp-infra (/root/.ssh/itpp-infra)
OS Debian 13 (trixie)
Disk 503 GB (6% used — plenty of room)

SSH Access

ssh root@152.53.192.33 -i /root/.ssh/itpp-infra

Services Running on Core

Service Type Port Status Command
Hermes Agent systemd 8081 systemctl status hermes
Hermes Browser (Chromium) systemd 9222 (localhost) systemctl status hermes-browser
Caddy Web Server systemd 80/443 systemctl status caddy
Ollama (LLM) systemd 11434 (localhost) systemctl status ollama
Shark Game systemd 8083 systemctl status shark-game
MySQL Tunnel systemd 33060 (localhost) systemctl status mysql-tunnel
StrongSwan (IPsec VPN) systemd 500/4500 UDP systemctl status strongswan-starter
xl2tpd (L2TP VPN) systemd 1701 UDP systemctl status xl2tpd
Tailscale systemd — systemctl status tailscaled
Twenty CRM Docker 3001 docker ps | grep twenty
Vaultwarden Docker 8080 docker ps | grep vaultwarden
DocuSeal Docker 3000 (localhost) docker ps | grep docuseal
SearXNG Docker 8888 (localhost) docker ps | grep searxng

Recovery from Full System Failure

If the server is completely dead (won't boot, drive failed, etc.):

  1. Order a new netcup RS 2000 from the Server Control Panel:

  2. Set up the basics (after new server is provisioned):

    # Add SSH key
mkdir -p /root/.ssh && chmod 700 /root/.ssh
# Copy your itpp-infra key into /root/.ssh/itpp-infra
# (from your own machine or from the standby)

# Install essentials
apt update && apt install -y curl wget git docker.io docker-compose-v2

  3. Restore Hermes (see Section 3)

  4. Restore Docker services — each one is documented below in Section 5.

  5. Restart Caddy:

    source /opt/awscli-venv/bin/activate
aws s3 cp s3://itpropartner-system-configs/caddy/Caddyfile /etc/caddy/Caddyfile \
  --endpoint-url https://s3.us-east-1.wasabisys.com
systemctl restart caddy

Checking System Health

# Disk space
df -h

# Memory
free -h

# Running Docker containers
docker ps

# All systemd services
systemctl list-unit-files --type=service | grep enabled

# Recent system logs
journalctl -xe -n 50

3. Hermes Agent (AI Assistant)

Hermes is the AI assistant running on the Core server. It powers Telegram chat, web search, email, and infrastructure management.

How to Check If Hermes Is Running

systemctl status hermes

Look for: Active: active (running)

How to Restart Hermes

systemctl restart hermes

Then verify: systemctl status hermes

How to View Hermes Logs

# Last 100 lines
journalctl -u hermes -n 100

# Live tail
journalctl -u hermes -n 50 -f

# Today's logs only
journalctl -u hermes --since today

Where Hermes Lives

Item Path
Config /root/.hermes/config.yaml
Environment/Secrets /root/.hermes/.env
State DB (conversations) /root/.hermes/state.db
Skills /root/.hermes/skills/
Scripts /root/.hermes/scripts/
Cron jobs /root/.hermes/cron/jobs.json
Systemd service /etc/systemd/system/hermes.service

Hermes Browser (Headless Chromium)

The browser service runs alongside Hermes for web automation:

systemctl status hermes-browser
systemctl restart hermes-browser

It runs Chrome on port 9222 (localhost only).

Restoring Hermes from S3 Backup

Step 1: Stop Hermes (if running)

systemctl stop hermes

Step 2: Restore the state database

source /opt/awscli-venv/bin/activate
aws s3 cp s3://hermes-vps-backups/live/state.db /root/.hermes/data/state.db \
  --endpoint-url https://s3.us-east-1.wasabisys.com

Step 3: Restore the full config

aws s3 sync s3://hermes-vps-backups/live/ /root/.hermes/ \
  --endpoint-url https://s3.us-east-1.wasabisys.com \
  --exclude "audio_cache/*" \
  --exclude "image_cache/*" \
  --exclude "cache/*" \
  --exclude "sandboxes/*"

Step 4: Start Hermes

systemctl start hermes

If Hermes won't start (broken binary or config):

  1. Reinstall from the official script:

    curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash

  2. Then restore config from S3 (Step 3 above).

  3. Restart:

    systemctl restart hermes

4. app1-bu Standby (Hetzner — 5.161.114.8)

What it is: A warm standby server on Hetzner. If the Core server (netcup) dies, this server takes over as Hermes.

Server Details

Detail Value
IP 5.161.114.8
Hostname app1-bu.itpropartner.com
Plan CPX11 (2C/2G/40G)
OS Ubuntu 26.04
SSH Key itpp-infra

SSH Access

ssh root@5.161.114.8 -i /root/.ssh/itpp-infra

What Runs on Standby (Normally Dormant)

Service Status Purpose
hermes-gateway Disabled (dormant) Hermes standby
hermes-standby Disabled Standby restore service
strongswan-starter Active IPsec VPN (always on)
xl2tpd Active L2TP tunnel (always on)
home-router-vpn Active Router VPN tunnel
home-router-keepalive Active Router watchdog

How Failover Works (Automatic)

The watchdog script runs every 5 minutes via cron (*/5 * * * * /root/.hermes/scripts/hermes-standby-watchdog.sh):

  1. Checks if Core server (152.53.192.33) is reachable via ping
  2. If Core is alive: exits silently (nothing to do)
  3. If Core is unreachable for ~3.5 minutes: confirms via 4 consecutive ping failures
  4. Takes over: syncs latest Hermes state from S3, starts Hermes gateway
  5. Sends alerts via Telegram and email to g@germainebrown.com

How to Manually Promote Standby (Failover)

If you need to manually switch to standby:

ssh root@5.161.114.8 -i /root/.ssh/itpp-infra 'systemctl start hermes-gateway'

How to Demote (Switch Back to Core)

After Core is restored:

ssh root@5.161.114.8 -i /root/.ssh/itpp-infra 'systemctl stop hermes-gateway'

Then make sure Hermes is running on Core:

systemctl restart hermes

Standby Sync

The standby syncs state from S3 every 10 minutes (*/10 * * * * /root/.hermes/scripts/hermes-standby-sync.sh) so it's always ready to take over.

5. Docker Services on Core

All Docker services live under /root/docker/<service>/. Each has its own docker-compose.yml and .env file.

General Docker Commands (Work for Any Service)

# List running containers
docker ps

# Restart a specific service
cd /root/docker/<service> && docker compose --env-file .env down && docker compose --env-file .env up -d

# View logs for a service
docker logs <container-name> -n 50

# Check health
docker ps --filter name=<service> --format '{{.Names}} {{.Status}}'

5.1 Twenty CRM

Purpose: Customer Relationship Management for Debt Recovery Experts
URL: https://crm.debtrecoveryexperts.com
Port: 3001 (maps to container port 3000)

Containers (4 total)

Container Image Purpose
twenty-server-1 twentycrm/twenty:latest NestJS API + frontend
twenty-worker-1 twentycrm/twenty:latest Background job processor
twenty-db-1 postgres:16 PostgreSQL database
twenty-redis-1 redis Queue + cache

Compose Location

/root/docker/twenty/docker-compose.yml /root/docker/twenty/.env

Data Locations

  • PostgreSQL data: Docker volume (twenty_pg_data or similar)
  • File uploads: Docker volume (twenty_server-local-data or similar)

Backup

# Export PostgreSQL database
docker exec twenty-db-1 pg_dump -U postgres default > /root/twenty-backup-$(date +%Y%m%d).sql

Restore from Scratch

cd /root/docker/twenty
docker compose --env-file .env down -v    # WARNING: -v removes volumes
docker compose --env-file .env up -d
# Then restore database:
cat /path/to/backup.sql | docker exec -i twenty-db-1 psql -U postgres default

Verify It's Working

curl -sI https://crm.debtrecoveryexperts.com/ | head -3
# Should return HTTP/2 200

5.2 Vaultwarden

Purpose: Password manager (compatible with Bitwarden apps)
URL: https://vaultwarden.tailc2f3b0.ts.net (Tailscale only -- not public internet)
Port: 8080 -> container port 80

Compose Location

/root/docker/vaultwarden/docker-compose.yml /root/docker/vaultwarden/.env

Data Location

/root/docker/vaultwarden/data/ (bind mount)

Key Config

Variable Value
DOMAIN https://vaultwarden.tailc2f3b0.ts.net
SIGNUPS_ALLOWED false
INVITATIONS_ALLOWED true
ADMIN_TOKEN Stored in .env

Restart

cd /root/docker/vaultwarden && docker compose --env-file .env down && docker compose --env-file .env up -d

Restore from S3

cd /root/docker/vaultwarden
source /opt/awscli-venv/bin/activate
aws s3 sync s3://hermes-vps-backups/live/docker/vaultwarden/ ./ \
  --endpoint-url https://s3.us-east-1.wasabisys.com
docker compose --env-file .env up -d

Verify

curl -sI http://127.0.0.1:8080 | head -3

5.3 DocuSeal

Purpose: Document signing (contracts, forms)
URL: https://sign.itpropartner.com
Port: 127.0.0.1:3000 (localhost only -- proxied by Caddy)

Compose Location

/root/docker/docuseal/docker-compose.yml

Data Location

/root/docker/docuseal/data/ (bind mount -- SQLite database)

Restart

cd /root/docker/docuseal && docker compose down && docker compose up -d

Restore from S3

cd /root/docker/docuseal
source /opt/awscli-venv/bin/activate
aws s3 sync s3://hermes-vps-backups/live/docker/docuseal/ ./ \
  --endpoint-url https://s3.us-east-1.wasabisys.com
docker compose up -d

Verify

curl -sI https://sign.itpropartner.com | head -3

5.4 SearXNG

Purpose: Private, self-hosted search engine (used by Hermes for web searches)
URL: http://127.0.0.1:8888 (localhost only)
Port: 127.0.0.1:8888

Compose Location

/root/docker/searxng/docker-compose.yml

Data Locations

  • /root/docker/searxng/searxng-data/ -- settings and config
  • /root/docker/searxng/searxng-themes/ -- theme overrides

Restart

cd /root/docker/searxng && docker compose down && docker compose up -d

Quick Test

curl -s "http://127.0.0.1:8888/search?q=test&format=json" | head -5

6. AI Services (ai.itpropartner.com)

IMPORTANT: This server hosts admin-ai.itpropartner.com (LiteLLM) which powers Sho'Nuff's deepseek-chat model. If this goes down, Hermes loses its brain.

DISK WARNING: 92% used (199G of 226G) -- needs attention soon.

Server Details

Detail Value
IP 178.156.167.181
Hostname ai
Provider Hetzner CPX41
SSH ssh root@178.156.167.181 -i /root/.ssh/itpp-infra
OS Ubuntu 24.04
Status Still on Hetzner (will migrate to netcup Monitor server)

Docker Containers (6 total)

Container Image Port Purpose
litellm ghcr.io/berriai/litellm:v1.84.0 4000 LLM proxy (currently unhealthy but working)
litellm_postgres postgres:16 5432 LiteLLM database
openwebui ghcr.io/open-webui/open-webui:0.9.6 8080 Chat interface at admin-ai.itpropartner.com
ollama ollama/ollama:latest 11434 Local LLM inference
qdrant qdrant/qdrant:latest 6333-6334 Vector storage for RAG
caddy caddy:latest 80/443 Reverse proxy

Compose Location

/opt/ai/docker-compose.yml

Restart All AI Services

ssh root@178.156.167.181 -i /root/.ssh/itpp-infra
cd /opt/ai && docker compose down && docker compose up -d

Check Health

# LiteLLM health
curl -s https://admin-ai.itpropartner.com/health/liveliness

# Check all containers
docker ps

Restore Open WebUI Database

The WebUI SQLite DB is at /app/backend/data/webui.db inside the container. To restore:

# Find the Open WebUI volume
docker volume inspect ai_open-webui || docker volume ls | grep webui

# To restore from a backup file:
docker run --rm -v ai_open-webui:/data -v /path/to/backup:/backup alpine sh -c \
  "cp /backup/webui.db /data/backend/data/webui.db"

Restore LiteLLM PostgreSQL

# Dump current DB
docker exec litellm_postgres pg_dump -U postgres litellm > /root/litellm-backup.sql

# Restore from backup
cat /root/litellm-backup.sql | docker exec -i litellm_postgres psql -U postgres litellm

Key User Accounts

  • Open WebUI: admin@itpropartner.com (password in /root/.hermes/.env or ask Sho'Nuff)
  • Root user: root via SSH key

7. All Other Hetzner Servers

7.1 Hudu

Purpose: IT documentation and knowledge base
URL: https://hudu.itpropartner.com
Server: 178.156.130.130 (CPX21)

ssh root@178.156.130.130 -i /root/.ssh/itpp-infra

Docker Containers (5)

Container Purpose Port
hudu2-app-1 Hudu application 3000
hudu2-worker-1 Background worker --
hudu2-db-1 PostgreSQL 5432
hudu2-redis-1 Cache 6379
letsencrypt (swag) Let's Encrypt + Nginx 80/443

Restart Everything

cd /opt/hudu2 && docker compose down && docker compose up -d

Native Services (also running)

  • MariaDB 11.4.12 (native, not Docker) -- port 3306
  • Postfix mail server -- port 25
  • RunCloud agent

7.2 UniFi

Purpose: Ubiquiti network controller -- manages all UniFi devices
URL: https://unifi.itpropartner.com:8443 (or direct IP)
Server: 178.156.131.57 (CPX21)

ssh root@178.156.131.57 -i /root/.ssh/itpp-infra

What Runs on It

Service Port Purpose
unifi.service 8080, 8443, 8843, 8880, etc. UniFi Network Controller (Java)
mariadb.service 3306 MariaDB database
MongoDB 27117 (localhost) UniFi internal database

Restart UniFi Controller

systemctl restart unifi

Check UniFi Status

systemctl status unifi

Data Locations

  • UniFi data: /usr/lib/unifi/data/
  • UniFi backups (auto-generated): /usr/lib/unifi/data/backup/

7.3 UNMS/UISP

Purpose: WISP management -- manages Ubiquiti wireless devices across the network
URL: https://unms.forefrontwireless.com
Server: 5.161.225.131 (CPX21)

ssh root@5.161.225.131 -i /root/.ssh/itpp-infra

Docker Containers (10)

UNMS is a complex stack. The key containers:

Container Image Purpose
unms-nginx ubnt/unms-nginx:3.0.140 Frontend proxy
unms-api ubnt/unms:3.0.140 API server
unms-device-ws-1/2 ubnt/unms:3.0.140 Device WebSocket connections
unms-postgres ubnt/unms-postgres:3.0.140 PostgreSQL database
unms-siridb ubnt/unms-siridb:3.0.140 Time-series database
unms-rabbitmq rabbitmq:3.7.28-alpine Message queue
unms-fluentd ubnt/unms-fluentd:3.0.140 Log collector
unms-netflow ubnt/unms-netflow:3.0.140 Netflow collector
ucrm ubnt/unms-crm:4.5.31 Customer management (CRM)

Restart UNMS

cd /opt/unms && docker compose down && docker compose up -d

Native Services

  • MariaDB 11.4.12 (native, port 3306) -- separate from Docker PostgreSQL
  • Postfix mail server

Backup

/root/.hermes/scripts/backup-uisp.sh

7.4 wphost02

Purpose: WordPress hosting (Apex Track Experience)
Server: 5.161.62.38 (CPX21)

ssh root@5.161.62.38 -i /root/.ssh/itpp-infra

What Runs on It

Service Purpose
nginx-rc (RunCloud) Nginx reverse proxy on 80/443
apache2-rc (RunCloud) Apache backend on port 81
php81rc-fpm PHP 8.1
mariadb.service MariaDB on 3306
postfix Mail server

Restart Web Server

systemctl restart nginx-rc
systemctl restart apache2-rc
systemctl restart php81rc-fpm

Restart All

runcloud restart

Data Locations

  • WordPress files: managed by RunCloud (see /home/runcloud/ or RunCloud dashboard)
  • MySQL databases: RunCloud-managed

MySQL Tunnel (from Core)

The Core server connects to this MySQL via AutoSSH tunnel on port 33060 (localhost):

systemctl status mysql-tunnel

7.5 app1.itpropartner.com (n8n)

Purpose: Workflow automation (n8n)
URL: https://app1.itpropartner.com
Server: 87.99.144.163 (CPX11)

ssh root@87.99.144.163 -i /root/.ssh/itpp-infra

Docker Containers (2)

Container Purpose Port
n8n Workflow automation 5678 (localhost)
n8n-postgres PostgreSQL backend 5432

Compose Location

/opt/n8n/docker-compose.yml

Restart n8n

cd /opt/n8n && docker compose down && docker compose up -d

Native Services

  • Nginx -- reverse proxy on 80/443 -> 127.0.0.1:5678
  • MariaDB -- port 3306

Verify

curl -sI https://app1.itpropartner.com | head -3

7.6 Docker Host (NPM/RustDesk)

Purpose: Nginx Proxy Manager, RustDesk remote desktop, Uptime-Kuma monitoring
Server: 178.156.168.35 (CPX11)

ssh root@178.156.168.35 -i /root/.ssh/itpp-infra

DISK WARNING: 73% used (26G of 38G) -- getting full.

Docker Containers (7+)

Container Purpose Port
nginx-proxy-manager Reverse proxy dashboard 80, 81, 443
uptime-kuma Status monitoring 3001
hbbr RustDesk relay server 21117, 21119
hbbs RustDesk signal server 21114, 21116, 21118
strongswan IPsec VPN 500/4500
runcloud_mariadb RunCloud MariaDB --
runcloud_php* RunCloud PHP-FPM --

Restart Nginx Proxy Manager

docker restart nginx-proxy-manager

Restart Uptime-Kuma

docker restart uptime-kuma

Compose Files

Services are at /opt/<name>/docker-compose.yml (not in /root/docker/):

  • /opt/rustdesk/docker-compose.yml
  • /opt/strongswan/docker-compose.yml
  • /opt/litellm/docker-compose.yml (legacy -- this is now on the AI server)

7.7 fleettracker360 (Traccar)

Purpose: GPS fleet tracking
Server: 178.156.149.32 (CPX11)

ssh root@178.156.149.32 -i /root/.ssh/itpp-infra

What Runs on It

Service Port Purpose
traccar.service 5001-5032, 8082 GPS tracking (Java)
nginx-rc (RunCloud) 80/443 Web frontend
apache2-rc (RunCloud) 81 Apache backend
mariadb.service 3306 MariaDB database

Restart Traccar

systemctl restart traccar

Check Traccar Status

systemctl status traccar
journalctl -u traccar -n 50

Data Locations

  • Traccar data: /opt/traccar/data/ or /opt/traccar/
  • Traccar config: /opt/traccar/conf/traccar.xml

8. S3 Backups (Wasabi)

All infrastructure backups go to Wasabi S3 (us-east-1).

Credentials

  • Config file: /root/.aws/credentials
  • CLI activation: source /opt/awscli-venv/bin/activate
  • Endpoint: https://s3.us-east-1.wasabisys.com

Buckets

Bucket Contents Backup Frequency
hermes-vps-backups Hermes state, config, Docker data, skills, scripts Every 15 min (live sync), daily (full backup)
mikrotik-ccr-backups Router (CCR) config exports Daily via wisp-backup
itpropartner-backups Home router config, legacy data Daily

Bucket Structure

hermes-vps-backups/
+-- live/                  <- 15-min sync (most recent Hermes state)
|   +-- state.db          <- Conversation history (1.1 GB)
|   +-- config.yaml       <- Hermes config
|   +-- .env              <- All secrets/API keys
|   +-- docker/           <- Docker service configs & data
|   |   +-- vaultwarden/
|   |   +-- docuseal/
|   |   +-- ...
|   +-- skills/           <- All skills
|   +-- scripts/          <- All scripts
|   +-- cron/             <- Cron job definitions
|   +-- ...
+-- live-sync/            <- Alternative sync target
+-- hermes-full-backup/   <- Daily tar.gz archives
|   +-- hermes-full-backup-2026-07-05.tar.gz
|   +-- hermes-full-backup-2026-07-04.tar.gz
+-- standby/              <- DR scripts and docs
+-- docker/               <- Docker volume backups (when configured)

mikrotik-ccr-backups/
+-- wisp-backups/         <- WISP router configs

itpropartner-backups/
+-- home-router/          <- Home gateway configs
+-- shonuff/              <- Sho'Nuff personal data

Checking Backups

source /opt/awscli-venv/bin/activate

# List Hermes backup folders
aws s3 ls s3://hermes-vps-backups/ --endpoint-url https://s3.us-east-1.wasabisys.com

# List recent Hermes state files
aws s3 ls s3://hermes-vps-backups/live/ --endpoint-url https://s3.us-east-1.wasabisys.com

# List router backups
aws s3 ls s3://mikrotik-ccr-backups/wisp-backups/ --endpoint-url https://s3.us-east-1.wasabisys.com

Sync Commands

# Full Hermes restore
aws s3 sync s3://hermes-vps-backups/live/ /root/.hermes/ \
  --endpoint-url https://s3.us-east-1.wasabisys.com

# Restore just the state DB
aws s3 cp s3://hermes-vps-backups/live/state.db /root/.hermes/state.db \
  --endpoint-url https://s3.us-east-1.wasabisys.com

Backup Scripts

These scripts run automatically on Core:

Script What It Backs Up When
hermes-live-sync.sh Full .hermes directory -> live/ Every 15 min
hermes-backup.sh Daily tar.gz archive -> hermes-full-backup/ Daily
hermes-docker-sync.sh Docker volumes Config-based
hermes-system-config-sync.sh Caddyfile and system configs Config-based
home-router-backup.sh Home gateway CCR config Daily
run-wisp-backup.sh WISP router configs Daily

9. Routers and Networking

Home Gateway (MikroTik CCR)

Detail Value
Model MikroTik CCR (or similar)
IP 10.77.0.2 (via WireGuard tunnel)
SSH Key /root/.ssh/wisp_rsa
SSH User admin@10.77.0.2
Connection WireGuard (preferred) or L2TP/IPsec

Backup the Router

Manual backup:

cd /root/.hermes/scripts/wisp-backup && python3 wisp-backup.py

Wrapper script (for cron):

bash /root/.hermes/scripts/run-wisp-backup.sh

Output: Backup files go to S3 at s3://mikrotik-ccr-backups/wisp-backups/

Restore Router from Backup

  1. SSH into the router:

    ssh -i /root/.ssh/wisp_rsa admin@10.77.0.2

  2. Download the latest backup from S3:

    source /opt/awscli-venv/bin/activate
aws s3 ls s3://mikrotik-ccr-backups/wisp-backups/ --endpoint-url https://s3.us-east-1.wasabisys.com
aws s3 cp s3://mikrotik-ccr-backups/wisp-backups/<latest-file> /tmp/ \
  --endpoint-url https://s3.us-east-1.wasabisys.com

  3. Upload to router via SCP:

    scp -i /root/.ssh/wisp_rsa /tmp/<backup-file> admin@10.77.0.2:/

  4. On the router, import the config:

    /import file-name=<backup-file>

WireGuard vs L2TP/IPsec

  • WireGuard is the preferred connection method -- faster, more reliable, simpler
  • L2TP/IPsec (StrongSwan + xl2tpd) is the fallback -- runs on both Core and app1-bu
  • If WireGuard goes down, the watchdog scripts automatically try the VPN tunnel

Home Router Watchdog

  • Runs on Core via: /usr/local/bin/home-router-watchdog.sh
  • Runs on app1-bu via: /root/.hermes/scripts/wisp-backup/home-router-keepalive.sh
  • Checks connection to 10.77.0.2 periodically
  • Restarts the VPN tunnel if the router is unreachable

10. Shared Credentials

IMPORTANT: These credentials are essential for recovery. Guard them carefully.

What Where Purpose
All API keys & secrets /root/.hermes/.env Telegram tokens, API keys, database passwords
Email password /root/.config/himalaya/shonuff.pass g@germainebrown.com IMAP/SMTP
Router SSH key /root/.ssh/wisp_rsa SSH to MikroTik CCR
Server SSH key /root/.ssh/itpp-infra SSH to ALL servers
S3 access /root/.aws/credentials Wasabi S3 access
Himalaya config /root/.config/himalaya/config.toml Email client config
netcup admin https://www.servercontrolpanel.de/SCP/ (Customer #389212) Order/manage netcup servers

What to Do If Credentials Are Lost

  1. S3 credentials -- stored in multiple places (S3 itself can be accessed via the web console at wasabi.com)
  2. SSH keys -- backed up in /root/.hermes/ which syncs to S3
  3. Email password -- stored in /root/.config/himalaya/shonuff.pass and in the .env file
  4. If everything is lost: contact Sho'Nuff for password resets

11. Quick Reference -- Common Commands

Service Management

What Command
Check Hermes systemctl status hermes
Restart Hermes systemctl restart hermes
Check Caddy systemctl status caddy
Restart Caddy systemctl reload caddy
Check shark game systemctl status shark-game
Restart shark game systemctl restart shark-game
Check all services systemctl list-unit-files --type=service | grep enabled
Check Docker docker ps
Restart all Docker See each service in Section 5

Logs

What Command
Hermes logs journalctl -u hermes -n 100
Hermes live log journalctl -u hermes -n 50 -f
Caddy logs journalctl -u caddy -n 50
Shark game logs journalctl -u shark-game -n 100
Any service logs journalctl -u <service> -n 100
Filter for errors journalctl -u <service> -n 200 | grep -i "error\|traceback\|exception"

Docker

What Command
List running containers docker ps
List all containers docker ps -a
View Docker logs docker logs <container> -n 50
Follow Docker logs docker logs <container> -n 50 -f
Restart a Docker service cd /root/docker/<name> && docker compose --env-file .env down && docker compose --env-file .env up -d

System

What Command
Check disk space df -h
Check memory free -h
Check network ports ss -tlnp
Check cron jobs crontab -l

S3 / Backups

What Command
Activate AWS CLI source /opt/awscli-venv/bin/activate
List backup buckets aws s3 ls s3://hermes-vps-backups/ --endpoint-url https://s3.us-east-1.wasabisys.com
Restore Hermes state aws s3 cp s3://hermes-vps-backups/live/state.db /root/.hermes/state.db --endpoint-url https://s3.us-east-1.wasabisys.com
Backup router bash /root/.hermes/scripts/run-wisp-backup.sh

Other

What Command
SSH to any server ssh root@<IP> -i /root/.ssh/itpp-infra
SSH to router ssh -i /root/.ssh/wisp_rsa admin@10.77.0.2
Check LiteLLM health curl -s https://admin-ai.itpropartner.com/health/liveliness
Check site response curl -I https://<domain>
netcup SCP (admin panel) https://www.servercontrolpanel.de/SCP/

12. Disaster Scenarios

Scenario A: "Sho'Nuff Is Gone" -- Full Recovery

If Sho'Nuff disappears and you need to recover everything without them:

  1. SSH into Core:

    ssh root@152.53.192.33 -i /root/.ssh/itpp-infra

  2. Check what's running:

    systemctl list-unit-files --type=service | grep enabled
docker ps

  3. If Core is down, failover to standby (see Scenario B below)

  4. Check S3 backups exist:

    source /opt/awscli-venv/bin/activate
aws s3 ls s3://hermes-vps-backups/ --endpoint-url https://s3.us-east-1.wasabisys.com

  5. Restore Hermes (if needed):

    systemctl stop hermes
aws s3 sync s3://hermes-vps-backups/live/ /root/.hermes/ \
  --endpoint-url https://s3.us-east-1.wasabisys.com \
  --exclude "audio_cache/*" --exclude "image_cache/*"
systemctl start hermes

  6. Restart each Docker service (see Section 5)

  7. Verify Caddy is proxying:

    systemctl restart caddy
curl -I https://core.itpropartner.com

  8. Back up the router if needed:

    bash /root/.hermes/scripts/run-wisp-backup.sh

Scenario B: "Core Server Is Dead" -- Failover to Standby

If 152.53.192.33 is offline and won't come back:

  1. Wait for automatic failover -- the watchdog on app1-bu will detect it and promote itself within ~5 minutes.

  2. If automatic failover doesn't trigger, do it manually:

    ssh root@5.161.114.8 -i /root/.ssh/itpp-infra 'systemctl start hermes-gateway'

  3. Verify Hermes is running on standby:

    ssh root@5.161.114.8 -i /root/.ssh/itpp-infra 'systemctl status hermes-gateway'

  4. Order a new netcup server:

  5. When new server arrives, set it up (see Section 2 -- Recovery from Full System Failure)

  6. Demote standby and restore Core as primary (see Section 4)

Scenario C: "Database Corruption"

If a database (SQLite or PostgreSQL) has corrupted data:

  1. Identify which service is affected from error messages or logs

  2. For SQLite-based services (DocuSeal, Hermes state.db):

    # Stop the service
systemctl stop hermes  # or: docker compose down

# Check integrity
sqlite3 /path/to/database.db "PRAGMA integrity_check;"

# Restore from S3
source /opt/awscli-venv/bin/activate
aws s3 cp s3://hermes-vps-backups/live/state.db /path/to/database.db \
  --endpoint-url https://s3.us-east-1.wasabisys.com

# Restart
systemctl start hermes

  3. For PostgreSQL-based services (Twenty CRM, n8n, LiteLLM):

    # Find the database container
docker ps | grep postgres

# Restore from backup
cat /path/to/backup.sql | docker exec -i <container> psql -U postgres <database>

Scenario D: "Accidental Deletion" -- Restore from S3 Versioning

Wasabi supports S3-compatible versioning. If a file was accidentally deleted:

source /opt/awscli-venv/bin/activate

# List versions of a deleted file
aws s3api list-object-versions --bucket hermes-vps-backups --prefix live/state.db \
  --endpoint-url https://s3.us-east-1.wasabisys.com

# Copy a specific version back
aws s3api get-object --bucket hermes-vps-backups --key live/state.db \
  --version-id <version-id> /root/.hermes/state.db \
  --endpoint-url https://s3.us-east-1.wasabisys.com

Or just sync from a different backup prefix:

# Try the previous sync
aws s3 sync s3://hermes-vps-backups/live-sync/ /root/.hermes/ \
  --endpoint-url https://s3.us-east-1.wasabisys.com

# Or try a daily backup
aws s3 cp s3://hermes-vps-backups/hermes-full-backup/hermes-full-backup-2026-07-05.tar.gz /tmp/
tar xzf /tmp/hermes-full-backup-2026-07-05.tar.gz -C /root/.hermes/

Scenario E: "ai.itpropartner.com Is Down" -- AI Services Unavailable

If the AI server (178.156.167.181) is offline:

  1. Check if it's just a Docker issue:

    ssh root@178.156.167.181 -i /root/.ssh/itpp-infra
docker ps
cd /opt/ai && docker compose up -d

  2. Check disk space (this server is at 92%):

    df -h
# If full, clean up old Docker images:
docker system prune -a

  3. If the server is completely dead, Hermes loses access to the deepseek-chat model. Sho'Nuff would need to:

    • Provision a new server
    • Restore LiteLLM PostgreSQL database
    • Restore Open WebUI data
    • Point DNS to the new server

Scenario F: "The Internet Is Down at Home" -- Router Issues

If the home internet (and therefore the MikroTik router) is unavailable:

  1. Check if the router is reachable via WireGuard:

    ping -c 3 10.77.0.2

  2. If unreachable, check the VPN tunnel:

    systemctl status strongswan-starter
systemctl status xl2tpd

  3. If VPN is down but Core is up, the rest of your infrastructure still works -- only home-related services are affected.

  4. Restore router from last known good config (see Section 9).

13. Cron Jobs -- Complete List

On Core (152.53.192.33)

TZ=America/New_York

* * * * *                /root/.hermes/scripts/dre-mail-poller.py
0 * * * *                /root/.hermes/scripts/boys-mail-monitor.py collect 2>&1 | logger -t boys-mail-collect
0 19 * * *               /root/.hermes/scripts/boys-mail-monitor.py summary 2>&1 | logger -t boys-mail-summary
0 8 * * *                /root/shark-game/scraper/run.sh
*/15 * * * *             /root/.hermes/scripts/shark-draft-reminder.sh 2>&1 | logger -t shark-draft-reminder

Hermes internal cron jobs (managed by Hermes itself):

  • Various ticker-based tasks defined in /root/.hermes/cron/jobs.json

On app1-bu (5.161.114.8)

*/5 * * * *    /root/.hermes/scripts/hermes-standby-watchdog.sh
*/10 * * * *   /root/.hermes/scripts/hermes-standby-sync.sh

On wphost02 (5.161.62.38)

0 3 * * *    /usr/local/bin/lynis-scan.sh

To View Cron Jobs

crontab -l                     # Current user's crontab
cat /root/.hermes/cron/jobs.json  # Hermes-managed cron jobs

This manual supersedes shark-game-recovery-manual.md.
For shark-game-specific details (scoring, push notifications, common player issues), see the legacy manual at /root/.hermes/references/shark-game-recovery-manual.md.
All infrastructure recovery procedures above take precedence.

Reference in New Issue View Git Blame Copy Permalink