disaster-recovery/itpp-recovery-manual.md

# 🏢 IT Pro Partner — Full Infrastructure Recovery Manual

**Author:** Sho'Nuff | **Audience:** Germaine  
**Last updated:** July 9, 2026  
**Purpose:** Recover ANY server, service, Docker container, or piece of ITPP infrastructure without Sho'Nuff.  
**Supersedes:** `shark-game-recovery-manual.md` — this manual covers everything, including the shark game.

---

## Table of Contents

1. [Architecture Overview](#1-architecture-overview)
2. [Core Server (netcup — 152.53.192.33)](#2-core-server-netcup--1525319233)
3. [Hermes Agent (AI Assistant)](#3-hermes-agent-ai-assistant)
4. [app1-bu Standby (Hetzner — 5.161.114.8)](#4-app1-bu-standby-hetzner--51611148)
5. [Docker Services on Core](#5-docker-services-on-core)
   - [5.1 Twenty CRM (crm.debtrecoveryexperts.com)](#51-twenty-crm)
   - [5.2 Vaultwarden Password Manager](#52-vaultwarden-password-manager)
   - [5.3 DocuSeal Document Signing](#53-docuseal-document-signing)
   - [5.4 SearXNG Private Search](#54-searxng-private-search)
6. [AI Services (ai.itpropartner.com — 178.156.167.181)](#6-ai-services-aiitpropartnercom)
7. [All Other Hetzner Servers](#7-all-other-hetzner-servers)
   - [7.1 hudu.itpropartner.com (Documentation)](#71-hudu)
   - [7.2 unifi (UniFi Controller)](#72-unifi-unifi-controller)
   - [7.3 unms.forefrontwireless.com (WISP/UISP)](#73-unmsforefrontwirelesscom)
   - [7.4 wphost02 (WordPress/Apex)](#74-wphost02-wordpress)
   - [7.5 app1.itpropartner.com (n8n Automation)](#75-app1itpropartnercom-n8n)
   - [7.6 docker host (NPM/RustDesk/Uptime-Kuma)](#76-docker-host-npmrustdeskuptime-kuma)
   - [7.7 fleettracker360 (Traccar GPS)](#77-fleettracker360-traccar)
8. [S3 Backups (Wasabi)](#8-s3-backups-wasabi)
9. [Routers and Networking](#9-routers-and-networking)
10. [Shared Credentials](#10-shared-credentials)
11. [Quick Reference — Common Commands](#11-quick-reference--common-commands)
12. [Disaster Scenarios](#12-disaster-scenarios)
13. [Cron Jobs — Complete List](#13-cron-jobs--complete-list)

---

## 1. Architecture Overview

### All Servers at a Glance

| # | Server | IP | Provider | Specs | OS | What It Runs |
|---|--------|----|----------|-------|----|-------------|
| **1** | **Core** (Primary) | 152.53.192.33 | netcup RS 2000 | 8C/15G/512G | Debian 13 | Hermes, Caddy, Docker (Twenty, Vaultwarden, Docuseal, SearXNG), Shark Game, Ollama, VPN |
| **2** | **app1-bu** (Standby) | 5.161.114.8 | Hetzner CPX11 | 2C/2G/40G | Ubuntu 26.04 | Hermes standby, StrongSwan, L2TP |
| **3** | **ai.itpropartner.com** | 178.156.167.181 | Hetzner CPX41 | 8C/16G/240G | Ubuntu 24.04 | LiteLLM, Open WebUI, Ollama, Qdrant |
| **4** | **hudu.itpropartner.com** | 178.156.130.130 | Hetzner CPX21 | 3C/4G/80G | Ubuntu 24.04 | Hudu Docs, SWAG, MariaDB |
| **5** | **unifi** | 178.156.131.57 | Hetzner CPX21 | 3C/4G/80G | Ubuntu 24.04 | UniFi Controller, MongoDB, MariaDB |
| **6** | **unms.forefrontwireless.com** | 5.161.225.131 | Hetzner CPX21 | 3C/4G/80G | Ubuntu 24.04 | UNMS/UISP, UCRM, PostgreSQL, RabbitMQ |
| **7** | **wphost02** | 5.161.62.38 | Hetzner CPX21 | 3C/4G/80G | Ubuntu 24.04 | RunCloud, WordPress (Apex), MariaDB |
| **8** | **app1.itpropartner.com** | 87.99.144.163 | Hetzner CPX11 | 2C/2G/40G | Ubuntu 24.04 | n8n, PostgreSQL, Nginx |
| **9** | **docker** | 178.156.168.35 | Hetzner CPX11 | 2C/2G/40G | Ubuntu 24.04 | NPM, RustDesk, Uptime-Kuma, StrongSwan |
| **10** | **fleettracker360** | 178.156.149.32 | Hetzner CPX11 | 2C/2G/40G | Ubuntu 24.04 | Traccar, RunCloud, MariaDB |

### Domain Map

| Domain | Server | Service |
|--------|--------|---------|
| crm.debtrecoveryexperts.com | Core (152.53.192.33) | Twenty CRM (port 3001) |
| sign.itpropartner.com | Core (152.53.192.33) | DocuSeal (port 3000) |
| app.itpropartner.com | Core (152.53.192.33) | Hermes Assistant PWA (port 8082) |
| core.itpropartner.com | Core (152.53.192.33) | API, static files, health checks |
| portal.debtrecoveryexperts.com | Core (152.53.192.33) | DRE Portal (static) |
| shark.iamgmb.com | Core (152.53.192.33) | Shark Game (port 8083) |
| shark.itpropartner.com | Core (152.53.192.33) | Shark Game (static) |
| ops.itpropartner.com | Core (152.53.192.33) | Ops Portal (static) |
| admin-ai.itpropartner.com | ai (178.156.167.181) | LiteLLM proxy |
| hudu.itpropartner.com | hudu (178.156.130.130) | Hudu documentation |
| unms.forefrontwireless.com | unms (5.161.225.131) | WISP management |
| app1.itpropartner.com | n8n (87.99.144.163) | n8n workflows |

---

## 2. Core Server (netcup — 152.53.192.33)

**This is the most important server.** It runs Hermes (your AI assistant), the web proxy (Caddy), Docker services, the shark game, and VPN services.

| Detail | Value |
|--------|-------|
| **IP** | 152.53.192.33 |
| **Hostname** | core (internal) |
| **Provider** | netcup RS 2000 G12 |
| **SSH Key** | `itpp-infra` (`/root/.ssh/itpp-infra`) |
| **OS** | Debian 13 (trixie) |
| **Disk** | 503 GB (6% used — plenty of room) |

### SSH Access

```bash
ssh root@152.53.192.33 -i /root/.ssh/itpp-infra
```

### Services Running on Core

| Service | Type | Port | Status Command |
|---------|------|------|----------------|
| Hermes Agent | systemd | 8081 | `systemctl status hermes` |
| Hermes Browser (Chromium) | systemd | 9222 (localhost) | `systemctl status hermes-browser` |
| Caddy Web Server | systemd | 80/443 | `systemctl status caddy` |
| Ollama (LLM) | systemd | 11434 (localhost) | `systemctl status ollama` |
| Shark Game | systemd | 8083 | `systemctl status shark-game` |
| MySQL Tunnel | systemd | 33060 (localhost) | `systemctl status mysql-tunnel` |
| StrongSwan (IPsec VPN) | systemd | 500/4500 UDP | `systemctl status strongswan-starter` |
| xl2tpd (L2TP VPN) | systemd | 1701 UDP | `systemctl status xl2tpd` |
| Tailscale | systemd | — | `systemctl status tailscaled` |
| Twenty CRM | Docker | 3001 | `docker ps \| grep twenty` |
| Vaultwarden | Docker | 8080 | `docker ps \| grep vaultwarden` |
| DocuSeal | Docker | 3000 (localhost) | `docker ps \| grep docuseal` |
| SearXNG | Docker | 8888 (localhost) | `docker ps \| grep searxng` |

### Recovery from Full System Failure

If the server is completely dead (won't boot, drive failed, etc.):

1. **Order a new netcup RS 2000** from the Server Control Panel:
   - https://www.servercontrolpanel.de/SCP/ (Customer #389212)
   - Pick the same plan (RS 2000 G12) — Debian 13

2. **Set up the basics** (after new server is provisioned):
   ```bash
   # Add SSH key
   mkdir -p /root/.ssh && chmod 700 /root/.ssh
   # Copy your itpp-infra key into /root/.ssh/itpp-infra
   # (from your own machine or from the standby)
   
   # Install essentials
   apt update && apt install -y curl wget git docker.io docker-compose-v2
   ```

3. **Restore Hermes** (see Section 3)

4. **Restore Docker services** — each one is documented below in Section 5.

5. **Restart Caddy:**
   ```bash
   source /opt/awscli-venv/bin/activate
   aws s3 cp s3://itpropartner-system-configs/caddy/Caddyfile /etc/caddy/Caddyfile \
     --endpoint-url https://s3.us-east-1.wasabisys.com
   systemctl restart caddy
   ```

### Checking System Health

```bash
# Disk space
df -h

# Memory
free -h

# Running Docker containers
docker ps

# All systemd services
systemctl list-unit-files --type=service | grep enabled

# Recent system logs
journalctl -xe -n 50
```

---

## 3. Hermes Agent (AI Assistant)

Hermes is the AI assistant running on the Core server. It powers Telegram chat, web search, email, and infrastructure management.

### How to Check If Hermes Is Running

```bash
systemctl status hermes
```

Look for: **Active: active (running)**

### How to Restart Hermes

```bash
systemctl restart hermes
```

Then verify: `systemctl status hermes`

### How to View Hermes Logs

```bash
# Last 100 lines
journalctl -u hermes -n 100

# Live tail
journalctl -u hermes -n 50 -f

# Today's logs only
journalctl -u hermes --since today
```

### Where Hermes Lives

| Item | Path |
|------|------|
| Config | `/root/.hermes/config.yaml` |
| Environment/Secrets | `/root/.hermes/.env` |
| State DB (conversations) | `/root/.hermes/state.db` |
| Skills | `/root/.hermes/skills/` |
| Scripts | `/root/.hermes/scripts/` |
| Cron jobs | `/root/.hermes/cron/jobs.json` |
| Systemd service | `/etc/systemd/system/hermes.service` |

### Hermes Browser (Headless Chromium)

The browser service runs alongside Hermes for web automation:

```bash
systemctl status hermes-browser
systemctl restart hermes-browser
```

It runs Chrome on port 9222 (localhost only).

### Restoring Hermes from S3 Backup

**Step 1: Stop Hermes** (if running)

```bash
systemctl stop hermes
```

**Step 2: Restore the state database**

```bash
source /opt/awscli-venv/bin/activate
aws s3 cp s3://hermes-vps-backups/live/state.db /root/.hermes/data/state.db \
  --endpoint-url https://s3.us-east-1.wasabisys.com
```

**Step 3: Restore the full config**

```bash
aws s3 sync s3://hermes-vps-backups/live/ /root/.hermes/ \
  --endpoint-url https://s3.us-east-1.wasabisys.com \
  --exclude "audio_cache/*" \
  --exclude "image_cache/*" \
  --exclude "cache/*" \
  --exclude "sandboxes/*"
```

**Step 4: Start Hermes**

```bash
systemctl start hermes
```

**If Hermes won't start** (broken binary or config):

1. Reinstall from the official script:
   ```bash
   curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
   ```

2. Then restore config from S3 (Step 3 above).

3. Restart:
   ```bash
   systemctl restart hermes
   ```

---

## 4. app1-bu Standby (Hetzner — 5.161.114.8)

**What it is:** A warm standby server on Hetzner. If the Core server (netcup) dies, this server takes over as Hermes.

### Server Details

| Detail | Value |
|--------|-------|
| **IP** | 5.161.114.8 |
| **Hostname** | app1-bu.itpropartner.com |
| **Plan** | CPX11 (2C/2G/40G) |
| **OS** | Ubuntu 26.04 |
| **SSH Key** | `itpp-infra` |

### SSH Access

```bash
ssh root@5.161.114.8 -i /root/.ssh/itpp-infra
```

### What Runs on Standby (Normally Dormant)

| Service | Status | Purpose |
|---------|--------|---------|
| hermes-gateway | Disabled (dormant) | Hermes standby |
| hermes-standby | Disabled | Standby restore service |
| strongswan-starter | Active | IPsec VPN (always on) |
| xl2tpd | Active | L2TP tunnel (always on) |
| home-router-vpn | Active | Router VPN tunnel |
| home-router-keepalive | Active | Router watchdog |

### How Failover Works (Automatic)

The watchdog script runs every 5 minutes via cron (`*/5 * * * * /root/.hermes/scripts/hermes-standby-watchdog.sh`):

1. **Checks if Core server (152.53.192.33) is reachable** via ping
2. **If Core is alive:** exits silently (nothing to do)
3. **If Core is unreachable for ~3.5 minutes:** confirms via 4 consecutive ping failures
4. **Takes over:** syncs latest Hermes state from S3, starts Hermes gateway
5. **Sends alerts** via Telegram and email to g@germainebrown.com

### How to Manually Promote Standby (Failover)

If you need to manually switch to standby:

```bash
ssh root@5.161.114.8 -i /root/.ssh/itpp-infra 'systemctl start hermes-gateway'
```

### How to Demote (Switch Back to Core)

After Core is restored:

```bash
ssh root@5.161.114.8 -i /root/.ssh/itpp-infra 'systemctl stop hermes-gateway'
```

Then make sure Hermes is running on Core:
```bash
systemctl restart hermes
```

### Standby Sync

The standby syncs state from S3 every 10 minutes (`*/10 * * * * /root/.hermes/scripts/hermes-standby-sync.sh`) so it's always ready to take over.

---

## 5. Docker Services on Core

All Docker services live under `/root/docker/<service>/`. Each has its own `docker-compose.yml` and `.env` file.

### General Docker Commands (Work for Any Service)

```bash
# List running containers
docker ps

# Restart a specific service
cd /root/docker/<service> && docker compose --env-file .env down && docker compose --env-file .env up -d

# View logs for a service
docker logs <container-name> -n 50

# Check health
docker ps --filter name=<service> --format '{{.Names}} {{.Status}}'
```

### 5.1 Twenty CRM

**Purpose:** Customer Relationship Management for Debt Recovery Experts  
**URL:** https://crm.debtrecoveryexperts.com  
**Port:** 3001 (maps to container port 3000)

#### Containers (4 total)

| Container | Image | Purpose |
|-----------|-------|---------|
| twenty-server-1 | twentycrm/twenty:latest | NestJS API + frontend |
| twenty-worker-1 | twentycrm/twenty:latest | Background job processor |
| twenty-db-1 | postgres:16 | PostgreSQL database |
| twenty-redis-1 | redis | Queue + cache |

#### Compose Location

`/root/docker/twenty/docker-compose.yml`
`/root/docker/twenty/.env`

#### Data Locations

- PostgreSQL data: Docker volume (`twenty_pg_data` or similar)
- File uploads: Docker volume (`twenty_server-local-data` or similar)

#### Backup

```bash
# Export PostgreSQL database
docker exec twenty-db-1 pg_dump -U postgres default > /root/twenty-backup-$(date +%Y%m%d).sql
```

#### Restore from Scratch

```bash
cd /root/docker/twenty
docker compose --env-file .env down -v    # WARNING: -v removes volumes
docker compose --env-file .env up -d
# Then restore database:
cat /path/to/backup.sql | docker exec -i twenty-db-1 psql -U postgres default
```

#### Verify It's Working

```bash
curl -sI https://crm.debtrecoveryexperts.com/ | head -3
# Should return HTTP/2 200
```

### 5.2 Vaultwarden

**Purpose:** Password manager (compatible with Bitwarden apps)  
**URL:** https://vaultwarden.tailc2f3b0.ts.net (Tailscale only -- not public internet)  
**Port:** 8080 -> container port 80

#### Compose Location

`/root/docker/vaultwarden/docker-compose.yml`
`/root/docker/vaultwarden/.env`

#### Data Location

`/root/docker/vaultwarden/data/` (bind mount)

#### Key Config

| Variable | Value |
|----------|-------|
| DOMAIN | https://vaultwarden.tailc2f3b0.ts.net |
| SIGNUPS_ALLOWED | false |
| INVITATIONS_ALLOWED | true |
| ADMIN_TOKEN | Stored in `.env` |

#### Restart

```bash
cd /root/docker/vaultwarden && docker compose --env-file .env down && docker compose --env-file .env up -d
```

#### Restore from S3

```bash
cd /root/docker/vaultwarden
source /opt/awscli-venv/bin/activate
aws s3 sync s3://hermes-vps-backups/live/docker/vaultwarden/ ./ \
  --endpoint-url https://s3.us-east-1.wasabisys.com
docker compose --env-file .env up -d
```

#### Verify

```bash
curl -sI http://127.0.0.1:8080 | head -3
```

### 5.3 DocuSeal

**Purpose:** Document signing (contracts, forms)  
**URL:** https://sign.itpropartner.com  
**Port:** 127.0.0.1:3000 (localhost only -- proxied by Caddy)

#### Compose Location

`/root/docker/docuseal/docker-compose.yml`

#### Data Location

`/root/docker/docuseal/data/` (bind mount -- SQLite database)

#### Restart

```bash
cd /root/docker/docuseal && docker compose down && docker compose up -d
```

#### Restore from S3

```bash
cd /root/docker/docuseal
source /opt/awscli-venv/bin/activate
aws s3 sync s3://hermes-vps-backups/live/docker/docuseal/ ./ \
  --endpoint-url https://s3.us-east-1.wasabisys.com
docker compose up -d
```

#### Verify

```bash
curl -sI https://sign.itpropartner.com | head -3
```

### 5.4 SearXNG

**Purpose:** Private, self-hosted search engine (used by Hermes for web searches)  
**URL:** http://127.0.0.1:8888 (localhost only)  
**Port:** 127.0.0.1:8888

#### Compose Location

`/root/docker/searxng/docker-compose.yml`

#### Data Locations

- `/root/docker/searxng/searxng-data/` -- settings and config
- `/root/docker/searxng/searxng-themes/` -- theme overrides

#### Restart

```bash
cd /root/docker/searxng && docker compose down && docker compose up -d
```

#### Quick Test

```bash
curl -s "http://127.0.0.1:8888/search?q=test&format=json" | head -5
```

---

## 6. AI Services (ai.itpropartner.com)

**IMPORTANT:** This server hosts `admin-ai.itpropartner.com` (LiteLLM) which powers Sho'Nuff's deepseek-chat model. If this goes down, Hermes loses its brain.

**DISK WARNING:** 92% used (199G of 226G) -- needs attention soon.

### Server Details

| Detail | Value |
|--------|-------|
| **IP** | 178.156.167.181 |
| **Hostname** | ai |
| **Provider** | Hetzner CPX41 |
| **SSH** | `ssh root@178.156.167.181 -i /root/.ssh/itpp-infra` |
| **OS** | Ubuntu 24.04 |
| **Status** | Still on Hetzner (will migrate to netcup Monitor server) |

### Docker Containers (6 total)

| Container | Image | Port | Purpose |
|-----------|-------|------|---------|
| litellm | ghcr.io/berriai/litellm:v1.84.0 | 4000 | LLM proxy (currently unhealthy but working) |
| litellm_postgres | postgres:16 | 5432 | LiteLLM database |
| openwebui | ghcr.io/open-webui/open-webui:0.9.6 | 8080 | Chat interface at admin-ai.itpropartner.com |
| ollama | ollama/ollama:latest | 11434 | Local LLM inference |
| qdrant | qdrant/qdrant:latest | 6333-6334 | Vector storage for RAG |
| caddy | caddy:latest | 80/443 | Reverse proxy |

### Compose Location

`/opt/ai/docker-compose.yml`

### Restart All AI Services

```bash
ssh root@178.156.167.181 -i /root/.ssh/itpp-infra
cd /opt/ai && docker compose down && docker compose up -d
```

### Check Health

```bash
# LiteLLM health
curl -s https://admin-ai.itpropartner.com/health/liveliness

# Check all containers
docker ps
```

### Restore Open WebUI Database

The WebUI SQLite DB is at `/app/backend/data/webui.db` inside the container. To restore:

```bash
# Find the Open WebUI volume
docker volume inspect ai_open-webui || docker volume ls | grep webui

# To restore from a backup file:
docker run --rm -v ai_open-webui:/data -v /path/to/backup:/backup alpine sh -c \
  "cp /backup/webui.db /data/backend/data/webui.db"
```

### Restore LiteLLM PostgreSQL

```bash
# Dump current DB
docker exec litellm_postgres pg_dump -U postgres litellm > /root/litellm-backup.sql

# Restore from backup
cat /root/litellm-backup.sql | docker exec -i litellm_postgres psql -U postgres litellm
```

### Key User Accounts

- **Open WebUI:** admin@itpropartner.com (password in /root/.hermes/.env or ask Sho'Nuff)
- Root user: root via SSH key

---

## 7. All Other Hetzner Servers

### 7.1 Hudu

**Purpose:** IT documentation and knowledge base  
**URL:** https://hudu.itpropartner.com  
**Server:** 178.156.130.130 (CPX21)

```bash
ssh root@178.156.130.130 -i /root/.ssh/itpp-infra
```

#### Docker Containers (5)

| Container | Purpose | Port |
|-----------|---------|------|
| hudu2-app-1 | Hudu application | 3000 |
| hudu2-worker-1 | Background worker | -- |
| hudu2-db-1 | PostgreSQL | 5432 |
| hudu2-redis-1 | Cache | 6379 |
| letsencrypt (swag) | Let's Encrypt + Nginx | 80/443 |

#### Restart Everything

```bash
cd /opt/hudu2 && docker compose down && docker compose up -d
```

#### Native Services (also running)

- **MariaDB** 11.4.12 (native, not Docker) -- port 3306
- **Postfix** mail server -- port 25
- **RunCloud** agent

---

### 7.2 UniFi

**Purpose:** Ubiquiti network controller -- manages all UniFi devices  
**URL:** https://unifi.itpropartner.com:8443 (or direct IP)  
**Server:** 178.156.131.57 (CPX21)

```bash
ssh root@178.156.131.57 -i /root/.ssh/itpp-infra
```

#### What Runs on It

| Service | Port | Purpose |
|---------|------|---------|
| unifi.service | 8080, 8443, 8843, 8880, etc. | UniFi Network Controller (Java) |
| mariadb.service | 3306 | MariaDB database |
| MongoDB | 27117 (localhost) | UniFi internal database |

#### Restart UniFi Controller

```bash
systemctl restart unifi
```

#### Check UniFi Status

```bash
systemctl status unifi
```

#### Data Locations

- UniFi data: `/usr/lib/unifi/data/`
- UniFi backups (auto-generated): `/usr/lib/unifi/data/backup/`

---

### 7.3 UNMS/UISP

**Purpose:** WISP management -- manages Ubiquiti wireless devices across the network  
**URL:** https://unms.forefrontwireless.com  
**Server:** 5.161.225.131 (CPX21)

```bash
ssh root@5.161.225.131 -i /root/.ssh/itpp-infra
```

#### Docker Containers (10)

UNMS is a complex stack. The key containers:

| Container | Image | Purpose |
|-----------|-------|---------|
| unms-nginx | ubnt/unms-nginx:3.0.140 | Frontend proxy |
| unms-api | ubnt/unms:3.0.140 | API server |
| unms-device-ws-1/2 | ubnt/unms:3.0.140 | Device WebSocket connections |
| unms-postgres | ubnt/unms-postgres:3.0.140 | PostgreSQL database |
| unms-siridb | ubnt/unms-siridb:3.0.140 | Time-series database |
| unms-rabbitmq | rabbitmq:3.7.28-alpine | Message queue |
| unms-fluentd | ubnt/unms-fluentd:3.0.140 | Log collector |
| unms-netflow | ubnt/unms-netflow:3.0.140 | Netflow collector |
| ucrm | ubnt/unms-crm:4.5.31 | Customer management (CRM) |

#### Restart UNMS

```bash
cd /opt/unms && docker compose down && docker compose up -d
```

#### Native Services

- **MariaDB** 11.4.12 (native, port 3306) -- separate from Docker PostgreSQL
- **Postfix** mail server

#### Backup

```bash
/root/.hermes/scripts/backup-uisp.sh
```

---

### 7.4 wphost02

**Purpose:** WordPress hosting (Apex Track Experience)  
**Server:** 5.161.62.38 (CPX21)

```bash
ssh root@5.161.62.38 -i /root/.ssh/itpp-infra
```

#### What Runs on It

| Service | Purpose |
|---------|---------|
| nginx-rc (RunCloud) | Nginx reverse proxy on 80/443 |
| apache2-rc (RunCloud) | Apache backend on port 81 |
| php81rc-fpm | PHP 8.1 |
| mariadb.service | MariaDB on 3306 |
| postfix | Mail server |

#### Restart Web Server

```bash
systemctl restart nginx-rc
systemctl restart apache2-rc
systemctl restart php81rc-fpm
```

#### Restart All

```bash
runcloud restart
```

#### Data Locations

- WordPress files: managed by RunCloud (see `/home/runcloud/` or RunCloud dashboard)
- MySQL databases: RunCloud-managed

#### MySQL Tunnel (from Core)

The Core server connects to this MySQL via AutoSSH tunnel on port 33060 (localhost):
```bash
systemctl status mysql-tunnel
```

---

### 7.5 app1.itpropartner.com (n8n)

**Purpose:** Workflow automation (n8n)  
**URL:** https://app1.itpropartner.com  
**Server:** 87.99.144.163 (CPX11)

```bash
ssh root@87.99.144.163 -i /root/.ssh/itpp-infra
```

#### Docker Containers (2)

| Container | Purpose | Port |
|-----------|---------|------|
| n8n | Workflow automation | 5678 (localhost) |
| n8n-postgres | PostgreSQL backend | 5432 |

#### Compose Location

`/opt/n8n/docker-compose.yml`

#### Restart n8n

```bash
cd /opt/n8n && docker compose down && docker compose up -d
```

#### Native Services

- **Nginx** -- reverse proxy on 80/443 -> 127.0.0.1:5678
- **MariaDB** -- port 3306

#### Verify

```bash
curl -sI https://app1.itpropartner.com | head -3
```

---

### 7.6 Docker Host (NPM/RustDesk)

**Purpose:** Nginx Proxy Manager, RustDesk remote desktop, Uptime-Kuma monitoring  
**Server:** 178.156.168.35 (CPX11)

```bash
ssh root@178.156.168.35 -i /root/.ssh/itpp-infra
```

**DISK WARNING:** 73% used (26G of 38G) -- getting full.

#### Docker Containers (7+)

| Container | Purpose | Port |
|-----------|---------|------|
| nginx-proxy-manager | Reverse proxy dashboard | 80, 81, 443 |
| uptime-kuma | Status monitoring | 3001 |
| hbbr | RustDesk relay server | 21117, 21119 |
| hbbs | RustDesk signal server | 21114, 21116, 21118 |
| strongswan | IPsec VPN | 500/4500 |
| runcloud_mariadb | RunCloud MariaDB | -- |
| runcloud_php* | RunCloud PHP-FPM | -- |

#### Restart Nginx Proxy Manager

```bash
docker restart nginx-proxy-manager
```

#### Restart Uptime-Kuma

```bash
docker restart uptime-kuma
```

#### Compose Files

Services are at `/opt/<name>/docker-compose.yml` (not in `/root/docker/`):
- `/opt/rustdesk/docker-compose.yml`
- `/opt/strongswan/docker-compose.yml`
- `/opt/litellm/docker-compose.yml` (legacy -- this is now on the AI server)

---

### 7.7 fleettracker360 (Traccar)

**Purpose:** GPS fleet tracking  
**Server:** 178.156.149.32 (CPX11)

```bash
ssh root@178.156.149.32 -i /root/.ssh/itpp-infra
```

#### What Runs on It

| Service | Port | Purpose |
|---------|------|---------|
| traccar.service | 5001-5032, 8082 | GPS tracking (Java) |
| nginx-rc (RunCloud) | 80/443 | Web frontend |
| apache2-rc (RunCloud) | 81 | Apache backend |
| mariadb.service | 3306 | MariaDB database |

#### Restart Traccar

```bash
systemctl restart traccar
```

#### Check Traccar Status

```bash
systemctl status traccar
journalctl -u traccar -n 50
```

#### Data Locations

- Traccar data: `/opt/traccar/data/` or `/opt/traccar/`
- Traccar config: `/opt/traccar/conf/traccar.xml`

---

## 8. S3 Backups (Wasabi)

All infrastructure backups go to Wasabi S3 (us-east-1).

### Credentials

- **Config file:** `/root/.aws/credentials`
- **CLI activation:** `source /opt/awscli-venv/bin/activate`
- **Endpoint:** `https://s3.us-east-1.wasabisys.com`

### Buckets

| Bucket | Contents | Backup Frequency |
|--------|----------|-----------------|
| `hermes-vps-backups` | Hermes state, config, Docker data, skills, scripts | Every 15 min (live sync), daily (full backup) |
| `mikrotik-ccr-backups` | Router (CCR) config exports | Daily via wisp-backup |
| `itpropartner-backups` | Home router config, legacy data | Daily |

### Bucket Structure

```
hermes-vps-backups/
+-- live/                  <- 15-min sync (most recent Hermes state)
|   +-- state.db          <- Conversation history (1.1 GB)
|   +-- config.yaml       <- Hermes config
|   +-- .env              <- All secrets/API keys
|   +-- docker/           <- Docker service configs & data
|   |   +-- vaultwarden/
|   |   +-- docuseal/
|   |   +-- ...
|   +-- skills/           <- All skills
|   +-- scripts/          <- All scripts
|   +-- cron/             <- Cron job definitions
|   +-- ...
+-- live-sync/            <- Alternative sync target
+-- hermes-full-backup/   <- Daily tar.gz archives
|   +-- hermes-full-backup-2026-07-05.tar.gz
|   +-- hermes-full-backup-2026-07-04.tar.gz
+-- standby/              <- DR scripts and docs
+-- docker/               <- Docker volume backups (when configured)

mikrotik-ccr-backups/
+-- wisp-backups/         <- WISP router configs

itpropartner-backups/
+-- home-router/          <- Home gateway configs
+-- shonuff/              <- Sho'Nuff personal data
```

### Checking Backups

```bash
source /opt/awscli-venv/bin/activate

# List Hermes backup folders
aws s3 ls s3://hermes-vps-backups/ --endpoint-url https://s3.us-east-1.wasabisys.com

# List recent Hermes state files
aws s3 ls s3://hermes-vps-backups/live/ --endpoint-url https://s3.us-east-1.wasabisys.com

# List router backups
aws s3 ls s3://mikrotik-ccr-backups/wisp-backups/ --endpoint-url https://s3.us-east-1.wasabisys.com
```

### Sync Commands

```bash
# Full Hermes restore
aws s3 sync s3://hermes-vps-backups/live/ /root/.hermes/ \
  --endpoint-url https://s3.us-east-1.wasabisys.com

# Restore just the state DB
aws s3 cp s3://hermes-vps-backups/live/state.db /root/.hermes/state.db \
  --endpoint-url https://s3.us-east-1.wasabisys.com
```

### Backup Scripts

These scripts run automatically on Core:

| Script | What It Backs Up | When |
|--------|------------------|------|
| `hermes-live-sync.sh` | Full .hermes directory -> `live/` | Every 15 min |
| `hermes-backup.sh` | Daily tar.gz archive -> `hermes-full-backup/` | Daily |
| `hermes-docker-sync.sh` | Docker volumes | Config-based |
| `hermes-system-config-sync.sh` | Caddyfile and system configs | Config-based |
| `home-router-backup.sh` | Home gateway CCR config | Daily |
| `run-wisp-backup.sh` | WISP router configs | Daily |

---

## 9. Routers and Networking

### Home Gateway (MikroTik CCR)

| Detail | Value |
|--------|-------|
| **Model** | MikroTik CCR (or similar) |
| **IP** | 10.77.0.2 (via WireGuard tunnel) |
| **SSH Key** | `/root/.ssh/wisp_rsa` |
| **SSH User** | `admin@10.77.0.2` |
| **Connection** | WireGuard (preferred) or L2TP/IPsec |

### Backup the Router

**Manual backup:**
```bash
cd /root/.hermes/scripts/wisp-backup && python3 wisp-backup.py
```

**Wrapper script (for cron):**
```bash
bash /root/.hermes/scripts/run-wisp-backup.sh
```

**Output:** Backup files go to S3 at `s3://mikrotik-ccr-backups/wisp-backups/`

### Restore Router from Backup

1. SSH into the router:
   ```bash
   ssh -i /root/.ssh/wisp_rsa admin@10.77.0.2
   ```

2. Download the latest backup from S3:
   ```bash
   source /opt/awscli-venv/bin/activate
   aws s3 ls s3://mikrotik-ccr-backups/wisp-backups/ --endpoint-url https://s3.us-east-1.wasabisys.com
   aws s3 cp s3://mikrotik-ccr-backups/wisp-backups/<latest-file> /tmp/ \
     --endpoint-url https://s3.us-east-1.wasabisys.com
   ```

3. Upload to router via SCP:
   ```bash
   scp -i /root/.ssh/wisp_rsa /tmp/<backup-file> admin@10.77.0.2:/
   ```

4. On the router, import the config:
   ```
   /import file-name=<backup-file>
   ```

### WireGuard vs L2TP/IPsec

- **WireGuard** is the preferred connection method -- faster, more reliable, simpler
- **L2TP/IPsec** (StrongSwan + xl2tpd) is the fallback -- runs on both Core and app1-bu
- If WireGuard goes down, the watchdog scripts automatically try the VPN tunnel

### Home Router Watchdog

- Runs on Core via: `/usr/local/bin/home-router-watchdog.sh`
- Runs on app1-bu via: `/root/.hermes/scripts/wisp-backup/home-router-keepalive.sh`
- Checks connection to 10.77.0.2 periodically
- Restarts the VPN tunnel if the router is unreachable

---

## 10. Shared Credentials

**IMPORTANT:** These credentials are essential for recovery. Guard them carefully.

| What | Where | Purpose |
|------|-------|---------|
| **All API keys & secrets** | `/root/.hermes/.env` | Telegram tokens, API keys, database passwords |
| **Email password** | `/root/.config/himalaya/shonuff.pass` | g@germainebrown.com IMAP/SMTP |
| **Router SSH key** | `/root/.ssh/wisp_rsa` | SSH to MikroTik CCR |
| **Server SSH key** | `/root/.ssh/itpp-infra` | SSH to ALL servers |
| **S3 access** | `/root/.aws/credentials` | Wasabi S3 access |
| **Himalaya config** | `/root/.config/himalaya/config.toml` | Email client config |
| **netcup admin** | https://www.servercontrolpanel.de/SCP/ (Customer #389212) | Order/manage netcup servers |

### What to Do If Credentials Are Lost

1. **S3 credentials** -- stored in multiple places (S3 itself can be accessed via the web console at wasabi.com)
2. **SSH keys** -- backed up in `/root/.hermes/` which syncs to S3
3. **Email password** -- stored in `/root/.config/himalaya/shonuff.pass` and in the `.env` file
4. **If everything is lost:** contact Sho'Nuff for password resets

---

## 11. Quick Reference -- Common Commands

### Service Management

| What | Command |
|------|---------|
| Check Hermes | `systemctl status hermes` |
| Restart Hermes | `systemctl restart hermes` |
| Check Caddy | `systemctl status caddy` |
| Restart Caddy | `systemctl reload caddy` |
| Check shark game | `systemctl status shark-game` |
| Restart shark game | `systemctl restart shark-game` |
| Check all services | `systemctl list-unit-files --type=service \| grep enabled` |
| Check Docker | `docker ps` |
| Restart all Docker | See each service in Section 5 |

### Logs

| What | Command |
|------|---------|
| Hermes logs | `journalctl -u hermes -n 100` |
| Hermes live log | `journalctl -u hermes -n 50 -f` |
| Caddy logs | `journalctl -u caddy -n 50` |
| Shark game logs | `journalctl -u shark-game -n 100` |
| Any service logs | `journalctl -u <service> -n 100` |
| Filter for errors | `journalctl -u <service> -n 200 \| grep -i "error\\|traceback\\|exception"` |

### Docker

| What | Command |
|------|---------|
| List running containers | `docker ps` |
| List all containers | `docker ps -a` |
| View Docker logs | `docker logs <container> -n 50` |
| Follow Docker logs | `docker logs <container> -n 50 -f` |
| Restart a Docker service | `cd /root/docker/<name> && docker compose --env-file .env down && docker compose --env-file .env up -d` |

### System

| What | Command |
|------|---------|
| Check disk space | `df -h` |
| Check memory | `free -h` |
| Check network ports | `ss -tlnp` |
| Check cron jobs | `crontab -l` |

### S3 / Backups

| What | Command |
|------|---------|
| Activate AWS CLI | `source /opt/awscli-venv/bin/activate` |
| List backup buckets | `aws s3 ls s3://hermes-vps-backups/ --endpoint-url https://s3.us-east-1.wasabisys.com` |
| Restore Hermes state | `aws s3 cp s3://hermes-vps-backups/live/state.db /root/.hermes/state.db --endpoint-url https://s3.us-east-1.wasabisys.com` |
| Backup router | `bash /root/.hermes/scripts/run-wisp-backup.sh` |

### Other

| What | Command |
|------|---------|
| SSH to any server | `ssh root@<IP> -i /root/.ssh/itpp-infra` |
| SSH to router | `ssh -i /root/.ssh/wisp_rsa admin@10.77.0.2` |
| Check LiteLLM health | `curl -s https://admin-ai.itpropartner.com/health/liveliness` |
| Check site response | `curl -I https://<domain>` |
| netcup SCP (admin panel) | https://www.servercontrolpanel.de/SCP/ |

---

## 12. Disaster Scenarios

### Scenario A: "Sho'Nuff Is Gone" -- Full Recovery

**If Sho'Nuff disappears and you need to recover everything without them:**

1. **SSH into Core:**
   ```bash
   ssh root@152.53.192.33 -i /root/.ssh/itpp-infra
   ```

2. **Check what's running:**
   ```bash
   systemctl list-unit-files --type=service | grep enabled
   docker ps
   ```

3. **If Core is down, failover to standby** (see Scenario B below)

4. **Check S3 backups exist:**
   ```bash
   source /opt/awscli-venv/bin/activate
   aws s3 ls s3://hermes-vps-backups/ --endpoint-url https://s3.us-east-1.wasabisys.com
   ```

5. **Restore Hermes** (if needed):
   ```bash
   systemctl stop hermes
   aws s3 sync s3://hermes-vps-backups/live/ /root/.hermes/ \
     --endpoint-url https://s3.us-east-1.wasabisys.com \
     --exclude "audio_cache/*" --exclude "image_cache/*"
   systemctl start hermes
   ```

6. **Restart each Docker service** (see Section 5)

7. **Verify Caddy is proxying:**
   ```bash
   systemctl restart caddy
   curl -I https://core.itpropartner.com
   ```

8. **Back up the router if needed:**
   ```bash
   bash /root/.hermes/scripts/run-wisp-backup.sh
   ```

### Scenario B: "Core Server Is Dead" -- Failover to Standby

**If 152.53.192.33 is offline and won't come back:**

1. **Wait for automatic failover** -- the watchdog on app1-bu will detect it and promote itself within ~5 minutes.

2. **If automatic failover doesn't trigger**, do it manually:
   ```bash
   ssh root@5.161.114.8 -i /root/.ssh/itpp-infra 'systemctl start hermes-gateway'
   ```

3. **Verify Hermes is running on standby:**
   ```bash
   ssh root@5.161.114.8 -i /root/.ssh/itpp-infra 'systemctl status hermes-gateway'
   ```

4. **Order a new netcup server:**
   - Go to https://www.servercontrolpanel.de/SCP/ (Customer #389212)
   - Order an RS 2000 G12 with Debian 13

5. **When new server arrives, set it up** (see Section 2 -- Recovery from Full System Failure)

6. **Demote standby and restore Core as primary** (see Section 4)

### Scenario C: "Database Corruption"

**If a database (SQLite or PostgreSQL) has corrupted data:**

1. **Identify which service is affected** from error messages or logs

2. **For SQLite-based services** (DocuSeal, Hermes state.db):
   ```bash
   # Stop the service
   systemctl stop hermes  # or: docker compose down
   
   # Check integrity
   sqlite3 /path/to/database.db "PRAGMA integrity_check;"
   
   # Restore from S3
   source /opt/awscli-venv/bin/activate
   aws s3 cp s3://hermes-vps-backups/live/state.db /path/to/database.db \
     --endpoint-url https://s3.us-east-1.wasabisys.com
   
   # Restart
   systemctl start hermes
   ```

3. **For PostgreSQL-based services** (Twenty CRM, n8n, LiteLLM):
   ```bash
   # Find the database container
   docker ps | grep postgres
   
   # Restore from backup
   cat /path/to/backup.sql | docker exec -i <container> psql -U postgres <database>
   ```

### Scenario D: "Accidental Deletion" -- Restore from S3 Versioning

Wasabi supports S3-compatible versioning. If a file was accidentally deleted:

```bash
source /opt/awscli-venv/bin/activate

# List versions of a deleted file
aws s3api list-object-versions --bucket hermes-vps-backups --prefix live/state.db \
  --endpoint-url https://s3.us-east-1.wasabisys.com

# Copy a specific version back
aws s3api get-object --bucket hermes-vps-backups --key live/state.db \
  --version-id <version-id> /root/.hermes/state.db \
  --endpoint-url https://s3.us-east-1.wasabisys.com
```

Or just sync from a different backup prefix:
```bash
# Try the previous sync
aws s3 sync s3://hermes-vps-backups/live-sync/ /root/.hermes/ \
  --endpoint-url https://s3.us-east-1.wasabisys.com

# Or try a daily backup
aws s3 cp s3://hermes-vps-backups/hermes-full-backup/hermes-full-backup-2026-07-05.tar.gz /tmp/
tar xzf /tmp/hermes-full-backup-2026-07-05.tar.gz -C /root/.hermes/
```

### Scenario E: "ai.itpropartner.com Is Down" -- AI Services Unavailable

**If the AI server (178.156.167.181) is offline:**

1. **Check if it's just a Docker issue:**
   ```bash
   ssh root@178.156.167.181 -i /root/.ssh/itpp-infra
   docker ps
   cd /opt/ai && docker compose up -d
   ```

2. **Check disk space** (this server is at 92%):
   ```bash
   df -h
   # If full, clean up old Docker images:
   docker system prune -a
   ```

3. **If the server is completely dead**, Hermes loses access to the deepseek-chat model. Sho'Nuff would need to:
   - Provision a new server
   - Restore LiteLLM PostgreSQL database
   - Restore Open WebUI data
   - Point DNS to the new server

### Scenario F: "The Internet Is Down at Home" -- Router Issues

**If the home internet (and therefore the MikroTik router) is unavailable:**

1. **Check if the router is reachable via WireGuard:**
   ```bash
   ping -c 3 10.77.0.2
   ```

2. **If unreachable**, check the VPN tunnel:
   ```bash
   systemctl status strongswan-starter
   systemctl status xl2tpd
   ```

3. **If VPN is down but Core is up**, the rest of your infrastructure still works -- only home-related services are affected.

4. **Restore router from last known good config** (see Section 9).

---

## 13. Cron Jobs -- Complete List

### On Core (152.53.192.33)

```
TZ=America/New_York

* * * * *                /root/.hermes/scripts/dre-mail-poller.py
0 * * * *                /root/.hermes/scripts/boys-mail-monitor.py collect 2>&1 | logger -t boys-mail-collect
0 19 * * *               /root/.hermes/scripts/boys-mail-monitor.py summary 2>&1 | logger -t boys-mail-summary
0 8 * * *                /root/shark-game/scraper/run.sh
*/15 * * * *             /root/.hermes/scripts/shark-draft-reminder.sh 2>&1 | logger -t shark-draft-reminder
```

**Hermes internal cron jobs** (managed by Hermes itself):
- Various ticker-based tasks defined in `/root/.hermes/cron/jobs.json`

### On app1-bu (5.161.114.8)

```
*/5 * * * *    /root/.hermes/scripts/hermes-standby-watchdog.sh
*/10 * * * *   /root/.hermes/scripts/hermes-standby-sync.sh
```

### On wphost02 (5.161.62.38)

```
0 3 * * *    /usr/local/bin/lynis-scan.sh
```

### To View Cron Jobs

```bash
crontab -l                     # Current user's crontab
cat /root/.hermes/cron/jobs.json  # Hermes-managed cron jobs
```

---

> **This manual supersedes** `shark-game-recovery-manual.md`.  
> For shark-game-specific details (scoring, push notifications, common player issues), see the legacy manual at `/root/.hermes/references/shark-game-recovery-manual.md`.  
> All infrastructure recovery procedures above take precedence.