hermes-recovery/references/restore-runbooks.md

Standalone Restore Runbooks

1. Primary Host (Core) — Hermes

Estimated time to complete: 15 minutes Rollback plan: Revert DNS to standby app1-bu (5.161.114.8). Wipe server and restart runbook.

Prerequisites

• SSH key: itpp-infra
• AWS S3 access credentials (for Wasabi)
• netcup panel access

Step 1: Provision server

Order a netcup RS 2000 G12 (8C EPYC, 16GB RAM, 512GB NVMe) running Debian 13.

Step 2: Base install

hostnamectl set-hostname core
apt update && apt upgrade -y
apt install -y fail2ban prometheus-node-exporter ufw docker.io docker-compose-v2
ufw allow ssh && ufw allow http && ufw allow https && ufw --force enable
useradd -m -G sudo ippadmin
mkdir -p /home/ippadmin/.ssh
echo "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAII4dxTH11aJkBqCY8lXl1kTfZ8yXWhTcthHnt1MtAuIE itpp-infra" >> /home/ippadmin/.ssh/authorized_keys
sed -i 's/^#PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config
systemctl restart sshd

Step 3: Restore data

curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
aws s3 cp s3://hermes-vps-backups/live/state.db /root/.hermes/state.db --endpoint-url https://s3.us-east-1.wasabisys.com
aws s3 cp s3://hermes-vps-backups/live/config.yaml /root/.hermes/config.yaml --endpoint-url https://s3.us-east-1.wasabisys.com
aws s3 cp s3://hermes-vps-backups/live/.env /root/.hermes/.env --endpoint-url https://s3.us-east-1.wasabisys.com
chmod 600 /root/.hermes/.env
aws s3 cp s3://hermes-vps-backups/live/config/Caddyfile /etc/caddy/Caddyfile --endpoint-url https://s3.us-east-1.wasabisys.com
aws s3 cp s3://hermes-vps-backups/core/latest/docker/ /root/docker/ --recursive --endpoint-url https://s3.us-east-1.wasabisys.com

Step 4: Start services

for dir in /root/docker/*/; do
  (cd "$dir" && docker compose up -d)
done
systemctl restart caddy
hermes gateway run --replace

Step 5: Verify

curl -s https://ops.itpropartner.com/api/health
curl -s https://sign.itpropartner.com
# Open Telegram and send /ping to Hermes bot; ensure it replies immediately.

---

2. AI Server (admin-ai) — LiteLLM + OpenWebUI + Ollama

Estimated time to complete: 2 hours Rollback plan: Use external model providers in LiteLLM config or fallback to Core's local Ollama. Wipe server and retry.

Prerequisites

• SSH key: itpp-infra
• AWS S3 access credentials
• Hetzner Cloud panel access

Step 1: Provision server

Order a Hetzner CPX41 running Debian 12 (or 13).

Step 2: Base install

hostnamectl set-hostname admin-ai
apt update && apt upgrade -y
curl -fsSL https://get.docker.com | bash
apt install -y ufw docker-compose-plugin
ufw allow ssh && ufw allow http && ufw allow https && ufw --force enable

Step 3: Restore data

aws s3 cp s3://hermes-vps-backups/admin-ai/latest/opt/ /opt/ --recursive --endpoint-url https://s3.us-east-1.wasabisys.com

Step 4: Start services

cd /opt/litellm && docker compose up -d
cd /opt/openwebui && docker compose up -d
cd /opt/qdrant && docker compose up -d
curl -fsSL https://ollama.com/install.sh | sh
ollama pull llama3.2:3b
# Update DNS to point admin-ai.itpropartner.com to the new IP address

Step 5: Verify

# Wait for DNS propagation, then test completion API
curl https://admin-ai.itpropartner.com/v1/chat/completions \
  -H "Authorization: Bearer $(grep ADMIN_AI ~/.hermes/.env | cut -d= -f2)" \
  -d '{"model":"deepseek-chat","messages":[{"role":"user","content":"ping"}]}'
# Log into OpenWebUI via browser to ensure UI and database are intact

---

3. Legacy Web Host (wphost02) — WordPress + RunCloud

Estimated time to complete: 4 hours Rollback plan: Delete the broken Hetzner VPS instance, revert DNS. Restore an older snapshot in RunCloud.

Prerequisites

• RunCloud panel access
• Hetzner Cloud panel access
• Credentials for email@redacted Postfix relay

Step 1: Provision server

Order a Hetzner CPX21 (3C/4G/80GB) running Debian 12.

Step 2: Base install

# Connect to RunCloud panel and add the new server IP.
# RunCloud will automatically provision nginx, MariaDB, and PHP.
# Deploy itpp-infra SSH key in the RunCloud panel for SSH access.
# In RunCloud firewall, whitelist Core IP: 152.53.192.33

Step 3: Restore data

# Go to RunCloud panel > Web Applications.
# Restore the WordPress site from the RunCloud daily backup.
# Manually restore the database if the backup does not include it.

Step 4: Start services

# Web services start automatically via RunCloud. Reconfigure Postfix relay:
postconf -e "relayhost = mail.germainebrown.com:2525"
postconf -e "smtp_sasl_auth_enable = yes"
echo "mail.germainebrown.com:2525 email@redacted:PASSWORD_HERE" > /etc/postfix/sasl_passwd
postmap /etc/postfix/sasl_passwd
systemctl restart postfix
postqueue -f

Step 5: Verify

curl -I https://apex-vreg-live.com # Or target domain, check for 200 OK
# Verify email delivery by triggering a password reset or test email from WordPress.

---

4. UNMS (UISP) — Docker stack

Estimated time to complete: 4 hours Rollback plan: docker compose down -v, clear the volume path, and re-pull the backup from an older date.

Prerequisites

• SSH key: itpp-infra
• AWS S3 access credentials
• Hetzner Cloud panel access

Step 1: Provision server

Order a Hetzner CPX21 (3C/4G/80GB) running Debian 12.

Step 2: Base install

hostnamectl set-hostname unms
apt update && apt upgrade -y
curl -fsSL https://get.docker.com | bash
apt install -y docker-compose-plugin ufw
ufw allow ssh && ufw allow 80 && ufw allow 443 && ufw allow 2055/udp && ufw --force enable

Step 3: Restore data

mkdir -p /opt/unms
aws s3 cp s3://hermes-vps-backups/unms/latest/ /opt/unms/ --recursive --endpoint-url https://s3.us-east-1.wasabisys.com

Step 4: Start services

cd /opt/unms
docker compose up -d
# Update DNS to point the UISP domain to the new IP address.

Step 5: Verify

# Navigate to the UNMS/UISP web interface.
# Log in and confirm the device list populates and shows towers/routers as connected.
# Check container health:
docker ps | grep unms

---

5. UniFi — Controller

Estimated time to complete: 4 hours Rollback plan: Factory reset UniFi via dpkg-reconfigure unifi and upload an older .unf backup file.

Prerequisites

• A recent UniFi .unf backup file (from S3 or local download)
• Hetzner Cloud panel access

Step 1: Provision server

Order a Hetzner CPX21 (3C/4G/80GB) running Debian 12.

Step 2: Base install

hostnamectl set-hostname unifi
apt update && apt upgrade -y
apt install -y ca-certificates apt-transport-https openjdk-17-jre-headless ufw
ufw allow ssh && ufw allow 8080 && ufw allow 8443 && ufw allow 3478/udp && ufw --force enable

Step 3: Restore data

# Add Ubiquiti repo and install UniFi (native, no Docker)
echo 'deb https://www.ui.com/downloads/unifi/debian stable ubiquiti' | tee /etc/apt/sources.list.d/100-ubnt-unifi.list
wget -O /etc/apt/trusted.gpg.d/unifi-repo.gpg https://dl.ui.com/unifi/unifi-repo.gpg
apt update && apt install -y unifi

Step 4: Start services

systemctl enable --now unifi
# Wait for the service to start (can take a few minutes)
# Access the web UI at https://<new-ip>:8443
# Select "Restore from a previous backup" and upload the .unf file

Step 5: Verify

systemctl status unifi
# Log into the UniFi UI
# Confirm access points and switches show as "Online" (Adopted).

---

6. n8n — Docker

Estimated time to complete: 8 hours Rollback plan: Stop Docker, delete /opt/n8n, sync an older backup snapshot from S3.

Prerequisites

• SSH key: itpp-infra
• AWS S3 access credentials
• Hetzner Cloud panel access

Step 1: Provision server

Order a Hetzner CPX11 (2C/2G/40GB) running Debian 12 or 13.

Step 2: Base install

hostnamectl set-hostname n8n
apt update && apt upgrade -y
curl -fsSL https://get.docker.com | bash
apt install -y docker-compose-plugin ufw
ufw allow ssh && ufw allow 80 && ufw allow 443 && ufw --force enable

Step 3: Restore data

mkdir -p /opt/n8n
aws s3 cp s3://hermes-vps-backups/n8n/latest/ /opt/n8n/ --recursive --endpoint-url https://s3.us-east-1.wasabisys.com

Step 4: Start services

cd /opt/n8n
docker compose up -d
# Update DNS to point the n8n domain to the new IP address.

Step 5: Verify

# Access the n8n web interface.
# Check that all workflows are present.
# Execute a test run of a non-critical workflow to verify Postgres connectivity.

---

7. Docker Box — Compose stacks

Estimated time to complete: 8 hours Rollback plan: Run docker compose down, remove volume data, download an older backup archive, and try starting again.

Prerequisites

• SSH key: itpp-infra
• AWS S3 access credentials
• Hetzner Cloud panel access

Step 1: Provision server

Order a Hetzner CPX11 (2C/2G/40GB) running Debian 12 or 13.

Step 2: Base install

hostnamectl set-hostname docker-box
apt update && apt upgrade -y
curl -fsSL https://get.docker.com | bash
apt install -y docker-compose-plugin ufw
ufw allow ssh && ufw allow 80 && ufw allow 443 && ufw allow 21115:21119/tcp && ufw allow 21116/udp && ufw --force enable

Step 3: Restore data

mkdir -p /opt
aws s3 cp s3://hermes-vps-backups/docker-box/latest/ /opt/ --recursive --endpoint-url https://s3.us-east-1.wasabisys.com

Step 4: Start services

# Start all 8 containers (NPM, RustDesk, Uptime Kuma, RunCloud, etc.)
for dir in /opt/*/; do
  if [ -f "$dir/docker-compose.yml" ]; then
    (cd "$dir" && docker compose up -d)
  fi
done
# Update DNS records for NPM, Uptime Kuma, and RustDesk relay to the new IP.

Step 5: Verify

# Log into Uptime Kuma web UI and verify monitors are running.
# Access Nginx Proxy Manager (NPM) UI and check proxy hosts.
# Open a RustDesk client and ensure it connects to the self-hosted relay.