hermes-recovery/references/restore-runbooks.md

# Standalone Restore Runbooks

## 1. Primary Host (Core) — Hermes
**Estimated time to complete:** 15 minutes
**Rollback plan:** Revert DNS to standby `app1-bu` (5.161.114.8). Wipe server and restart runbook.

### Prerequisites
- SSH key: `itpp-infra`
- AWS S3 access credentials (for Wasabi)
- netcup panel access

### Step 1: Provision server
Order a **netcup RS 2000 G12** (8C EPYC, 16GB RAM, 512GB NVMe) running Debian 13.

### Step 2: Base install
```bash
hostnamectl set-hostname core
apt update && apt upgrade -y
apt install -y fail2ban prometheus-node-exporter ufw docker.io docker-compose-v2
ufw allow ssh && ufw allow http && ufw allow https && ufw --force enable
useradd -m -G sudo ippadmin
mkdir -p /home/ippadmin/.ssh
echo "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAII4dxTH11aJkBqCY8lXl1kTfZ8yXWhTcthHnt1MtAuIE itpp-infra" >> /home/ippadmin/.ssh/authorized_keys
sed -i 's/^#PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config
systemctl restart sshd
```

### Step 3: Restore data
```bash
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
aws s3 cp s3://hermes-vps-backups/live/state.db /root/.hermes/state.db --endpoint-url https://s3.us-east-1.wasabisys.com
aws s3 cp s3://hermes-vps-backups/live/config.yaml /root/.hermes/config.yaml --endpoint-url https://s3.us-east-1.wasabisys.com
aws s3 cp s3://hermes-vps-backups/live/.env /root/.hermes/.env --endpoint-url https://s3.us-east-1.wasabisys.com
chmod 600 /root/.hermes/.env
aws s3 cp s3://hermes-vps-backups/live/config/Caddyfile /etc/caddy/Caddyfile --endpoint-url https://s3.us-east-1.wasabisys.com
aws s3 cp s3://hermes-vps-backups/core/latest/docker/ /root/docker/ --recursive --endpoint-url https://s3.us-east-1.wasabisys.com
```

### Step 4: Start services
```bash
for dir in /root/docker/*/; do
  (cd "$dir" && docker compose up -d)
done
systemctl restart caddy
hermes gateway run --replace
```

### Step 5: Verify
```bash
curl -s https://ops.itpropartner.com/api/health
curl -s https://sign.itpropartner.com
# Open Telegram and send /ping to Hermes bot; ensure it replies immediately.
```

---

## 2. AI Server (admin-ai) — LiteLLM + OpenWebUI + Ollama
**Estimated time to complete:** 2 hours
**Rollback plan:** Use external model providers in LiteLLM config or fallback to Core's local Ollama. Wipe server and retry.

### Prerequisites
- SSH key: `itpp-infra`
- AWS S3 access credentials
- Hetzner Cloud panel access

### Step 1: Provision server
Order a **Hetzner CPX41** running Debian 12 (or 13). 

### Step 2: Base install
```bash
hostnamectl set-hostname admin-ai
apt update && apt upgrade -y
curl -fsSL https://get.docker.com | bash
apt install -y ufw docker-compose-plugin
ufw allow ssh && ufw allow http && ufw allow https && ufw --force enable
```

### Step 3: Restore data
```bash
aws s3 cp s3://hermes-vps-backups/admin-ai/latest/opt/ /opt/ --recursive --endpoint-url https://s3.us-east-1.wasabisys.com
```

### Step 4: Start services
```bash
cd /opt/litellm && docker compose up -d
cd /opt/openwebui && docker compose up -d
cd /opt/qdrant && docker compose up -d
curl -fsSL https://ollama.com/install.sh | sh
ollama pull llama3.2:3b
# Update DNS to point admin-ai.itpropartner.com to the new IP address
```

### Step 5: Verify
```bash
# Wait for DNS propagation, then test completion API
curl https://admin-ai.itpropartner.com/v1/chat/completions \
  -H "Authorization: Bearer $(grep ADMIN_AI ~/.hermes/.env | cut -d= -f2)" \
  -d '{"model":"deepseek-chat","messages":[{"role":"user","content":"ping"}]}'
# Log into OpenWebUI via browser to ensure UI and database are intact
```

---

## 3. Legacy Web Host (wphost02) — WordPress + RunCloud
**Estimated time to complete:** 4 hours
**Rollback plan:** Delete the broken Hetzner VPS instance, revert DNS. Restore an older snapshot in RunCloud.

### Prerequisites
- RunCloud panel access
- Hetzner Cloud panel access
- Credentials for `email@redacted` Postfix relay

### Step 1: Provision server
Order a **Hetzner CPX21** (3C/4G/80GB) running Debian 12.

### Step 2: Base install
```bash
# Connect to RunCloud panel and add the new server IP.
# RunCloud will automatically provision nginx, MariaDB, and PHP.
# Deploy itpp-infra SSH key in the RunCloud panel for SSH access.
# In RunCloud firewall, whitelist Core IP: 152.53.192.33
```

### Step 3: Restore data
```bash
# Go to RunCloud panel > Web Applications.
# Restore the WordPress site from the RunCloud daily backup.
# Manually restore the database if the backup does not include it.
```

### Step 4: Start services
```bash
# Web services start automatically via RunCloud. Reconfigure Postfix relay:
postconf -e "relayhost = mail.germainebrown.com:2525"
postconf -e "smtp_sasl_auth_enable = yes"
echo "mail.germainebrown.com:2525 email@redacted:PASSWORD_HERE" > /etc/postfix/sasl_passwd
postmap /etc/postfix/sasl_passwd
systemctl restart postfix
postqueue -f
```

### Step 5: Verify
```bash
curl -I https://apex-vreg-live.com # Or target domain, check for 200 OK
# Verify email delivery by triggering a password reset or test email from WordPress.
```

---

## 4. UNMS (UISP) — Docker stack
**Estimated time to complete:** 4 hours
**Rollback plan:** `docker compose down -v`, clear the volume path, and re-pull the backup from an older date.

### Prerequisites
- SSH key: `itpp-infra`
- AWS S3 access credentials
- Hetzner Cloud panel access

### Step 1: Provision server
Order a **Hetzner CPX21** (3C/4G/80GB) running Debian 12.

### Step 2: Base install
```bash
hostnamectl set-hostname unms
apt update && apt upgrade -y
curl -fsSL https://get.docker.com | bash
apt install -y docker-compose-plugin ufw
ufw allow ssh && ufw allow 80 && ufw allow 443 && ufw allow 2055/udp && ufw --force enable
```

### Step 3: Restore data
```bash
mkdir -p /opt/unms
aws s3 cp s3://hermes-vps-backups/unms/latest/ /opt/unms/ --recursive --endpoint-url https://s3.us-east-1.wasabisys.com
```

### Step 4: Start services
```bash
cd /opt/unms
docker compose up -d
# Update DNS to point the UISP domain to the new IP address.
```

### Step 5: Verify
```bash
# Navigate to the UNMS/UISP web interface.
# Log in and confirm the device list populates and shows towers/routers as connected.
# Check container health:
docker ps | grep unms
```

---

## 5. UniFi — Controller
**Estimated time to complete:** 4 hours
**Rollback plan:** Factory reset UniFi via `dpkg-reconfigure unifi` and upload an older `.unf` backup file.

### Prerequisites
- A recent UniFi `.unf` backup file (from S3 or local download)
- Hetzner Cloud panel access

### Step 1: Provision server
Order a **Hetzner CPX21** (3C/4G/80GB) running Debian 12.

### Step 2: Base install
```bash
hostnamectl set-hostname unifi
apt update && apt upgrade -y
apt install -y ca-certificates apt-transport-https openjdk-17-jre-headless ufw
ufw allow ssh && ufw allow 8080 && ufw allow 8443 && ufw allow 3478/udp && ufw --force enable
```

### Step 3: Restore data
```bash
# Add Ubiquiti repo and install UniFi (native, no Docker)
echo 'deb https://www.ui.com/downloads/unifi/debian stable ubiquiti' | tee /etc/apt/sources.list.d/100-ubnt-unifi.list
wget -O /etc/apt/trusted.gpg.d/unifi-repo.gpg https://dl.ui.com/unifi/unifi-repo.gpg
apt update && apt install -y unifi
```

### Step 4: Start services
```bash
systemctl enable --now unifi
# Wait for the service to start (can take a few minutes)
# Access the web UI at https://<new-ip>:8443
# Select "Restore from a previous backup" and upload the .unf file
```

### Step 5: Verify
```bash
systemctl status unifi
# Log into the UniFi UI
# Confirm access points and switches show as "Online" (Adopted).
```

---

## 6. n8n — Docker
**Estimated time to complete:** 8 hours
**Rollback plan:** Stop Docker, delete `/opt/n8n`, sync an older backup snapshot from S3.

### Prerequisites
- SSH key: `itpp-infra`
- AWS S3 access credentials
- Hetzner Cloud panel access

### Step 1: Provision server
Order a **Hetzner CPX11** (2C/2G/40GB) running Debian 12 or 13.

### Step 2: Base install
```bash
hostnamectl set-hostname n8n
apt update && apt upgrade -y
curl -fsSL https://get.docker.com | bash
apt install -y docker-compose-plugin ufw
ufw allow ssh && ufw allow 80 && ufw allow 443 && ufw --force enable
```

### Step 3: Restore data
```bash
mkdir -p /opt/n8n
aws s3 cp s3://hermes-vps-backups/n8n/latest/ /opt/n8n/ --recursive --endpoint-url https://s3.us-east-1.wasabisys.com
```

### Step 4: Start services
```bash
cd /opt/n8n
docker compose up -d
# Update DNS to point the n8n domain to the new IP address.
```

### Step 5: Verify
```bash
# Access the n8n web interface.
# Check that all workflows are present.
# Execute a test run of a non-critical workflow to verify Postgres connectivity.
```

---

## 7. Docker Box — Compose stacks
**Estimated time to complete:** 8 hours
**Rollback plan:** Run `docker compose down`, remove volume data, download an older backup archive, and try starting again.

### Prerequisites
- SSH key: `itpp-infra`
- AWS S3 access credentials
- Hetzner Cloud panel access

### Step 1: Provision server
Order a **Hetzner CPX11** (2C/2G/40GB) running Debian 12 or 13.

### Step 2: Base install
```bash
hostnamectl set-hostname docker-box
apt update && apt upgrade -y
curl -fsSL https://get.docker.com | bash
apt install -y docker-compose-plugin ufw
ufw allow ssh && ufw allow 80 && ufw allow 443 && ufw allow 21115:21119/tcp && ufw allow 21116/udp && ufw --force enable
```

### Step 3: Restore data
```bash
mkdir -p /opt
aws s3 cp s3://hermes-vps-backups/docker-box/latest/ /opt/ --recursive --endpoint-url https://s3.us-east-1.wasabisys.com
```

### Step 4: Start services
```bash
# Start all 8 containers (NPM, RustDesk, Uptime Kuma, RunCloud, etc.)
for dir in /opt/*/; do
  if [ -f "$dir/docker-compose.yml" ]; then
    (cd "$dir" && docker compose up -d)
  fi
done
# Update DNS records for NPM, Uptime Kuma, and RustDesk relay to the new IP.
```

### Step 5: Verify
```bash
# Log into Uptime Kuma web UI and verify monitors are running.
# Access Nginx Proxy Manager (NPM) UI and check proxy hosts.
# Open a RustDesk client and ensure it connects to the self-hosted relay.
```