Initial resurrection kit — 81 scripts, 62 references, configs, systemd units, crons, Docker compose files, Caddy config, master README
This commit is contained in:
root
@@ -0,0 +1,324 @@
|
||||
# Standalone Restore Runbooks
|
||||
|
||||
## 1. Primary Host (Core) — Hermes
|
||||
**Estimated time to complete:** 15 minutes
|
||||
**Rollback plan:** Revert DNS to standby `app1-bu` (5.161.114.8). Wipe server and restart runbook.
|
||||
|
||||
### Prerequisites
|
||||
- SSH key: `itpp-infra`
|
||||
- AWS S3 access credentials (for Wasabi)
|
||||
- netcup panel access
|
||||
|
||||
### Step 1: Provision server
|
||||
Order a **netcup RS 2000 G12** (8C EPYC, 16GB RAM, 512GB NVMe) running Debian 13.
|
||||
|
||||
### Step 2: Base install
|
||||
```bash
|
||||
hostnamectl set-hostname core
|
||||
apt update && apt upgrade -y
|
||||
apt install -y fail2ban prometheus-node-exporter ufw docker.io docker-compose-v2
|
||||
ufw allow ssh && ufw allow http && ufw allow https && ufw --force enable
|
||||
useradd -m -G sudo ippadmin
|
||||
mkdir -p /home/ippadmin/.ssh
|
||||
echo "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAII4dxTH11aJkBqCY8lXl1kTfZ8yXWhTcthHnt1MtAuIE itpp-infra" >> /home/ippadmin/.ssh/authorized_keys
|
||||
sed -i 's/^#PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config
|
||||
systemctl restart sshd
|
||||
```
|
||||
|
||||
### Step 3: Restore data
|
||||
```bash
|
||||
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
|
||||
aws s3 cp s3://hermes-vps-backups/live/state.db /root/.hermes/state.db --endpoint-url https://s3.us-east-1.wasabisys.com
|
||||
aws s3 cp s3://hermes-vps-backups/live/config.yaml /root/.hermes/config.yaml --endpoint-url https://s3.us-east-1.wasabisys.com
|
||||
aws s3 cp s3://hermes-vps-backups/live/.env /root/.hermes/.env --endpoint-url https://s3.us-east-1.wasabisys.com
|
||||
chmod 600 /root/.hermes/.env
|
||||
aws s3 cp s3://hermes-vps-backups/live/config/Caddyfile /etc/caddy/Caddyfile --endpoint-url https://s3.us-east-1.wasabisys.com
|
||||
aws s3 cp s3://hermes-vps-backups/core/latest/docker/ /root/docker/ --recursive --endpoint-url https://s3.us-east-1.wasabisys.com
|
||||
```
|
||||
|
||||
### Step 4: Start services
|
||||
```bash
|
||||
for dir in /root/docker/*/; do
|
||||
(cd "$dir" && docker compose up -d)
|
||||
done
|
||||
systemctl restart caddy
|
||||
hermes gateway run --replace
|
||||
```
|
||||
|
||||
### Step 5: Verify
|
||||
```bash
|
||||
curl -s https://ops.itpropartner.com/api/health
|
||||
curl -s https://sign.itpropartner.com
|
||||
# Open Telegram and send /ping to Hermes bot; ensure it replies immediately.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. AI Server (admin-ai) — LiteLLM + OpenWebUI + Ollama
|
||||
**Estimated time to complete:** 2 hours
|
||||
**Rollback plan:** Use external model providers in LiteLLM config or fallback to Core's local Ollama. Wipe server and retry.
|
||||
|
||||
### Prerequisites
|
||||
- SSH key: `itpp-infra`
|
||||
- AWS S3 access credentials
|
||||
- Hetzner Cloud panel access
|
||||
|
||||
### Step 1: Provision server
|
||||
Order a **Hetzner CPX41** running Debian 12 (or 13).
|
||||
|
||||
### Step 2: Base install
|
||||
```bash
|
||||
hostnamectl set-hostname admin-ai
|
||||
apt update && apt upgrade -y
|
||||
curl -fsSL https://get.docker.com | bash
|
||||
apt install -y ufw docker-compose-plugin
|
||||
ufw allow ssh && ufw allow http && ufw allow https && ufw --force enable
|
||||
```
|
||||
|
||||
### Step 3: Restore data
|
||||
```bash
|
||||
aws s3 cp s3://hermes-vps-backups/admin-ai/latest/opt/ /opt/ --recursive --endpoint-url https://s3.us-east-1.wasabisys.com
|
||||
```
|
||||
|
||||
### Step 4: Start services
|
||||
```bash
|
||||
cd /opt/litellm && docker compose up -d
|
||||
cd /opt/openwebui && docker compose up -d
|
||||
cd /opt/qdrant && docker compose up -d
|
||||
curl -fsSL https://ollama.com/install.sh | sh
|
||||
ollama pull llama3.2:3b
|
||||
# Update DNS to point admin-ai.itpropartner.com to the new IP address
|
||||
```
|
||||
|
||||
### Step 5: Verify
|
||||
```bash
|
||||
# Wait for DNS propagation, then test completion API
|
||||
curl https://admin-ai.itpropartner.com/v1/chat/completions \
|
||||
-H "Authorization: Bearer $(grep ADMIN_AI ~/.hermes/.env | cut -d= -f2)" \
|
||||
-d '{"model":"deepseek-chat","messages":[{"role":"user","content":"ping"}]}'
|
||||
# Log into OpenWebUI via browser to ensure UI and database are intact
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Legacy Web Host (wphost02) — WordPress + RunCloud
|
||||
**Estimated time to complete:** 4 hours
|
||||
**Rollback plan:** Delete the broken Hetzner VPS instance, revert DNS. Restore an older snapshot in RunCloud.
|
||||
|
||||
### Prerequisites
|
||||
- RunCloud panel access
|
||||
- Hetzner Cloud panel access
|
||||
- Credentials for `email@redacted` Postfix relay
|
||||
|
||||
### Step 1: Provision server
|
||||
Order a **Hetzner CPX21** (3C/4G/80GB) running Debian 12.
|
||||
|
||||
### Step 2: Base install
|
||||
```bash
|
||||
# Connect to RunCloud panel and add the new server IP.
|
||||
# RunCloud will automatically provision nginx, MariaDB, and PHP.
|
||||
# Deploy itpp-infra SSH key in the RunCloud panel for SSH access.
|
||||
# In RunCloud firewall, whitelist Core IP: 152.53.192.33
|
||||
```
|
||||
|
||||
### Step 3: Restore data
|
||||
```bash
|
||||
# Go to RunCloud panel > Web Applications.
|
||||
# Restore the WordPress site from the RunCloud daily backup.
|
||||
# Manually restore the database if the backup does not include it.
|
||||
```
|
||||
|
||||
### Step 4: Start services
|
||||
```bash
|
||||
# Web services start automatically via RunCloud. Reconfigure Postfix relay:
|
||||
postconf -e "relayhost = mail.germainebrown.com:2525"
|
||||
postconf -e "smtp_sasl_auth_enable = yes"
|
||||
echo "mail.germainebrown.com:2525 email@redacted:PASSWORD_HERE" > /etc/postfix/sasl_passwd
|
||||
postmap /etc/postfix/sasl_passwd
|
||||
systemctl restart postfix
|
||||
postqueue -f
|
||||
```
|
||||
|
||||
### Step 5: Verify
|
||||
```bash
|
||||
curl -I https://apex-vreg-live.com # Or target domain, check for 200 OK
|
||||
# Verify email delivery by triggering a password reset or test email from WordPress.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. UNMS (UISP) — Docker stack
|
||||
**Estimated time to complete:** 4 hours
|
||||
**Rollback plan:** `docker compose down -v`, clear the volume path, and re-pull the backup from an older date.
|
||||
|
||||
### Prerequisites
|
||||
- SSH key: `itpp-infra`
|
||||
- AWS S3 access credentials
|
||||
- Hetzner Cloud panel access
|
||||
|
||||
### Step 1: Provision server
|
||||
Order a **Hetzner CPX21** (3C/4G/80GB) running Debian 12.
|
||||
|
||||
### Step 2: Base install
|
||||
```bash
|
||||
hostnamectl set-hostname unms
|
||||
apt update && apt upgrade -y
|
||||
curl -fsSL https://get.docker.com | bash
|
||||
apt install -y docker-compose-plugin ufw
|
||||
ufw allow ssh && ufw allow 80 && ufw allow 443 && ufw allow 2055/udp && ufw --force enable
|
||||
```
|
||||
|
||||
### Step 3: Restore data
|
||||
```bash
|
||||
mkdir -p /opt/unms
|
||||
aws s3 cp s3://hermes-vps-backups/unms/latest/ /opt/unms/ --recursive --endpoint-url https://s3.us-east-1.wasabisys.com
|
||||
```
|
||||
|
||||
### Step 4: Start services
|
||||
```bash
|
||||
cd /opt/unms
|
||||
docker compose up -d
|
||||
# Update DNS to point the UISP domain to the new IP address.
|
||||
```
|
||||
|
||||
### Step 5: Verify
|
||||
```bash
|
||||
# Navigate to the UNMS/UISP web interface.
|
||||
# Log in and confirm the device list populates and shows towers/routers as connected.
|
||||
# Check container health:
|
||||
docker ps | grep unms
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. UniFi — Controller
|
||||
**Estimated time to complete:** 4 hours
|
||||
**Rollback plan:** Factory reset UniFi via `dpkg-reconfigure unifi` and upload an older `.unf` backup file.
|
||||
|
||||
### Prerequisites
|
||||
- A recent UniFi `.unf` backup file (from S3 or local download)
|
||||
- Hetzner Cloud panel access
|
||||
|
||||
### Step 1: Provision server
|
||||
Order a **Hetzner CPX21** (3C/4G/80GB) running Debian 12.
|
||||
|
||||
### Step 2: Base install
|
||||
```bash
|
||||
hostnamectl set-hostname unifi
|
||||
apt update && apt upgrade -y
|
||||
apt install -y ca-certificates apt-transport-https openjdk-17-jre-headless ufw
|
||||
ufw allow ssh && ufw allow 8080 && ufw allow 8443 && ufw allow 3478/udp && ufw --force enable
|
||||
```
|
||||
|
||||
### Step 3: Restore data
|
||||
```bash
|
||||
# Add Ubiquiti repo and install UniFi (native, no Docker)
|
||||
echo 'deb https://www.ui.com/downloads/unifi/debian stable ubiquiti' | tee /etc/apt/sources.list.d/100-ubnt-unifi.list
|
||||
wget -O /etc/apt/trusted.gpg.d/unifi-repo.gpg https://dl.ui.com/unifi/unifi-repo.gpg
|
||||
apt update && apt install -y unifi
|
||||
```
|
||||
|
||||
### Step 4: Start services
|
||||
```bash
|
||||
systemctl enable --now unifi
|
||||
# Wait for the service to start (can take a few minutes)
|
||||
# Access the web UI at https://<new-ip>:8443
|
||||
# Select "Restore from a previous backup" and upload the .unf file
|
||||
```
|
||||
|
||||
### Step 5: Verify
|
||||
```bash
|
||||
systemctl status unifi
|
||||
# Log into the UniFi UI
|
||||
# Confirm access points and switches show as "Online" (Adopted).
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. n8n — Docker
|
||||
**Estimated time to complete:** 8 hours
|
||||
**Rollback plan:** Stop Docker, delete `/opt/n8n`, sync an older backup snapshot from S3.
|
||||
|
||||
### Prerequisites
|
||||
- SSH key: `itpp-infra`
|
||||
- AWS S3 access credentials
|
||||
- Hetzner Cloud panel access
|
||||
|
||||
### Step 1: Provision server
|
||||
Order a **Hetzner CPX11** (2C/2G/40GB) running Debian 12 or 13.
|
||||
|
||||
### Step 2: Base install
|
||||
```bash
|
||||
hostnamectl set-hostname n8n
|
||||
apt update && apt upgrade -y
|
||||
curl -fsSL https://get.docker.com | bash
|
||||
apt install -y docker-compose-plugin ufw
|
||||
ufw allow ssh && ufw allow 80 && ufw allow 443 && ufw --force enable
|
||||
```
|
||||
|
||||
### Step 3: Restore data
|
||||
```bash
|
||||
mkdir -p /opt/n8n
|
||||
aws s3 cp s3://hermes-vps-backups/n8n/latest/ /opt/n8n/ --recursive --endpoint-url https://s3.us-east-1.wasabisys.com
|
||||
```
|
||||
|
||||
### Step 4: Start services
|
||||
```bash
|
||||
cd /opt/n8n
|
||||
docker compose up -d
|
||||
# Update DNS to point the n8n domain to the new IP address.
|
||||
```
|
||||
|
||||
### Step 5: Verify
|
||||
```bash
|
||||
# Access the n8n web interface.
|
||||
# Check that all workflows are present.
|
||||
# Execute a test run of a non-critical workflow to verify Postgres connectivity.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. Docker Box — Compose stacks
|
||||
**Estimated time to complete:** 8 hours
|
||||
**Rollback plan:** Run `docker compose down`, remove volume data, download an older backup archive, and try starting again.
|
||||
|
||||
### Prerequisites
|
||||
- SSH key: `itpp-infra`
|
||||
- AWS S3 access credentials
|
||||
- Hetzner Cloud panel access
|
||||
|
||||
### Step 1: Provision server
|
||||
Order a **Hetzner CPX11** (2C/2G/40GB) running Debian 12 or 13.
|
||||
|
||||
### Step 2: Base install
|
||||
```bash
|
||||
hostnamectl set-hostname docker-box
|
||||
apt update && apt upgrade -y
|
||||
curl -fsSL https://get.docker.com | bash
|
||||
apt install -y docker-compose-plugin ufw
|
||||
ufw allow ssh && ufw allow 80 && ufw allow 443 && ufw allow 21115:21119/tcp && ufw allow 21116/udp && ufw --force enable
|
||||
```
|
||||
|
||||
### Step 3: Restore data
|
||||
```bash
|
||||
mkdir -p /opt
|
||||
aws s3 cp s3://hermes-vps-backups/docker-box/latest/ /opt/ --recursive --endpoint-url https://s3.us-east-1.wasabisys.com
|
||||
```
|
||||
|
||||
### Step 4: Start services
|
||||
```bash
|
||||
# Start all 8 containers (NPM, RustDesk, Uptime Kuma, RunCloud, etc.)
|
||||
for dir in /opt/*/; do
|
||||
if [ -f "$dir/docker-compose.yml" ]; then
|
||||
(cd "$dir" && docker compose up -d)
|
||||
fi
|
||||
done
|
||||
# Update DNS records for NPM, Uptime Kuma, and RustDesk relay to the new IP.
|
||||
```
|
||||
|
||||
### Step 5: Verify
|
||||
```bash
|
||||
# Log into Uptime Kuma web UI and verify monitors are running.
|
||||
# Access Nginx Proxy Manager (NPM) UI and check proxy hosts.
|
||||
# Open a RustDesk client and ensure it connects to the self-hosted relay.
|
||||
```
|
||||
Reference in New Issue
Block a user