ippadmin/itpp-infrastructure
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

DR docs audit fix (2026-08-08): fix stale dates, path errors, add RPO/RTO, unbacked services, restore testing, rollback procedure, S3 access instructions

Browse Source
This commit is contained in:
root
2026-08-08 17:25:27 -04:00
parent a522d11c35
commit c4c6609a81
3 changed files with 225 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backup-plan.md
+49 -5
View File
@@ -1,9 +1,10 @@
# ITPP Backup Plan # ITPP Backup Plan
> **Last updated:** 2026-07-28 > **Last updated:** 2026-08-08
> **Scope:** All ITPP infrastructure backups — Hermes core, application servers, routers, and external services. > **Scope:** All ITPP infrastructure backups — Hermes core, application servers, routers, and external services.
> **Storage:** Wasabi S3 (us-east-1) via `--endpoint-url https://s3.us-east-1.wasabisys.com` > **Storage:** Wasabi S3 (us-east-1) via `--endpoint-url https://s3.us-east-1.wasabisys.com`
> **Cron backend:** Hybrid — system crontab + Hermes cron jobs > **Cron backend:** Hybrid — system crontab + Hermes cron jobs (both on Core)
> **Verification audit:** [DR issue log](/root/.hermes/references/dr-issue-log.md)
--- ---
@@ -53,7 +54,7 @@
| MySQL (all DBs) | `app3-backup.sh` — mysqldump | `s3://hermes-vps-backups/app3/mysql/` | 3:00 AM | 2026-07-28 | | MySQL (all DBs) | `app3-backup.sh` — mysqldump | `s3://hermes-vps-backups/app3/mysql/` | 3:00 AM | 2026-07-28 |
| WordPress Files | `app3-backup.sh` — wp-content tar.gz | `s3://hermes-vps-backups/app3/wordpress/` | 3:00 AM | 2026-07-28 | | WordPress Files | `app3-backup.sh` — wp-content tar.gz | `s3://hermes-vps-backups/app3/wordpress/` | 3:00 AM | 2026-07-28 |
| Nginx Configs | `app3-backup.sh` — sites-enabled + config | `s3://hermes-vps-backups/app3/config/` | 3:00 AM | 2026-07-28 | | Nginx Configs | `app3-backup.sh` — sites-enabled + config | `s3://hermes-vps-backups/app3/config/` | 3:00 AM | 2026-07-28 |
| WordPress Snapshots | `/opt/backup-restore/snapshot.sh` — per-site tar.gz | `/opt/backup-restore/snapshots/` (local, 30-day retention) | 6 AM / 6 PM | 2026-07-28 13:00 | | WordPress Snapshots | `/opt/backup-restore/snapshot.sh` — per-site tar.gz | `/opt/backup-restore/snapshots/` (local, 30-day retention) | 1 AM / 1 PM | 2026-08-08 |
### wphost02 (5.161.62.38) — Hetzner CPX21 ### wphost02 (5.161.62.38) — Hetzner CPX21
@@ -82,7 +83,7 @@
| Every 15 min | Hermes session state | Hermes cron | `hermes-live-sync` | | Every 15 min | Hermes session state | Hermes cron | `hermes-live-sync` |
| 1:00 AM | Full Hermes backup | crontab | `hermes-backup.sh` | | 1:00 AM | Full Hermes backup | crontab | `hermes-backup.sh` |
| 1:30 AM | Grafana, Uptime Kuma, Docker volumes, Prometheus | crontab | `core-services-backup.sh` | | 1:30 AM | Grafana, Uptime Kuma, Docker volumes, Prometheus | crontab | `core-services-backup.sh` |
| 2:00 AM | Open WebUI, n8n, MCP configs (App1) | crontab | `app1-backup.sh` | | 2:00 AM | Open WebUI, n8n, MCP configs (App1) + **UniFi sync** (App2) | crontab | `app1-backup.sh`, `unifi-backup-sync.sh` |
| 2:30 AM | Vaultwarden (App1) + Traccar (App2) | Hermes cron / crontab | `vaultwarden-backup.sh`, `app2-backup.sh` | | 2:30 AM | Vaultwarden (App1) + Traccar (App2) | Hermes cron / crontab | `vaultwarden-backup.sh`, `app2-backup.sh` |
| 3:00 AM | /root essentials + App3 (CloudPanel, MySQL, WP, Nginx) | crontab | `root-essentials-backup.sh`, `app3-backup.sh` | | 3:00 AM | /root essentials + App3 (CloudPanel, MySQL, WP, Nginx) | crontab | `root-essentials-backup.sh`, `app3-backup.sh` |
| 3:30 AM | LiteLLM (App1) | Hermes cron | `litellm-backup.sh` | | 3:30 AM | LiteLLM (App1) | Hermes cron | `litellm-backup.sh` |
@@ -91,7 +92,7 @@
| 4:15 AM | Twenty CRM (App1) | Hermes cron | `twenty-backup.sh` | | 4:15 AM | Twenty CRM (App1) | Hermes cron | `twenty-backup.sh` |
| 5:00 AM | wphost02 WordPress (SSH) | crontab | `wphost02-backup` | | 5:00 AM | wphost02 WordPress (SSH) | crontab | `wphost02-backup` |
| 6:00 AM | MikroTik CCR + UNMS sync | Hermes cron | `run-wisp-backup.sh`, `unms-backup-sync.sh` | | 6:00 AM | MikroTik CCR + UNMS sync | Hermes cron | `run-wisp-backup.sh`, `unms-backup-sync.sh` |
| 6 AM / 6 PM | WordPress per-site snapshots (App3, local) | App3 crontab | `/opt/backup-restore/snapshot.sh` | | 1 AM / 1 PM | WordPress per-site snapshots (App3, local) | App3 crontab | `/opt/backup-restore/snapshot.sh` |
| 7:00 AM | Hudu (App2) | Hermes cron | `hudu-backup.sh` | | 7:00 AM | Hudu (App2) | Hermes cron | `hudu-backup.sh` |
| 8:00 AM | Gitea (App2) | Hermes cron | `gitea-backup.sh` | | 8:00 AM | Gitea (App2) | Hermes cron | `gitea-backup.sh` |
| Mon 5:00 AM | Hetzner weekly snapshots | Hermes cron | `snapshot-hetzner.py` | | Mon 5:00 AM | Hetzner weekly snapshots | Hermes cron | `snapshot-hetzner.py` |
@@ -189,6 +190,49 @@ mikrotik-ccr-backups/
--- ---
## Recovery Objectives (RPO / RTO)
| Tier | Services | RPO | RTO | Notes |
|---|---|---|---|---|
| **Critical** | Hermes Agent, Gitea, Traccar, UISP | ≤ 1 hour | ≤ 4 hours | Live sync + daily backups; restore from S3 then replay live-sync |
| **High** | LiteLLM, n8n, Open WebUI, Vaultwarden, Twenty CRM | 24 hours | ≤ 8 hours | Daily backups; restore from previous night's dump |
| **Medium** | Hudu, UniFi, Komodo, DocuSeal, App3 WP sites | 24 hours | ≤ 24 hours | Daily backups only; acceptable overnight gap |
| **Low** | Grafana, Uptime Kuma, Prometheus, MikroTik CCR | 24 hours | ≤ 48 hours | Monitoring data is nice-to-have, not blocking |
| **Unbacked** | Dawarich, RAGFlow, Technitium DNS | N/A | N/A | No backup exists — see Unbacked Services below |
## Restore Testing Cadence
**Quarterly:** Pick one random backup per tier, restore to a staging location, verify integrity.
**After any major infra change:** Test the affected service's restore path.
**Annual:** Full DR simulation — restore all Critical + High tier services to staging from S3.
## Stale S3 Paths — Cleanup Queue
These paths contain data from services that migrated off Core (Jul 28, 2026) or were removed:
| Path | Status | Action |
|---|---|---|
| `core/vaultwarden/` | Stale 11 days | **Safe to delete** — Vaultwarden migrated to App1; new backups at `app1/vaultwarden/` |
| `core/twenty/` | Stale 11 days | **Safe to delete** — Twenty CRM migrated to App1; new backups at `app1/twenty/` |
| `core/searxng/` | Stale 11 days | **Safe to delete** — SearXNG removed; replaced by Super Search |
| `core/komodo/` | Stale 11 days | **Safe to delete** — Komodo migrated to App1; new backups at `app1/komodo/` |
| `caddy/` | Unused | **Safe to delete** — Never populated |
| `snapshots/` | Unused | **Safe to delete** — Never populated |
## Unbacked Services
These services are running in production with **zero backup coverage**:
| Service | Server | Data Path | Risk |
|---|---|---|---|
| **Dawarich** | app2 | Docker: `dawarich_db`, `dawarich_app` (4 containers) | Location history lost on disk failure |
| **RAGFlow** | app2 | Docker: `ragflow-cpu-1` | RAG pipeline state lost |
| **Technitium DNS** | app2 | `/root/docker/technitium/data` (bind mount) | All DNS zones lost on disk failure; zone files must be recreated manually |
> **Action needed:** Create backup scripts for all three. For Technitium, the data directory can be tar'd and sent to `s3://hermes-vps-backups/app2/technitium/`. For Dawarich and RAGFlow, determine database engines before designing backup strategy.
---
## Migration History (Jul 28, 2026) ## Migration History (Jul 28, 2026)
All the following services were migrated from Core to App1 in a single session: All the following services were migrated from Core to App1 in a single session:
docs/app2-recovery-runbook.md
+172
View File
@@ -0,0 +1,172 @@
# app2 Recovery Runbook
**Server:** app2 (netcup RS 4000, 8 vCPU/16 GB/512 GB, 152.53.39.202)
**Status:** Production — no warm standby
**Last Updated:** 2026-08-08
---
## 1. Services Hosted (Impact if Down)
| Service | Domain | Criticality | Impact |
|---|---|---|---|
| Gitea | git.itpropartner.com | **High** | All source code repos unreachable. No git push/pull. |
| UISP (UNMS) | unms.forefrontwireless.com | **High** | WISP CCR tower backups, network management offline. |
| Traccar | fleettracker360.com | **High** | Client-facing GPS tracking product down. |
| Technitium DNS | dns1.itpropartner.com | Medium | Authoritative DNS for internal zones. All ITPP servers use Tailscale MagicDNS (100.100.100.100) for resolution, so internal DNS is NOT affected. Only external clients querying zones hosted on Technitium would be impacted. ⚠ No backup. |
| Hudu | hudu.itpropartner.com | Medium | IT documentation unavailable. Important but not blocking. |
| UniFi Controller | unifi.itpropartner.com | Medium | Wi-Fi management offline. APs continue operating in standalone mode. |
| Dawarich | timeline.iamgmb.com | Low | Personal location tracking. ⚠ No backup. |
| RAGFlow | ragflow.itpropartner.com | Low | RAG pipeline. ⚠ No backup. |
---
## 2. Pre-Flight Checks (Before Recovery)
Before assuming app2 is truly down, verify from Core:
```bash
# 1. Ping check
ping -c 3 -W 2 152.53.39.202
# 2. SSH check (with timeout to avoid hanging)
ssh -i /root/.ssh/itpp-infra -o ConnectTimeout=10 root@152.53.39.202 'uptime'
# 3. Docker health
ssh -i /root/.ssh/itpp-infra -o ConnectTimeout=10 root@152.53.39.202 'docker ps --format "{{.Names}} {{.Status}}" | grep -v "Up"'
```
---
## 3. Recovery Scenarios
### Scenario A: app2 is reachable but Docker services are down
```bash
# SSH in, check what happened
ssh -i /root/.ssh/itpp-infra -o ConnectTimeout=10 root@152.53.39.202
# Check disk space (common cause)
df -h
# Check Docker status
systemctl status docker
docker ps -a
# Restart critical services first
docker restart gitea hudu-app-1 traccar
docker restart technitium
```
### Scenario B: app2 is completely unreachable (server crash/hung)
1. **Log into netcup CCP** (Server Control Panel) at https://www.servercontrolpanel.de
2. **Check server status** — if hung, send ACPI shutdown + cold boot
3. **If boot fails:** Request KVM console from netcup support (or use integrated KVM if available)
4. **Boot into rescue mode if needed**, check filesystem:
```bash
fsck -f /dev/vda4
mount /dev/vda4 /mnt
# Check logs
cat /mnt/var/log/syslog | tail -100
```
### Scenario C: Complete server failure (hardware, unrecoverable)
1. **Order new RS 4000 from netcup** — provision with same specs (8 vCPU, 16 GB RAM, 512 GB SSD). Netcup provisioning typically takes **2-4 hours** for existing customers.
2. **Access S3 backups** to download restore data:
```bash
# Credentials are on Core at /root/.aws/credentials (wasabi profile)
# Or use the awscli venv:
source /opt/awscli-venv/bin/activate
# List available backups
aws s3 ls --endpoint-url https://s3.us-east-1.wasabisys.com s3://hermes-vps-backups/app2/
aws s3 ls --endpoint-url https://s3.us-east-1.wasabisys.com s3://hermes-vps-backups/gitea/
aws s3 ls --endpoint-url https://s3.us-east-1.wasabisys.com s3://hermes-vps-backups/hudu/
aws s3 ls --endpoint-url https://s3.us-east-1.wasabisys.com s3://hermes-vps-backups/unms/
aws s3 ls --endpoint-url https://s3.us-east-1.wasabisys.com s3://hermes-vps-backups/unifi/
# Download latest backup for each service
aws s3 sync --endpoint-url https://s3.us-east-1.wasabisys.com \
s3://hermes-vps-backups/app2/ ./restore/app2/
```
3. **Re-deploy Docker services** using compose files from restored data
4. **Restore databases** from S3 dumps
5. **Update DNS** for app2's new IP (if netcup assigns a different one)
6. **Reinstall Tailscale** and re-approve in admin console
7. **Note:** Netcup typically assigns IPs from the same subnet on re-provision, but this is not guaranteed.
---
## 4. Service-Specific Recovery
### Gitea
- Data: Docker volume at `/var/lib/docker/volumes/gitea_data`
- DB: **SQLite** (`gitea.db`) — file-based, no separate database container. Backup copies the SQLite file directly.
- Restore: `docker restart gitea` is usually sufficient; if data is corrupted, restore `gitea.db` from S3 and restart.
- Backup exists at `s3://hermes-vps-backups/gitea/daily/`
### UISP (UNMS)
- Stack: 9 containers (postgres, siridb, rabbitmq, fluentd, nginx, netflow, api, device-ws x8)
- Data: PostgreSQL at `unms-postgres`, SiridB at `unms-siridb`
- Restore: `cd /root/unms && docker compose up -d`
- Backup: `s3://hermes-vps-backups/unms/`
### Traccar
- Data: H2 database (embedded, no separate DB container)
- Config: XML at `/opt/traccar/conf/traccar.xml`
- Restore: `docker restart traccar`
- Backup: `s3://hermes-vps-backups/app2/traccar/`
### Technitium DNS
- Data: Docker bind mount at `/root/docker/technitium/data` → `/etc/dns`
- **⚠ No S3 backup** — zones exist only on disk. On full server loss, zones must be recreated manually.
- Restore: `docker restart technitium` (zones auto-load from volume on restart)
- Mitigation: Export zone files manually and commit to git for DR coverage until automated backup is implemented.
### Dawarich
- Stack: 4 containers (`dawarich_app`, `dawarich_sidekiq`, `dawarich_db`, `dawarich_redis`)
- **⚠ No backup** — location history data has zero DR coverage.
- Database: PostgreSQL (container `dawarich_db`). To create a backup: `docker exec dawarich_db pg_dump -U postgres dawarich > dawarich.sql`
- Restore: `cd /root/dawarich && docker compose up -d`
### RAGFlow
- Stack: 1 container (`docker-ragflow-cpu-1`)
- **⚠ No backup** — RAG pipeline state has zero DR coverage.
- Data: Docker volume; database engine TBD (needs inspection)
- Restore: `cd /root/ragflow && docker compose up -d`
### Hudu
- Stack: app + db + redis + worker
- Data: PostgreSQL at `hudu-db-1`
- Restore: `cd /root/hudu && docker compose up -d`
- Backup: `s3://hermes-vps-backups/hudu/`
---
## 5. Rollback Procedure
If a recovery attempt makes things worse (wrong backup restored, config mismatch, cascading failures):
1. **Stop affected containers:** `docker stop <container>`
2. **Identify the last known-good backup** from S3 timestamps
3. **Restore from the previous day's backup** (never overwrite the good backup while troubleshooting)
4. **Start containers one at a time** — verify each before starting the next
5. **If services still fail:** Do NOT attempt additional restores. Escalate to Germaine and document what was tried.
> **Golden rule:** The backup you're about to overwrite is your safety net. Copy it aside before restoring over it.
---
## 6. Verification Checklist (Post-Recovery)
- [ ] SSH to app2 works
- [ ] All Docker containers show "Up" in `docker ps`
- [ ] `git clone git.itpropartner.com/ippadmin/itpp-infrastructure.git` succeeds
- [ ] `https://git.itpropartner.com` loads in browser
- [ ] `https://hudu.itpropartner.com` loads in browser
- [ ] `https://unms.forefrontwireless.com` loads in browser
- [ ] `https://unifi.itpropartner.com` loads in browser
- [ ] `https://fleettracker360.com` loads in browser
- [ ] `dig @152.53.39.202 itpropartner.com` returns authoritative answer (DNS)
docs/backup-restore/ARCHITECTURE.md
+4 -2
View File
@@ -113,9 +113,11 @@ Browser shows success toast → Restore History updates
### 3. Snapshot Storage (`/opt/backup-restore/snapshots/`) ### 3. Snapshot Storage (`/opt/backup-restore/snapshots/`)
- Structure: `/<domain>/<YYYY-MM-DD_HHMMSS>/` - Structure: `/<domain>/<YYYY-MM-DD_HHMMSS>/`
- 9 WordPress domains, 10 snapshots each (10 days retention shown) - 9 WordPress domains, 10 snapshots each (10 days shown in UI)
- Retention: **30 days** — `snapshot.sh` auto-deletes snapshots older than 30 days via cron
- Average snapshot size: 16MB files + 74KB database - Average snapshot size: 16MB files + 74KB database
- Total: ~1.4GB for full snapshot set - Total: ~1.4GB for full snapshot set
- **⚠ Local only** — not synced to S3. If app3 fails, all local snapshots are lost. Daily 3 AM S3 backup (`app3-backup.sh`) provides coarser off-site coverage.
### 4. Restore Log (`/opt/backup-restore/logs/restore.log`) ### 4. Restore Log (`/opt/backup-restore/logs/restore.log`)
- Pipe-delimited format: `timestamp|domain|snapshot_id|status` - Pipe-delimited format: `timestamp|domain|snapshot_id|status`
@@ -155,4 +157,4 @@ All served by CloudPanel on app3, backed up by this system:
3. **Tar + mysqldump over rsync:** Snapshots are point-in-time archives, not incremental backups. Each snapshot is self-contained (files.tar.gz + database.sql). Restore is a single operation with no dependency chain. 3. **Tar + mysqldump over rsync:** Snapshots are point-in-time archives, not incremental backups. Each snapshot is self-contained (files.tar.gz + database.sql). Restore is a single operation with no dependency chain.
4. **No auth on backup API:** The endpoints have no authentication. Access is controlled by Caddy routing — only requests through my.itpropartner.com reach the app. Internal network only. 4. **No auth on backup API:** The endpoints have no authentication. The UI at `my.itpropartner.com/backups/` is publicly accessible through Core's Caddy. Access control relies on obscurity (the domain is not widely known) and Caddy's TLS termination. For production use, consider adding IP whitelisting or Caddy basic auth.