ippadmin/itpp-infrastructure
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

DR docs audit fix (2026-08-08): fix stale dates, path errors, add RPO/RTO, unbacked services, restore testing, rollback procedure, S3 access instructions

Browse Source
This commit is contained in:
root
2026-08-08 17:25:27 -04:00
parent a522d11c35
commit c4c6609a81
3 changed files with 225 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backup-plan.md
+49 -5
View File
@@ -1,9 +1,10 @@
# ITPP Backup Plan
> **Last updated:** 2026-07-28
> **Last updated:** 2026-08-08
> **Scope:** All ITPP infrastructure backups — Hermes core, application servers, routers, and external services.
> **Storage:** Wasabi S3 (us-east-1) via `--endpoint-url https://s3.us-east-1.wasabisys.com`
> **Cron backend:** Hybrid — system crontab + Hermes cron jobs
> **Cron backend:** Hybrid — system crontab + Hermes cron jobs (both on Core)
> **Verification audit:** [DR issue log](/root/.hermes/references/dr-issue-log.md)
---
@@ -53,7 +54,7 @@
| MySQL (all DBs) | `app3-backup.sh` — mysqldump | `s3://hermes-vps-backups/app3/mysql/` | 3:00 AM | 2026-07-28 |
| WordPress Files | `app3-backup.sh` — wp-content tar.gz | `s3://hermes-vps-backups/app3/wordpress/` | 3:00 AM | 2026-07-28 |
| Nginx Configs | `app3-backup.sh` — sites-enabled + config | `s3://hermes-vps-backups/app3/config/` | 3:00 AM | 2026-07-28 |
| WordPress Snapshots | `/opt/backup-restore/snapshot.sh` — per-site tar.gz | `/opt/backup-restore/snapshots/` (local, 30-day retention) | 6 AM / 6 PM | 2026-07-28 13:00 |
| WordPress Snapshots | `/opt/backup-restore/snapshot.sh` — per-site tar.gz | `/opt/backup-restore/snapshots/` (local, 30-day retention) | 1 AM / 1 PM | 2026-08-08 |
### wphost02 (5.161.62.38) — Hetzner CPX21
@@ -82,7 +83,7 @@
| Every 15 min | Hermes session state | Hermes cron | `hermes-live-sync` |
| 1:00 AM | Full Hermes backup | crontab | `hermes-backup.sh` |
| 1:30 AM | Grafana, Uptime Kuma, Docker volumes, Prometheus | crontab | `core-services-backup.sh` |
| 2:00 AM | Open WebUI, n8n, MCP configs (App1) | crontab | `app1-backup.sh` |
| 2:00 AM | Open WebUI, n8n, MCP configs (App1) + **UniFi sync** (App2) | crontab | `app1-backup.sh`, `unifi-backup-sync.sh` |
| 2:30 AM | Vaultwarden (App1) + Traccar (App2) | Hermes cron / crontab | `vaultwarden-backup.sh`, `app2-backup.sh` |
| 3:00 AM | /root essentials + App3 (CloudPanel, MySQL, WP, Nginx) | crontab | `root-essentials-backup.sh`, `app3-backup.sh` |
| 3:30 AM | LiteLLM (App1) | Hermes cron | `litellm-backup.sh` |
@@ -91,7 +92,7 @@
| 4:15 AM | Twenty CRM (App1) | Hermes cron | `twenty-backup.sh` |
| 5:00 AM | wphost02 WordPress (SSH) | crontab | `wphost02-backup` |
| 6:00 AM | MikroTik CCR + UNMS sync | Hermes cron | `run-wisp-backup.sh`, `unms-backup-sync.sh` |
| 6 AM / 6 PM | WordPress per-site snapshots (App3, local) | App3 crontab | `/opt/backup-restore/snapshot.sh` |
| 1 AM / 1 PM | WordPress per-site snapshots (App3, local) | App3 crontab | `/opt/backup-restore/snapshot.sh` |
| 7:00 AM | Hudu (App2) | Hermes cron | `hudu-backup.sh` |
| 8:00 AM | Gitea (App2) | Hermes cron | `gitea-backup.sh` |
| Mon 5:00 AM | Hetzner weekly snapshots | Hermes cron | `snapshot-hetzner.py` |
@@ -189,6 +190,49 @@ mikrotik-ccr-backups/
---
## Recovery Objectives (RPO / RTO)
| Tier | Services | RPO | RTO | Notes |
|---|---|---|---|---|
| **Critical** | Hermes Agent, Gitea, Traccar, UISP | ≤ 1 hour | ≤ 4 hours | Live sync + daily backups; restore from S3 then replay live-sync |
| **High** | LiteLLM, n8n, Open WebUI, Vaultwarden, Twenty CRM | 24 hours | ≤ 8 hours | Daily backups; restore from previous night's dump |
| **Medium** | Hudu, UniFi, Komodo, DocuSeal, App3 WP sites | 24 hours | ≤ 24 hours | Daily backups only; acceptable overnight gap |
| **Low** | Grafana, Uptime Kuma, Prometheus, MikroTik CCR | 24 hours | ≤ 48 hours | Monitoring data is nice-to-have, not blocking |
| **Unbacked** | Dawarich, RAGFlow, Technitium DNS | N/A | N/A | No backup exists — see Unbacked Services below |
## Restore Testing Cadence
**Quarterly:** Pick one random backup per tier, restore to a staging location, verify integrity.
**After any major infra change:** Test the affected service's restore path.
**Annual:** Full DR simulation — restore all Critical + High tier services to staging from S3.
## Stale S3 Paths — Cleanup Queue
These paths contain data from services that migrated off Core (Jul 28, 2026) or were removed:
| Path | Status | Action |
|---|---|---|
| `core/vaultwarden/` | Stale 11 days | **Safe to delete** — Vaultwarden migrated to App1; new backups at `app1/vaultwarden/` |
| `core/twenty/` | Stale 11 days | **Safe to delete** — Twenty CRM migrated to App1; new backups at `app1/twenty/` |
| `core/searxng/` | Stale 11 days | **Safe to delete** — SearXNG removed; replaced by Super Search |
| `core/komodo/` | Stale 11 days | **Safe to delete** — Komodo migrated to App1; new backups at `app1/komodo/` |
| `caddy/` | Unused | **Safe to delete** — Never populated |
| `snapshots/` | Unused | **Safe to delete** — Never populated |
## Unbacked Services
These services are running in production with **zero backup coverage**:
| Service | Server | Data Path | Risk |
|---|---|---|---|
| **Dawarich** | app2 | Docker: `dawarich_db`, `dawarich_app` (4 containers) | Location history lost on disk failure |
| **RAGFlow** | app2 | Docker: `ragflow-cpu-1` | RAG pipeline state lost |
| **Technitium DNS** | app2 | `/root/docker/technitium/data` (bind mount) | All DNS zones lost on disk failure; zone files must be recreated manually |
> **Action needed:** Create backup scripts for all three. For Technitium, the data directory can be tar'd and sent to `s3://hermes-vps-backups/app2/technitium/`. For Dawarich and RAGFlow, determine database engines before designing backup strategy.
---
## Migration History (Jul 28, 2026)
All the following services were migrated from Core to App1 in a single session: