ippadmin/itpp-infrastructure
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

CRITICAL #4: Auth API + Stack Auth backups — auth-api-backup.sh (SQLite hot), hexclave-backup.sh (PG dump + compose). Auth API Core 3:15 AM, Hexclave app3 3:30 AM. Update RPO/RTO tiers. Remove Technitium from Unbacked (now in Low tier).

Browse Source
This commit is contained in:
root
2026-08-08 18:20:03 -04:00
parent be5af418ae
commit 81762a33bd
1 changed files with 15 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backup-plan.md
+15 -8
View File
@@ -21,6 +21,7 @@
| Uptime Kuma | `core-services-backup.sh` — SQLite DB dump | `s3://hermes-vps-backups/core/uptime-kuma/` | 1:30 AM | 2026-07-28 | | Uptime Kuma | `core-services-backup.sh` — SQLite DB dump | `s3://hermes-vps-backups/core/uptime-kuma/` | 1:30 AM | 2026-07-28 |
| Docker Volumes | `core-services-backup.sh` — tar of key compose volumes | `s3://hermes-vps-backups/volumes/` | 1:30 AM | 2026-07-28 | | Docker Volumes | `core-services-backup.sh` — tar of key compose volumes | `s3://hermes-vps-backups/volumes/` | 1:30 AM | 2026-07-28 |
| Prometheus | `core-services-backup.sh` — TSDB snapshot | `s3://hermes-vps-backups/core/prometheus/` | 1:30 AM | 2026-07-28 | | Prometheus | `core-services-backup.sh` — TSDB snapshot | `s3://hermes-vps-backups/core/prometheus/` | 1:30 AM | 2026-07-28 |
| Auth API | `auth-api-backup.sh` — SQLite .backup + .env | `s3://hermes-vps-backups/core/auth-api/` | 3:15 AM | 2026-08-08 |
### App1 (152.53.36.131) — RS 4000 ### App1 (152.53.36.131) — RS 4000
@@ -45,6 +46,7 @@
| UNMS | `unms-backup-sync.sh` — auto-backup .unms | `s3://hermes-vps-backups/unms-backups/live/backups/` | 6:00 AM | 2026-07-28 | | UNMS | `unms-backup-sync.sh` — auto-backup .unms | `s3://hermes-vps-backups/unms-backups/live/backups/` | 6:00 AM | 2026-07-28 |
| UniFi | `unifi-backup-sync.sh` — auto-backup .unf | `s3://hermes-vps-backups/unifi-backups/` | 2:00 AM | 2026-07-18 | | UniFi | `unifi-backup-sync.sh` — auto-backup .unf | `s3://hermes-vps-backups/unifi-backups/` | 2:00 AM | 2026-07-18 |
| Traccar | `app2-backup.sh` — H2 DB + config | `s3://hermes-vps-backups/app2/traccar/` | 2:30 AM | 2026-07-28 | | Traccar | `app2-backup.sh` — H2 DB + config | `s3://hermes-vps-backups/app2/traccar/` | 2:30 AM | 2026-07-28 |
| Technitium DNS | `technitium-backup.sh` — data dir + compose | `s3://hermes-vps-backups/app2/technitium/` | 2:45 AM | 2026-08-08 |
### App3 (152.53.241.111) — RS 4000 ### App3 (152.53.241.111) — RS 4000
@@ -55,6 +57,7 @@
| WordPress Files | `app3-backup.sh` — wp-content tar.gz | `s3://hermes-vps-backups/app3/wordpress/` | 3:00 AM | 2026-07-28 | | WordPress Files | `app3-backup.sh` — wp-content tar.gz | `s3://hermes-vps-backups/app3/wordpress/` | 3:00 AM | 2026-07-28 |
| Nginx Configs | `app3-backup.sh` — sites-enabled + config | `s3://hermes-vps-backups/app3/config/` | 3:00 AM | 2026-07-28 | | Nginx Configs | `app3-backup.sh` — sites-enabled + config | `s3://hermes-vps-backups/app3/config/` | 3:00 AM | 2026-07-28 |
| WordPress Snapshots | `/opt/backup-restore/snapshot.sh` — per-site tar.gz | `/opt/backup-restore/snapshots/` (local, 30-day retention) | 1 AM / 1 PM | 2026-08-08 | | WordPress Snapshots | `/opt/backup-restore/snapshot.sh` — per-site tar.gz | `/opt/backup-restore/snapshots/` (local, 30-day retention) | 1 AM / 1 PM | 2026-08-08 |
| Hexclave (Stack Auth) | `hexclave-backup.sh` — PG dump + compose + env | `s3://hermes-vps-backups/app3/hexclave/` | 3:30 AM | 2026-08-08 |
### wphost02 (5.161.62.38) — Hetzner CPX21 ### wphost02 (5.161.62.38) — Hetzner CPX21
@@ -85,9 +88,11 @@
| 1:30 AM | Grafana, Uptime Kuma, Docker volumes, Prometheus | crontab | `core-services-backup.sh` | | 1:30 AM | Grafana, Uptime Kuma, Docker volumes, Prometheus | crontab | `core-services-backup.sh` |
| 2:00 AM | Open WebUI, n8n, MCP configs (App1) + **UniFi sync** (App2) | crontab | `app1-backup.sh`, `unifi-backup-sync.sh` | | 2:00 AM | Open WebUI, n8n, MCP configs (App1) + **UniFi sync** (App2) | crontab | `app1-backup.sh`, `unifi-backup-sync.sh` |
| 2:30 AM | Vaultwarden (App1) + Traccar (App2) | Hermes cron / crontab | `vaultwarden-backup.sh`, `app2-backup.sh` | | 2:30 AM | Vaultwarden (App1) + Traccar (App2) | Hermes cron / crontab | `vaultwarden-backup.sh`, `app2-backup.sh` |
| 2:45 AM | Technitium DNS (App2) | Hermes cron | `technitium-backup.sh` |
| 3:00 AM | /root essentials + App3 (CloudPanel, MySQL, WP, Nginx) | crontab | `root-essentials-backup.sh`, `app3-backup.sh` | | 3:00 AM | /root essentials + App3 (CloudPanel, MySQL, WP, Nginx) | crontab | `root-essentials-backup.sh`, `app3-backup.sh` |
| 3:30 AM | LiteLLM (App1) | Hermes cron | `litellm-backup.sh` | | 3:15 AM | Auth API (Core) | Hermes cron | `auth-api-backup.sh` |
| 3:45 AM | Komodo (App1) | Hermes cron | `komodo-backup.sh` | | 3:30 AM | Hexclave / Stack Auth (App3) + LiteLLM (App1) | Hermes cron / crontab | `hexclave-backup.sh`, `litellm-backup.sh` |
| 3:45 AM | Komodo (App1) | crontab | `komodo-backup.sh` |
| 4:00 AM | DocuSeal (App1) | Hermes cron | `docuseal-backup.sh` | | 4:00 AM | DocuSeal (App1) | Hermes cron | `docuseal-backup.sh` |
| 4:15 AM | Twenty CRM (App1) | Hermes cron | `twenty-backup.sh` | | 4:15 AM | Twenty CRM (App1) | Hermes cron | `twenty-backup.sh` |
| 5:00 AM | wphost02 WordPress (SSH) | crontab | `wphost02-backup` | | 5:00 AM | wphost02 WordPress (SSH) | crontab | `wphost02-backup` |
@@ -116,7 +121,10 @@
| `twenty-backup.sh` | Twenty CRM Postgres dump (SSH to App1) | Hermes cron 4:15 AM | | `twenty-backup.sh` | Twenty CRM Postgres dump (SSH to App1) | Hermes cron 4:15 AM |
| `app1-backup.sh` | Open WebUI, n8n, MCP configs (SSH to App1) | App1 crontab 2:00 AM | | `app1-backup.sh` | Open WebUI, n8n, MCP configs (SSH to App1) | App1 crontab 2:00 AM |
| `app2-backup.sh` | Traccar backup (SSH to App2) | App2 crontab 2:30 AM | | `app2-backup.sh` | Traccar backup (SSH to App2) | App2 crontab 2:30 AM |
| `technitium-backup.sh` | Technitium DNS backup (SSH to App2) | Hermes cron 2:45 AM |
| `app3-backup.sh` | CloudPanel, MySQL, WP, Nginx (SSH to App3) | crontab 3:00 AM | | `app3-backup.sh` | CloudPanel, MySQL, WP, Nginx (SSH to App3) | crontab 3:00 AM |
| `auth-api-backup.sh` | Auth API SQLite + config (Core local) | Hermes cron 3:15 AM |
| `hexclave-backup.sh` | Stack Auth PG dump + compose + env (SSH to App3) | Hermes cron 3:30 AM |
| `wphost02-backup.sh` | SSH to wphost02, backup all sites | crontab 5:00 AM | | `wphost02-backup.sh` | SSH to wphost02, backup all sites | crontab 5:00 AM |
| `run-wisp-backup.sh` | MikroTik CCR config export | Hermes cron 6:00 AM | | `run-wisp-backup.sh` | MikroTik CCR config export | Hermes cron 6:00 AM |
| `unms-backup-sync.sh` | UNMS auto-backup sync to S3 | Hermes cron 6:00 AM | | `unms-backup-sync.sh` | UNMS auto-backup sync to S3 | Hermes cron 6:00 AM |
@@ -196,9 +204,9 @@ mikrotik-ccr-backups/
|---|---|---|---|---| |---|---|---|---|---|
| **Critical** | Hermes Agent, Gitea, Traccar, UISP | ≤ 1 hour | ≤ 4 hours | Live sync + daily backups; restore from S3 then replay live-sync | | **Critical** | Hermes Agent, Gitea, Traccar, UISP | ≤ 1 hour | ≤ 4 hours | Live sync + daily backups; restore from S3 then replay live-sync |
| **High** | LiteLLM, n8n, Open WebUI, Vaultwarden, Twenty CRM | 24 hours | ≤ 8 hours | Daily backups; restore from previous night's dump | | **High** | LiteLLM, n8n, Open WebUI, Vaultwarden, Twenty CRM | 24 hours | ≤ 8 hours | Daily backups; restore from previous night's dump |
| **Medium** | Hudu, UniFi, Komodo, DocuSeal, App3 WP sites | 24 hours | ≤ 24 hours | Daily backups only; acceptable overnight gap | || **Medium** | Hudu, UniFi, Komodo, DocuSeal, App3 WP sites, Auth API, Hexclave (Stack Auth) | 24 hours | ≤ 24 hours | Daily backups only; acceptable overnight gap; Auth API backs all SSO; Hexclave is customer-facing auth |
| **Low** | Grafana, Uptime Kuma, Prometheus, MikroTik CCR | 24 hours | ≤ 48 hours | Monitoring data is nice-to-have, not blocking | | **Low** | Grafana, Uptime Kuma, Prometheus, MikroTik CCR, Technitium DNS | 24 hours | ≤ 48 hours | Monitoring data is nice-to-have, not blocking; Technitium DNS is caching-only (no authoritative zones — public DNS is SiteGround/Cloudflare, servers use Tailscale MagicDNS) |
| **Unbacked** | Dawarich, RAGFlow, Technitium DNS | N/A | N/A | No backup exists — see Unbacked Services below | | **Unbacked** | Dawarich, RAGFlow | N/A | N/A | No backup exists — see Unbacked Services below |
## Restore Testing Cadence ## Restore Testing Cadence
@@ -226,10 +234,9 @@ These services are running in production with **zero backup coverage**:
| Service | Server | Data Path | Risk | | Service | Server | Data Path | Risk |
|---|---|---|---| |---|---|---|---|
| **Dawarich** | app2 | Docker: `dawarich_db`, `dawarich_app` (4 containers) | Location history lost on disk failure | | **Dawarich** | app2 | Docker: `dawarich_db`, `dawarich_app` (4 containers) | Location history lost on disk failure |
| **RAGFlow** | app2 | Docker: `ragflow-cpu-1` | RAG pipeline state lost | || **RAGFlow** | app2 | Docker: `ragflow-cpu-1` | RAG pipeline state lost |
| **Technitium DNS** | app2 | `/root/docker/technitium/data` (bind mount) | All DNS zones lost on disk failure; zone files must be recreated manually |
> **Action needed:** Create backup scripts for all three. For Technitium, the data directory can be tar'd and sent to `s3://hermes-vps-backups/app2/technitium/`. For Dawarich and RAGFlow, determine database engines before designing backup strategy. > **Action needed:** Create backup scripts for both remaining unbacked services. For Dawarich and RAGFlow, determine database engines before designing backup strategy.
--- ---