ippadmin/itpp-infrastructure
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: ippadmin/itpp-infrastructure:c4c6609a817be744c9cc36ca8cbe898db5c0e21a
Branches Tags
ippadmin/itpp-infrastructure:master ippadmin/itpp-infrastructure:main
..
compare: ippadmin/itpp-infrastructure:81762a33bd4178186ccb2fa680c0ae109bbb6c36
Branches Tags
ippadmin/itpp-infrastructure:main ippadmin/itpp-infrastructure:master
2 Commits
c4c6609a81 .. 81762a33bd
Author SHA1 Message Date
root 81762a33bd CRITICAL #4: Auth API + Stack Auth backups — auth-api-backup.sh (SQLite hot), hexclave-backup.sh (PG dump + compose). Auth API Core 3:15 AM, Hexclave app3 3:30 AM. Update RPO/RTO tiers. Remove Technitium from Unbacked (now in Low tier). 2026-08-08 18:20:03 -04:00
root be5af418ae CRITICAL #1: Rotate Grafana off admin/admin to random credential stored in Vaultwarden 2026-08-08 18:09:18 -04:00
2 changed files with 16 additions and 9 deletions
Expand all files Collapse all files
README.md
+1 -1
View File
@@ -218,7 +218,7 @@ unifi.itpropartner.com → :8443 (UniFi)
| Open WebUI | https://ai.itpropartner.com | app1 | Chat UI |
| Open WebUI Admin | https://admin-ai.itpropartner.com/ui | app1 | user: admin, pw: LITELLM_MASTER_KEY |
| Ops Portal | https://ops.itpropartner.com | Core | Internal dashboard |
| Grafana | http://core.itpropartner.com:3002 | Core | admin/admin |
| Grafana | http://core.itpropartner.com:3002 | Core | admin / stored in Vaultwarden |
| Uptime Kuma | https://uptimekuma.itpropartner.com | Core | Service monitoring |
| Vaultwarden | https://vault.iamgmb.com | app1 | Password vault |
| CloudPanel | https://panel.itpropartner.com | app3 | user: gmb / SQLite auth |
backup-plan.md
+15 -8
View File
@@ -21,6 +21,7 @@
| Uptime Kuma | `core-services-backup.sh` — SQLite DB dump | `s3://hermes-vps-backups/core/uptime-kuma/` | 1:30 AM | 2026-07-28 |
| Docker Volumes | `core-services-backup.sh` — tar of key compose volumes | `s3://hermes-vps-backups/volumes/` | 1:30 AM | 2026-07-28 |
| Prometheus | `core-services-backup.sh` — TSDB snapshot | `s3://hermes-vps-backups/core/prometheus/` | 1:30 AM | 2026-07-28 |
| Auth API | `auth-api-backup.sh` — SQLite .backup + .env | `s3://hermes-vps-backups/core/auth-api/` | 3:15 AM | 2026-08-08 |
### App1 (152.53.36.131) — RS 4000
@@ -45,6 +46,7 @@
| UNMS | `unms-backup-sync.sh` — auto-backup .unms | `s3://hermes-vps-backups/unms-backups/live/backups/` | 6:00 AM | 2026-07-28 |
| UniFi | `unifi-backup-sync.sh` — auto-backup .unf | `s3://hermes-vps-backups/unifi-backups/` | 2:00 AM | 2026-07-18 |
| Traccar | `app2-backup.sh` — H2 DB + config | `s3://hermes-vps-backups/app2/traccar/` | 2:30 AM | 2026-07-28 |
| Technitium DNS | `technitium-backup.sh` — data dir + compose | `s3://hermes-vps-backups/app2/technitium/` | 2:45 AM | 2026-08-08 |
### App3 (152.53.241.111) — RS 4000
@@ -55,6 +57,7 @@
| WordPress Files | `app3-backup.sh` — wp-content tar.gz | `s3://hermes-vps-backups/app3/wordpress/` | 3:00 AM | 2026-07-28 |
| Nginx Configs | `app3-backup.sh` — sites-enabled + config | `s3://hermes-vps-backups/app3/config/` | 3:00 AM | 2026-07-28 |
| WordPress Snapshots | `/opt/backup-restore/snapshot.sh` — per-site tar.gz | `/opt/backup-restore/snapshots/` (local, 30-day retention) | 1 AM / 1 PM | 2026-08-08 |
| Hexclave (Stack Auth) | `hexclave-backup.sh` — PG dump + compose + env | `s3://hermes-vps-backups/app3/hexclave/` | 3:30 AM | 2026-08-08 |
### wphost02 (5.161.62.38) — Hetzner CPX21
@@ -85,9 +88,11 @@
| 1:30 AM | Grafana, Uptime Kuma, Docker volumes, Prometheus | crontab | `core-services-backup.sh` |
| 2:00 AM | Open WebUI, n8n, MCP configs (App1) + **UniFi sync** (App2) | crontab | `app1-backup.sh`, `unifi-backup-sync.sh` |
| 2:30 AM | Vaultwarden (App1) + Traccar (App2) | Hermes cron / crontab | `vaultwarden-backup.sh`, `app2-backup.sh` |
| 2:45 AM | Technitium DNS (App2) | Hermes cron | `technitium-backup.sh` |
| 3:00 AM | /root essentials + App3 (CloudPanel, MySQL, WP, Nginx) | crontab | `root-essentials-backup.sh`, `app3-backup.sh` |
| 3:30 AM | LiteLLM (App1) | Hermes cron | `litellm-backup.sh` |
| 3:45 AM | Komodo (App1) | Hermes cron | `komodo-backup.sh` |
| 3:15 AM | Auth API (Core) | Hermes cron | `auth-api-backup.sh` |
| 3:30 AM | Hexclave / Stack Auth (App3) + LiteLLM (App1) | Hermes cron / crontab | `hexclave-backup.sh`, `litellm-backup.sh` |
| 3:45 AM | Komodo (App1) | crontab | `komodo-backup.sh` |
| 4:00 AM | DocuSeal (App1) | Hermes cron | `docuseal-backup.sh` |
| 4:15 AM | Twenty CRM (App1) | Hermes cron | `twenty-backup.sh` |
| 5:00 AM | wphost02 WordPress (SSH) | crontab | `wphost02-backup` |
@@ -116,7 +121,10 @@
| `twenty-backup.sh` | Twenty CRM Postgres dump (SSH to App1) | Hermes cron 4:15 AM |
| `app1-backup.sh` | Open WebUI, n8n, MCP configs (SSH to App1) | App1 crontab 2:00 AM |
| `app2-backup.sh` | Traccar backup (SSH to App2) | App2 crontab 2:30 AM |
| `technitium-backup.sh` | Technitium DNS backup (SSH to App2) | Hermes cron 2:45 AM |
| `app3-backup.sh` | CloudPanel, MySQL, WP, Nginx (SSH to App3) | crontab 3:00 AM |
| `auth-api-backup.sh` | Auth API SQLite + config (Core local) | Hermes cron 3:15 AM |
| `hexclave-backup.sh` | Stack Auth PG dump + compose + env (SSH to App3) | Hermes cron 3:30 AM |
| `wphost02-backup.sh` | SSH to wphost02, backup all sites | crontab 5:00 AM |
| `run-wisp-backup.sh` | MikroTik CCR config export | Hermes cron 6:00 AM |
| `unms-backup-sync.sh` | UNMS auto-backup sync to S3 | Hermes cron 6:00 AM |
@@ -196,9 +204,9 @@ mikrotik-ccr-backups/
|---|---|---|---|---|
| **Critical** | Hermes Agent, Gitea, Traccar, UISP | ≤ 1 hour | ≤ 4 hours | Live sync + daily backups; restore from S3 then replay live-sync |
| **High** | LiteLLM, n8n, Open WebUI, Vaultwarden, Twenty CRM | 24 hours | ≤ 8 hours | Daily backups; restore from previous night's dump |
| **Medium** | Hudu, UniFi, Komodo, DocuSeal, App3 WP sites | 24 hours | ≤ 24 hours | Daily backups only; acceptable overnight gap |
| **Low** | Grafana, Uptime Kuma, Prometheus, MikroTik CCR | 24 hours | ≤ 48 hours | Monitoring data is nice-to-have, not blocking |
| **Unbacked** | Dawarich, RAGFlow, Technitium DNS | N/A | N/A | No backup exists — see Unbacked Services below |
|| **Medium** | Hudu, UniFi, Komodo, DocuSeal, App3 WP sites, Auth API, Hexclave (Stack Auth) | 24 hours | ≤ 24 hours | Daily backups only; acceptable overnight gap; Auth API backs all SSO; Hexclave is customer-facing auth |
| **Low** | Grafana, Uptime Kuma, Prometheus, MikroTik CCR, Technitium DNS | 24 hours | ≤ 48 hours | Monitoring data is nice-to-have, not blocking; Technitium DNS is caching-only (no authoritative zones — public DNS is SiteGround/Cloudflare, servers use Tailscale MagicDNS) |
| **Unbacked** | Dawarich, RAGFlow | N/A | N/A | No backup exists — see Unbacked Services below |
## Restore Testing Cadence
@@ -226,10 +234,9 @@ These services are running in production with **zero backup coverage**:
| Service | Server | Data Path | Risk |
|---|---|---|---|
| **Dawarich** | app2 | Docker: `dawarich_db`, `dawarich_app` (4 containers) | Location history lost on disk failure |
| **RAGFlow** | app2 | Docker: `ragflow-cpu-1` | RAG pipeline state lost |
| **Technitium DNS** | app2 | `/root/docker/technitium/data` (bind mount) | All DNS zones lost on disk failure; zone files must be recreated manually |
|| **RAGFlow** | app2 | Docker: `ragflow-cpu-1` | RAG pipeline state lost |
> **Action needed:** Create backup scripts for all three. For Technitium, the data directory can be tar'd and sent to `s3://hermes-vps-backups/app2/technitium/`. For Dawarich and RAGFlow, determine database engines before designing backup strategy.
> **Action needed:** Create backup scripts for both remaining unbacked services. For Dawarich and RAGFlow, determine database engines before designing backup strategy.
---