ippadmin/itpp-infrastructure
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
13b48e09d7d697329c72de0f3bcf9a5641db165f
itpp-infrastructure/backup-plan.md
T

293 lines
19 KiB
Markdown
Raw Blame History

# ITPP Backup Plan
> **Last updated:** 2026-09-11
> **Scope:** All ITPP infrastructure backups — Hermes core, application servers, routers, and external services.
> **Storage:** Wasabi S3 (us-east-1) via `--endpoint-url https://s3.us-east-1.wasabisys.com`
> **Cron backend:** Hybrid — system crontab + Hermes cron jobs (both on Core)
> **Verification audit:** [DR issue log](/root/.hermes/references/dr-issue-log.md)
---
## Backup Inventory (27 targets)
### Core (152.53.192.33) — RS 2000
| Service | Method | Destination | Schedule | Last Verified |
|---------|--------|-------------|----------|---------------|
| Hermes Agent (full) | `hermes-backup.sh` — tar.gz of config, skills, profiles, sessions | `s3://hermes-vps-backups/hermes-full-backup/` | 1:00 AM | 2026-07-28 |
| Hermes Live Sync | `hermes-live-sync` cron — session state + profiles DB | `s3://hermes-vps-backups/live/` | Every 15 min | 2026-07-28 20:42 |
| /root Essentials | `root-essentials-backup.sh` — dotfiles, keys, scripts | `s3://hermes-vps-backups/root-backup/` | 3:00 AM | 2026-07-28 |
| Grafana | `core-services-backup.sh` — SQLite DB dump | `s3://hermes-vps-backups/core/grafana/` | 1:30 AM | 2026-07-28 |
| Uptime Kuma | `core-services-backup.sh` — SQLite DB dump | `s3://hermes-vps-backups/core/uptime-kuma/` | 1:30 AM | 2026-07-28 |
| Docker Volumes | `core-services-backup.sh` — tar of key compose volumes | `s3://hermes-vps-backups/volumes/` | 1:30 AM | 2026-07-28 |
| Prometheus | `core-services-backup.sh` — TSDB snapshot | `s3://hermes-vps-backups/core/prometheus/` | 1:30 AM | 2026-07-28 |
| Auth API | `auth-api-backup.sh` — SQLite .backup + .env | `s3://hermes-vps-backups/core/auth-api/` | 3:15 AM | 2026-08-08 |
### App1 (152.53.36.131) — RS 4000
| Service | Method | Destination | Schedule | Last Verified |
|---------|--------|-------------|----------|---------------|
| Open WebUI | `app1-backup.sh` — data dir tar.gz | `s3://hermes-vps-backups/app1/openwebui/` | 2:00 AM | 2026-07-28 |
| LiteLLM | `litellm-backup.sh` — Postgres DB dump + .env | `s3://hermes-vps-backups/app1/litellm/` | 3:30 AM | 2026-07-28 |
| n8n | `app1-backup.sh` — Postgres DB dump | `s3://hermes-vps-backups/app1/n8n/` | 2:00 AM | 2026-07-28 |
| MCP Server Configs | `app1-backup.sh` — MCP settings files | `s3://hermes-vps-backups/app1/mcp/` | 2:00 AM | 2026-07-28 |
| Vaultwarden | `vaultwarden-backup.sh` — SQLite DB dump | `s3://hermes-vps-backups/app1/vaultwarden/` | 2:30 AM | 2026-07-28 |
| Komodo | `komodo-backup.sh` — MongoDB dump + compose + keys | `s3://hermes-vps-backups/app1/komodo/` | 3:45 AM | 2026-07-28 |
| DocuSeal | `docuseal-backup.sh` — SQLite DB + attachments | `s3://hermes-vps-backups/app1/docuseal/` | 4:00 AM | 2026-07-28 |
| Twenty CRM | `twenty-backup.sh` — Postgres dump + .env + compose | `s3://hermes-vps-backups/app1/twenty/` | 4:15 AM | 2026-07-28 |
| Kokoro TTS | *(stateless — no backup needed)* | — | — | — |
### App2 (152.53.39.202) — RS 4000
| Service | Method | Destination | Schedule | Last Verified |
|---------|--------|-------------|----------|---------------|
| Hudu | `hudu-backup.sh` — Postgres dump | `s3://hermes-vps-backups/hudu/backups/` | 7:00 AM | 2026-07-28 |
| Gitea | `gitea-backup.sh` — repos + DB dump | `s3://hermes-vps-backups/gitea/daily/` | 8:00 AM | 2026-07-28 |
| UNMS | `unms-backup-sync.sh` — auto-backup .unms | `s3://hermes-vps-backups/unms-backups/live/backups/` | 6:00 AM | 2026-07-28 |
| UniFi | `unifi-backup-sync.sh` — auto-backup .unf | `s3://hermes-vps-backups/unifi-backups/` | 2:00 AM | 2026-07-18 |
| Traccar | `app2-backup.sh` — H2 DB + config | `s3://hermes-vps-backups/app2/traccar/` | 2:30 AM | 2026-07-28 |
| Technitium DNS | `technitium-backup.sh` — data dir + compose | `s3://hermes-vps-backups/app2/technitium/` | 2:45 AM | 2026-08-08 |
| Dawarich | `dawarich-backup.sh` — PostgreSQL dump (remote SSH) | `s3://hermes-vps-backups/app2/dawarich/` | 4:00 AM | 2026-08-08 |
| RAGFlow | `ragflow-backup.sh` — MySQL dump (remote SSH) | `s3://hermes-vps-backups/app2/ragflow/` | 4:15 AM | 2026-08-08 |
### App3 (152.53.241.111) — RS 4000
| Service | Method | Destination | Schedule | Last Verified |
|---------|--------|-------------|----------|---------------|
| CloudPanel DB | `app3-backup.sh` — SQLite DB | `s3://hermes-vps-backups/app3/cloudpanel/` | 3:00 AM | 2026-07-28 |
| MySQL (all DBs) | `app3-backup.sh` — mysqldump | `s3://hermes-vps-backups/app3/mysql/` | 3:00 AM | 2026-07-28 |
| WordPress Files | `app3-backup.sh` — wp-content tar.gz | `s3://hermes-vps-backups/app3/wordpress/` | 3:00 AM | 2026-07-28 |
| Nginx Configs | `app3-backup.sh` — sites-enabled + config | `s3://hermes-vps-backups/app3/config/` | 3:00 AM | 2026-07-28 |
| Static Sites | `app3-backup.sh` — all non-WordPress htdocs (modelortho.com, verdicttank, transitpin, katiewatts, etc.) | `s3://hermes-vps-backups/app3/static/` | 3:00 AM | 2026-08-08 |
| WordPress Snapshots | `/opt/backup-restore/snapshot.sh` — per-site tar.gz | `/opt/backup-restore/snapshots/` (local, 30-day retention) | 1 AM / 1 PM | 2026-08-08 |
| Hexclave (Stack Auth) | `hexclave-backup.sh` — PG dump + compose + env | `s3://hermes-vps-backups/app3/hexclave/` | 3:30 AM | 2026-08-08 |
| modelortho.com | `modelortho-backup.sh` — htdocs tar.gz + nginx configs | `s3://hermes-vps-backups/app3/modelortho/` | 4:30 AM | 2026-08-08 |
### wphost02 (5.161.62.38) — Hetzner CPX21 — **DECOMMISSIONED (2026-08-28)**
wphost02 was deleted from the Hetzner account on 2026-08-28. Its backup job has been removed; sites now live on app3 (covered under the App3 backup section above).
### anita-mnz (159.195.16.30) - netcup (Anita's dedicated box, cut over 2026-09-11)
| Service | Method | Destination | Schedule | Last Verified |
|---------|--------|-------------|----------|---------------|
| /root Essentials | `root-essentials-backup.sh` - dotfiles, profile, keys, scripts (**excludes `*.db` by design**) | `s3://hermes-vps-backups/root-backup/anita-mnz/` | 3:00 AM | 2026-09-11 |
| Hermes Profile DBs | `hermes-db-backup.sh` - `sqlite3 .backup` snapshots of state.db + cron + wisdom + notepad + verification DBs, `quick_check` per snapshot, upload then download-and-verify | `s3://hermes-vps-backups/root-backup/anita-mnz/db/` | 3:10 AM | 2026-09-11 (restore test passed: quick_check ok, 99,285 messages) |
### Home Router
| Service | Method | Destination | Schedule | Last Verified |
|---------|--------|-------------|----------|---------------|
| MikroTik CCR2004 | `run-wisp-backup.sh` — export .rsc via SSH | `s3://mikrotik-ccr-backups/wisp-backups/configs/home/` | 6:00 AM | 2026-07-28 |
### External
| Service | Method | Destination | Schedule | Last Verified |
|---------|--------|-------------|----------|---------------|
| Hetzner Snapshots | `snapshot-hetzner.py` — API-driven disk snapshots | Hetzner Cloud (weekly) | Mon 5:00 AM | — |
---
## Schedules Summary (ET)
| Time | What | Runner | Script |
|------|------|--------|--------|
| Every 15 min | Hermes session state | Hermes cron | `hermes-live-sync` |
| 1:00 AM | Full Hermes backup | crontab | `hermes-backup.sh` |
| 1:30 AM | Grafana, Uptime Kuma, Docker volumes, Prometheus | crontab | `core-services-backup.sh` |
| 2:00 AM | Open WebUI, n8n, MCP configs (App1) + **UniFi sync** (App2) | crontab | `app1-backup.sh`, `unifi-backup-sync.sh` |
| 2:30 AM | Vaultwarden (App1) + Traccar (App2) | Hermes cron / crontab | `vaultwarden-backup.sh`, `app2-backup.sh` |
| 2:45 AM | Technitium DNS (App2) | Hermes cron | `technitium-backup.sh` |
| 3:00 AM | /root essentials + App3 (CloudPanel, MySQL, WP, Nginx) | crontab | `root-essentials-backup.sh`, `app3-backup.sh` |
| 3:00 AM | **anita-mnz** /root essentials | Anita crontab | `root-essentials-backup.sh` |
| 3:10 AM | **anita-mnz** Hermes profile DB snapshots | Anita crontab | `hermes-db-backup.sh` |
| 3:15 AM | Auth API (Core) | Hermes cron | `auth-api-backup.sh` |
| 3:30 AM | Hexclave / Stack Auth (App3) + LiteLLM (App1) | Hermes cron / crontab | `hexclave-backup.sh`, `litellm-backup.sh` |
| 3:45 AM | Komodo (App1) | crontab | `komodo-backup.sh` |
| 4:00 AM | **Dawarich (App2)** | Hermes cron | `dawarich-backup.sh` |
| 4:00 AM | DocuSeal (App1) | Hermes cron | `docuseal-backup.sh` |
| 4:15 AM | **RAGFlow (App2)** | Hermes cron | `ragflow-backup.sh` |
| 4:15 AM | Twenty CRM (App1) | Hermes cron | `twenty-backup.sh` |
| 6:00 AM | MikroTik CCR + UNMS sync | Hermes cron | `run-wisp-backup.sh`, `unms-backup-sync.sh` |
| 1 AM / 1 PM | WordPress per-site snapshots (App3, local) | App3 crontab | `/opt/backup-restore/snapshot.sh` |
| 7:00 AM | Hudu (App2) | Hermes cron | `hudu-backup.sh` |
| 8:00 AM | Gitea (App2) | Hermes cron | `gitea-backup.sh` |
| Mon 5:00 AM | Hetzner weekly snapshots | Hermes cron | `snapshot-hetzner.py` |
---
## Script Inventory
### On Core (`/root/.hermes/scripts/`)
| Script | Purpose | Runs |
|--------|---------|------|
| `hermes-backup.sh` | Full Hermes tar.gz to S3 | crontab 1:00 AM |
| `core-services-backup.sh` | Grafana, Uptime Kuma, volumes, Prometheus | crontab 1:30 AM |
| `root-essentials-backup.sh` | /root keys, configs, scripts | crontab 3:00 AM |
| `backup-audit-check.sh` | Verify recent backup timestamps | crontab 2:00 AM |
| `vaultwarden-backup.sh` | Vaultwarden SQLite dump (SSH to App1) | Hermes cron 2:30 AM |
| `litellm-backup.sh` | LiteLLM Postgres dump + config (SSH to App1) | Hermes cron 3:30 AM |
| `komodo-backup.sh` | Komodo MongoDB dump (SSH to App1) | Hermes cron 3:45 AM |
| `docuseal-backup.sh` | DocuSeal SQLite + attachments (SSH to App1) | Hermes cron 4:00 AM |
| `twenty-backup.sh` | Twenty CRM Postgres dump (SSH to App1) | Hermes cron 4:15 AM |
| `app1-backup.sh` | Open WebUI, n8n, MCP configs (SSH to App1) | App1 crontab 2:00 AM |
| `app2-backup.sh` | Traccar backup (SSH to App2) | App2 crontab 2:30 AM |
| `technitium-backup.sh` | Technitium DNS backup (SSH to App2) | Hermes cron 2:45 AM |
| `app3-backup.sh` | CloudPanel, MySQL, WP, Nginx (SSH to App3) | crontab 3:00 AM |
| `auth-api-backup.sh` | Auth API SQLite + config (Core local) | Hermes cron 3:15 AM |
| `hexclave-backup.sh` | Stack Auth PG dump + compose + env (SSH to App3) | Hermes cron 3:30 AM |
| `dawarich-backup.sh` | Dawarich PostgreSQL dump + compose + env (SSH to App2) | Hermes cron 4:00 AM |
| `ragflow-backup.sh` | RAGFlow MySQL dump + compose + env (SSH to App2) | Hermes cron 4:15 AM |
| `run-wisp-backup.sh` | MikroTik CCR config export | Hermes cron 6:00 AM |
| `unms-backup-sync.sh` | UNMS auto-backup sync to S3 | Hermes cron 6:00 AM |
| `unifi-backup-sync.sh` | UniFi auto-backup sync to S3 | Hermes cron 2:00 AM |
| `hudu-backup.sh` | Hudu Postgres dump to S3 | Hermes cron 7:00 AM |
| `gitea-backup.sh` | Gitea repos + DB dump to S3 | Hermes cron 8:00 AM |
| `snapshot-hetzner.py` | Hetzner server disk snapshots via API | Hermes cron Mon 5:00 AM |
### On App3 (`/opt/backup-restore/`)
| Script | Purpose | Runs |
|--------|---------|------|
| `snapshot.sh` | Per-site WordPress tarball + DB | App3 crontab 6 AM / 6 PM |
### On anita-mnz (`/root/`)
| Script | Purpose | Runs |
|--------|---------|------|
| `root-essentials-backup.sh` | /root essentials (profile, keys, scripts, configs) | crontab 3:00 AM |
| `hermes-db-backup.sh` | `sqlite3 .backup` snapshots of Hermes profile DBs + per-snapshot quick_check + restore test | crontab 3:10 AM |
---
## S3 Bucket Structure
```
hermes-vps-backups/
├── hermes-full-backup/ — Full Hermes daily (tar.gz)
├── live/ — Hermes live sync (every 15 min)
├── decommissioned/ — archived copies of decommissioned systems (profile archives)
├── live-sync/ — Old sync format (deprecated)
├── root-backup/ — /root essentials (Core uses the flat legacy prefix)
│ └── <host>/ — per-host prefix (`anita-mnz`, ...)
│ └── db/ — SQLite `.backup` snapshots (`hermes-db-backup.sh`)
├── standby/ — Standby configs + recovery bundle
├── core/
│ ├── grafana/ — Grafana SQLite DB
│ ├── uptime-kuma/ — Uptime Kuma SQLite DB
│ ├── prometheus/ — Prometheus TSDB snapshots
│ ├── vaultwarden/ — [STALE — service migrated to App1]
│ ├── twenty/ — [STALE — service migrated to App1]
│ ├── searxng/ — [STALE — service removed]
│ └── komodo/ — [STALE — service migrated to App1]
├── app1/
│ ├── openwebui/ — Open WebUI data
│ ├── litellm/ — LiteLLM Postgres dump + config
│ ├── n8n/ — n8n Postgres dump
│ ├── mcp/ — MCP server configs
│ ├── vaultwarden/ — Vaultwarden SQLite dump
│ ├── komodo/ — Komodo MongoDB dump
│ ├── docuseal/ — DocuSeal SQLite + attachments
│ └── twenty/ — Twenty CRM Postgres dump
├── app2/
│ └── traccar/ — Traccar H2 DB + config
├── app3/
│ ├── cloudpanel/ — CloudPanel SQLite DB
│ ├── mysql/ — All MySQL DBs
│ ├── wordpress/ — wp-content tar.gz
│ └── config/ — Nginx configs
├── hudu/backups/ — Hudu Postgres dump
├── gitea/daily/ — Gitea repos + DB
├── unms-backups/live/ — UNMS auto-backups
├── unifi-backups/ — UniFi controller backups
├── volumes/ — Docker volume dumps
├── caddy/ — (unused)
├── assets/ — Static assets
├── snapshots/ — (unused)
└── system-configs-*.tar.gz — Historical system config snapshots
mikrotik-ccr-backups/
└── wisp-backups/configs/
└── home/ — CCR2004 export .rsc
```
---
## Recovery Objectives (RPO / RTO)
| Tier | Services | RPO | RTO | Notes |
|---|---|---|---|---|
| **Critical** | Hermes Agent, Gitea, Traccar, UISP | ≤ 1 hour | ≤ 4 hours | Live sync + daily backups; restore from S3 then replay live-sync |
| **High** | LiteLLM, n8n, Open WebUI, Vaultwarden, Twenty CRM | 24 hours | ≤ 8 hours | Daily backups; restore from previous night's dump |
|| **Medium** | Hudu, UniFi, Komodo, DocuSeal, App3 WP sites, Auth API, Hexclave (Stack Auth) | 24 hours | ≤ 24 hours | Daily backups only; acceptable overnight gap; Auth API backs all SSO; Hexclave is customer-facing auth |
|| **Low** | Grafana, Uptime Kuma, Prometheus, MikroTik CCR, Technitium DNS, Dawarich, RAGFlow | 24 hours | ≤ 48 hours | Monitoring data is nice-to-have; Dawarich is personal location tracking; RAGFlow is dev-stage RAG pipeline |
## Restore Testing Cadence
**Quarterly:** Pick one random backup per tier, restore to a staging location, verify integrity.
**After any major infra change:** Test the affected service's restore path.
**Annual:** Full DR simulation — restore all Critical + High tier services to staging from S3.
## Stale S3 Paths — Cleanup Queue
These paths contain data from services that migrated off Core (Jul 28, 2026) or were removed:
| Path | Status | Action |
|---|---|---|
| `core/vaultwarden/` | Stale 11 days | **Safe to delete** — Vaultwarden migrated to App1; new backups at `app1/vaultwarden/` |
| `core/twenty/` | Stale 11 days | **Safe to delete** — Twenty CRM migrated to App1; new backups at `app1/twenty/` |
| `core/searxng/` | Stale 11 days | **Safe to delete** — SearXNG removed; replaced by Super Search |
| `core/komodo/` | Stale 11 days | **Safe to delete** — Komodo migrated to App1; new backups at `app1/komodo/` |
| `caddy/` | Unused | **Safe to delete** — Never populated |
| `snapshots/` | Unused | **Safe to delete** — Never populated |
| `live/profiles/anita/` | Purged 2026-09-11 | **Deleted** — 20,551 objects / 5.0 GB of the frozen Core copy of Anita's profile, pushed by the 15-minute sync before it was paused Sep 3. Superseded by `decommissioned/anita-core-frozen-profile-20260911-1732.tar.gz` |
## Unbacked Services
These services are running in production with **zero backup coverage**:
| Unbacked | *(none)* | N/A | N/A | All services are backed up as of 2026-08-08 |
**Done 2026-09-11 (Anita migration closeout).** The frozen Core copy `/root/.hermes/profiles/anita` (8.0 GB, excluded from live sync after the 15:53 cutover) was archived to
`s3://hermes-vps-backups/decommissioned/anita-core-frozen-profile-20260911-1732.tar.gz` — 2,786,082,561 B, sha256 `b457fa7f…b3dc`, 21,206 entries — then deleted, reclaiming 8 GB (125 GB → 117 GB used).
The archive was proven by downloading the object back and matching the sha256 against the local tarball **before** anything was removed. The 8 GB was almost entirely quarantine
artifacts from the Sep 3/9/10 SQLite corruption events (`corrupt-20260903/09/10`, `pre-rebuild-20260910`, `recovered-20260910`); the live profile had already been proven migrated
(memories, `.env` and the six cron job IDs identical on the new box, which carries one more skill and two more messages).
**Gap found and closed 2026-09-11 (anita-mnz):** `root-essentials-backup.sh` correctly excludes `*.db` so it never archives a
live SQLite file, but nothing replaced it, so after the box was provisioned Anita's conversation store (974 MB), the cron
execution DB, notepad, wisdom and verification DBs had **zero** backup coverage. Core has `hermes-backup.sh` doing this job;
the migrated box had no equivalent. `hermes-db-backup.sh` (3:10 AM) now snapshots each DB with the sqlite3 `.backup` API,
quick_checks every snapshot, uploads one dated tarball, then downloads it back and verifies the restored store. Verified
2026-09-11 17:04: restore test passed, `quick_check=ok`, 99,285 messages. Any box migrated with this script set needs the
same check, because the `*.db` exclusion is silent.
> **6 previously-unbacked services now backed up (Auth API, Technitium DNS, Dawarich, RAGFlow, Stack Auth/Hexclave, app3 static sites).** The earlier count of "14" was an error — no source document supports that number. The actual delta since July 28 is 6.
---
## Migration History (Jul 28, 2026)
All the following services were migrated from Core to App1 in a single session:
| Service | Old Home | New Home | Backup Before | Backup After |
|---------|----------|----------|--------------|-------------|
| Vaultwarden | Core (:8080) | App1 (:8081) | `core/vaultwarden/` (stale) | `app1/vaultwarden/` ✅ |
| SearXNG | Core (:8888) | Removed | `core/searxng/` (stale) | N/A (replaced by Super Search) |
| Twenty CRM | Core (:3003) | App1 (:3003) | `core/twenty/` (stale) | `app1/twenty/` ✅ |
| Komodo | Core (:9120) | App1 (:9120) | `core/komodo/` (stale) | `app1/komodo/` ✅ |
| DocuSeal | Core (:3000) | App1 (:3002) | No backup existed | `app1/docuseal/` ✅ |
| Kokoro TTS | Core (:8880) | App1 (:8880) | N/A (stateless) | N/A (stateless) |
---
## Disaster Recovery
- **Live Hermes:** netcup VPS — `core.itpropartner.com` — `152.53.192.33`
- **Standby Hermes:** Hetzner CPX21 — `app1-bu.itpropartner.com` — `5.161.225.131`
- Cron checks live box every 10 min, takes over if down
- Auto-restores from `s3://hermes-vps-backups/hermes-full-backup/`
- Provider diversity: netcup outage won't kill both Core and standby
- **Recovery priority:** Hermes first → infrastructure monitoring (Uptime Kuma, Grafana) → App1 services → App2 services
- **Backup scripts:** 24 scripts on Core, 1 on App3 (wphost02 DECOMMISSIONED 2026-08-28; its backup script and S3 path were removed)
Reference in New Issue View Git Blame Copy Permalink