Update backup plan doc with Jul 28 migration changes

- Remove stale core-services (Vaultwarden, Twenty, SearXNG, Komodo) - Add new App1 services (Komodo, DocuSeal, Twenty CRM backups) - Update S3 path table with canonical locations - Add migration history section (Jul 28) - Note stale core/ subfolder paths in S3 for reference - Coverage: 26 backup targets across 6 servers + router
2026-07-28 20:51:48 -04:00
parent 6e0d793696
commit fcc9863232
1 changed files with 178 additions and 122 deletions
@@ -1,159 +1,215 @@
-# Backup Plan — All Servers
+# ITPP Backup Plan

-**Last Updated:** July 16, 2026
-**S3 Provider:** Wasabi (s3.us-east-1.wasabisys.com)
-**Buckets:** hermes-vps-backups, itpropartner-backups, mikrotik-ccr-backups
-**Versioning:** ON (all buckets)
+> **Last updated:** 2026-07-28
+> **Scope:** All ITPP infrastructure backups — Hermes core, application servers, routers, and external services.
+> **Storage:** Wasabi S3 (us-east-1) via `--endpoint-url https://s3.us-east-1.wasabisys.com`
+> **Cron backend:** Hybrid — system crontab + Hermes cron jobs

 ---

-## Schedule (ET timezone)
+## Backup Inventory (26 targets)

-| Time | Server | Script | What |
-|---|---|---|---|
-| Every 15 min | Core | hermes-live-sync | Hermes session state → S3 live/ |
-| 1:00 AM | Core | hermes-backup.sh | Full Hermes backup |
-| 1:30 AM | Core | core-services-backup.sh | Vaultwarden, Twenty CRM, SearXNG, Komodo, Prometheus, Grafana, Uptime Kuma |
-| 2:00 AM | Core | backup-audit-check.sh | Verify all backups |
-| 2:00 AM | app1 | /root/.hermes/scripts/app1-backup.sh | LiteLLM, Open WebUI, n8n, MCP configs, Mattermost |
-| 2:30 AM | app2 | /root/.hermes/scripts/app2-backup.sh | Traccar, Gitea, Hudu, UNMS, UniFi |
-| 3:00 AM | Core | root-essentials-backup.sh | /root essentials |
-| 3:00 AM | app3 | /root/.hermes/scripts/app3-backup.sh | CloudPanel, MySQL, WordPress, Nginx configs |
-| 5:00 AM | Core | wphost02-backup (cron) | wphost02 webapps + databases → S3 |
-| 1:00 AM + 1:00 PM | app3 | /opt/backup-restore/snapshot.sh | Per-site WordPress snapshots (files + DB), 60-day retention |
-| 3:00 AM | Core | docker-volume-sync.sh | Docker volumes |
-| 4:00 AM | Core | system-config-sync.sh | System configs |
-| 6:00 AM | Core | run-wisp-backup.sh | MikroTik CCR config (via wisp-backup.py) |
-| Every 10 min | core-bu | warm-standby-sync | Pull from S3 live/ → standby readiness |
+### Core (152.53.192.33) — RS 2000
+
+| Service | Method | Destination | Schedule | Last Verified |
+|---------|--------|-------------|----------|---------------|
+| Hermes Agent (full) | `hermes-backup.sh` — tar.gz of config, skills, profiles, sessions | `s3://hermes-vps-backups/hermes-full-backup/` | 1:00 AM | 2026-07-28 |
+| Hermes Live Sync | `hermes-live-sync` cron — session state + profiles DB | `s3://hermes-vps-backups/live/` | Every 15 min | 2026-07-28 20:42 |
+| /root Essentials | `root-essentials-backup.sh` — dotfiles, keys, scripts | `s3://hermes-vps-backups/root-backup/` | 3:00 AM | 2026-07-28 |
+| Grafana | `core-services-backup.sh` — SQLite DB dump | `s3://hermes-vps-backups/core/grafana/` | 1:30 AM | 2026-07-28 |
+| Uptime Kuma | `core-services-backup.sh` — SQLite DB dump | `s3://hermes-vps-backups/core/uptime-kuma/` | 1:30 AM | 2026-07-28 |
+| Docker Volumes | `core-services-backup.sh` — tar of key compose volumes | `s3://hermes-vps-backups/volumes/` | 1:30 AM | 2026-07-28 |
+| Prometheus | `core-services-backup.sh` — TSDB snapshot | `s3://hermes-vps-backups/core/prometheus/` | 1:30 AM | 2026-07-28 |
+
+### App1 (152.53.36.131) — RS 4000
+
+| Service | Method | Destination | Schedule | Last Verified |
+|---------|--------|-------------|----------|---------------|
+| Open WebUI | `app1-backup.sh` — data dir tar.gz | `s3://hermes-vps-backups/app1/openwebui/` | 2:00 AM | 2026-07-28 |
+| LiteLLM | `litellm-backup.sh` — Postgres DB dump + .env | `s3://hermes-vps-backups/app1/litellm/` | 3:30 AM | 2026-07-28 |
+| n8n | `app1-backup.sh` — Postgres DB dump | `s3://hermes-vps-backups/app1/n8n/` | 2:00 AM | 2026-07-28 |
+| MCP Server Configs | `app1-backup.sh` — MCP settings files | `s3://hermes-vps-backups/app1/mcp/` | 2:00 AM | 2026-07-28 |
+| Vaultwarden | `vaultwarden-backup.sh` — SQLite DB dump | `s3://hermes-vps-backups/app1/vaultwarden/` | 2:30 AM | 2026-07-28 |
+| Komodo | `komodo-backup.sh` — MongoDB dump + compose + keys | `s3://hermes-vps-backups/app1/komodo/` | 3:45 AM | 2026-07-28 |
+| DocuSeal | `docuseal-backup.sh` — SQLite DB + attachments | `s3://hermes-vps-backups/app1/docuseal/` | 4:00 AM | 2026-07-28 |
+| Twenty CRM | `twenty-backup.sh` — Postgres dump + .env + compose | `s3://hermes-vps-backups/app1/twenty/` | 4:15 AM | 2026-07-28 |
+| Kokoro TTS | *(stateless — no backup needed)* | — | — | — |
+
+### App2 (152.53.39.202) — RS 4000
+
+| Service | Method | Destination | Schedule | Last Verified |
+|---------|--------|-------------|----------|---------------|
+| Hudu | `hudu-backup.sh` — Postgres dump | `s3://hermes-vps-backups/hudu/backups/` | 7:00 AM | 2026-07-28 |
+| Gitea | `gitea-backup.sh` — repos + DB dump | `s3://hermes-vps-backups/gitea/daily/` | 8:00 AM | 2026-07-28 |
+| UNMS | `unms-backup-sync.sh` — auto-backup .unms | `s3://hermes-vps-backups/unms-backups/live/backups/` | 6:00 AM | 2026-07-28 |
+| UniFi | `unifi-backup-sync.sh` — auto-backup .unf | `s3://hermes-vps-backups/unifi-backups/` | 2:00 AM | 2026-07-18 |
+| Traccar | `app2-backup.sh` — H2 DB + config | `s3://hermes-vps-backups/app2/traccar/` | 2:30 AM | 2026-07-28 |
+
+### App3 (152.53.241.111) — RS 4000
+
+| Service | Method | Destination | Schedule | Last Verified |
+|---------|--------|-------------|----------|---------------|
+| CloudPanel DB | `app3-backup.sh` — SQLite DB | `s3://hermes-vps-backups/app3/cloudpanel/` | 3:00 AM | 2026-07-28 |
+| MySQL (all DBs) | `app3-backup.sh` — mysqldump | `s3://hermes-vps-backups/app3/mysql/` | 3:00 AM | 2026-07-28 |
+| WordPress Files | `app3-backup.sh` — wp-content tar.gz | `s3://hermes-vps-backups/app3/wordpress/` | 3:00 AM | 2026-07-28 |
+| Nginx Configs | `app3-backup.sh` — sites-enabled + config | `s3://hermes-vps-backups/app3/config/` | 3:00 AM | 2026-07-28 |
+| WordPress Snapshots | `/opt/backup-restore/snapshot.sh` — per-site tar.gz | `/opt/backup-restore/snapshots/` (local, 30-day retention) | 6 AM / 6 PM | 2026-07-28 13:00 |
+
+### wphost02 (5.161.62.38) — Hetzner CPX21
+
+| Service | Method | Destination | Schedule | Last Verified |
+|---------|--------|-------------|----------|---------------|
+| WordPress (7 sites) | `ssh → /root/backup.sh` — per-site tar.gz + all DBs | `s3://hermes-vps-backups/wphost02-backup/` | 5:00 AM | 2026-07-28 |
+
+### Home Router
+
+| Service | Method | Destination | Schedule | Last Verified |
+|---------|--------|-------------|----------|---------------|
+| MikroTik CCR2004 | `run-wisp-backup.sh` — export .rsc via SSH | `s3://mikrotik-ccr-backups/wisp-backups/configs/home/` | 6:00 AM | 2026-07-28 |
+
+### External
+
+| Service | Method | Destination | Schedule | Last Verified |
+|---------|--------|-------------|----------|---------------|
+| Hetzner Snapshots | `snapshot-hetzner.py` — API-driven disk snapshots | Hetzner Cloud (weekly) | Mon 5:00 AM | — |

 ---

-## Coverage by Server
+## Schedules Summary (ET)

-### Core (152.53.192.33)
+| Time | What | Runner | Script |
+|------|------|--------|--------|
+| Every 15 min | Hermes session state | Hermes cron | `hermes-live-sync` |
+| 1:00 AM | Full Hermes backup | crontab | `hermes-backup.sh` |
+| 1:30 AM | Grafana, Uptime Kuma, Docker volumes, Prometheus | crontab | `core-services-backup.sh` |
+| 2:00 AM | Open WebUI, n8n, MCP configs (App1) | crontab | `app1-backup.sh` |
+| 2:30 AM | Vaultwarden (App1) + Traccar (App2) | Hermes cron / crontab | `vaultwarden-backup.sh`, `app2-backup.sh` |
+| 3:00 AM | /root essentials + App3 (CloudPanel, MySQL, WP, Nginx) | crontab | `root-essentials-backup.sh`, `app3-backup.sh` |
+| 3:30 AM | LiteLLM (App1) | Hermes cron | `litellm-backup.sh` |
+| 3:45 AM | Komodo (App1) | Hermes cron | `komodo-backup.sh` |
+| 4:00 AM | DocuSeal (App1) | Hermes cron | `docuseal-backup.sh` |
+| 4:15 AM | Twenty CRM (App1) | Hermes cron | `twenty-backup.sh` |
+| 5:00 AM | wphost02 WordPress (SSH) | crontab | `wphost02-backup` |
+| 6:00 AM | MikroTik CCR + UNMS sync | Hermes cron | `run-wisp-backup.sh`, `unms-backup-sync.sh` |
+| 6 AM / 6 PM | WordPress per-site snapshots (App3, local) | App3 crontab | `/opt/backup-restore/snapshot.sh` |
+| 7:00 AM | Hudu (App2) | Hermes cron | `hudu-backup.sh` |
+| 8:00 AM | Gitea (App2) | Hermes cron | `gitea-backup.sh` |
+| Mon 5:00 AM | Hetzner weekly snapshots | Hermes cron | `snapshot-hetzner.py` |

-| Service | Method | RPO |
-|---|---|---|
-| Hermes Agent | tar.gz → S3 (daily) + live sync (15 min) | 15 min |
-| Vaultwarden | SQLite dump → S3 (daily) | 24 hr |
-| Twenty CRM | Postgres pg_dump → S3 (daily) | 24 hr |
-| SearXNG | settings.yml → S3 (daily) | 24 hr |
-| Komodo | config dir → S3 (daily) | 24 hr |
-| Prometheus | TSDB snapshot → S3 (daily) | 24 hr |
-| Grafana | SQLite DB → S3 (daily) | 24 hr |
-| Uptime Kuma | SQLite DB → S3 (daily) | 24 hr |
-| DocuSeal | Covered by docker-volume-sync | 24 hr |
-| Caddy config | system-config-sync (daily) | 24 hr |
-| /root essentials | root-essentials-backup (daily) | 24 hr |
+---

-### app1 (152.53.36.131)
+## Script Inventory

-| Service | Method | RPO |
-|---|---|---|
-| LiteLLM | Postgres pg_dump + config.yaml → S3 (daily) | 24 hr |
-| Open WebUI | /app/backend/data → S3 (daily) | 24 hr |
-| n8n | Postgres pg_dump → S3 (daily) | 24 hr |
-| MCP servers | Configs in Git + Docker volumes | 24 hr |
-| Mattermost | Postgres DB + data volume → S3 (daily) | 24 hr |
+### On Core (`/root/.hermes/scripts/`)

-### app2 (152.53.39.202)
+| Script | Purpose | Runs |
+|--------|---------|------|
+| `hermes-backup.sh` | Full Hermes tar.gz to S3 | crontab 1:00 AM |
+| `core-services-backup.sh` | Grafana, Uptime Kuma, volumes, Prometheus | crontab 1:30 AM |
+| `root-essentials-backup.sh` | /root keys, configs, scripts | crontab 3:00 AM |
+| `backup-audit-check.sh` | Verify recent backup timestamps | crontab 2:00 AM |
+| `vaultwarden-backup.sh` | Vaultwarden SQLite dump (SSH to App1) | Hermes cron 2:30 AM |
+| `litellm-backup.sh` | LiteLLM Postgres dump + config (SSH to App1) | Hermes cron 3:30 AM |
+| `komodo-backup.sh` | Komodo MongoDB dump (SSH to App1) | Hermes cron 3:45 AM |
+| `docuseal-backup.sh` | DocuSeal SQLite + attachments (SSH to App1) | Hermes cron 4:00 AM |
+| `twenty-backup.sh` | Twenty CRM Postgres dump (SSH to App1) | Hermes cron 4:15 AM |
+| `app1-backup.sh` | Open WebUI, n8n, MCP configs (SSH to App1) | App1 crontab 2:00 AM |
+| `app2-backup.sh` | Traccar backup (SSH to App2) | App2 crontab 2:30 AM |
+| `app3-backup.sh` | CloudPanel, MySQL, WP, Nginx (SSH to App3) | crontab 3:00 AM |
+| `wphost02-backup.sh` | SSH to wphost02, backup all sites | crontab 5:00 AM |
+| `run-wisp-backup.sh` | MikroTik CCR config export | Hermes cron 6:00 AM |
+| `unms-backup-sync.sh` | UNMS auto-backup sync to S3 | Hermes cron 6:00 AM |
+| `unifi-backup-sync.sh` | UniFi auto-backup sync to S3 | Hermes cron 2:00 AM |
+| `hudu-backup.sh` | Hudu Postgres dump to S3 | Hermes cron 7:00 AM |
+| `gitea-backup.sh` | Gitea repos + DB dump to S3 | Hermes cron 8:00 AM |
+| `snapshot-hetzner.py` | Hetzner server disk snapshots via API | Hermes cron Mon 5:00 AM |

-| Service | Method | RPO |
-|---|---|---|
-| Traccar | H2 database + conf → S3 (daily) | 24 hr |
-| Gitea | Repos + DB → S3 (daily) | 24 hr |
-| Hudu | Docker volume → S3 (daily, 30-day retention) | 24 hr |
-| UNMS | S3 sync (daily) | 24 hr |
-| UniFi | Autobackup → S3 (daily) | 24 hr |
+### On App3 (`/opt/backup-restore/`)

-### app3 (152.53.241.111)
+| Script | Purpose | Runs |
+|--------|---------|------|
+| `snapshot.sh` | Per-site WordPress tarball + DB | App3 crontab 6 AM / 6 PM |

-| Service | Method | RPO |
-|---|---|---|
-| CloudPanel CE | SQLite DB → S3 (daily) | 24 hr |
-| MySQL | All databases mysqldump → S3 (daily) | 24 hr |
-| WordPress | Files + wp-config → S3 (daily) | 24 hr |
-| Nginx | /etc/nginx + /etc/cloudpanel → S3 (daily) | 24 hr |
-| **WordPress Snapshots** | **Per-site tarball + MySQL dump (2x daily, 60-day retention)** | **12 hr** |
-| **Backup Restore UI** | **Flask app port 8090 at my.itpropartner.com/backups** | **Instant** |
-| voipsimplicity.com | Covered by MySQL + WordPress file backup | 24 hr |
-| my.voipsimplicity.com | Covered by Git (static HTML) | N/A |
+### On wphost02 (`/root/`)
+
+| Script | Purpose | Runs |
+|--------|---------|------|
+| `backup.sh` | All 7 WordPress sites + all MySQL DBs | Triggered via SSH from Core |

 ---

 ## S3 Bucket Structure

 ```
-s3://hermes-vps-backups/
-├── live/                          # 15-min Hermes state sync
-├── live-sync/                     # Live sync artifacts
-├── hermes-full-backup/            # Daily full Hermes backups
-├── snapshots/                     # Hermes snapshots
-├── standby/                       # Warm standby scripts
-├── root-backup/                   # /root essentials
+hermes-vps-backups/
+├── hermes-full-backup/     — Full Hermes daily (tar.gz)
+├── live/                   — Hermes live sync (every 15 min)
+├── live-sync/              — Old sync format (deprecated)
+├── root-backup/            — /root essentials
+├── standby/                — Standby configs + recovery bundle
 ├── core/
-│   ├── vaultwarden/
-│   ├── twenty/
-│   ├── searxng/
-│   └── komodo/
+│   ├── grafana/            — Grafana SQLite DB
+│   ├── uptime-kuma/        — Uptime Kuma SQLite DB
+│   ├── prometheus/         — Prometheus TSDB snapshots
+│   ├── vaultwarden/        — [STALE — service migrated to App1]
+│   ├── twenty/             — [STALE — service migrated to App1]
+│   ├── searxng/            — [STALE — service removed]
+│   └── komodo/             — [STALE — service migrated to App1]
 ├── app1/
-│   ├── litellm/
-│   ├── openwebui/
-│   ├── n8n/
-│   ├── mcp/
-│   └── ollama/
+│   ├── openwebui/          — Open WebUI data
+│   ├── litellm/            — LiteLLM Postgres dump + config
+│   ├── n8n/                — n8n Postgres dump
+│   ├── mcp/                — MCP server configs
+│   ├── vaultwarden/        — Vaultwarden SQLite dump
+│   ├── komodo/             — Komodo MongoDB dump
+│   ├── docuseal/           — DocuSeal SQLite + attachments
+│   └── twenty/             — Twenty CRM Postgres dump
 ├── app2/
-│   ├── traccar/
-│   ├── gitea/
-│   ├── hudu/
-│   ├── unms-backups/
-│   └── unifi-backups/
+│   └── traccar/            — Traccar H2 DB + config
 ├── app3/
-│   ├── cloudpanel/
-│   ├── mysql/
-│   ├── wordpress/
-│   └── config/
-├── volumes/                       # Docker volume syncs
-├── caddy/                         # Caddy configs
-├── env/                           # Environment files
-└── assets/                        # Static assets
+│   ├── cloudpanel/         — CloudPanel SQLite DB
+│   ├── mysql/              — All MySQL DBs
+│   ├── wordpress/          — wp-content tar.gz
+│   └── config/             — Nginx configs
+├── hudu/backups/           — Hudu Postgres dump
+├── gitea/daily/            — Gitea repos + DB
+├── unms-backups/live/      — UNMS auto-backups
+├── unifi-backups/          — UniFi controller backups
+├── wphost02-backup/        — wphost02 per-site + DB
+├── volumes/                — Docker volume dumps
+├── caddy/                  — (unused)
+├── assets/                 — Static assets
+├── snapshots/              — (unused)
+└── system-configs-*.tar.gz — Historical system config snapshots

-s3://itpropartner-backups/
-├── home-router/                   # MikroTik CCR configs
-└── shonuff/                       # Personal backups
-
-s3://mikrotik-ccr-backups/         # Home CCR2004 configs
+mikrotik-ccr-backups/
+└── wisp-backups/configs/
+    └── home/               — CCR2004 export .rsc
 ```

 ---

-## Restoration
+## Migration History (Jul 28, 2026)

-### Single Service Restore
-```bash
-# Example: restore LiteLLM database
-SERVICE=litellm; DATE=2026-07-16
-aws s3 cp s3://hermes-vps-backups/app1/$SERVICE/$SERVICE-$DATE.sql.gz . \
-  --endpoint-url https://s3.us-east-1.wasabisys.com
-gunzip $SERVICE-$DATE.sql.gz
-# Load into Postgres
-```
+All the following services were migrated from Core to App1 in a single session:

-### Full Server Restore
-Each server's backup directory contains everything needed to rebuild that server from scratch.
-
-### Warm Standby (Core only)
-core-bu (5.161.225.131) automatically syncs from S3 every 10 minutes. To activate:
-1. Power on via Hetzner Cloud API
-2. Hermes starts from latest S3 state
+| Service | Old Home | New Home | Backup Before | Backup After |
+|---------|----------|----------|--------------|-------------|
+| Vaultwarden | Core (:8080) | App1 (:8081) | `core/vaultwarden/` (stale) | `app1/vaultwarden/` ✅ |
+| SearXNG | Core (:8888) | Removed | `core/searxng/` (stale) | N/A (replaced by Super Search) |
+| Twenty CRM | Core (:3003) | App1 (:3003) | `core/twenty/` (stale) | `app1/twenty/` ✅ |
+| Komodo | Core (:9120) | App1 (:9120) | `core/komodo/` (stale) | `app1/komodo/` ✅ |
+| DocuSeal | Core (:3000) | App1 (:3002) | No backup existed | `app1/docuseal/` ✅ |
+| Kokoro TTS | Core (:8880) | App1 (:8880) | N/A (stateless) | N/A (stateless) |

 ---

-## Known Gaps
+## Disaster Recovery

-| Gap | Impact | Plan |
-|---|---|---|
-| No WAL archiving | RPO is 24 hours for most databases | Add WAL-G for Postgres services |
-| No restore testing | Don't know if backups actually restore | Schedule quarterly restore drill |
-| core-bu not tested since upgrade | Standby might not work | Schedule failover test |
-| wphost02 S3 upload in progress | First backup running now | Monitor completion |
+- **Live Hermes:** netcup VPS — `core.itpropartner.com` — `152.53.192.33`
+- **Standby Hermes:** Hetzner CPX21 — `app1-bu.itpropartner.com` — `5.161.225.131`
+  - Cron checks live box every 10 min, takes over if down
+  - Auto-restores from `s3://hermes-vps-backups/hermes-full-backup/`
+  - Provider diversity: netcup outage won't kill both Core and standby
+- **Recovery priority:** Hermes first → infrastructure monitoring (Uptime Kuma, Grafana) → App1 services → App2 services
+- **24 backup scripts** on Core, 1 on App3, 1 on wphost02