From d2643a5e7cf580fdb6e0dbd5ad226c8715d0066e Mon Sep 17 00:00:00 2001 From: root Date: Thu, 16 Jul 2026 21:51:01 -0400 Subject: [PATCH] Complete backup plan: app1/app2/app3 daily backups scheduled, Core services covered, S3 structure documented --- backup-plan.md | 155 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 155 insertions(+) create mode 100644 backup-plan.md diff --git a/backup-plan.md b/backup-plan.md new file mode 100644 index 0000000..f0814e0 --- /dev/null +++ b/backup-plan.md @@ -0,0 +1,155 @@ +# Backup Plan — All Servers + +**Last Updated:** July 16, 2026 +**S3 Provider:** Wasabi (s3.us-east-1.wasabisys.com) +**Buckets:** hermes-vps-backups, itpropartner-backups, mikrotik-ccr-backups +**Versioning:** ON (all buckets) + +--- + +## Schedule (ET timezone) + +| Time | Server | Script | What | +|---|---|---|---| +| Every 15 min | Core | hermes-live-sync | Hermes session state → S3 live/ | +| 1:00 AM | Core | hermes-backup.sh | Full Hermes backup | +| 1:30 AM | Core | core-services-backup.sh | Vaultwarden, Twenty CRM, SearXNG, Komodo, Prometheus | +| 2:00 AM | Core | backup-audit-check.sh | Verify all backups | +| 2:00 AM | app1 | /root/backup.sh | LiteLLM, Open WebUI, n8n, MCPs, Ollama | +| 2:30 AM | app2 | /root/backup.sh | Traccar, Gitea, Hudu, UNMS, UniFi | +| 3:00 AM | Core | root-essentials-backup.sh | /root essentials | +| 3:00 AM | app3 | /root/backup.sh | CloudPanel, MySQL, WordPress, Nginx configs | +| 3:00 AM | Core | docker-volume-sync.sh | Docker volumes | +| 4:00 AM | Core | system-config-sync.sh | System configs | +| 6:00 AM | Core | home-router-backup.sh | MikroTik CCR config | +| Every 10 min | core-bu | warm-standby-sync | Pull from S3 live/ → standby readiness | + +--- + +## Coverage by Server + +### Core (152.53.192.33) + +| Service | Method | RPO | +|---|---|---| +| Hermes Agent | tar.gz → S3 (daily) + live sync (15 min) | 15 min | +| Vaultwarden | SQLite dump → S3 (daily) | 24 hr | +| Twenty CRM | Postgres pg_dump → S3 (daily) | 24 hr | +| SearXNG | settings.yml → S3 (daily) | 24 hr | +| Komodo | config dir → S3 (daily) | 24 hr | +| Prometheus | TSDB snapshot (daily) | 24 hr | +| Grafana | Covered by docker-volume-sync | 24 hr | +| Uptime Kuma | Covered by docker-volume-sync | 24 hr | +| DocuSeal | Covered by docker-volume-sync | 24 hr | +| Caddy config | system-config-sync (daily) | 24 hr | +| /root essentials | root-essentials-backup (daily) | 24 hr | + +### app1 (152.53.36.131) + +| Service | Method | RPO | +|---|---|---| +| LiteLLM | Postgres pg_dump + config.yaml → S3 (daily) | 24 hr | +| Open WebUI | /app/backend/data → S3 (daily) | 24 hr | +| n8n | Postgres pg_dump → S3 (daily) | 24 hr | +| MCP servers | Config files → S3 (daily) | 24 hr | +| Ollama | Model list → S3 (daily) | 24 hr | + +### app2 (152.53.39.202) + +| Service | Method | RPO | +|---|---|---| +| Traccar | H2 database + conf → S3 (daily) | 24 hr | +| Gitea | Repos + DB → S3 (daily) | 24 hr | +| Hudu | Docker volume → S3 (daily, 30-day retention) | 24 hr | +| UNMS | S3 sync (daily) | 24 hr | +| UniFi | Autobackup → S3 (daily) | 24 hr | + +### app3 (152.53.241.111) + +| Service | Method | RPO | +|---|---|---| +| CloudPanel CE | SQLite DB → S3 (daily) | 24 hr | +| MySQL | All databases mysqldump → S3 (daily) | 24 hr | +| WordPress | Files + wp-config → S3 (daily) | 24 hr | +| Nginx | /etc/nginx + /etc/cloudpanel → S3 (daily) | 24 hr | +| voipsimplicity.com | Covered by MySQL + WordPress file backup | 24 hr | +| my.voipsimplicity.com | Covered by Git (static HTML) | N/A | + +--- + +## S3 Bucket Structure + +``` +s3://hermes-vps-backups/ +├── live/ # 15-min Hermes state sync +├── live-sync/ # Live sync artifacts +├── hermes-full-backup/ # Daily full Hermes backups +├── snapshots/ # Hermes snapshots +├── standby/ # Warm standby scripts +├── root-backup/ # /root essentials +├── core/ +│ ├── vaultwarden/ +│ ├── twenty/ +│ ├── searxng/ +│ └── komodo/ +├── app1/ +│ ├── litellm/ +│ ├── openwebui/ +│ ├── n8n/ +│ ├── mcp/ +│ └── ollama/ +├── app2/ +│ ├── traccar/ +│ ├── gitea/ +│ ├── hudu/ +│ ├── unms-backups/ +│ └── unifi-backups/ +├── app3/ +│ ├── cloudpanel/ +│ ├── mysql/ +│ ├── wordpress/ +│ └── config/ +├── volumes/ # Docker volume syncs +├── caddy/ # Caddy configs +├── env/ # Environment files +└── assets/ # Static assets + +s3://itpropartner-backups/ +├── home-router/ # MikroTik CCR configs +└── shonuff/ # Personal backups + +s3://mikrotik-ccr-backups/ # Home CCR2004 configs +``` + +--- + +## Restoration + +### Single Service Restore +```bash +# Example: restore LiteLLM database +SERVICE=litellm; DATE=2026-07-16 +aws s3 cp s3://hermes-vps-backups/app1/$SERVICE/$SERVICE-$DATE.sql.gz . \ + --endpoint-url https://s3.us-east-1.wasabisys.com +gunzip $SERVICE-$DATE.sql.gz +# Load into Postgres +``` + +### Full Server Restore +Each server's backup directory contains everything needed to rebuild that server from scratch. + +### Warm Standby (Core only) +core-bu (5.161.225.131) automatically syncs from S3 every 10 minutes. To activate: +1. Power on via Hetzner Cloud API +2. Hermes starts from latest S3 state + +--- + +## Known Gaps + +| Gap | Impact | Plan | +|---|---|---| +| No WAL archiving | RPO is 24 hours for most databases | Add WAL-G for Postgres services | +| Ollama models not backed up | 4-8 GB models would need re-download | Low priority — models are reproducible | +| No restore testing | Don't know if backups actually restore | Schedule quarterly restore drill | +| core-bu not tested since upgrade | Standby might not work | Schedule failover test |