160 lines
5.8 KiB
Markdown
160 lines
5.8 KiB
Markdown
# Backup Plan — All Servers
|
|
|
|
**Last Updated:** July 16, 2026
|
|
**S3 Provider:** Wasabi (s3.us-east-1.wasabisys.com)
|
|
**Buckets:** hermes-vps-backups, itpropartner-backups, mikrotik-ccr-backups
|
|
**Versioning:** ON (all buckets)
|
|
|
|
---
|
|
|
|
## Schedule (ET timezone)
|
|
|
|
| Time | Server | Script | What |
|
|
|---|---|---|---|
|
|
| Every 15 min | Core | hermes-live-sync | Hermes session state → S3 live/ |
|
|
| 1:00 AM | Core | hermes-backup.sh | Full Hermes backup |
|
|
| 1:30 AM | Core | core-services-backup.sh | Vaultwarden, Twenty CRM, SearXNG, Komodo, Prometheus, Grafana, Uptime Kuma |
|
|
| 2:00 AM | Core | backup-audit-check.sh | Verify all backups |
|
|
| 2:00 AM | app1 | /root/.hermes/scripts/app1-backup.sh | LiteLLM, Open WebUI, n8n, MCP configs, Mattermost |
|
|
| 2:30 AM | app2 | /root/.hermes/scripts/app2-backup.sh | Traccar, Gitea, Hudu, UNMS, UniFi |
|
|
| 3:00 AM | Core | root-essentials-backup.sh | /root essentials |
|
|
| 3:00 AM | app3 | /root/.hermes/scripts/app3-backup.sh | CloudPanel, MySQL, WordPress, Nginx configs |
|
|
| 5:00 AM | Core | wphost02-backup (cron) | wphost02 webapps + databases → S3 |
|
|
| 1:00 AM + 1:00 PM | app3 | /opt/backup-restore/snapshot.sh | Per-site WordPress snapshots (files + DB), 60-day retention |
|
|
| 3:00 AM | Core | docker-volume-sync.sh | Docker volumes |
|
|
| 4:00 AM | Core | system-config-sync.sh | System configs |
|
|
| 6:00 AM | Core | run-wisp-backup.sh | MikroTik CCR config (via wisp-backup.py) |
|
|
| Every 10 min | core-bu | warm-standby-sync | Pull from S3 live/ → standby readiness |
|
|
|
|
---
|
|
|
|
## Coverage by Server
|
|
|
|
### Core (152.53.192.33)
|
|
|
|
| Service | Method | RPO |
|
|
|---|---|---|
|
|
| Hermes Agent | tar.gz → S3 (daily) + live sync (15 min) | 15 min |
|
|
| Vaultwarden | SQLite dump → S3 (daily) | 24 hr |
|
|
| Twenty CRM | Postgres pg_dump → S3 (daily) | 24 hr |
|
|
| SearXNG | settings.yml → S3 (daily) | 24 hr |
|
|
| Komodo | config dir → S3 (daily) | 24 hr |
|
|
| Prometheus | TSDB snapshot → S3 (daily) | 24 hr |
|
|
| Grafana | SQLite DB → S3 (daily) | 24 hr |
|
|
| Uptime Kuma | SQLite DB → S3 (daily) | 24 hr |
|
|
| DocuSeal | Covered by docker-volume-sync | 24 hr |
|
|
| Caddy config | system-config-sync (daily) | 24 hr |
|
|
| /root essentials | root-essentials-backup (daily) | 24 hr |
|
|
|
|
### app1 (152.53.36.131)
|
|
|
|
| Service | Method | RPO |
|
|
|---|---|---|
|
|
| LiteLLM | Postgres pg_dump + config.yaml → S3 (daily) | 24 hr |
|
|
| Open WebUI | /app/backend/data → S3 (daily) | 24 hr |
|
|
| n8n | Postgres pg_dump → S3 (daily) | 24 hr |
|
|
| MCP servers | Configs in Git + Docker volumes | 24 hr |
|
|
| Mattermost | Postgres DB + data volume → S3 (daily) | 24 hr |
|
|
|
|
### app2 (152.53.39.202)
|
|
|
|
| Service | Method | RPO |
|
|
|---|---|---|
|
|
| Traccar | H2 database + conf → S3 (daily) | 24 hr |
|
|
| Gitea | Repos + DB → S3 (daily) | 24 hr |
|
|
| Hudu | Docker volume → S3 (daily, 30-day retention) | 24 hr |
|
|
| UNMS | S3 sync (daily) | 24 hr |
|
|
| UniFi | Autobackup → S3 (daily) | 24 hr |
|
|
|
|
### app3 (152.53.241.111)
|
|
|
|
| Service | Method | RPO |
|
|
|---|---|---|
|
|
| CloudPanel CE | SQLite DB → S3 (daily) | 24 hr |
|
|
| MySQL | All databases mysqldump → S3 (daily) | 24 hr |
|
|
| WordPress | Files + wp-config → S3 (daily) | 24 hr |
|
|
| Nginx | /etc/nginx + /etc/cloudpanel → S3 (daily) | 24 hr |
|
|
| **WordPress Snapshots** | **Per-site tarball + MySQL dump (2x daily, 60-day retention)** | **12 hr** |
|
|
| **Backup Restore UI** | **Flask app port 8090 at my.itpropartner.com/backups** | **Instant** |
|
|
| voipsimplicity.com | Covered by MySQL + WordPress file backup | 24 hr |
|
|
| my.voipsimplicity.com | Covered by Git (static HTML) | N/A |
|
|
|
|
---
|
|
|
|
## S3 Bucket Structure
|
|
|
|
```
|
|
s3://hermes-vps-backups/
|
|
├── live/ # 15-min Hermes state sync
|
|
├── live-sync/ # Live sync artifacts
|
|
├── hermes-full-backup/ # Daily full Hermes backups
|
|
├── snapshots/ # Hermes snapshots
|
|
├── standby/ # Warm standby scripts
|
|
├── root-backup/ # /root essentials
|
|
├── core/
|
|
│ ├── vaultwarden/
|
|
│ ├── twenty/
|
|
│ ├── searxng/
|
|
│ └── komodo/
|
|
├── app1/
|
|
│ ├── litellm/
|
|
│ ├── openwebui/
|
|
│ ├── n8n/
|
|
│ ├── mcp/
|
|
│ └── ollama/
|
|
├── app2/
|
|
│ ├── traccar/
|
|
│ ├── gitea/
|
|
│ ├── hudu/
|
|
│ ├── unms-backups/
|
|
│ └── unifi-backups/
|
|
├── app3/
|
|
│ ├── cloudpanel/
|
|
│ ├── mysql/
|
|
│ ├── wordpress/
|
|
│ └── config/
|
|
├── volumes/ # Docker volume syncs
|
|
├── caddy/ # Caddy configs
|
|
├── env/ # Environment files
|
|
└── assets/ # Static assets
|
|
|
|
s3://itpropartner-backups/
|
|
├── home-router/ # MikroTik CCR configs
|
|
└── shonuff/ # Personal backups
|
|
|
|
s3://mikrotik-ccr-backups/ # Home CCR2004 configs
|
|
```
|
|
|
|
---
|
|
|
|
## Restoration
|
|
|
|
### Single Service Restore
|
|
```bash
|
|
# Example: restore LiteLLM database
|
|
SERVICE=litellm; DATE=2026-07-16
|
|
aws s3 cp s3://hermes-vps-backups/app1/$SERVICE/$SERVICE-$DATE.sql.gz . \
|
|
--endpoint-url https://s3.us-east-1.wasabisys.com
|
|
gunzip $SERVICE-$DATE.sql.gz
|
|
# Load into Postgres
|
|
```
|
|
|
|
### Full Server Restore
|
|
Each server's backup directory contains everything needed to rebuild that server from scratch.
|
|
|
|
### Warm Standby (Core only)
|
|
core-bu (5.161.225.131) automatically syncs from S3 every 10 minutes. To activate:
|
|
1. Power on via Hetzner Cloud API
|
|
2. Hermes starts from latest S3 state
|
|
|
|
---
|
|
|
|
## Known Gaps
|
|
|
|
| Gap | Impact | Plan |
|
|
|---|---|---|
|
|
| No WAL archiving | RPO is 24 hours for most databases | Add WAL-G for Postgres services |
|
|
| No restore testing | Don't know if backups actually restore | Schedule quarterly restore drill |
|
|
| core-bu not tested since upgrade | Standby might not work | Schedule failover test |
|
|
| wphost02 S3 upload in progress | First backup running now | Monitor completion |
|