Complete backup plan: app1/app2/app3 daily backups scheduled, Core services covered, S3 structure documented
This commit is contained in:
root
+155
@@ -0,0 +1,155 @@
|
|||||||
|
# Backup Plan — All Servers
|
||||||
|
|
||||||
|
**Last Updated:** July 16, 2026
|
||||||
|
**S3 Provider:** Wasabi (s3.us-east-1.wasabisys.com)
|
||||||
|
**Buckets:** hermes-vps-backups, itpropartner-backups, mikrotik-ccr-backups
|
||||||
|
**Versioning:** ON (all buckets)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Schedule (ET timezone)
|
||||||
|
|
||||||
|
| Time | Server | Script | What |
|
||||||
|
|---|---|---|---|
|
||||||
|
| Every 15 min | Core | hermes-live-sync | Hermes session state → S3 live/ |
|
||||||
|
| 1:00 AM | Core | hermes-backup.sh | Full Hermes backup |
|
||||||
|
| 1:30 AM | Core | core-services-backup.sh | Vaultwarden, Twenty CRM, SearXNG, Komodo, Prometheus |
|
||||||
|
| 2:00 AM | Core | backup-audit-check.sh | Verify all backups |
|
||||||
|
| 2:00 AM | app1 | /root/backup.sh | LiteLLM, Open WebUI, n8n, MCPs, Ollama |
|
||||||
|
| 2:30 AM | app2 | /root/backup.sh | Traccar, Gitea, Hudu, UNMS, UniFi |
|
||||||
|
| 3:00 AM | Core | root-essentials-backup.sh | /root essentials |
|
||||||
|
| 3:00 AM | app3 | /root/backup.sh | CloudPanel, MySQL, WordPress, Nginx configs |
|
||||||
|
| 3:00 AM | Core | docker-volume-sync.sh | Docker volumes |
|
||||||
|
| 4:00 AM | Core | system-config-sync.sh | System configs |
|
||||||
|
| 6:00 AM | Core | home-router-backup.sh | MikroTik CCR config |
|
||||||
|
| Every 10 min | core-bu | warm-standby-sync | Pull from S3 live/ → standby readiness |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Coverage by Server
|
||||||
|
|
||||||
|
### Core (152.53.192.33)
|
||||||
|
|
||||||
|
| Service | Method | RPO |
|
||||||
|
|---|---|---|
|
||||||
|
| Hermes Agent | tar.gz → S3 (daily) + live sync (15 min) | 15 min |
|
||||||
|
| Vaultwarden | SQLite dump → S3 (daily) | 24 hr |
|
||||||
|
| Twenty CRM | Postgres pg_dump → S3 (daily) | 24 hr |
|
||||||
|
| SearXNG | settings.yml → S3 (daily) | 24 hr |
|
||||||
|
| Komodo | config dir → S3 (daily) | 24 hr |
|
||||||
|
| Prometheus | TSDB snapshot (daily) | 24 hr |
|
||||||
|
| Grafana | Covered by docker-volume-sync | 24 hr |
|
||||||
|
| Uptime Kuma | Covered by docker-volume-sync | 24 hr |
|
||||||
|
| DocuSeal | Covered by docker-volume-sync | 24 hr |
|
||||||
|
| Caddy config | system-config-sync (daily) | 24 hr |
|
||||||
|
| /root essentials | root-essentials-backup (daily) | 24 hr |
|
||||||
|
|
||||||
|
### app1 (152.53.36.131)
|
||||||
|
|
||||||
|
| Service | Method | RPO |
|
||||||
|
|---|---|---|
|
||||||
|
| LiteLLM | Postgres pg_dump + config.yaml → S3 (daily) | 24 hr |
|
||||||
|
| Open WebUI | /app/backend/data → S3 (daily) | 24 hr |
|
||||||
|
| n8n | Postgres pg_dump → S3 (daily) | 24 hr |
|
||||||
|
| MCP servers | Config files → S3 (daily) | 24 hr |
|
||||||
|
| Ollama | Model list → S3 (daily) | 24 hr |
|
||||||
|
|
||||||
|
### app2 (152.53.39.202)
|
||||||
|
|
||||||
|
| Service | Method | RPO |
|
||||||
|
|---|---|---|
|
||||||
|
| Traccar | H2 database + conf → S3 (daily) | 24 hr |
|
||||||
|
| Gitea | Repos + DB → S3 (daily) | 24 hr |
|
||||||
|
| Hudu | Docker volume → S3 (daily, 30-day retention) | 24 hr |
|
||||||
|
| UNMS | S3 sync (daily) | 24 hr |
|
||||||
|
| UniFi | Autobackup → S3 (daily) | 24 hr |
|
||||||
|
|
||||||
|
### app3 (152.53.241.111)
|
||||||
|
|
||||||
|
| Service | Method | RPO |
|
||||||
|
|---|---|---|
|
||||||
|
| CloudPanel CE | SQLite DB → S3 (daily) | 24 hr |
|
||||||
|
| MySQL | All databases mysqldump → S3 (daily) | 24 hr |
|
||||||
|
| WordPress | Files + wp-config → S3 (daily) | 24 hr |
|
||||||
|
| Nginx | /etc/nginx + /etc/cloudpanel → S3 (daily) | 24 hr |
|
||||||
|
| voipsimplicity.com | Covered by MySQL + WordPress file backup | 24 hr |
|
||||||
|
| my.voipsimplicity.com | Covered by Git (static HTML) | N/A |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## S3 Bucket Structure
|
||||||
|
|
||||||
|
```
|
||||||
|
s3://hermes-vps-backups/
|
||||||
|
├── live/ # 15-min Hermes state sync
|
||||||
|
├── live-sync/ # Live sync artifacts
|
||||||
|
├── hermes-full-backup/ # Daily full Hermes backups
|
||||||
|
├── snapshots/ # Hermes snapshots
|
||||||
|
├── standby/ # Warm standby scripts
|
||||||
|
├── root-backup/ # /root essentials
|
||||||
|
├── core/
|
||||||
|
│ ├── vaultwarden/
|
||||||
|
│ ├── twenty/
|
||||||
|
│ ├── searxng/
|
||||||
|
│ └── komodo/
|
||||||
|
├── app1/
|
||||||
|
│ ├── litellm/
|
||||||
|
│ ├── openwebui/
|
||||||
|
│ ├── n8n/
|
||||||
|
│ ├── mcp/
|
||||||
|
│ └── ollama/
|
||||||
|
├── app2/
|
||||||
|
│ ├── traccar/
|
||||||
|
│ ├── gitea/
|
||||||
|
│ ├── hudu/
|
||||||
|
│ ├── unms-backups/
|
||||||
|
│ └── unifi-backups/
|
||||||
|
├── app3/
|
||||||
|
│ ├── cloudpanel/
|
||||||
|
│ ├── mysql/
|
||||||
|
│ ├── wordpress/
|
||||||
|
│ └── config/
|
||||||
|
├── volumes/ # Docker volume syncs
|
||||||
|
├── caddy/ # Caddy configs
|
||||||
|
├── env/ # Environment files
|
||||||
|
└── assets/ # Static assets
|
||||||
|
|
||||||
|
s3://itpropartner-backups/
|
||||||
|
├── home-router/ # MikroTik CCR configs
|
||||||
|
└── shonuff/ # Personal backups
|
||||||
|
|
||||||
|
s3://mikrotik-ccr-backups/ # Home CCR2004 configs
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Restoration
|
||||||
|
|
||||||
|
### Single Service Restore
|
||||||
|
```bash
|
||||||
|
# Example: restore LiteLLM database
|
||||||
|
SERVICE=litellm; DATE=2026-07-16
|
||||||
|
aws s3 cp s3://hermes-vps-backups/app1/$SERVICE/$SERVICE-$DATE.sql.gz . \
|
||||||
|
--endpoint-url https://s3.us-east-1.wasabisys.com
|
||||||
|
gunzip $SERVICE-$DATE.sql.gz
|
||||||
|
# Load into Postgres
|
||||||
|
```
|
||||||
|
|
||||||
|
### Full Server Restore
|
||||||
|
Each server's backup directory contains everything needed to rebuild that server from scratch.
|
||||||
|
|
||||||
|
### Warm Standby (Core only)
|
||||||
|
core-bu (5.161.225.131) automatically syncs from S3 every 10 minutes. To activate:
|
||||||
|
1. Power on via Hetzner Cloud API
|
||||||
|
2. Hermes starts from latest S3 state
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Known Gaps
|
||||||
|
|
||||||
|
| Gap | Impact | Plan |
|
||||||
|
|---|---|---|
|
||||||
|
| No WAL archiving | RPO is 24 hours for most databases | Add WAL-G for Postgres services |
|
||||||
|
| Ollama models not backed up | 4-8 GB models would need re-download | Low priority — models are reproducible |
|
||||||
|
| No restore testing | Don't know if backups actually restore | Schedule quarterly restore drill |
|
||||||
|
| core-bu not tested since upgrade | Standby might not work | Schedule failover test |
|
||||||
Reference in New Issue
Block a user