# Backup Plan — All Servers

**Last Updated:** July 16, 2026
**S3 Provider:** Wasabi (s3.us-east-1.wasabisys.com)
**Buckets:** hermes-vps-backups, itpropartner-backups, mikrotik-ccr-backups
**Versioning:** ON (all buckets)

---

## Schedule (ET timezone)

| Time | Server | Script | What |
|---|---|---|---|
| Every 15 min | Core | hermes-live-sync | Hermes session state → S3 live/ |
| 1:00 AM | Core | hermes-backup.sh | Full Hermes backup |
| 1:30 AM | Core | core-services-backup.sh | Vaultwarden, Twenty CRM, SearXNG, Komodo, Prometheus, Grafana, Uptime Kuma |
| 2:00 AM | Core | backup-audit-check.sh | Verify all backups |
| 2:00 AM | app1 | /root/.hermes/scripts/app1-backup.sh | LiteLLM, Open WebUI, n8n, MCP configs, Mattermost |
| 2:30 AM | app2 | /root/.hermes/scripts/app2-backup.sh | Traccar, Gitea, Hudu, UNMS, UniFi |
| 3:00 AM | Core | root-essentials-backup.sh | /root essentials |
| 3:00 AM | app3 | /root/.hermes/scripts/app3-backup.sh | CloudPanel, MySQL, WordPress, Nginx configs |
| 5:00 AM | Core | wphost02-backup (cron) | wphost02 webapps + databases → S3 |
| 1:00 AM + 1:00 PM | app3 | /opt/backup-restore/snapshot.sh | Per-site WordPress snapshots (files + DB), 60-day retention |
| 3:00 AM | Core | docker-volume-sync.sh | Docker volumes |
| 4:00 AM | Core | system-config-sync.sh | System configs |
| 6:00 AM | Core | run-wisp-backup.sh | MikroTik CCR config (via wisp-backup.py) |
| Every 10 min | core-bu | warm-standby-sync | Pull from S3 live/ → standby readiness |

---

## Coverage by Server

### Core (152.53.192.33)

| Service | Method | RPO |
|---|---|---|
| Hermes Agent | tar.gz → S3 (daily) + live sync (15 min) | 15 min |
| Vaultwarden | SQLite dump → S3 (daily) | 24 hr |
| Twenty CRM | Postgres pg_dump → S3 (daily) | 24 hr |
| SearXNG | settings.yml → S3 (daily) | 24 hr |
| Komodo | config dir → S3 (daily) | 24 hr |
| Prometheus | TSDB snapshot → S3 (daily) | 24 hr |
| Grafana | SQLite DB → S3 (daily) | 24 hr |
| Uptime Kuma | SQLite DB → S3 (daily) | 24 hr |
| DocuSeal | Covered by docker-volume-sync | 24 hr |
| Caddy config | system-config-sync (daily) | 24 hr |
| /root essentials | root-essentials-backup (daily) | 24 hr |

### app1 (152.53.36.131)

| Service | Method | RPO |
|---|---|---|
| LiteLLM | Postgres pg_dump + config.yaml → S3 (daily) | 24 hr |
| Open WebUI | /app/backend/data → S3 (daily) | 24 hr |
| n8n | Postgres pg_dump → S3 (daily) | 24 hr |
| MCP servers | Configs in Git + Docker volumes | 24 hr |
| Mattermost | Postgres DB + data volume → S3 (daily) | 24 hr |

### app2 (152.53.39.202)

| Service | Method | RPO |
|---|---|---|
| Traccar | H2 database + conf → S3 (daily) | 24 hr |
| Gitea | Repos + DB → S3 (daily) | 24 hr |
| Hudu | Docker volume → S3 (daily, 30-day retention) | 24 hr |
| UNMS | S3 sync (daily) | 24 hr |
| UniFi | Autobackup → S3 (daily) | 24 hr |

### app3 (152.53.241.111)

| Service | Method | RPO |
|---|---|---|
| CloudPanel CE | SQLite DB → S3 (daily) | 24 hr |
| MySQL | All databases mysqldump → S3 (daily) | 24 hr |
| WordPress | Files + wp-config → S3 (daily) | 24 hr |
| Nginx | /etc/nginx + /etc/cloudpanel → S3 (daily) | 24 hr |
| **WordPress Snapshots** | **Per-site tarball + MySQL dump (2x daily, 60-day retention)** | **12 hr** |
| **Backup Restore UI** | **Flask app port 8090 at my.itpropartner.com/backups** | **Instant** |
| voipsimplicity.com | Covered by MySQL + WordPress file backup | 24 hr |
| my.voipsimplicity.com | Covered by Git (static HTML) | N/A |

---

## S3 Bucket Structure

```
s3://hermes-vps-backups/
├── live/                          # 15-min Hermes state sync
├── live-sync/                     # Live sync artifacts
├── hermes-full-backup/            # Daily full Hermes backups
├── snapshots/                     # Hermes snapshots
├── standby/                       # Warm standby scripts
├── root-backup/                   # /root essentials
├── core/
│   ├── vaultwarden/
│   ├── twenty/
│   ├── searxng/
│   └── komodo/
├── app1/
│   ├── litellm/
│   ├── openwebui/
│   ├── n8n/
│   ├── mcp/
│   └── ollama/
├── app2/
│   ├── traccar/
│   ├── gitea/
│   ├── hudu/
│   ├── unms-backups/
│   └── unifi-backups/
├── app3/
│   ├── cloudpanel/
│   ├── mysql/
│   ├── wordpress/
│   └── config/
├── volumes/                       # Docker volume syncs
├── caddy/                         # Caddy configs
├── env/                           # Environment files
└── assets/                        # Static assets

s3://itpropartner-backups/
├── home-router/                   # MikroTik CCR configs
└── shonuff/                       # Personal backups

s3://mikrotik-ccr-backups/         # Home CCR2004 configs
```

---

## Restoration

### Single Service Restore
```bash
# Example: restore LiteLLM database
SERVICE=litellm; DATE=2026-07-16
aws s3 cp s3://hermes-vps-backups/app1/$SERVICE/$SERVICE-$DATE.sql.gz . \
  --endpoint-url https://s3.us-east-1.wasabisys.com
gunzip $SERVICE-$DATE.sql.gz
# Load into Postgres
```

### Full Server Restore
Each server's backup directory contains everything needed to rebuild that server from scratch.

### Warm Standby (Core only)
core-bu (5.161.225.131) automatically syncs from S3 every 10 minutes. To activate:
1. Power on via Hetzner Cloud API
2. Hermes starts from latest S3 state

---

## Known Gaps

| Gap | Impact | Plan |
|---|---|---|
| No WAL archiving | RPO is 24 hours for most databases | Add WAL-G for Postgres services |
| No restore testing | Don't know if backups actually restore | Schedule quarterly restore drill |
| core-bu not tested since upgrade | Standby might not work | Schedule failover test |
| wphost02 S3 upload in progress | First backup running now | Monitor completion |