Backup Plan — All Servers

Last Updated: July 16, 2026 S3 Provider: Wasabi (s3.us-east-1.wasabisys.com) Buckets: hermes-vps-backups, itpropartner-backups, mikrotik-ccr-backups Versioning: ON (all buckets)

Schedule (ET timezone)

Time	Server	Script	What
Every 15 min	Core	hermes-live-sync	Hermes session state → S3 live/
1:00 AM	Core	hermes-backup.sh	Full Hermes backup
1:30 AM	Core	core-services-backup.sh	Vaultwarden, Twenty CRM, SearXNG, Komodo, Prometheus, Grafana, Uptime Kuma
2:00 AM	Core	backup-audit-check.sh	Verify all backups
2:00 AM	app1	/root/.hermes/scripts/app1-backup.sh	LiteLLM, Open WebUI, n8n, MCP configs, Mattermost
2:30 AM	app2	/root/.hermes/scripts/app2-backup.sh	Traccar, Gitea, Hudu, UNMS, UniFi
3:00 AM	Core	root-essentials-backup.sh	/root essentials
3:00 AM	app3	/root/.hermes/scripts/app3-backup.sh	CloudPanel, MySQL, WordPress, Nginx configs
5:00 AM	Core	wphost02-backup (cron)	wphost02 webapps + databases → S3
1:00 AM + 1:00 PM	app3	/opt/backup-restore/snapshot.sh	Per-site WordPress snapshots (files + DB), 60-day retention
3:00 AM	Core	docker-volume-sync.sh	Docker volumes
4:00 AM	Core	system-config-sync.sh	System configs
6:00 AM	Core	run-wisp-backup.sh	MikroTik CCR config (via wisp-backup.py)
Every 10 min	core-bu	warm-standby-sync	Pull from S3 live/ → standby readiness

Coverage by Server

Core (152.53.192.33)

Service	Method	RPO
Hermes Agent	tar.gz → S3 (daily) + live sync (15 min)	15 min
Vaultwarden	SQLite dump → S3 (daily)	24 hr
Twenty CRM	Postgres pg_dump → S3 (daily)	24 hr
SearXNG	settings.yml → S3 (daily)	24 hr
Komodo	config dir → S3 (daily)	24 hr
Prometheus	TSDB snapshot → S3 (daily)	24 hr
Grafana	SQLite DB → S3 (daily)	24 hr
Uptime Kuma	SQLite DB → S3 (daily)	24 hr
DocuSeal	Covered by docker-volume-sync	24 hr
Caddy config	system-config-sync (daily)	24 hr
/root essentials	root-essentials-backup (daily)	24 hr

app1 (152.53.36.131)

Service	Method	RPO
LiteLLM	Postgres pg_dump + config.yaml → S3 (daily)	24 hr
Open WebUI	/app/backend/data → S3 (daily)	24 hr
n8n	Postgres pg_dump → S3 (daily)	24 hr
MCP servers	Configs in Git + Docker volumes	24 hr
Mattermost	Postgres DB + data volume → S3 (daily)	24 hr

app2 (152.53.39.202)

Service	Method	RPO
Traccar	H2 database + conf → S3 (daily)	24 hr
Gitea	Repos + DB → S3 (daily)	24 hr
Hudu	Docker volume → S3 (daily, 30-day retention)	24 hr
UNMS	S3 sync (daily)	24 hr
UniFi	Autobackup → S3 (daily)	24 hr

app3 (152.53.241.111)

Service	Method	RPO
CloudPanel CE	SQLite DB → S3 (daily)	24 hr
MySQL	All databases mysqldump → S3 (daily)	24 hr
WordPress	Files + wp-config → S3 (daily)	24 hr
Nginx	/etc/nginx + /etc/cloudpanel → S3 (daily)	24 hr
WordPress Snapshots	Per-site tarball + MySQL dump (2x daily, 60-day retention)	12 hr
Backup Restore UI	Flask app port 8090 at my.itpropartner.com/backups	Instant
voipsimplicity.com	Covered by MySQL + WordPress file backup	24 hr
my.voipsimplicity.com	Covered by Git (static HTML)	N/A

S3 Bucket Structure

s3://hermes-vps-backups/
├── live/                          # 15-min Hermes state sync
├── live-sync/                     # Live sync artifacts
├── hermes-full-backup/            # Daily full Hermes backups
├── snapshots/                     # Hermes snapshots
├── standby/                       # Warm standby scripts
├── root-backup/                   # /root essentials
├── core/
│   ├── vaultwarden/
│   ├── twenty/
│   ├── searxng/
│   └── komodo/
├── app1/
│   ├── litellm/
│   ├── openwebui/
│   ├── n8n/
│   ├── mcp/
│   └── ollama/
├── app2/
│   ├── traccar/
│   ├── gitea/
│   ├── hudu/
│   ├── unms-backups/
│   └── unifi-backups/
├── app3/
│   ├── cloudpanel/
│   ├── mysql/
│   ├── wordpress/
│   └── config/
├── volumes/                       # Docker volume syncs
├── caddy/                         # Caddy configs
├── env/                           # Environment files
└── assets/                        # Static assets

s3://itpropartner-backups/
├── home-router/                   # MikroTik CCR configs
└── shonuff/                       # Personal backups

s3://mikrotik-ccr-backups/         # Home CCR2004 configs

Restoration

Single Service Restore

# Example: restore LiteLLM database
SERVICE=litellm; DATE=2026-07-16
aws s3 cp s3://hermes-vps-backups/app1/$SERVICE/$SERVICE-$DATE.sql.gz . \
  --endpoint-url https://s3.us-east-1.wasabisys.com
gunzip $SERVICE-$DATE.sql.gz
# Load into Postgres

Full Server Restore

Each server's backup directory contains everything needed to rebuild that server from scratch.

Warm Standby (Core only)

core-bu (5.161.225.131) automatically syncs from S3 every 10 minutes. To activate:

Power on via Hetzner Cloud API
Hermes starts from latest S3 state

Known Gaps

Gap	Impact	Plan
No WAL archiving	RPO is 24 hours for most databases	Add WAL-G for Postgres services
No restore testing	Don't know if backups actually restore	Schedule quarterly restore drill
core-bu not tested since upgrade	Standby might not work	Schedule failover test
wphost02 S3 upload in progress	First backup running now	Monitor completion

5.8 KiB Raw Blame History