itpp-infrastructure/backup-plan.md

ITPP Backup Plan

Last updated: 2026-08-08 Scope: All ITPP infrastructure backups — Hermes core, application servers, routers, and external services. Storage: Wasabi S3 (us-east-1) via --endpoint-url https://s3.us-east-1.wasabisys.com Cron backend: Hybrid — system crontab + Hermes cron jobs (both on Core) Verification audit: DR issue log

---

Backup Inventory (26 targets)

Core (152.53.192.33) — RS 2000

Service	Method	Destination	Schedule	Last Verified
Hermes Agent (full)	hermes-backup.sh — tar.gz of config, skills, profiles, sessions	s3://hermes-vps-backups/hermes-full-backup/	1:00 AM	2026-07-28
Hermes Live Sync	hermes-live-sync cron — session state + profiles DB	s3://hermes-vps-backups/live/	Every 15 min	2026-07-28 20:42
/root Essentials	root-essentials-backup.sh — dotfiles, keys, scripts	s3://hermes-vps-backups/root-backup/	3:00 AM	2026-07-28
Grafana	core-services-backup.sh — SQLite DB dump	s3://hermes-vps-backups/core/grafana/	1:30 AM	2026-07-28
Uptime Kuma	core-services-backup.sh — SQLite DB dump	s3://hermes-vps-backups/core/uptime-kuma/	1:30 AM	2026-07-28
Docker Volumes	core-services-backup.sh — tar of key compose volumes	s3://hermes-vps-backups/volumes/	1:30 AM	2026-07-28
Prometheus	core-services-backup.sh — TSDB snapshot	s3://hermes-vps-backups/core/prometheus/	1:30 AM	2026-07-28
Auth API	auth-api-backup.sh — SQLite .backup + .env	s3://hermes-vps-backups/core/auth-api/	3:15 AM	2026-08-08

App1 (152.53.36.131) — RS 4000

Service	Method	Destination	Schedule	Last Verified
Open WebUI	app1-backup.sh — data dir tar.gz	s3://hermes-vps-backups/app1/openwebui/	2:00 AM	2026-07-28
LiteLLM	litellm-backup.sh — Postgres DB dump + .env	s3://hermes-vps-backups/app1/litellm/	3:30 AM	2026-07-28
n8n	app1-backup.sh — Postgres DB dump	s3://hermes-vps-backups/app1/n8n/	2:00 AM	2026-07-28
MCP Server Configs	app1-backup.sh — MCP settings files	s3://hermes-vps-backups/app1/mcp/	2:00 AM	2026-07-28
Vaultwarden	vaultwarden-backup.sh — SQLite DB dump	s3://hermes-vps-backups/app1/vaultwarden/	2:30 AM	2026-07-28
Komodo	komodo-backup.sh — MongoDB dump + compose + keys	s3://hermes-vps-backups/app1/komodo/	3:45 AM	2026-07-28
DocuSeal	docuseal-backup.sh — SQLite DB + attachments	s3://hermes-vps-backups/app1/docuseal/	4:00 AM	2026-07-28
Twenty CRM	twenty-backup.sh — Postgres dump + .env + compose	s3://hermes-vps-backups/app1/twenty/	4:15 AM	2026-07-28
Kokoro TTS	(stateless — no backup needed)	—	—	—

App2 (152.53.39.202) — RS 4000

Service	Method	Destination	Schedule	Last Verified
Hudu	hudu-backup.sh — Postgres dump	s3://hermes-vps-backups/hudu/backups/	7:00 AM	2026-07-28
Gitea	gitea-backup.sh — repos + DB dump	s3://hermes-vps-backups/gitea/daily/	8:00 AM	2026-07-28
UNMS	unms-backup-sync.sh — auto-backup .unms	s3://hermes-vps-backups/unms-backups/live/backups/	6:00 AM	2026-07-28
UniFi	unifi-backup-sync.sh — auto-backup .unf	s3://hermes-vps-backups/unifi-backups/	2:00 AM	2026-07-18
Traccar	app2-backup.sh — H2 DB + config	s3://hermes-vps-backups/app2/traccar/	2:30 AM	2026-07-28
Technitium DNS	technitium-backup.sh — data dir + compose	s3://hermes-vps-backups/app2/technitium/	2:45 AM	2026-08-08
Dawarich	dawarich-backup.sh — PostgreSQL dump (remote SSH)	s3://hermes-vps-backups/app2/dawarich/	4:00 AM	2026-08-08
RAGFlow	ragflow-backup.sh — MySQL dump (remote SSH)	s3://hermes-vps-backups/app2/ragflow/	4:15 AM	2026-08-08

App3 (152.53.241.111) — RS 4000

Service	Method	Destination	Schedule	Last Verified
CloudPanel DB	app3-backup.sh — SQLite DB	s3://hermes-vps-backups/app3/cloudpanel/	3:00 AM	2026-07-28
MySQL (all DBs)	app3-backup.sh — mysqldump	s3://hermes-vps-backups/app3/mysql/	3:00 AM	2026-07-28
WordPress Files	app3-backup.sh — wp-content tar.gz	s3://hermes-vps-backups/app3/wordpress/	3:00 AM	2026-07-28
Nginx Configs	app3-backup.sh — sites-enabled + config	s3://hermes-vps-backups/app3/config/	3:00 AM	2026-07-28
WordPress Snapshots	/opt/backup-restore/snapshot.sh — per-site tar.gz	/opt/backup-restore/snapshots/ (local, 30-day retention)	1 AM / 1 PM	2026-08-08
Hexclave (Stack Auth)	hexclave-backup.sh — PG dump + compose + env	s3://hermes-vps-backups/app3/hexclave/	3:30 AM	2026-08-08

wphost02 (5.161.62.38) — Hetzner CPX21

Service	Method	Destination	Schedule	Last Verified
WordPress (7 sites)	ssh → /root/backup.sh — per-site tar.gz + all DBs	s3://hermes-vps-backups/wphost02-backup/	5:00 AM	2026-07-28

Home Router

Service	Method	Destination	Schedule	Last Verified
MikroTik CCR2004	run-wisp-backup.sh — export .rsc via SSH	s3://mikrotik-ccr-backups/wisp-backups/configs/home/	6:00 AM	2026-07-28

External

Service	Method	Destination	Schedule	Last Verified
Hetzner Snapshots	snapshot-hetzner.py — API-driven disk snapshots	Hetzner Cloud (weekly)	Mon 5:00 AM	—

---

Schedules Summary (ET)

Time	What	Runner	Script
Every 15 min	Hermes session state	Hermes cron	hermes-live-sync
1:00 AM	Full Hermes backup	crontab	hermes-backup.sh
1:30 AM	Grafana, Uptime Kuma, Docker volumes, Prometheus	crontab	core-services-backup.sh
2:00 AM	Open WebUI, n8n, MCP configs (App1) + UniFi sync (App2)	crontab	app1-backup.sh, unifi-backup-sync.sh
2:30 AM	Vaultwarden (App1) + Traccar (App2)	Hermes cron / crontab	vaultwarden-backup.sh, app2-backup.sh
2:45 AM	Technitium DNS (App2)	Hermes cron	technitium-backup.sh
3:00 AM	/root essentials + App3 (CloudPanel, MySQL, WP, Nginx)	crontab	root-essentials-backup.sh, app3-backup.sh
3:15 AM	Auth API (Core)	Hermes cron	auth-api-backup.sh
3:30 AM	Hexclave / Stack Auth (App3) + LiteLLM (App1)	Hermes cron / crontab	hexclave-backup.sh, litellm-backup.sh
3:45 AM	Komodo (App1)	crontab	komodo-backup.sh
4:00 AM	DocuSeal (App1)	Hermes cron	docuseal-backup.sh
4:15 AM	Twenty CRM (App1)	Hermes cron	twenty-backup.sh
5:00 AM	wphost02 WordPress (SSH)	crontab	wphost02-backup
6:00 AM	MikroTik CCR + UNMS sync	Hermes cron	run-wisp-backup.sh, unms-backup-sync.sh
1 AM / 1 PM	WordPress per-site snapshots (App3, local)	App3 crontab	/opt/backup-restore/snapshot.sh
7:00 AM	Hudu (App2)	Hermes cron	hudu-backup.sh
8:00 AM	Gitea (App2)	Hermes cron	gitea-backup.sh
Mon 5:00 AM	Hetzner weekly snapshots	Hermes cron	snapshot-hetzner.py

---

Script Inventory

On Core (/root/.hermes/scripts/)

Script	Purpose	Runs
hermes-backup.sh	Full Hermes tar.gz to S3	crontab 1:00 AM
core-services-backup.sh	Grafana, Uptime Kuma, volumes, Prometheus	crontab 1:30 AM
root-essentials-backup.sh	/root keys, configs, scripts	crontab 3:00 AM
backup-audit-check.sh	Verify recent backup timestamps	crontab 2:00 AM
vaultwarden-backup.sh	Vaultwarden SQLite dump (SSH to App1)	Hermes cron 2:30 AM
litellm-backup.sh	LiteLLM Postgres dump + config (SSH to App1)	Hermes cron 3:30 AM
komodo-backup.sh	Komodo MongoDB dump (SSH to App1)	Hermes cron 3:45 AM
docuseal-backup.sh	DocuSeal SQLite + attachments (SSH to App1)	Hermes cron 4:00 AM
twenty-backup.sh	Twenty CRM Postgres dump (SSH to App1)	Hermes cron 4:15 AM
app1-backup.sh	Open WebUI, n8n, MCP configs (SSH to App1)	App1 crontab 2:00 AM
app2-backup.sh	Traccar backup (SSH to App2)	App2 crontab 2:30 AM
technitium-backup.sh	Technitium DNS backup (SSH to App2)	Hermes cron 2:45 AM
app3-backup.sh	CloudPanel, MySQL, WP, Nginx (SSH to App3)	crontab 3:00 AM
auth-api-backup.sh	Auth API SQLite + config (Core local)	Hermes cron 3:15 AM
hexclave-backup.sh	Stack Auth PG dump + compose + env (SSH to App3)	Hermes cron 3:30 AM
dawarich-backup.sh	Dawarich PostgreSQL dump + compose + env (SSH to App2)	Hermes cron 4:00 AM
ragflow-backup.sh	RAGFlow MySQL dump + compose + env (SSH to App2)	Hermes cron 4:15 AM
wphost02-backup.sh	SSH to wphost02, backup all sites	crontab 5:00 AM
run-wisp-backup.sh	MikroTik CCR config export	Hermes cron 6:00 AM
unms-backup-sync.sh	UNMS auto-backup sync to S3	Hermes cron 6:00 AM
unifi-backup-sync.sh	UniFi auto-backup sync to S3	Hermes cron 2:00 AM
hudu-backup.sh	Hudu Postgres dump to S3	Hermes cron 7:00 AM
gitea-backup.sh	Gitea repos + DB dump to S3	Hermes cron 8:00 AM
snapshot-hetzner.py	Hetzner server disk snapshots via API	Hermes cron Mon 5:00 AM

On App3 (/opt/backup-restore/)

Script	Purpose	Runs
snapshot.sh	Per-site WordPress tarball + DB	App3 crontab 6 AM / 6 PM

On wphost02 (/root/)

Script	Purpose	Runs
backup.sh	All 7 WordPress sites + all MySQL DBs	Triggered via SSH from Core

---

S3 Bucket Structure

hermes-vps-backups/
├── hermes-full-backup/     — Full Hermes daily (tar.gz)
├── live/                   — Hermes live sync (every 15 min)
├── live-sync/              — Old sync format (deprecated)
├── root-backup/            — /root essentials
├── standby/                — Standby configs + recovery bundle
├── core/
│   ├── grafana/            — Grafana SQLite DB
│   ├── uptime-kuma/        — Uptime Kuma SQLite DB
│   ├── prometheus/         — Prometheus TSDB snapshots
│   ├── vaultwarden/        — [STALE — service migrated to App1]
│   ├── twenty/             — [STALE — service migrated to App1]
│   ├── searxng/            — [STALE — service removed]
│   └── komodo/             — [STALE — service migrated to App1]
├── app1/
│   ├── openwebui/          — Open WebUI data
│   ├── litellm/            — LiteLLM Postgres dump + config
│   ├── n8n/                — n8n Postgres dump
│   ├── mcp/                — MCP server configs
│   ├── vaultwarden/        — Vaultwarden SQLite dump
│   ├── komodo/             — Komodo MongoDB dump
│   ├── docuseal/           — DocuSeal SQLite + attachments
│   └── twenty/             — Twenty CRM Postgres dump
├── app2/
│   └── traccar/            — Traccar H2 DB + config
├── app3/
│   ├── cloudpanel/         — CloudPanel SQLite DB
│   ├── mysql/              — All MySQL DBs
│   ├── wordpress/          — wp-content tar.gz
│   └── config/             — Nginx configs
├── hudu/backups/           — Hudu Postgres dump
├── gitea/daily/            — Gitea repos + DB
├── unms-backups/live/      — UNMS auto-backups
├── unifi-backups/          — UniFi controller backups
├── wphost02-backup/        — wphost02 per-site + DB
├── volumes/                — Docker volume dumps
├── caddy/                  — (unused)
├── assets/                 — Static assets
├── snapshots/              — (unused)
└── system-configs-*.tar.gz — Historical system config snapshots

mikrotik-ccr-backups/
└── wisp-backups/configs/
    └── home/               — CCR2004 export .rsc

---

Recovery Objectives (RPO / RTO)

Tier	Services	RPO	RTO	Notes
Critical	Hermes Agent, Gitea, Traccar, UISP	≤ 1 hour	≤ 4 hours	Live sync + daily backups; restore from S3 then replay live-sync
High	LiteLLM, n8n, Open WebUI, Vaultwarden, Twenty CRM	24 hours	≤ 8 hours	Daily backups; restore from previous night's dump
	Medium	Hudu, UniFi, Komodo, DocuSeal, App3 WP sites, Auth API, Hexclave (Stack Auth)	24 hours	≤ 24 hours
	Low	Grafana, Uptime Kuma, Prometheus, MikroTik CCR, Technitium DNS, Dawarich, RAGFlow	24 hours	≤ 48 hours

Restore Testing Cadence

Quarterly: Pick one random backup per tier, restore to a staging location, verify integrity. After any major infra change: Test the affected service's restore path. Annual: Full DR simulation — restore all Critical + High tier services to staging from S3.

Stale S3 Paths — Cleanup Queue

These paths contain data from services that migrated off Core (Jul 28, 2026) or were removed:

Path	Status	Action
core/vaultwarden/	Stale 11 days	Safe to delete — Vaultwarden migrated to App1; new backups at app1/vaultwarden/
core/twenty/	Stale 11 days	Safe to delete — Twenty CRM migrated to App1; new backups at app1/twenty/
core/searxng/	Stale 11 days	Safe to delete — SearXNG removed; replaced by Super Search
core/komodo/	Stale 11 days	Safe to delete — Komodo migrated to App1; new backups at app1/komodo/
caddy/	Unused	Safe to delete — Never populated
snapshots/	Unused	Safe to delete — Never populated

Unbacked Services

These services are running in production with zero backup coverage: | Unbacked | (none) | N/A | N/A | All services are backed up as of 2026-08-08 |

All 14 previously-unbacked services are now backed up.

---

Migration History (Jul 28, 2026)

All the following services were migrated from Core to App1 in a single session:

Service	Old Home	New Home	Backup Before	Backup After
Vaultwarden	Core (:8080)	App1 (:8081)	core/vaultwarden/ (stale)	app1/vaultwarden/ ✅
SearXNG	Core (:8888)	Removed	core/searxng/ (stale)	N/A (replaced by Super Search)
Twenty CRM	Core (:3003)	App1 (:3003)	core/twenty/ (stale)	app1/twenty/ ✅
Komodo	Core (:9120)	App1 (:9120)	core/komodo/ (stale)	app1/komodo/ ✅
DocuSeal	Core (:3000)	App1 (:3002)	No backup existed	app1/docuseal/ ✅
Kokoro TTS	Core (:8880)	App1 (:8880)	N/A (stateless)	N/A (stateless)

---

Disaster Recovery

• Live Hermes: netcup VPS — core.itpropartner.com — 152.53.192.33
• Standby Hermes: Hetzner CPX21 — app1-bu.itpropartner.com — 5.161.225.131
  • Cron checks live box every 10 min, takes over if down
  • Auto-restores from s3://hermes-vps-backups/hermes-full-backup/
  • Provider diversity: netcup outage won't kill both Core and standby
• Recovery priority: Hermes first → infrastructure monitoring (Uptime Kuma, Grafana) → App1 services → App2 services
• 24 backup scripts on Core, 1 on App3, 1 on wphost02