Files

T

root 81762a33bd CRITICAL #4 : Auth API + Stack Auth backups — auth-api-backup.sh (SQLite hot), hexclave-backup.sh (PG dump + compose). Auth API Core 3:15 AM, Hexclave app3 3:30 AM. Update RPO/RTO tiers. Remove Technitium from Unbacked (now in Low tier).

2026-08-08 18:20:03 -04:00

16 KiB

Raw Blame History

ITPP Backup Plan

Last updated: 2026-08-08 Scope: All ITPP infrastructure backups — Hermes core, application servers, routers, and external services. Storage: Wasabi S3 (us-east-1) via --endpoint-url https://s3.us-east-1.wasabisys.com Cron backend: Hybrid — system crontab + Hermes cron jobs (both on Core) Verification audit: DR issue log

Backup Inventory (26 targets)

Core (152.53.192.33) — RS 2000

Service	Method	Destination	Schedule	Last Verified
Hermes Agent (full)	`hermes-backup.sh` — tar.gz of config, skills, profiles, sessions	`s3://hermes-vps-backups/hermes-full-backup/`	1:00 AM	2026-07-28
Hermes Live Sync	`hermes-live-sync` cron — session state + profiles DB	`s3://hermes-vps-backups/live/`	Every 15 min	2026-07-28 20:42
/root Essentials	`root-essentials-backup.sh` — dotfiles, keys, scripts	`s3://hermes-vps-backups/root-backup/`	3:00 AM	2026-07-28
Grafana	`core-services-backup.sh` — SQLite DB dump	`s3://hermes-vps-backups/core/grafana/`	1:30 AM	2026-07-28
Uptime Kuma	`core-services-backup.sh` — SQLite DB dump	`s3://hermes-vps-backups/core/uptime-kuma/`	1:30 AM	2026-07-28
Docker Volumes	`core-services-backup.sh` — tar of key compose volumes	`s3://hermes-vps-backups/volumes/`	1:30 AM	2026-07-28
Prometheus	`core-services-backup.sh` — TSDB snapshot	`s3://hermes-vps-backups/core/prometheus/`	1:30 AM	2026-07-28
Auth API	`auth-api-backup.sh` — SQLite .backup + .env	`s3://hermes-vps-backups/core/auth-api/`	3:15 AM	2026-08-08

App1 (152.53.36.131) — RS 4000

Service	Method	Destination	Schedule	Last Verified
Open WebUI	`app1-backup.sh` — data dir tar.gz	`s3://hermes-vps-backups/app1/openwebui/`	2:00 AM	2026-07-28
LiteLLM	`litellm-backup.sh` — Postgres DB dump + .env	`s3://hermes-vps-backups/app1/litellm/`	3:30 AM	2026-07-28
n8n	`app1-backup.sh` — Postgres DB dump	`s3://hermes-vps-backups/app1/n8n/`	2:00 AM	2026-07-28
MCP Server Configs	`app1-backup.sh` — MCP settings files	`s3://hermes-vps-backups/app1/mcp/`	2:00 AM	2026-07-28
Vaultwarden	`vaultwarden-backup.sh` — SQLite DB dump	`s3://hermes-vps-backups/app1/vaultwarden/`	2:30 AM	2026-07-28
Komodo	`komodo-backup.sh` — MongoDB dump + compose + keys	`s3://hermes-vps-backups/app1/komodo/`	3:45 AM	2026-07-28
DocuSeal	`docuseal-backup.sh` — SQLite DB + attachments	`s3://hermes-vps-backups/app1/docuseal/`	4:00 AM	2026-07-28
Twenty CRM	`twenty-backup.sh` — Postgres dump + .env + compose	`s3://hermes-vps-backups/app1/twenty/`	4:15 AM	2026-07-28
Kokoro TTS	(stateless — no backup needed)	—	—	—

App2 (152.53.39.202) — RS 4000

Service	Method	Destination	Schedule	Last Verified
Hudu	`hudu-backup.sh` — Postgres dump	`s3://hermes-vps-backups/hudu/backups/`	7:00 AM	2026-07-28
Gitea	`gitea-backup.sh` — repos + DB dump	`s3://hermes-vps-backups/gitea/daily/`	8:00 AM	2026-07-28
UNMS	`unms-backup-sync.sh` — auto-backup .unms	`s3://hermes-vps-backups/unms-backups/live/backups/`	6:00 AM	2026-07-28
UniFi	`unifi-backup-sync.sh` — auto-backup .unf	`s3://hermes-vps-backups/unifi-backups/`	2:00 AM	2026-07-18
Traccar	`app2-backup.sh` — H2 DB + config	`s3://hermes-vps-backups/app2/traccar/`	2:30 AM	2026-07-28
Technitium DNS	`technitium-backup.sh` — data dir + compose	`s3://hermes-vps-backups/app2/technitium/`	2:45 AM	2026-08-08

App3 (152.53.241.111) — RS 4000

Service	Method	Destination	Schedule	Last Verified
CloudPanel DB	`app3-backup.sh` — SQLite DB	`s3://hermes-vps-backups/app3/cloudpanel/`	3:00 AM	2026-07-28
MySQL (all DBs)	`app3-backup.sh` — mysqldump	`s3://hermes-vps-backups/app3/mysql/`	3:00 AM	2026-07-28
WordPress Files	`app3-backup.sh` — wp-content tar.gz	`s3://hermes-vps-backups/app3/wordpress/`	3:00 AM	2026-07-28
Nginx Configs	`app3-backup.sh` — sites-enabled + config	`s3://hermes-vps-backups/app3/config/`	3:00 AM	2026-07-28
WordPress Snapshots	`/opt/backup-restore/snapshot.sh` — per-site tar.gz	`/opt/backup-restore/snapshots/` (local, 30-day retention)	1 AM / 1 PM	2026-08-08
Hexclave (Stack Auth)	`hexclave-backup.sh` — PG dump + compose + env	`s3://hermes-vps-backups/app3/hexclave/`	3:30 AM	2026-08-08

wphost02 (5.161.62.38) — Hetzner CPX21

Service	Method	Destination	Schedule	Last Verified
WordPress (7 sites)	`ssh → /root/backup.sh` — per-site tar.gz + all DBs	`s3://hermes-vps-backups/wphost02-backup/`	5:00 AM	2026-07-28

Home Router

Service	Method	Destination	Schedule	Last Verified
MikroTik CCR2004	`run-wisp-backup.sh` — export .rsc via SSH	`s3://mikrotik-ccr-backups/wisp-backups/configs/home/`	6:00 AM	2026-07-28

External

Service	Method	Destination	Schedule	Last Verified
Hetzner Snapshots	`snapshot-hetzner.py` — API-driven disk snapshots	Hetzner Cloud (weekly)	Mon 5:00 AM	—

Schedules Summary (ET)

Time	What	Runner	Script
Every 15 min	Hermes session state	Hermes cron	`hermes-live-sync`
1:00 AM	Full Hermes backup	crontab	`hermes-backup.sh`
1:30 AM	Grafana, Uptime Kuma, Docker volumes, Prometheus	crontab	`core-services-backup.sh`
2:00 AM	Open WebUI, n8n, MCP configs (App1) + UniFi sync (App2)	crontab	`app1-backup.sh`, `unifi-backup-sync.sh`
2:30 AM	Vaultwarden (App1) + Traccar (App2)	Hermes cron / crontab	`vaultwarden-backup.sh`, `app2-backup.sh`
2:45 AM	Technitium DNS (App2)	Hermes cron	`technitium-backup.sh`
3:00 AM	/root essentials + App3 (CloudPanel, MySQL, WP, Nginx)	crontab	`root-essentials-backup.sh`, `app3-backup.sh`
3:15 AM	Auth API (Core)	Hermes cron	`auth-api-backup.sh`
3:30 AM	Hexclave / Stack Auth (App3) + LiteLLM (App1)	Hermes cron / crontab	`hexclave-backup.sh`, `litellm-backup.sh`
3:45 AM	Komodo (App1)	crontab	`komodo-backup.sh`
4:00 AM	DocuSeal (App1)	Hermes cron	`docuseal-backup.sh`
4:15 AM	Twenty CRM (App1)	Hermes cron	`twenty-backup.sh`
5:00 AM	wphost02 WordPress (SSH)	crontab	`wphost02-backup`
6:00 AM	MikroTik CCR + UNMS sync	Hermes cron	`run-wisp-backup.sh`, `unms-backup-sync.sh`
1 AM / 1 PM	WordPress per-site snapshots (App3, local)	App3 crontab	`/opt/backup-restore/snapshot.sh`
7:00 AM	Hudu (App2)	Hermes cron	`hudu-backup.sh`
8:00 AM	Gitea (App2)	Hermes cron	`gitea-backup.sh`
Mon 5:00 AM	Hetzner weekly snapshots	Hermes cron	`snapshot-hetzner.py`

Script Inventory

On Core (`/root/.hermes/scripts/`)

Script	Purpose	Runs
`hermes-backup.sh`	Full Hermes tar.gz to S3	crontab 1:00 AM
`core-services-backup.sh`	Grafana, Uptime Kuma, volumes, Prometheus	crontab 1:30 AM
`root-essentials-backup.sh`	/root keys, configs, scripts	crontab 3:00 AM
`backup-audit-check.sh`	Verify recent backup timestamps	crontab 2:00 AM
`vaultwarden-backup.sh`	Vaultwarden SQLite dump (SSH to App1)	Hermes cron 2:30 AM
`litellm-backup.sh`	LiteLLM Postgres dump + config (SSH to App1)	Hermes cron 3:30 AM
`komodo-backup.sh`	Komodo MongoDB dump (SSH to App1)	Hermes cron 3:45 AM
`docuseal-backup.sh`	DocuSeal SQLite + attachments (SSH to App1)	Hermes cron 4:00 AM
`twenty-backup.sh`	Twenty CRM Postgres dump (SSH to App1)	Hermes cron 4:15 AM
`app1-backup.sh`	Open WebUI, n8n, MCP configs (SSH to App1)	App1 crontab 2:00 AM
`app2-backup.sh`	Traccar backup (SSH to App2)	App2 crontab 2:30 AM
`technitium-backup.sh`	Technitium DNS backup (SSH to App2)	Hermes cron 2:45 AM
`app3-backup.sh`	CloudPanel, MySQL, WP, Nginx (SSH to App3)	crontab 3:00 AM
`auth-api-backup.sh`	Auth API SQLite + config (Core local)	Hermes cron 3:15 AM
`hexclave-backup.sh`	Stack Auth PG dump + compose + env (SSH to App3)	Hermes cron 3:30 AM
`wphost02-backup.sh`	SSH to wphost02, backup all sites	crontab 5:00 AM
`run-wisp-backup.sh`	MikroTik CCR config export	Hermes cron 6:00 AM
`unms-backup-sync.sh`	UNMS auto-backup sync to S3	Hermes cron 6:00 AM
`unifi-backup-sync.sh`	UniFi auto-backup sync to S3	Hermes cron 2:00 AM
`hudu-backup.sh`	Hudu Postgres dump to S3	Hermes cron 7:00 AM
`gitea-backup.sh`	Gitea repos + DB dump to S3	Hermes cron 8:00 AM
`snapshot-hetzner.py`	Hetzner server disk snapshots via API	Hermes cron Mon 5:00 AM

On App3 (`/opt/backup-restore/`)

Script	Purpose	Runs
`snapshot.sh`	Per-site WordPress tarball + DB	App3 crontab 6 AM / 6 PM

On wphost02 (`/root/`)

Script	Purpose	Runs
`backup.sh`	All 7 WordPress sites + all MySQL DBs	Triggered via SSH from Core

S3 Bucket Structure

hermes-vps-backups/
├── hermes-full-backup/     — Full Hermes daily (tar.gz)
├── live/                   — Hermes live sync (every 15 min)
├── live-sync/              — Old sync format (deprecated)
├── root-backup/            — /root essentials
├── standby/                — Standby configs + recovery bundle
├── core/
│   ├── grafana/            — Grafana SQLite DB
│   ├── uptime-kuma/        — Uptime Kuma SQLite DB
│   ├── prometheus/         — Prometheus TSDB snapshots
│   ├── vaultwarden/        — [STALE — service migrated to App1]
│   ├── twenty/             — [STALE — service migrated to App1]
│   ├── searxng/            — [STALE — service removed]
│   └── komodo/             — [STALE — service migrated to App1]
├── app1/
│   ├── openwebui/          — Open WebUI data
│   ├── litellm/            — LiteLLM Postgres dump + config
│   ├── n8n/                — n8n Postgres dump
│   ├── mcp/                — MCP server configs
│   ├── vaultwarden/        — Vaultwarden SQLite dump
│   ├── komodo/             — Komodo MongoDB dump
│   ├── docuseal/           — DocuSeal SQLite + attachments
│   └── twenty/             — Twenty CRM Postgres dump
├── app2/
│   └── traccar/            — Traccar H2 DB + config
├── app3/
│   ├── cloudpanel/         — CloudPanel SQLite DB
│   ├── mysql/              — All MySQL DBs
│   ├── wordpress/          — wp-content tar.gz
│   └── config/             — Nginx configs
├── hudu/backups/           — Hudu Postgres dump
├── gitea/daily/            — Gitea repos + DB
├── unms-backups/live/      — UNMS auto-backups
├── unifi-backups/          — UniFi controller backups
├── wphost02-backup/        — wphost02 per-site + DB
├── volumes/                — Docker volume dumps
├── caddy/                  — (unused)
├── assets/                 — Static assets
├── snapshots/              — (unused)
└── system-configs-*.tar.gz — Historical system config snapshots

mikrotik-ccr-backups/
└── wisp-backups/configs/
    └── home/               — CCR2004 export .rsc

Recovery Objectives (RPO / RTO)

Tier	Services	RPO	RTO	Notes
Critical	Hermes Agent, Gitea, Traccar, UISP	≤ 1 hour	≤ 4 hours	Live sync + daily backups; restore from S3 then replay live-sync
High	LiteLLM, n8n, Open WebUI, Vaultwarden, Twenty CRM	24 hours	≤ 8 hours	Daily backups; restore from previous night's dump
	Medium	Hudu, UniFi, Komodo, DocuSeal, App3 WP sites, Auth API, Hexclave (Stack Auth)	24 hours	≤ 24 hours
Low	Grafana, Uptime Kuma, Prometheus, MikroTik CCR, Technitium DNS	24 hours	≤ 48 hours	Monitoring data is nice-to-have, not blocking; Technitium DNS is caching-only (no authoritative zones — public DNS is SiteGround/Cloudflare, servers use Tailscale MagicDNS)
Unbacked	Dawarich, RAGFlow	N/A	N/A	No backup exists — see Unbacked Services below

Restore Testing Cadence

Quarterly: Pick one random backup per tier, restore to a staging location, verify integrity. After any major infra change: Test the affected service's restore path. Annual: Full DR simulation — restore all Critical + High tier services to staging from S3.

Stale S3 Paths — Cleanup Queue

These paths contain data from services that migrated off Core (Jul 28, 2026) or were removed:

Path	Status	Action
`core/vaultwarden/`	Stale 11 days	Safe to delete — Vaultwarden migrated to App1; new backups at `app1/vaultwarden/`
`core/twenty/`	Stale 11 days	Safe to delete — Twenty CRM migrated to App1; new backups at `app1/twenty/`
`core/searxng/`	Stale 11 days	Safe to delete — SearXNG removed; replaced by Super Search
`core/komodo/`	Stale 11 days	Safe to delete — Komodo migrated to App1; new backups at `app1/komodo/`
`caddy/`	Unused	Safe to delete — Never populated
`snapshots/`	Unused	Safe to delete — Never populated

Unbacked Services

These services are running in production with zero backup coverage:

Service	Server	Data Path	Risk
Dawarich	app2	Docker: `dawarich_db`, `dawarich_app` (4 containers)	Location history lost on disk failure
	RAGFlow	app2	Docker: `ragflow-cpu-1`

Action needed: Create backup scripts for both remaining unbacked services. For Dawarich and RAGFlow, determine database engines before designing backup strategy.

Migration History (Jul 28, 2026)

All the following services were migrated from Core to App1 in a single session:

Service	Old Home	New Home	Backup Before	Backup After
Vaultwarden	Core (:8080)	App1 (:8081)	`core/vaultwarden/` (stale)	`app1/vaultwarden/` ✅
SearXNG	Core (:8888)	Removed	`core/searxng/` (stale)	N/A (replaced by Super Search)
Twenty CRM	Core (:3003)	App1 (:3003)	`core/twenty/` (stale)	`app1/twenty/` ✅
Komodo	Core (:9120)	App1 (:9120)	`core/komodo/` (stale)	`app1/komodo/` ✅
DocuSeal	Core (:3000)	App1 (:3002)	No backup existed	`app1/docuseal/` ✅
Kokoro TTS	Core (:8880)	App1 (:8880)	N/A (stateless)	N/A (stateless)

Disaster Recovery

Live Hermes: netcup VPS — core.itpropartner.com — 152.53.192.33
Standby Hermes: Hetzner CPX21 — app1-bu.itpropartner.com — 5.161.225.131
- Cron checks live box every 10 min, takes over if down
- Auto-restores from s3://hermes-vps-backups/hermes-full-backup/
- Provider diversity: netcup outage won't kill both Core and standby
Recovery priority: Hermes first → infrastructure monitoring (Uptime Kuma, Grafana) → App1 services → App2 services
24 backup scripts on Core, 1 on App3, 1 on wphost02

16 KiB Raw Blame History