ippadmin/itpp-infrastructure
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
77a7cb172a6b573daf446868068f1f6791d61f80
itpp-infrastructure/backup-plan.md
T

16 KiB
Raw Blame History

ITPP Backup Plan

Last updated: 2026-08-08 Scope: All ITPP infrastructure backups — Hermes core, application servers, routers, and external services. Storage: Wasabi S3 (us-east-1) via --endpoint-url https://s3.us-east-1.wasabisys.com Cron backend: Hybrid — system crontab + Hermes cron jobs (both on Core) Verification audit: DR issue log

Backup Inventory (27 targets)

Core (152.53.192.33) — RS 2000

Service Method Destination Schedule Last Verified
Hermes Agent (full) hermes-backup.sh — tar.gz of config, skills, profiles, sessions s3://hermes-vps-backups/hermes-full-backup/ 1:00 AM 2026-07-28
Hermes Live Sync hermes-live-sync cron — session state + profiles DB s3://hermes-vps-backups/live/ Every 15 min 2026-07-28 20:42
/root Essentials root-essentials-backup.sh — dotfiles, keys, scripts s3://hermes-vps-backups/root-backup/ 3:00 AM 2026-07-28
Grafana core-services-backup.sh — SQLite DB dump s3://hermes-vps-backups/core/grafana/ 1:30 AM 2026-07-28
Uptime Kuma core-services-backup.sh — SQLite DB dump s3://hermes-vps-backups/core/uptime-kuma/ 1:30 AM 2026-07-28
Docker Volumes core-services-backup.sh — tar of key compose volumes s3://hermes-vps-backups/volumes/ 1:30 AM 2026-07-28
Prometheus core-services-backup.sh — TSDB snapshot s3://hermes-vps-backups/core/prometheus/ 1:30 AM 2026-07-28
Auth API auth-api-backup.sh — SQLite .backup + .env s3://hermes-vps-backups/core/auth-api/ 3:15 AM 2026-08-08

App1 (152.53.36.131) — RS 4000

Service Method Destination Schedule Last Verified
Open WebUI app1-backup.sh — data dir tar.gz s3://hermes-vps-backups/app1/openwebui/ 2:00 AM 2026-07-28
LiteLLM litellm-backup.sh — Postgres DB dump + .env s3://hermes-vps-backups/app1/litellm/ 3:30 AM 2026-07-28
n8n app1-backup.sh — Postgres DB dump s3://hermes-vps-backups/app1/n8n/ 2:00 AM 2026-07-28
MCP Server Configs app1-backup.sh — MCP settings files s3://hermes-vps-backups/app1/mcp/ 2:00 AM 2026-07-28
Vaultwarden vaultwarden-backup.sh — SQLite DB dump s3://hermes-vps-backups/app1/vaultwarden/ 2:30 AM 2026-07-28
Komodo komodo-backup.sh — MongoDB dump + compose + keys s3://hermes-vps-backups/app1/komodo/ 3:45 AM 2026-07-28
DocuSeal docuseal-backup.sh — SQLite DB + attachments s3://hermes-vps-backups/app1/docuseal/ 4:00 AM 2026-07-28
Twenty CRM twenty-backup.sh — Postgres dump + .env + compose s3://hermes-vps-backups/app1/twenty/ 4:15 AM 2026-07-28
Kokoro TTS (stateless — no backup needed) — — —

App2 (152.53.39.202) — RS 4000

Service Method Destination Schedule Last Verified
Hudu hudu-backup.sh — Postgres dump s3://hermes-vps-backups/hudu/backups/ 7:00 AM 2026-07-28
Gitea gitea-backup.sh — repos + DB dump s3://hermes-vps-backups/gitea/daily/ 8:00 AM 2026-07-28
UNMS unms-backup-sync.sh — auto-backup .unms s3://hermes-vps-backups/unms-backups/live/backups/ 6:00 AM 2026-07-28
UniFi unifi-backup-sync.sh — auto-backup .unf s3://hermes-vps-backups/unifi-backups/ 2:00 AM 2026-07-18
Traccar app2-backup.sh — H2 DB + config s3://hermes-vps-backups/app2/traccar/ 2:30 AM 2026-07-28
Technitium DNS technitium-backup.sh — data dir + compose s3://hermes-vps-backups/app2/technitium/ 2:45 AM 2026-08-08
Dawarich dawarich-backup.sh — PostgreSQL dump (remote SSH) s3://hermes-vps-backups/app2/dawarich/ 4:00 AM 2026-08-08
RAGFlow ragflow-backup.sh — MySQL dump (remote SSH) s3://hermes-vps-backups/app2/ragflow/ 4:15 AM 2026-08-08

App3 (152.53.241.111) — RS 4000

Service Method Destination Schedule Last Verified
CloudPanel DB app3-backup.sh — SQLite DB s3://hermes-vps-backups/app3/cloudpanel/ 3:00 AM 2026-07-28
MySQL (all DBs) app3-backup.sh — mysqldump s3://hermes-vps-backups/app3/mysql/ 3:00 AM 2026-07-28
WordPress Files app3-backup.sh — wp-content tar.gz s3://hermes-vps-backups/app3/wordpress/ 3:00 AM 2026-07-28
Nginx Configs app3-backup.sh — sites-enabled + config s3://hermes-vps-backups/app3/config/ 3:00 AM 2026-07-28
Static Sites app3-backup.sh — all non-WordPress htdocs (modelortho.com, verdicttank, transitpin, katiewatts, etc.) s3://hermes-vps-backups/app3/static/ 3:00 AM 2026-08-08
WordPress Snapshots /opt/backup-restore/snapshot.sh — per-site tar.gz /opt/backup-restore/snapshots/ (local, 30-day retention) 1 AM / 1 PM 2026-08-08
Hexclave (Stack Auth) hexclave-backup.sh — PG dump + compose + env s3://hermes-vps-backups/app3/hexclave/ 3:30 AM 2026-08-08
modelortho.com modelortho-backup.sh — htdocs tar.gz + nginx configs s3://hermes-vps-backups/app3/modelortho/ 4:30 AM 2026-08-08

wphost02 (5.161.62.38) — Hetzner CPX21 — DECOMMISSIONED (2026-08-28)

wphost02 was deleted from the Hetzner account on 2026-08-28. Its backup job has been removed; sites now live on app3 (covered under the App3 backup section above).

Home Router

Service Method Destination Schedule Last Verified
MikroTik CCR2004 run-wisp-backup.sh — export .rsc via SSH s3://mikrotik-ccr-backups/wisp-backups/configs/home/ 6:00 AM 2026-07-28

External

Service Method Destination Schedule Last Verified
Hetzner Snapshots snapshot-hetzner.py — API-driven disk snapshots Hetzner Cloud (weekly) Mon 5:00 AM —

Schedules Summary (ET)

Time What Runner Script
Every 15 min Hermes session state Hermes cron hermes-live-sync
1:00 AM Full Hermes backup crontab hermes-backup.sh
1:30 AM Grafana, Uptime Kuma, Docker volumes, Prometheus crontab core-services-backup.sh
2:00 AM Open WebUI, n8n, MCP configs (App1) + UniFi sync (App2) crontab app1-backup.sh, unifi-backup-sync.sh
2:30 AM Vaultwarden (App1) + Traccar (App2) Hermes cron / crontab vaultwarden-backup.sh, app2-backup.sh
2:45 AM Technitium DNS (App2) Hermes cron technitium-backup.sh
3:00 AM /root essentials + App3 (CloudPanel, MySQL, WP, Nginx) crontab root-essentials-backup.sh, app3-backup.sh
3:15 AM Auth API (Core) Hermes cron auth-api-backup.sh
3:30 AM Hexclave / Stack Auth (App3) + LiteLLM (App1) Hermes cron / crontab hexclave-backup.sh, litellm-backup.sh
3:45 AM Komodo (App1) crontab komodo-backup.sh
4:00 AM Dawarich (App2) Hermes cron dawarich-backup.sh
4:00 AM DocuSeal (App1) Hermes cron docuseal-backup.sh
4:15 AM RAGFlow (App2) Hermes cron ragflow-backup.sh
4:15 AM Twenty CRM (App1) Hermes cron twenty-backup.sh
6:00 AM MikroTik CCR + UNMS sync Hermes cron run-wisp-backup.sh, unms-backup-sync.sh
1 AM / 1 PM WordPress per-site snapshots (App3, local) App3 crontab /opt/backup-restore/snapshot.sh
7:00 AM Hudu (App2) Hermes cron hudu-backup.sh
8:00 AM Gitea (App2) Hermes cron gitea-backup.sh
Mon 5:00 AM Hetzner weekly snapshots Hermes cron snapshot-hetzner.py

Script Inventory

On Core (/root/.hermes/scripts/)

Script Purpose Runs
hermes-backup.sh Full Hermes tar.gz to S3 crontab 1:00 AM
core-services-backup.sh Grafana, Uptime Kuma, volumes, Prometheus crontab 1:30 AM
root-essentials-backup.sh /root keys, configs, scripts crontab 3:00 AM
backup-audit-check.sh Verify recent backup timestamps crontab 2:00 AM
vaultwarden-backup.sh Vaultwarden SQLite dump (SSH to App1) Hermes cron 2:30 AM
litellm-backup.sh LiteLLM Postgres dump + config (SSH to App1) Hermes cron 3:30 AM
komodo-backup.sh Komodo MongoDB dump (SSH to App1) Hermes cron 3:45 AM
docuseal-backup.sh DocuSeal SQLite + attachments (SSH to App1) Hermes cron 4:00 AM
twenty-backup.sh Twenty CRM Postgres dump (SSH to App1) Hermes cron 4:15 AM
app1-backup.sh Open WebUI, n8n, MCP configs (SSH to App1) App1 crontab 2:00 AM
app2-backup.sh Traccar backup (SSH to App2) App2 crontab 2:30 AM
technitium-backup.sh Technitium DNS backup (SSH to App2) Hermes cron 2:45 AM
app3-backup.sh CloudPanel, MySQL, WP, Nginx (SSH to App3) crontab 3:00 AM
auth-api-backup.sh Auth API SQLite + config (Core local) Hermes cron 3:15 AM
hexclave-backup.sh Stack Auth PG dump + compose + env (SSH to App3) Hermes cron 3:30 AM
dawarich-backup.sh Dawarich PostgreSQL dump + compose + env (SSH to App2) Hermes cron 4:00 AM
ragflow-backup.sh RAGFlow MySQL dump + compose + env (SSH to App2) Hermes cron 4:15 AM
run-wisp-backup.sh MikroTik CCR config export Hermes cron 6:00 AM
unms-backup-sync.sh UNMS auto-backup sync to S3 Hermes cron 6:00 AM
unifi-backup-sync.sh UniFi auto-backup sync to S3 Hermes cron 2:00 AM
hudu-backup.sh Hudu Postgres dump to S3 Hermes cron 7:00 AM
gitea-backup.sh Gitea repos + DB dump to S3 Hermes cron 8:00 AM
snapshot-hetzner.py Hetzner server disk snapshots via API Hermes cron Mon 5:00 AM

On App3 (/opt/backup-restore/)

Script Purpose Runs
snapshot.sh Per-site WordPress tarball + DB App3 crontab 6 AM / 6 PM

S3 Bucket Structure

hermes-vps-backups/
├── hermes-full-backup/     — Full Hermes daily (tar.gz)
├── live/                   — Hermes live sync (every 15 min)
├── live-sync/              — Old sync format (deprecated)
├── root-backup/            — /root essentials
├── standby/                — Standby configs + recovery bundle
├── core/
│   ├── grafana/            — Grafana SQLite DB
│   ├── uptime-kuma/        — Uptime Kuma SQLite DB
│   ├── prometheus/         — Prometheus TSDB snapshots
│   ├── vaultwarden/        — [STALE — service migrated to App1]
│   ├── twenty/             — [STALE — service migrated to App1]
│   ├── searxng/            — [STALE — service removed]
│   └── komodo/             — [STALE — service migrated to App1]
├── app1/
│   ├── openwebui/          — Open WebUI data
│   ├── litellm/            — LiteLLM Postgres dump + config
│   ├── n8n/                — n8n Postgres dump
│   ├── mcp/                — MCP server configs
│   ├── vaultwarden/        — Vaultwarden SQLite dump
│   ├── komodo/             — Komodo MongoDB dump
│   ├── docuseal/           — DocuSeal SQLite + attachments
│   └── twenty/             — Twenty CRM Postgres dump
├── app2/
│   └── traccar/            — Traccar H2 DB + config
├── app3/
│   ├── cloudpanel/         — CloudPanel SQLite DB
│   ├── mysql/              — All MySQL DBs
│   ├── wordpress/          — wp-content tar.gz
│   └── config/             — Nginx configs
├── hudu/backups/           — Hudu Postgres dump
├── gitea/daily/            — Gitea repos + DB
├── unms-backups/live/      — UNMS auto-backups
├── unifi-backups/          — UniFi controller backups
├── volumes/                — Docker volume dumps
├── caddy/                  — (unused)
├── assets/                 — Static assets
├── snapshots/              — (unused)
└── system-configs-*.tar.gz — Historical system config snapshots

mikrotik-ccr-backups/
└── wisp-backups/configs/
    └── home/               — CCR2004 export .rsc

Recovery Objectives (RPO / RTO)

Tier Services RPO RTO Notes
Critical Hermes Agent, Gitea, Traccar, UISP ≤ 1 hour ≤ 4 hours Live sync + daily backups; restore from S3 then replay live-sync
High LiteLLM, n8n, Open WebUI, Vaultwarden, Twenty CRM 24 hours ≤ 8 hours Daily backups; restore from previous night's dump
Medium Hudu, UniFi, Komodo, DocuSeal, App3 WP sites, Auth API, Hexclave (Stack Auth) 24 hours ≤ 24 hours
Low Grafana, Uptime Kuma, Prometheus, MikroTik CCR, Technitium DNS, Dawarich, RAGFlow 24 hours ≤ 48 hours

Restore Testing Cadence

Quarterly: Pick one random backup per tier, restore to a staging location, verify integrity. After any major infra change: Test the affected service's restore path. Annual: Full DR simulation — restore all Critical + High tier services to staging from S3.

Stale S3 Paths — Cleanup Queue

These paths contain data from services that migrated off Core (Jul 28, 2026) or were removed:

Path Status Action
core/vaultwarden/ Stale 11 days Safe to delete — Vaultwarden migrated to App1; new backups at app1/vaultwarden/
core/twenty/ Stale 11 days Safe to delete — Twenty CRM migrated to App1; new backups at app1/twenty/
core/searxng/ Stale 11 days Safe to delete — SearXNG removed; replaced by Super Search
core/komodo/ Stale 11 days Safe to delete — Komodo migrated to App1; new backups at app1/komodo/
caddy/ Unused Safe to delete — Never populated
snapshots/ Unused Safe to delete — Never populated

Unbacked Services

These services are running in production with zero backup coverage: | Unbacked | (none) | N/A | N/A | All services are backed up as of 2026-08-08 |

6 previously-unbacked services now backed up (Auth API, Technitium DNS, Dawarich, RAGFlow, Stack Auth/Hexclave, app3 static sites). The earlier count of "14" was an error — no source document supports that number. The actual delta since July 28 is 6.

Migration History (Jul 28, 2026)

All the following services were migrated from Core to App1 in a single session:

Service Old Home New Home Backup Before Backup After
Vaultwarden Core (:8080) App1 (:8081) core/vaultwarden/ (stale) app1/vaultwarden/ ✅
SearXNG Core (:8888) Removed core/searxng/ (stale) N/A (replaced by Super Search)
Twenty CRM Core (:3003) App1 (:3003) core/twenty/ (stale) app1/twenty/ ✅
Komodo Core (:9120) App1 (:9120) core/komodo/ (stale) app1/komodo/ ✅
DocuSeal Core (:3000) App1 (:3002) No backup existed app1/docuseal/ ✅
Kokoro TTS Core (:8880) App1 (:8880) N/A (stateless) N/A (stateless)

Disaster Recovery

  • Live Hermes: netcup VPS — core.itpropartner.com — 152.53.192.33
  • Standby Hermes: Hetzner CPX21 — app1-bu.itpropartner.com — 5.161.225.131
    • Cron checks live box every 10 min, takes over if down
    • Auto-restores from s3://hermes-vps-backups/hermes-full-backup/
    • Provider diversity: netcup outage won't kill both Core and standby
  • Recovery priority: Hermes first → infrastructure monitoring (Uptime Kuma, Grafana) → App1 services → App2 services
  • Backup scripts: 24 scripts on Core, 1 on App3 (wphost02 DECOMMISSIONED 2026-08-28; its backup script and S3 path were removed)
Reference in New Issue View Git Blame Copy Permalink