ippadmin/itpp-infrastructure
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
af80a8cb7b1064006fa37558b75512a381cfea94
itpp-infrastructure/backup-plan.md
T

19 KiB
Raw Blame History

ITPP Backup Plan

Last updated: 2026-09-11 Scope: All ITPP infrastructure backups — Hermes core, application servers, routers, and external services. Storage: Wasabi S3 (us-east-1) via --endpoint-url https://s3.us-east-1.wasabisys.com Cron backend: Hybrid — system crontab + Hermes cron jobs (both on Core) Verification audit: DR issue log

Backup Inventory (27 targets)

Core (152.53.192.33) — RS 2000

Service Method Destination Schedule Last Verified
Hermes Agent (full) hermes-backup.sh — tar.gz of config, skills, profiles, sessions s3://hermes-vps-backups/hermes-full-backup/ 1:00 AM 2026-07-28
Hermes Live Sync hermes-live-sync cron — session state + profiles DB s3://hermes-vps-backups/live/ Every 15 min 2026-07-28 20:42
/root Essentials root-essentials-backup.sh — dotfiles, keys, scripts s3://hermes-vps-backups/root-backup/ 3:00 AM 2026-07-28
Grafana core-services-backup.sh — SQLite DB dump s3://hermes-vps-backups/core/grafana/ 1:30 AM 2026-07-28
Uptime Kuma core-services-backup.sh — SQLite DB dump s3://hermes-vps-backups/core/uptime-kuma/ 1:30 AM 2026-07-28
Docker Volumes core-services-backup.sh — tar of key compose volumes s3://hermes-vps-backups/volumes/ 1:30 AM 2026-07-28
Prometheus core-services-backup.sh — TSDB snapshot s3://hermes-vps-backups/core/prometheus/ 1:30 AM 2026-07-28
Auth API auth-api-backup.sh — SQLite .backup + .env s3://hermes-vps-backups/core/auth-api/ 3:15 AM 2026-08-08

App1 (152.53.36.131) — RS 4000

Service Method Destination Schedule Last Verified
Open WebUI app1-backup.sh — data dir tar.gz s3://hermes-vps-backups/app1/openwebui/ 2:00 AM 2026-07-28
LiteLLM litellm-backup.sh — Postgres DB dump + .env s3://hermes-vps-backups/app1/litellm/ 3:30 AM 2026-07-28
n8n app1-backup.sh — Postgres DB dump s3://hermes-vps-backups/app1/n8n/ 2:00 AM 2026-07-28
MCP Server Configs app1-backup.sh — MCP settings files s3://hermes-vps-backups/app1/mcp/ 2:00 AM 2026-07-28
Vaultwarden vaultwarden-backup.sh — SQLite DB dump s3://hermes-vps-backups/app1/vaultwarden/ 2:30 AM 2026-07-28
Komodo komodo-backup.sh — MongoDB dump + compose + keys s3://hermes-vps-backups/app1/komodo/ 3:45 AM 2026-07-28
DocuSeal docuseal-backup.sh — SQLite DB + attachments s3://hermes-vps-backups/app1/docuseal/ 4:00 AM 2026-07-28
Twenty CRM twenty-backup.sh — Postgres dump + .env + compose s3://hermes-vps-backups/app1/twenty/ 4:15 AM 2026-07-28
Kokoro TTS (stateless — no backup needed) — — —

App2 (152.53.39.202) — RS 4000

Service Method Destination Schedule Last Verified
Hudu hudu-backup.sh — Postgres dump s3://hermes-vps-backups/hudu/backups/ 7:00 AM 2026-07-28
Gitea gitea-backup.sh — repos + DB dump s3://hermes-vps-backups/gitea/daily/ 8:00 AM 2026-07-28
UNMS unms-backup-sync.sh — auto-backup .unms s3://hermes-vps-backups/unms-backups/live/backups/ 6:00 AM 2026-07-28
UniFi unifi-backup-sync.sh — auto-backup .unf s3://hermes-vps-backups/unifi-backups/ 2:00 AM 2026-07-18
Traccar app2-backup.sh — H2 DB + config s3://hermes-vps-backups/app2/traccar/ 2:30 AM 2026-07-28
Technitium DNS technitium-backup.sh — data dir + compose s3://hermes-vps-backups/app2/technitium/ 2:45 AM 2026-08-08
Dawarich dawarich-backup.sh — PostgreSQL dump (remote SSH) s3://hermes-vps-backups/app2/dawarich/ 4:00 AM 2026-08-08
RAGFlow ragflow-backup.sh — MySQL dump (remote SSH) s3://hermes-vps-backups/app2/ragflow/ 4:15 AM 2026-08-08

App3 (152.53.241.111) — RS 4000

Service Method Destination Schedule Last Verified
CloudPanel DB app3-backup.sh — SQLite DB s3://hermes-vps-backups/app3/cloudpanel/ 3:00 AM 2026-07-28
MySQL (all DBs) app3-backup.sh — mysqldump s3://hermes-vps-backups/app3/mysql/ 3:00 AM 2026-07-28
WordPress Files app3-backup.sh — wp-content tar.gz s3://hermes-vps-backups/app3/wordpress/ 3:00 AM 2026-07-28
Nginx Configs app3-backup.sh — sites-enabled + config s3://hermes-vps-backups/app3/config/ 3:00 AM 2026-07-28
Static Sites app3-backup.sh — all non-WordPress htdocs (modelortho.com, verdicttank, transitpin, katiewatts, etc.) s3://hermes-vps-backups/app3/static/ 3:00 AM 2026-08-08
WordPress Snapshots /opt/backup-restore/snapshot.sh — per-site tar.gz /opt/backup-restore/snapshots/ (local, 30-day retention) 1 AM / 1 PM 2026-08-08
Hexclave (Stack Auth) hexclave-backup.sh — PG dump + compose + env s3://hermes-vps-backups/app3/hexclave/ 3:30 AM 2026-08-08
modelortho.com modelortho-backup.sh — htdocs tar.gz + nginx configs s3://hermes-vps-backups/app3/modelortho/ 4:30 AM 2026-08-08

wphost02 (5.161.62.38) — Hetzner CPX21 — DECOMMISSIONED (2026-08-28)

wphost02 was deleted from the Hetzner account on 2026-08-28. Its backup job has been removed; sites now live on app3 (covered under the App3 backup section above).

anita-mnz (159.195.16.30) - netcup (Anita's dedicated box, cut over 2026-09-11)

Service Method Destination Schedule Last Verified
/root Essentials root-essentials-backup.sh - dotfiles, profile, keys, scripts (excludes *.db by design) s3://hermes-vps-backups/root-backup/anita-mnz/ 3:00 AM 2026-09-11
Hermes Profile DBs hermes-db-backup.sh - sqlite3 .backup snapshots of state.db + cron + wisdom + notepad + verification DBs, quick_check per snapshot, upload then download-and-verify s3://hermes-vps-backups/root-backup/anita-mnz/db/ 3:10 AM 2026-09-11 (restore test passed: quick_check ok, 99,285 messages)

Home Router

Service Method Destination Schedule Last Verified
MikroTik CCR2004 run-wisp-backup.sh — export .rsc via SSH s3://mikrotik-ccr-backups/wisp-backups/configs/home/ 6:00 AM 2026-07-28

External

Service Method Destination Schedule Last Verified
Hetzner Snapshots snapshot-hetzner.py — API-driven disk snapshots Hetzner Cloud (weekly) Mon 5:00 AM —

Schedules Summary (ET)

Time What Runner Script
Every 15 min Hermes session state Hermes cron hermes-live-sync
1:00 AM Full Hermes backup crontab hermes-backup.sh
1:30 AM Grafana, Uptime Kuma, Docker volumes, Prometheus crontab core-services-backup.sh
2:00 AM Open WebUI, n8n, MCP configs (App1) + UniFi sync (App2) crontab app1-backup.sh, unifi-backup-sync.sh
2:30 AM Vaultwarden (App1) + Traccar (App2) Hermes cron / crontab vaultwarden-backup.sh, app2-backup.sh
2:45 AM Technitium DNS (App2) Hermes cron technitium-backup.sh
3:00 AM /root essentials + App3 (CloudPanel, MySQL, WP, Nginx) crontab root-essentials-backup.sh, app3-backup.sh
3:00 AM anita-mnz /root essentials Anita crontab root-essentials-backup.sh
3:10 AM anita-mnz Hermes profile DB snapshots Anita crontab hermes-db-backup.sh
3:15 AM Auth API (Core) Hermes cron auth-api-backup.sh
3:30 AM Hexclave / Stack Auth (App3) + LiteLLM (App1) Hermes cron / crontab hexclave-backup.sh, litellm-backup.sh
3:45 AM Komodo (App1) crontab komodo-backup.sh
4:00 AM Dawarich (App2) Hermes cron dawarich-backup.sh
4:00 AM DocuSeal (App1) Hermes cron docuseal-backup.sh
4:15 AM RAGFlow (App2) Hermes cron ragflow-backup.sh
4:15 AM Twenty CRM (App1) Hermes cron twenty-backup.sh
6:00 AM MikroTik CCR + UNMS sync Hermes cron run-wisp-backup.sh, unms-backup-sync.sh
1 AM / 1 PM WordPress per-site snapshots (App3, local) App3 crontab /opt/backup-restore/snapshot.sh
7:00 AM Hudu (App2) Hermes cron hudu-backup.sh
8:00 AM Gitea (App2) Hermes cron gitea-backup.sh
Mon 5:00 AM Hetzner weekly snapshots Hermes cron snapshot-hetzner.py

Script Inventory

On Core (/root/.hermes/scripts/)

Script Purpose Runs
hermes-backup.sh Full Hermes tar.gz to S3 crontab 1:00 AM
core-services-backup.sh Grafana, Uptime Kuma, volumes, Prometheus crontab 1:30 AM
root-essentials-backup.sh /root keys, configs, scripts crontab 3:00 AM
backup-audit-check.sh Verify recent backup timestamps crontab 2:00 AM
vaultwarden-backup.sh Vaultwarden SQLite dump (SSH to App1) Hermes cron 2:30 AM
litellm-backup.sh LiteLLM Postgres dump + config (SSH to App1) Hermes cron 3:30 AM
komodo-backup.sh Komodo MongoDB dump (SSH to App1) Hermes cron 3:45 AM
docuseal-backup.sh DocuSeal SQLite + attachments (SSH to App1) Hermes cron 4:00 AM
twenty-backup.sh Twenty CRM Postgres dump (SSH to App1) Hermes cron 4:15 AM
app1-backup.sh Open WebUI, n8n, MCP configs (SSH to App1) App1 crontab 2:00 AM
app2-backup.sh Traccar backup (SSH to App2) App2 crontab 2:30 AM
technitium-backup.sh Technitium DNS backup (SSH to App2) Hermes cron 2:45 AM
app3-backup.sh CloudPanel, MySQL, WP, Nginx (SSH to App3) crontab 3:00 AM
auth-api-backup.sh Auth API SQLite + config (Core local) Hermes cron 3:15 AM
hexclave-backup.sh Stack Auth PG dump + compose + env (SSH to App3) Hermes cron 3:30 AM
dawarich-backup.sh Dawarich PostgreSQL dump + compose + env (SSH to App2) Hermes cron 4:00 AM
ragflow-backup.sh RAGFlow MySQL dump + compose + env (SSH to App2) Hermes cron 4:15 AM
run-wisp-backup.sh MikroTik CCR config export Hermes cron 6:00 AM
unms-backup-sync.sh UNMS auto-backup sync to S3 Hermes cron 6:00 AM
unifi-backup-sync.sh UniFi auto-backup sync to S3 Hermes cron 2:00 AM
hudu-backup.sh Hudu Postgres dump to S3 Hermes cron 7:00 AM
gitea-backup.sh Gitea repos + DB dump to S3 Hermes cron 8:00 AM
snapshot-hetzner.py Hetzner server disk snapshots via API Hermes cron Mon 5:00 AM

On App3 (/opt/backup-restore/)

Script Purpose Runs
snapshot.sh Per-site WordPress tarball + DB App3 crontab 6 AM / 6 PM

On anita-mnz (/root/)

Script Purpose Runs
root-essentials-backup.sh /root essentials (profile, keys, scripts, configs) crontab 3:00 AM
hermes-db-backup.sh sqlite3 .backup snapshots of Hermes profile DBs + per-snapshot quick_check + restore test crontab 3:10 AM

S3 Bucket Structure

hermes-vps-backups/
├── hermes-full-backup/     — Full Hermes daily (tar.gz)
├── live/                   — Hermes live sync (every 15 min)
├── decommissioned/         — archived copies of decommissioned systems (profile archives)
├── live-sync/              — Old sync format (deprecated)
├── root-backup/            — /root essentials (Core uses the flat legacy prefix)
│   └── <host>/             — per-host prefix (`anita-mnz`, ...)
│       └── db/             — SQLite `.backup` snapshots (`hermes-db-backup.sh`)
├── standby/                — Standby configs + recovery bundle
├── core/
│   ├── grafana/            — Grafana SQLite DB
│   ├── uptime-kuma/        — Uptime Kuma SQLite DB
│   ├── prometheus/         — Prometheus TSDB snapshots
│   ├── vaultwarden/        — [STALE — service migrated to App1]
│   ├── twenty/             — [STALE — service migrated to App1]
│   ├── searxng/            — [STALE — service removed]
│   └── komodo/             — [STALE — service migrated to App1]
├── app1/
│   ├── openwebui/          — Open WebUI data
│   ├── litellm/            — LiteLLM Postgres dump + config
│   ├── n8n/                — n8n Postgres dump
│   ├── mcp/                — MCP server configs
│   ├── vaultwarden/        — Vaultwarden SQLite dump
│   ├── komodo/             — Komodo MongoDB dump
│   ├── docuseal/           — DocuSeal SQLite + attachments
│   └── twenty/             — Twenty CRM Postgres dump
├── app2/
│   └── traccar/            — Traccar H2 DB + config
├── app3/
│   ├── cloudpanel/         — CloudPanel SQLite DB
│   ├── mysql/              — All MySQL DBs
│   ├── wordpress/          — wp-content tar.gz
│   └── config/             — Nginx configs
├── hudu/backups/           — Hudu Postgres dump
├── gitea/daily/            — Gitea repos + DB
├── unms-backups/live/      — UNMS auto-backups
├── unifi-backups/          — UniFi controller backups
├── volumes/                — Docker volume dumps
├── caddy/                  — (unused)
├── assets/                 — Static assets
├── snapshots/              — (unused)
└── system-configs-*.tar.gz — Historical system config snapshots

mikrotik-ccr-backups/
└── wisp-backups/configs/
    └── home/               — CCR2004 export .rsc

Recovery Objectives (RPO / RTO)

Tier Services RPO RTO Notes
Critical Hermes Agent, Gitea, Traccar, UISP ≤ 1 hour ≤ 4 hours Live sync + daily backups; restore from S3 then replay live-sync
High LiteLLM, n8n, Open WebUI, Vaultwarden, Twenty CRM 24 hours ≤ 8 hours Daily backups; restore from previous night's dump
Medium Hudu, UniFi, Komodo, DocuSeal, App3 WP sites, Auth API, Hexclave (Stack Auth) 24 hours ≤ 24 hours
Low Grafana, Uptime Kuma, Prometheus, MikroTik CCR, Technitium DNS, Dawarich, RAGFlow 24 hours ≤ 48 hours

Restore Testing Cadence

Quarterly: Pick one random backup per tier, restore to a staging location, verify integrity. After any major infra change: Test the affected service's restore path. Annual: Full DR simulation — restore all Critical + High tier services to staging from S3.

Stale S3 Paths — Cleanup Queue

These paths contain data from services that migrated off Core (Jul 28, 2026) or were removed:

Path Status Action
core/vaultwarden/ Stale 11 days Safe to delete — Vaultwarden migrated to App1; new backups at app1/vaultwarden/
core/twenty/ Stale 11 days Safe to delete — Twenty CRM migrated to App1; new backups at app1/twenty/
core/searxng/ Stale 11 days Safe to delete — SearXNG removed; replaced by Super Search
core/komodo/ Stale 11 days Safe to delete — Komodo migrated to App1; new backups at app1/komodo/
caddy/ Unused Safe to delete — Never populated
snapshots/ Unused Safe to delete — Never populated
live/profiles/anita/ Purged 2026-09-11 Deleted — 20,551 objects / 5.0 GB of the frozen Core copy of Anita's profile, pushed by the 15-minute sync before it was paused Sep 3. Superseded by decommissioned/anita-core-frozen-profile-20260911-1732.tar.gz

Unbacked Services

These services are running in production with zero backup coverage: | Unbacked | (none) | N/A | N/A | All services are backed up as of 2026-08-08 |

Done 2026-09-11 (Anita migration closeout). The frozen Core copy /root/.hermes/profiles/anita (8.0 GB, excluded from live sync after the 15:53 cutover) was archived to s3://hermes-vps-backups/decommissioned/anita-core-frozen-profile-20260911-1732.tar.gz — 2,786,082,561 B, sha256 b457fa7f…b3dc, 21,206 entries — then deleted, reclaiming 8 GB (125 GB → 117 GB used). The archive was proven by downloading the object back and matching the sha256 against the local tarball before anything was removed. The 8 GB was almost entirely quarantine artifacts from the Sep 3/9/10 SQLite corruption events (corrupt-20260903/09/10, pre-rebuild-20260910, recovered-20260910); the live profile had already been proven migrated (memories, .env and the six cron job IDs identical on the new box, which carries one more skill and two more messages).

Gap found and closed 2026-09-11 (anita-mnz): root-essentials-backup.sh correctly excludes *.db so it never archives a live SQLite file, but nothing replaced it, so after the box was provisioned Anita's conversation store (974 MB), the cron execution DB, notepad, wisdom and verification DBs had zero backup coverage. Core has hermes-backup.sh doing this job; the migrated box had no equivalent. hermes-db-backup.sh (3:10 AM) now snapshots each DB with the sqlite3 .backup API, quick_checks every snapshot, uploads one dated tarball, then downloads it back and verifies the restored store. Verified 2026-09-11 17:04: restore test passed, quick_check=ok, 99,285 messages. Any box migrated with this script set needs the same check, because the *.db exclusion is silent.

6 previously-unbacked services now backed up (Auth API, Technitium DNS, Dawarich, RAGFlow, Stack Auth/Hexclave, app3 static sites). The earlier count of "14" was an error — no source document supports that number. The actual delta since July 28 is 6.

Migration History (Jul 28, 2026)

All the following services were migrated from Core to App1 in a single session:

Service Old Home New Home Backup Before Backup After
Vaultwarden Core (:8080) App1 (:8081) core/vaultwarden/ (stale) app1/vaultwarden/ ✅
SearXNG Core (:8888) Removed core/searxng/ (stale) N/A (replaced by Super Search)
Twenty CRM Core (:3003) App1 (:3003) core/twenty/ (stale) app1/twenty/ ✅
Komodo Core (:9120) App1 (:9120) core/komodo/ (stale) app1/komodo/ ✅
DocuSeal Core (:3000) App1 (:3002) No backup existed app1/docuseal/ ✅
Kokoro TTS Core (:8880) App1 (:8880) N/A (stateless) N/A (stateless)

Disaster Recovery

  • Live Hermes: netcup VPS — core.itpropartner.com — 152.53.192.33
  • Standby Hermes: Hetzner CPX21 — app1-bu.itpropartner.com — 5.161.225.131
    • Cron checks live box every 10 min, takes over if down
    • Auto-restores from s3://hermes-vps-backups/hermes-full-backup/
    • Provider diversity: netcup outage won't kill both Core and standby
  • Recovery priority: Hermes first → infrastructure monitoring (Uptime Kuma, Grafana) → App1 services → App2 services
  • Backup scripts: 24 scripts on Core, 1 on App3 (wphost02 DECOMMISSIONED 2026-08-28; its backup script and S3 path were removed)
Reference in New Issue View Git Blame Copy Permalink