ippadmin/itpp-infrastructure
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
58de741d94eb57531966e7882e740947fb006afa
itpp-infrastructure/backup-plan.md
T

23 KiB
Raw Blame History

ITPP Backup Plan

Last updated: 2026-09-11 Scope: All ITPP infrastructure backups — Hermes core, application servers, routers, and external services. Storage: Wasabi S3 (us-east-1) via --endpoint-url https://s3.us-east-1.wasabisys.com Cron backend: Hybrid — system crontab + Hermes cron jobs (both on Core) Verification audit: DR issue log

Backup Inventory (27 targets)

Core (152.53.192.33) — RS 2000

Service Method Destination Schedule Last Verified
Hermes Agent (full) hermes-backup.sh — tar.gz of config, skills, profiles, sessions s3://hermes-vps-backups/hermes-full-backup/ 1:00 AM 2026-07-28
Hermes Live Sync hermes-live-sync cron — session state + profiles DB s3://hermes-vps-backups/live/ Every 15 min 2026-07-28 20:42
/root Essentials root-essentials-backup.sh — dotfiles, keys, scripts s3://hermes-vps-backups/root-backup/ 3:00 AM 2026-07-28
Grafana core-services-backup.sh — SQLite DB dump s3://hermes-vps-backups/core/grafana/ 1:30 AM 2026-07-28
Uptime Kuma core-services-backup.sh — SQLite DB dump s3://hermes-vps-backups/core/uptime-kuma/ 1:30 AM 2026-07-28
Docker Stack Definitions docker-stack-defs-backup.sh — compose, .env, configs, service code under /root/docker s3://hermes-vps-backups/docker-stack-defs/ Core 1:40 AM, app4 5:10 AM 2026-09-15
Core App Data + Code app-data-backup.sh — 10 SQLite DBs via .backup, pg_dump hotnow, app code s3://hermes-vps-backups/app-data/ 2:15 AM, Sun 2:50 AM 2026-09-15
Prometheus core-services-backup.sh — TSDB snapshot s3://hermes-vps-backups/core/prometheus/ 1:30 AM 2026-07-28
Auth API auth-api-backup.sh — SQLite .backup + .env s3://hermes-vps-backups/core/auth-api/ 3:15 AM 2026-08-08

App1 (152.53.36.131) — RS 4000

Service Method Destination Schedule Last Verified
Open WebUI app1-backup.sh — data dir tar.gz s3://hermes-vps-backups/app1/openwebui/ 2:00 AM 2026-07-28
LiteLLM litellm-backup.sh — Postgres DB dump + .env s3://hermes-vps-backups/app1/litellm/ 3:30 AM 2026-07-28
n8n app1-backup.sh — Postgres DB dump s3://hermes-vps-backups/app1/n8n/ 2:00 AM 2026-07-28
MCP Server Configs app1-backup.sh — MCP settings files s3://hermes-vps-backups/app1/mcp/ 2:00 AM 2026-07-28
Vaultwarden vaultwarden-backup.sh — SQLite DB dump s3://hermes-vps-backups/app1/vaultwarden/ 2:30 AM 2026-07-28
Komodo komodo-backup.sh — MongoDB dump + compose + keys s3://hermes-vps-backups/app1/komodo/ 3:45 AM 2026-07-28
DocuSeal docuseal-backup.sh — SQLite DB + attachments s3://hermes-vps-backups/app1/docuseal/ 4:00 AM 2026-07-28
Twenty CRM twenty-backup.sh — Postgres dump + .env + compose s3://hermes-vps-backups/app1/twenty/ 4:15 AM 2026-07-28
Kokoro TTS (stateless — no backup needed) — — —

App2 (152.53.39.202) — RS 4000

Service Method Destination Schedule Last Verified
Hudu hudu-backup.sh — Postgres dump s3://hermes-vps-backups/hudu/backups/ 7:00 AM 2026-07-28
Gitea gitea-backup.sh — repos + DB dump s3://hermes-vps-backups/gitea/daily/ 8:00 AM 2026-07-28
UNMS unms-backup-sync.sh — auto-backup .unms s3://hermes-vps-backups/unms-backups/live/backups/ 6:00 AM 2026-07-28
UniFi unifi-backup-sync.sh — auto-backup .unf s3://hermes-vps-backups/unifi-backups/ 2:00 AM 2026-07-18
Traccar app2-backup.sh — H2 DB + config s3://hermes-vps-backups/app2/traccar/ 2:30 AM 2026-07-28
Technitium DNS technitium-backup.sh — data dir + compose s3://hermes-vps-backups/app2/technitium/ 2:45 AM 2026-08-08
Dawarich dawarich-backup.sh — PostgreSQL dump (remote SSH) s3://hermes-vps-backups/app2/dawarich/ 4:00 AM 2026-08-08
RAGFlow ragflow-backup.sh — MySQL dump (remote SSH) s3://hermes-vps-backups/app2/ragflow/ 4:15 AM 2026-08-08

App3 (152.53.241.111) — RS 4000

Service Method Destination Schedule Last Verified
CloudPanel DB app3-backup.sh — SQLite DB s3://hermes-vps-backups/app3/cloudpanel/ 3:00 AM 2026-07-28
MySQL (all DBs) app3-backup.sh — mysqldump s3://hermes-vps-backups/app3/mysql/ 3:00 AM 2026-07-28
WordPress Files app3-backup.sh — wp-content tar.gz s3://hermes-vps-backups/app3/wordpress/ 3:00 AM 2026-07-28
Nginx Configs app3-backup.sh — sites-enabled + config s3://hermes-vps-backups/app3/config/ 3:00 AM 2026-07-28
Static Sites app3-backup.sh — all non-WordPress htdocs (modelortho.com, verdicttank, transitpin, katiewatts, etc.) s3://hermes-vps-backups/app3/static/ 3:00 AM 2026-08-08
WordPress Snapshots /opt/backup-restore/snapshot.sh — per-site tar.gz /opt/backup-restore/snapshots/ (local, 30-day retention) 1 AM / 1 PM 2026-08-08
Hexclave (Stack Auth) hexclave-backup.sh — PG dump + compose + env s3://hermes-vps-backups/app3/hexclave/ 3:30 AM 2026-08-08
modelortho.com modelortho-backup.sh — htdocs tar.gz + nginx configs s3://hermes-vps-backups/app3/modelortho/ 4:30 AM 2026-08-08

wphost02 (5.161.62.38) — Hetzner CPX21 — DECOMMISSIONED (2026-08-28)

wphost02 was deleted from the Hetzner account on 2026-08-28. Its backup job has been removed; sites now live on app3 (covered under the App3 backup section above).

anita-mnz (159.195.16.30) - netcup (Anita's dedicated box, cut over 2026-09-11)

Service Method Destination Schedule Last Verified
/root Essentials root-essentials-backup.sh - dotfiles, profile, keys, scripts (excludes *.db by design) s3://hermes-vps-backups/root-backup/anita-mnz/ 3:00 AM 2026-09-11
Hermes Profile DBs hermes-db-backup.sh - sqlite3 .backup snapshots of state.db + cron + wisdom + notepad + verification DBs, quick_check per snapshot, upload then download-and-verify s3://hermes-vps-backups/root-backup/anita-mnz/db/ 3:10 AM 2026-09-11 (restore test passed: quick_check ok, 99,285 messages)
/root Essentials root-essentials-backup.sh — dotfiles, keys, scripts (excludes *.db by design) s3://hermes-vps-backups/root-backup/app4/ 4:45 AM 2026-09-15 (first run verified: upload + download/extract OK)
/root Essentials root-essentials-backup.sh — dotfiles, keys, scripts (excludes *.db by design) s3://hermes-vps-backups/root-backup/core-bu/ 5:15 AM 2026-09-15 (first run verified: upload + download/extract OK)

Home Router

Service Method Destination Schedule Last Verified
MikroTik CCR2004 run-wisp-backup.sh — export .rsc via SSH s3://mikrotik-ccr-backups/wisp-backups/configs/home/ 6:00 AM 2026-07-28

External

Service Method Destination Schedule Last Verified
Hetzner Snapshots snapshot-hetzner.py — API-driven disk snapshots Hetzner Cloud (weekly) Mon 5:00 AM —

Schedules Summary (ET)

Time What Runner Script
Every 15 min Hermes session state Hermes cron hermes-live-sync
1:00 AM Full Hermes backup crontab hermes-backup.sh
1:30 AM Grafana, Uptime Kuma, Prometheus crontab core-services-backup.sh
1:40 AM Docker stack definitions (/root/docker) crontab docker-stack-defs-backup.sh
2:15 AM Core app data: 10 SQLite DBs + hotnow pg_dump + app code crontab app-data-backup.sh
Sun 2:50 AM Core app upload/attachment blobs (~195 MB) crontab app-data-backup.sh --full
2:00 AM Open WebUI, n8n, MCP configs (App1) + UniFi sync (App2) crontab app1-backup.sh, unifi-backup-sync.sh
2:30 AM Vaultwarden (App1) + Traccar (App2) Hermes cron / crontab vaultwarden-backup.sh, app2-backup.sh
2:45 AM Technitium DNS (App2) Hermes cron technitium-backup.sh
3:00 AM /root essentials + App3 (CloudPanel, MySQL, WP, Nginx) crontab root-essentials-backup.sh, app3-backup.sh
3:00 AM anita-mnz /root essentials Anita crontab root-essentials-backup.sh
3:10 AM anita-mnz Hermes profile DB snapshots Anita crontab hermes-db-backup.sh
3:15 AM Auth API (Core) Hermes cron auth-api-backup.sh
3:30 AM Hexclave / Stack Auth (App3) + LiteLLM (App1) Hermes cron / crontab hexclave-backup.sh, litellm-backup.sh
3:45 AM Komodo (App1) crontab komodo-backup.sh
4:00 AM Dawarich (App2) Hermes cron dawarich-backup.sh
4:00 AM DocuSeal (App1) Hermes cron docuseal-backup.sh
4:15 AM RAGFlow (App2) Hermes cron ragflow-backup.sh
4:15 AM Twenty CRM (App1) Hermes cron twenty-backup.sh
4:45 AM app4 /root essentials app4 crontab root-essentials-backup.sh
5:15 AM core-bu /root essentials core-bu crontab root-essentials-backup.sh
6:00 AM MikroTik CCR + UNMS sync Hermes cron run-wisp-backup.sh, unms-backup-sync.sh
1 AM / 1 PM WordPress per-site snapshots (App3, local) App3 crontab /opt/backup-restore/snapshot.sh
7:00 AM Hudu (App2) Hermes cron hudu-backup.sh
8:00 AM Gitea (App2) Hermes cron gitea-backup.sh
Mon 5:00 AM Hetzner weekly snapshots Hermes cron snapshot-hetzner.py

Script Inventory

On Core (/root/.hermes/scripts/)

Script Purpose Runs
hermes-backup.sh Full Hermes tar.gz to S3 crontab 1:00 AM
core-services-backup.sh Grafana, Uptime Kuma, Prometheus crontab 1:30 AM
docker-stack-defs-backup.sh /root/docker stack definitions (compose, .env, configs, code) crontab 1:40 AM (app4 5:10 AM)
app-data-backup.sh /opt + /root app SQLite DBs, hotnow pg_dump, app code (+ blobs Sun) crontab 2:15 AM / Sun 2:50 AM
root-essentials-backup.sh /root keys, configs, scripts crontab 3:00 AM
backup-audit-check.sh Verify recent backup timestamps crontab 2:00 AM
vaultwarden-backup.sh Vaultwarden SQLite dump (SSH to App1) Hermes cron 2:30 AM
litellm-backup.sh LiteLLM Postgres dump + config (SSH to App1) Hermes cron 3:30 AM
komodo-backup.sh Komodo MongoDB dump (SSH to App1) Hermes cron 3:45 AM
docuseal-backup.sh DocuSeal SQLite + attachments (SSH to App1) Hermes cron 4:00 AM
twenty-backup.sh Twenty CRM Postgres dump (SSH to App1) Hermes cron 4:15 AM
app1-backup.sh Open WebUI, n8n, MCP configs (SSH to App1) App1 crontab 2:00 AM
app2-backup.sh Traccar backup (SSH to App2) App2 crontab 2:30 AM
technitium-backup.sh Technitium DNS backup (SSH to App2) Hermes cron 2:45 AM
app3-backup.sh CloudPanel, MySQL, WP, Nginx (SSH to App3) crontab 3:00 AM
auth-api-backup.sh Auth API SQLite + config (Core local) Hermes cron 3:15 AM
hexclave-backup.sh Stack Auth PG dump + compose + env (SSH to App3) Hermes cron 3:30 AM
dawarich-backup.sh Dawarich PostgreSQL dump + compose + env (SSH to App2) Hermes cron 4:00 AM
ragflow-backup.sh RAGFlow MySQL dump + compose + env (SSH to App2) Hermes cron 4:15 AM
run-wisp-backup.sh MikroTik CCR config export Hermes cron 6:00 AM
unms-backup-sync.sh UNMS auto-backup sync to S3 Hermes cron 6:00 AM
unifi-backup-sync.sh UniFi auto-backup sync to S3 Hermes cron 2:00 AM
hudu-backup.sh Hudu Postgres dump to S3 Hermes cron 7:00 AM
gitea-backup.sh Gitea repos + DB dump to S3 Hermes cron 8:00 AM
snapshot-hetzner.py Hetzner server disk snapshots via API Hermes cron Mon 5:00 AM

On App3 (/opt/backup-restore/)

Script Purpose Runs
snapshot.sh Per-site WordPress tarball + DB App3 crontab 6 AM / 6 PM

On anita-mnz (/root/)

Script Purpose Runs
root-essentials-backup.sh /root essentials (profile, keys, scripts, configs) crontab 3:00 AM
hermes-db-backup.sh sqlite3 .backup snapshots of Hermes profile DBs + per-snapshot quick_check + restore test crontab 3:10 AM

S3 Bucket Structure

hermes-vps-backups/
├── hermes-full-backup/     — Full Hermes daily (tar.gz)
├── live/                   — Hermes live sync (every 15 min)
├── decommissioned/         — archived copies of decommissioned systems (profile archives)
├── live-sync/              — Old sync format (deprecated)
├── root-backup/            — /root essentials (Core uses the flat legacy prefix)
│   └── <host>/             — per-host prefix (`anita-mnz`, ...)
│       └── db/             — SQLite `.backup` snapshots (`hermes-db-backup.sh`)
├── standby/                — Standby configs + recovery bundle
├── core/
│   ├── grafana/            — Grafana SQLite DB
│   ├── uptime-kuma/        — Uptime Kuma SQLite DB
│   ├── prometheus/         — Prometheus TSDB snapshots
│   ├── vaultwarden/        — [STALE — service migrated to App1]
│   ├── twenty/             — [STALE — service migrated to App1]
│   ├── searxng/            — [STALE — service removed]
│   └── komodo/             — [STALE — service migrated to App1]
├── app1/
│   ├── openwebui/          — Open WebUI data
│   ├── litellm/            — LiteLLM Postgres dump + config
│   ├── n8n/                — n8n Postgres dump
│   ├── mcp/                — MCP server configs
│   ├── vaultwarden/        — Vaultwarden SQLite dump
│   ├── komodo/             — Komodo MongoDB dump
│   ├── docuseal/           — DocuSeal SQLite + attachments
│   └── twenty/             — Twenty CRM Postgres dump
├── app2/
│   └── traccar/            — Traccar H2 DB + config
├── app3/
│   ├── cloudpanel/         — CloudPanel SQLite DB
│   ├── mysql/              — All MySQL DBs
│   ├── wordpress/          — wp-content tar.gz
│   └── config/             — Nginx configs
├── hudu/backups/           — Hudu Postgres dump
├── gitea/daily/            — Gitea repos + DB
├── unms-backups/live/      — UNMS auto-backups
├── unifi-backups/          — UniFi controller backups
├── volumes/                — Docker volume dumps
├── caddy/                  — (unused)
├── assets/                 — Static assets
├── snapshots/              — (unused)
└── system-configs-*.tar.gz — Historical system config snapshots

mikrotik-ccr-backups/
└── wisp-backups/configs/
    └── home/               — CCR2004 export .rsc

Recovery Objectives (RPO / RTO)

Tier Services RPO RTO Notes
Critical Hermes Agent, Gitea, Traccar, UISP ≤ 1 hour ≤ 4 hours Live sync + daily backups; restore from S3 then replay live-sync
High LiteLLM, n8n, Open WebUI, Vaultwarden, Twenty CRM 24 hours ≤ 8 hours Daily backups; restore from previous night's dump
Medium Hudu, UniFi, Komodo, DocuSeal, App3 WP sites, Auth API, Hexclave (Stack Auth) 24 hours ≤ 24 hours
Low Grafana, Uptime Kuma, Prometheus, MikroTik CCR, Technitium DNS, Dawarich, RAGFlow 24 hours ≤ 48 hours

Restore Testing Cadence

Quarterly: Pick one random backup per tier, restore to a staging location, verify integrity. After any major infra change: Test the affected service's restore path. Annual: Full DR simulation — restore all Critical + High tier services to staging from S3.

Stale S3 Paths — Cleanup Queue

These paths contain data from services that migrated off Core (Jul 28, 2026) or were removed:

Path Status Action
core/vaultwarden/ Stale 11 days Safe to delete — Vaultwarden migrated to App1; new backups at app1/vaultwarden/
core/twenty/ Stale 11 days Safe to delete — Twenty CRM migrated to App1; new backups at app1/twenty/
core/searxng/ Stale 11 days Safe to delete — SearXNG removed; replaced by Super Search
core/komodo/ Stale 11 days Safe to delete — Komodo migrated to App1; new backups at app1/komodo/
caddy/ Unused Safe to delete — Never populated
snapshots/ Unused Safe to delete — Never populated
live/profiles/anita/ Purged 2026-09-11 Deleted — 20,551 objects / 5.0 GB of the frozen Core copy of Anita's profile, pushed by the 15-minute sync before it was paused Sep 3. Superseded by decommissioned/anita-core-frozen-profile-20260911-1732.tar.gz

Unbacked Services

Verified 2026-09-15 — two real gaps found and closed.

Gap 1: /root/docker had no coverage. The plan previously claimed a "Docker Volumes" leg writing to s3://hermes-vps-backups/volumes/. That leg does not exist in core-services-backup.sh (which only covers Grafana, Uptime Kuma and Prometheus), and the two scripts that would do it — docker-volume-sync.sh and hermes-docker-sync.sh — are present on Core but scheduled nowhere. Both are orphans. Closed with docker-stack-defs-backup.sh: 557 files / 682 KB covering the definitions of all 9 stacks, deliberately excluding venv/, node_modules/, data/, database/, packages/, caches and anything ≥ 256 KiB, because /root/docker is 5.8 GB of mostly reproducible installs and volume state that has its own per-service backups. Restore proven 2026-09-15: .env, settings.yml and prometheus.yml all recovered.

Gap 2: the non-Docker apps under /opt and /root had zero coverage. Live databases with no backup: ops-portal/ops.db, rally/data/rally.db, dre-portal/data/dre.db, seemytrip/data/seemytrip.db, verdicttank/users.db, pta-survey/survey.db, pta-registration/pta.db, mooresunnydaze/db.sqlite, intelsight-api/intelsight.db, projects/pipeline/pipeline.db, plus the host PostgreSQL database hotnow and irreplaceable upload/attachment blobs (182 MB of SeeMyTrip uploads alone). Three of these apps run from /root, not /opt, and were missed by every earlier sweep: hotnow-api, intelsight-api, pipeline. Closed with app-data-backup.sh — daily DBs + Postgres + app code, weekly blobs. SQLite is snapshotted with the .backup API and quick_checked, never copied raw. Restore proven 2026-09-15: 10/10 DBs quick_check=ok, hotnow.dump a valid pg_restore archive (52 TOC entries), 275 code files recovered with ops-portal/server.py byte-identical to live.

Pitfall recorded: GNU find -size -1M rounds up, so it matches only empty files — a backup built with that predicate archives 240 zero-byte files and none of the configs, while still reporting success. Size filters in backup scripts must use byte units (-size -262144c).

Still open: the /opt apps that are not git checkouts now have code, data and configs captured, but nothing captures the deployment state (systemd unit files) for them; and the app1-bu retirement means its backup set must be inherited by the replacement US box.

Done 2026-09-11 (Anita migration closeout). The frozen Core copy /root/.hermes/profiles/anita (8.0 GB, excluded from live sync after the 15:53 cutover) was archived to s3://hermes-vps-backups/decommissioned/anita-core-frozen-profile-20260911-1732.tar.gz — 2,786,082,561 B, sha256 b457fa7f…b3dc, 21,206 entries — then deleted, reclaiming 8 GB (125 GB → 117 GB used). The archive was proven by downloading the object back and matching the sha256 against the local tarball before anything was removed. The 8 GB was almost entirely quarantine artifacts from the Sep 3/9/10 SQLite corruption events (corrupt-20260903/09/10, pre-rebuild-20260910, recovered-20260910); the live profile had already been proven migrated (memories, .env and the six cron job IDs identical on the new box, which carries one more skill and two more messages).

Gap found and closed 2026-09-11 (anita-mnz): root-essentials-backup.sh correctly excludes *.db so it never archives a live SQLite file, but nothing replaced it, so after the box was provisioned Anita's conversation store (974 MB), the cron execution DB, notepad, wisdom and verification DBs had zero backup coverage. Core has hermes-backup.sh doing this job; the migrated box had no equivalent. hermes-db-backup.sh (3:10 AM) now snapshots each DB with the sqlite3 .backup API, quick_checks every snapshot, uploads one dated tarball, then downloads it back and verifies the restored store. Verified 2026-09-11 17:04: restore test passed, quick_check=ok, 99,285 messages. Any box migrated with this script set needs the same check, because the *.db exclusion is silent.

6 previously-unbacked services now backed up (Auth API, Technitium DNS, Dawarich, RAGFlow, Stack Auth/Hexclave, app3 static sites). The earlier count of "14" was an error — no source document supports that number. The actual delta since July 28 is 6.

Migration History (Jul 28, 2026)

All the following services were migrated from Core to App1 in a single session:

Service Old Home New Home Backup Before Backup After
Vaultwarden Core (:8080) App1 (:8081) core/vaultwarden/ (stale) app1/vaultwarden/ ✅
SearXNG Core (:8888) Removed core/searxng/ (stale) N/A (replaced by Super Search)
Twenty CRM Core (:3003) App1 (:3003) core/twenty/ (stale) app1/twenty/ ✅
Komodo Core (:9120) App1 (:9120) core/komodo/ (stale) app1/komodo/ ✅
DocuSeal Core (:3000) App1 (:3002) No backup existed app1/docuseal/ ✅
Kokoro TTS Core (:8880) App1 (:8880) N/A (stateless) N/A (stateless)

Disaster Recovery

  • Live Hermes: netcup VPS — core.itpropartner.com — 152.53.192.33
  • Standby Hermes: Hetzner CPX21 — app1-bu.itpropartner.com — 5.161.225.131
    • Cron checks live box every 10 min, takes over if down
    • Auto-restores from s3://hermes-vps-backups/hermes-full-backup/
    • Provider diversity: netcup outage won't kill both Core and standby
  • Recovery priority: Hermes first → infrastructure monitoring (Uptime Kuma, Grafana) → App1 services → App2 services
  • Backup scripts: 24 scripts on Core, 1 on App3 (wphost02 DECOMMISSIONED 2026-08-28; its backup script and S3 path were removed)
Reference in New Issue View Git Blame Copy Permalink