ippadmin/itpp-infrastructure
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

anita-mnz: close DB backup coverage gap, document her box in the backup plan

Browse Source 
root-essentials-backup.sh excludes *.db by design; Core backstops that with hermes-backup.sh,
the migrated box did not. Her conversation store had zero backup coverage. Added
hermes-db-backup.sh (03:10 daily, sqlite3 .backup + quick_check + download-and-verify),
restore test passed (291 MB, quick_check ok, 99285 messages). Backup plan had no entry for
her box at all, now it does.
This commit is contained in:
Sho'Nuff
2026-09-11 17:32:26 -04:00
parent 77a7cb172a
commit 3132475f83
2 changed files with 36 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backup-plan.md
+28 -2
View File
@@ -1,6 +1,6 @@
# ITPP Backup Plan
> **Last updated:** 2026-08-08
> **Last updated:** 2026-09-11
> **Scope:** All ITPP infrastructure backups — Hermes core, application servers, routers, and external services.
> **Storage:** Wasabi S3 (us-east-1) via `--endpoint-url https://s3.us-east-1.wasabisys.com`
> **Cron backend:** Hybrid — system crontab + Hermes cron jobs (both on Core)
@@ -67,6 +67,13 @@
wphost02 was deleted from the Hetzner account on 2026-08-28. Its backup job has been removed; sites now live on app3 (covered under the App3 backup section above).
### anita-mnz (159.195.16.30) - netcup (Anita's dedicated box, cut over 2026-09-11)
| Service | Method | Destination | Schedule | Last Verified |
|---------|--------|-------------|----------|---------------|
| /root Essentials | `root-essentials-backup.sh` - dotfiles, profile, keys, scripts (**excludes `*.db` by design**) | `s3://hermes-vps-backups/root-backup/anita-mnz/` | 3:00 AM | 2026-09-11 |
| Hermes Profile DBs | `hermes-db-backup.sh` - `sqlite3 .backup` snapshots of state.db + cron + wisdom + notepad + verification DBs, `quick_check` per snapshot, upload then download-and-verify | `s3://hermes-vps-backups/root-backup/anita-mnz/db/` | 3:10 AM | 2026-09-11 (restore test passed: quick_check ok, 99,285 messages) |
### Home Router
| Service | Method | Destination | Schedule | Last Verified |
@@ -92,6 +99,8 @@ wphost02 was deleted from the Hetzner account on 2026-08-28. Its backup job has
| 2:30 AM | Vaultwarden (App1) + Traccar (App2) | Hermes cron / crontab | `vaultwarden-backup.sh`, `app2-backup.sh` |
| 2:45 AM | Technitium DNS (App2) | Hermes cron | `technitium-backup.sh` |
| 3:00 AM | /root essentials + App3 (CloudPanel, MySQL, WP, Nginx) | crontab | `root-essentials-backup.sh`, `app3-backup.sh` |
| 3:00 AM | **anita-mnz** /root essentials | Anita crontab | `root-essentials-backup.sh` |
| 3:10 AM | **anita-mnz** Hermes profile DB snapshots | Anita crontab | `hermes-db-backup.sh` |
| 3:15 AM | Auth API (Core) | Hermes cron | `auth-api-backup.sh` |
| 3:30 AM | Hexclave / Stack Auth (App3) + LiteLLM (App1) | Hermes cron / crontab | `hexclave-backup.sh`, `litellm-backup.sh` |
| 3:45 AM | Komodo (App1) | crontab | `komodo-backup.sh` |
@@ -143,6 +152,13 @@ wphost02 was deleted from the Hetzner account on 2026-08-28. Its backup job has
|--------|---------|------|
| `snapshot.sh` | Per-site WordPress tarball + DB | App3 crontab 6 AM / 6 PM |
### On anita-mnz (`/root/`)
| Script | Purpose | Runs |
|--------|---------|------|
| `root-essentials-backup.sh` | /root essentials (profile, keys, scripts, configs) | crontab 3:00 AM |
| `hermes-db-backup.sh` | `sqlite3 .backup` snapshots of Hermes profile DBs + per-snapshot quick_check + restore test | crontab 3:10 AM |
---
## S3 Bucket Structure
@@ -152,7 +168,9 @@ hermes-vps-backups/
├── hermes-full-backup/ — Full Hermes daily (tar.gz)
├── live/ — Hermes live sync (every 15 min)
├── live-sync/ — Old sync format (deprecated)
├── root-backup/ — /root essentials
├── root-backup/ — /root essentials (Core uses the flat legacy prefix)
│ └── <host>/ — per-host prefix (`anita-mnz`, ...)
│ └── db/ — SQLite `.backup` snapshots (`hermes-db-backup.sh`)
├── standby/ — Standby configs + recovery bundle
├── core/
│ ├── grafana/ — Grafana SQLite DB
@@ -228,6 +246,14 @@ These paths contain data from services that migrated off Core (Jul 28, 2026) or
These services are running in production with **zero backup coverage**:
| Unbacked | *(none)* | N/A | N/A | All services are backed up as of 2026-08-08 |
**Gap found and closed 2026-09-11 (anita-mnz):** `root-essentials-backup.sh` correctly excludes `*.db` so it never archives a
live SQLite file, but nothing replaced it, so after the box was provisioned Anita's conversation store (974 MB), the cron
execution DB, notepad, wisdom and verification DBs had **zero** backup coverage. Core has `hermes-backup.sh` doing this job;
the migrated box had no equivalent. `hermes-db-backup.sh` (3:10 AM) now snapshots each DB with the sqlite3 `.backup` API,
quick_checks every snapshot, uploads one dated tarball, then downloads it back and verifies the restored store. Verified
2026-09-11 17:04: restore test passed, `quick_check=ok`, 99,285 messages. Any box migrated with this script set needs the
same check, because the `*.db` exclusion is silent.
> **6 previously-unbacked services now backed up (Auth API, Technitium DNS, Dawarich, RAGFlow, Stack Auth/Hexclave, app3 static sites).** The earlier count of "14" was an error — no source document supports that number. The actual delta since July 28 is 6.
---