ippadmin/itpp-infrastructure
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

anita-mnz: close DB backup coverage gap, document her box in the backup plan

Browse Source 
root-essentials-backup.sh excludes *.db by design; Core backstops that with hermes-backup.sh,
the migrated box did not. Her conversation store had zero backup coverage. Added
hermes-db-backup.sh (03:10 daily, sqlite3 .backup + quick_check + download-and-verify),
restore test passed (291 MB, quick_check ok, 99285 messages). Backup plan had no entry for
her box at all, now it does.
This commit is contained in:
Sho'Nuff
2026-09-11 17:32:26 -04:00
parent 77a7cb172a
commit 3132475f83
2 changed files with 36 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+8 -1
View File
@@ -6,12 +6,19 @@
- **Why:** repeated `state.db` corruption on Core under co-tenant memory pressure (4th recurrence, `system_prompts` then `sessions` pages). A dedicated box removes the co-tenancy.
- **Method:** clean staged base + row-tolerant live-tail graft, so the corrupt live file never transferred. Signatures matched exactly: `99283/99283/166244414/157` (messages/maxid/contentbytes/sessions), `integrity_check ok`.
- **Post-move state:** Core `hermes-gateway-anita.service` stopped and **disabled** (no double-poller risk; standby app1-bu carries no anita unit); target gateway `active`+`enabled`, telegram `connected`, 0 `getUpdates` conflicts.
- **Backup:** her own box now runs the 3 AM root-essentials backup → `s3://hermes-vps-backups/root-backup/anita-mnz/`. Full run tested end-to-end 2026-09-11 15:56 (163 MB, upload + download/extract verify OK).
- **Backup:** her own box runs the 3 AM root-essentials backup → `s3://hermes-vps-backups/root-backup/anita-mnz/`. Full run tested end-to-end 2026-09-11 15:56 (170 MB, upload + download/extract verify OK).
- **⚠ Correction to the line above (17:35, same day):** that archive is config only. `root-essentials-backup.sh` excludes `*.db` by design, and nothing on the new box replaced that exclusion, so her conversation store, cron execution DB, notepad, wisdom and verification DBs had **zero** backup coverage. Fixed the same evening — details below.
- **Core change:** `hermes-live-sync` no longer snapshots the frozen `profiles/anita` copy (it would advertise a stale db as "live").
- **MCP servers:** stripped from her profile (decision 2026-09-11: "She doesn't need access to those"). The `mcp_servers` block (`dre`, `osint-person`, `super-search` → `127.0.0.1:8900/8902/8899`) was removed from `anita/config.yaml`, which is why her log had been retry-parking those three every ~5 minutes since the move. Backed up first as `config.yaml.bak-mcpstrip-*`; 13 top-level keys verified intact. Takes effect on her next gateway restart.
- **Retired / order cancelled:** Nuremberg `89.58.44.96` (`v2202609377162518632.nicesrv.de`). **Correction to how this was first written here:** it does not "hold nothing". It holds a bare default-profile Hermes install: no `state.db`, empty `sessions/` and `cron/`, no systemd user units, no gateway process (only node_exporter, containerd, sshd). No unique data, so nothing needs preserving before cancellation. Recorded in `/root/.hermes/references/decommissioned-hosts.json`.
- **Monitoring fix:** `health-master-watchdog.py` watch-listed `hermes-gateway-anita.service` as a LOCAL user unit on Core, so it would have alerted forever once that unit was disabled. Local check removed; `anita-mnz` (`159.195.16.30`) added to `REMOTE_SERVERS` and a new `REMOTE_USER_UNITS` remote user-unit check added (SSH + `XDG_RUNTIME_DIR`). Verified live: no false alert for her gateway, and her box answers as active.
- **Docs:** full incident + pitfalls in `/root/.hermes/references/dr-issue-log.md`; transferable procedure in the `hermes-migration` skill.
- **Backup coverage gap (found 17:20, fixed 17:35):** the nightly archive on her box was green and 170 MB, but a restore test showed it contains no `.db` file at all. `root-essentials-backup.sh` excludes `*.db` (correct — never `tar` a live SQLite file), which on Core is backstopped by `hermes-backup.sh`. The migrated box got the script set without the backstop. **Silent failure mode**: every indicator said "backed up" while her 974 MB store had no copy anywhere except the frozen Core dir being deleted.
- **Fix:** new `hermes-db-backup.sh` on `anita-mnz` (`/root/hermes-db-backup.sh`, chmod 750), cron `10 3 * * *`. Per-DB `sqlite3 .backup` (safe on a live DB), `PRAGMA quick_check` on every snapshot before it is accepted, one dated tarball, upload to `s3://hermes-vps-backups/root-backup/anita-mnz/db/`, then **downloads the object back, extracts and re-verifies**. 14-day retention.
- **Verified:** first run 17:04 EDT, 291 MB uploaded; restore test passed — extracted, `quick_check=ok`, 99,285 messages.
- **Migration completeness proven before deleting the Core copy:** `memories/MEMORY.md`, `memories/USER.md` and `.env` are md5-identical between the frozen Core copy and `anita-mnz`; `cron/jobs.json` holds the same six job IDs; skills 125 on MNZ vs 124 frozen; store 99,285 vs 99,283 messages. `config.yaml` differs only by the deliberate MCP strip.
- **Frozen Core copy removed:** `/root/.hermes/profiles/anita` (8.0 GB) was 7.9 GB of corrupt-DB corpses (`corrupt-20260903/09/10`, `pre-rebuild-20260910`, `recovered-20260910`). Archived with sha256 to `s3://hermes-vps-backups/decommissioned/`, size-verified, then deleted. Also removes that copy from Core's nightly archive, which is why Core's backup had grown to 2.97 GB.
- **Same-day correction on job status:** "Germaine inbox watch" is `enabled=false` on `anita-mnz`. It was `enabled=false` in the frozen Core copy too — the migration did not disable it.
## 2026-08-17 - Scirium v2 Proposal Deployed to /scirium/
backup-plan.md
+28 -2
View File
@@ -1,6 +1,6 @@
# ITPP Backup Plan
> **Last updated:** 2026-08-08
> **Last updated:** 2026-09-11
> **Scope:** All ITPP infrastructure backups — Hermes core, application servers, routers, and external services.
> **Storage:** Wasabi S3 (us-east-1) via `--endpoint-url https://s3.us-east-1.wasabisys.com`
> **Cron backend:** Hybrid — system crontab + Hermes cron jobs (both on Core)
@@ -67,6 +67,13 @@
wphost02 was deleted from the Hetzner account on 2026-08-28. Its backup job has been removed; sites now live on app3 (covered under the App3 backup section above).
### anita-mnz (159.195.16.30) - netcup (Anita's dedicated box, cut over 2026-09-11)
| Service | Method | Destination | Schedule | Last Verified |
|---------|--------|-------------|----------|---------------|
| /root Essentials | `root-essentials-backup.sh` - dotfiles, profile, keys, scripts (**excludes `*.db` by design**) | `s3://hermes-vps-backups/root-backup/anita-mnz/` | 3:00 AM | 2026-09-11 |
| Hermes Profile DBs | `hermes-db-backup.sh` - `sqlite3 .backup` snapshots of state.db + cron + wisdom + notepad + verification DBs, `quick_check` per snapshot, upload then download-and-verify | `s3://hermes-vps-backups/root-backup/anita-mnz/db/` | 3:10 AM | 2026-09-11 (restore test passed: quick_check ok, 99,285 messages) |
### Home Router
| Service | Method | Destination | Schedule | Last Verified |
@@ -92,6 +99,8 @@ wphost02 was deleted from the Hetzner account on 2026-08-28. Its backup job has
| 2:30 AM | Vaultwarden (App1) + Traccar (App2) | Hermes cron / crontab | `vaultwarden-backup.sh`, `app2-backup.sh` |
| 2:45 AM | Technitium DNS (App2) | Hermes cron | `technitium-backup.sh` |
| 3:00 AM | /root essentials + App3 (CloudPanel, MySQL, WP, Nginx) | crontab | `root-essentials-backup.sh`, `app3-backup.sh` |
| 3:00 AM | **anita-mnz** /root essentials | Anita crontab | `root-essentials-backup.sh` |
| 3:10 AM | **anita-mnz** Hermes profile DB snapshots | Anita crontab | `hermes-db-backup.sh` |
| 3:15 AM | Auth API (Core) | Hermes cron | `auth-api-backup.sh` |
| 3:30 AM | Hexclave / Stack Auth (App3) + LiteLLM (App1) | Hermes cron / crontab | `hexclave-backup.sh`, `litellm-backup.sh` |
| 3:45 AM | Komodo (App1) | crontab | `komodo-backup.sh` |
@@ -143,6 +152,13 @@ wphost02 was deleted from the Hetzner account on 2026-08-28. Its backup job has
|--------|---------|------|
| `snapshot.sh` | Per-site WordPress tarball + DB | App3 crontab 6 AM / 6 PM |
### On anita-mnz (`/root/`)
| Script | Purpose | Runs |
|--------|---------|------|
| `root-essentials-backup.sh` | /root essentials (profile, keys, scripts, configs) | crontab 3:00 AM |
| `hermes-db-backup.sh` | `sqlite3 .backup` snapshots of Hermes profile DBs + per-snapshot quick_check + restore test | crontab 3:10 AM |
---
## S3 Bucket Structure
@@ -152,7 +168,9 @@ hermes-vps-backups/
├── hermes-full-backup/ — Full Hermes daily (tar.gz)
├── live/ — Hermes live sync (every 15 min)
├── live-sync/ — Old sync format (deprecated)
├── root-backup/ — /root essentials
├── root-backup/ — /root essentials (Core uses the flat legacy prefix)
│ └── <host>/ — per-host prefix (`anita-mnz`, ...)
│ └── db/ — SQLite `.backup` snapshots (`hermes-db-backup.sh`)
├── standby/ — Standby configs + recovery bundle
├── core/
│ ├── grafana/ — Grafana SQLite DB
@@ -228,6 +246,14 @@ These paths contain data from services that migrated off Core (Jul 28, 2026) or
These services are running in production with **zero backup coverage**:
| Unbacked | *(none)* | N/A | N/A | All services are backed up as of 2026-08-08 |
**Gap found and closed 2026-09-11 (anita-mnz):** `root-essentials-backup.sh` correctly excludes `*.db` so it never archives a
live SQLite file, but nothing replaced it, so after the box was provisioned Anita's conversation store (974 MB), the cron
execution DB, notepad, wisdom and verification DBs had **zero** backup coverage. Core has `hermes-backup.sh` doing this job;
the migrated box had no equivalent. `hermes-db-backup.sh` (3:10 AM) now snapshots each DB with the sqlite3 `.backup` API,
quick_checks every snapshot, uploads one dated tarball, then downloads it back and verifies the restored store. Verified
2026-09-11 17:04: restore test passed, `quick_check=ok`, 99,285 messages. Any box migrated with this script set needs the
same check, because the `*.db` exclusion is silent.
> **6 previously-unbacked services now backed up (Auth API, Technitium DNS, Dawarich, RAGFlow, Stack Auth/Hexclave, app3 static sites).** The earlier count of "14" was an error — no source document supports that number. The actual delta since July 28 is 6.
---