ippadmin/itpp-infrastructure
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

anita-mnz: close DB backup coverage gap, document her box in the backup plan

Browse Source 
root-essentials-backup.sh excludes *.db by design; Core backstops that with hermes-backup.sh,
the migrated box did not. Her conversation store had zero backup coverage. Added
hermes-db-backup.sh (03:10 daily, sqlite3 .backup + quick_check + download-and-verify),
restore test passed (291 MB, quick_check ok, 99285 messages). Backup plan had no entry for
her box at all, now it does.
This commit is contained in:
Sho'Nuff
2026-09-11 17:32:26 -04:00
parent 77a7cb172a
commit 3132475f83
2 changed files with 36 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+8 -1
View File
@@ -6,12 +6,19 @@
- **Why:** repeated `state.db` corruption on Core under co-tenant memory pressure (4th recurrence, `system_prompts` then `sessions` pages). A dedicated box removes the co-tenancy.
- **Method:** clean staged base + row-tolerant live-tail graft, so the corrupt live file never transferred. Signatures matched exactly: `99283/99283/166244414/157` (messages/maxid/contentbytes/sessions), `integrity_check ok`.
- **Post-move state:** Core `hermes-gateway-anita.service` stopped and **disabled** (no double-poller risk; standby app1-bu carries no anita unit); target gateway `active`+`enabled`, telegram `connected`, 0 `getUpdates` conflicts.
- **Backup:** her own box now runs the 3 AM root-essentials backup → `s3://hermes-vps-backups/root-backup/anita-mnz/`. Full run tested end-to-end 2026-09-11 15:56 (163 MB, upload + download/extract verify OK).
- **Backup:** her own box runs the 3 AM root-essentials backup → `s3://hermes-vps-backups/root-backup/anita-mnz/`. Full run tested end-to-end 2026-09-11 15:56 (170 MB, upload + download/extract verify OK).
- **⚠ Correction to the line above (17:35, same day):** that archive is config only. `root-essentials-backup.sh` excludes `*.db` by design, and nothing on the new box replaced that exclusion, so her conversation store, cron execution DB, notepad, wisdom and verification DBs had **zero** backup coverage. Fixed the same evening — details below.
- **Core change:** `hermes-live-sync` no longer snapshots the frozen `profiles/anita` copy (it would advertise a stale db as "live").
- **MCP servers:** stripped from her profile (decision 2026-09-11: "She doesn't need access to those"). The `mcp_servers` block (`dre`, `osint-person`, `super-search` → `127.0.0.1:8900/8902/8899`) was removed from `anita/config.yaml`, which is why her log had been retry-parking those three every ~5 minutes since the move. Backed up first as `config.yaml.bak-mcpstrip-*`; 13 top-level keys verified intact. Takes effect on her next gateway restart.
- **Retired / order cancelled:** Nuremberg `89.58.44.96` (`v2202609377162518632.nicesrv.de`). **Correction to how this was first written here:** it does not "hold nothing". It holds a bare default-profile Hermes install: no `state.db`, empty `sessions/` and `cron/`, no systemd user units, no gateway process (only node_exporter, containerd, sshd). No unique data, so nothing needs preserving before cancellation. Recorded in `/root/.hermes/references/decommissioned-hosts.json`.
- **Monitoring fix:** `health-master-watchdog.py` watch-listed `hermes-gateway-anita.service` as a LOCAL user unit on Core, so it would have alerted forever once that unit was disabled. Local check removed; `anita-mnz` (`159.195.16.30`) added to `REMOTE_SERVERS` and a new `REMOTE_USER_UNITS` remote user-unit check added (SSH + `XDG_RUNTIME_DIR`). Verified live: no false alert for her gateway, and her box answers as active.
- **Docs:** full incident + pitfalls in `/root/.hermes/references/dr-issue-log.md`; transferable procedure in the `hermes-migration` skill.
- **Backup coverage gap (found 17:20, fixed 17:35):** the nightly archive on her box was green and 170 MB, but a restore test showed it contains no `.db` file at all. `root-essentials-backup.sh` excludes `*.db` (correct — never `tar` a live SQLite file), which on Core is backstopped by `hermes-backup.sh`. The migrated box got the script set without the backstop. **Silent failure mode**: every indicator said "backed up" while her 974 MB store had no copy anywhere except the frozen Core dir being deleted.
- **Fix:** new `hermes-db-backup.sh` on `anita-mnz` (`/root/hermes-db-backup.sh`, chmod 750), cron `10 3 * * *`. Per-DB `sqlite3 .backup` (safe on a live DB), `PRAGMA quick_check` on every snapshot before it is accepted, one dated tarball, upload to `s3://hermes-vps-backups/root-backup/anita-mnz/db/`, then **downloads the object back, extracts and re-verifies**. 14-day retention.
- **Verified:** first run 17:04 EDT, 291 MB uploaded; restore test passed — extracted, `quick_check=ok`, 99,285 messages.
- **Migration completeness proven before deleting the Core copy:** `memories/MEMORY.md`, `memories/USER.md` and `.env` are md5-identical between the frozen Core copy and `anita-mnz`; `cron/jobs.json` holds the same six job IDs; skills 125 on MNZ vs 124 frozen; store 99,285 vs 99,283 messages. `config.yaml` differs only by the deliberate MCP strip.
- **Frozen Core copy removed:** `/root/.hermes/profiles/anita` (8.0 GB) was 7.9 GB of corrupt-DB corpses (`corrupt-20260903/09/10`, `pre-rebuild-20260910`, `recovered-20260910`). Archived with sha256 to `s3://hermes-vps-backups/decommissioned/`, size-verified, then deleted. Also removes that copy from Core's nightly archive, which is why Core's backup had grown to 2.97 GB.
- **Same-day correction on job status:** "Germaine inbox watch" is `enabled=false` on `anita-mnz`. It was `enabled=false` in the frozen Core copy too — the migration did not disable it.
## 2026-08-17 - Scirium v2 Proposal Deployed to /scirium/