diff --git a/CHANGELOG.md b/CHANGELOG.md index 1d718bc..7d5f23d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,12 +6,19 @@ - **Why:** repeated `state.db` corruption on Core under co-tenant memory pressure (4th recurrence, `system_prompts` then `sessions` pages). A dedicated box removes the co-tenancy. - **Method:** clean staged base + row-tolerant live-tail graft, so the corrupt live file never transferred. Signatures matched exactly: `99283/99283/166244414/157` (messages/maxid/contentbytes/sessions), `integrity_check ok`. - **Post-move state:** Core `hermes-gateway-anita.service` stopped and **disabled** (no double-poller risk; standby app1-bu carries no anita unit); target gateway `active`+`enabled`, telegram `connected`, 0 `getUpdates` conflicts. -- **Backup:** her own box now runs the 3 AM root-essentials backup → `s3://hermes-vps-backups/root-backup/anita-mnz/`. Full run tested end-to-end 2026-09-11 15:56 (163 MB, upload + download/extract verify OK). +- **Backup:** her own box runs the 3 AM root-essentials backup → `s3://hermes-vps-backups/root-backup/anita-mnz/`. Full run tested end-to-end 2026-09-11 15:56 (170 MB, upload + download/extract verify OK). +- **⚠ Correction to the line above (17:35, same day):** that archive is config only. `root-essentials-backup.sh` excludes `*.db` by design, and nothing on the new box replaced that exclusion, so her conversation store, cron execution DB, notepad, wisdom and verification DBs had **zero** backup coverage. Fixed the same evening — details below. - **Core change:** `hermes-live-sync` no longer snapshots the frozen `profiles/anita` copy (it would advertise a stale db as "live"). - **MCP servers:** stripped from her profile (decision 2026-09-11: "She doesn't need access to those"). The `mcp_servers` block (`dre`, `osint-person`, `super-search` → `127.0.0.1:8900/8902/8899`) was removed from `anita/config.yaml`, which is why her log had been retry-parking those three every ~5 minutes since the move. Backed up first as `config.yaml.bak-mcpstrip-*`; 13 top-level keys verified intact. Takes effect on her next gateway restart. - **Retired / order cancelled:** Nuremberg `89.58.44.96` (`v2202609377162518632.nicesrv.de`). **Correction to how this was first written here:** it does not "hold nothing". It holds a bare default-profile Hermes install: no `state.db`, empty `sessions/` and `cron/`, no systemd user units, no gateway process (only node_exporter, containerd, sshd). No unique data, so nothing needs preserving before cancellation. Recorded in `/root/.hermes/references/decommissioned-hosts.json`. - **Monitoring fix:** `health-master-watchdog.py` watch-listed `hermes-gateway-anita.service` as a LOCAL user unit on Core, so it would have alerted forever once that unit was disabled. Local check removed; `anita-mnz` (`159.195.16.30`) added to `REMOTE_SERVERS` and a new `REMOTE_USER_UNITS` remote user-unit check added (SSH + `XDG_RUNTIME_DIR`). Verified live: no false alert for her gateway, and her box answers as active. - **Docs:** full incident + pitfalls in `/root/.hermes/references/dr-issue-log.md`; transferable procedure in the `hermes-migration` skill. +- **Backup coverage gap (found 17:20, fixed 17:35):** the nightly archive on her box was green and 170 MB, but a restore test showed it contains no `.db` file at all. `root-essentials-backup.sh` excludes `*.db` (correct — never `tar` a live SQLite file), which on Core is backstopped by `hermes-backup.sh`. The migrated box got the script set without the backstop. **Silent failure mode**: every indicator said "backed up" while her 974 MB store had no copy anywhere except the frozen Core dir being deleted. +- **Fix:** new `hermes-db-backup.sh` on `anita-mnz` (`/root/hermes-db-backup.sh`, chmod 750), cron `10 3 * * *`. Per-DB `sqlite3 .backup` (safe on a live DB), `PRAGMA quick_check` on every snapshot before it is accepted, one dated tarball, upload to `s3://hermes-vps-backups/root-backup/anita-mnz/db/`, then **downloads the object back, extracts and re-verifies**. 14-day retention. +- **Verified:** first run 17:04 EDT, 291 MB uploaded; restore test passed — extracted, `quick_check=ok`, 99,285 messages. +- **Migration completeness proven before deleting the Core copy:** `memories/MEMORY.md`, `memories/USER.md` and `.env` are md5-identical between the frozen Core copy and `anita-mnz`; `cron/jobs.json` holds the same six job IDs; skills 125 on MNZ vs 124 frozen; store 99,285 vs 99,283 messages. `config.yaml` differs only by the deliberate MCP strip. +- **Frozen Core copy removed:** `/root/.hermes/profiles/anita` (8.0 GB) was 7.9 GB of corrupt-DB corpses (`corrupt-20260903/09/10`, `pre-rebuild-20260910`, `recovered-20260910`). Archived with sha256 to `s3://hermes-vps-backups/decommissioned/`, size-verified, then deleted. Also removes that copy from Core's nightly archive, which is why Core's backup had grown to 2.97 GB. +- **Same-day correction on job status:** "Germaine inbox watch" is `enabled=false` on `anita-mnz`. It was `enabled=false` in the frozen Core copy too — the migration did not disable it. ## 2026-08-17 - Scirium v2 Proposal Deployed to /scirium/ diff --git a/backup-plan.md b/backup-plan.md index e8407f6..bd9c68b 100644 --- a/backup-plan.md +++ b/backup-plan.md @@ -1,6 +1,6 @@ # ITPP Backup Plan -> **Last updated:** 2026-08-08 +> **Last updated:** 2026-09-11 > **Scope:** All ITPP infrastructure backups — Hermes core, application servers, routers, and external services. > **Storage:** Wasabi S3 (us-east-1) via `--endpoint-url https://s3.us-east-1.wasabisys.com` > **Cron backend:** Hybrid — system crontab + Hermes cron jobs (both on Core) @@ -67,6 +67,13 @@ wphost02 was deleted from the Hetzner account on 2026-08-28. Its backup job has been removed; sites now live on app3 (covered under the App3 backup section above). +### anita-mnz (159.195.16.30) - netcup (Anita's dedicated box, cut over 2026-09-11) + +| Service | Method | Destination | Schedule | Last Verified | +|---------|--------|-------------|----------|---------------| +| /root Essentials | `root-essentials-backup.sh` - dotfiles, profile, keys, scripts (**excludes `*.db` by design**) | `s3://hermes-vps-backups/root-backup/anita-mnz/` | 3:00 AM | 2026-09-11 | +| Hermes Profile DBs | `hermes-db-backup.sh` - `sqlite3 .backup` snapshots of state.db + cron + wisdom + notepad + verification DBs, `quick_check` per snapshot, upload then download-and-verify | `s3://hermes-vps-backups/root-backup/anita-mnz/db/` | 3:10 AM | 2026-09-11 (restore test passed: quick_check ok, 99,285 messages) | + ### Home Router | Service | Method | Destination | Schedule | Last Verified | @@ -92,6 +99,8 @@ wphost02 was deleted from the Hetzner account on 2026-08-28. Its backup job has | 2:30 AM | Vaultwarden (App1) + Traccar (App2) | Hermes cron / crontab | `vaultwarden-backup.sh`, `app2-backup.sh` | | 2:45 AM | Technitium DNS (App2) | Hermes cron | `technitium-backup.sh` | | 3:00 AM | /root essentials + App3 (CloudPanel, MySQL, WP, Nginx) | crontab | `root-essentials-backup.sh`, `app3-backup.sh` | +| 3:00 AM | **anita-mnz** /root essentials | Anita crontab | `root-essentials-backup.sh` | +| 3:10 AM | **anita-mnz** Hermes profile DB snapshots | Anita crontab | `hermes-db-backup.sh` | | 3:15 AM | Auth API (Core) | Hermes cron | `auth-api-backup.sh` | | 3:30 AM | Hexclave / Stack Auth (App3) + LiteLLM (App1) | Hermes cron / crontab | `hexclave-backup.sh`, `litellm-backup.sh` | | 3:45 AM | Komodo (App1) | crontab | `komodo-backup.sh` | @@ -143,6 +152,13 @@ wphost02 was deleted from the Hetzner account on 2026-08-28. Its backup job has |--------|---------|------| | `snapshot.sh` | Per-site WordPress tarball + DB | App3 crontab 6 AM / 6 PM | +### On anita-mnz (`/root/`) + +| Script | Purpose | Runs | +|--------|---------|------| +| `root-essentials-backup.sh` | /root essentials (profile, keys, scripts, configs) | crontab 3:00 AM | +| `hermes-db-backup.sh` | `sqlite3 .backup` snapshots of Hermes profile DBs + per-snapshot quick_check + restore test | crontab 3:10 AM | + --- ## S3 Bucket Structure @@ -152,7 +168,9 @@ hermes-vps-backups/ ├── hermes-full-backup/ — Full Hermes daily (tar.gz) ├── live/ — Hermes live sync (every 15 min) ├── live-sync/ — Old sync format (deprecated) -├── root-backup/ — /root essentials +├── root-backup/ — /root essentials (Core uses the flat legacy prefix) +│ └── / — per-host prefix (`anita-mnz`, ...) +│ └── db/ — SQLite `.backup` snapshots (`hermes-db-backup.sh`) ├── standby/ — Standby configs + recovery bundle ├── core/ │ ├── grafana/ — Grafana SQLite DB @@ -228,6 +246,14 @@ These paths contain data from services that migrated off Core (Jul 28, 2026) or These services are running in production with **zero backup coverage**: | Unbacked | *(none)* | N/A | N/A | All services are backed up as of 2026-08-08 | +**Gap found and closed 2026-09-11 (anita-mnz):** `root-essentials-backup.sh` correctly excludes `*.db` so it never archives a +live SQLite file, but nothing replaced it, so after the box was provisioned Anita's conversation store (974 MB), the cron +execution DB, notepad, wisdom and verification DBs had **zero** backup coverage. Core has `hermes-backup.sh` doing this job; +the migrated box had no equivalent. `hermes-db-backup.sh` (3:10 AM) now snapshots each DB with the sqlite3 `.backup` API, +quick_checks every snapshot, uploads one dated tarball, then downloads it back and verifies the restored store. Verified +2026-09-11 17:04: restore test passed, `quick_check=ok`, 99,285 messages. Any box migrated with this script set needs the +same check, because the `*.db` exclusion is silent. + > **6 previously-unbacked services now backed up (Auth API, Technitium DNS, Dawarich, RAGFlow, Stack Auth/Hexclave, app3 static sites).** The earlier count of "14" was an error — no source document supports that number. The actual delta since July 28 is 6. ---