docs: close out Anita migration (archive verified, frozen copy deleted, stale S3 prefix purged, backup monitor findings)

2026-09-11 17:41:53 -04:00
parent 3132475f83
commit af80a8cb7b
2 changed files with 11 additions and 1 deletions
@@ -17,7 +17,9 @@
 - **Fix:** new `hermes-db-backup.sh` on `anita-mnz` (`/root/hermes-db-backup.sh`, chmod 750), cron `10 3 * * *`. Per-DB `sqlite3 .backup` (safe on a live DB), `PRAGMA quick_check` on every snapshot before it is accepted, one dated tarball, upload to `s3://hermes-vps-backups/root-backup/anita-mnz/db/`, then **downloads the object back, extracts and re-verifies**. 14-day retention.
 - **Verified:** first run 17:04 EDT, 291 MB uploaded; restore test passed — extracted, `quick_check=ok`, 99,285 messages.
 - **Migration completeness proven before deleting the Core copy:** `memories/MEMORY.md`, `memories/USER.md` and `.env` are md5-identical between the frozen Core copy and `anita-mnz`; `cron/jobs.json` holds the same six job IDs; skills 125 on MNZ vs 124 frozen; store 99,285 vs 99,283 messages. `config.yaml` differs only by the deliberate MCP strip.
- **Frozen Core copy removed:** `/root/.hermes/profiles/anita` (8.0 GB) was 7.9 GB of corrupt-DB corpses (`corrupt-20260903/09/10`, `pre-rebuild-20260910`, `recovered-20260910`). Archived with sha256 to `s3://hermes-vps-backups/decommissioned/`, size-verified, then deleted. Also removes that copy from Core's nightly archive, which is why Core's backup had grown to 2.97 GB.
+- **Frozen Core copy removed (17:41):** `/root/.hermes/profiles/anita` (8.0 GB) was 7.9 GB of corrupt-DB corpses (`corrupt-20260903/09/10`, `pre-rebuild-20260910`, `recovered-20260910`). Archived to `s3://hermes-vps-backups/decommissioned/anita-core-frozen-profile-20260911-1732.tar.gz` (2,786,082,561 B, 21,206 entries), **verified by downloading the object back and matching sha256** `b457fa7f0cb6f546f36e53a338e90b7b940f51cf29d5a73d0cceb97f49c9b3dc` against the local tarball, then deleted. Nothing unique was destroyed: memories, `.env` and the six cron job IDs were identical on `anita-mnz`, which carries one more skill and two more messages. Core disk 125 GB → 117 GB used; `/root/.hermes/profiles` is now empty. Also removes that copy from Core's nightly archive, which is why Core's backup had grown to 2.97 GB.
 - **Stale S3 copy purged (17:41):** `s3://hermes-vps-backups/live/profiles/anita/` held 20,551 objects / 5.0 GB that the then-running 15-minute sync had pushed before it was paused on Sep 3, including a stale-looking `state.db` that would have been advertised as "live". Superseded by the archive above and removed; `live/profiles/` is now empty.
 - **Backup monitor checked (17:38):** its single CRITICAL was `hermes-live-sync: DISABLED/PAUSED`, which is the pause from Sep 3 and explains the monitor's 8-day exit-1 streak (it exits 1 only on CRITICAL, never on warnings). Its three WARNINGs are false positives, verified by content rather than size: Wazuh manager tarballs are sha256-identical for 3 days (static config, job ran 03:15 today), LiteLLM's flagged object is a 333 B config yaml that never changes (the real backup is a 37.9 MB `litellm-backup-*.tar.gz` from 03:30 today, and there are 47 of them), and the voipsimplicity dump is the same 6,834,636 B each day but a **different** sha256 each day (valid gzip, 73 tables). The size-uniqueness heuristic cannot tell static-but-fine from stalled; the three entries should be reclassified as "unchanged content" rather than SUSPICIOUS.
 - **Same-day correction on job status:** "Germaine inbox watch" is `enabled=false` on `anita-mnz`. It was `enabled=false` in the frozen Core copy too — the migration did not disable it.
 ## 2026-08-17 - Scirium v2 Proposal Deployed to /scirium/
@@ -167,6 +167,7 @@ wphost02 was deleted from the Hetzner account on 2026-08-28. Its backup job has
 hermes-vps-backups/
 ├── hermes-full-backup/     — Full Hermes daily (tar.gz)
 ├── live/                   — Hermes live sync (every 15 min)
 ├── decommissioned/         — archived copies of decommissioned systems (profile archives)
 ├── live-sync/              — Old sync format (deprecated)
 ├── root-backup/            — /root essentials (Core uses the flat legacy prefix)
 │   └── <host>/             — per-host prefix (`anita-mnz`, ...)
@@ -240,12 +241,19 @@ These paths contain data from services that migrated off Core (Jul 28, 2026) or
 | `core/komodo/` | Stale 11 days | **Safe to delete** — Komodo migrated to App1; new backups at `app1/komodo/` |
 | `caddy/` | Unused | **Safe to delete** — Never populated |
 | `snapshots/` | Unused | **Safe to delete** — Never populated |
 | `live/profiles/anita/` | Purged 2026-09-11 | **Deleted** — 20,551 objects / 5.0 GB of the frozen Core copy of Anita's profile, pushed by the 15-minute sync before it was paused Sep 3. Superseded by `decommissioned/anita-core-frozen-profile-20260911-1732.tar.gz` |
 ## Unbacked Services
 These services are running in production with **zero backup coverage**:
 | Unbacked | *(none)* | N/A | N/A | All services are backed up as of 2026-08-08 |
 **Done 2026-09-11 (Anita migration closeout).** The frozen Core copy `/root/.hermes/profiles/anita` (8.0 GB, excluded from live sync after the 15:53 cutover) was archived to
 `s3://hermes-vps-backups/decommissioned/anita-core-frozen-profile-20260911-1732.tar.gz` — 2,786,082,561 B, sha256 `b457fa7f…b3dc`, 21,206 entries — then deleted, reclaiming 8 GB (125 GB → 117 GB used).
 The archive was proven by downloading the object back and matching the sha256 against the local tarball **before** anything was removed. The 8 GB was almost entirely quarantine
 artifacts from the Sep 3/9/10 SQLite corruption events (`corrupt-20260903/09/10`, `pre-rebuild-20260910`, `recovered-20260910`); the live profile had already been proven migrated
 (memories, `.env` and the six cron job IDs identical on the new box, which carries one more skill and two more messages).
 **Gap found and closed 2026-09-11 (anita-mnz):** `root-essentials-backup.sh` correctly excludes `*.db` so it never archives a
 live SQLite file, but nothing replaced it, so after the box was provisioned Anita's conversation store (974 MB), the cron
 execution DB, notepad, wisdom and verification DBs had **zero** backup coverage. Core has `hermes-backup.sh` doing this job;