From af80a8cb7b1064006fa37558b75512a381cfea94 Mon Sep 17 00:00:00 2001 From: Sho'Nuff Date: Fri, 11 Sep 2026 17:41:53 -0400 Subject: [PATCH] docs: close out Anita migration (archive verified, frozen copy deleted, stale S3 prefix purged, backup monitor findings) --- CHANGELOG.md | 4 +++- backup-plan.md | 8 ++++++++ 2 files changed, 11 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7d5f23d..6d88cd1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,7 +17,9 @@ - **Fix:** new `hermes-db-backup.sh` on `anita-mnz` (`/root/hermes-db-backup.sh`, chmod 750), cron `10 3 * * *`. Per-DB `sqlite3 .backup` (safe on a live DB), `PRAGMA quick_check` on every snapshot before it is accepted, one dated tarball, upload to `s3://hermes-vps-backups/root-backup/anita-mnz/db/`, then **downloads the object back, extracts and re-verifies**. 14-day retention. - **Verified:** first run 17:04 EDT, 291 MB uploaded; restore test passed — extracted, `quick_check=ok`, 99,285 messages. - **Migration completeness proven before deleting the Core copy:** `memories/MEMORY.md`, `memories/USER.md` and `.env` are md5-identical between the frozen Core copy and `anita-mnz`; `cron/jobs.json` holds the same six job IDs; skills 125 on MNZ vs 124 frozen; store 99,285 vs 99,283 messages. `config.yaml` differs only by the deliberate MCP strip. -- **Frozen Core copy removed:** `/root/.hermes/profiles/anita` (8.0 GB) was 7.9 GB of corrupt-DB corpses (`corrupt-20260903/09/10`, `pre-rebuild-20260910`, `recovered-20260910`). Archived with sha256 to `s3://hermes-vps-backups/decommissioned/`, size-verified, then deleted. Also removes that copy from Core's nightly archive, which is why Core's backup had grown to 2.97 GB. +- **Frozen Core copy removed (17:41):** `/root/.hermes/profiles/anita` (8.0 GB) was 7.9 GB of corrupt-DB corpses (`corrupt-20260903/09/10`, `pre-rebuild-20260910`, `recovered-20260910`). Archived to `s3://hermes-vps-backups/decommissioned/anita-core-frozen-profile-20260911-1732.tar.gz` (2,786,082,561 B, 21,206 entries), **verified by downloading the object back and matching sha256** `b457fa7f0cb6f546f36e53a338e90b7b940f51cf29d5a73d0cceb97f49c9b3dc` against the local tarball, then deleted. Nothing unique was destroyed: memories, `.env` and the six cron job IDs were identical on `anita-mnz`, which carries one more skill and two more messages. Core disk 125 GB → 117 GB used; `/root/.hermes/profiles` is now empty. Also removes that copy from Core's nightly archive, which is why Core's backup had grown to 2.97 GB. +- **Stale S3 copy purged (17:41):** `s3://hermes-vps-backups/live/profiles/anita/` held 20,551 objects / 5.0 GB that the then-running 15-minute sync had pushed before it was paused on Sep 3, including a stale-looking `state.db` that would have been advertised as "live". Superseded by the archive above and removed; `live/profiles/` is now empty. +- **Backup monitor checked (17:38):** its single CRITICAL was `hermes-live-sync: DISABLED/PAUSED`, which is the pause from Sep 3 and explains the monitor's 8-day exit-1 streak (it exits 1 only on CRITICAL, never on warnings). Its three WARNINGs are false positives, verified by content rather than size: Wazuh manager tarballs are sha256-identical for 3 days (static config, job ran 03:15 today), LiteLLM's flagged object is a 333 B config yaml that never changes (the real backup is a 37.9 MB `litellm-backup-*.tar.gz` from 03:30 today, and there are 47 of them), and the voipsimplicity dump is the same 6,834,636 B each day but a **different** sha256 each day (valid gzip, 73 tables). The size-uniqueness heuristic cannot tell static-but-fine from stalled; the three entries should be reclassified as "unchanged content" rather than SUSPICIOUS. - **Same-day correction on job status:** "Germaine inbox watch" is `enabled=false` on `anita-mnz`. It was `enabled=false` in the frozen Core copy too — the migration did not disable it. ## 2026-08-17 - Scirium v2 Proposal Deployed to /scirium/ diff --git a/backup-plan.md b/backup-plan.md index bd9c68b..80796ac 100644 --- a/backup-plan.md +++ b/backup-plan.md @@ -167,6 +167,7 @@ wphost02 was deleted from the Hetzner account on 2026-08-28. Its backup job has hermes-vps-backups/ ├── hermes-full-backup/ — Full Hermes daily (tar.gz) ├── live/ — Hermes live sync (every 15 min) +├── decommissioned/ — archived copies of decommissioned systems (profile archives) ├── live-sync/ — Old sync format (deprecated) ├── root-backup/ — /root essentials (Core uses the flat legacy prefix) │ └── / — per-host prefix (`anita-mnz`, ...) @@ -240,12 +241,19 @@ These paths contain data from services that migrated off Core (Jul 28, 2026) or | `core/komodo/` | Stale 11 days | **Safe to delete** — Komodo migrated to App1; new backups at `app1/komodo/` | | `caddy/` | Unused | **Safe to delete** — Never populated | | `snapshots/` | Unused | **Safe to delete** — Never populated | +| `live/profiles/anita/` | Purged 2026-09-11 | **Deleted** — 20,551 objects / 5.0 GB of the frozen Core copy of Anita's profile, pushed by the 15-minute sync before it was paused Sep 3. Superseded by `decommissioned/anita-core-frozen-profile-20260911-1732.tar.gz` | ## Unbacked Services These services are running in production with **zero backup coverage**: | Unbacked | *(none)* | N/A | N/A | All services are backed up as of 2026-08-08 | +**Done 2026-09-11 (Anita migration closeout).** The frozen Core copy `/root/.hermes/profiles/anita` (8.0 GB, excluded from live sync after the 15:53 cutover) was archived to +`s3://hermes-vps-backups/decommissioned/anita-core-frozen-profile-20260911-1732.tar.gz` — 2,786,082,561 B, sha256 `b457fa7f…b3dc`, 21,206 entries — then deleted, reclaiming 8 GB (125 GB → 117 GB used). +The archive was proven by downloading the object back and matching the sha256 against the local tarball **before** anything was removed. The 8 GB was almost entirely quarantine +artifacts from the Sep 3/9/10 SQLite corruption events (`corrupt-20260903/09/10`, `pre-rebuild-20260910`, `recovered-20260910`); the live profile had already been proven migrated +(memories, `.env` and the six cron job IDs identical on the new box, which carries one more skill and two more messages). + **Gap found and closed 2026-09-11 (anita-mnz):** `root-essentials-backup.sh` correctly excludes `*.db` so it never archives a live SQLite file, but nothing replaced it, so after the box was provisioned Anita's conversation store (974 MB), the cron execution DB, notepad, wisdom and verification DBs had **zero** backup coverage. Core has `hermes-backup.sh` doing this job;