# Incident: Core Hermes Store Corruption (not-a-database) and Recovery **Date:** September 11, 2026 **Severity:** High. Core's agent lost its conversation store mid-day, which disabled `session_search`, `delegate_task`, and every cron job that reads history. **Status:** Recovered. Repair staged and verified, installed the same evening. **Root cause:** The SQLite store was written while it was being checkpointed, so the file header was destroyed. The store stopped being a database at all. **Amplifier:** A 1.9 GB store that had never been pruned, failing three times in three days, with backups that faithfully captured the corruption because they tarred the live file. > Ground truth for this report was re-verified at 17:45 EDT on September 11, 2026. Numbers below are from the live box, not from a summary. --- ## 1. What happened At 12:49:22 EDT the default profile store `/root/.hermes/state.db` took its last write. Five seconds later the gateway restarted and found the store unreadable. Every tool that depends on history began failing with `file is not a database`: ``` session_search -> {"success": false, "error": "Session database not available: DatabaseError: file is not a database."} delegate_task -> Error executing tool: ... file is not a database ``` The file is 1,984,344,064 bytes. Its first bytes are **not** `SQLite format 3`, so SQLite rejects it before reading a single page. It is not a marginally damaged database, it is a file that no longer has a database header. ## 2. Impact - **Capability loss:** `session_search` and `delegate_task` were down for the rest of the day. Delegation is a load-bearing part of how this box operates. - **Cron surface:** any job reading history was at risk. The daily backup monitor kept failing for eight days, though that turned out to be a separate, already identified cause (section 6). - **Data loss window:** 11:49 to 12:49 EDT. No readable copy of that hour exists anywhere. Anything learned in that window is only recoverable from durable artifacts (DR issue log, CHANGELOG, skills). - **Backup lineage contamination:** the Sep 10 and Sep 11 archives captured the live, corrupt file. A restore from the newest archive would have restored the corruption. This is the single most dangerous detail in the incident. ## 3. Timeline (all times EDT) | When | Event | |---|---| | Sep 9 | WAL damage appears in the live store. First corruption event. | | Sep 10 12:18 | A backup is written that is already malformed. | | Sep 10 12:20 | A repair attempt fails. | | Sep 10 14:48 | A corrupt copy is preserved (`state.db.corrupt-20260910`). | | Sep 9 to 11 | Daily archives grow 1.76 GB to 2.34 GB to 3.52 GB. The growth is the unpruned store plus quarantine artifacts. | | Sep 11 01:00 | Nightly archive taken. This copy turns out to be clean: 1,939,980,288 B, `quick_check ok`, 107,588 messages, max id 321,545. | | Sep 11 11:49 | A snapshot is taken that is valid SQLite (1.96 GB) but has one bad page. It carries roughly 985 messages the 01:00 archive does not. | | Sep 11 12:49:22 | Last write to the live store. | | Sep 11 12:49:27 | Gateway restarts and cannot open the store. | | Sep 11 16:33 | Grafted candidate built and verified (`state.working.db`). | | Sep 11 16:51 | Install path hardened (`chmod +x`, explicit `/bin/bash`, dry run passed). | | Sep 11 18:56 | Repair scheduled to install via detached root crontab one shot. | ## 4. Root cause **Proximate cause:** a write that overlapped WAL checkpointing during the 12:49 shutdown. The store's header was overwritten, leaving an 1.85 GiB file with no valid database header. There is no evidence of filesystem damage; other files on the same volume are intact and the disk reports no errors. **What it was NOT, checked and excluded:** - **Not OOM.** The box had headroom. At the time of this report Core runs 15 GB total with 4 GB used and 11 GB available, 366 GB free disk. - **Not a cron job touching the database.** No scheduled job writes to `state.db` directly. The 15 minute `hermes-live-sync` job uses `.backup` snapshots and excludes live files, and it had been paused since Sep 3. - **Not a bad restore.** Nothing replaced the store on Sep 11 before the 12:49 failure. **What is still unknown, and stays unknown until proven:** which process requested the 12:49 gateway restart. The store was written and then the gateway came back and found it broken, and the caller was not recorded anywhere durable. That gap is itself a finding (section 7, item 5). **Why it was able to hurt this much:** the store is 1.9 GB and has never been pruned. Every checkpoint, backup, and recovery operation on a store that size is slow, IO heavy, and exposed to exactly this failure mode. Three corruption events in three days (Sep 9, 10, 11) is not bad luck, it is a store operating outside safe limits. The corrupt copy from Sep 3 through Sep 10 in Anita's frozen profile shows the same pattern on a second profile, which points at the host and the store size rather than at one profile's content. ## 5. Recovery **Strategy:** do not try to repair the corrupt file body. Build a new store from the newest clean copy and graft forward the messages that only exist in newer ones. 1. **Base:** the 01:00 archive store, verified clean: `quick_check ok`, 107,588 messages, max id 321,545. 2. **Graft:** 985 messages from the 11:49 snapshot (valid SQLite, one bad page) into the base. Grafting is row by row, so a single unreadable page costs a row, not the migration. 3. **Result:** `/root/db-forensics/state.working.db`, 1,961,385,984 B, verified at 17:45 EDT: ``` PRAGMA quick_check -> ok PRAGMA integrity_check -> ok 108,573 messages / max id 322,530 / 242 sessions FTS intact: messages_fts and messages_fts_trigram with all supporting tables ``` 985 grafted rows against 107,588 base rows reconciles exactly to 108,573. **Install path:** the job runs as a detached root crontab one shot writing to `/root/db-forensics/cron-invoke.log`, not `systemd-run` and not `/etc/cron.d` (both blocked by the lifecycle guard on this box). **First attempt failed for a boring reason worth recording:** `install.sh` was not executable, so cron forked it and it died within a second, writing nothing anywhere. The fix is three parts: `chmod +x`, invoke explicitly as `/bin/bash