Incident: Core Hermes Store Corruption (not-a-database) and Recovery

Date: September 11, 2026 Severity: High. Core's agent lost its conversation store mid-day, which disabled session_search, delegate_task, and every cron job that reads history. Status: Recovered. Repair staged and verified, installed the same evening. Root cause: The SQLite store was written while it was being checkpointed, so the file header was destroyed. The store stopped being a database at all. Amplifier: A 1.9 GB store that had never been pruned, failing three times in three days, with backups that faithfully captured the corruption because they tarred the live file.

Ground truth for this report was re-verified at 17:45 EDT on September 11, 2026. Numbers below are from the live box, not from a summary.

1. What happened

At 12:49:22 EDT the default profile store /root/.hermes/state.db took its last write. Five seconds later the gateway restarted and found the store unreadable. Every tool that depends on history began failing with file is not a database:

session_search -> {"success": false, "error": "Session database not available: DatabaseError: file is not a database."}
delegate_task  -> Error executing tool: ... file is not a database

The file is 1,984,344,064 bytes. Its first bytes are not SQLite format 3, so SQLite rejects it before reading a single page. It is not a marginally damaged database, it is a file that no longer has a database header.

2. Impact

Capability loss: session_search and delegate_task were down for the rest of the day. Delegation is a load-bearing part of how this box operates.
Cron surface: any job reading history was at risk. The daily backup monitor kept failing for eight days, though that turned out to be a separate, already identified cause (section 6).
Data loss window: 11:49 to 12:49 EDT. No readable copy of that hour exists anywhere. Anything learned in that window is only recoverable from durable artifacts (DR issue log, CHANGELOG, skills).
Backup lineage contamination: the Sep 10 and Sep 11 archives captured the live, corrupt file. A restore from the newest archive would have restored the corruption. This is the single most dangerous detail in the incident.

3. Timeline (all times EDT)

When	Event
Sep 9	WAL damage appears in the live store. First corruption event.
Sep 10 12:18	A backup is written that is already malformed.
Sep 10 12:20	A repair attempt fails.
Sep 10 14:48	A corrupt copy is preserved (`state.db.corrupt-20260910`).
Sep 9 to 11	Daily archives grow 1.76 GB to 2.34 GB to 3.52 GB. The growth is the unpruned store plus quarantine artifacts.
Sep 11 01:00	Nightly archive taken. This copy turns out to be clean: 1,939,980,288 B, `quick_check ok`, 107,588 messages, max id 321,545.
Sep 11 11:49	A snapshot is taken that is valid SQLite (1.96 GB) but has one bad page. It carries roughly 985 messages the 01:00 archive does not.
Sep 11 12:49:22	Last write to the live store.
Sep 11 12:49:27	Gateway restarts and cannot open the store.
Sep 11 16:33	Grafted candidate built and verified (`state.working.db`).
Sep 11 16:51	Install path hardened (`chmod +x`, explicit `/bin/bash`, dry run passed).
Sep 11 18:56	Repair scheduled to install via detached root crontab one shot.

4. Root cause

Proximate cause: a write that overlapped WAL checkpointing during the 12:49 shutdown. The store's header was overwritten, leaving an 1.85 GiB file with no valid database header. There is no evidence of filesystem damage; other files on the same volume are intact and the disk reports no errors.

What it was NOT, checked and excluded:

Not OOM. The box had headroom. At the time of this report Core runs 15 GB total with 4 GB used and 11 GB available, 366 GB free disk.
Not a cron job touching the database. No scheduled job writes to state.db directly. The 15 minute hermes-live-sync job uses .backup snapshots and excludes live files, and it had been paused since Sep 3.
Not a bad restore. Nothing replaced the store on Sep 11 before the 12:49 failure.

What is still unknown, and stays unknown until proven: which process requested the 12:49 gateway restart. The store was written and then the gateway came back and found it broken, and the caller was not recorded anywhere durable. That gap is itself a finding (section 7, item 5).

Why it was able to hurt this much: the store is 1.9 GB and has never been pruned. Every checkpoint, backup, and recovery operation on a store that size is slow, IO heavy, and exposed to exactly this failure mode. Three corruption events in three days (Sep 9, 10, 11) is not bad luck, it is a store operating outside safe limits. The corrupt copy from Sep 3 through Sep 10 in Anita's frozen profile shows the same pattern on a second profile, which points at the host and the store size rather than at one profile's content.

5. Recovery

Strategy: do not try to repair the corrupt file body. Build a new store from the newest clean copy and graft forward the messages that only exist in newer ones.

Base: the 01:00 archive store, verified clean: quick_check ok, 107,588 messages, max id 321,545.
Graft: 985 messages from the 11:49 snapshot (valid SQLite, one bad page) into the base. Grafting is row by row, so a single unreadable page costs a row, not the migration.
Result: /root/db-forensics/state.working.db, 1,961,385,984 B, verified at 17:45 EDT:

PRAGMA quick_check     -> ok
PRAGMA integrity_check -> ok
108,573 messages / max id 322,530 / 242 sessions
FTS intact: messages_fts and messages_fts_trigram with all supporting tables

985 grafted rows against 107,588 base rows reconciles exactly to 108,573.

Install path: the job runs as a detached root crontab one shot writing to /root/db-forensics/cron-invoke.log, not systemd-run and not /etc/cron.d (both blocked by the lifecycle guard on this box).

First attempt failed for a boring reason worth recording: install.sh was not executable, so cron forked it and it died within a second, writing nothing anywhere. The fix is three parts: chmod +x, invoke explicitly as /bin/bash <script>, and log stdout and stderr to a file that can be read afterwards. It was then exercised in --dry-run before being armed for real.

6. Side finding: the backup monitor was crying wolf for eight days

Backup-Health-Monitor (daily 09:00) had failure_streak=8 and last_status=error.

The one real CRITICAL was hermes-live-sync: DISABLED/PAUSED (job 61cd31eec51c, paused 2026-09-03 15:49, the same window as the first corruption event). The script exits 1 only when total_critical > 0, never for warnings, so that single pause accounts for the entire streak.
Its three WARNINGs are false positives, proven by content rather than by size:
- Wazuh Manager (app1): the tarballs for Sep 9, 10 and 11 are all sha256 0e65a2364cdc0795..., identical. The content is a static config, and the 03:15 job did run today. Not stalled.
- LiteLLM Config (app1): the flagged object is a 333 byte config yaml that legitimately never changes. The actual data backup is app1/litellm/litellm-backup-*.tar.gz, 47 of them, newest 37,894,223 bytes at 03:30 today, and it carries the Postgres dump.
- MySQL voipsimplicity (app3): identical size 6,834,636 bytes each day but a different sha256 each day (cb1ce9f8, 92c5eb70, b83ea28f), valid gzip, 73 tables, real mysqldump 8.4.10. Content is changing, the size simply coincides.
Conclusion: the size uniqueness heuristic cannot tell "static but fine" from "stalled". Those three checks should be reclassified as "unchanged content" rather than SUSPICIOUS. Not yet changed.

7. Prevention

Prune the store. 1.9 GB unpruned is the amplifier behind all three failures. Needs a retention policy and a size cap, sized before anything is deleted (session count, cron and watcher noise measured first).
Snapshot, never tar, live databases. Excluding *.db from the essentials archive is correct. What was missing was the replacement, now supplied by hermes-db-backup.sh, which takes a sqlite3 .backup per database, quick_check each snapshot, uploads, then downloads and re-verifies. Deployed on the Anita box with cron 10 3 * * *, proven by round trip (291 MB snapshot, quick_check ok, 99,285 messages).
Never trust a backup that has not been restored. Every archive is now suspect until an object has been downloaded, extracted, and opened. The Sep 10 and Sep 11 archives would both have restored the corruption.
Check integrity on the newest snapshot daily, not just its existence and size. PRAGMA quick_check on the newest object is cheap and would have caught the Sep 10 and Sep 11 contamination on the day it happened.
Record who restarts the gateway. The 12:49 caller is still unknown. The restart path should write caller identity, reason, and timestamp to a durable log so this question is answerable next time.
Resume the 15 minute checkpoint only after the repaired store verifies healthy, and prove the resumed job by downloading its object and running quick_check on it.
Keep quarantine artifacts for a retention window. The only unique bytes in the deleted frozen profile were the Sep 3, 9 and 10 quarantine files. They are archived, not discarded, so a future investigator can compare failure signatures.

8. Evidence (re-verified 2026-09-11 17:45 EDT)

live      /root/.hermes/state.db                      1,984,344,064 B   mtime 2026-09-11 12:49:22   header: NOT SQLite
donor     /root/sqlite_tmp/sep11/hermes-backup-2026-09-11/state.db
                                                      1,939,980,288 B   quick_check ok  107,588 msgs  max id 321,545
candidate /root/db-forensics/state.working.db         1,961,385,984 B   quick_check ok  integrity_check ok
                                                                       108,573 msgs  max id 322,530  242 sessions
disk      503 GB volume, 117 GB used, 366 GB free
memory    15 GB total, 4 GB used, 11 GB available

9. Lesson

The failure was survivable because the 01:00 archive happened to be clean and the 11:49 snapshot happened to be readable enough to graft from. That is luck, not architecture. Two things made it luck: backups that copied the live store, and no pruning on a store big enough that every operation on it is a window of risk. Both are fixed or scheduled. The remaining exposure is the same store size, which stays live until a retention policy is agreed.

10 KiB Raw Permalink Blame History