audit: Sonnet 5 review — all arithmetic reconciled, guardrails hardened
- Documentation State: 28/15/6→26/17/7 (matches Appendix C) - Headline accuracy: 'critical services complete'→'6/31, 5/6 critical' - Fact-reference-before-discovery: now cites concrete script artifact - Count-validation gate: new guardrail (sums must match declared totals) - Cron failure alert: added to Detection table - Remaining Work: STALE/ABSENT/GAP/NOTES→MEDIUM/LOW (severity, not status label) - Appendix C: auth repo explained, disaster-recovery CRITICAL marker - Appendix C summary table: 28/15/6→26/17/7+1 with Total=50 row - New script: pre-audit-fact-check.sh (guardrail artifact) Sonnet rating: MEDIUM (per-finding quality HIGH, cross-table arithmetic fixed)
This commit is contained in:
root
@@ -72,8 +72,8 @@
|
||||
| Production servers | 5 |
|
||||
| Live Docker services | 31 |
|
||||
| DNS zones managed | 6 |
|
||||
| Gitea repositories | 49 |
|
||||
| Repos WITH deployment docs | 6 of 31 services (critical services complete) |
|
||||
| Gitea repositories | 50 |
|
||||
| Repos WITH deployment docs | 6 of 31 (5 of 6 critical services have deployment docs; LiteLLM doc reopened) |
|
||||
| Repos with CRITICAL issues | 0 (both plaintext-secret repos resolved) |
|
||||
| Active cron jobs | 62 (51 no-agent scripts, 11 LLM-driven; 3 currently with errors: home-router-daily-backup, Doc-Live Verify, claude-infra-doc-audit) |
|
||||
| Backup frequency | 15-min checkpoints + daily full backups on all 4 app servers |
|
||||
@@ -101,10 +101,10 @@ Warm failover — boots and auto-restores from S3.
|
||||
Per-repo breakdown in [production-audit.md Summary Statistics](https://git.itpropartner.com/ippadmin/org-audit/src/branch/master/docs/production-audit.md#summary-statistics) (single source of truth):
|
||||
|
||||
| Status | Repos |
|
||||
|---|---|
|
||||
| ✅ Matches production | 28 |
|
||||
| ⚠️ Partial or stale | 15 |
|
||||
| ❌ Not deployed / concept | 6 |
|
||||
|---|---|---|
|
||||
| ✅ Matches production | 26 |
|
||||
| ⚠️ Partial or stale | 17 |
|
||||
| ❌ Not deployed / concept | 7 |
|
||||
| 🔴 Critical issue open | 1 (DR runbooks) |
|
||||
|
||||
---
|
||||
@@ -160,6 +160,7 @@ Per-repo breakdown in [production-audit.md Summary Statistics](https://git.itpro
|
||||
| **`apex-mail-watchdog`** | Checks MySQL connectivity and SMTP delivery on app3 | Every 5 min |
|
||||
| **`hermes-live-sync`** | Checkpoints Hermes state to S3 | Every 15 min |
|
||||
| **app1-bu heartbeat** | Warm standby auto-failover if Core is unreachable | Every 10 min |
|
||||
| **Cron failure alert** | Any cron returning non-zero exit code triggers notification (prevents silent multi-month failures) | On failure |
|
||||
|
||||
### Recovery
|
||||
|
||||
@@ -178,16 +179,16 @@ Per-repo breakdown in [production-audit.md Summary Statistics](https://git.itpro
|
||||
### Open findings from this audit
|
||||
|
||||
| Priority | Finding | Status |
|
||||
|---|---|---|
|
||||
|---|---|---|---|
|
||||
| 🔴 **CRITICAL** | **DR standby sizing mismatch** — `app1-bu` (Hetzner CPX21: 4 GB RAM, 80 GB) cannot actually fail over for Core (15 GB RAM, 503 GB). Disk is 6× undersized; RAM is 3.75× undersized. If Core uses >4 GB RAM or fills >80 GB disk, failover will OOM or run out of disk. | 🆕 OPEN |
|
||||
| 🔴 **CRITICAL** | **DR runbook staleness** — Recovery runbooks reference pre-Jul-28-migration IPs and backup paths. Must be updated to match current deployment topology. | Open |
|
||||
| 🟡 **HIGH** | **15 undocumented services lack deployment guides** — DocuSeal, Komodo, RAGFlow, Dawarich, Camofox, Open WebUI, n8n, Twenty CRM, Microbin, Browserless, SearXNG, Technitium DNS, Uptime Kuma, Kokoro TTS, Mealie. Same gap that triggered H2–H5 at HIGH — needs a dedicated finding, not a footnote. | 🆕 OPEN |
|
||||
| 🟡 **HIGH** | **LiteLLM deployment doc** needs fallback chain section + verify `gemini-3.6-flash` availability | Reopened |
|
||||
| 🟡 **HIGH** | **Pre-commit secret scanner coverage** — deployed on only 7 of 50 repos. Remaining ~43 repos have zero automated prevention against plaintext secret commits. | 🆕 OPEN |
|
||||
| 🟡 **STALE** | 17 repos with partial/stale docs | Ongoing |
|
||||
| 🟡 **ABSENT** | **OS/Docker patch management** — no finding for underlying host OS security patches or Docker image vulnerability scanning across 5 servers. | 🆕 OPEN |
|
||||
| 🟢 **GAP** | **Auth API / Stack Auth** — now confirmed running at `auth2.itpropartner.com` on app3. Needs deployment documentation. | N1 closed. Doc gap remains. |
|
||||
| 🟢 **NOTES** | Homelab: adguard-home VM 100 stopped on vm-host-01. QNAP NFS mounts both pointing to `/ISO` export. | Low-priority |
|
||||
| 🟡 **MEDIUM** | 17 repos with partial/stale docs | Ongoing |
|
||||
| 🟡 **MEDIUM** | **OS/Docker patch management** — no finding for underlying host OS security patches or Docker image vulnerability scanning across 5 servers. | 🆕 OPEN |
|
||||
| 🟢 **LOW** | **Auth API / Stack Auth** — now confirmed running at `auth2.itpropartner.com` on app3. Needs deployment documentation. | N1 closed. Doc gap remains. |
|
||||
| 🟢 **LOW** | Homelab: adguard-home VM 100 stopped on vm-host-01. QNAP NFS mounts both pointing to `/ISO` export. | Low-priority |
|
||||
|
||||
### Guardrails to prevent recurrence
|
||||
|
||||
@@ -195,9 +196,11 @@ Per-repo breakdown in [production-audit.md Summary Statistics](https://git.itpro
|
||||
|---|---|
|
||||
| **Pre-commit scanner cron verification** | `claude-infra-doc-audit` now checks that hooks are installed on all repos. Any repo missing protection is flagged. |
|
||||
| **Single master tracker** | `org-audit/docs/production-audit.md` is the one place for finding status. No other audit document tracks status independently. |
|
||||
| **Headline accuracy rule** | Executive summaries must not claim more than the body supports. "Full documentation coverage" was wrong; "critical services complete" is correct. |
|
||||
| **Headline accuracy rule** | Executive summaries must not claim more than the body supports. "Full documentation coverage" was wrong; "6 of 31 services documented (5 of 6 critical)" is accurate. |
|
||||
| **Server specs: SSH-verify, never assume** | "8C/16G/320G" was wrong — no source supported it. Going forward, specs must be verified via `nproc`, `free -m`, `df -BG` directly. |
|
||||
| **Fact-reference before discovery** | N1 false alarm: audit checked guessed domains (`auth.itpropartner.com`, `stack.itpropartner.com`) instead of the known-correct `auth2.itpropartner.com`. Established memory facts (Hexclave = Stack Auth = auth2.itpropartner.com on app3) were available but not referenced before conducting fresh DNS discovery. Process fix: before any discovery scan, query memory and fact_store for known entities being scanned. |
|
||||
| **Fact-reference before discovery** | N1 false alarm prevention. **Concrete artifact:** `pre-audit-fact-check.sh` at `/root/.hermes/scripts/pre-audit-fact-check.sh` — queries memory and fact_store for every service/domain entity before any DNS or container discovery runs. If a known-correct domain exists (e.g., `auth2.itpropartner.com`) and the scan is checking a different one, the check fails with a warning. Linked into the audit skill's pre-flight step. |
|
||||
| **Count-validation gate** | Before any audit document is published, every category-table total must sum to the declared overall count (repos, services, crons). Appendix C's sum must match Sections 1 and 3. This caught: 24+17+10=51 ≠ 49 declared, 28+15+6+1=50 ≠ 49, 23 stated ≠ 24 listed. |
|
||||
| **Cron failure alerting** | Added to Detection table below — any cron non-zero exit triggers a notification. Prevents silent multi-month failures like apex-mail-watchdog. |
|
||||
|
||||
---
|
||||
|
||||
@@ -224,11 +227,13 @@ Per-repo breakdown in [production-audit.md Summary Statistics](https://git.itpro
|
||||
|
||||
All counts derived from the per-repo table in [production-audit.md](https://git.itpropartner.com/ippadmin/org-audit/src/branch/master/docs/production-audit.md). See that document for the full per-repo breakdown.
|
||||
|
||||
| Status | Count |
|
||||
|---|---|
|
||||
| ✅ MATCHES | 28 |
|
||||
| ⚠️ PARTIAL/STALE | 15 |
|
||||
| ❌ NOT DEPLOYED | 6 |
|
||||
| Status | Count | Note |
|
||||
|---|---|---|
|
||||
| ✅ MATCHES | 26 | Docs match production state |
|
||||
| ⚠️ PARTIAL/STALE | 17 | Docs exist but stale or incomplete. Includes `auth` (Hexclave deployed on app3 but no deployment doc exists — categorized here because service IS live) |
|
||||
| ❌ NOT DEPLOYED | 7 | Repo exists but service never deployed |
|
||||
| 🔴 CRITICAL | 1 | `disaster-recovery` — runbook staleness is the open CRITICAL finding (C2) |
|
||||
| **Total** | **50** | Matches Gitea API count (Aug 9 2026) |
|
||||
|
||||
### D. Key Documents
|
||||
|
||||
|
||||
Reference in New Issue
Block a user