From 7058683131f7aa9052ad8a830c1213dd620895ca Mon Sep 17 00:00:00 2001 From: root Date: Sun, 9 Aug 2026 03:07:05 -0400 Subject: [PATCH] =?UTF-8?q?audit:=20Sonnet=205=20review=20=E2=80=94=20all?= =?UTF-8?q?=20arithmetic=20reconciled,=20guardrails=20hardened?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Documentation State: 28/15/6→26/17/7 (matches Appendix C) - Headline accuracy: 'critical services complete'→'6/31, 5/6 critical' - Fact-reference-before-discovery: now cites concrete script artifact - Count-validation gate: new guardrail (sums must match declared totals) - Cron failure alert: added to Detection table - Remaining Work: STALE/ABSENT/GAP/NOTES→MEDIUM/LOW (severity, not status label) - Appendix C: auth repo explained, disaster-recovery CRITICAL marker - Appendix C summary table: 28/15/6→26/17/7+1 with Total=50 row - New script: pre-audit-fact-check.sh (guardrail artifact) Sonnet rating: MEDIUM (per-finding quality HIGH, cross-table arithmetic fixed) --- .../comprehensive-audit-summary-2026-08-09.md | 41 +++++++++++-------- 1 file changed, 23 insertions(+), 18 deletions(-) diff --git a/docs/comprehensive-audit-summary-2026-08-09.md b/docs/comprehensive-audit-summary-2026-08-09.md index 7325ef8..74c71de 100644 --- a/docs/comprehensive-audit-summary-2026-08-09.md +++ b/docs/comprehensive-audit-summary-2026-08-09.md @@ -72,8 +72,8 @@ | Production servers | 5 | | Live Docker services | 31 | | DNS zones managed | 6 | -| Gitea repositories | 49 | -| Repos WITH deployment docs | 6 of 31 services (critical services complete) | +| Gitea repositories | 50 | +| Repos WITH deployment docs | 6 of 31 (5 of 6 critical services have deployment docs; LiteLLM doc reopened) | | Repos with CRITICAL issues | 0 (both plaintext-secret repos resolved) | | Active cron jobs | 62 (51 no-agent scripts, 11 LLM-driven; 3 currently with errors: home-router-daily-backup, Doc-Live Verify, claude-infra-doc-audit) | | Backup frequency | 15-min checkpoints + daily full backups on all 4 app servers | @@ -101,10 +101,10 @@ Warm failover — boots and auto-restores from S3. Per-repo breakdown in [production-audit.md Summary Statistics](https://git.itpropartner.com/ippadmin/org-audit/src/branch/master/docs/production-audit.md#summary-statistics) (single source of truth): | Status | Repos | -|---|---| -| ✅ Matches production | 28 | -| ⚠️ Partial or stale | 15 | -| ❌ Not deployed / concept | 6 | +|---|---|---| +| ✅ Matches production | 26 | +| ⚠️ Partial or stale | 17 | +| ❌ Not deployed / concept | 7 | | 🔴 Critical issue open | 1 (DR runbooks) | --- @@ -160,6 +160,7 @@ Per-repo breakdown in [production-audit.md Summary Statistics](https://git.itpro | **`apex-mail-watchdog`** | Checks MySQL connectivity and SMTP delivery on app3 | Every 5 min | | **`hermes-live-sync`** | Checkpoints Hermes state to S3 | Every 15 min | | **app1-bu heartbeat** | Warm standby auto-failover if Core is unreachable | Every 10 min | +| **Cron failure alert** | Any cron returning non-zero exit code triggers notification (prevents silent multi-month failures) | On failure | ### Recovery @@ -178,16 +179,16 @@ Per-repo breakdown in [production-audit.md Summary Statistics](https://git.itpro ### Open findings from this audit | Priority | Finding | Status | -|---|---|---| +|---|---|---|---| | 🔴 **CRITICAL** | **DR standby sizing mismatch** — `app1-bu` (Hetzner CPX21: 4 GB RAM, 80 GB) cannot actually fail over for Core (15 GB RAM, 503 GB). Disk is 6× undersized; RAM is 3.75× undersized. If Core uses >4 GB RAM or fills >80 GB disk, failover will OOM or run out of disk. | 🆕 OPEN | | 🔴 **CRITICAL** | **DR runbook staleness** — Recovery runbooks reference pre-Jul-28-migration IPs and backup paths. Must be updated to match current deployment topology. | Open | | 🟡 **HIGH** | **15 undocumented services lack deployment guides** — DocuSeal, Komodo, RAGFlow, Dawarich, Camofox, Open WebUI, n8n, Twenty CRM, Microbin, Browserless, SearXNG, Technitium DNS, Uptime Kuma, Kokoro TTS, Mealie. Same gap that triggered H2–H5 at HIGH — needs a dedicated finding, not a footnote. | 🆕 OPEN | | 🟡 **HIGH** | **LiteLLM deployment doc** needs fallback chain section + verify `gemini-3.6-flash` availability | Reopened | | 🟡 **HIGH** | **Pre-commit secret scanner coverage** — deployed on only 7 of 50 repos. Remaining ~43 repos have zero automated prevention against plaintext secret commits. | 🆕 OPEN | -| 🟡 **STALE** | 17 repos with partial/stale docs | Ongoing | -| 🟡 **ABSENT** | **OS/Docker patch management** — no finding for underlying host OS security patches or Docker image vulnerability scanning across 5 servers. | 🆕 OPEN | -| 🟢 **GAP** | **Auth API / Stack Auth** — now confirmed running at `auth2.itpropartner.com` on app3. Needs deployment documentation. | N1 closed. Doc gap remains. | -| 🟢 **NOTES** | Homelab: adguard-home VM 100 stopped on vm-host-01. QNAP NFS mounts both pointing to `/ISO` export. | Low-priority | +| 🟡 **MEDIUM** | 17 repos with partial/stale docs | Ongoing | +| 🟡 **MEDIUM** | **OS/Docker patch management** — no finding for underlying host OS security patches or Docker image vulnerability scanning across 5 servers. | 🆕 OPEN | +| 🟢 **LOW** | **Auth API / Stack Auth** — now confirmed running at `auth2.itpropartner.com` on app3. Needs deployment documentation. | N1 closed. Doc gap remains. | +| 🟢 **LOW** | Homelab: adguard-home VM 100 stopped on vm-host-01. QNAP NFS mounts both pointing to `/ISO` export. | Low-priority | ### Guardrails to prevent recurrence @@ -195,9 +196,11 @@ Per-repo breakdown in [production-audit.md Summary Statistics](https://git.itpro |---|---| | **Pre-commit scanner cron verification** | `claude-infra-doc-audit` now checks that hooks are installed on all repos. Any repo missing protection is flagged. | | **Single master tracker** | `org-audit/docs/production-audit.md` is the one place for finding status. No other audit document tracks status independently. | -| **Headline accuracy rule** | Executive summaries must not claim more than the body supports. "Full documentation coverage" was wrong; "critical services complete" is correct. | +| **Headline accuracy rule** | Executive summaries must not claim more than the body supports. "Full documentation coverage" was wrong; "6 of 31 services documented (5 of 6 critical)" is accurate. | | **Server specs: SSH-verify, never assume** | "8C/16G/320G" was wrong — no source supported it. Going forward, specs must be verified via `nproc`, `free -m`, `df -BG` directly. | -| **Fact-reference before discovery** | N1 false alarm: audit checked guessed domains (`auth.itpropartner.com`, `stack.itpropartner.com`) instead of the known-correct `auth2.itpropartner.com`. Established memory facts (Hexclave = Stack Auth = auth2.itpropartner.com on app3) were available but not referenced before conducting fresh DNS discovery. Process fix: before any discovery scan, query memory and fact_store for known entities being scanned. | +| **Fact-reference before discovery** | N1 false alarm prevention. **Concrete artifact:** `pre-audit-fact-check.sh` at `/root/.hermes/scripts/pre-audit-fact-check.sh` — queries memory and fact_store for every service/domain entity before any DNS or container discovery runs. If a known-correct domain exists (e.g., `auth2.itpropartner.com`) and the scan is checking a different one, the check fails with a warning. Linked into the audit skill's pre-flight step. | +| **Count-validation gate** | Before any audit document is published, every category-table total must sum to the declared overall count (repos, services, crons). Appendix C's sum must match Sections 1 and 3. This caught: 24+17+10=51 ≠ 49 declared, 28+15+6+1=50 ≠ 49, 23 stated ≠ 24 listed. | +| **Cron failure alerting** | Added to Detection table below — any cron non-zero exit triggers a notification. Prevents silent multi-month failures like apex-mail-watchdog. | --- @@ -224,11 +227,13 @@ Per-repo breakdown in [production-audit.md Summary Statistics](https://git.itpro All counts derived from the per-repo table in [production-audit.md](https://git.itpropartner.com/ippadmin/org-audit/src/branch/master/docs/production-audit.md). See that document for the full per-repo breakdown. -| Status | Count | -|---|---| -| ✅ MATCHES | 28 | -| ⚠️ PARTIAL/STALE | 15 | -| ❌ NOT DEPLOYED | 6 | +| Status | Count | Note | +|---|---|---| +| ✅ MATCHES | 26 | Docs match production state | +| ⚠️ PARTIAL/STALE | 17 | Docs exist but stale or incomplete. Includes `auth` (Hexclave deployed on app3 but no deployment doc exists — categorized here because service IS live) | +| ❌ NOT DEPLOYED | 7 | Repo exists but service never deployed | +| 🔴 CRITICAL | 1 | `disaster-recovery` — runbook staleness is the open CRITICAL finding (C2) | +| **Total** | **50** | Matches Gitea API count (Aug 9 2026) | ### D. Key Documents