diff --git a/docs/comprehensive-audit-summary-2026-08-09.md b/docs/comprehensive-audit-summary-2026-08-09.md index 2137221..7325ef8 100644 --- a/docs/comprehensive-audit-summary-2026-08-09.md +++ b/docs/comprehensive-audit-summary-2026-08-09.md @@ -12,7 +12,7 @@ ## 1. Audit Scope **Date:** August 9, 2026 -**Coverage:** 49 Gitea repositories, 5 production servers, 24 live services, 6 DNS zones +**Coverage:** 50 Gitea repositories, 5 production servers, 31 live services, 6 DNS zones **Methodology:** - Cross-referenced every repository's documentation against live production state via SSH @@ -70,12 +70,12 @@ | Metric | Count | |---|---| | Production servers | 5 | -| Live Docker services | 24 | +| Live Docker services | 31 | | DNS zones managed | 6 | | Gitea repositories | 49 | -| Repos WITH deployment docs | 6 of 24 services (critical services complete) | +| Repos WITH deployment docs | 6 of 31 services (critical services complete) | | Repos with CRITICAL issues | 0 (both plaintext-secret repos resolved) | -| Active cron jobs | 62 (59 no-agent scripts, 3 LLM-driven — 3 currently with errors) | +| Active cron jobs | 62 (51 no-agent scripts, 11 LLM-driven; 3 currently with errors: home-router-daily-backup, Doc-Live Verify, claude-infra-doc-audit) | | Backup frequency | 15-min checkpoints + daily full backups on all 4 app servers | | Pre-commit secret scanner | Deployed on 7 repos | @@ -98,11 +98,13 @@ Warm failover — boots and auto-restores from S3. ### Documentation State +Per-repo breakdown in [production-audit.md Summary Statistics](https://git.itpropartner.com/ippadmin/org-audit/src/branch/master/docs/production-audit.md#summary-statistics) (single source of truth): + | Status | Repos | |---|---| -| ✅ Matches production | 20 | -| ⚠️ Partial or stale | 14 | -| ❌ Not deployed / concept | 10 | +| ✅ Matches production | 28 | +| ⚠️ Partial or stale | 15 | +| ❌ Not deployed / concept | 6 | | 🔴 Critical issue open | 1 (DR runbooks) | --- @@ -177,10 +179,13 @@ Warm failover — boots and auto-restores from S3. | Priority | Finding | Status | |---|---|---| +| 🔴 **CRITICAL** | **DR standby sizing mismatch** — `app1-bu` (Hetzner CPX21: 4 GB RAM, 80 GB) cannot actually fail over for Core (15 GB RAM, 503 GB). Disk is 6× undersized; RAM is 3.75× undersized. If Core uses >4 GB RAM or fills >80 GB disk, failover will OOM or run out of disk. | 🆕 OPEN | | 🔴 **CRITICAL** | **DR runbook staleness** — Recovery runbooks reference pre-Jul-28-migration IPs and backup paths. Must be updated to match current deployment topology. | Open | +| 🟡 **HIGH** | **15 undocumented services lack deployment guides** — DocuSeal, Komodo, RAGFlow, Dawarich, Camofox, Open WebUI, n8n, Twenty CRM, Microbin, Browserless, SearXNG, Technitium DNS, Uptime Kuma, Kokoro TTS, Mealie. Same gap that triggered H2–H5 at HIGH — needs a dedicated finding, not a footnote. | 🆕 OPEN | | 🟡 **HIGH** | **LiteLLM deployment doc** needs fallback chain section + verify `gemini-3.6-flash` availability | Reopened | -| 🟡 **STALE** | 14 repos with partial/stale docs | Ongoing | -| 🟡 **UNDOCUMENTED** | 15 production services lack deployment guides (DocuSeal, Komodo, RAGFlow, Dawarich, Camofox, Open WebUI, n8n, etc.) | Not started | +| 🟡 **HIGH** | **Pre-commit secret scanner coverage** — deployed on only 7 of 50 repos. Remaining ~43 repos have zero automated prevention against plaintext secret commits. | 🆕 OPEN | +| 🟡 **STALE** | 17 repos with partial/stale docs | Ongoing | +| 🟡 **ABSENT** | **OS/Docker patch management** — no finding for underlying host OS security patches or Docker image vulnerability scanning across 5 servers. | 🆕 OPEN | | 🟢 **GAP** | **Auth API / Stack Auth** — now confirmed running at `auth2.itpropartner.com` on app3. Needs deployment documentation. | N1 closed. Doc gap remains. | | 🟢 **NOTES** | Homelab: adguard-home VM 100 stopped on vm-host-01. QNAP NFS mounts both pointing to `/ISO` export. | Low-priority | @@ -217,11 +222,13 @@ Warm failover — boots and auto-restores from S3. ### C. Cross-Reference: Every Repo vs Production -| Status | Count | Repos | -|---|---|---| -| ✅ MATCHES | 23 | itpp-infrastructure, ops-portal, ops-reports, backup-restore, hudu, unifi, unms, hermes-skills, hermes-recovery, mcp-servers, mcp-browser, mcp-email, mcp-filesystem, mcp-git, super-search, shonuff-caller, nvr-shield, dre, fleettracker360, voipsimplicity, voipsimplicity-manual, shark-game, gift-a-roast, apextrack | -| ⚠️ PARTIAL/STALE | 17 | disaster-recovery, scripts, model-fallback, homelab, auth, pipeline, research-search-mcp, voice-agent, osint-tool, content-creation-pipeline, pry, itpropartner-website, forefront-wireless-portal, cartmylist, mooresunnydaze, msp-forms, boxpilot | -| ❌ NOT DEPLOYED | 10 | super-search-business, launchcheck, verdicttank, track-a-flock, transitpin, digital-signage, personal-assistant, hotnow.io, IntelSight, SchoolCart | +All counts derived from the per-repo table in [production-audit.md](https://git.itpropartner.com/ippadmin/org-audit/src/branch/master/docs/production-audit.md). See that document for the full per-repo breakdown. + +| Status | Count | +|---|---| +| ✅ MATCHES | 28 | +| ⚠️ PARTIAL/STALE | 15 | +| ❌ NOT DEPLOYED | 6 | ### D. Key Documents