audit: N1 resolved (auth2 on app3), Core storage 503G, Appendix C counts fixed, model info added

docs/architecture.md

@@ -10,7 +10,7 @@
 
 | Server | IP | Specs | Provider | Role |
 |---|---|---|---|---|
-| **Core** | 152.53.192.33 | RS 2000 G9.5 (8 vCPU EPYC 9645, 15 GB RAM, 256 GB SSD) | netcup | Hermes agent host, monitoring, Caddy reverse proxy (26 sites) |
+| **Core** | 152.53.192.33 | RS 2000 G9.5 (8 vCPU EPYC 9645, 15 GB RAM, 503 GB SSD) | netcup | Hermes agent host, monitoring, Caddy reverse proxy (26 sites) |
 | **app1** | 152.53.36.131 | RS 4000 G9.5 (12 vCPU EPYC, 32 GB RAM, 1 TB SSD) | netcup | Service hub — AI gateway, CRM, signing, TTS, auth, automation |
 | **app2** | 152.53.39.202 | RS 4000 G9.5 (12 vCPU EPYC, 32 GB RAM, 1 TB SSD) | netcup | Infrastructure — Gitea, Hudu, Ubiquiti controllers, Traccar, DNS, SIEM |
 | **app3** | 152.53.241.111 | RS 4000 G9.5 (12 vCPU EPYC, 32 GB RAM, 1 TB SSD) | netcup | Web hosting — CloudPanel CE (WordPress/static/PHP), client sites |

docs/comprehensive-audit-summary-2026-08-09.md

@@ -3,6 +3,7 @@
 
 **Prepared for:** External Review
 **Auditor:** Sho'Nuff (Hermes Agent)
+**Model:** DeepSeek V4 Pro via admin-ai.itpropartner.com (LiteLLM gateway) — full audit, issue resolution, and follow-up task orchestration
 **Master tracker:** [org-audit/docs/production-audit.md](https://git.itpropartner.com/ippadmin/org-audit/src/branch/master/docs/production-audit.md) (single source of truth)
 **Narrative companion:** [itpp-infrastructure/docs/post-audit-report-2026-08-09.md](https://git.itpropartner.com/ippadmin/itpp-infrastructure/src/branch/main/docs/post-audit-report-2026-08-09.md)
 
@@ -24,7 +25,7 @@
 
 | Server | IP | Specs (SSH-verified) | Provider |
 |---|---|---|---|
-| Core | 152.53.192.33 | 8 vCPU EPYC 9645, 15 GB RAM, 256 GB | netcup RS 2000 |
+| Core | 152.53.192.33 | 8 vCPU EPYC 9645, 15 GB RAM, 503 GB | netcup RS 2000 |
 | app1 | 152.53.36.131 | 12 vCPU EPYC, 32 GB RAM, 1 TB | netcup RS 4000 |
 | app2 | 152.53.39.202 | 12 vCPU EPYC, 32 GB RAM, 1 TB | netcup RS 4000 |
 | app3 | 152.53.241.111 | 12 vCPU EPYC, 32 GB RAM, 1 TB | netcup RS 4000 |
@@ -57,7 +58,7 @@
 |---|---|---|---|
 | R1 | **fleettracker360.com DNS** | Flagged as broken DNS | ✅ **RESOLVED** — false positive. Cloudflare orange-cloud proxy IPs (188.114.x.x) are expected. Site returns HTTP/2 200 through proxy. |
 | R2 | **itpp-infrastructure stale docs** | `master-apps-services.md` listed defunct servers | ✅ **RESOLVED** — file deleted. `architecture.md` is now authoritative, updated with verified specs. |
-| N1 | **Auth API / Stack Auth gap** | Not in audit scope | 🆕 **OPEN** — `auth.itpropartner.com` resolves to Core but nothing listens. No Docker containers for auth/hexclave/stack-auth anywhere. DNS for `stack.itpropartner.com` and `hexclave.itpropartner.com` doesn't exist. `auth.iamgmb.com` decommissioned. **Not deployed.** |
+| N1 | **Auth API / Stack Auth / Hexclave** | Not in audit scope, flagged as missing | ✅ **RESOLVED (false alarm, closed 2026-08-09).** `auth2.itpropartner.com` (app3) is live. Hexclave (formerly Stack Auth) Docker containers confirmed: `hexclave-server`, `hexclave-cron`, `hexclave-postgres`, `hexclave-clickhouse`. Daily backups at 3:15 AM and 3:30 AM. **Root cause:** Audit checked wrong domains (`auth.itpropartner.com`, `stack.itpropartner.com`) instead of the known-correct `auth2.itpropartner.com`. Container search was scoped to app1 only, missing app3. Process gap: established facts weren't referenced before fresh discovery scans. |
 | N2 | **Gitea deployment docs** | No deployment doc existed | ✅ **DOCUMENTED** (565 lines). The service hosting all documentation is now itself documented. |
 
 ---
@@ -74,7 +75,7 @@
 | Gitea repositories | 49 |
 | Repos WITH deployment docs | 6 of 24 services (critical services complete) |
 | Repos with CRITICAL issues | 0 (both plaintext-secret repos resolved) |
-| Active cron jobs | 39 (38 silent, 1 alert-capable) |
+| Active cron jobs | 62 (59 no-agent scripts, 3 LLM-driven — 3 currently with errors) |
 | Backup frequency | 15-min checkpoints + daily full backups on all 4 app servers |
 | Pre-commit secret scanner | Deployed on 7 repos |
 
@@ -180,7 +181,7 @@ Warm failover — boots and auto-restores from S3.
 | 🟡 **HIGH** | **LiteLLM deployment doc** needs fallback chain section + verify `gemini-3.6-flash` availability | Reopened |
 | 🟡 **STALE** | 14 repos with partial/stale docs | Ongoing |
 | 🟡 **UNDOCUMENTED** | 15 production services lack deployment guides (DocuSeal, Komodo, RAGFlow, Dawarich, Camofox, Open WebUI, n8n, etc.) | Not started |
-| 🟢 **GAP** | Auth API / Stack Auth — not deployed, not documented. Needs decision: build or remove DNS entries. | Open |
+| 🟢 **GAP** | **Auth API / Stack Auth** — now confirmed running at `auth2.itpropartner.com` on app3. Needs deployment documentation. | N1 closed. Doc gap remains. |
 | 🟢 **NOTES** | Homelab: adguard-home VM 100 stopped on vm-host-01. QNAP NFS mounts both pointing to `/ISO` export. | Low-priority |
 
 ### Guardrails to prevent recurrence
@@ -191,6 +192,7 @@ Warm failover — boots and auto-restores from S3.
 | **Single master tracker** | `org-audit/docs/production-audit.md` is the one place for finding status. No other audit document tracks status independently. |
 | **Headline accuracy rule** | Executive summaries must not claim more than the body supports. "Full documentation coverage" was wrong; "critical services complete" is correct. |
 | **Server specs: SSH-verify, never assume** | "8C/16G/320G" was wrong — no source supported it. Going forward, specs must be verified via `nproc`, `free -m`, `df -BG` directly. |
+| **Fact-reference before discovery** | N1 false alarm: audit checked guessed domains (`auth.itpropartner.com`, `stack.itpropartner.com`) instead of the known-correct `auth2.itpropartner.com`. Established memory facts (Hexclave = Stack Auth = auth2.itpropartner.com on app3) were available but not referenced before conducting fresh DNS discovery. Process fix: before any discovery scan, query memory and fact_store for known entities being scanned. |
 
 ---
 
@@ -217,8 +219,8 @@ Warm failover — boots and auto-restores from S3.
 
 | Status | Count | Repos |
 |---|---|---|
-| ✅ MATCHES | 20 | itpp-infrastructure, ops-portal, ops-reports, backup-restore, hudu, unifi, unms, hermes-skills, hermes-recovery, mcp-servers, mcp-browser, mcp-email, mcp-filesystem, mcp-git, super-search, shonuff-caller, nvr-shield, shark-game, gift-a-roast, dre, fleettracker360, voipsimplicity, voipsimplicity-manual, apextrack, startup-studio |
+| ✅ MATCHES | 23 | itpp-infrastructure, ops-portal, ops-reports, backup-restore, hudu, unifi, unms, hermes-skills, hermes-recovery, mcp-servers, mcp-browser, mcp-email, mcp-filesystem, mcp-git, super-search, shonuff-caller, nvr-shield, dre, fleettracker360, voipsimplicity, voipsimplicity-manual, shark-game, gift-a-roast, apextrack |
-| ⚠️ PARTIAL/STALE | 14 | disaster-recovery, scripts, model-fallback, homelab, auth, pipeline, research-search-mcp, voice-agent, osint-tool, launchcheck, verdicttank, track-a-flock, transitpin, personal-assistant, content-creation-pipeline, pry, itpropartner-website, forefront-wireless-portal, cartmylist, mooresunnydaze, msp-forms, boxpilot, super-search-business, digital-signage |
+| ⚠️ PARTIAL/STALE | 17 | disaster-recovery, scripts, model-fallback, homelab, auth, pipeline, research-search-mcp, voice-agent, osint-tool, content-creation-pipeline, pry, itpropartner-website, forefront-wireless-portal, cartmylist, mooresunnydaze, msp-forms, boxpilot |
 | ❌ NOT DEPLOYED | 10 | super-search-business, launchcheck, verdicttank, track-a-flock, transitpin, digital-signage, personal-assistant, hotnow.io, IntelSight, SchoolCart |
 
 ### D. Key Documents