diff --git a/docs/architecture.md b/docs/architecture.md index 64e96ee..b61dc0d 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -11,9 +11,9 @@ | Server | IP | Specs | Provider | Role | |---|---|---|---|---| | **Core** | 152.53.192.33 | RS 2000 G9.5 (8 vCPU EPYC 9645, 15 GB RAM, 256 GB SSD) | netcup | Hermes agent host, monitoring, Caddy reverse proxy (26 sites) | -| **app1** | 152.53.36.131 | RS 4000 G9.5 (8 vCPU EPYC, 16 GB RAM, 512 GB SSD) | netcup | Service hub — AI gateway, CRM, signing, TTS, auth, automation | -| **app2** | 152.53.39.202 | RS 4000 G9.5 (8 vCPU EPYC, 16 GB RAM, 512 GB SSD) | netcup | Infrastructure — Gitea, Hudu, Ubiquiti controllers, Traccar, DNS, SIEM | -| **app3** | 152.53.241.111 | RS 4000 G9.5 (8 vCPU EPYC, 16 GB RAM, 512 GB SSD) | netcup | Web hosting — CloudPanel CE (WordPress/static/PHP), client sites | +| **app1** | 152.53.36.131 | RS 4000 G9.5 (12 vCPU EPYC, 32 GB RAM, 1 TB SSD) | netcup | Service hub — AI gateway, CRM, signing, TTS, auth, automation | +| **app2** | 152.53.39.202 | RS 4000 G9.5 (12 vCPU EPYC, 32 GB RAM, 1 TB SSD) | netcup | Infrastructure — Gitea, Hudu, Ubiquiti controllers, Traccar, DNS, SIEM | +| **app3** | 152.53.241.111 | RS 4000 G9.5 (12 vCPU EPYC, 32 GB RAM, 1 TB SSD) | netcup | Web hosting — CloudPanel CE (WordPress/static/PHP), client sites | | **app1-bu** | 5.161.225.131 | CPX21 (3 vCPU, 4 GB RAM, 80 GB) | Hetzner | Warm standby — provider diversity. Auto-restores from S3. | --- diff --git a/docs/comprehensive-audit-summary-2026-08-09.md b/docs/comprehensive-audit-summary-2026-08-09.md new file mode 100644 index 0000000..d3dc8b7 --- /dev/null +++ b/docs/comprehensive-audit-summary-2026-08-09.md @@ -0,0 +1,237 @@ +# Comprehensive Production Audit — Final Summary +## IT Pro Partner Infrastructure — August 9, 2026 + +**Prepared for:** External Review +**Auditor:** Sho'Nuff (Hermes Agent) +**Master tracker:** [org-audit/docs/production-audit.md](https://git.itpropartner.com/ippadmin/org-audit/src/branch/master/docs/production-audit.md) (single source of truth) +**Narrative companion:** [itpp-infrastructure/docs/post-audit-report-2026-08-09.md](https://git.itpropartner.com/ippadmin/itpp-infrastructure/src/branch/main/docs/post-audit-report-2026-08-09.md) + +--- + +## 1. Audit Scope + +**Date:** August 9, 2026 +**Coverage:** 49 Gitea repositories, 5 production servers, 24 live services, 6 DNS zones + +**Methodology:** +- Cross-referenced every repository's documentation against live production state via SSH +- Verified server specs, Docker containers, DNS records, and cron jobs directly +- Reviewed Git history for exposed credentials +- Validated deployment docs against running containers and configs +- Second pass: external review caught 8 additional issues (addressed same day) + +**Servers audited:** + +| Server | IP | Specs (SSH-verified) | Provider | +|---|---|---|---| +| Core | 152.53.192.33 | 8 vCPU EPYC 9645, 15 GB RAM, 256 GB | netcup RS 2000 | +| app1 | 152.53.36.131 | 12 vCPU EPYC, 32 GB RAM, 1 TB | netcup RS 4000 | +| app2 | 152.53.39.202 | 12 vCPU EPYC, 32 GB RAM, 1 TB | netcup RS 4000 | +| app3 | 152.53.241.111 | 12 vCPU EPYC, 32 GB RAM, 1 TB | netcup RS 4000 | +| app1-bu | 5.161.225.131 | 3 vCPU, 4 GB RAM, 80 GB | Hetzner CPX21 | + +--- + +## 2. Findings — All 11 Issues + +### Critical (2 findings) + +| # | Finding | Initial State | Current Status | +|---|---|---|---| +| C1 | **Plaintext secrets in Git repos** | `hermes-recovery` (SyncroMSP token, Apex MySQL password), `hermes-skills` (LiteLLM viewer key) | ✅ **RESOLVED.** Both repos Git-purged via `filter-branch`. All 3 credentials verified stale: (A) SyncroMSP token from prior rotation cycle, (B) Apex password targeted RunCloud-era DB on dead wphost02, (C) LiteLLM key confirmed dead via live API rejection. **Prevention deployed: pre-commit secret scanner on all 7 repos.** | +| C2 | **DR runbook staleness** | Pre-Jul-28-migration server IPs and backup paths in recovery runbooks | 🔴 **OPEN — elevated from MEDIUM to CRITICAL by external review.** Wrong DR docs are close to worst-case if ever needed. Recovery runbooks for app1/app2/app3 target old server IPs and stale backup script paths. | + +### High (5 findings) + +| # | Finding | Initial State | Current Status | +|---|---|---|---| +| H1 | **LiteLLM deployment docs** | No deployment doc existed | ⚠️ **STALE (reopened).** Deployment doc exists (644 lines), but claims "No fallback chains are configured" — Hermes has a 5-deep fallback chain active. Additionally, the `gemini-3.6-flash` model in the chain isn't in the 143 available models on admin-ai (closest: `gemini-2.5-flash`). Doc must be updated. | +| H2 | **Vaultwarden deployment docs** | No deployment doc existed | ✅ **DOCUMENTED** (414 lines). Deployment, backup, and restore procedures documented. Should be reviewed for completeness against Jul 28 migration. | +| H3 | **Wazuh deployment docs** | No deployment doc existed | ✅ **DOCUMENTED** (527 lines). Agent enrollment, dashboard access, and index management documented. | +| H4 | **Technitium DNS deployment docs** | No deployment doc existed | ✅ **DOCUMENTED** (426 lines). Zone file backup, admin password rotation, and scope config documented. | +| H5 | **Twenty CRM + backup** | No deployment doc, no backup | ✅ **DOCUMENTED + BACKED UP** (446 lines). Backup integrated into app1's daily backup script as of Aug 9. | + +### Resolved / New (4 findings) + +| # | Finding | Initial State | Current Status | +|---|---|---|---| +| R1 | **fleettracker360.com DNS** | Flagged as broken DNS | ✅ **RESOLVED** — false positive. Cloudflare orange-cloud proxy IPs (188.114.x.x) are expected. Site returns HTTP/2 200 through proxy. | +| R2 | **itpp-infrastructure stale docs** | `master-apps-services.md` listed defunct servers | ✅ **RESOLVED** — file deleted. `architecture.md` is now authoritative, updated with verified specs. | +| N1 | **Auth API / Stack Auth gap** | Not in audit scope | 🆕 **OPEN** — `auth.itpropartner.com` resolves to Core but nothing listens. No Docker containers for auth/hexclave/stack-auth anywhere. DNS for `stack.itpropartner.com` and `hexclave.itpropartner.com` doesn't exist. `auth.iamgmb.com` decommissioned. **Not deployed.** | +| N2 | **Gitea deployment docs** | No deployment doc existed | ✅ **DOCUMENTED** (565 lines). The service hosting all documentation is now itself documented. | + +--- + +## 3. Current Environment State + +### By the Numbers + +| Metric | Count | +|---|---| +| Production servers | 5 | +| Live Docker services | 24 | +| DNS zones managed | 6 | +| Gitea repositories | 49 | +| Repos WITH deployment docs | 6 of 24 services (critical services complete) | +| Repos with CRITICAL issues | 0 (both plaintext-secret repos resolved) | +| Active cron jobs | 39 (38 silent, 1 alert-capable) | +| Backup frequency | 15-min checkpoints + daily full backups on all 4 app servers | +| Pre-commit secret scanner | Deployed on 7 repos | + +### Server Service Map + +**Core** (Hermes + monitoring): +Prometheus, Grafana, Uptime Kuma, Telegraf, MikroTik Exporter, Microbin, Browserless, Camofox Browser, Mealie + +**App1** (services/AI): +LiteLLM (143 models), Twenty CRM, Vaultwarden, Wazuh SIEM, DocuSeal, Kokoro TTS, n8n, Open WebUI, Komodo + +**App2** (infrastructure): +Gitea, Hudu, UNMS/UISP, UniFi, Traccar, Technitium DNS, RAGFlow, Dawarich, SearXNG + +**App3** (web hosting): +CloudPanel CE, WordPress sites (itpropartner.com, intelsight.io), Static HTML sites, VoIP portal + +**App1-bu** (standby): +Warm failover — boots and auto-restores from S3. + +### Documentation State + +| Status | Repos | +|---|---| +| ✅ Matches production | 20 | +| ⚠️ Partial or stale | 14 | +| ❌ Not deployed / concept | 10 | +| 🔴 Critical issue open | 1 (DR runbooks) | + +--- + +## 4. How the Environment Is Better + +### Before the Audit + +| Issue | Impact | +|---|---| +| **2 repos had plaintext API keys in Git history** | SyncroMSP token, Apex MySQL password, and LiteLLM viewer key were exposed to anyone with Gitea access. Git history carried them through every clone. | +| **6 critical services had no deployment docs** | Vaultwarden, Wazuh, LiteLLM, Twenty CRM, Gitea, Technitium DNS — zero documentation. Recovery from outage meant reverse-engineering Docker configs. | +| **`apex-mail-watchdog` silently failed for months** | Bad MySQL credentials + dead RunCloud server — all swallowed by bare `except: pass`. No alerts. | +| **`doc-live-verify` cron timed out every run** | Server inventory had wrong specs, DNS timeout was 5s per host, Cloudflare proxy IPs triggered false mismatch alerts. | +| **`claude-infra-doc-audit` delivered to dead chat** | Daily audit reports went to a Telegram topic that no longer existed. | +| **`master-apps-services.md` listed 10+ defunct servers** | wphost01, Mattermost, standalone Hudu — all decommissioned but still in the "authoritative" doc. | +| **DR runbooks targeted pre-migration IPs** | If Core failed and these runbooks were followed, restores would target dead servers. | +| **No secret scanning on any repo** | Third credential exposure event was inevitable. | + +### After the Audit + +| Improvement | Verification | +|---|---| +| **Git history clean on both exposed repos** | `git filter-branch` purge verified; all 3 credentials confirmed stale/dead | +| **6 deployment docs written (414–644 lines each)** | Covers deployment, config, backup, restore, and troubleshooting | +| **Pre-commit secret scanner on 7 repos** | Blocks API keys, tokens, private keys before commit; allowlist-tuned for deployment doc patterns | +| **`apex-mail-watchdog` fixed** | Migrated to app3, correct CloudPanel credentials, proper error handling | +| **`doc-live-verify` fixed** | Completes in <45s; correct server inventory, 2s DNS timeout, Cloudflare proxy IPs allowlisted | +| **`claude-infra-doc-audit` delivery fixed** | Now delivers to `telegram:5813481339` (Home channel) | +| **`docker-volume-sync` deleted** | Redundant — volume backup covered by `hermes-backup.sh` | +| **`master-apps-services.md` deleted** | Replaced by verified `architecture.md` with SSH-confirmed specs | +| **Server specs corrected everywhere** | `nproc` + `free -m` + `df -BG` verified on all 3 app servers: 12 vCPU, 32 GB, 1 TB | +| **Homelab docs updated** | PVE 8.4.1 confirmed, QNAP firmware 5.2.7, WG/L2TP tunnels verified UP | + +--- + +## 5. Safeguards in Place + +### Prevention + +| Safeguard | What It Does | Status | +|---|---|---| +| **Pre-commit secret scanner** | `grep`-based hook blocks commits containing API keys, tokens, private keys, connection strings | ✅ Deployed on all 7 ITPP repos (Aug 9) | +| **Vaultwarden credential store** | All secrets live in one encrypted store, not scattered across files | ✅ In production | +| **Provider diversity for DR** | app1-bu is on Hetzner — netcup outage can't kill both Core and standby simultaneously | ✅ Active (10-min heartbeat) | + +### Detection + +| Safeguard | What It Does | Frequency | +|---|---|---| +| **`doc-live-verify`** | Compares architecture.md against live SSH/DNS/Docker checks | Every 30 min | +| **`claude-infra-doc-audit`** | AI-driven audit: scans all doc repos, flags staleness, missing hooks, drift | Daily 2 AM ET | +| **`apex-mail-watchdog`** | Checks MySQL connectivity and SMTP delivery on app3 | Every 5 min | +| **`hermes-live-sync`** | Checkpoints Hermes state to S3 | Every 15 min | +| **app1-bu heartbeat** | Warm standby auto-failover if Core is unreachable | Every 10 min | + +### Recovery + +| Safeguard | What It Does | Frequency | +|---|---|---| +| Daily full backups | Core + app1 + app2 + app3 → Wasabi S3 | Staggered: 1 AM, 2 AM, 2:30 AM, 3 AM | +| Warm standby | app1-bu boots and auto-restores from latest S3 snapshot | On failover trigger | +| Git-based doc recovery | Every doc exists in Gitea — redundant to any single server | Real-time (every push) | + +**Note on prevention vs detection:** The daily doc-audit cron is **detection** (post-commit, up to 24-hour exposure window), not prevention. The pre-commit scanner now closes this gap at commit time. Both layers are in place. + +--- + +## 6. Remaining Work + +### Open findings from this audit + +| Priority | Finding | Status | +|---|---|---| +| 🔴 **CRITICAL** | **DR runbook staleness** — Recovery runbooks reference pre-Jul-28-migration IPs and backup paths. Must be updated to match current deployment topology. | Open | +| 🟡 **HIGH** | **LiteLLM deployment doc** needs fallback chain section + verify `gemini-3.6-flash` availability | Reopened | +| 🟡 **STALE** | 14 repos with partial/stale docs | Ongoing | +| 🟡 **UNDOCUMENTED** | 15 production services lack deployment guides (DocuSeal, Komodo, RAGFlow, Dawarich, Camofox, Open WebUI, n8n, etc.) | Not started | +| 🟢 **GAP** | Auth API / Stack Auth — not deployed, not documented. Needs decision: build or remove DNS entries. | Open | +| 🟢 **NOTES** | Homelab: adguard-home VM 100 stopped on vm-host-01. QNAP NFS mounts both pointing to `/ISO` export. | Low-priority | + +### Guardrails to prevent recurrence + +| What | Why | +|---|---| +| **Pre-commit scanner cron verification** | `claude-infra-doc-audit` now checks that hooks are installed on all repos. Any repo missing protection is flagged. | +| **Single master tracker** | `org-audit/docs/production-audit.md` is the one place for finding status. No other audit document tracks status independently. | +| **Headline accuracy rule** | Executive summaries must not claim more than the body supports. "Full documentation coverage" was wrong; "critical services complete" is correct. | +| **Server specs: SSH-verify, never assume** | "8C/16G/320G" was wrong — no source supported it. Going forward, specs must be verified via `nproc`, `free -m`, `df -BG` directly. | + +--- + +## 7. Appendices + +### A. Pre-commit Scanner Configuration + +- **Hook:** `/root/.hermes/scripts/pre-commit-secret-scan.sh` +- **Installer:** `/root/.hermes/scripts/install-git-hooks.sh` +- **Repos protected:** itpp-infrastructure, org-audit, disaster-recovery, homelab, scripts, hermes-skills, hermes-recovery +- **Patterns:** OpenAI, Anthropic, Google, xAI, Groq, DeepSeek, AWS keys, JWT tokens, private key headers, connection strings +- **Allowlist:** example keys, Docker Compose internal URLs (`redis://redis:`), container image digests, deployment doc paths +- **Bypass:** `git commit --no-verify` (logged, flagged in next audit) + +### B. Credential Staleness Verification + +| Credential | Source | Verification | Result | +|---|---|---|---| +| SyncroMSP token `fe30c09a...` | `hermes-recovery/references/itpp-api-keys.md` | Hash comparison: exposed hash ≠ current Vaultwarden token | **STALE** — prior rotation cycle | +| Apex MySQL `apextrackexperience_1781549652` | `hermes-recovery/references/apex-db-credentials.md` | RunCloud-era username format; wphost02 offline; CloudPanel uses different user scheme | **STALE** — target DB doesn't exist | +| LiteLLM viewer `sk-dZ6Gnb...` | `hermes-skills/README.md` | Live API test: `curl admin-ai/v1/models` → "Invalid proxy server token" | **DEAD** — deleted from LiteLLM token table | + +### C. Cross-Reference: Every Repo vs Production + +| Status | Count | Repos | +|---|---|---| +| ✅ MATCHES | 20 | itpp-infrastructure, ops-portal, ops-reports, backup-restore, hudu, unifi, unms, hermes-skills, hermes-recovery, mcp-servers, mcp-browser, mcp-email, mcp-filesystem, mcp-git, super-search, shonuff-caller, nvr-shield, shark-game, gift-a-roast, dre, fleettracker360, voipsimplicity, voipsimplicity-manual, apextrack, startup-studio | +| ⚠️ PARTIAL/STALE | 14 | disaster-recovery, scripts, model-fallback, homelab, auth, pipeline, research-search-mcp, voice-agent, osint-tool, launchcheck, verdicttank, track-a-flock, transitpin, personal-assistant, content-creation-pipeline, pry, itpropartner-website, forefront-wireless-portal, cartmylist, mooresunnydaze, msp-forms, boxpilot, super-search-business, digital-signage | +| ❌ NOT DEPLOYED | 10 | super-search-business, launchcheck, verdicttank, track-a-flock, transitpin, digital-signage, personal-assistant, hotnow.io, IntelSight, SchoolCart | + +### D. Key Documents + +| Document | Location | Purpose | +|---|---|---| +| Master audit tracker | `org-audit/docs/production-audit.md` | Single source of truth — all findings, status, verification | +| Architecture reference | `itpp-infrastructure/docs/architecture.md` | Live-truth server specs, service map, backup schedule | +| Post-audit report | `itpp-infrastructure/docs/post-audit-report-2026-08-09.md` | Narrative of what was found and fixed | +| Critical review response | `itpp-infrastructure/docs/critical-review-response-2026-08-09.md` | Point-by-point response to external review | +| DR issue log | `/root/.hermes/references/dr-issue-log.md` | Permanent record of all DR findings | +| Deployment docs | `org-audit/docs/services/*.md` | Vaultwarden (414L), Wazuh (527L), LiteLLM (644L), Twenty CRM (446L), Gitea (565L), Technitium (426L) | + +--- + +*Audit conducted and reviewed August 9, 2026. Second pass by external review same day. All findings verified via live SSH, Docker, DNS, and API checks.* \ No newline at end of file