docs: incident report for 2026-09-12 Core state.db recovery

- WAL-unlink root cause: interactive root hermes doctor --fix / optimize-storage under a live gateway

- store rebuilt via snapshot -> .recover -> quarantine -> install; 1,902 MB -> 1,108 MB; integrity ok

- live-sync coverage resumed and verified

.gitignore

@@ -6,3 +6,7 @@
 .env.*
 *.key
 *.pem
+
+# Nested standalone repos (own Gitea remotes)
+projects/seo-tool/
+projects/venturebuilt/

CHANGELOG.md

@@ -1,5 +1,42 @@
 # itpp-infrastructure — CHANGELOG
 
+## 2026-09-11 — Anita's Hermes Profile Moved to a Dedicated Box (anita-mnz)
+
+- **Infra move:** Anita's assistant profile moved off shared Core (`152.53.241.111`) to a dedicated box **anita-mnz `159.195.16.30`** (netcup, Manassas VA, 8 vCPU / 15 GB / 503 GB). Cutover 15:53 EDT, ~2 minutes dark, **zero messages lost**. She keeps the same Telegram bot and chat.
+- **Why:** repeated `state.db` corruption on Core under co-tenant memory pressure (4th recurrence, `system_prompts` then `sessions` pages). A dedicated box removes the co-tenancy.
+- **Method:** clean staged base + row-tolerant live-tail graft, so the corrupt live file never transferred. Signatures matched exactly: `99283/99283/166244414/157` (messages/maxid/contentbytes/sessions), `integrity_check ok`.
+- **Post-move state:** Core `hermes-gateway-anita.service` stopped and **disabled** (no double-poller risk; standby app1-bu carries no anita unit); target gateway `active`+`enabled`, telegram `connected`, 0 `getUpdates` conflicts.
+- **Backup:** her own box runs the 3 AM root-essentials backup → `s3://hermes-vps-backups/root-backup/anita-mnz/`. Full run tested end-to-end 2026-09-11 15:56 (170 MB, upload + download/extract verify OK).
+- **⚠ Correction to the line above (17:35, same day):** that archive is config only. `root-essentials-backup.sh` excludes `*.db` by design, and nothing on the new box replaced that exclusion, so her conversation store, cron execution DB, notepad, wisdom and verification DBs had **zero** backup coverage. Fixed the same evening — details below.
+- **Core change:** `hermes-live-sync` no longer snapshots the frozen `profiles/anita` copy (it would advertise a stale db as "live").
+- **MCP servers:** stripped from her profile (decision 2026-09-11: "She doesn't need access to those"). The `mcp_servers` block (`dre`, `osint-person`, `super-search` → `127.0.0.1:8900/8902/8899`) was removed from `anita/config.yaml`, which is why her log had been retry-parking those three every ~5 minutes since the move. Backed up first as `config.yaml.bak-mcpstrip-*`; 13 top-level keys verified intact. Takes effect on her next gateway restart.
+- **Retired / order cancelled:** Nuremberg `89.58.44.96` (`v2202609377162518632.nicesrv.de`). **Correction to how this was first written here:** it does not "hold nothing". It holds a bare default-profile Hermes install: no `state.db`, empty `sessions/` and `cron/`, no systemd user units, no gateway process (only node_exporter, containerd, sshd). No unique data, so nothing needs preserving before cancellation. Recorded in `/root/.hermes/references/decommissioned-hosts.json`.
+- **Monitoring fix:** `health-master-watchdog.py` watch-listed `hermes-gateway-anita.service` as a LOCAL user unit on Core, so it would have alerted forever once that unit was disabled. Local check removed; `anita-mnz` (`159.195.16.30`) added to `REMOTE_SERVERS` and a new `REMOTE_USER_UNITS` remote user-unit check added (SSH + `XDG_RUNTIME_DIR`). Verified live: no false alert for her gateway, and her box answers as active.
+- **Docs:** full incident + pitfalls in `/root/.hermes/references/dr-issue-log.md`; transferable procedure in the `hermes-migration` skill.
+- **Backup coverage gap (found 17:20, fixed 17:35):** the nightly archive on her box was green and 170 MB, but a restore test showed it contains no `.db` file at all. `root-essentials-backup.sh` excludes `*.db` (correct — never `tar` a live SQLite file), which on Core is backstopped by `hermes-backup.sh`. The migrated box got the script set without the backstop. **Silent failure mode**: every indicator said "backed up" while her 974 MB store had no copy anywhere except the frozen Core dir being deleted.
+- **Fix:** new `hermes-db-backup.sh` on `anita-mnz` (`/root/hermes-db-backup.sh`, chmod 750), cron `10 3 * * *`. Per-DB `sqlite3 .backup` (safe on a live DB), `PRAGMA quick_check` on every snapshot before it is accepted, one dated tarball, upload to `s3://hermes-vps-backups/root-backup/anita-mnz/db/`, then **downloads the object back, extracts and re-verifies**. 14-day retention.
+- **Verified:** first run 17:04 EDT, 291 MB uploaded; restore test passed — extracted, `quick_check=ok`, 99,285 messages.
+- **Migration completeness proven before deleting the Core copy:** `memories/MEMORY.md`, `memories/USER.md` and `.env` are md5-identical between the frozen Core copy and `anita-mnz`; `cron/jobs.json` holds the same six job IDs; skills 125 on MNZ vs 124 frozen; store 99,285 vs 99,283 messages. `config.yaml` differs only by the deliberate MCP strip.
+- **Incident report written:** `docs/incidents/2026-09-11-core-state-db-corruption.md` documents the store corruption (header destroyed at 12:49:22), the recovery that built a new store from the clean 01:00 archive and grafted 985 newer messages into it (108,573 messages, max id 322,530, 242 sessions, `quick_check` and `integrity_check` both `ok`), the eight day backup monitor false positive streak, and the prevention list (prune the store, snapshot instead of tar, restore-test every archive, integrity check the newest snapshot, record who restarts the gateway).
+- **Frozen Core copy removed (17:41):** `/root/.hermes/profiles/anita` (8.0 GB) was 7.9 GB of corrupt-DB corpses (`corrupt-20260903/09/10`, `pre-rebuild-20260910`, `recovered-20260910`). Archived to `s3://hermes-vps-backups/decommissioned/anita-core-frozen-profile-20260911-1732.tar.gz` (2,786,082,561 B, 21,206 entries), **verified by downloading the object back and matching sha256** `b457fa7f0cb6f546f36e53a338e90b7b940f51cf29d5a73d0cceb97f49c9b3dc` against the local tarball, then deleted. Nothing unique was destroyed: memories, `.env` and the six cron job IDs were identical on `anita-mnz`, which carries one more skill and two more messages. Core disk 125 GB → 117 GB used; `/root/.hermes/profiles` is now empty. Also removes that copy from Core's nightly archive, which is why Core's backup had grown to 2.97 GB.
+- **Stale S3 copy purged (17:41):** `s3://hermes-vps-backups/live/profiles/anita/` held 20,551 objects / 5.0 GB that the then-running 15-minute sync had pushed before it was paused on Sep 3, including a stale-looking `state.db` that would have been advertised as "live". Superseded by the archive above and removed; `live/profiles/` is now empty.
+- **Backup monitor checked (17:38):** its single CRITICAL was `hermes-live-sync: DISABLED/PAUSED`, which is the pause from Sep 3 and explains the monitor's 8-day exit-1 streak (it exits 1 only on CRITICAL, never on warnings). Its three WARNINGs are false positives, verified by content rather than size: Wazuh manager tarballs are sha256-identical for 3 days (static config, job ran 03:15 today), LiteLLM's flagged object is a 333 B config yaml that never changes (the real backup is a 37.9 MB `litellm-backup-*.tar.gz` from 03:30 today, and there are 47 of them), and the voipsimplicity dump is the same 6,834,636 B each day but a **different** sha256 each day (valid gzip, 73 tables). The size-uniqueness heuristic cannot tell static-but-fine from stalled; the three entries should be reclassified as "unchanged content" rather than SUSPICIOUS.
+- **Same-day correction on job status:** "Germaine inbox watch" is `enabled=false` on `anita-mnz`. It was `enabled=false` in the frozen Core copy too — the migration did not disable it.
+
+## 2026-08-17 - Scirium v2 Proposal Deployed to /scirium/
+
+- **v2 proposal deployed** to `proposals.itpropartner.com/scirium/` (index.html + 04-business-proposal-v2.md + critical-review.html). Assembled from 4 parallel remediation teams (marketing, technical, financial, legal), SOM reconciled with Financial as authority, build cost corrected to ~$167K (was $68K), verdict: GO with conditions (churn gate, acquisition-maturation gate, trademark clearance, Phase 0 DLP spike). Status: DRAFT FOR REVIEW pending Germaine review before VerdictTank resubmission.
+- **URL rename completed:** v1 (codename Wall-O) frozen at `proposals.itpropartner.com/wall-o/` with a SUPERSEDED banner pointing to `/scirium/`. v2 is live at `/scirium/`. This closes the pending item from the 2026-08-16 changelog entry.
+- **Sources:** v1 at `projects/scirium/04-business-proposal.md`; v2 at `projects/scirium/04-business-proposal-v2.md`; team remediation sections under `/tmp/scirium-v2/output/` (not repo-bound).
+
+## 2026-08-16 — Wall-O Renamed to Scirium
+
+- **Product renamed Wall-O → Scirium** (coined from Latin "scire" = to know). Applies going forward; "Wall-O" retired to internal codename history only.
+- **Domain:** `scirium.com` selected. `.com`/`.io`/`.ai`/`.co`/`.app` all available (RDAP 404 + empty NS cross-check). Trademarkia: 0 results for "scirium".
+- **Cloudflare at-cost pricing (verified 2026-08-16):** `.com` $10.44/yr, `.io` $50/yr (renewal ~$51.75), `.ai` $70/yr (min 2-year term = $140; rising to $80/yr on 2026-03-05), `.co` $15 first yr / $30 renewal, `.app` $14.20/yr, `.dev` $10.18/yr.
+- **Source folder moved** `projects/wall-o/` → `projects/scirium/`. Legacy 4 docs still carry "Wall-O" internally; rebranded by the docs team as part of the v1/v2 documentation package.
+- **Deployed proposal URL** (`proposals.itpropartner.com/wall-o/`) unchanged pending redeploy under `/scirium/`.
+
 ## 2026-08-12 — app3 Web Docroot Migration to Per-Site Users
 
 - **Change:** every app3 nginx vhost moved off the shared `/home/ippadmin/htdocs/` root to a per-site dedicated Linux user with docroot `/home/<site-user>/htdocs/<domain>` (security hardening — no more single-owner web tree).

README.md

@@ -88,9 +88,11 @@
 ### Legacy / Decommissioned
 - **old-ai:** 178.156.167.181 (Hetzner CPX41) — **decommissioned** (LiteLLM migrated to app1)
 - **old app1:** 87.99.144.163 (Hetzner CPX11) — **deleted**
-- **wphost02:** 5.161.62.38 (Hetzner CPX21) — **migrated to app3**
+- **wphost02:** 5.161.62.38 (Hetzner CPX21) — **DECOMMISSIONED (2026-08-28)** — deleted from Hetzner account; sites migrated to app3
 - **Ollama:** Removed from Core (systemd) and app1 (Docker) Jul 17
 
+> **Hetzner Cloud (current):** As of 2026-08-28, the Hetzner Cloud API returns exactly **one** server — **app1-bu / core-bu** (`5.161.225.131`, CPX21, warm standby for Core). All other Hetzner boxes (old-ai, wphost02) are decommissioned/deleted.
+
 ---
 
 ## Model Fallback Chain
@@ -171,7 +173,7 @@ Direct API keys for all providers. Claude Sonnet 5 for primary quality, then dir
 | app1-backup | Daily 2 AM | s3://hermes-vps-backups/app1/ | LiteLLM, n8n, OpenWebUI, MCP configs |
 | app2-backup | Daily 2:30 AM | s3://hermes-vps-backups/app2/ | Traccar, Gitea, Hudu, UNMS, UniFi |
 | app3-backup | Daily 3 AM | s3://hermes-vps-backups/app3/ | CloudPanel, MySQL, WordPress |
-| wphost02-backup | Daily 5 AM | s3://hermes-vps-backups/wphost02-backup/ | Webapps + MySQL |
+| ~~wphost02-backup~~ | ~~Daily 5 AM~~ | ~~s3://hermes-vps-backups/wphost02-backup/~~ | **REMOVED — wphost02 decommissioned 2026-08-28** |
 | warm-standby-sync | Every 10 min | core-bu ← S3 | DR readiness |
 
 ---

api-master-list.md

@@ -100,7 +100,7 @@
 | SyncroMSP | SyncroMSP | RMM/PSA, client asset management |
 | Bitdefender GravityZone | Bitdefender | Endpoint security, client AV |
 | Cloudflare | Cloudflare | DNS zones, domains, records |
-| Hetzner Cloud | Hetzner | app1-bu standby server, wphost02 |
+| Hetzner Cloud | Hetzner | app1-bu standby server (sole online Hetzner box as of 2026-08-28; wphost02 decommissioned) |
 | netcup | netcup | Core/app1/app2/app3 servers |
 | Wasabi S3 | Wasabi | All backups (hermes-vps-backups bucket, app backups) |
 | Firecrawl | Firecrawl | Web extraction (Super Search fallback) |

audit/phase-one/brief.md

@@ -0,0 +1,92 @@
+# Phase One: ITPP Infrastructure Audit - Operational Brief
+
+**Engagement:** Discovery, audit, and documentation only. **READ-ONLY.**
+**Status:** GO authorized by Germaine Brown on 2026-08-13.
+**Conductor:** Sho'Nuff (deepseek-v4-pro). Hands-on discovery runs through fresh subagents only.
+
+## ABSOLUTE RULES (every subagent, no exceptions)
+
+1. **READ-ONLY.** No config changes, restarts, patches, credential rotations, firewall edits, or any live modification - regardless of how beneficial. Every remediation is written up as a Phase Two finding.
+2. If validating a finding requires a non-read-only action (test change, restart, failover test), **do not do it** - document the limitation and flag for Phase Two.
+3. **Verify against live systems, never assume from docs/memory.** No finding based solely on existing documentation without confirming against what is actually running.
+4. Flag any area where access/credentials/visibility were insufficient - do not guess.
+5. Never emit credentials. Use `[REDACTED]` everywhere, including paths that contain secrets.
+6. No em dashes. Write findings in plain English a technical-but-not-infra-SME reader understands - explain *why* a finding matters, not just what it is.
+
+## KNOWN INFRASTRUCTURE FOOTPRINT (verify independently, do not trust)
+
+Hosting: **Netcup** (4 servers) + **Hetzner** (2 servers) = 6 total.
+
+| Server | Provider | IP | Role (to verify) |
+|---|---|---|---|
+| Core | Netcup RS 2000 | (this host) | Hermes host + core services: Grafana :3002, Prometheus, Super Search MCP :8899, backup scripts |
+| app1 | Netcup RS 4000 | 152.53.36.131 | LiteLLM/admin-ai (docker), Caddy, super-search, mcp-*, browserless |
+| app2 | Netcup RS 4000 | 152.53.39.202 | Hudu, UNMS, UniFi, Traccar, Gitea, Dawarich, Technitium DNS (docker) |
+| app3 | Netcup RS 4000 | 152.53.241.111 | CloudPanel static+PHP, WordPress sunset |
+| app1-bu | Hetzner CPX21 | 5.161.225.131 | Warm standby for Core, Hermes failover |
+| wphost02 | Hetzner | (resolve) | Legacy WordPress/RunCloud host |
+
+**Also in scope:** MikroTik CCR tower backups, UniFi/UISP edge, git.itpropartner.com (Gitea), docs.itpropartner.com (to be created), Wasabi S3 backups.
+
+**SSH access:** `ssh -i /root/.ssh/itpp-infra root@<ip>`. Key at `/root/.ssh/itpp-infra` on Core.
+**Verify more than this list** - surface shadow IT, forgotten instances, additional accounts.
+
+## TEAM ROSTER & BATCHES
+
+| # | Member | Role | Model | Owns | NOT covering |
+|---|---|---|---|---|---|
+| 0 | Conductor | Audit lead | deepseek-v4-pro (orchestration) + claude-sonnet-5 (report/QA synthesis) | Synthesis, QA, report, policy doc, skill spec, disagreement log | Hands-on discovery |
+| 1 | NetEng-A | Network Eng | deepseek-v4-pro | Firewall/ports/DNS/VPN all 6 + MikroTik + UniFi/UISP edge | Segmentation policy |
+| 2 | NetEng-B | Network Eng | claude-sonnet-5 | Segmentation & blast-radius (internal vs client vs product/dev) | Raw port/DNS enum |
+| 3 | Sec-A | InfoSec | claude-sonnet-5 | IAM + credentials/secrets inventory (standalone) | Hardening/patch |
+| 4 | Sec-B | InfoSec | claude-sonnet-5 | Hardening, patch, MFA, logging (Wazuh), CIS/NIST | Secrets inventory |
+| 5 | Sys-A | Sysadmin | deepseek-v4-pro | Core + app1 (highest risk) | app2/app3/app1-bu/wphost02 |
+| 6 | Sys-B | Sysadmin | deepseek-v4-pro | app2 + app3 + app1-bu + wphost02 | Core/app1 |
+| 7 | Sys-C | Sysadmin | claude-sonnet-5 | Backup/DR verification - 27 targets vs live S3, restore-test history, RTO/RPO | Service config drift |
+| 8 | Git-A | Repo auditor | claude-sonnet-5 | git.itpropartner.com inventory, classification, lineage, secrets-in-history | Docs repo build |
+| 9 | Docs-W | Documentation | claude-sonnet-5 | Docs repo creation + docs.itpropartner.com publish | Git audit findings |
+| 10 | Indep | Independent reviewer | claude-sonnet-5 | Bias-check severity + disagreement log (separate model) | Full re-audit |
+
+## BATCH PLAN (conductor executes)
+
+- **Batch 1 (mechanical, deepseek-v4-pro):** NetEng-A, Sys-A, Sys-B
+- **Batch 2 (reasoning, claude-sonnet-5):** NetEng-B, Sec-B, Sys-C, Git-A, Docs-W (concurrent, max 3 at a time)
+- **Batch 2b (premium, claude-sonnet-5):** Sec-A (standalone IAM/secrets deliverable)
+- **Batch 3 (independence check, claude-sonnet-5):** Indep - reviews severity ratings + disagreement log AFTER conductor synthesizes
+
+## DISCOVERY SCOPE (per subagent - see individual briefs for exact deliverables)
+
+Each subagent returns a **structured findings list** with severity (Critical/High/Medium/Low), evidence (command output, file paths, timestamps), and written rationale. Coverage:
+
+1. **Network (NetEng-A, NetEng-B):** topology, routing, firewalls, open ports, DNS records, VPNs, VLAN/segmentation, wireless, public exposure surface, blast-radius between internal/client/product-dev.
+2. **InfoSec (Sec-A, Sec-B):** IAM/accounts/MFA, privileged access, secrets location & plaintext/hardcoded/unrotated flags, patch posture, hardening (CIS/NIST/SOC 2 lens), logging/monitoring.
+3. **Sysadmin (Sys-A, Sys-B):** per-server per-service granular inventory: identity/version/purpose/dependencies (dependency map), backup status + restore-test history, config file paths + drift vs expected, cron jobs (schedule/user/deps/failure-mode), estimated RTO/RPO for critical services.
+4. **Backup/DR (Sys-C):** verify all 27 backup targets against live S3 (Wasabi), retention, restore-test history, evidence-based RTO/RPO per critical service, 3-2-1 rule compliance.
+5. **Git (Git-A):** inventory every repo: name/purpose/last commit/primary branch/deploy target; classify active vs stale vs superseded vs orphaned; lineage; secrets-in-history; disposition recommendation per stale repo.
+6. **Docs (Docs-W):** determine existing docs; if present audit say-do gap; if absent create. New dedicated repo on git.itpropartner.com + publish to docs.itpropartner.com. Delineate internal vs client vs product/dev.
+
+## DELIVERABLES (conductor produces at end)
+
+**A. Audit Report** (table of contents + exec brief for Germaine, plain-English):
+1. Discovery Summary (verified inventory)
+2. Findings (prioritized, severity tiers with rationale)
+3. Recommendations (mapped to findings, effort estimate)
+4. Documentation Status
+5. Infrastructure Separation Assessment
+6. Git/Docs Reorganization Recommendation (actual proposed tree, plain-English rationale, where existing repos land)
+7. Disagreements section (documented, not resolved - Germaine resolves)
+8. Independence-check appendix (Indep's severity review)
+
+**B. Policy & Procedure Document** (change mgmt, docs-sync, backup/DR standards, access/offboarding, segmentation for new entities, recurring audit cadence).
+
+**C. Skill Spec (documentation only)** for a Hermes policy-adherence skill: flag requests falling outside policy, only Germaine authorizes exemptions, running Exemptions Document schema (date, requester, provision departed, justification, Germaine approval, one-time vs ongoing, follow-up).
+
+## COST
+
+Estimate (approved): subtotal ~$5.40, realistic $8-10, ceiling ~$13. Under $20 cap. No opus-tier models used.
+Track actual via LiteLLM SpendLogs (now verified accruing). Include estimate-vs-actual in final report.
+
+## NOTIFICATION POLICY
+
+- **Silent during run.** Only surface: (1) questions needing Germaine input, (2) Critical/High findings that cannot wait, (3) completion.
+- Completion = final report + policy doc + skill spec **emailed to g@germainebrown.com**.

audit/phase-one/findings/docs-w.md

@@ -0,0 +1,169 @@
+# Docs-W Findings: Documentation Audit (Phase One, Read-Only)
+
+**Auditor:** Docs-W (documentation auditor)
+**Scope:** docs.itpropartner.com (live MkDocs Material site on app3) and its stale duplicate; cross-referenced against neteng-a.md, neteng-b.md, sec-b.md, sys-a.md, sys-b.md, sys-c.md.
+**Method:** Read-only SSH to app3 (152.53.241.111), filesystem inspection, HTML content extraction of every section/page and CHANGELOG. No live docs, nginx, or DNS were modified.
+**Date:** 2026-08-13
+
+---
+
+## 1. Site Inventory
+
+**Platform:** MkDocs Material, built and copied as static HTML into two htdocs trees on app3. No `.git` in either served tree, so the live docs root itself carries no version history or build provenance.
+
+**Served copy:** `/home/docs/htdocs/docs.itpropartner.com` (owned by user `docs`, referenced by `/etc/nginx/sites-enabled/docs.itpropartner.com.conf`, most recent build timestamp ~2026-08-12 19:31 local).
+**Unserved duplicate:** `/home/ippadmin/htdocs/docs.itpropartner.com` (root-owned, not referenced by any nginx vhost).
+
+**Access control:** Since 2026-08-10, the entire site is gated by `docs-auth-validator.service` (JWT + email allowlist against auth2/Stack Auth). Before that date the site was open access. See section 3 (say-do gaps) for the implication.
+
+### 1.1 Top-level sections (12), status, and last-update evidence (from each section's CHANGELOG)
+
+| Section | Claims to document | Last CHANGELOG entry | Age vs audit date (2026-08-13) | Status |
+|---|---|---|---|---|
+| ITPP Infrastructure | Server inventory, DNS, architecture, key inventory, model routing, cost controls, ops portal, backup-restore, legal, project log | 2026-08-10 (Docs Auth Gate) | 3 days | Current |
+| ITPP Standards | Documentation templates, CI (docs-check, docs-publish) | 2026-08-09 (Initial) | 4 days | Current |
+| TransitPin | White-label transportation portal, dispatch, driver PWA | 2026-08-09 (migrate to Git); content dated 2026-07-29 | 4 days (changelog) / 15 days (content) | Current (changelog) |
+| VerdictTank (+ Architecture) | Product review/validation platform, pricing, pipeline review | 2026-08-10 (v3 proposal + architecture) | 3 days | Current |
+| HomeLab | Home lab infrastructure automation | "2026-07" (state snapshot 2026-07-21) | ~3+ weeks | Stale |
+| Scripts | Operations/automation script catalog | 2026-07-21 | 3 weeks | Stale |
+| FleetTracker360 | GPS fleet tracking platform | 2026-07-16 | 4 weeks | Stale |
+| LaunchCheck | Startup validation product | 2026-07-25 (Project Inception, single entry) | 3 weeks, never updated since inception | Stale |
+| Shark Game | Shark Attack Fantasy League | 2026-07-10 | 5 weeks | Stale |
+| Apex Track | Track event management | 2026-07-10 ("Created project repository," single entry) | 5 weeks, stub only | Stale |
+| BoxPilot | Logistics operations platform | 2026-07-10 ("Created project repository," single entry) | 5 weeks, stub only | Stale |
+| OSINT Tool | OSINT people search / skip tracing | 2026-07-10 (Documentation migration) | 5 weeks | Stale |
+
+**Section count: 12. Current: 4. Stale: 8.**
+
+### 1.2 Notable sub-pages
+
+- `itpp-infrastructure/`: Overview, Docs Auth Gate, Key Inventory, Model Chain, Cost Control Rollout (2026-07-24), Super Search CF Bypass (2026-07-21), app2 Caddyfile Audit (2026-07-21), Backup-Restore (+Architecture), Ops Portal (+Architecture), **Legal (empty page, no content rendered)**, Project Log, Projects Master README.
+- `verdicttank/`: Overview, Architecture, plus markdown source files for v3.5/v3.6 architecture and a legal-framework document.
+- `launchcheck/`: Competitive Analysis, Business Proposal (in addition to the section root).
+
+---
+
+## 2. Say-Do Gap Audit (cross-referenced against neteng-a/b, sec-b, sys-a/b/c)
+
+| # | Doc claim | Live reality (per findings file) | Severity |
+|---|---|---|---|
+| 1 | `key-inventory` states secrets were "Sanitized... plaintext secrets replaced with storage references" (2026-07-23) | sys-a F-2: plaintext JWT_SECRET/DEEPSEEK_API_KEY/ADMIN_AI_KEY hardcoded in world-readable systemd units on Core (rally, seemytrip) and app1 (giftaroast). sys-b C2: `/root/.hermes/.env` on app1-bu (mode 644) holds ~20 plaintext secrets including root passwords for app1/app2/app3. sys-b C3: MySQL root password hardcoded in plaintext in two app3 scripts, one world-readable (775). | **Critical** - the one doc whose entire purpose is to assure the org that secrets are handled safely is contradicted by the live estate in at least three independent locations. |
+| 2 | `ops-portal` CHANGELOG (2026-07-20) publishes a line reading "Admin credentials: [old username]/[old password] -> ippadmin (password moved to Vaultwarden)" | The docs site was open access (no auth gate) until 2026-08-10 per the site's own `docs-auth-gate` doc. This means a real historical admin credential sat in plaintext on an unauthenticated public web page for roughly three weeks. | **Critical** - a credential exposure inside the documentation itself, not just the infrastructure. |
+| 3 | `docs-auth-gate` describes a JWT+allowlist access-control layer protecting the whole site, implying documentation (including key-inventory and the credential above) is now access-controlled | None of the six live-verified findings files enumerate or test `docs-auth-validator.service`/port 8099 on app3. sec-b's MFA/hardening coverage table (11 admin surfaces reviewed) does not include it. Its actual enforcement is undocumented outside its own self-description. | High - an access control the org depends on for a page containing credential-adjacent content has never been independently verified by the security or network auditors. |
+| 4 | `projects-master-readme` lists Apex Track and BoxPilot as **"(PLANNED)"** | Both already have fully generated dedicated MkDocs sections (nav entries, CHANGELOG, "Created project repository and directory structure" stub content) on the same site. | Medium - internal self-contradiction on project status, visible on two pages of the same docs tree, no external cross-reference needed. |
+| 5 | `itpp-infrastructure/legal` is a live nav entry titled "Legal" | Page renders with no content at all. | Medium - a documented, linked page with nothing behind it; reads as either an abandoned stub or a missed publish step. |
+| 6 | `homelab` documents host inventory, DNS chain, and Docker service catalog for the separate home-lab network as if it were part of the audited estate | HomeLab (vm-host-01/02, QNAP, MikroTik home router) is **outside the scope** of neteng-a/b, sec-b, sys-a/b/c, which cover only Core/app1/app2/app3/app1-bu/wphost02. There is no live-verified findings coverage to check this section against at all. | Medium - a documented area of the estate with zero Phase One audit visibility; a structural blind spot rather than a specific factual error. |
+| 7 | `model-chain` (2026-07-24) documents the AI model fallback chain and the admin-ai virtual key's daily budget cap, implying this is a governed, protected control | sys-a F-1: the LiteLLM Postgres database that stores this exact routing/budget/key configuration is **never backed up** (dump targets a nonexistent database name and fails silently every night). The governance the doc describes has zero disaster-recovery coverage, a fact the doc itself does not mention. | High - doc describes a control without disclosing that its backing store is unprotected. |
+| 8 | `app2-caddyfile-audit-2026-07-21` documents a "final Caddyfile (validated)" for app2 routing UNMS, Gitea, UniFi, dns1, FleetTracker360 through Caddy | neteng-a APP2-1 through APP2-5 show UniFi, UNMS, Gitea SSH, Technitium DNS, and several other app2 services are **also** reachable directly via Docker's UFW bypass, i.e. the "validated" Caddy-only routing picture in the doc is incomplete: the real exposure includes direct Docker-published ports the doc never mentions. | High - the doc documents the intended path but is silent on (and therefore implicitly contradicts) the actual public exposure discovered live. |
+| 9 | `backup-restore` doc describes a 30-day retention snapshot system and a documented restore API for 9 WordPress domains | sys-c's live restore-test audit found **zero** restore tests have ever been run against this specific mechanism (only Gitea and Vaultwarden have been restore-tested estate-wide), and sys-c SYSC-10 notes this snapshot layer is local-disk-only with no offsite copy - a single point of failure the doc does not disclose. | Medium - doc presents the system as complete/operational; live audit shows "backed up" without "restore-verified," which is exactly the gap Sys-C's engagement rule was written to catch. |
+
+**Top 5 say-do gaps (for summary):** #1, #2, #3, #7, #8 above - ranked by blast radius and the fact that each represents documentation actively asserting a safety property (secrets sanitized, admin credential rotated, access gated, budget governed, routing validated) that the independently-verified infrastructure findings show is false, incomplete, or unverifiable.
+
+---
+
+## 3. Split-Brain: Two Copies of docs.itpropartner.com
+
+**Finding:** Two nearly-identical full copies of the docs site exist on app3:
+
+- `/home/docs/htdocs/docs.itpropartner.com` - owned by `docs`, referenced by the live nginx vhost, most recent build ~2026-08-12 19:31 local. **This is the authoritative, served copy.**
+- `/home/ippadmin/htdocs/docs.itpropartner.com` - owned by `root`, **not referenced by any nginx config**, most recent build ~2026-08-10 (roughly 29 hours older than the served copy).
+
+**Verification performed:** file-list diff between the two trees returns zero differences (93 files each, identical filenames/paths), and `index.html` content is byte-identical. The only measurable difference is per-file modification time, consistently ~29 hours older on the `ippadmin` copy across sampled files. This confirms both are build outputs of the same MkDocs source, generated by the same pipeline at two different times, not two diverging content sources.
+
+**Root cause (most likely, not confirmed via any log evidence during this read-only audit):** an early build/deploy of the docs site landed in the default `ippadmin` home path (the operator's own home directory, `root`-owned), and a later correction properly deployed to a dedicated `docs` system user matching the nginx vhost root. The stale copy was never cleaned up.
+
+**Risk:**
+- No `.git` exists in either tree, so there is no audit trail proving which copy is "true" beyond nginx's own vhost pointer and file mtimes; a future config regeneration (this is a CloudPanel-managed host) or an operator mistake pointing the vhost root back at `/home/ippadmin/...` would silently roll the live site back ~29 hours, re-serving stale content (e.g., pre-dating the Aug 10 auth-gate documentation and VerdictTank v3 updates) with no visible error.
+- The stale copy is `root`-owned while every other operational convention on this host uses a dedicated service user; this is itself a hygiene deviation worth correcting regardless of the duplication.
+- There is currently no single documented deploy path (git repo -> build -> publish target) for this site, so a repeat of this duplication is only prevented by operator memory.
+
+**Recommendation (Phase Two):** delete or archive `/home/ippadmin/htdocs/docs.itpropartner.com`, and formalize the build/publish pipeline (ITPP Standards already defines a `docs-publish` Gitea Actions workflow template; wire the live docs site to that pipeline with the `docs` htdocs path as its sole target) so there is exactly one deploy destination, git-tracked, going forward.
+
+---
+
+## 4. Audience Delineation: Internal vs Client vs Product/Dev
+
+**Current state: no delineation exists.** All 12 sections live under one flat, undifferentiated MkDocs nav, gated by a single email allowlist behind one auth layer. Concretely mixed on the same site, one click apart:
+
+- **Internal-only, high-sensitivity operational material:** Key Inventory (SSH key fingerprints/locations), Model Chain (LiteLLM virtual key hash, budget caps), Cost Control Rollout, app2 Caddyfile audit, Ops Portal / Backup-Restore internals (including the exposed historical credential noted in gap #2 above).
+- **Internal engineering process docs:** ITPP Standards (CI/templates), Scripts catalog, HomeLab.
+- **Product/dev documentation for ITPP's own ventures:** VerdictTank, TransitPin, FleetTracker360, Apex Track, BoxPilot, OSINT Tool, LaunchCheck, Shark Game - these describe products being built for eventual external users or customers, not ITPP's own infrastructure.
+- **Client-facing:** none currently exists as a distinct, intentionally-scoped audience. Nothing in the current tree is written for or safe to hand to an actual paying customer of VerdictTank, FleetTracker360, or TransitPin - yet the auth gate's allowlist (`g@germainebrown.com`, `info@itpropartner.com`) suggests the site is intended for internal ITPP staff only, which conflicts with product docs plausibly needing outside readers eventually (contractors, investors, or customers).
+
+**Assessment:** the site currently serves one audience (internal ITPP operators) under one login, with no mechanism to safely expose a subset of content (e.g., product architecture for a contractor, or end-user help docs for a VerdictTank customer) without also exposing Key Inventory, Model Chain, and Ops Portal internals. This is a structural risk, not just an organizational tidiness issue: the moment anyone outside the current two-person allowlist needs access to any single product doc, the only lever available is "grant them access to everything," including SSH key inventories and credential-adjacent operational docs.
+
+**Recommendation:** split into three distinct trust zones (detailed in the reorg tree below) before granting any doc access to anyone outside the current internal allowlist.
+
+---
+
+## 5. Proposed Documentation Reorg Tree (Deliverable A.6)
+
+Plain-English rationale first, then the tree.
+
+**Rationale:**
+1. **Separate the credential-adjacent operational core from everything else.** Key Inventory, Model Chain, Cost Control, Ops Portal/Backup-Restore internals, and the app2 Caddyfile audit should live in a zone that is never extended to anyone outside the current internal allowlist, regardless of what else changes.
+2. **Separate product/dev docs from internal ops docs**, because product docs (VerdictTank, TransitPin, etc.) have a plausible future need for a wider audience (contractors, eventual customers) that internal ops docs never will.
+3. **Reserve an explicit, currently-empty client-facing zone** rather than pretending the current site serves that purpose. If ITPP wants client-facing documentation (e.g., a VerdictTank user guide), it should be a deliberately separate publish target (different subdomain or path with its own, lighter-weight auth model), not a corner of the internal-only site.
+4. **Retire dead stubs rather than let them accumulate.** Apex Track, BoxPilot, and (arguably) Shark Game and OSINT Tool have single-entry, 5-week-stale changelogs with no real content beyond "created project repository." Continuing to display them as live nav entries between actively-maintained sections misrepresents the site's currency.
+5. **Fix the self-contradiction between `projects-master-readme` and the sections it describes** by making project status ("PLANNED" / "IN DEVELOPMENT" / "LIVE") a single generated field rather than manually duplicated free text in two places.
+
+```
+docs.itpropartner.com/
+├── internal/ [existing allowlist gate stays here, unchanged scope]
+│ ├── infrastructure/ (from itpp-infrastructure, minus product-adjacent items below)
+│ │ ├── overview
+│ │ ├── key-inventory <- flag: rewrite "sanitized" claim or actually remediate first (gap #1)
+│ │ ├── model-chain
+│ │ ├── cost-control-rollout-2026-07-24
+│ │ ├── app2-caddyfile-audit-2026-07-21
+│ │ ├── super-search-cf-bypass
+│ │ ├── ops-portal (+ architecture)
+│ │ ├── backup-restore (+ architecture)
+│ │ ├── docs-auth-gate
+│ │ ├── project-log
+│ │ └── audit/ <- NEW: link Phase One (and future Phase Two) findings for internal transparency
+│ ├── standards/ (itpp-standards, unchanged)
+│ ├── homelab/ (unchanged; flag as "out of Phase One audit scope" until a homelab-specific audit exists)
+│ └── scripts/ (unchanged)
+│
+├── products/ [same or a separate, slightly wider internal+contractor gate]
+│ ├── verdicttank/ (+ architecture)
+│ ├── transitpin/
+│ ├── fleettracker360/
+│ ├── apex-track/ <- ARCHIVE unless real content is added; currently a dead stub
+│ ├── boxpilot/ <- ARCHIVE unless real content is added; currently a dead stub
+│ ├── osint-tool/
+│ ├── launchcheck/ (+ competitive-analysis, business-proposal)
+│ └── shark-game/ <- review: 5-week-stale, confirm still active before keeping live
+│
+├── client/ [NEW, does not exist today - separate publish target/subdomain,
+│ its own lightweight auth or fully public, populated only with
+│ content explicitly written for external readers]
+│ └── (empty until ITPP decides which product needs a customer-facing doc set)
+│
+└── legal/ [NEW, single location - currently a dangling empty page under
+ itpp-infrastructure; either populate with real legal/compliance
+ content or remove the nav entry entirely]
+```
+
+**What moves where:**
+- `itpp-infrastructure/*` -> `internal/infrastructure/*` (unchanged content, new path only).
+- `itpp-standards`, `homelab`, `scripts` -> `internal/*` (unchanged).
+- `verdicttank`, `transitpin`, `fleettracker360`, `apex-track`, `boxpilot`, `osint-tool`, `launchcheck`, `shark-game` -> `products/*` (unchanged content, new path, plus an explicit staleness review for the four dead/near-dead stubs).
+- `itpp-infrastructure/legal` -> either populated and moved to a top-level `legal/` or removed.
+- `itpp-infrastructure/project-log` and `projects-master-readme` -> reconcile into a single generated project-status view under `internal/infrastructure/project-log`, sourced from each project's own README/CHANGELOG status field rather than hand-maintained twice.
+- New `internal/infrastructure/audit/` -> add Phase One (and future Phase Two) findings summaries so the org has one place that tracks "what the docs say" next to "what the audits found."
+- `client/` -> created empty; populated only when/if ITPP commits to publishing customer-facing docs for a specific product, with its own auth model decided at that time.
+
+**What gets archived (not deleted, moved to an `archive/` prefix or removed from nav):**
+- `apex-track` and `boxpilot`: both are single-entry "created project repository" stubs, 5 weeks stale, indistinguishable from placeholders. Either bring them current or pull them from the live nav so the site does not imply active documentation where none exists.
+- The stale `/home/ippadmin/htdocs/docs.itpropartner.com` duplicate (not a doc section, but the entire stale build) should be deleted per section 3.
+
+---
+
+## Summary of Evidence Sources
+
+- Live site content: extracted via SSH from `/home/docs/htdocs/docs.itpropartner.com` on app3 (152.53.241.111), CHANGELOG and index pages per section.
+- Split-brain comparison: `find`/`diff`/`stat` against both `/home/docs/htdocs/...` and `/home/ippadmin/htdocs/...` on app3.
+- Cross-reference findings: neteng-a.md, neteng-b.md, sec-b.md, sys-a.md, sys-b.md, sys-c.md (all read in full for this audit).
+- No configuration, DNS, nginx, or docs content was modified. All recommendations are deferred to Phase Two per the audit brief's absolute rules.

audit/phase-one/findings/git-a.md

@@ -0,0 +1,238 @@
+# Git-A Findings: Gitea Repository Audit (git.itpropartner.com)
+
+**Auditor:** Git-A (claude-sonnet-5, subagent under Phase One ITPP Infrastructure Audit)
+**Scope:** Full repo inventory, local/remote cross-reference, classification, lineage, secrets-in-history, disposition recommendations.
+**Method:** READ-ONLY. All findings verified against the live Gitea API and local clone git history. Nothing was modified, deleted, renamed, force-pushed, or rotated. All remediation is written up below as a Phase Two recommendation.
+**Date:** 2026-08-13
+
+---
+
+## 1. Full Inventory
+
+Enumerated via `GET /api/v1/user/repos?token=...&limit=100`, paginated (2 pages, page 3 empty). **Total repos on Gitea: 56.** All owned by `ippadmin`. None are flagged `empty: true`.
+
+| Repo | Private | Size (KB) | Default Branch | Last Updated |
+|---|---|---|---|---|
+| apex-track | No | 28 | main | 2026-08-09 |
+| auth | Yes | 42 | main | 2026-08-08 |
+| backup-restore | Yes | 29 | main | 2026-08-08 |
+| boxpilot | No | 27 | main | 2026-08-09 |
+| cartmylist | No | 70 | main | 2026-08-05 |
+| competitive-landscape-research | No | 75 | main | 2026-08-10 |
+| content-creation-pipeline | Yes | 27 | main | 2026-08-08 |
+| digital-signage | No | 82 | main | 2026-08-08 |
+| disaster-recovery | No | 82 | main | 2026-08-08 |
+| dre | No | 117 | main | 2026-08-08 |
+| fleettracker360 | No | 26 | main | 2026-08-08 |
+| forefront-wireless-portal | No | 57 | main | 2026-08-08 |
+| furniture-pos | Yes | 56 | main | 2026-08-12 |
+| gift-a-roast | No | 28 | main | 2026-08-08 |
+| hermes-recovery | Yes | 455 | main | 2026-08-09 |
+| hermes-skills | No | 13807 | main | 2026-08-08 |
+| homelab | Yes | 42 | main | 2026-08-09 |
+| hudu | No | 32 | main | 2026-08-08 |
+| itpp-docs | No | 946 | main | 2026-08-10 |
+| itpp-infrastructure | No | 727 | **master** | 2026-08-12 |
+| itpp-standards | No | 28 | main | 2026-08-09 |
+| itpropartner-website | Yes | 36 | main | 2026-08-08 |
+| launchcheck | No | 59 | main | 2026-08-09 |
+| mcp-browser | No | 27 | master | 2026-08-08 |
+| mcp-email | No | 26 | master | 2026-08-08 |
+| mcp-filesystem | No | 26 | master | 2026-08-08 |
+| mcp-git | No | 26 | master | 2026-08-08 |
+| mcp-servers | No | 27 | main | 2026-08-08 |
+| model-fallback | Yes | 43 | main | 2026-08-08 |
+| mooresunnydaze | No | 107 | main | 2026-08-08 |
+| msp-forms | Yes | 44 | master | 2026-08-08 |
+| nvr-shield | No | 33 | main | 2026-08-08 |
+| ops-portal | Yes | 31 | main | 2026-08-08 |
+| ops-reports | Yes | 44 | main | 2026-08-12 |
+| org-audit | Yes | 472 | master | 2026-08-09 |
+| osint-tool | No | 37 | main | 2026-08-09 |
+| personal-assistant | Yes | 82 | main | 2026-08-08 |
+| pipeline | Yes | 41 | main | 2026-08-08 |
+| pry | Yes | 39 | master | 2026-08-08 |
+| research-search-mcp | Yes | 63 | main | 2026-08-08 |
+| scripts | Yes | 75 | main | 2026-08-08 |
+| seo-tool | No | 43 | master | 2026-08-10 |
+| shark-game | No | 30 | main | 2026-08-08 |
+| shonuff-caller | Yes | 31 | main | 2026-08-08 |
+| startup-studio | Yes | 43 | main | 2026-08-08 |
+| super-search | No | 39 | master | 2026-08-08 |
+| super-search-business | No | 57 | main | 2026-08-08 |
+| track-a-flock | Yes | 35 | main | 2026-08-08 |
+| transitpin | Yes | 129 | main | 2026-08-09 |
+| unifi | No | 27 | main | 2026-08-08 |
+| unms | No | 27 | main | 2026-08-08 |
+| venturebuilt | No | 47 | master | 2026-08-10 |
+| verdicttank | No | 145 | main | 2026-08-12 |
+| voice-agent | Yes | 32 | master | 2026-08-08 |
+| voipsimplicity | No | 68 | main | 2026-08-08 |
+| voipsimplicity-manual | Yes | 448 | main | 2026-08-08 |
+
+Note: `itpp-infra` (singular, no "structure") does **not** appear in this list. Confirmed separately below (Section 4, Lineage) that it has been deleted server-side since the last local clone was made.
+
+---
+
+## 2. Cross-Reference: Gitea vs Local Clones
+
+Local clones live under `/root/projects/`. 67 top-level directories exist there; 48 are real git repos (have a `.git` directory), 19 are plain project folders (docs, scratch dirs, or research artifacts with no version control).
+
+### 2a. Gitea repos with NO local clone (11 repos)
+
+These exist server-side but Git-A found no corresponding directory under `/root/projects/`. Cannot verify their content/purpose without a local clone; flagged as an access/visibility gap per Brief Rule 4.
+
+| Repo | Private | Size | Notes |
+|---|---|---|---|
+| itpp-docs | No | 946KB | Aggregated MkDocs site - likely owned by Docs-W's workstream |
+| mcp-browser | No | 27KB | MCP server source, `master` branch |
+| mcp-email | No | 26KB | MCP server source, `master` branch |
+| mcp-filesystem | No | 26KB | MCP server source, `master` branch |
+| mcp-git | No | 26KB | MCP server source, `master` branch |
+| msp-forms | Yes | 44KB | `master` branch |
+| pry | Yes | 39KB | "pry service", `master` branch, no description |
+| seo-tool | No | 43KB | ITPP SEO audit tool, `master` branch |
+| super-search | No | 39KB | MCP server source (distinct from `super-search-business`) |
+| venturebuilt | No | 47KB | AI business dev platform, `master` branch |
+| voice-agent | Yes | 32KB | `master` branch |
+
+**Why it matters:** Git-A cannot confirm what's in these 11 repos beyond Gitea's own metadata (name/description/size). If Sys-A/Sys-B or Sec-A need to verify what's actually deployed from these repos, they'll need a fresh clone - this audit's local-clone cross-reference has a blind spot here. All 11 also share the pattern of being either MCP micro-services or product-dev tools that may have been scaffolded once and forgotten (small size, single description, no CHANGELOG visible from metadata alone).
+
+### 2b. Local repos with NO Gitea remote match ("local-only", no off-server backup)
+
+None found with truly missing remotes - `deep-scan.py` reported **0 local-only repos** (every `.git`-bearing directory under `/root/projects/` has a configured `origin`). However, two categories of concern:
+
+- **`itpp-infra`** - has an `origin` pointing at `git.itpropartner.com/ippadmin/itpp-infra.git`, but the repo **no longer exists on the Gitea server** (`git ls-remote` returns `remote: Repository not found`, HTTP 404 on the repos API). This is an **orphaned local clone of a deleted remote** - effectively local-only now, with 3 commits of history that exist nowhere else. See Section 4.
+- **2 repos use GitHub, not Gitea, as their remote**: `msp-claude-skills` (github.com/RTFM-IT-Services-LLC) and `viral-content-system` (github.com/swaroop2004/Proven-Viral-Content-System - a fork of an external template, not an ITPP-authored repo). These are outside Gitea's scope entirely; flagged for awareness, not an audit failure.
+
+### 2c. Non-repo project folders under /root/projects/ (19 total, no git at all)
+
+`asher-and-rye-m365-consolidation`, `competitive-analysis`, `diglocate`, `forefront-broadband-map`, `giftaroast`, `kids-school-calendar`, `mautic-multitenant`, `mcp-planning`, `mcp-registration`, `mikrotik-monitoring`, `obsidian-selfhost`, `paperless-ngx`, `personal-shopping-assistant`, `portal-design-system`, `rfptank`, `school-calendar-2026-2027`, `twilio-10dlc`, `udm-tailscale`, `village-express`.
+
+These are working directories, research scratch space, or planning docs - not under version control at all, so they carry **zero off-server backup**. Not directly this audit's remit (no Git history to inventory), but worth flagging to Sys-C/the conductor: any of these with real deliverable content (`giftaroast` has working Python files, e.g.) has no backup whatsoever, git or otherwise, beyond whatever the general server backup captures.
+
+---
+
+## 3. Classification
+
+Classified by **last-commit recency** and **inferred purpose/domain**. "Active" = commit within last 5 days as of audit date (2026-08-13) AND has ongoing purpose; "Stale" = >5 days idle but still relevant; "Superseded" = replaced by a newer/renamed repo; "Orphaned" = remote gone, dangling reference, or abandoned scaffold.
+
+### Active (commits within last ~5 days, live purpose)
+
+itpp-infrastructure (age 0d, 90 commits - actively growing), furniture-pos (0d), ops-reports (0d), verdicttank (0d), competitive-landscape-research (2d), transitpin (3d), homelab (3d), itpp-standards (3d), org-audit (3d, private audit-artifact repo - still being appended to for this very engagement).
+
+**Domain split:** itpp-infrastructure / homelab / itpp-standards / org-audit = internal infra. furniture-pos / transitpin / forefront-wireless-portal = client-facing. verdicttank / competitive-landscape-research / launchcheck = product-dev micro-SaaS.
+
+### Stale (idle 4+ days, no evidence of abandonment - most repos)
+
+The overwhelming majority of the 46 Gitea-backed local clones sit at **exactly 2026-08-08, ~13:06-13:08** - a single mass-scaffolding event, not organic development. This includes: apex-track, auth, backup-restore, boxpilot, cartmylist, content-creation-pipeline, digital-signage, dre, fleettracker360, forefront-wireless-portal, gift-a-roast, hermes-recovery, hermes-skills, hudu, itpp-infra, launchcheck, mcp-servers, model-fallback, mooresunnydaze, msp-claude-skills, nvr-shield, ops-portal, osint-tool, personal-assistant, pipeline, research-search-mcp, scripts, shark-game, shonuff-caller, startup-studio, super-search-business, track-a-flock, unifi, unms, viral-content-system, voipsimplicity, voipsimplicity-manual.
+
+This pattern (dozens of repos all touched in the same 2-minute window on Aug 8) strongly suggests a batch `.gitignore`/scaffolding pass (consistent with the git-audit skill's own Step 9 "batch remediation" - likely a prior audit's cleanup run) rather than genuine feature work. **Domain split:** roughly half internal-infra/ops tooling (auth, backup-restore, hudu, unifi, unms, model-fallback, ops-portal, personal-assistant, pipeline, shonuff-caller, mcp-servers), half product-dev micro-SaaS prototypes (apex-track, boxpilot, dre, fleettracker360, gift-a-roast, launchcheck, mooresunnydaze, nvr-shield, osint-tool, shark-game, startup-studio, super-search-business, track-a-flock, voipsimplicity, voipsimplicity-manual) plus a couple of client-facing scaffolds (digital-signage, forefront-wireless-portal, cartmylist).
+
+### Superseded / Duplicate
+
+- **itpp-infra** superseded by **itpp-infrastructure** (see Lineage below - do not confuse them per skill pitfall).
+- **cartmylist-repo** (local dir name) is the same project as Gitea's **cartmylist** - naming drift, not two repos (confirmed: same remote URL). Not a true duplicate but flagged for local-directory-name hygiene.
+
+### Orphaned
+
+- **itpp-infra** - remote deleted server-side; local clone is now a dangling reference to nothing. 3 commits of unique history exist only in this local clone.
+
+---
+
+## 4. Lineage: Duplicates, Forks, Naming Collisions
+
+| Pair | Status | Verdict |
+|---|---|---|
+| **itpp-infra** vs **itpp-infrastructure** | itpp-infra: local clone exists (3 commits, "Initial commit -- audit Jul 24 2026" as its most substantive commit), but the **Gitea remote no longer exists** - confirmed via `git ls-remote` (`Repository not found`) and direct API call (404). itpp-infrastructure: 90 commits, actively updated as of today (2026-08-13), 727KB, is the working audit/documentation repo, default branch on Gitea is `master` while local clone tracks `main` (branch mismatch - see below). | **itpp-infra is dead/orphaned.** itpp-infrastructure is the live, correct repo. Do not confuse the two per skill's known pitfall - confirmed still true. |
+| **cartmylist** (Gitea) vs **cartmylist-repo** (local dir name) | Same repo, same remote URL (`git.itpropartner.com/ippadmin/cartmylist.git`), local clone directory is just named differently (`cartmylist-repo`). Not a real duplication - it's local naming drift. | No consolidation needed; rename local directory for clarity in Phase Two, non-urgent. |
+| **gift-a-roast** (Gitea+git) vs **giftaroast** (local, no git) | Two different local directories. `gift-a-roast` is the real git-tracked repo (28KB, matches Gitea). `giftaroast` is an *untracked scratch directory* containing loose Python fix-scripts (`fix_auth.py`, `fix_dict.py`, `fix_final.py`) and an `index.html` - looks like ad-hoc debugging output that predates or parallels the real repo, never committed anywhere. | `giftaroast` (no-git) should be reviewed and either merged into `gift-a-roast`'s history or deleted as scratch work - currently has zero backup. |
+| **super-search** vs **super-search-business** | Two distinct Gitea repos. `super-search` = "super-search MCP server source code" (39KB, `master` branch, no local clone found). `super-search-business` = "Super Search for Business - multi-tenant competitive intelligence & OSINT SaaS platform" (57KB, `main` branch, cloned locally). Different products (infra tool vs. product-dev SaaS), not a duplicate - naming is just confusingly similar. | Not a collision requiring merge; recommend renaming one for clarity (e.g. `super-search-mcp` vs `super-search-business`) in Phase Two docs pass. |
+| **mcp-servers** vs **mcp-browser / mcp-email / mcp-filesystem / mcp-git** | `mcp-servers` (cloned locally, "MCP servers for Open WebUI") appears to be a monorepo/aggregator. The four `mcp-*` singles (browser/email/filesystem/git) are NOT cloned locally, all sit on `master` branch (older default, out of step with the `main` convention used elsewhere), and are small (26-27KB each) with generic "<name> MCP server source code" descriptions. | Consolidation candidate: verify whether `mcp-servers` already contains these four as subdirectories (would need a fresh clone to confirm - Section 2a gap). If duplicated, the four standalone repos are consolidation-into-mcp-servers candidates. |
+| **itpp-docs** vs **itpp-standards** vs **itpp-infrastructure** | Three separate repos all touching "ITPP documentation": itpp-docs = "Aggregated MkDocs documentation site for all IT Pro Partner projects" (946KB, not cloned locally); itpp-standards = "ITPP documentation standards, templates, and CI workflows" (28KB, 1 commit only); itpp-infrastructure = the working infra/audit repo with actual runbooks (727KB, 90 commits). | Not true duplicates (each has distinct scope: standards/templates vs published docs site vs raw infra runbooks) but worth flagging to Docs-W for the Git/Docs reorg recommendation - three repos with "docs" in scope invites confusion about which is canonical for what. |
+
+**Branch mismatch flag (per skill's known pitfall, independently reverified today):** `itpp-infrastructure`'s local clone tracks `main` while Gitea's `default_branch` is `master`. This is not cosmetic - it means anyone cloning fresh from Gitea without specifying a branch lands on `master`, which may be stale relative to the `main`-tracking local clone that's actually being worked in daily. Confirmed both branches exist on the remote (`origin/master` and `origin/main` both resolve). **This should be fixed in Phase Two**: pick one branch, make it Gitea's default, delete the other.
+
+Six other repos still default to `master` on Gitea rather than the `main` convention used everywhere else: `mcp-browser`, `mcp-email`, `mcp-filesystem`, `mcp-git`, `msp-forms`, `pry`, `seo-tool`, `super-search`, `venturebuilt`, `voice-agent`, and locally `org-audit`. Consistent naming convention is a Phase Two hygiene item, not a security risk.
+
+---
+
+## 5. Secrets-in-History
+
+Scanned via the skill's regex pattern across full git history (`git log -p --all`) for all 46 Gitea-backed local clones, filtered against the false-positive list (`publicKeyToken`, doc-example/placeholder values, `?token=` in URLs). Manual review of every raw hit below; only REAL, exploitable values are recorded with `[REDACTED]` substituted for the actual secret.
+
+### CRITICAL
+
+**1. `scripts` repo - hardcoded Windows admin passwords in provisioning script, repo is PRIVATE but still exposed to anyone with repo access/token leak**
+- Repo: `scripts` (private=true)
+- Commits: `ec6e0e1b9894b9c7e16793aafacc4681f988bf86` ("Liberty: add ippadmin MSP backdoor account alongside liberty-admin customer admin"), `fef88f8d634957c538b3a800c9db1093bb9c3520` ("Refactor to two-script architecture..."), `2497f4c0c03b1435dcd9f59161a25d3b9383cc9f` ("Standard onboard v2...")
+- File: `dell-reimage-kit/standard-onboard.ps1`
+- Value: `[REDACTED]` - plaintext local-admin passwords for an "ippadmin MSP backdoor account" (used across all client onboards) and a "liberty-admin" customer admin account, embedded directly in a PowerShell provisioning script.
+- **Why it matters:** This is a credential used to provision a hidden admin account on every client machine imaged with this kit - a live, reusable master-key password baked into source control. Even though the repo is private, anyone with read access to Gitea (or a leaked API token, see finding 3 below) gets a password that likely still works on production client endpoints today.
+- **Disposition:** Rotate the password at the source (change it on any machine it was actually used on), then scrub history per the skill's Step 8 workflow, force-push, and verify.
+
+### HIGH
+
+**2. `itpp-infrastructure` repo - same passwords re-exposed, but this time in a PUBLIC repo**
+- Repo: `itpp-infrastructure` (private=**false**)
+- Commits: `a269a17b1f40460b0ef96ca234233167564b404d` ("git-audit: 42-repo hygiene audit Aug 8..."), `de0190283b6c89b399a9089dbaa1865e0b4cb337` ("docs: Git structure audit -- 40 Gitea repos...")
+- Files: `docs/git-audit-2026-08-08.md`, `docs/git-audit-2026-08-07.md`
+- Value: `[REDACTED]` - the same "ippadmin"/"liberty-admin" MSP backdoor passwords from Finding 1, quoted verbatim inside a **prior audit report** that a previous Git-A wrote and committed to this repo as evidence of the finding.
+- **Why it matters:** This is worse than Finding 1 in one dimension: `itpp-infrastructure` is a **public** repo. A previous audit correctly identified the `scripts` repo credential leak, but then re-leaked the same credential by quoting it verbatim into a report and pushing that report to a public repo. Anyone on the internet who finds `git.itpropartner.com/ippadmin/itpp-infrastructure` can clone it and `git log -p` these two commits to get the same admin password. This is a textbook example of why the Brief's Rule 5 ("emit `[REDACTED]` everywhere, including paths that contain secrets") exists - Git-A is following it here; a prior pass did not.
+- **Disposition:** Rotate the same credential (covers both findings 1 and 2 at once), then scrub history from `itpp-infrastructure` specifically since it's public-facing, force-push, verify with a fresh clone. Treat as higher urgency than Finding 1 purely because of public exposure, even though it's the same underlying secret.
+
+**3. `hermes-recovery` repo - live database password + the actual Gitea API token used for this very audit, committed to history**
+- Repo: `hermes-recovery` (private=true)
+- Commit: `ae056eaf83b3d9ed273b68bce635aef8fdd1d665` ("Initial resurrection kit - 81 scripts, 62 references, configs, systemd units, crons, Docker compose files, Caddy config, master README")
+- Files: `scripts/apex-mail-watchdog.py` (MySQL password for user `apextrackexperience_1781549652`), `configs/config.yaml` (a Gitea API token value)
+- Value: `[REDACTED]` (both)
+- **Why it matters:** Two separate live secrets in one commit: a MySQL credential for what looks like a production WordPress/mail-watchdog database, and a Gitea API token embedded directly in a backup config file. The repo is private, which limits blast radius to whoever has Gitea access - but "resurrection kit" repos exist specifically to be handed to whoever needs to rebuild the server, meaning this file is likely to be read, copied, and pasted elsewhere by design. That's exactly how a scoped secret becomes an unscoped one.
+- **Disposition:** Rotate both the DB password and the Gitea token, scrub history, verify. Because this token may be the *same* token used to authenticate this very audit run (structurally identical format), Phase Two should explicitly confirm whether it's live before assuming it's already been rotated.
+
+### MEDIUM (documentation artifacts, not live secrets - recorded for completeness, not actionable as leaks)
+
+- `hermes-skills` repo: 15 regex hits, all reviewed - every one is a documentation example (`SOME_API_KEY: "value"`, `secret: "generate-a-strong-secret-here"`, shell snippets showing *how* to extract a key from a config file rather than an actual key value). **No real secret.** Two large files flagged separately (`skills/.curator_backups/.../skills.tar.gz` at 2.7MB and `skills/.hub/index-cache/hermes-index.json` at 38MB) - per the skill's known pitfall, this is a legitimate mirror repo and these are cache artifacts that should be gitignored, not evidence of misuse.
+- `org-audit` repo: 3 regex hits, all reviewed - `SMTP_PASSWORD: '<REDACTED>'` is literally the placeholder string `<REDACTED>` already in the source (a prior audit's own sanitized report), `ADMIN_TOKEN` hit is a comment noting no token is set, `temporary-password-here` is a literal placeholder. **No real secret** - this repo is itself sanitized audit output and correctly follows the redaction convention.
+
+### Summary table
+
+| Severity | Repo | Public/Private | Real secret? | Commits |
+|---|---|---|---|---|
+| CRITICAL | scripts | Private | Yes - admin passwords | 3 commits |
+| HIGH | itpp-infrastructure | **Public** | Yes - same admin passwords, re-leaked | 2 commits |
+| HIGH | hermes-recovery | Private | Yes - DB password + Gitea API token | 1 commit |
+| Info-only | hermes-skills | Public | No - doc examples only | n/a |
+| Info-only | org-audit | Private | No - already redacted in source | n/a |
+
+**No live secrets found in any of the 46 scanned repos beyond the three real findings above.** The 11 repos in Section 2a (no local clone) could not be scanned and remain an access/visibility gap - flag for Phase Two follow-up if their content needs verification.
+
+---
+
+## 6. Disposition Recommendations
+
+| Repo | Classification | Recommendation | Rationale |
+|---|---|---|---|
+| itpp-infra | Orphaned (remote deleted) | **Archive/delete local clone** | Remote no longer exists; 3 commits of unique local history should be reviewed once for anything not already in itpp-infrastructure, then the local clone can be safely removed. Not backed up anywhere else. |
+| itpp-infrastructure | Active | **Keep-active; fix branch default** | Live, growing, canonical infra/audit repo. Fix the `main`/`master` default-branch mismatch first (Phase Two). |
+| cartmylist-repo (local dir) | Active (as `cartmylist` on Gitea) | **Keep-active; rename local dir** | Same repo as Gitea's `cartmylist`, just named differently locally. Cosmetic fix only. |
+| giftaroast (local, no git) | Untracked scratch work | **Merge-into gift-a-roast or delete** | Loose debugging scripts with no version control and no backup. If content is still useful, commit into `gift-a-roast`'s history; otherwise delete as scratch. |
+| mcp-browser / mcp-email / mcp-filesystem / mcp-git | Stale, not locally verifiable | **Merge-into mcp-servers (pending verification)** | Small, single-purpose repos with generic descriptions on the outdated `master` branch. Likely duplicated inside the `mcp-servers` monorepo - needs a fresh clone to confirm before consolidating. |
+| super-search | Stale, not locally verifiable | **Keep-active but rename for clarity** | Distinct product from `super-search-business` (infra MCP tool vs. SaaS product) - not a true duplicate, but the near-identical name is a standing source of confusion. |
+| The ~35 "Aug 8, 13:06-13:08" batch-scaffolded repos (apex-track, boxpilot, dre, fleettracker360, gift-a-roast, launchcheck, mooresunnydaze, nvr-shield, osint-tool, shark-game, startup-studio, super-search-business, track-a-flock, voipsimplicity, voipsimplicity-manual, and the internal-infra set: auth, backup-restore, hudu, unifi, unms, model-fallback, ops-portal, personal-assistant, pipeline, shonuff-caller, mcp-servers) | Stale | **Case-by-case review, default keep-active** | These are 4+ days idle but represent real, distinct project scaffolds (verified non-trivial directory structure in every sampled case), not abandoned stubs. No action needed unless the business decides a given micro-SaaS idea is dead - that's a product decision, not a Git hygiene one. Flagging as stale is informational, not a call to archive. |
+| itpp-docs, itpp-standards | Active/Stale, overlapping scope with itpp-infrastructure | **Consolidation review with Docs-W** | Three "ITPP docs" repos with overlapping but distinct scope (standards/templates vs. published site vs. raw runbooks) invite confusion. Recommend Docs-W's Git/Docs reorg proposal explicitly define which repo owns what, rather than merging outright. |
+| scripts | Active/stale, CRITICAL secret | **Keep-active, rotate + scrub immediately in Phase Two** | Real provisioning tooling still in use; the credential leak is the problem, not the repo's existence. |
+| hermes-recovery | Stale, HIGH secret | **Keep-active, rotate + scrub immediately in Phase Two** | Legitimate disaster-recovery kit; same treatment - fix the leak, keep the repo. |
+| org-audit | Active (still being written to for this engagement) | **Keep-active** | This is this audit's own output repo. No issue found. |
+
+---
+
+## 7. Access/Visibility Gaps (Brief Rule 4)
+
+- **11 Gitea repos have no local clone** (Section 2a) - their contents could not be scanned for secrets or verified against description metadata. Flag for Phase Two: clone and re-scan `mcp-browser`, `mcp-email`, `mcp-filesystem`, `mcp-git`, `msp-forms`, `pry`, `seo-tool`, `super-search`, `venturebuilt`, `voice-agent`, `itpp-docs`.
+- **itpp-infra's Gitea history is gone.** If that repo held anything not duplicated in `itpp-infrastructure`, it is now unrecoverable from Gitea - only the local clone's 3 commits remain. Recommend preserving that local clone as-is (do not delete) until someone confirms nothing of value is unique to it.
+
+---
+
+## 8. Estimate vs Actual
+
+This subagent run is a re-run after a prior Git-A instance hit a LiteLLM budget 429 before writing output. No cost data is available to Git-A directly; defer to the conductor's LiteLLM SpendLogs reconciliation for this run's actual token/dollar cost.

audit/phase-one/findings/indep-review.md

@@ -0,0 +1,71 @@
+# Independent Severity Review (Indep) - ITPP Phase One Audit
+
+**Reviewer:** Indep (claude-sonnet-5), independent QA pass
+**Scope:** Re-score every Critical/High finding in the nine findings files and cross-check against report.md Sections 3 and 7. Read-only. No infrastructure was touched to produce this review; all conclusions are drawn from the raw findings files already on disk.
+**Method:** Read all 9 findings files (neteng-a, neteng-b, sec-a, sec-b, sys-a, sys-b, sys-c, git-a, docs-w) and report.md in full, then independently judged each Critical/High rating against its own stated evidence, without deferring to the conductor's synthesis.
+
+---
+
+## 1. Severity re-score table
+
+| ID | Finding (short) | Conductor rating | My rating | Verdict |
+|---|---|---|---|---|
+| D1 / Sys-B H1 | Standby watchdog pings "wrong" IP 152.53.192.33 | Resolved as false positive (conductor) | False positive | FALSE-POSITIVE (agree) |
+| D2 / Sys-B C1 | Gitea/Hudu/UNMS/UniFi "no effective backup" | Downgraded to High (conductor) | High for Hudu/UNMS/UniFi; not-a-finding for Gitea | DOWNGRADE-to-High, with the added correction that Gitea should be dropped from this finding entirely |
+| Report C1 (NetEng-A) | Docker/UFW bypass, ~20 consoles public | Critical | Critical | AGREE |
+| Report C2 (NetEng-B NETB-1) | No network segmentation anywhere | Critical | Critical | AGREE |
+| Report C3 (NetEng-A APP1-1 + Sec-B-01) | Wazuh public + zero enrolled agents | Critical | Critical | AGREE |
+| Report C4 (Sec-A-01, Sys-A F-2, Sys-B C2/C3, Docs-W #1) | Plaintext credentials estate-wide | Critical | Critical | AGREE, but see "under-weighted" note on Git-A Finding 1 below, which is thin in this writeup |
+| Report C5 (Sys-A F-1) | LiteLLM Postgres never backed up | Critical | Critical | AGREE |
+| Report C6 (NetEng-B NETB-3) | app3 single shared MySQL, ~24 sites | Critical | Critical | AGREE |
+| Report C7 (Sec-B-02 + NetEng-A CORE-1) | Grafana default admin/admin, public, no MFA | Critical | Critical | AGREE |
+| Report C8 (Sys-B C4) | wphost02 backup gap, 6 of 8 DBs unprotected | Critical | Critical | AGREE |
+| Report C9 (Sys-B C5 / Sys-C SYSC-01) | Warm standby not data-ready | Critical | Critical | AGREE |
+| Sec-A-02 | Single SSH key = root on 5 of 6 hosts, passwordless sudo on top | Critical in sec-a.md, silently downgraded to "High findings (representative)" in report Section 3.2, no Section 7 entry | Critical | UPGRADE-to-Critical (restore original rating; also flag the undocumented downgrade as a process gap) |
+| NetEng-B NETB-6 | Same underlying fact as Sec-A-02, stated as High in neteng-b.md itself | High | Critical | UPGRADE-to-Critical (same reasoning as Sec-A-02; this is one finding described twice, not two findings) |
+| Sec-B-03 | Technitium DNS `DNS_SERVER_ADMIN_PASSWORD=changeme` in container env | Critical in sec-b.md, silently shown as High in report Section 3.2, no Section 7 entry | High, with an explicit evidence caveat | DOWNGRADE-to-High (agree with the report's de facto number, disagree with doing it silently) |
+| Sys-C SYSC-02 | Duplicate/conflicting auth-api-backup cron jobs | Critical in sys-c.md; not mentioned anywhere in report Section 3 | Medium/High | DOWNGRADE-to-Medium-or-High, and separately flag as omitted from the consolidated report |
+| Sys-C SYSC-04 | WISP tower router (DR-017) has zero backup coverage at all | High in sys-c.md; not mentioned anywhere in report Section 3 | High | AGREE with sys-c's rating, but flag as MISSED from the consolidated report |
+| Git-A Finding 1 | `scripts` repo: hardcoded MSP-backdoor admin password reused across client onboards | Critical in git-a.md; report's C4 write-up only names the downstream public re-leak (Finding 2/D3), not this original Critical | Critical | AGREE with git-a's rating; flag that report C4's evidence bullets omit this specific item and should name it explicitly, since rotating it is required independent of the D3 policy call on the public repo |
+| Git-A Finding 2 | Same password re-leaked inside a PUBLIC repo (`itpp-infrastructure`) | High in git-a.md; treated as Critical-tier in report's C4/D3 framing | Critical | UPGRADE-to-Critical (agree with the report's implicit escalation over git-a's own High rating; public exposure of a live, reusable credential is worse than the private-repo case, and the severity legend the report itself uses supports Critical here) |
+| Git-A Finding 3 | `hermes-recovery` (private): live MySQL password + a live Gitea API token in history | High | High | AGREE |
+
+---
+
+## 2. False positives
+
+1. **Sys-B H1 (confirmed false positive - this is D1, see Section 4).** The claim that the standby watchdog targets the "wrong IP" is wrong. 152.53.192.33 is Core's real public IP per sys-a.md's own host profile table (Core Public IP row) and per report Section 2.1. 152.53.36.131 is app1, not Core. Sys-B conflated the two hosts. The watchdog is correctly configured.
+
+2. **Sys-B C1, as applied to Gitea specifically (this is part of D2, see Section 4).** Sys-B's claim of "no effective backup, a loss would be unrecoverable" for Gitea is contradicted by sys-c.md's live evidence: Gitea's backup was restore-tested PASS on 2026-08-10 (117 DB tables, 52 repos, 3 sampled repos restored with valid git history). "Unrecoverable" is factually wrong for Gitea. This part of C1 should be dropped, not just downgraded.
+
+No other Critical/High finding in the nine files was found to be factually wrong on re-read. The rest of C1 (Hudu/UNMS/UniFi backups being untested, see below) is a real gap, just not the "Critical, unrecoverable" framing Sys-B originally gave it.
+
+---
+
+## 3. Under-weighted or missed
+
+1. **Sec-A-02 / NetEng-B NETB-6 (single SSH key = root on 5 of 6 hosts).** Sec-A rated this Critical in its own file. Report Section 3.2 lists the same fact under "High findings (representative)" with no corresponding Section 7 disagreement entry explaining the downgrade. Using the report's own severity legend ("Critical = ... single-compromise = estate-wide blast"), a single key that unlocks passwordless root on 5 of 6 servers, with no MFA and no network segmentation to contain it, meets that bar. I recommend restoring this to Critical. Separately, the fact that it was downgraded without being logged as a disagreement (the way D1/D2/D3 were) is itself a process gap worth naming to Germaine: any time the conductor changes a source auditor's severity, it should show up in Section 7, even if the conductor believes the change is obviously correct.
+
+2. **Sys-C SYSC-04 (WISP tower router, zero backup coverage, DR-017 still open).** Rated High in sys-c.md with clear evidence (`s3://mikrotik-ccr-backups/wisp-backups/configs/tower*` returns zero objects, versus 30+ dailies for the home gateway at the same prefix pattern). This does not appear anywhere in report.md Section 3 (Critical or High), and is not folded into any of the C1-C9 themes since it is a standalone network-device gap, not a Docker/backup-script issue. This is a genuinely missed High finding: an operational device with a total absence of configuration backup, not just an untested one.
+
+3. **Git-A Finding 1 (`scripts` repo, hardcoded MSP-backdoor admin password used across client onboards).** Rated Critical in git-a.md, correctly. Report's C4 write-up (the Critical bucket for plaintext credentials) lists key-inventory.md copies, app1-bu's `.env`, systemd units, app3's MySQL password, and the itpp-infrastructure public re-leak, but never names this specific finding, the one that is arguably the most consequential of the group because it is a live password reused across production client machines, not just infrastructure secrets. It is mentioned only indirectly through D3 (which covers the re-leak, not the original). Recommend the report name Finding 1 explicitly in C4's evidence list.
+
+4. **Sys-C SYSC-02 (duplicate/conflicting auth-api-backup cron jobs).** Rated Critical in sys-c.md. On re-read, I think this is overstated: the working 03:15 job succeeds every night, and the second 04:35 job is a leftover that fails visibly. The real risk here is alert fatigue (a failing job that nobody investigates because "the cron always shows an error") rather than a live data-loss condition today. I would score this Medium, with a note that it could become a real gap if the good job silently breaks later. Separately, whatever its severity, it is not mentioned anywhere in report.md Section 3 and should be, since it currently reads as fully resolved (it is not).
+
+5. **Sec-B-03 (Technitium DNS `DNS_SERVER_ADMIN_PASSWORD=changeme`).** Rated Critical in sec-b.md. I think Critical overstates the confidence level here. Technitium (like many similar tools) typically only applies an admin-password environment variable on first bootstrap of its config; once a config already exists, subsequent container restarts do not necessarily re-apply that env var to the live credential. Sec-A's own file explicitly says it "could not confirm the live in-app credential value without an authenticated read." Sec-B's own rationale acknowledges this too ("Even if the operational credential has since been changed inside the app's own database..."). Given that acknowledged uncertainty, I would score this High rather than Critical: the finding (a default-credential string persisting in a live container env, on the estate's authoritative DNS) is a legitimate and important hardening signal regardless of whether it is literally the current password, but "Critical" implies a confirmed, exploitable credential, which this audit did not verify. The report's own Section 3.2 already lists this as High, so the net number matches what I'd recommend, but again, that downgrade from sec-b.md's own Critical rating was made silently, with no Section 7 entry.
+
+---
+
+## 4. Verdict on D1, D2, and D3
+
+**D1 (Sys-B H1, watchdog "wrong IP"): I agree with the conductor's resolution.** Core's public IP is confirmed as 152.53.192.33 in sys-a.md's host profile table and in report.md's Discovery Summary (Section 2.1). 152.53.36.131 belongs to app1. Sys-B's H1 conflated the two hosts and its underlying claim is false. This is a clean false positive, not a judgment call. No action needed beyond correcting Sys-B's file for the record.
+
+**D2 (Sys-B C1, Gitea/Hudu/UNMS/UniFi "no effective backup"): I agree with the direction of the conductor's resolution (downgrade), and I'd go slightly further on the details.** Sys-C's live S3 evidence shows all four services have current, on-schedule backups running through a different mechanism than the one Sys-B checked (Core-side scripts scheduled via Hermes's own cron system, not the app2-local scripts Sys-B examined, which genuinely are missing). That distinction matters: Sys-B's observation that the specific scripts referenced in app2's own `/root/backup.sh` do not exist is accurate and worth keeping as a hygiene finding (a redundant, broken, misleading logging path), but the conclusion that these four services have "no effective backup" and "a loss would be unrecoverable" is not supported by the live evidence. For Gitea specifically, there is a passing restore test, so I would remove it from this finding entirely rather than just downgrading its severity. For Hudu, UNMS, and UniFi, the accurate framing is "backups exist and are current, but have never been restore-tested," which is a real gap, appropriately High, not Critical. This also overlaps with Sys-C's own broader Critical finding (SYSC-03: 94%+ of all backup targets estate-wide have never been restore-tested), so Hudu/UNMS/UniFi's specific gap is really a subset of an already-Critical estate-wide pattern rather than its own independent Critical.
+
+**D3 (Git-A public repo credential re-leak, Germaine deferred remediation): I do not have grounds to disagree with the underlying finding, and deferral is Germaine's call to make, not mine to override.** The finding itself is factually solid: git-a.md independently confirmed the same admin password sits in git history in a public repo, verified against live Gitea API data. Where I'd add value here is on severity, not on the remediation decision: git-a.md itself rated this specific finding (Finding 2) as High, but the report's consolidated C4 treats it as Critical-tier alongside the other plaintext-credential findings. I agree with the report's implicit escalation, a live, reusable credential sitting in a searchable public repository is a worse exposure than the same secret in a private repo, so Critical is the more defensible rating even though the source auditor called it High. Germaine's decision to leave the repo alone for now is a risk-acceptance call made with full knowledge of the finding; I have no evidence that the decision was made on a mistaken understanding of severity, so I am not overriding it, I am only flagging that the underlying risk is live and, if anything, slightly under-stated by git-a.md's own severity label.
+
+---
+
+## 5. Overall confidence statement
+
+Confidence in this review is high for the two flagged disagreements (D1 is unambiguous, D2 is well-supported by Sys-C's independent live S3 check) and reasonably high for the severity re-scores involving the shared SSH key and the Technitium default-credential finding, since those turn on the report's own stated severity legend and on an explicit evidence gap the source auditors themselves called out, not on speculation. Confidence is lower, and explicitly flagged as such, on SYSC-02's exact severity (Medium vs High is a closer call than Critical vs Medium) and on whether Sec-B-03's live Technitium credential is actually still the default, since neither this review nor any of the nine original findings files could confirm the live value without an authenticated read, which was correctly out of scope for a read-only audit. Where evidence was insufficient to fully confirm or refute a claim, I have said so explicitly rather than guessing, consistent with the audit's own read-only, no-assumption rules. I found no evidence of systematic severity inflation or deflation across the nine files; the two confirmed issues (D1, D2) and the additional items surfaced here are individual scoring errors and one process gap (severity downgrades happening without a corresponding Section 7 entry), not a pattern that should cast doubt on the audit's other 50+ Critical and High findings, which were consistently well-evidenced with specific file paths, command output, or cross-referenced live checks.

audit/phase-one/findings/neteng-a.md

@@ -0,0 +1,268 @@
+# NetEng-A - Network Exposure Inventory (Phase One, Read-Only)
+
+**Auditor:** NetEng-A (network enumeration)
+**Date:** 2026-08-13
+**Scope:** Firewalls, open/listening ports, public DNS, VPN/private paths, and reverse-proxy/ingress surface across the ITPP estate.
+**Method:** Read-only discovery only (`ss -tulpn`, `ip`, `ufw status`, `iptables -L/-t nat`, `dig`, `docker ps`, `docker inspect`, `cat` of config files). No configuration was modified on any host.
+
+**Severity legend:** Critical = publicly reachable management/security console or control-plane with broad impact. High = significant unintended public exposure or broken DNS control. Medium = defense-in-depth gap or weak configuration. Low = hygiene/minor.
+
+---
+
+## 1. Executive summary
+
+- **Hosts reachable:** 6 of 6 servers in scope (Core, app1, app2, app3, app1-bu, wphost02) - all accepted the `itpp-infra` SSH key.
+- **Total open listening sockets (bound to non-loopback/public addresses):** ~390 across the estate. app2 alone accounts for ~330 of them (Traccar's `5000-5150` device range published on both TCP and UDP = 302 sockets).
+- **Single most important finding:** Docker published-port rules bypass UFW on Core, app1, app2, and app3. Security and management consoles (Wazuh, UniFi, UNMS, Grafana, CloudPanel) are reachable from the public internet even though UFW's allow-lists do not include their ports.
+- **DNS hygiene is poor:** the `itpropartner.com` apex A record and ~10 legacy subdomains still resolve to a decommissioned GCP host (`35.212.86.161`), and the SPF record is malformed (two concatenated `v=spf1` strings with a truncated IP).
+
+---
+
+## 2. Cross-cutting finding: Docker port publishing bypasses UFW (Critical)
+
+**Evidence (app1, identical mechanism on Core/app2/app3):**
+```
+# iptables -t nat -L DOCKER -n
+DNAT tcp 0.0.0.0/0 0.0.0.0/0 tcp dpt:5601 to:172.26.0.4:5601
+DNAT tcp 0.0.0.0/0 0.0.0.0/0 tcp dpt:9200 to:172.26.0.2:9200
+# iptables -L DOCKER -n
+ACCEPT tcp 0.0.0.0/0 172.26.0.4 tcp dpt:5601
+ACCEPT tcp 0.0.0.0/0 172.26.0.2 tcp dpt:9200
+```
+UFW's `ALLOW` list on app1 contains only `22, 80, 443, 1514, 1515` (and `3006` from Core). Yet `5601`, `9200`, `55000`, `514/udp`, `3003`, `9120` are all published by Docker as `0.0.0.0:<port>`. Docker inserts DNAT rules into `nat/PREROUTING` and ACCEPT rules into the `filter/FORWARD` DOCKER chain, which are processed *before* UFW's `filter/INPUT` chain. UFW's default-deny therefore never sees these packets.
+
+**Why it matters:** every operator on these boxes believes UFW is the security boundary, but any `docker run -p <port>` silently punches a public hole. On app2, ~20 services (UniFi controller, UNMS/UISP, Gitea SSH, BookStack, MinIO console, RAGFlow, Technitium DNS, support API, Infinity DB) are publicly reachable despite none of their ports appearing in UFW. This is a systemic, high-impact control failure.
+
+**Remediation (Phase Two):** bind Docker publishes to `127.0.0.1` (e.g. `-p 127.0.0.1:5601:5601`) and route through the reverse proxy, or enable `ufw-docker`/`DOCKER-USER` chain rules.
+
+---
+
+## 3. Core (localhost / 152.53.192.33)
+
+### 3.1 Open ports (non-loopback)
+
+| Port | Proto | Process | Bound | UFW | Publicly reachable? |
+|---|---|---|---|---|---|
+| 22 | tcp | sshd | 0.0.0.0 | ALLOW Anywhere | Yes (intended) |
+| 80/443 | tcp/udp | caddy | 152.53.192.33 | ALLOW Anywhere | Yes (intended ingress) |
+| 3000 | tcp | browserless (docker) | 0.0.0.0 | **not listed** | **Yes - bypass** |
+| 3001 | tcp | uptime-kuma (docker) | 0.0.0.0 | **not listed** | **Yes - bypass** |
+| 3002 | tcp | grafana | `*` | ALLOW Anywhere | **Yes (intentional)** |
+| 9377 | tcp | camofox-browser (docker) | 0.0.0.0 | **not listed** | **Yes - bypass** |
+| 8080 | tcp | `python3 -m http.server` | 0.0.0.0 | tailscale0 only | No (UFW), bound 0.0.0.0 |
+| 8083 | tcp | shark-game backend | 0.0.0.0 | not listed | No (UFW) |
+| 8105 | tcp | rally backend | 0.0.0.0 | not listed | No (UFW) |
+| 8787 | tcp | socat → 127.0.0.1:8642 (Hermes) | 0.0.0.0 | not listed | No (UFW) |
+| 8899 | tcp | super-search MCP | 0.0.0.0 | 172.17.0.0/16 only | No |
+| 8934, 9876 | tcp | `python3 -m http.server` | 0.0.0.0 | not listed | No (UFW) |
+| 9090 | tcp | prometheus | `*` | not listed | No (UFW) |
+| 9100 | tcp | node_exporter | `*` | not listed | No (UFW) |
+| 9119 | tcp | hermes gateway | 0.0.0.0 | not listed | No (UFW) |
+| 9273 | tcp | telegraf | `*` | not listed | No (UFW) |
+| 34239 | tcp | act_runner | `*` | not listed | No (UFW) |
+| 1701 | udp | xl2tpd (L2TP) | 0.0.0.0 | not listed | No (UFW) |
+| 51821 | udp | wireguard | 0.0.0.0 | ALLOW Anywhere | Yes (intended) |
+| 5353 | udp | avahi-daemon | 0.0.0.0 + mcast | not listed | No (multicast) |
+
+### 3.2 Firewall
+UFW **active**, default deny incoming/allow outgoing/deny routed. Allows `22, 80, 443, 51821/udp, 8890, 3002`, `8080 on tailscale0`, `8899 from 172.17.0.0/16`. Baseline is sound, but the Docker bypass (section 2) undermines it for `3000/3001/9377`.
+
+### 3.3 Findings
+
+| ID | Severity | Finding | Evidence | Why it matters |
+|---|---|---|---|---|
+| CORE-1 | High | **Grafana (3002) publicly exposed by explicit UFW rule** | `ufw: 3002/tcp ALLOW Anywhere`; `ss: *:3002 grafana` | Grafana holds dashboards of the entire monitoring estate; a public, unauthenticated-able Grafana (or one with weak creds) leaks ops data and is a frequent CVE target. |
+| CORE-2 | High | **Headless-browser proxies publicly reachable via Docker bypass** | `docker: browserless 0.0.0.0:3000`, `camofox-browser 0.0.0.0:9377`; `nat DOCKER DNAT 0.0.0.0/0 dpt:3000`, `dpt:9377` | browserless/camofox render arbitrary URLs; a public instance is an SSRF / internal-network pivot primitive. |
+| CORE-3 | Medium | **Monitoring exporters bound to `0.0.0.0`** | `ss: *:9090 prometheus`, `*:9100 node_exporter`, `*:9273 telegraf` | These expose metrics (hostnames, labels, sometimes secrets in scrape configs) if UFW is ever disabled. Should be loopback/private-only. |
+| CORE-4 | Medium | **Three ad-hoc `python3 -m http.server` on 0.0.0.0** | PIDs 2191495 (:8080), 1601850 (:8934), 1607064 (:9876) | Unauthenticated static file servers serving unknown directories on all interfaces. |
+| CORE-5 | Medium | **Hermes control API exposed via socat on 0.0.0.0:8787** | `socat TCP-LISTEN:8787,fork,reuseaddr TCP:127.0.0.1:8642` | A control/agent API reachable on all interfaces; UFW currently blocks it, but the binding is needlessly broad. |
+| CORE-6 | Low | **avahi/mDNS (5353) running on public interface** | `ss: 0.0.0.0:5353 avahi-daemon` | Multicast name resolution leaks hostnames/services to the local segment. |
+
+---
+
+## 4. app1 (152.53.36.131, Netcup RS 4000) - Wazuh / AI / CRM host
+
+### 4.1 Open ports (non-loopback)
+
+| Port | Proto | Process | Bound | UFW | Publicly reachable? |
+|---|---|---|---|---|---|
+| 22 | tcp | sshd | 0.0.0.0 | ALLOW | Yes |
+| 80/443 | tcp | caddy | `*` | ALLOW | Yes (intended) |
+| 3006 | tcp | caddy (browserless proxy) | `*` | ALLOW from Core only | No (source-restricted) |
+| 1514/1515 | tcp | Wazuh manager | 0.0.0.0 | ALLOW Anywhere | **Yes (intentional but risky)** |
+| 514 | udp | Wazuh syslog | 0.0.0.0 | **not listed** | **Yes - bypass** |
+| 55000 | tcp | Wazuh manager API | 0.0.0.0 | **not listed** | **Yes - bypass** |
+| 9200 | tcp | Wazuh indexer (Elasticsearch) | 0.0.0.0 | **not listed** | **Yes - bypass** |
+| 5601 | tcp | Wazuh dashboard | 0.0.0.0 | **not listed** | **Yes - bypass** |
+| 3003 | tcp | Twenty CRM | 0.0.0.0 | **not listed** | **Yes - bypass** |
+| 9120 | tcp | Komodo core | 0.0.0.0 | **not listed** | **Yes - bypass** |
+
+### 4.2 Findings
+
+| ID | Severity | Finding | Evidence | Why it matters |
+|---|---|---|---|---|
+| APP1-1 | **Critical** | **Wazuh security stack publicly exposed via Docker bypass** | `docker: single-node-wazuh.indexer-1 0.0.0.0:9200`, `dashboard-1 0.0.0.0:5601`, `manager-1 0.0.0.0:55000, 0.0.0.0:514/udp`; UFW lists none of these | The Wazuh indexer holds all security-event telemetry for the estate and the dashboard is the admin console. Both are on the public internet with no UFW gate. |
+| APP1-2 | High | **Wazuh agent enrollment ports (1514/1515) open to the world by explicit UFW rule** | `ufw: 1514/tcp, 1515/tcp ALLOW Anywhere` | Enrollment should be restricted to agent source ranges; a public enrollment port invites rogue agent registration into the SIEM. |
+| APP1-3 | High | **Twenty CRM (3003) and Komodo core (9120) publicly reachable via Docker bypass** | `docker: twenty-server-1 0.0.0.0:3003`, `komodo-core-1 0.0.0.0:9120` | Client CRM data (PII) and the Komodo deployment/automation control plane are public, bypassing UFW. |
+| APP1-4 | Info | Intended public surface is correctly proxied | `/etc/caddy/Caddyfile` maps `vault`, `n8n`, `ai`, `admin-ai`, `noc`, `wz`, `sign.iamgmb.com`, `giftaroast.com`, `crm.debtrecoveryexperts.com`, `komodo.iamgmb.com`, `transitpin.iamgmb.com` → 127.0.0.1 backends | Reverse proxy is doing its job; the leaks are at the Docker layer, not Caddy. |
+
+---
+
+## 5. app2 (152.53.39.202, Netcup RS 4000) - management/self-hosted stack
+
+### 5.1 Open ports (non-loopback)
+
+| Port(s) | Proto | Service | UFW | Publicly reachable? |
+|---|---|---|---|---|
+| 22 | tcp | sshd | ALLOW | Yes |
+| 80/443 | tcp | caddy | ALLOW | Yes (intended) |
+| 53 | tcp/udp | Technitium DNS (dns1.itpropartner.com) | ALLOW **only from 76.195.7.60** | **Yes - bypass (open resolver)** |
+| 81, 8089, 8444 | tcp | UNMS/UISP nginx | **not listed** | **Yes - bypass** |
+| 8080, 8443, 8843, 8880 | tcp | UniFi controller | **not listed** | **Yes - bypass** |
+| 3478, 10001 | udp | UniFi STUN/discovery | **not listed** | **Yes - bypass** |
+| 2055 | udp | UNMS NetFlow | **not listed** | **Yes - bypass** |
+| 3022 | tcp | Gitea SSH | **not listed** | **Yes - bypass** |
+| 6875 | tcp | BookStack | **not listed** | **Yes - bypass** |
+| 6880 | tcp | support-api | **not listed** | **Yes - bypass** |
+| 8082 | tcp | Traccar web UI | **not listed** | **Yes - bypass** |
+| 5000-5150 | tcp+udp | Traccar device listeners (302 sockets) | **ALLOW Anywhere (explicit)** | **Yes (intentional)** |
+| 9001 | tcp | MinIO console | **not listed** | **Yes - bypass** |
+| 9380-9384, 9392 | tcp | RAGFlow | **not listed** | **Yes - bypass** |
+| 23817, 23820 | tcp | Infinity database | **not listed** | **Yes - bypass** |
+
+### 5.2 Findings
+
+| ID | Severity | Finding | Evidence | Why it matters |
+|---|---|---|---|---|
+| APP2-1 | **Critical** | **UniFi Network Controller publicly exposed via Docker bypass** | `docker: unifi-controller 0.0.0.0:8080,8443,8843,8880`; `nat DOCKER DNAT 0.0.0.0/0 dpt:8443 to:172.17.0.2:8443`; UFW lists none | The UniFi controller manages the tower/Wi-Fi network. Its 8443 web UI and 8080 device-inform endpoint are public. Compromise = control of the wireless/tower estate. |
+| APP2-2 | **Critical** | **UNMS/UISP (ISP management) publicly exposed via Docker bypass** | `docker: unms-nginx 0.0.0.0:81,8089,8444`; `nat DNAT 0.0.0.0/0 dpt:8444 to:172.18.251.5:443` | UISP is the entire WISP network-management plane (device inventory, configs, customers). Publicly reachable despite UFW. |
+| APP2-3 | High | **Technitium DNS published 0.0.0.0:53 bypasses source restriction** | UFW allows `53 only from 76.195.7.60`, but `docker: technitium 0.0.0.0:53->53 tcp+udp` and `DOCKER chain ACCEPT 0.0.0.0/0 dpt:53` | Operator clearly intended DNS to be reachable only from the home router, yet the Docker publish makes it a public open resolver (DNS amplification/abuse risk). |
+| APP2-4 | High | **Traccar device range (5000-5150, ~302 sockets) fully public by explicit UFW rule** | `ufw: 5000:5150/tcp+udp ALLOW Anywhere` | Fleet-tracking device protocol listeners are open to the world; large attack surface for protocol-specific exploits and data injection. |
+| APP2-5 | High | **Gitea SSH (3022), BookStack (6875), support-api (6880), MinIO console (9001), RAGFlow (9380-9392), Infinity DB (23817/23820) all public via bypass** | `docker ps --format '{{.Ports}}'` shows all bound `0.0.0.0` | Source-code host, internal wiki, support API, object-storage console, and a vector database are each on the public internet. |
+| APP2-6 | Info | `git.itpropartner.com`, `hudu.itpropartner.com`, `unifi.itpropartner.com`, `unms.forefrontwireless.com`, `ragflow.itpropartner.com`, `gps.fleettracker360.com` proxied via Caddy | `/etc/caddy/Caddyfile` | The reverse proxy is correctly terminating TLS for the intended public names; the exposure is the direct Docker port publish bypassing it. |
+
+---
+
+## 6. app3 (152.53.241.111, Netcup RS 4000) - CloudPanel shared web host
+
+### 6.1 Open ports (non-loopback)
+
+| Port | Proto | Process | Bound | UFW | Publicly reachable? |
+|---|---|---|---|---|---|
+| 22 | tcp | sshd | 0.0.0.0 | ALLOW | Yes |
+| 80/443 | tcp/udp | nginx (CloudPanel) | 0.0.0.0 | ALLOW | Yes (intended) |
+| 8443 | tcp | nginx (CloudPanel panel) | 0.0.0.0 | ALLOW 8433:8443 | **Yes (intentional)** |
+| 8090 | tcp | backup-restore web app (python) | 0.0.0.0 | ALLOW Anywhere | **Yes (intentional)** |
+| 21 | tcp | proftpd FTP | 0.0.0.0 | not listed | No (UFW), bound 0.0.0.0 |
+| 25 | tcp | postfix SMTP | 0.0.0.0 | not listed | No (UFW) |
+| 3000 | tcp | buzz-prod-relay (docker) | 0.0.0.0 | **not listed** | **Yes - bypass** |
+| 6081 | tcp | varnish | 0.0.0.0 | not listed | No (UFW) |
+| 8080 | tcp | nginx (alt vhost) | 0.0.0.0 | not listed | No (UFW) |
+| 33060 | tcp | mysqld (MySQL X) | `*` | not listed | No (UFW), bound `*` |
+
+### 6.2 Findings
+
+| ID | Severity | Finding | Evidence | Why it matters |
+|---|---|---|---|---|
+| APP3-1 | High | **CloudPanel admin panel (8443) publicly exposed** | `ufw: 8433:8443/tcp ALLOW Anywhere`; `ss: 0.0.0.0:8443 nginx` | CloudPanel is the control plane for ~30 client websites on this box. Publicly exposing the admin panel (even with auth) is a high-value target. |
+| APP3-2 | High | **Backup-restore web UI (8090) publicly exposed** | `ufw: 8090/tcp ALLOW Anywhere`; `ss: 0.0.0.0:8090 python /opt/backup-restore/app/app.py` | A web UI that can trigger restores is a destructive-capability surface and should be internal/Tailscale-only. |
+| APP3-3 | High | **Buzz relay (3000) public via Docker bypass** | `docker: buzz-prod-relay-1 0.0.0.0:3000`; `nat DNAT 0.0.0.0/0 dpt:3000 to:172.19.0.5:3000`; UFW does not list 3000 | The Block/Buzz relay is publicly reachable with no UFW gate. |
+| APP3-4 | Medium | **MySQL X protocol (33060) bound to `*`** | `ss: *:33060 mysqld` | Database protocol listener on all interfaces (currently UFW-blocked); should be loopback. |
+| APP3-5 | Medium | **FTP (21), SMTP (25), Varnish (6081) bound to 0.0.0.0** | `ss` output | FTP is cleartext (credential leakage if ever allowed); SMTP bound publicly invites relay abuse; Varnish cache admin not needed externally. |
+
+---
+
+## 7. app1-bu (5.161.225.131, Hetzner CPX21) - warm standby
+
+Minimal footprint. Only `22/tcp` (sshd) and `41641/udp` (tailscaled) listening on non-loopback. UFW active: `22/tcp` and `51821/udp` allowed.
+
+| ID | Severity | Finding | Evidence | Why it matters |
+|---|---|---|---|---|
+| APP1BU-1 | Low | **UFW allows 51821/udp (WireGuard) but no WireGuard service is installed/listening** | `ufw: 51821/udp ALLOW`; `ss` shows no 51821; `wg: command not found` | Leftover rule opens a port with no service behind it; indicates incomplete standby bring-up. |
+| APP1BU-2 | Info | Tailscale device name drift | `tailscale status` lists `app1-bu` (100.112.23.21, offline ~28d) and `app1-bu-1` (100.95.212.28, online) | The live standby is registered as `app1-bu-1`; the old `app1-bu` node is stale on the tailnet. |
+
+---
+
+## 8. wphost02 (5.161.62.38, Hetzner) - legacy WordPress/RunCloud
+
+**Still live and serving traffic.** Not decommissioned.
+
+| Port | Proto | Process | Bound | UFW | Publicly reachable? |
+|---|---|---|---|---|---|
+| 22 | tcp | sshd | 0.0.0.0 | ALLOW | Yes |
+| 80/443 | tcp | nginx-rc | 0.0.0.0 | ALLOW | Yes (legacy sites) |
+| 25 | tcp | postfix | 0.0.0.0 | not listed | No (UFW) |
+| 34210 | tcp | runcloud agent | `*` | not listed | No (UFW) |
+| 9100 | tcp | node_exporter | `*` | not listed | No (UFW) |
+
+| ID | Severity | Finding | Evidence | Why it matters |
+|---|---|---|---|---|
+| WPHOST-1 | Medium | **Legacy WordPress host still publicly serving on 80/443** | `ss: 0.0.0.0:80,443 nginx-rc`; `ufw: 80,443 ALLOW` | Flagged as possibly decommissioned in scope; it is still a live public attack surface (legacy WordPress) that should be verified against the migration plan and decommissioned or hardened. |
+| WPHOST-2 | Low | **RunCloud agent (34210) and node_exporter (9100) bound to `*`** | `ss: *:34210 runcloud`, `*:9100 node_exporter` | Management agent and metrics exporter on all interfaces (currently UFW-blocked). |
+
+---
+
+## 9. Public DNS - itpropartner.com and related domains
+
+Nameservers: `ns1/ns2.siteground.net`. All lookups against `1.1.1.1`.
+
+| ID | Severity | Finding | Evidence | Why it matters |
+|---|---|---|---|---|
+| DNS-1 | **High** | **Apex A record and ~10 legacy subdomains point to decommissioned GCP host 35.212.86.161** | `itpropartner.com A → 35.212.86.161` (no live server uses this IP). Also `ssh`, `mail`, `ftp`, `autodiscover`, `autoconfig`, `clientmedia`, `www.clientmedia`, `media`, `www.media`, `apx`, `www.apx`, `www` all → `35.212.86.161` | The primary domain apex resolves to a retired host. Visitors and services hitting the apex go nowhere (or to an attacker if the IP is later reassigned). Live subdomains (`ops`, `core`, `app1`, `git`, `panel`, etc.) point to the correct hosts. |
+| DNS-2 | High | **SPF record is malformed (concatenated `v=spf1` + truncated IP)** | `"v=spf1 +a +mx +ip4:35.209.36v=spf1 +a +mx +ip4:35.212.110.90 include:... ~all"` | Two SPF records were merged and an IP (`35.209.36`) is truncated. Broken SPF breaks delivery and can allow spoofing depending on how receivers parse it. |
+| DNS-3 | Medium | **DMARC is `p=none` (monitoring only), and four related domains have no DMARC at all** | `_dmarc.itpropartner.com → "v=DMARC1; p=none; aspf=r; adkim=r"`; `fleettracker360.com`, `voipsimplicity.com`, `iamgmb.com`, `debtrecoveryexperts.com` → empty | No enforcement means the domain can be spoofed with no receiver-side protection. |
+| DNS-4 | Medium | **fleettracker360.com has no MX, no SPF, no DMARC** | `dig MX/TXT` all empty (NS = Cloudflare) | A live product domain with no mail/DMARC records is trivially spoofable. |
+| DNS-5 | Low | **voipsimplicity.com apex has no A record** | `dig A voipsimplicity.com → (empty)` | Apex resolves to nothing; subdomain `my.voipsimplicity.com` works but the root does not. |
+| DNS-6 | Info | **DKIM present (dnssmarthost); wildcard absent; MX correct** | `default._domainkey.itpropartner.com TXT → v=DKIM1...`; `*.itpropartner.com → empty`; `MX → mx10/20/30.antispam.mailspamprotection.com` | Good: DKIM configured, no wildcard, MX routes through SiteGround antispam. |
+| DNS-7 | Info | **Live subdomain map verified** | `ops/core/my/sign/uptimekuma/app/status → 152.53.192.33`; `app1/n8n/ai/admin-ai/vault/wz/noc → 152.53.36.131`; `git/hudu/unifi → 152.53.39.202`; `panel/mainwp/support/auth2/docs/forms/mockups/proposals → 152.53.241.111`; `app1-bu → 5.161.225.131` | Subdomains are correctly mapped to live hosts; only the apex + legacy names are stale. |
+
+---
+
+## 10. VPN / private network paths
+
+| Path | Technology | Endpoints | State |
+|---|---|---|---|
+| Server mesh | Tailscale | Core 100.71.155.7, app1 100.90.186.109, app3 100.72.15.12, app1-bu-1 100.95.212.28 (+ app2, personal devices) | Up, full mesh |
+| Core → home network | WireGuard `wg0` (10.77.0.1/24) | peer `home-gateway` 10.77.0.2, endpoint 76.195.7.60:443 | Up |
+| Core → home lab / tower subnets | WireGuard routed | `10.1.0.0/16`, `10.2.0.0/16`, `172.16.1.0/24`, `172.18.18.0/24` via wg0 | Up |
+| Core → WISP towers | L2TP/IPsec (`ppp0`) | server 76.195.7.60; routes `10.199.1-4.0/24`, `10.199.100.0/24`, `192.168.88.0/24` | **Down** (charon not running, no ppp0, no 10.199 routes) |
+
+| ID | Severity | Finding | Evidence | Why it matters |
+|---|---|---|---|---|
+| VPN-1 | Medium | **Tower VPN (L2TP/IPsec) uses IKEv1 + AES128/SHA1/MODP1024** | `home-router-vpn.sh`: `ike=aes128-sha1-modp1024`, `esp=aes128-sha1-modp1024`, `keyexchange=ikev1` | Legacy crypto for the path into the WISP tower network; weak and cryptographically dated. |
+| VPN-2 | Medium | **VPN credentials stored in plaintext config** | `/root/.hermes/scripts/wisp-backup/config.yaml` holds L2TP `psk`, `username`, `password` in cleartext (values `[REDACTED]`) | A file-readable compromise of Core yields credentials to the home gateway and tower network. Cross-ref Sec-A for secret management. |
+| VPN-3 | Info | Tower subnets only reachable while the nightly-backup VPN is up | `ipsec status` → charon refused; no `10.199.*` routes present | Towers (T01-T04, MP100) are not persistently reachable; enumeration of their live config was not possible this session. |
+
+---
+
+## 11. Edge devices (MikroTik CCR towers / UniFi / UISP)
+
+Discoverable but **not directly enumerated this session** (VPN down; read-only constraint).
+
+- **Home MikroTik gateway:** public endpoint `76.195.7.60` (WireGuard :443, L2TP/IPsec :1701/udp). SSH reachable via WireGuard as `home-gateway` 10.77.0.2 (`wisp_rsa` key, user `shonuff`).
+- **WISP tower routers (CCR):** `T01-RTR 10.199.1.4`, `T02-RTR 10.199.2.4`, `T03-RTR 10.199.3.4`, `T04-RTR 10.199.4.4`, `MP100-RTR 10.199.100.4` (source: `wisp-backup/config.yaml`).
+- **UniFi controller** runs on app2 (publicly exposed, see APP2-1). **UNMS/UISP** runs on app2 (publicly exposed, see APP2-2). Backup key deployment for towers is via `deploy-key.rsc` (SSH pubkey, not secret).
+- No edge-device config was modified; only inventory/paths recorded.
+
+---
+
+## 12. Access limitations
+
+| Host | Status |
+|---|---|
+| Core, app1, app2, app3, app1-bu, wphost02 | Reachable via `itpp-infra` key; full enumeration completed |
+| MikroTik towers T01-T04, MP100 | **Not reachable** - L2TP/IPsec VPN down (charon not running, no `10.199.*` routes) |
+| Home gateway 76.195.7.60 | Public endpoint confirmed reachable at network level (WG/L2TP listeners); not SSH-enumerated this session |
+
+---
+
+## 13. Consolidated severity summary
+
+| Severity | Count | Highlights |
+|---|---|---|
+| Critical | 3 | Docker bypass exposing Wazuh stack (APP1-1); UniFi controller public (APP2-1); UNMS/UISP public (APP2-2) |
+| High | 11 | Grafana public, browserless/camofox public, Wazuh 1514/1515 public, Twenty/Komodo public, Technitium open resolver, Traccar range public, Gitea/BookStack/MinIO/RAGFlow public, CloudPanel public, backup-restore UI public, Buzz relay public, apex DNS stale, SPF broken |
+| Medium | 11 | Monitoring exporters on 0.0.0.0, ad-hoc http.server, socat→Hermes, MySQL X on `*`, FTP/SMTP/Varnish on 0.0.0.0, DMARC p=none / missing, weak L2TP crypto, plaintext VPN creds, legacy wphost02 live |
+| Low | 4 | avahi on public iface, app1-bu stale WG rule + tailscale name drift, runcloud/node_exporter on wphost02 |
+
+**Note on remediation:** all items above are Phase Two candidates. No firewall, service, DNS, or VPN configuration was changed during this audit.

audit/phase-one/findings/neteng-b.md

@@ -0,0 +1,197 @@
+# NetEng-B - Network Segmentation and Blast-Radius Assessment (Phase One, Read-Only)
+
+**Auditor:** NetEng-B
+**Date:** 2026-08-13
+**Scope:** Tier classification, inter-host segmentation, blast-radius chains, and product/tenant isolation across the ITPP estate (Core, app1, app2, app3, app1-bu, wphost02).
+**Method:** Read-only discovery only (`ip`, `ufw status verbose`, `ss`, `tailscale status --json`, `docker network ls`, `docker inspect`, `cat` of config files, `curl`/TCP reachability probes between hosts). Builds directly on NetEng-A's raw port/DNS inventory (`neteng-a.md`) - no port/DNS re-enumeration performed here. No configuration was changed on any host.
+
+**Severity legend:** Critical = a single compromise gives an attacker control of most or all of the estate, or of a system holding client/security data with no compensating boundary. High = a compromise crosses a trust boundary that should exist (internal-to-client, product-to-product) with material impact. Medium = defense-in-depth gap that increases blast radius but requires a second failure to be catastrophic. Low = hygiene issue with limited blast-radius effect.
+
+---
+
+## 1. Executive summary
+
+- **There is no network segmentation between Core, app1, app2, app3, app1-bu, and wphost02.** All six hosts sit on public Netcup/Hetzner IP space with no VLAN, no subnetting, and no firewall boundary between each other. The only inter-host overlay is a **single flat Tailscale mesh** with **no ACL tags applied to any node** - every server and every personal device (phone, laptop, home router) is in one undifferentiated group with implicit allow-all reachability.
+- Because UFW is bypassed by Docker on 4 of 6 hosts (confirmed by NetEng-A and re-verified here), the "segmentation" that exists on paper (UFW allow-lists) does not match what is actually reachable. Verified independently: Core -> app1 and app1 -> app2 both reach each other over their **public IPs**, not just Tailscale, with no filtering in between.
+- **Compromise of any one of the six hosts gives an attacker a foothold that can reach every other host** over the public internet (all are mutually pingable/routable on public IPs) and, once inside, over the flat Tailscale mesh as well. There is no host that is network-isolated from the rest.
+- **Products and internal ITPP operations are not isolated.** app1 runs the internal LLM gateway (LiteLLM/admin-ai), the internal CRM (Twenty), the security stack (Wazuh), and the reverse proxy for two products (Komodo, TransitPin) all as sibling Docker containers on one host with one Caddy instance. app3 runs a **single shared MySQL/Percona instance** behind ~24 CloudPanel sites that mix internal ops (mainwp, support, panel, ippadmin), client sites (katiewatts, modelortho, vigilanttac, boxpilotlogistics, timapta), and products (transitpin, myverdicttank, buzz, hexclave) with no per-tenant database server or credential vault separating them.
+- **Tier misclassification is widespread**: management/admin consoles (Wazuh dashboard+indexer, UniFi controller, UNMS/UISP, CloudPanel admin, backup-restore UI, MinIO console, RAGFlow, Infinity DB, Komodo) are all internet-reachable, several unintentionally via the Docker/UFW bypass NetEng-A documented. These are control planes; none should be directly public.
+
+---
+
+## 2. Tier classification
+
+Legend: **INT** = Internal-only (should never be public), **CLI** = Client-facing (legitimately public), **PROD** = Product/dev (micro-SaaS), **MGMT** = Management/admin control plane.
+
+| Service | Host | Correct tier | Actually public? | Tier violation? |
+|---|---|---|---|---|
+| SSH (22) all hosts | all | MGMT | Yes (intended, key-only) | No - acceptable exposure |
+| Caddy/nginx 80/443 (ingress) | Core, app1, app2, app3, wphost02 | CLI | Yes (intended) | No |
+| Grafana :3002 | Core | MGMT | **Yes (explicit UFW allow)** | **Yes - dashboards public** |
+| Prometheus/node_exporter/telegraf | Core | INT | No (UFW blocks; Docker bypass N/A, native binaries) | No, but bound 0.0.0.0 (Medium hygiene per NetEng-A) |
+| browserless / camofox | Core | INT (SSRF-capable tool) | **Yes - Docker bypass** | **Yes** |
+| WireGuard :51821 | Core, app1-bu | MGMT/VPN | Yes (intended) | No |
+| Wazuh manager 1514/1515 | app1 | MGMT | Yes (intended, but should be source-restricted) | Partial - over-broad |
+| Wazuh indexer :9200, dashboard :5601, API :55000, syslog 514/udp | app1 | MGMT | **Yes - Docker bypass** | **Yes - critical control plane public** |
+| Twenty CRM :3003 | app1 | INT (holds client PII) | **Yes - Docker bypass** | **Yes** |
+| Komodo core :9120 | app1 | MGMT (deploy/automation control plane) | **Yes - Docker bypass**, also proxied at komodo.iamgmb.com | **Yes** |
+| LiteLLM/admin-ai (via Caddy) | app1 | MGMT | Yes (intended, proxied) | No (proxy correct; bypass risk is at Docker layer for adjacent services) |
+| TransitPin static site | app1 | PROD | Yes (intended) | No |
+| giftaroast.com site + API | app1 | PROD/CLI | Yes (intended) | No |
+| UniFi controller 8080/8443/8843/8880 | app2 | MGMT | **Yes - Docker bypass** | **Yes - wireless/tower control plane public** |
+| UNMS/UISP 81/8089/8444 | app2 | MGMT | **Yes - Docker bypass** | **Yes - entire WISP management plane public** |
+| Technitium DNS :53 | app2 | INT (should be source-restricted to home router only) | **Yes, open resolver - Docker bypass defeats UFW source restriction** | **Yes** |
+| Gitea SSH :3022 | app2 | MGMT (source code) | **Yes - Docker bypass** | **Yes** |
+| BookStack :6875 | app2 | INT (internal wiki) | **Yes - Docker bypass** | **Yes** |
+| support-api :6880 | app2 | INT/CLI boundary (unclear which) | **Yes - Docker bypass** | Flag for clarification |
+| MinIO console :9001 | app2 | MGMT | **Yes - Docker bypass** | **Yes** |
+| RAGFlow :9380-9392 | app2 | PROD/INT (unclear) | **Yes - Docker bypass** | Flag for clarification |
+| Infinity DB :23817/23820 | app2 | INT (database) | **Yes - Docker bypass** | **Yes** |
+| Traccar device listeners 5000-5150 | app2 | PROD (device protocol, must be public) | Yes (intended) | No - legitimate, but see APP2-4 in NetEng-A for hardening |
+| Hudu, Dawarich | app2 | INT (internal IT docs / location tracking) | Proxied via Caddy (correctly) at hudu.itpropartner.com | No tier issue found at proxy layer |
+| CloudPanel admin panel :8443 | app3 | MGMT | **Yes (explicit UFW allow)** | **Yes - hosting control plane for ~24 sites public** |
+| backup-restore web UI :8090 | app3 | MGMT (destructive capability) | **Yes (explicit UFW allow)** | **Yes** |
+| buzz-prod-relay :3000 | app3 | PROD | **Yes - Docker bypass** | Bypass mechanism wrong even though public exposure may be intended |
+| MySQL X protocol :33060 | app3 | INT (database) | Bound `*` but currently UFW-blocked | No public exposure confirmed, hygiene flag only |
+| CloudPanel client/product sites (24 total) | app3 | Mixed CLI/PROD/INT (see section 5) | Yes (intended, various) | See section 5 for isolation gap, not tier gap |
+| wphost02 nginx-rc 80/443 | wphost02 | CLI (legacy) | Yes (intended, legacy) | No tier issue, but should be decommissioned per NetEng-A |
+
+### Tier-classification counts (from table above)
+- **INTERNAL-ONLY services incorrectly public: 12** (Grafana, browserless, camofox, Wazuh indexer/dashboard/API/syslog treated as one group counted once = Wazuh stack, Twenty CRM, Technitium open resolver, BookStack, MinIO console, Infinity DB, plus MySQL X hygiene flag)
+- **MANAGEMENT/ADMIN consoles incorrectly public: 6** (Wazuh dashboard, UniFi controller, UNMS/UISP, CloudPanel admin, backup-restore UI, Komodo core)
+- **CLIENT-FACING correctly public: 6** (Core/app1/app2/app3/wphost02 web ingress, Traccar device range)
+- **PRODUCT/DEV correctly public (by design): 4** (TransitPin, giftaroast.com, buzz relay -- though via wrong mechanism, Traccar)
+- **Ambiguous tier, needs Germaine clarification: 2** (support-api, RAGFlow)
+
+---
+
+## 3. Segmentation map
+
+```
+ PUBLIC INTERNET
+ |
+ +--------------------------------+--------------------------------+
+ | | | | |
+ Core app1 (Netcup) app2 (Netcup) app3 (Netcup) app1-bu / wphost02
+152.53.192.33 152.53.36.131 152.53.39.202 152.53.241.111 (Hetzner, separate /32s)
+ | | | | |
+ |<===============+===============+===============+================|
+ FLAT PUBLIC IP MESH -- every host reaches every other host's
+ public IP directly. No VLAN. No inter-host firewall. Verified:
+ Core->app1:*, app1->app2:* all reachable on PUBLIC IPs, unfiltered
+ for whatever Docker/UFW leaves open on the receiving end.
+ | | | | |
+ +----------------+---------------+---------------+----------------+
+ |
+ TAILSCALE MESH (100.x.x.x/32 each)
+ tailscale status --json: NO "Tags" field on ANY
+ node (Core, app1, app2, app3, app1-bu-1). Default
+ tailnet ACL = allow-all between all nodes.
+ app3 has no tailscaled at all listed in this scan --
+ wait, confirmed: app3 100.72.15.12 IS on tailnet.
+ wphost02 has NO tailscale client installed at all --
+ it is reachable ONLY over the flat public internet.
+ |
+ +---------------+----------------+------------------+
+ | | | |
+ Core app1 app2 app3
+ (+ personal devices: iphone, m4-mac-mini, ipp-g-lap, home
+ router "liberty-udm-pro" -- ALL in the SAME flat tailnet group
+ as the six production servers, no tag-based isolation)
+
+ app1-bu-1 (Hetzner standby) is ALSO in this same flat tailnet.
+
+ wphost02 -- OUTSIDE the tailnet entirely. Reachable from Core/app1/
+ app2/app3 only via public internet + SSH key. One-directional trust:
+ Core has an autossh reverse tunnel INTO wphost02 (mysql-tunnel,
+ 127.0.0.1:33060 -> wphost02:3306) plus a root SSH cron backup job.
+ wphost02 has no client that can reach back into Core/app1/app2/app3.
+
+INSIDE EACH HOST: dozens of isolated Docker bridge networks per
+compose stack (e.g. app1 has 15 separate bridge networks: litellm,
+twenty, komodo, n8n, docuseal, etc.) -- this is REAL intra-host
+container isolation. But it is undermined by the Docker-publish-vs-UFW
+bypass NetEng-A documented: many of those "isolated" containers punch
+a hole straight to 0.0.0.0 on the host's public interface, which
+erases the isolation the bridge network was providing.
+```
+
+**Bottom line: there is effectively ONE trust zone across the entire estate.** The only segmentation primitives in play (UFW allow-lists, per-stack Docker bridge networks, Tailscale) are all either bypassed (UFW/Docker), unused for isolation (Tailscale has no ACL tags), or absent entirely between hosts (no VLAN/subnet separation of any kind exists between Core/app1/app2/app3/app1-bu; wphost02 is flat-public with an SSH+tunnel trust relationship back to Core).
+
+---
+
+## 4. Per-host blast-radius table
+
+| Host | If compromised, attacker gets... | Falls in blast radius | Evidence |
+|---|---|---|---|
+| **Core** | Root on the Hermes agent host: WireGuard keys to home network + WISP towers (10.77.0.0/24, 10.1.0.0/16, 10.2.0.0/16, 172.16.1.0/24, 172.18.18.0/24 routed via wg0), plaintext L2TP/IPsec creds to home gateway, the `itpp-infra` SSH private key (used to reach ALL 6 hosts), the autossh tunnel credential path into wphost02's MySQL, Grafana/Prometheus telemetry, Tailscale identity (can pivot to every tailnet peer incl. personal devices) | app1, app2, app3, app1-bu, wphost02 (via SSH key + tunnels), home network + WISP towers (via WireGuard), personal devices on tailnet (phone, laptop, home router) | `ip route` shows wg0 routes; `/root/.ssh/itpp-infra` used identically against all 6 hosts in this audit; crontab shows nightly root SSH job to wphost02; NetEng-A VPN-2 documents plaintext creds |
+| **app1** | Root on the box hosting Wazuh (SIEM for the whole estate), Twenty CRM (client PII), LiteLLM/admin-ai (the AI control plane and its API keys), Komodo (deployment/automation control plane), n8n (workflow automation with stored credentials) | Every host Wazuh agents report from (SIEM blast radius = estate-wide visibility loss/tamper), every client whose PII sits in Twenty, every downstream system Komodo can deploy to, every credential n8n workflows hold | Docker inspect showed `PG_DATABASE_URL`, `APP_SECRET`, `REDIS_URL` for Twenty; `KOMODO_JWT_SECRET`/`KOMODO_WEBHOOK_SECRET` for Komodo; `DATABASE_URL` for LiteLLM postgres; Caddyfile proxies vault/n8n/ai/admin-ai/noc/wz/komodo/transitpin all from this one host |
+| **app2** | Root on the box running UniFi controller (wireless/tower control), UNMS/UISP (entire WISP customer/device management plane), Gitea (source code + SSH deploy keys), Hudu (IT documentation, likely holds more credentials/secrets), Traccar (fleet GPS data), BookStack, RAGFlow, Infinity DB, MinIO | The WISP tower network and its customers, all git repos + CI secrets on Gitea, all documented IT credentials in Hudu, fleet-tracking customer data, any data indexed in RAGFlow/Infinity | `docker inspect` showed `UNMS_PG_HOST`, `SECURE_LINK_SECRET`, Hudu `SECRET_KEY_BASE`/`S3_SECRET_ACCESS_KEY`, Gitea SSH port 3022 public per NetEng-A |
+| **app3** | Root on the box running CloudPanel (control plane for ~24 sites) and ONE shared MySQL/Percona instance backing internal ops sites, client sites, and product sites simultaneously | All ~24 CloudPanel-hosted sites: internal (mainwp, support, panel, ippadmin), clients (katiewatts, modelortho, vigilanttac, boxpilotlogistics, timapta), products (transitpin, myverdicttank, buzz, hexclave) - a single DB-engine compromise threatens every one of them at once | `ss -tlnp` showed one `mysqld` process on 127.0.0.1:3306/*:33060; `docker ps` on app3 showed buzz-prod-postgres, hexclave-postgres as separate containers, but CloudPanel's own PHP/static sites run against the single host-level Percona instance shown above |
+| **app1-bu** | Root on the warm-standby box. Minimal live footprint (per NetEng-A). Sync direction is **pull-only**: `hermes-standby-sync.sh` pulls from Wasabi S3, does not push to or read live secrets directly off Core over the network | Limited - compromise here does not directly expose Core, because sync is one-way pull from S3, not a live network tunnel to Core. Attacker would get whatever is in the last S3 snapshot (which may include full Hermes state/secrets) | `hermes-standby-sync.sh`: `aws s3 sync s3://hermes-vps-backups/live/ ...`; `authorized_keys` on app1-bu matches Core's `itpp-infra.pub`, meaning Core (not app1-bu) is the initiator of any direct SSH, consistent with pull-based design |
+| **wphost02** | Root on legacy WordPress/RunCloud host. Directly exposes MySQL 3306 to Core via the standing autossh tunnel (`-L 127.0.0.1:33060:localhost:3306`) | Core's tunneled MySQL access (`apextrackexperience` DB per `service-health-check.sh`); any WordPress sites still live here | `ps aux` on Core shows `autossh ... -L 127.0.0.1:33060:localhost:3306 -N root@5.161.62.38`; `service-health-check.sh` references `MySQL SSH tunnel (wphost02)` and `MySQL database (apextrackexperience)` |
+
+---
+
+## 5. Blast-radius dependency chains (worst-case, critical services)
+
+1. **Core mysql-tunnel -> wphost02**: Core maintains a permanent autossh reverse tunnel (`127.0.0.1:33060 -> wphost02:3306`) plus a nightly cron root-SSH backup job to wphost02. If Core is compromised, the attacker inherits this tunnel and reaches wphost02's database directly, no additional credential theft required. If wphost02 is compromised first, the exposed MySQL on `localhost:3306` there is only reachable over the SSH tunnel Core already holds the key for - so wphost02 compromise does not directly threaten Core, but Core compromise fully threatens wphost02's DB.
+2. **app1 super-search / MCP tooling -> Core**: app1 hosts multiple MCP servers (`mcp-browser`, `mcp-email`, `mcp-git`, `mcp-filesystem`, `super-search`) that are part of the same automation fabric as Core's `super-search` (:8899, allow-listed to `172.17.0.0/16` on Core). A compromised app1 MCP container that can reach Core's Docker subnet peer range could attempt to reach Core's super-search MCP if any cross-host trust exists in the MCP orchestration layer. Full confirmation of live cross-host MCP calls was not possible read-only this session - **flagged for Phase Two verification**, but the underlying network path (Core and app1 both reachable from each other over public IP and Tailscale with no ACL) means the network layer would not stop such a call if the application layer permits it.
+3. **Core Wazuh agents -> app1 Wazuh manager**: every host in the estate almost certainly runs a Wazuh agent reporting to app1's manager (1514/1515, intentionally public per NetEng-A APP1-2). If app1's Wazuh stack is compromised (already publicly reachable via the Docker/UFW bypass - APP1-1, Critical), the attacker controls the SIEM for the entire estate: they can see all security telemetry and potentially inject false negatives, blinding detection across Core/app2/app3/app1-bu/wphost02 simultaneously. This is the single highest-leverage compromise in the estate.
+4. **app3 shared Percona -> all 24 CloudPanel sites**: any SQL injection, credential leak, or root compromise on app3 threatens every site's database in one blast, because there is one MySQL engine process backing internal, client, and product sites alike (see section 6).
+5. **app1-bu standby -> S3 snapshot, not live Core**: because sync is pull-only from Wasabi S3 rather than a live tunnel to Core, app1-bu compromise does NOT directly hand an attacker live access to Core. This is a **positive isolation finding** - call it out as something already done right, worth preserving in Phase Two hardening (don't accidentally add a live push tunnel later).
+
+---
+
+## 6. Product/tenant isolation findings
+
+- **app3 (CloudPanel) hosts ~24 sites behind one shared MySQL/Percona instance** with no per-tenant database server, container, or credential vault: `apx, boxpilotlogistics, buzz, clp, debtreecoveryexperts, docs, drecovery, forms, gmb, hexclave-api, hexclave-dash, iAmGMB, intelsight, ippadmin, katiewatts, mainwp, mockups, modelortho, modelorthowww, myverdicttank, myvoip, panel, proposals, support, timapta, transitpin, transitpin-dash, vigilanttac, voipsimplicity`. This list mixes **internal ITPP ops** (mainwp, support, panel, ippadmin, docs, proposals, mockups), **client sites** (katiewatts, modelortho/modelorthowww, vigilanttac, boxpilotlogistics, timapta), and **products** (transitpin/transitpin-dash, myverdicttank, buzz, hexclave-api/dash) on the same host, same web server, same database engine. Some products (buzz, hexclave) do run their own dedicated Postgres containers alongside the shared Percona instance, which is better isolation for those two, but the CloudPanel-native PHP/static sites (the majority) share the one host-level MySQL.
+- **app1 mixes internal control-plane services with two client-facing products in the same Docker/Caddy stack**: Wazuh (SIEM), Twenty CRM (internal, holds PII for debt-recovery-experts.com), LiteLLM/admin-ai (AI gateway), and Komodo (deploy automation) run as containers alongside TransitPin and giftaroast.com (products) and crm.debtrecoveryexperts.com (client-branded CRM instance). A single Caddy instance and a single Docker host boundary is the only thing separating "internal admin tooling" from "product the client interacts with directly."
+- **No evidence of per-product credential vaults.** Each product/service has its own DB password baked into its own container's env (Twenty's `APP_SECRET`, Komodo's `KOMODO_JWT_SECRET`, buzz's `BUZZ_S3_SECRET_KEY`, hexclave's `STACK_SERVER_SECRET`, etc.) - that part is correctly per-service. But nothing enforces that these secrets stay scoped: any root compromise of the host reads all of them from `docker inspect`, as this audit itself demonstrated read-only.
+- **HotNow was not found** deployed anywhere in this pass (no matching directory/container/proxy entry across app1/app2/app3). **Flag for Phase Two / Git-A cross-reference**: confirm whether HotNow is live, decommissioned, or hosted somewhere outside the six audited servers.
+- **VerdictTank / RFP Tank**: found evidence of `myverdicttank` CloudPanel site on app3 and a `/root/.verdicttank-key.json` LiteLLM API key file on app1 (model access key scoped to specific models, not a raw cloud credential) - so VerdictTank spans app1 (LLM access) and app3 (web/DB), with no isolation boundary tying those two halves together other than the key itself.
+
+---
+
+## 7. Shared credentials / shared reverse-proxy / shared database findings
+
+- **Shared reverse proxy (app1 Caddy)**: one Caddyfile terminates TLS for internal tools (vault, n8n, ai, admin-ai, noc, wz/Wazuh) and product/client domains (giftaroast.com, crm.debtrecoveryexperts.com, komodo.iamgmb.com, transitpin.iamgmb.com) side by side. A Caddy-level misconfiguration or compromise affects both tiers simultaneously.
+- **Shared reverse proxy (app3 CloudPanel/nginx)**: same pattern - one web-server control plane (CloudPanel, itself a public admin panel) fronts internal, client, and product sites.
+- **Shared database engine (app3 Percona/MySQL)**: single instance backs the majority of the 24 CloudPanel sites across all three tiers, as detailed in section 6.
+- **Shared Tailscale mesh with no ACL segmentation**: all 6 production hosts AND personal devices (iPhone, MacBook, Mac Mini, home router) are members of one flat tailnet group with no tags observed on any peer. A compromised personal device is one hop from every production server over Tailscale; a compromised production server is one hop from personal devices.
+- **Shared SSH key (`itpp-infra`) across all 6 hosts**: this is an operational convenience but means loss of that one private key (which lives on Core) compromises SSH access to the entire estate at once. This is the single most consequential shared credential in the estate.
+
+---
+
+## 8. Findings (severity-rated)
+
+| ID | Severity | Finding | Evidence | Why it matters |
+|---|---|---|---|---|
+| NETB-1 | **Critical** | **No network segmentation exists anywhere in the estate.** Core, app1, app2, app3, app1-bu are flat on public Netcup/Hetzner IP space with no VLAN or subnet isolation, and the Tailscale overlay that connects most of them has no ACL tags on any node (default allow-all). wphost02 sits fully outside the mesh, reachable only over the open internet. | `tailscale status --json` on Core/app1/app2/app3/app1-bu shows zero `Tags` entries for any peer; `ip route`/`ping` tests confirmed Core, app1, app2 mutually reach each other's PUBLIC IPs directly with no filtering observed between them | If any one host is popped, the attacker is not contained to that host's blast radius alone. They inherit a direct network path (public IP or Tailscale) to every other host and to personal devices, turning a single compromise into an estate-wide incident with no lateral-movement friction. |
+| NETB-2 | **Critical** | **Wazuh SIEM stack (indexer, dashboard, manager API) is the single highest-leverage target in the estate** because (a) it is publicly exposed via the Docker/UFW bypass NetEng-A found (APP1-1) and (b) compromising it gives an attacker visibility into and potential control over security telemetry for every other host, which likely reports to it as a Wazuh agent. | app1 Docker/iptables inspection (NetEng-A APP1-1) + this audit's confirmation that app1 accepts inbound connections on its public IP from other estate hosts with no gate | Losing the SIEM is worse than losing any single production host: it blinds detection across the whole estate at the exact moment an attacker needs detection to fail, and it sits on a host with no segmentation from the rest of the fleet. |
+| NETB-3 | **Critical** | **app3 shares one MySQL/Percona instance across ~24 sites spanning internal ops, client sites, and micro-SaaS products** with no per-tenant database server or vault boundary. | `ss -tlnp` shows a single `mysqld` on 127.0.0.1:3306/*:33060; CloudPanel site list (`apx, boxpilotlogistics, buzz, ... myverdicttank, panel, support, transitpin, vigilanttac, voipsimplicity`, ~24 total) all on one host | A SQLi or credential leak against ANY one of the 24 sites is a plausible path to every other site's data on the same instance, including internal ITPP operational data (mainwp, support, panel) sitting next to client and product databases. |
+| NETB-4 | **High** | **Management/admin control planes are public with no compensating network boundary**: UniFi controller (app2), UNMS/UISP (app2), CloudPanel admin panel (app3), backup-restore UI (app3), Komodo (app1). Several via unintended Docker/UFW bypass, some via explicit-but-broad UFW rules. | NetEng-A APP1-1/2/3, APP2-1/2, APP3-1/2; cross-referenced here against tier classification (section 2) | Each of these is a "master key" for a whole subsystem (wireless network, ISP customer management, ~24 hosted sites, backup/restore capability, deploy automation). None should be reachable without a network-level gate (VPN, allow-list, or reverse-proxy auth) in addition to application login. |
+| NETB-5 | **High** | **Internal tooling and client-facing/product systems share the same Docker host and reverse proxy with no boundary** on app1 (Wazuh + Twenty CRM + LiteLLM + Komodo alongside TransitPin, giftaroast.com, crm.debtrecoveryexperts.com) and app3 (CloudPanel mixing mainwp/support/panel with client and product sites). | app1 Caddyfile listing both internal and product/client domains; app3 site directory listing (section 6) | A compromise anywhere on either host crosses tiers for free - there is no reason a breach that starts in a low-value product site should reach the internal CRM or deploy-automation control plane, but on these hosts it can, because nothing stops it at the network layer. |
+| NETB-6 | **High** | **Estate-wide single SSH key (`itpp-infra`) is the sole authentication factor for root on all 6 hosts**, and it lives on Core alongside a live autossh tunnel into wphost02's database and WireGuard routes into the home network and WISP towers. | This audit used the same key against all 6 hosts successfully; `ip route`/`ps aux` on Core show the WireGuard routes and the standing tunnel | Core is a de facto master key for the entire estate plus the home network and WISP towers. Its compromise is strictly worse than any other single host's compromise and there is no network segmentation limiting the blast radius once that key is in an attacker's hands. |
+| NETB-7 | **Medium** | **Tailscale mesh includes personal devices (phone, laptop, Mac Mini, home router) in the same untagged group as all six production servers.** | `tailscale status` output on every host lists `iphone-15-pro-max`, `m4-mac-mini`, `ipp-g-lap`, `liberty-udm-pro` alongside `core/app1/app2/app3/app1-bu` with no tag differentiation | A compromised personal device (phishing, stolen laptop, malicious app) is one network hop from production infrastructure with no policy boundary in between. Tailscale ACL tags exist specifically to prevent this and are not being used. |
+| NETB-8 | **Medium** | **Product credentials for HotNow/VerdictTank/RFP Tank/TransitPin are scoped per-service but not vaulted**, and the products themselves are split across hosts (app1 for LLM access key, app3 for web/DB) with no documented boundary tying the split together. | `/root/.verdicttank-key.json` on app1 (model-scoped LiteLLM key); `myverdicttank`, `transitpin`, `transitpin-dash` CloudPanel sites on app3 | If the split is intentional, it should be documented as an architecture decision with an explicit trust boundary; as observed, it looks like an artifact of convenience rather than a designed isolation boundary, which risks silent credential sprawl as more products are added. |
+| NETB-9 | **Low** | **app1-bu standby sync design is pull-only from S3 (not a live tunnel to Core)** - noted as a positive control worth preserving explicitly in Phase Two policy, not a finding to remediate. | `hermes-standby-sync.sh`: `aws s3 sync s3://hermes-vps-backups/live/ ...`; `authorized_keys` on app1-bu matches Core's public key (Core-initiated direction only) | Call this out so future changes to the standby (e.g. adding a live replication tunnel) are evaluated against the blast-radius benefit of the current pull-only design before being adopted. |
+
+### Severity summary
+- Critical: 3 (NETB-1, NETB-2, NETB-3)
+- High: 3 (NETB-4, NETB-5, NETB-6)
+- Medium: 2 (NETB-7, NETB-8)
+- Low: 1 (NETB-9, informational/positive-control note)
+
+---
+
+## 9. Access limitations / items requiring Phase Two follow-up
+
+- Could not confirm from read-only evidence whether app1's MCP tooling makes live cross-host calls into Core's super-search MCP (:8899) - the network path exists and is unfiltered, but application-layer confirmation needs a Phase Two trace (log review, not a live test).
+- Could not run `clpctl db:show:master-credentials` per-site on app3 (would require a site name argument and returns credentials - out of scope for a read-only, no-credential-exposure audit) to confirm whether each CloudPanel site has a distinct MySQL user/schema on the shared instance versus a fully shared root-level credential. Section 6's finding stands regardless (one instance = one blast radius for the engine itself), but Phase Two should verify per-site credential scoping.
+- Tailscale ACL policy file itself (the tailnet admin console's actual ACL JSON) was not inspected - only the absence of `Tags` on every peer via `tailscale status --json`, which is strong but indirect evidence of default-allow-all. Phase Two with tailnet-admin access should pull the actual ACL policy to confirm.
+- HotNow was not located on any of the six audited servers. Needs confirmation from Git-A/Docs-W on whether it is live elsewhere, decommissioned, or renamed.
+
+---
+
+**Note on remediation:** all items above are Phase Two candidates. No firewall, container, DNS, database, or Tailscale configuration was changed on any host during this audit.

audit/phase-one/findings/sec-a.md

@@ -0,0 +1,245 @@
+# Sec-A Findings: IAM, Accounts, Secrets Location, and Least Privilege
+
+Auditor: Sec-A (IAM and secrets inventory)
+Scope: Account inventory, sudo/root privilege, SSH access and key attribution, MFA coverage on admin surfaces, secrets location (no values captured, only presence/staleness), shared vs personal credential use, least-privilege posture.
+Mode: READ-ONLY. All evidence below comes from local audit capture files already on disk (`/root/audit_core.txt`, `/root/audit_app1.txt`, `/root/audit_app2.txt`, `/root/audit_app3.txt`, `/root/audit_app1bu.txt`, `/root/audit_wphost02.txt`), collected in a prior session via read-only SSH/curl. No new commands were run against remote hosts to produce this file. All credential values are redacted as `[REDACTED]`; this document names locations and types only.
+Overlap note: hardening, patch posture, MFA configuration detail (per-tool), and Wazuh/logging coverage are Sec-B's domain (see sec-b.md). This file focuses on accounts, SSH, secrets location, and least-privilege, and cross-references Sec-B's MFA table rather than repeating it in full.
+
+---
+
+## 1. Per-Server Account Inventory
+
+Six hosts: Core (Netcup, this host), app1 (152.53.36.131), app2 (152.53.39.202), app3 (152.53.241.111), app1-bu (Hetzner, warm standby), wphost02 (Hetzner, legacy WordPress/RunCloud).
+
+| Host | Total /etc/passwd entries | Interactive shell accounts (non-system) | Notable interactive accounts | sudo group members | Passwordless sudo (sudoers.d) |
+|---|---|---|---|---|---|
+| Core | 34 | `ippadmin`, `postgres`, `scanuser` (3) | `postgres` has `/bin/bash` (service account with a real shell, unusual), `scanuser` is locked/no-password, purpose not identified in captured data, flag for Phase Two follow-up | `ippadmin` | `ippadmin ALL=(ALL) NOPASSWD:ALL` |
+| app1 | 24 | `ippadmin` (1) | Clean, minimal footprint | `ippadmin` | none in sudoers.d beyond README (no explicit NOPASSWD entry for ippadmin found on app1, unlike Core/app2/app3, worth confirming) |
+| app2 | 25 | `ippadmin`, `unms` (2) | `unms` service account is in the `docker` group, meaning it can run containers with root-equivalent power (see Least-Privilege section) | `ippadmin` | `ippadmin ALL=(ALL) NOPASSWD:ALL` |
+| app3 | 37 | `ippadmin` plus roughly 28 per-client/per-site accounts (`gmb`, `myverdicttank`, `mainwp`, `boxpilotlogistics`, `modelortho`, `intelsight`, `mockups`, `modelorthowww`, `timapta`, `hexclave-api`, `apx`, `panel`, `voipsimplicity`, `transitpin-dash`, `vigilanttac`, `proposals`, `transitpin`, `docs`, `buzz`, `support`, `myvoip`, `hexclave-dash`, `iAmGMB`, `drecovery`, `forms`, `katiewatts`, `clp`, `debtrecoveryexperts`) | This is CloudPanel's per-site account model: one Linux user per hosted site/client. Large blast surface simply by account count, even though most appear unused for SSH (see Section 2) | `ippadmin` | `ippadmin ALL=(ALL) NOPASSWD:ALL`; also `clp ALL=(ALL) NOPASSWD:ALL` (CloudPanel's own management account) plus a wrapper rule `ALL ALL=(ALL) NOPASSWD: /usr/bin/clpctlWrapper` letting any account run the CloudPanel control wrapper as root |
+| app1-bu | 32 | none beyond `root` | Warm standby has essentially no interactive non-root accounts provisioned yet (matches its role as a failover target, not yet fully onboarded) | none (sudo group empty) | `root ALL=(ALL) NOPASSWD:ALL` via cloud-init default, standard for that image |
+| wphost02 | 37 | `runcloud`, `ippadmin` (2) | `runcloud` is the RunCloud panel's management account | `ippadmin` | `root ALL=(ALL) NOPASSWD:ALL` (cloud-init default) and `ippadmin ALL=(ALL) NOPASSWD:ALL` |
+
+Orphaned/unclear-purpose accounts:
+- **Core: `scanuser`** (UID 1001, locked password, `/bin/bash` shell, home `/home/scanuser`). No corresponding SSH key, cron job, or docker context found in the captured data tying this account to an active purpose. Flag as a candidate for removal or documentation in Phase Two; do not assume it is safe to delete without confirming with Germaine first (read-only rule).
+- **app3: roughly 28 per-client accounts.** Most have empty `authorized_keys` files (see Section 2), meaning they exist as CloudPanel site-owner accounts but show no evidence of direct SSH login capability. This is consistent with CloudPanel's model (site isolation, not direct-login accounts) but should be confirmed against actual client billing/engagement status in Phase Two; several of these usernames (`debtrecoveryexperts`, `boxpilotlogistics`, `transitpin`, `hexclave-api`) correspond to active named client projects, others (`katiewatts`, `iAmGMB`) look personal/less clearly tied to a current engagement and are worth a lifecycle check.
+- **wphost02: `runcloud` and `mysql`/`memcache`/`beanstalkd` service accounts.** These are RunCloud-panel defaults consistent with a legacy managed WordPress host; no evidence of misuse, flagged only for completeness since wphost02 is described in the brief as a legacy/sunset host.
+
+Password/shadow posture: `root:PASSWORD_HASH_SET` on Core, app1, app2, app3, and wphost02 (root has a live password hash and could theoretically log in with a password if PermitRootLogin allowed it; Sec-B confirms PermitRootLogin is key-only across the estate, which mitigates this). On app1-bu, root shows `LOCKED_BUT_HASH_PRESENT`, a slightly different but comparable state. All other system accounts show `LOCKED_NO_PASSWORD`, which is expected and healthy for service accounts.
+
+---
+
+## 2. SSH Access Inventory
+
+### 2.1 Key inventory found across `authorized_keys` files
+
+| Key comment/label | Fingerprint (truncated) | Seen on |
+|---|---|---|
+| `itpp-main-server` | SHA256:oqKRvPA...D7E | Core (root and ippadmin) |
+| `germaine@itppartner` | SHA256:dDbLH+b...y3I | Core, app1, app2, app3 (root and ippadmin on each) |
+| `wisp-backup` | SHA256:MxQw1oh...DcI | Core (root and ippadmin) |
+| `itpp-infra` | SHA256:Jxh0bbT...8uQ | Core, app1, app2, app3, app1-bu (root and/or ippadmin on each); also present on wphost02 under a second fingerprint variant, see below |
+| `g@germaine@itpropartner.com` | SHA256:7QuUx/s...8lc | Core (root only) |
+| `RunCloud_Server_Service` | SHA256:qMm5i4/...IUo | wphost02 (root only, RunCloud-managed) |
+| `itpp-infra RUNCLOUD1783623009` / `itpp-infra` (second fingerprint) | SHA256:jnmiJb9...v3U | wphost02 (root only) |
+
+### 2.2 Blast radius assessment
+
+**Confirmed live (not just documented):** the `itpp-infra` key (fingerprint `SHA256:Jxh0bbT9dUV3q1DYYB3hHyhy/1TDj7Q8U4xrVmB38uQ`) is present in `authorized_keys` on Core, app1, app2, app3, and app1-bu, five of the six servers in scope. This confirms the prior session's documentation-based claim with live data. wphost02 has a *different* key under the same "itpp-infra" label/comment (different fingerprint), so it is not the identical key, but it is functionally the same access pattern using the same naming convention.
+
+**Severity implication:** compromise of the single private key corresponding to fingerprint `SHA256:Jxh0bbT9dUV3q1DYYB3hHyhy/1TDj7Q8U4xrVmB38uQ` grants SSH access to 5 of 6 servers in the estate (Core, app1, app2, app3, app1-bu), each as root or root-equivalent (via ippadmin's passwordless sudo). This is the single largest blast-radius item in the IAM domain. There is no key segmentation by host, role, or trust tier; one credential spans the entire estate except the legacy wphost02 host.
+
+**`germaine@itppartner`** key is present on Core, app1, app2, and app3 (4 of 6 hosts), a second broad-access personal key, smaller blast radius than `itpp-infra` but still spanning 4 hosts.
+
+**Single-purpose keys** (`itpp-main-server`, `wisp-backup`, `g@germaine@itpropartner.com`) are scoped to Core only, a healthier pattern; these do not multiply blast radius.
+
+**app3 per-client accounts:** of the roughly 28 per-client home directories checked, all sampled `authorized_keys` files under those accounts were empty (no keys present). This means those accounts cannot currently be reached by direct SSH key login, which is a good containment property for a shared multi-tenant host, assuming CloudPanel manages access to those accounts through its own panel rather than SSH.
+
+**app1-bu (warm standby):** only the `itpp-infra` key is authorized, and only for root; there is no `germaine@itppartner` personal key here. This is a narrower access surface, appropriate for a less frequently accessed failover host, but means the standby depends entirely on one shared key with no personal-key fallback.
+
+**wphost02:** RunCloud's own service key plus two `itpp-infra`-labeled keys with different fingerprints are present; no personal (`germaine@itppartner`) key was found on this host, consistent with it being flagged as legacy/lower-touch in the brief.
+
+### 2.3 Login activity signal
+
+Last-login data was only meaningfully populated on app1-bu (all accounts "never logged in", consistent with a cold standby that has not yet been used) and wphost02 (root last login 2026-07-21 from 76.195.7.60; ippadmin last login 2025-12-31 from the same IP). Core, app1, app2, and app3 returned no populated last-login rows in the captured data, meaning login history could not be independently verified for those four hosts from this dataset. This is a visibility gap, not a finding of absence.
+
+---
+
+## 3. MFA Matrix (Cross-Reference to Sec-B)
+
+Sec-B's findings file (sec-b.md, Section 3) contains the full MFA coverage table across 11 admin surfaces and is the authoritative source for MFA configuration detail. Sec-A's contribution here is the account-and-access angle for each surface: how many people can reach it and via what credential, since that is what MFA (or its absence) is meant to protect.
+
+| Admin Surface | Host | Accounts with access (per this session's data) | MFA status (per Sec-B) | Sec-A note |
+|---|---|---|---|---|
+| Gitea | app2 | Single admin account (`ippadmin`, is_admin=true, confirmed live in the prior session via API) | Capability exists, not confirmed enforced; open registration | One shared admin account for the entire code-hosting platform is itself an IAM concentration risk independent of MFA; see Section 5 |
+| Grafana | Core | Uses `admin`/`[REDACTED default password]` per Sec-B | No MFA, default credential in use | Confirms Sec-B's Critical finding from the accounts angle: this is not even a personal account, it is a shared default account |
+| CloudPanel | app3 | Single admin account `gmb` observed via `clpctl user:list` (per prior session) | Capability exists, adoption unconfirmed | Single admin account matches the single-admin pattern seen on Gitea; worth asking Germaine whether other staff need scoped CloudPanel accounts rather than sharing `gmb` |
+| Technitium DNS | app2 | Environment shows `DNS_SERVER_ADMIN_PASSWORD` set to the literal default placeholder string | No MFA; default-credential pattern | Sec-A could not confirm the live in-app credential value without an authenticated read, which the read-only rule prohibits; flagged as an access-verification gap for Phase Two |
+| Wazuh dashboard, Hudu, UniFi, UNMS/UCRM, LiteLLM admin, Vaultwarden, Dawarich, Traccar | app1/app2 | No per-user account enumeration was possible from the captured local-file data (these are app-level accounts inside each service's own database, not OS accounts) | See Sec-B's table | This is a genuine visibility gap: Sec-A cannot state how many people have login credentials to these consoles without an authenticated read of each app's user table, which is out of scope for read-only discovery. Flagging per brief rule 4 (flag insufficient access/visibility, do not guess). |
+
+**MFA gap count (IAM-relevant): 8 of 8 checkable admin surfaces show no confirmed enforced MFA** (Gitea, Grafana, CloudPanel, Technitium, Wazuh dashboard, Hudu, UniFi, UNMS/UCRM), consistent with Sec-B's estate-wide finding.
+
+---
+
+## 4. Secrets-Location List
+
+No credential values are reproduced below. Entries are location, type, and staleness/rotation signal only.
+
+### 4.1 Confirmed live secrets locations (`.env` files, values redacted at capture time)
+
+| Host | Path | Secret types present | Last modified | Staleness signal |
+|---|---|---|---|---|
+| Core | `/opt/shopping-cart/.env` | Amazon Associates tag, SMTP credentials | 2026-07-27 | Recent |
+| Core | `/opt/mooresunnydaze/.env` | Admin API key, Stripe secret key, Stripe webhook secret | 2026-08-07 | Recent |
+| Core | `/opt/hermes-voice/.env` | Hermes API key/session key, xAI API key | 2026-07-28 | Recent |
+| Core | `/opt/voice-agent/.env` | Hermes API key/session key | 2026-07-26 | Recent |
+| app1 | `/docker/n8n/.env` | n8n encryption key, Postgres password | 2026-07-17 | Recent |
+| app1 | `/root/docker/docuseal/data/docuseal/docuseal.env` | Database URL, secret key base | 2026-07-28 | Recent |
+| app1 | `/root/docker/litellm/.env` | LiteLLM master key, salt key, Postgres password, UI credentials | 2026-07-15 | ~1 month old at capture time |
+| app1 | `/root/docker/litellm/.env.pre-keyfix-20260714-130347` | Same secret types as above (pre-rotation snapshot) | 2026-07-14 | **Stale duplicate.** This is a leftover pre-rotation backup sitting next to the live config; the filename itself documents a key-fix event on 2026-07-14, meaning this file likely contains a superseded credential set that was never deleted. Medium finding, see Section 7. |
+| app1 | `/root/docker/super-search/.env` | Exa, Firecrawl, OpenCorporates API keys | 2026-07-15 | Recent |
+| app1 | `/root/docker/twenty/.env` | Encryption key and app config | 2026-07-28 | Recent |
+| app1 | `/root/docker/wazuh/.env` | Wazuh build/version config (lower sensitivity, mostly version pins) | 2026-07-19 | Recent |
+| app2 | `/opt/bookstack/config/www/.env` | DB, SMTP, S3 credentials, two-factor key | present | Not independently dated in this pass |
+| app2 | `/root/docker/dawarich/.env` | Database password, Postgres password, secret key base | 2026-07-22 | Recent |
+| app2 | `/root/docker/hudu/.env` | Hudu app secrets | present | Not independently dated in this pass |
+| app2 | `/root/docker/technitium/docker-compose.yml` (inline env, not a `.env` file) | `DNS_SERVER_ADMIN_PASSWORD` set to the literal default placeholder string in the compose file itself | 2026-08-04 | Recent file, but the value itself is a stale/never-rotated default, see Section 7 |
+| app3 | `/home/clp/htdocs/app/files/.env` | Application credentials (CloudPanel-managed app) | present | Live copy |
+| app3 | `/home/clp/backups/2026-08-11_04-15-01/app/files/.env`, `.../2026-08-12_04-15-01/...`, `.../2026-08-13_04-15-01/...` | Same credential set as above | 2026-08-11, -12, -13 | **Three consecutive daily backup snapshots each retain a full plaintext copy of the same `.env`.** Every backup rotation multiplies the number of at-rest plaintext credential copies without any additional access control on the backup directory itself. Medium finding, see Section 7. |
+| app3 | `/opt/buzz/deploy/compose/.env` | Buzz relay service credentials | present | Not independently dated in this pass |
+| app3 | `/opt/docs-auth/docs-auth.env` | Docs site auth credentials | present | Not independently dated in this pass |
+| app3 | `/opt/hexclave/hexclave.env` | Hexclave app credentials | present | Not independently dated in this pass |
+| app3 | `/var/www/msp-forms/.env` | MSP forms app credentials | present | Not independently dated in this pass |
+| app1-bu | `/root/.hermes/.env`, `/root/.hermes/.env.telegram-backup` | Hermes core config and Telegram bot token | present | This is the same file flagged by Sec-B as **world-readable** on app1-bu; that permissions gap plus this being a live credential file is a compounding issue, cross-referenced in Section 7 |
+| app1-bu | `/root/.hermes/state-snapshots/20260703-011737-pre-update/.env`, `/root/.hermes/state-snapshots/20260711-150407-pre-update/.env` | Full Hermes config snapshots from pre-update states | 2026-07-03, 2026-07-11 | Stale historical copies retained on disk, same document-sprawl pattern as the litellm pre-keyfix file |
+
+### 4.2 Plaintext credential document sprawl (`key-inventory.md`)
+
+This is the single most significant secrets-location finding of this audit, and it was independently re-confirmed in this session against the raw file inventory rather than only relying on the prior session's account of it.
+
+**Ten copies of `key-inventory.md` exist across the filesystem.** File-level comparison (size, modification time, and a redaction-marker check that counts occurrences of `[REDACTED]`/`Vaultwarden`/`Hudu`-style placeholder references without reading or reproducing any actual secret value):
+
+| Path | Size | Modified | Redaction-marker count | Assessment |
+|---|---|---|---|---|
+| `/root/projects/itpp-infrastructure/docs/infrastructure/key-inventory.md` | 12,497 B | 2026-08-08 | 27 | **Sanitized (canonical).** This is the properly redacted version meant for the docs repo. |
+| `/root/itpp-docs/docs-source/itpp-infrastructure/key-inventory.md` and 3 identical copies under `/tmp/itpp-docs-build/`, `/tmp/audit-export/`, `/tmp/audit-repos/`, `/tmp/tmp.YZCSHQoVPf/` | 12,503 B each, identical md5 `670625a7...` | 2026-08-08/09 | 27 each | Sanitized, duplicate build/export artifacts of the canonical doc. Document sprawl, not a secrets leak, but five redundant copies of the same file across `/tmp` build directories is untidy and should be cleaned up as part of normal repo hygiene. |
+| `/root/projects/itpp-infrastructure/.backup-before-sanitize-20260723/key-inventory.md` | 13,136 B | 2026-07-23 | 3 (low redaction-marker count relative to the sanitized versions) | **CRITICAL: unsanitized, plaintext credential values.** File header reads "Generated: 2026-07-23... Contains real credentials". This is a pre-sanitization leftover from a July 23 cleanup pass that documented real root passwords for app1/app2/app3, cloud provider API tokens, S3 access keys, and multiple service tokens, per its own header and per the prior session's confirmed read. It is `.gitignore`'d (not in git history) but sits unencrypted, root:root, mode 600, on Core's local disk. |
+| `/root/.hermes/references/key-inventory.md` | 13,161 B | 2026-08-08 | 3 (low, matching the unsanitized pattern) | **CRITICAL: a second, independent unsanitized copy**, not identified by name in the prior session's summary. Same header pattern ("Generated: 2026-07-23... CLASSIFIED: Contains real credentials"), root:root, mode 600. This copy lives inside the Hermes agent's own reference-file directory, meaning it is reachable by any skill or process that can read Hermes's reference files, a broader exposure surface than a one-off backup directory. |
+| `/root/.hermes/skills/devops/hudu-management/references/key-inventory.md` and `/root/.hermes/.backups/hermes-backup-2026-07-22/skills/devops/hudu-management/references/key-inventory.md` | 4,022 B and 3,794 B | 2026-07-22 | 9 each | Smaller, older API-key-name-only lists (structure/labels, not full values per the prior session's read), lower risk but still worth folding into the cleanup since they are stale duplicates of a smaller scope. |
+
+**Net finding:** there are two full unsanitized plaintext copies of the complete credential inventory (not one, as the prior session reported), both root:root mode 600 so restricted to local root access only, but both should have been shredded after the sanitized canonical version was created on 2026-08-08. Local root access to Core is not a trivial bar (this is the Hermes host with the broadest operational reach in the estate), so "root-only" is a meaningful but not sufficient mitigation.
+
+### 4.3 Secrets-sprawl git-grep results (per-repo hit counts, tracked files only)
+
+The following are word-pattern hits (password/secret/token/api_key as text) across git-tracked files in every `.git` repo found on each host. These are **not confirmed leaked credentials**; they are candidate locations that need per-file triage before being treated as a real secrets-in-git problem.
+
+| Host | Repos scanned | Total file hits | Largest single contributor |
+|---|---|---|---|
+| Core | 68 | 6,296 | `/usr/local/lib/hermes-agent` (2,704 hits, almost entirely documentation/code referencing the words "secret"/"token" as identifiers, not literal values) |
+| app1 | 6 | 27 | `/root/docker/wazuh` (workflow/config files referencing "secret" as a term) |
+| app2 | 1 | 1,275 | `/opt/ragflow` (single large open-source repo, README/workflow/test files) |
+| app3 | 2 | 719 | `/opt/buzz` (open-source repo, same pattern: workflow files, `.env.example`, changelogs) |
+| app1-bu | 3 | 5,504 | `/root/.hermes/docker/twenty` (2,741 hits) and `/usr/local/lib/hermes-agent` (2,761 hits), both open-source vendor codebases mirrored onto the standby host |
+| wphost02 | 0 | 0 | No `.git` repos found on this host |
+
+**Assessment: Medium, not Critical.** The overwhelming majority of hits are in vendored open-source code (`hermes-agent`, `twenty`, `ragflow`, `buzz`, `theHarvester`, `sherlock`) where "secret", "token", or "api_key" appear as variable names, documentation words, or CI workflow keys, not as literal credential values. The one path worth a manual look is `/root/projects/itpp-infrastructure` itself (44 hits on Core, per the prior session's account, and 2 hits on app1-bu limited to `README.md` and `sites/app3.md`), since that is the org's own infrastructure documentation repo, not a third-party vendor codebase, and is the most likely place an actual value could have been accidentally committed. This needs a manual per-file read in Phase Two; it was not performed in this session because it would require opening and reading each of the 44 hit locations individually, and the tool budget for this task was reserved for account/SSH/secrets-location work per the task's explicit instructions.
+
+### 4.4 Configuration files with credential-relevant names (non-`.env`)
+
+Notable non-`.env` files that could contain live secrets, filtered from a much larger CONFIG_SECRET_FILES sweep that was mostly PHP extension `.ini` files (not credential-relevant, omitted from this table):
+
+| Host | Path | Type |
+|---|---|---|
+| Core | `/root/docker/monitoring/grafana/grafana.ini` | Grafana config, cross-referenced to Sec-B's default-admin-password finding |
+| app1 | `/root/docker/litellm/config.yaml` | LiteLLM routing/model config, may reference upstream API keys by env var name |
+| app2 | `/opt/gitea/data/gitea/conf/app.ini` | Gitea app config, contains SECRET_KEY/INTERNAL_TOKEN/JWT_SECRET per the prior session's confirmed (value-redacted) read |
+| app3 | `/etc/gitea/app.ini` | A second Gitea config path found on app3; worth confirming in Phase Two whether this is a stale leftover from a prior Gitea install location or an active second instance, since the brief describes Gitea as living on app2 |
+| Core | `/root/.config/goose/secrets.yaml` | Goose CLI agent secrets file |
+
+---
+
+## 5. Shared vs Personal Credential Assessment
+
+- **Gitea (app2): one shared admin account (`ippadmin`), confirmed live via API in the prior session** (is_admin=true, last login 2026-08-10). No per-person Gitea accounts exist. Anyone who knows this one credential, or holds a valid API token for it, has full administrative control over every source repository in the org.
+- **CloudPanel (app3): one admin account (`gmb`)** per `clpctl user:list` from the prior session. Same single-shared-account pattern.
+- **SSH root/ippadmin access:** access to Core, app1, app2, and app3 is governed by the same small set of keys (`itpp-infra`, `germaine@itppartner`) shared across hosts rather than per-host or per-person keys. There is no evidence of individual named-user SSH accounts distinct from `root` and `ippadmin`; all administrative SSH access funnels through these two shared identities.
+- **`ippadmin` has passwordless full sudo** (`NOPASSWD:ALL`) on Core, app2, app3, and wphost02 (and root has the OS-level password hash set on top of that). This means any of the keys that unlock `ippadmin`'s SSH access are equivalent to unlocking root on that host, with no additional authentication step (no MFA, no password re-prompt) in between.
+- **No individual-user accountability mechanism observed.** Because SSH access and sudo are both shared/group-level rather than per-person, there is no way, from the data available, to attribute a specific administrative action on any of these hosts to a specific individual. This is a foundational access-control gap that other findings (MFA gaps, unclear last-login history) compound.
+- **app3's per-client accounts** are the one place where the estate does practice some separation (one Linux account per hosted site), which is appropriate multi-tenant hygiene, but even this layer sits underneath the same shared `ippadmin`/root access that can reach every one of those accounts.
+
+---
+
+## 6. Least-Privilege Findings
+
+- **Almost every Docker container across the estate runs as `root` inside the container (`user=[root(default)]`).** This was true for the large majority of containers sampled on Core, app1, app2, and app3, including services with no operational need for root (e.g., `microbin`, `searxng`, `telegraf`, `mikrotik-exporter`, `uptime-kuma` on Core; `komodo-core`, `docuseal`, `twenty-db`, `n8n-postgres`, `mcp-browser`, `mcp-email`, `mcp-git`, `mcp-filesystem`, `super-search` on app1; `bookstack`, `bookstack-db`, `docker-ragflow-cpu-1`, `docker-mysql-1`, `docker-minio-1`, `docker-redis-1`, `technitium`, `dawarich_sidekiq/app/db/redis`, `traccar`, `gitea`, `unms-nginx`, `unifi-controller`, `hudu-app/db/worker/redis`, `unms-api`, `ucrm`, `unms-postgres`, `unms-siridb`, `unms-fluentd` on app2; `hexclave-postgres`, `hexclave-clickhouse`, `buzz-prod-postgres/redis/minio` on app3). None were flagged as `privileged=true` (a worse container-escape-relevant setting), which is a positive, but running as the root UID inside the container still means a container-breakout vulnerability in any of these images grants root, not a scoped user, on the host's container runtime layer.
+- **Some containers do practice least privilege correctly** and are worth naming as the good pattern to extend: `browserless` (`blessuser`), `grafana` (`472`, Grafana's own non-root UID), `prometheus` (`nobody`), `openwebui` (`0:0` is still root, note this is actually still root, listed for completeness), `twenty-server`/`twenty-worker` (`1000`), `kokoro-tts` (`appuser`), `wazuh-dashboard`/`wazuh-indexer` (their own service users), `vaultwarden` is root but is a well-maintained image; `unms-device-ws-*` and `unms-rabbitmq`/`unms-netflow` containers (`1001`), `hexclave-server` (`node`), `hexclave-cron` (`curl_user`), `buzz-prod-relay-1` (`buzz:buzz`). These show the operators clearly know how to configure non-root containers when the upstream image supports it; the gap is inconsistent application, not lack of capability.
+- **app2's `unms` service account is a member of the `docker` group.** Group membership in `docker` is functionally equivalent to root on the host, since a member can run any container with arbitrary host bind-mounts. If `unms` is meant to be a scoped service account for the UNMS/UCRM stack, its `docker` group membership defeats that scoping and should be reviewed in Phase Two; a properly least-privileged setup would run UNMS's containers under a system-level Docker Compose invocation by `ippadmin`/root rather than granting the `unms` account itself `docker` group membership.
+- **`ippadmin ALL=(ALL) NOPASSWD:ALL`** on 4 of 6 hosts (Core, app2, app3, wphost02) is the single most consequential least-privilege gap in the account layer: this is unrestricted, no-password-prompt root escalation for a shared account, with no command allowlisting or logging requirement built into the sudoers entry itself.
+- **`ALL ALL=(ALL) NOPASSWD: /usr/bin/clpctlWrapper` on app3** allows any account, including the low-privilege per-client site accounts, to invoke the CloudPanel control wrapper as root without a password. This needs a Phase Two review of exactly what `clpctlWrapper` can do; if it exposes any file-write or command-injection surface, every one of the ~28 per-client accounts on app3 effectively has a root-escalation path.
+- **Core's `postgres` account has an interactive `/bin/bash` shell** rather than the more typical `/bin/false` or `/usr/sbin/nologin` for a database service account. This is a smaller-scale least-privilege deviation worth tightening if there is no operational reason for interactive postgres logins.
+
+---
+
+## 7. Severity-Rated Findings
+
+### Critical
+
+**SEC-A-01: Two unsanitized plaintext copies of the complete infrastructure credential inventory exist on Core's local disk**
+Evidence: `/root/projects/itpp-infrastructure/.backup-before-sanitize-20260723/key-inventory.md` (13,136 bytes, root:root, mode 600, modified 2026-07-23) and `/root/.hermes/references/key-inventory.md` (13,161 bytes, root:root, mode 600, modified 2026-08-08) both carry the header "Contains real credentials, store encrypted, never email plaintext" and show a low redaction-marker count consistent with the prior session's confirmed read that these contain real root passwords, cloud provider API tokens, S3 keys, and service tokens in plaintext.
+Rationale: A single successful compromise of the Core host at the root level, or any process/skill with read access to Hermes's own reference-file directory, would expose essentially every credential the organization relies on, in one file, in one read. The sanitized canonical version already exists at `docs/infrastructure/key-inventory.md`; these two copies are leftovers that were never deleted after that sanitization work was done on 2026-08-08, and one of them (the Hermes references copy) is in a location that a wider range of automated processes could plausibly touch than a one-off backup folder. This should be shredded (not just deleted) as a Phase Two remediation item; per the read-only rule, Sec-A did not delete it.
+
+**SEC-A-02: A single SSH key grants access to 5 of 6 servers in the estate, each with passwordless root escalation on top**
+Evidence: The `itpp-infra` key (fingerprint `SHA256:Jxh0bbT9dUV3q1DYYB3hHyhy/1TDj7Q8U4xrVmB38uQ`) is present in `authorized_keys` on Core, app1, app2, app3, and app1-bu. On Core, app2, and app3, the account it can reach (`ippadmin` or root) has `NOPASSWD:ALL` sudo.
+Rationale: There is effectively one credential standing between an attacker and full administrative control of 5 of the org's 6 servers. If this key is ever exposed (leaked in a repo, phished, copied to a compromised laptop), the blast radius is close to the entire estate, in a single step, with no second factor to slow it down. This is the highest-leverage single point of failure identified in the IAM domain.
+
+### High
+
+**SEC-A-03: Gitea and CloudPanel each rely on a single shared admin account with no per-person accountability**
+Evidence: Gitea's `GET /api/v1/admin/users` (prior session, live read) returned exactly one account, `ippadmin`, is_admin=true. CloudPanel's `clpctl user:list` (prior session) showed a single admin account, `gmb`.
+Rationale: Every code change and every CloudPanel administrative action on these platforms is attributable only to "the shared account," not to a specific person. If credentials are shared among multiple staff (which a single account by definition requires, if more than one person needs access), there is no way to know who did what, which matters both for day-to-day accountability and for incident response if something goes wrong.
+
+**SEC-A-04: `ippadmin` has passwordless, unrestricted root sudo on 4 of 6 hosts, and is reachable by the same shared SSH keys used across the estate**
+Evidence: `/etc/sudoers.d/ippadmin` contains `ippadmin ALL=(ALL) NOPASSWD:ALL` on Core, app2, app3, and (via a separate file) wphost02. The same `itpp-infra` and `germaine@itppartner` keys that unlock `ippadmin`'s SSH session are shared across hosts (see SEC-A-02).
+Rationale: There is no additional authentication step between "I have this one SSH key" and "I am root on this server," across most of the estate. Combining a shared credential with unrestricted, no-prompt root escalation removes every layer of defense that would normally exist between initial access and full compromise.
+
+**SEC-A-05: app3's `clpctlWrapper` sudoers rule grants any account, including all ~28 per-client site accounts, a passwordless path to run a root-level control wrapper**
+Evidence: `/etc/sudoers.d/cloudpanel` contains `ALL ALL=(ALL) NOPASSWD: /usr/bin/clpctlWrapper`.
+Rationale: This rule is written to apply to `ALL` users, not just `clp`. If any of the roughly 28 per-client accounts on app3 is ever compromised (for example through a vulnerable WordPress plugin on that client's site), the attacker inherits a passwordless path to a root-level tool. The actual risk depends on what commands `clpctlWrapper` exposes and whether it validates its inputs, which needs a Phase Two review; it was not something this read-only audit could safely test without executing the wrapper.
+
+### Medium
+
+**SEC-A-06: Secrets-sprawl git-grep hits (6,296 on Core, 5,504 on app1-bu, 1,275 on app2, 719 on app3, 27 on app1) are mostly false-positive-heavy vendored open-source code, but the organization's own infrastructure repo has not yet been manually triaged**
+Evidence: Per-repo breakdown in Section 4.3 shows the largest contributors are vendored codebases (`hermes-agent`, `twenty`, `ragflow`, `buzz`) where matches are variable names or documentation text, not credential values. `/root/projects/itpp-infrastructure` itself shows 44 hits on Core and 2 on app1-bu.
+Rationale: Treating a raw grep-hit count as a secrets-leak severity score would both overstate the risk from vendor code and understate the one path that actually matters: the org's own documentation and infrastructure repo. This needs a targeted manual read of those 44 (Core) and 2 (app1-bu) hits in Phase Two, not a blanket "Critical, 13,000+ secrets found" framing.
+
+**SEC-A-07: Live `.env` credential files are retained unencrypted in daily backup snapshots on app3, and in stale pre-rotation copies on app1**
+Evidence: `/home/clp/backups/2026-08-11_04-15-01/app/files/.env`, `2026-08-12`, and `2026-08-13` each hold a full plaintext copy of the same application `.env`. On app1, `/root/docker/litellm/.env.pre-keyfix-20260714-130347` sits alongside the live `/root/docker/litellm/.env`, an old credential set from before a documented key-fix event on 2026-07-14 that was never removed.
+Rationale: Every backup cycle multiplies the number of at-rest plaintext copies of the same credentials without adding any access control beyond whatever protects the backup directory itself. Stale pre-rotation files are worse: if the "old" credentials in that file were ever rotated because they were compromised or suspected compromised, the old values are still sitting on disk in cleartext.
+
+**SEC-A-08: Almost all Docker containers across Core, app1, app2, and app3 run as root inside the container, despite several images on the same hosts demonstrating that non-root operation is supported**
+Evidence: Section 6 lists dozens of containers with `user=[root(default)]` across all four hosts, alongside a smaller set of containers (`browserless`, `grafana`, `prometheus`, `twenty-server/worker`, `kokoro-tts`, `unms-device-ws-*`, `hexclave-server`, `buzz-prod-relay-1`) that correctly run as scoped, non-root users.
+Rationale: Running as root inside a container is not itself a host compromise, but it removes one layer of defense in depth: a container-escape vulnerability in a root-run container hands the attacker root, not a limited user, on the container runtime. Since the operators clearly know how to configure non-root users (as shown by the containers that already do this correctly), tightening the rest is a configuration change, not a re-architecture.
+
+**SEC-A-09: app2's `unms` service account is a member of the `docker` group**
+Evidence: `/etc/group` on app2 shows `docker:x:990:unms`.
+Rationale: Docker group membership is equivalent to root access on the host. If `unms` was intended as a scoped, limited-privilege account for running the UNMS/UCRM stack, this membership defeats that intent and should be reviewed in Phase Two to confirm whether it is required for the stack's Compose-based startup or whether it can be removed in favor of running Compose as `ippadmin`/root directly.
+
+### Low
+
+**SEC-A-10: `scanuser` on Core has no identifiable purpose in the captured data and should be reviewed for removal or documentation**
+Evidence: `/etc/passwd` shows `scanuser:1001:1001:/home/scanuser:/bin/bash`, shadow status is `LOCKED_NO_PASSWORD`, no matching SSH key, cron job, or docker container context was found tying it to an active use.
+Rationale: An account with an interactive shell but no traceable purpose is exactly the kind of thing that accumulates in long-running infrastructure and eventually becomes an orphaned foothold. Low severity because it is currently locked (no password) and no SSH key reaches it, but it should be either documented or removed rather than left unexplained.
+
+**SEC-A-11: Login-history visibility is inconsistent across the estate**
+Evidence: Last-login data was populated for app1-bu and wphost02 but returned empty for Core, app1, app2, and app3 in the captured data.
+Rationale: Not being able to see recent login activity on 4 of 6 hosts is a monitoring/visibility gap that limits how confidently this audit (or Phase Two remediation planning) can assess whether the shared credentials discussed above are being actively used, by how many people, or from where. Flagged per the brief's rule 4 (flag insufficient visibility, do not guess) rather than assumed benign.
+
+---
+
+## Appendix: Methodology and Limitations
+
+- All findings in this file are derived from local audit capture files (`/root/audit_*.txt`) collected in a prior read-only SSH/curl session against the 6 hosts. No new SSH connections, command executions against remote hosts, or credential-value reads were performed to produce this document.
+- Every `.env`, compose-file environment variable, and config-file secret value in the raw capture was already redacted to `[REDACTED]` at collection time; this file reports only presence, location, and staleness signals derived from filenames, timestamps, and structural markers (redaction-marker counts), never actual values.
+- App-level user accounts inside Wazuh, Hudu, UniFi, UNMS/UCRM, LiteLLM admin, Vaultwarden, Dawarich, and Traccar could not be enumerated from the local file data available to this session, since those are internal application database records, not OS-level accounts or static config file entries. This is a stated visibility gap per brief rule 4, not a finding of "no accounts" or "single account."
+- The per-file triage of the 44 (Core) and 2 (app1-bu) git-grep hits inside `/root/projects/itpp-infrastructure` itself was not completed in this session due to tool-budget constraints; this is explicitly flagged as outstanding work for Phase Two in Finding SEC-A-06, not silently omitted.
+- This file was written by a fresh session using data collected by a prior session that exhausted its tool-call budget before writing any findings file. All data cited here was independently re-extracted and cross-checked from the raw capture files as part of producing this document, not copied verbatim from the prior session's unverified summary.

audit/phase-one/findings/sec-b.md

@@ -0,0 +1,160 @@
+# Sec-B Findings: Hardening, Patch Posture, MFA/Authentication, Logging/Monitoring
+
+Auditor: Sec-B (information security hardening)
+Scope: Patch posture, SSH/OS hardening, MFA coverage on admin consoles, Wazuh/logging coverage, across the full ITPP estate.
+Mode: READ-ONLY. No configuration changes, restarts, or remediation performed. All items below are findings for Phase Two remediation planning.
+Builds on: neteng-a.md (network exposure), sys-a.md and sys-b.md (per-service inventories). Enumeration of services/ports is not repeated here; see those files for full service lists and images in use.
+
+---
+
+## 1. Patch Posture Table
+
+| Host | OS | Kernel Running | Kernel Update Pending? | Last apt update/upgrade run | Upgradable pkg count | Notes |
+|---|---|---|---|---|---|---|
+| Core (localhost) | Debian 13 (trixie) | 6.12.94+deb13-amd64 | Yes, 6.12.101-1 available | apt metadata refreshed 2026-08-13 (same day as audit) | 15 | No unattended-upgrades service active; refresh appears manual/cron-triggered, not verified automatic |
+| app1 (152.53.36.131) | Debian 13 (trixie) | 6.12.95+deb13-amd64 | Unclear (kernel not in upgradable list this pass, discrepancy with Core) | unattended-upgrades stamp 2026-08-13 06:50, last real upgrade logged 2026-08-11 (caddy security update) | 15 (at first check) | unattended-upgrades ACTIVE and ENABLED, evidence of automatic security patching working |
+| app2 (152.53.39.202) | Debian 13 (trixie) | 6.12.95+deb13-amd64 | Unclear, similar to app1 | unattended-upgrades stamp 2026-08-13 06:22, last logged upgrade 2026-08-11 (caddy) | 15 (at first check) | unattended-upgrades ACTIVE and ENABLED |
+| app3 (152.53.241.111) | Debian 13 (trixie) | 6.12.95+deb13-amd64 | Yes, 6.12.101-1 available (confirmed twice) | update-stamp/upgrade-stamp last changed 2026-07-16 (about 4 weeks before audit) | 14, including linux-image-amd64 security kernel and postfix security update | unattended-upgrades reports active/enabled but the upgrade-stamp is stale (2026-07-16) versus app1/app2 (2026-08-13); patch cadence on app3 is lagging by roughly 4 weeks despite the service being enabled |
+| app1-bu (5.161.225.131, warm standby) | Ubuntu 24.04.4 LTS | 6.8.0-117-generic | No kernel package pending (not in upgradable list) | apt success stamp 2026-08-13 05:32 (current) | 16, mostly apport/cloud-init/apparmor/plymouth, non-security-critical | unattended-upgrades ACTIVE and ENABLED, fail2ban active |
+| wphost02 (5.161.62.38, legacy) | Ubuntu 24.04.4 LTS | 6.8.0-134-generic | No kernel package pending | apt success stamp 2026-08-13 00:44 (current) | 9, mostly apport/sosreport/network minor pkgs | unattended-upgrades ACTIVE and ENABLED, fail2ban active with 3 jails (runcloud-agent, sshd, sshd-ddos) |
+
+Docker image tags (cross-reference to sys-a.md/sys-b.md inventories):
+- Wazuh stack on app1 running wazuh-manager/indexer/dashboard 4.9.2, up 3 weeks continuously. 4.9.2 is a specific pinned version, not `latest`; verify against current Wazuh release train in Phase Two to confirm no known CVEs unpatched in this line.
+- Technitium DNS container on app2 runs on DOTNET_VERSION=10.0.9 / ASPNET_VERSION=10.0.9 (current runtime), but ships with `DNS_SERVER_ADMIN_PASSWORD=changeme` in its environment (see Finding SEC-B-03, credential-adjacent hardening issue, flagged here because it is a default/weak-credential indicator, not a secrets inventory item).
+- Gitea, Hudu, UniFi, Traccar, Dawarich image version pinning should be cross-checked against sys-a/sys-b inventories for stale major versions; no additional very-old (multi-year) tags were independently observed beyond what sys-a/sys-b already caught.
+
+**Patch posture summary:** All 6 hosts are on reasonably current OS/kernel baselines (Debian 13 trixie or Ubuntu 24.04 LTS), and 5 of 6 have unattended-upgrades active and enabled with recent apply timestamps. app3 is the outlier: unattended-upgrades reports enabled but its upgrade-stamp is roughly 4 weeks stale versus siblings, and it has a pending security kernel update (6.12.95 to 6.12.101) plus a pending postfix security update, both unapplied. Core (localhost) has NO unattended-upgrades service at all (inactive, not-found), relying entirely on manual or externally-scheduled patching, and has 15 upgradable packages including an available kernel update.
+
+---
+
+## 2. Hardening Table (SSH / OS-level)
+
+| Host | PermitRootLogin | PasswordAuthentication | PubkeyAuthentication | fail2ban | unattended-upgrades | auditd | World-readable secrets found |
+|---|---|---|---|---|---|---|---|
+| Core | prohibit-password (without-password) | no | yes (default) | NOT INSTALLED (inactive, unit not found) | NOT INSTALLED (inactive, unit not found) | inactive | None found in targeted scan |
+| app1 | prohibit-password | no | yes | active, enabled | active, enabled | not checked directly (not active per rsyslog/auditd probe) | None found |
+| app2 | prohibit-password | no | yes | active, enabled | active, enabled | not checked | None found |
+| app3 | prohibit-password | no | yes | active, enabled, 1 jail (sshd) | active, enabled (but stale upgrade cadence, see above) | not checked | None found |
+| app1-bu | prohibit-password | no | yes | active, enabled, 1 jail (sshd) | active, enabled | not checked | /root/.hermes/.env is world-readable (flagged, but this is credential-adjacent; full secrets review is Sec-A's domain, flagged here purely as a file-permission hardening gap) |
+| wphost02 | prohibit-password | no | yes | active, enabled, 3 jails (runcloud-agent, sshd, sshd-ddos) | active, enabled | not checked | None found |
+
+Additional hardening observations:
+- No empty-password accounts found on Core or app1 in the /etc/shadow scan performed.
+- SSH host key files (`ssh_host_*_key`) are not world-readable on any host checked.
+- Root login is uniformly configured as `prohibit-password` / `without-password` across all 6 hosts, meaning root can only log in via SSH key, not password. This is a reasonably strong baseline, but root login is still permitted at all (as opposed to fully disabled with a dedicated sudo-only admin account), which is a CIS/NIST deviation worth flagging as Low/Medium depending on the org's risk appetite.
+- Core has neither fail2ban nor unattended-upgrades installed. This is the weakest OS hardening posture of the 6 hosts, despite Core running Hermes (the orchestration agent), Grafana, Prometheus, and the Super Search MCP, i.e. a high-value control-plane host.
+- auditd was checked and found inactive on Core and app1; not independently verified on the remaining 4 hosts, but given fail2ban/journald are the only audit trail sources evidenced, assume auditd is similarly absent estate-wide unless Phase Two proves otherwise.
+- journald.conf reviewed on all 6 hosts: no [Journal] section overrides present anywhere (defaults are in force), meaning log rotation/retention limits are whatever the distro default is, not an explicit organizational retention policy. No rsyslog remote forwarding (`@@host` or `@host` directives) was found configured on any host, confirming logs are local-only and not centrally shipped from the OS layer.
+
+---
+
+## 3. MFA / Authentication Coverage Table
+
+| Admin Console / Service | Host | MFA Evidence Found | Status |
+|---|---|---|---|
+| Grafana | Core :3002 | `GF_SECURITY_ADMIN_PASSWORD=admin` (default password in use); grafana.ini shows only commented-out OAuth/LDAP stanzas, no active SSO/2FA config; `disable_login_form` not set to true, so local form-based login remains the primary path | NO MFA, plus default admin credentials in active use (Critical) |
+| Wazuh Dashboard/API | app1 | wazuh-wui API credential in ossec.conf uses local basic auth (`username: wazuh-wui`) with no OAuth/SAML/OIDC integration observed; dashboard is behind Caddy per neteng-a.md but no MFA layer identified | NO MFA identified |
+| Gitea | app2 | app.ini shows `ENABLE_CAPTCHA = false`, `REQUIRE_SIGNIN_VIEW = false`, `DISABLE_REGISTRATION = false`. Gitea supports per-user TOTP 2FA natively but enforcement/adoption was not verified (would require login, out of scope); registration being open plus no captcha is a related hardening gap independent of MFA | MFA capability exists but not confirmed enabled/enforced; registration is open, which is itself a risk |
+| Hudu | app2 | Env vars show only SMTP/upload settings; no OMNIAUTH/SAML/SSO/MFA environment flags present | NO MFA evidence found |
+| UniFi Controller | app2 | `system.properties` grep for 2fa/mfa/auth returned nothing; Ubiquiti UniFi supports MFA via Ubiquiti SSO cloud account when cloud-linked, but no evidence this controller is cloud-linked (local admin only per config inspected) | NO MFA evidence found (local-only auth assumed) |
+| UNMS/UCRM | app2 | Env vars show `UBNT_OAUTH_SERVICE_URL=null`, meaning Ubiquiti cloud SSO/MFA path is disabled/unconfigured | NO MFA, and the SSO path that would enable MFA is explicitly nulled out |
+| Technitium DNS | app2 | `DNS_SERVER_ADMIN_PASSWORD=changeme` present in container env, i.e. the DEFAULT PASSWORD PLACEHOLDER STRING is literally set as the variable value. This does not by itself prove the live credential equals "changeme" (Sec-A's domain to confirm), but the presence of the literal default string in the running environment is itself a hardening red flag. No MFA support is native to Technitium's basic auth | NO MFA; default-credential-pattern flag (Critical, pending Sec-A confirmation of live value) |
+| CloudPanel | app3 | CloudPanel CE (v6.0.8 running) supports optional per-user TOTP 2FA (confirmed via vendor docs), but it is opt-in per account and not enforceable/mandated centrally; no evidence found in local files that the sole `gmb` admin account (per `clpctl user:list`) has 2FA enabled, and this cannot be confirmed without interactive login (out of scope) | MFA capability exists but adoption unconfirmed; single admin account, no organizational enforcement mechanism |
+| LiteLLM / admin-ai UI | app1 | Only `LITELLM_MASTER_KEY` env found (redacted); no SSO/MFA env vars present | NO MFA evidence found |
+| Vaultwarden | app1 | `SIGNUPS_ALLOWED=false` (good, closed registration); Vaultwarden supports WebAuthn/TOTP 2FA per-user natively but no admin-enforced policy env vars (e.g., `ADMIN_TOKEN`) were found configured, meaning the /admin panel protection state is unclear | Per-user 2FA capability exists; admin panel protection unconfirmed |
+| Dawarich, Traccar | app2 | No MFA/2FA/OTP/OAuth-related env vars found on either container | NO MFA evidence found |
+
+**MFA-coverage summary:** Of roughly 11 distinct admin surfaces reviewed, ZERO were confirmed to have MFA actively enforced. Several platforms (Gitea, CloudPanel, Vaultwarden) have native 2FA capability that is opt-in/per-user and not centrally mandated, meaning coverage depends entirely on individual admins choosing to enable it, unverifiable from config alone. Grafana is actively using default admin credentials with no MFA, the most severe finding in this category. UNMS explicitly has its cloud SSO/MFA integration path disabled. Technitium DNS shows a literal "changeme" default-password string still present in its running configuration.
+
+---
+
+## 4. Logging / Monitoring (Wazuh) Coverage Table
+
+| Host | Wazuh Agent Installed? | Wazuh Manager/Stack Present? | Local Journald Only? | Remote Log Forwarding (rsyslog)? |
+|---|---|---|---|---|
+| Core | NOT INSTALLED (no wazuh-agent unit, no /var/ossec, no wazuh package) | No (manager lives on app1) | Yes, journald default config only | None configured |
+| app1 | Manager stack runs here (wazuh-manager, wazuh-indexer, wazuh-dashboard, all image 4.9.2, up 3 weeks) BUT `agent_control -l` shows only ONE registered agent: ID 000, the manager's own local agent (127.0.0.1) | Yes (this IS the manager) | Journald default | None configured |
+| app2 | NOT INSTALLED | No | Journald default | None configured |
+| app3 | NOT INSTALLED | No | Journald default | None configured |
+| app1-bu | NOT INSTALLED | No | Journald default | None configured |
+| wphost02 | NOT INSTALLED | No | Journald default | None configured |
+
+Wazuh manager health check detail: the manager and indexer/dashboard containers are up and running (3 weeks uptime, no restarts observed), and the indexer's REST API responds (401 without credentials, meaning it is alive and enforcing auth, not down). However, `agent_control -l` returning only the manager's own loopback agent (ID 000) confirms that **not a single remote host across the 6-server estate is enrolled as a Wazuh agent**. The Wazuh deployment is effectively monitoring only itself.
+
+**Wazuh/logging coverage summary:** Wazuh is deployed and technically healthy (containers up, API responsive) but has ZERO externally enrolled agents. Coverage is 1 of 6 hosts (16.7%), and that one host (app1) is only monitoring its own loopback, not even its own host-level OS logs via a real agent enrollment path (the "agent" is the manager's built-in local one, not a deployed endpoint agent). All 6 hosts rely solely on local, unshipped journald logs with default retention and no remote forwarding. There is no centralized log aggregation for authentication events, admin console access, container events, or intrusion indicators anywhere in the estate. This is the single largest blind spot identified in this audit.
+
+---
+
+## 5. Cross-Cutting: Unmanaged / Blind-Spot Hosts
+
+Applying the combined lens of (no Wazuh agent) + (no/weak local monitoring) + (patch posture gaps):
+
+- **Core (localhost):** No Wazuh agent, no fail2ban, no unattended-upgrades, no auditd. This is the most under-hardened host in the estate at the OS-control layer, notable because it hosts Hermes (agent orchestration), Grafana, Prometheus, and Super Search MCP, i.e., core operational tooling with broad reach. Prometheus/Grafana provide metrics-level visibility but not security-event-level visibility (no auth log shipping, no file integrity monitoring, no brute-force protection).
+- **app2, app3, app1-bu, wphost02:** No Wazuh agent on any of them; they retain fail2ban and unattended-upgrades as baseline compensating controls, which Core lacks entirely, but none have any form of centralized security log visibility. If any of these hosts is compromised, there is currently no telemetry path that would surface it to a central SIEM; detection depends entirely on someone manually reviewing local journald logs after the fact.
+- **app1 (Wazuh manager host):** Ironically the host running the SIEM has the SIEM monitoring nothing but itself. This is a significant program gap: the org has invested in deploying Wazuh infrastructure but has not completed the agent rollout, so the investment currently delivers near-zero detection value across the estate.
+
+---
+
+## 6. Severity-Rated Findings
+
+### Critical
+
+**SEC-B-01: Wazuh SIEM deployed but zero agents enrolled across the estate**
+Evidence: `agent_control -l` on the wazuh-manager container (app1) lists only agent ID 000 (the manager's own loopback), no remote agents. No `/var/ossec` or wazuh-agent service found on Core, app2, app3, app1-bu, or wphost02.
+Rationale: A SIEM with no enrolled agents provides no actual detection capability for the 5 non-manager hosts, which include internet-facing services (CloudPanel/WordPress, Hudu, UniFi, Gitea, LiteLLM). Security incidents on any of these hosts would go undetected by the org's own monitoring investment. This is the single highest-impact gap in the estate.
+
+**SEC-B-02: Grafana running with default admin credentials and no MFA**
+Evidence: `docker inspect grafana` shows `GF_SECURITY_ADMIN_PASSWORD=admin` alongside `GF_SECURITY_ADMIN_USER=admin`; grafana.ini has no active OAuth/LDAP/SSO configuration, only commented-out templates.
+Rationale: Default admin/admin credentials on an internet-reachable (per neteng-a.md) observability console is one of the most well-known, automatically-scanned-for misconfigurations on the internet. Combined with no MFA, this is a near-zero-effort compromise path for anyone who finds the port.
+
+**SEC-B-03: Technitium DNS container running with literal "changeme" default password string in live environment**
+Evidence: `docker inspect technitium` shows `DNS_SERVER_ADMIN_PASSWORD=changeme`.
+Rationale: Even if the operational credential has since been changed inside the app's own database (a possibility Sec-A should confirm, since this crosses into credential inventory), the fact that the container's own environment variable retains the literal placeholder value is a strong hardening/deployment-hygiene signal that default-credential practices may be in use elsewhere too. DNS admin compromise has estate-wide blast radius (this is itpropartner.com's DNS).
+
+### High
+
+**SEC-B-04: app3 patch cadence stale by approximately 4 weeks, with an unapplied security kernel and postfix update**
+Evidence: unattended-upgrades stamp files on app3 last touched 2026-07-16, versus 2026-08-13 (current, same day) on app1 and app2. `apt list --upgradable` on app3 shows `linux-image-amd64/stable-security 6.12.101-1` pending (currently on 6.12.95-1) and `postfix/stable-security` pending.
+Rationale: app3 hosts CloudPanel, WordPress, and MySQL, an internet-facing content and mail-adjacent stack (postfix present). An unapplied 4-week-old security kernel patch and a pending postfix security update on a public-facing host is a meaningful exposure window, especially since neteng-a.md would have already flagged what ports are open here.
+
+**SEC-B-05: Core has no fail2ban, no unattended-upgrades, and no auditd**
+Evidence: `systemctl is-active fail2ban` and `unattended-upgrades` both return inactive with "not-found" for enabled state; `dpkg -l` shows neither package installed; auditd inactive.
+Rationale: Core is the control-plane host for Hermes and observability tooling. Lacking brute-force protection and automatic security patching on a host with this level of operational privilege is disproportionate risk relative to its role, especially compared to the other 5 hosts which all have these controls.
+
+**SEC-B-06: No admin console in the estate has confirmed, enforced MFA**
+Evidence: Across Grafana, Wazuh dashboard, Gitea, Hudu, UniFi, UNMS/UCRM, Technitium, CloudPanel, LiteLLM, and Vaultwarden, no environment variable, config file, or system property indicated an active, enforced MFA/SSO integration. Several tools have opt-in per-user 2FA capability (Gitea, CloudPanel, Vaultwarden) but no evidence of organizational enforcement.
+Rationale: Any single compromised admin credential (phishing, credential stuffing, reused password) grants full access to that console with no second factor to stop it. This is a systemic authentication-hardening gap across the entire estate, not a one-off.
+
+### Medium
+
+**SEC-B-07: UNMS/UCRM has its cloud SSO integration explicitly disabled (`UBNT_OAUTH_SERVICE_URL=null`)**
+Evidence: `docker inspect ucrm` env output.
+Rationale: Ubiquiti's cloud SSO path is one of the few routes to MFA for this product family; explicitly nulling it out removes that option, leaving local-only authentication as the sole path.
+
+**SEC-B-08: No centralized log forwarding (rsyslog remote or journald shipping) configured anywhere in the estate**
+Evidence: grep for `@@host`/`@host` rsyslog forwarding directives returned empty on all 6 hosts; journald.conf shows default config (no [Journal] section overrides) on all 6 hosts.
+Rationale: Even setting aside Wazuh agent enrollment, there is no other mechanism (rsyslog, journald forwarding, or otherwise) shipping logs off-host anywhere. If a host is compromised and its local logs are tampered with or deleted, there is no off-host copy to fall back on for forensics.
+
+**SEC-B-09: Gitea has open registration and disabled CAPTCHA**
+Evidence: app.ini shows `DISABLE_REGISTRATION = false`, `ENABLE_CAPTCHA = false`, `REQUIRE_SIGNIN_VIEW = false`.
+Rationale: An internet-reachable Gitea instance (confirm exposure via neteng-a.md) with open self-registration and no CAPTCHA is exposed to automated account creation/spam and widens the attack surface for credential-based attacks against a code hosting platform.
+
+### Low
+
+**SEC-B-10: Root login permitted (key-only) rather than fully disabled across all 6 hosts**
+Evidence: `sshd -T` on all 6 hosts shows `permitrootlogin without-password` (uniformly).
+Rationale: While password-based root login is correctly disabled everywhere (good baseline), CIS/NIST guidance generally recommends disabling root SSH login entirely in favor of named-user + sudo, to preserve accountability/audit trail for privileged actions. Not urgent given key-only enforcement, but a durable hardening improvement for Phase Two.
+
+**SEC-B-11: /root/.hermes/.env world-readable on app1-bu**
+Evidence: file permission scan on app1-bu found `/root/.hermes/.env` matching world-readable pattern.
+Rationale: World-readable environment files can leak configuration/secrets to any local process or unprivileged account with filesystem access. Flagged here as a hardening/permissions gap; if it contains credentials, that overlaps with Sec-A's remit and should be cross-referenced with their inventory.
+
+---
+
+## Appendix: Methodology Notes
+
+- All SSH access used `-o ConnectTimeout=10 -o BatchMode=yes` per brief, no interactive prompts triggered.
+- No login attempts were made against any web UI (Grafana, Wazuh dashboard, Gitea, CloudPanel, etc.); all MFA determinations were made from static configuration files, environment variables, and system properties only, per the brief's read-only constraint.
+- `sshd -T` (dump effective config) was used in addition to raw grep of sshd_config to catch settings inherited from Included files or compiled defaults.
+- Wazuh index listing (`_cat/indices`) returned a 401 (auth required) rather than a connection failure, confirming the indexer is alive and reachable, just not accessible without credentials, which were not attempted per the read-only/no-credential-testing constraint.
+- No package installs, service restarts, or file writes were performed on any remote host.

audit/phase-one/findings/sys-a.md

@@ -0,0 +1,388 @@
+# Sys-A Phase One Inventory & Findings: Core + app1
+
+Auditor: Sys-A (sysadmin inventory auditor)
+Scope: Core (this host, localhost) and app1 (152.53.36.131)
+Date: 2026-08-13
+Mode: READ-ONLY. No mutation performed. All remediations are Phase Two items.
+
+---
+
+## 1. Host Profiles
+
+| Attribute | Core (localhost) | app1 (152.53.36.131) |
+|---|---|---|
+| Provider / model | Netcup RS 2000 G12 | Netcup RS 4000 G12 |
+| OS | Debian 13 (trixie) | Debian 13 (trixie) |
+| Kernel | 6.12.94+deb13-amd64 | 6.12.95 |
+| vCPU | 8 | 12 |
+| RAM | 15 GB, NO swap | 31 GB, NO swap |
+| RAM in use | ~12 GB used / 467 MB free (2.8 GB available) | (healthy) |
+| Disk | 503 GB, 73 GB used (15%) | 1007 GB, 107 GB used (11%) |
+| Uptime | not recorded | not recorded |
+| Public IP | 152.53.192.33 | 152.53.36.131 |
+| Docker | present | present |
+| Firewall | ufw active (default policy not confirmed) | ufw active |
+
+Total services inventoried: Core ~55 systemd units (running) + 10 Docker containers; app1 19 systemd units (running) + 23 Docker containers. Counts below are of running/active workloads only.
+
+---
+
+## 2. Core Service Inventory (localhost)
+
+### 2.1 Critical / Tier-0 services
+
+| Service | Software + Version | Port | Runs as | Restart | Backup | Notes |
+|---|---|---|---|---|---|---|
+| Hermes gateway | hermes-agent (python) | 8642, 8787 (socat), 9119 (dashboard) | root | user unit enabled | hermes-full-backup daily 01:00 + live-sync | runs under systemd USER manager (user@0), not a system unit; 3.7 GB RSS, 182 tasks |
+| Caddy | caddy (deb) | 80/443 on 152.53.192.33, 2019 admin 127.0.0.1 | caddy | systemd enabled | Caddyfile in hermes-full-backup + system-config-sync | single ingress for Core |
+| Grafana | grafana/grafana:11.4.0 | 3002 (0.0.0.0) | root (container) | always | grafana db in core-services-backup 01:30 | image 11.4.0 is ~20 months old |
+| Prometheus | prom/prometheus:latest | 9090 (0.0.0.0) | root (container) | always | prometheus_data volume in core-services-backup | config /root/docker/monitoring/prometheus/prometheus.yml |
+| Super Search MCP | /root/docker/super-search (python) | 8899 (0.0.0.0) | root | systemd | NOT covered by any backup script (see F-18) | depends on searxng + exa/firecrawl/open-corporates APIs |
+| PostgreSQL | postgres (deb) | 5432 (127.0.0.1) | postgres | systemd | no dedicated dump; only hotnow-app DB implied | single instance, no replica |
+| Redis | redis (deb) | 6379 (127.0.0.1) | redis | systemd | none found | single instance |
+
+### 2.2 Hermes ecosystem systemd units (all as root unless noted)
+
+| Unit | Purpose | Port | Notes |
+|---|---|---|---|
+| hermes-assistant | Hermes Assistant PWA backend | (8080-ish) | root |
+| hermes-browser | Headless Chromium CDP | 9222 (127.0.0.1) | runs chrome with `--no-sandbox` as root |
+| hermes-control-deck | Control Deck backend API | 8200 | **port collides with pipeline-api** (F-10) |
+| hermes-socat-8787 | port forward 8787 -> localhost:8642 | 8787 (0.0.0.0) | `After=hermes-gateway.service` references non-existent SYSTEM unit (gateway is a user unit) |
+| hermes-voice | Hermes Voice (SvelteKit) | 4331 (127.0.0.1) | runs as non-root (best practice) |
+| hermes-gateway (user) | gateway run | 8642 | root, user manager |
+| hermes-gateway-anita (user) | Anita messaging gateway | - | root, user manager |
+
+### 2.3 Application / MCP / API systemd units (all as root)
+
+| Unit | Purpose | Port | Hardcoded secret? |
+|---|---|---|---|
+| auth-api | auth backend | 8500 | no (uses /root/projects/auth/.env) |
+| ops-portal | operations portal | 8090 | - |
+| osint-api | OSINT API | 8100 | - |
+| osint-person | OSINT person MCP | 8902 | - |
+| diglocate-api | diglocate | 8000 | - |
+| intelsight-api | intelsight | 8099 | - |
+| hotnow-api | HotNow backend | 8001 | - (Postgres + Redis) |
+| shopping-cart | shopping cart | 8101 (127.0.0.1) | - |
+| seemytrip | SeeMyTrip backend | 8113 | **ADMIN_AI_KEY hardcoded** |
+| rally | rally backend | 8105 (0.0.0.0) | **JWT_SECRET, DEEPSEEK_API_KEY, ADMIN_AI_KEY hardcoded** |
+| shark-game | shark-game backend | 8083 (0.0.0.0) | - |
+| pipeline-api | pipeline API | 8200 | **collides with hermes-control-deck** |
+| verdicttank-api | verdicttank | 8201 | - |
+| verdicttank-worker | worker | - | - |
+| status-page / node | status page | 8210 | node |
+| pry | PRY API | 8905 | - |
+| dre-mcp | DRE MCP | 8900/8901 | - |
+| ft360-mcp | FleetTracker360 MCP | 8903 | - |
+| twilio-mcp | Twilio MCP | 8910/8911 | - |
+| crawl4ai | crawler | - | - |
+| voice-agent | voice agent | 9101 | - |
+| voice-agent-stt | voice STT | 9000 | - |
+| gitea-runner | Gitea actions runner | - | - |
+| host-metrics-export | metrics textfile | - | - |
+| mysql-tunnel | SSH tunnel to wphost02 MySQL | 33060 (127.0.0.1) | `StrictHostKeyChecking=no`, `UserKnownHostsFile=/dev/null` |
+| hear-read | audio TTS/read | 8240 (127.0.0.1) | - |
+
+### 2.4 Core Docker containers
+
+| Container | Image:Tag | Port | Image age | Restart | Purpose |
+|---|---|---|---|---|---|
+| browserless | browserless/chrome:latest | 3000 (0.0.0.0) | 2 years | always | headless chrome API |
+| uptime-kuma | louislam/uptime-kuma:1 | 3001 (0.0.0.0) | current | always | status monitoring |
+| grafana | grafana/grafana:11.4.0 | 3002 | 20 months | always | dashboards |
+| prometheus | prom/prometheus:latest | 9090 | 6 weeks | always | metrics |
+| searxng | searxng/searxng:latest | 8888 (127.0.0.1) | 6 weeks | always | meta search (Super Search dep) |
+| timetrex | timetrex | 8085 (127.0.0.1) | - | always | time tracking |
+| microbin | microbin | 8260 (127.0.0.1) | - | always | paste bin |
+| camofox | camofox | 9377 (0.0.0.0) | - | always | stealth browsing |
+| mikrotik-exporter | mikrotik-exporter | 9436 (127.0.0.1) | - | always | router metrics |
+| node_exporter (host) | prom/node-exporter | 9100 (0.0.0.0) | - | systemd | node metrics |
+| telegraf (host) | telegraf | 9273 (0.0.0.0) | - | systemd | metrics |
+
+### 2.5 Ad-hoc / unmanaged processes (Core)
+
+| Process | Port | CWD | Concern |
+|---|---|---|---|
+| `python3 -m http.server 8080` | 8080 (0.0.0.0) | /var/www/mockup/anita-consulting | ad-hoc web server as root, no systemd |
+| `python3 -m http.server 8934` | 8934 (0.0.0.0) | /var/www/mockup/itpropartner | ad-hoc web server as root |
+| `python3 -m http.server 9876` | 9876 (0.0.0.0) | /tmp | **serving /tmp as root, public bind** |
+| hermes dashboard | 9119 (0.0.0.0) | - | part of Hermes |
+
+---
+
+## 3. app1 Service Inventory (152.53.36.131)
+
+### 3.1 Critical / Tier-0 services
+
+| Service | Software + Version | Port | Runs as | Restart | Backup | Notes |
+|---|---|---|---|---|---|---|
+| LiteLLM / admin-ai | ghcr.io/berriai/litellm:v1.92.0 | 4000 | root (container) | always | **DB NOT backed up** (F-1); config YAML daily | master key + admin_ai provider |
+| Caddy | caddy (systemd) | 80/443 | caddy | enabled | **Caddyfile NOT backed up** (F-12) | single ingress for all app1 |
+| Open WebUI | ghcr.io/open-webui/open-webui:latest | 3000 (0.0.0.0) | root (container) | always | daily 1.47 GB | depends on LiteLLM |
+| Wazuh manager/indexer/dashboard | wazuh 4.9.2 (indexer, 21 months old) | 1514/1515, 9200, 5601 | root | always | daily | SIEM/XDR |
+| n8n | n8n | 5678 | root (container) | always | daily | automation |
+| Vaultwarden | vaultwarden latest | 8081 | root (container) | always | daily | password manager |
+| Twenty CRM | twentycrm | 3003 | root (container) | always | daily | CRM |
+| Komodo | komodo-core | 9120 (0.0.0.0) | root (container) | always | daily | infra automation |
+| Browserless | browserless/chrome:latest | 3005 (0.0.0.0), 3006 (ufw-limited) | root (container) | always | - | 2 year old image |
+| super-search (host) | /root/docker/super-search | 8899 | root | systemd | .env only (373 bytes) | depends on searxng (Core) |
+| giftaroast | /root/giftaroast | 8100 | root | systemd | - | **runaway fix_dict.py** (F-8) |
+
+### 3.2 app1 Docker containers (full list)
+
+LiteLLM v1.92.0 + litellm_postgres, mcp-browser / mcp-email / mcp-git / mcp-filesystem / mcp-super-search, litellm-super-search, open-webui, n8n + n8n-postgres, twenty-server + twenty-db + twenty-redis, komodo-core + komodo-mongo, vaultwarden, wazuh single-node (manager+indexer+dashboard), browserless, docuseal.
+
+Key stale/very-old image tags on app1: browserless/chrome:latest (2 years), wazuh/wazuh-indexer:4.9.2 (21 months), vaultwarden/server:1.33.2 (18 months, old tag present alongside latest), ollama/ollama (5 weeks, not running), mattermost (not running).
+
+### 3.3 app1 systemd units
+
+super-search (8899), giftaroast (8100), caddy, sshd, docker, containerd, fail2ban, cron, rsyslog, ufw, unattended-upgrades, qemu-guest-agent, chrony, plus base system units.
+
+### 3.4 Runaway / orphaned processes (app1)
+
+| PID | Process | CWD | Elapsed | CPU |
+|---|---|---|---|---|
+| 3656093 | python3 fix_dict.py | /root/giftaroast | 14d 17h | 99.4% |
+| 3657668 | python3 fix_dict.py | /root/giftaroast | 14d 17h | 99.4% |
+| 3658103 | python3 fix_dict.py | /root/giftaroast | 14d 17h | 99.4% |
+
+Parent of PID 3656093 is `bash -c "cd /root/giftaroast && python3 fix_dict.py 2>&1 echo '=== Restart service ===' systemctl dae..."`, an aborted/interrupted manual deployment. Three cores permanently pegged for 14+ days.
+
+---
+
+## 4. Dependency Map
+
+### 4.1 Core (dot)
+
+```dot
+digraph Core {
+ // Ingress
+ caddy -> ops_portal; caddy -> osint_api; caddy -> diglocate; caddy -> intelsight;
+ caddy -> hotnow_api; caddy -> shopping_cart; caddy -> seemytrip; caddy -> rally;
+ caddy -> shark_game; caddy -> pipeline_api; caddy -> verdicttank_api; caddy -> auth_api;
+ caddy -> hermes_voice; caddy -> voice_agent; caddy -> pry; caddy -> uptime_kuma;
+ caddy -> microbin; caddy -> timetrex; caddy -> status_page;
+
+ // Data layer
+ hotnow_api -> postgres; hotnow_api -> redis;
+ osint_api -> redis;
+
+ // Hermes gateway fan-out
+ hermes_gateway -> super_search; hermes_gateway -> dre_mcp; hermes_gateway -> ft360_mcp;
+ hermes_gateway -> osint_person; hermes_gateway -> twilio_mcp; hermes_gateway -> clearfront_mcp;
+ hermes_gateway -> browserless; hermes_gateway -> camofox; hermes_gateway -> chrome_cdp;
+
+ // Search chain
+ super_search -> searxng; super_search -> "exa/firecrawl/open-corporates APIs";
+
+ // Voice chain
+ voice_agent -> voice_agent_stt;
+
+ // Observability
+ grafana -> prometheus;
+ prometheus -> node_exporter; prometheus -> telegraf; prometheus -> mikrotik_exporter; prometheus -> snmp;
+
+ // Cross-host tunnel
+ mysql_tunnel -> "wphost02 MySQL (5.161.62.38, decommissioned 2026-08-28)";
+}
+```
+
+### 4.2 app1 (dot)
+
+```dot
+digraph app1 {
+ caddy -> litellm; caddy -> open_webui; caddy -> n8n; caddy -> vaultwarden;
+ caddy -> docuseal; caddy -> twenty; caddy -> komodo; caddy -> wazuh_dashboard;
+ caddy -> giftaroast; caddy -> browserless;
+
+ litellm -> litellm_postgres;
+ litellm -> mcp_browser; litellm -> mcp_email; litellm -> mcp_git; litellm -> mcp_filesystem;
+ litellm -> mcp_super_search;
+ open_webui -> litellm;
+
+ n8n -> n8n_postgres;
+ twenty_server -> twenty_db; twenty_server -> twenty_redis;
+ komodo_core -> komodo_mongo;
+ wazuh_manager -> wazuh_indexer; wazuh_dashboard -> wazuh_indexer;
+
+ super_search_host -> "searxng on Core (152.53.192.33:8888)";
+ mcp_super_search -> super_search_host;
+}
+```
+
+### 4.3 Dependency map highlights
+
+- Single ingress per host: Caddy is the sole HTTP(S) entry point. A Caddy misconfiguration or crash takes down every web service on that host (single point of failure).
+- LiteLLM is the AI backbone: Open WebUI, all mcp-* tool servers, and Super Search (via MCP) route through it. admin-ai.itpropartner.com is the model gateway for every AI consumer.
+- Hermes gateway has a wide fan-out (8+ MCP/tool servers, browserless, camofox, Chrome CDP). Its failure cascades into all agent tooling.
+- Cross-host dependency: app1 super-search depends on Core searxng (port 8888) over the WAN. If Core is down, app1 search is degraded.
+- mysql-tunnel (Core) depends on wphost02 (5.161.62.38, decommissioned 2026-08-28) reachability; a host key change breaks it silently.
+- HotNow is the only app with a real DB dependency chain (Postgres + Redis) on Core.
+
+---
+
+## 5. Cron Jobs
+
+### 5.1 Core crontab (root)
+
+| Schedule | Job | User | Purpose | Failure mode if silent |
+|---|---|---|---|---|
+| 01:00 daily | hermes-full-backup.sh | root | full Hermes home + Caddyfile + systemd units + state.db to Wasabi | no DR copy of Hermes state; RPO grows unbounded |
+| */15 min | hermes-live-sync.sh | root | aws s3 sync of .hermes to `live/` prefix | incremental state lost (mitigated by daily full) |
+| 01:30 daily | core-services-backup.sh | root | Grafana/Prometheus/etc volumes | monitoring history loss |
+| 02:00 daily | root-essentials-backup.sh | root | .hermes/.ssh/.aws/himalaya/shark-game/projects | credential/config loss |
+| 03:00 daily | docker-volume-sync.sh | root | **script does not exist** (F-13) | always fails silently |
+| 00:05 daily | system-config-sync.sh | root | system configs to Wasabi | config drift backups lost |
+| 15:00 daily | status-page-refresh.sh | root | status page data | stale status |
+| various | wphost02-backup.sh | root | SSHes to 5.161.62.38 (wphost02, decommissioned 2026-08-28) with root key | wphost02 backups lost |
+| various | docuseal/timetrex/gitea/hudu/dawarich/ragflow/twenty/stack-auth/hexclave/komodo/litellm/technitium/unifi/unms/vaultwarden-backup.sh | root | per-service backups (many target remote hosts) | per-service backup loss |
+
+Hermes-managed jobs live in /root/.hermes/cron/jobs.json (jobs.json + per-job dirs). Parsing jobs.json programmatically failed during the audit (malformed JSON); a grep fallback confirmed the job list above. The exact schedule set should be re-verified by the DR owner.
+
+### 5.2 app1 crontab (root)
+
+| Schedule | Job | Purpose | Failure mode |
+|---|---|---|---|
+| 02:00 daily | /root/backup.sh | litellm config, n8n, openwebui, mcp .env, ollama, mattermost, wazuh, twenty | see F-1 (litellm DB), F-12 (Caddyfile) |
+
+app1 /root/backup.sh references `mattermost-backup.sh` and `wazuh-backup.sh` in /root/.hermes/scripts/. Mattermost is not running (no container), so that backup is a no-op. Ollama is not installed, so its backup step fails each night.
+
+### 5.3 /etc/cron.* (both hosts)
+
+Standard Debian logrotate/man-db/dpkg jobs. No custom third-party cron drops observed beyond the system crontabs above. No evidence of a rotating backup retention job; S3 listings show daily objects accumulating without obvious lifecycle policy (Phase Two item).
+
+---
+
+## 6. Backup Status Matrix
+
+| Service | Backup script | Last S3 object | Restore test documented? |
+|---|---|---|---|
+| Hermes (full) | hermes-full-backup.sh | 2026-08-13 (daily tar.gz ~1.43 GB) | restore.sh generated; DR-PLAN.md present (600) |
+| Hermes (live) | hermes-live-sync.sh -> `live/` prefix | last state.db `live/` unclear; `live-sync/` (old prefix) stale since 2026-07-05 | partial |
+| LiteLLM config | app1 /root/backup.sh | 2026-08-13 (litellm-config-*.yaml, 333 B) | no |
+| **LiteLLM Postgres DB** | (should be pg_dump) | **ZERO .sql.gz objects ever** (F-1) | no |
+| Grafana/Prometheus | core-services-backup.sh | 2026-08-13 (grafana 55 KB, prometheus 51 MB) | no |
+| Open WebUI | app1 /root/backup.sh | 2026-08-13 (1.47 GB) | no |
+| n8n / Twenty / Vaultwarden / Wazuh | app1 /root/backup.sh | 2026-08-13 | no |
+| Core Caddyfile | hermes-full-backup + system-config-sync | 2026-08-13 | via restore.sh |
+| **app1 Caddyfile** | **none** | **never** (F-12) | no |
+| Super Search (both hosts) | none meaningful | .env only (373 B) | no |
+| Postgres / Redis (Core) | none dedicated | none | no |
+
+Restore-test documentation: DR-PLAN.md and migration-recovery.md exist on Core (root-only, mode 600) but no evidence of an actual periodic restore drill being executed (no drill logs found in backup dirs).
+
+---
+
+## 7. Severity-Rated Findings
+
+### CRITICAL
+
+**F-1. LiteLLM Postgres database is not being backed up (data-loss risk).**
+Evidence: `aws s3 ls s3://hermes-vps-backups/app1/litellm/` shows only `litellm-config-*.yaml` objects (240-333 bytes). Zero `.sql`/`.sql.gz` objects since inception. app1 /root/backup.sh dumps `pg_dump -U litellm litellm`, but the live config (`config.yaml`) sets database_url to database `litellm_db`, so the dump targets a nonexistent database and fails silently every night.
+Rationale: LiteLLM's Postgres holds every API key, model routing table, spend/budget records, and the admin-ai provider config. A database or volume failure means total loss of the AI gateway state. This is the single highest-impact finding.
+
+**F-2. Plaintext secrets hardcoded in world-readable systemd unit files.**
+Evidence: `/etc/systemd/system/rally.service` contains `JWT_SECRET`, `DEEPSEEK_API_KEY`, and `ADMIN_AI_KEY` as literal `Environment=` values; `/etc/systemd/system/seemytrip.service` contains `ADMIN_AI_KEY`; `/etc/systemd/system/giftaroast.service` (app1) contains Twilio `AUTH_TOKEN`/`SID` and a `ADMIN_AI_KEY`. Unit files are 0644 (world-readable).
+Rationale: Any local user (or any service compromise) can read live production API keys for DeepSeek, the admin-ai gateway, and Twilio. Secrets belong in root-only env files (600), not unit files.
+
+### HIGH
+
+**F-3. Every custom service runs as root.**
+Evidence: docker inspect of all Core/app1 containers shows `User=""` (root); all `python3`/`node` listeners show `user root` in ss/ps. Only hermes-voice and Caddy run non-root.
+Rationale: A single compromised service (e.g. a 2-year-old browserless) yields full root on the host, no privilege boundary.
+
+**F-4. Very old / unpatched image tags in active service.**
+Evidence: browserless/chrome:latest = 2 years old on both hosts; grafana/grafana:11.4.0 = ~20 months old (Core, internet-exposed via ufw); wazuh/wazuh-indexer:4.9.2 = 21 months old (app1); vaultwarden/server:1.33.2 = 18 months old tag still present on app1.
+Rationale: These images predate many published CVEs and receive no updates. Browserless (headless Chrome) is a high-value attack surface and is also a Hermes tool dependency.
+
+**F-5. Hermes gateway (most critical service) supervised by root's systemd USER manager, not a system unit.**
+Evidence: `systemctl --user status hermes-gateway.service` shows active (user@0), while `systemctl status hermes-gateway.service` is `not-found`. `hermes-socat-8787.service` declares `After=hermes-gateway.service` against a unit that does not exist at the system level.
+Rationale: The gateway is the core of ITPP automation. Its lifecycle depends on a user session staying alive (no lingering system-level restart guarantee) and its unit wiring is inconsistent (socat references a nonexistent system unit). Fragile single point of failure.
+
+**F-6. Port 8200 collision between hermes-control-deck and pipeline-api.**
+Evidence: both units are `active (running)` and both configure 127.0.0.1:8200. `ss -tlnp` shows 8200 held by pipeline-api (PID 3787644). hermes-control-deck Main PID (1962662) is not the socket owner.
+Rationale: One of the two services is silently shadowed (the Control Deck API). Traffic routed by Caddy reaches whichever holds the port; the other is effectively down while appearing healthy.
+
+**F-7. Three runaway `fix_dict.py` processes pegging 3 cores for 14+ days on app1.**
+Evidence: PIDs 3656093/3657668/3658103 at 99.4% CPU, elapsed 14d 17h, cwd /root/giftaroast. Parent is an aborted `bash -c` deployment (truncated command visible in /proc).
+Rationale: 25% of app1 CPU permanently wasted; indicates a bug in fix_dict.py and an interrupted deployment that was never cleaned up. Load average 3.26 on 12 cores.
+
+**F-8. app1 Caddyfile (ingress for every app1 service) is not backed up.**
+Evidence: grep of /root/backup.sh for Caddyfile returns nothing; no app1 script references /etc/caddy. Only Core's Caddyfile is backed up (by Core scripts).
+Rationale: app1's entire reverse-proxy routing config (every site, TLS policy, upstream mapping) would need to be reconstructed by hand after a host failure. High blast radius, zero coverage.
+
+### MEDIUM
+
+**F-9. Ad-hoc `python -m http.server` running as root on public interfaces, one serving /tmp.**
+Evidence: `python3 -m http.server 8080` (cwd /var/www/mockup/anita-consulting), `... 8934` (/var/www/mockup/itpropartner), `... 9876` (/tmp), all bound 0.0.0.0, none under systemd.
+Rationale: Undocumented, unsupervised web servers run as root; serving /tmp is a direct path to accidental data exposure. These are "shadow IT" that bypasses the standard service lifecycle.
+
+**F-10. Grafana exposed to Anywhere by firewall rule.**
+Evidence: `ufw status` shows `3002/tcp ALLOW IN Anywhere` (Grafana). Prometheus (9090) and node_exporter (9100) also bind 0.0.0.0.
+Rationale: Grafana (an old 11.4.0 with auth) and raw metrics endpoints are reachable from the public internet, not just the Tailscale/management network.
+
+**F-11. World-readable credential files.**
+Evidence: /root/projects/auth/.env = 0644, /etc/caddy/dre-passwd = 0644 (basic-auth password), /root/anita-key.json = 0644 (LiteLLM key alias), on app1.
+Rationale: Live credentials readable by any local account, compounding F-3 (everything is root anyway, but defense-in-depth is absent).
+
+**F-12. mysql-tunnel disables host key verification.**
+Evidence: unit uses `-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null` to wphost02.
+Rationale: The MySQL tunnel to wphost02 is susceptible to MITM/host-spoofing; a changed host would be silently accepted.
+
+**F-13. Dead cron job: docker-volume-sync.sh does not exist.**
+Evidence: crontab references `/root/.hermes/scripts/docker-volume-sync.sh` at 03:00 daily; the file does not exist (`ls` fails). Docker volume sync was reportedly moved to hermes-docker-sync.sh (different bucket), but the old crontab entry was never removed.
+Rationale: A scheduled job that can never succeed, silently. Indicates cron hygiene drift.
+
+**F-14. Pending security updates on both hosts (including kernel).**
+Evidence: Core has 14 upgradable packages including linux-image-amd64 6.12.94 -> 6.12.101 (security) and chromium security update; app1 has 14 including docker-ce 29.6.1 -> 29.7.2.
+Rationale: Kernel security updates pending on both hosts. Docker engine on app1 is a minor version behind. (Both run unattended-upgrades, but security packages remain pending.)
+
+**F-15. No swap on either host.**
+Evidence: `free -h` shows Swap 0 on both. Core sits at ~12 GB/15 GB used with 467 MB free.
+Rationale: Under memory pressure the OOM killer will terminate arbitrary services (likely the biggest consumer: the 3.7 GB Hermes gateway). No graceful pressure relief.
+
+**F-16. Dead reverse-proxy route: noc.itpropartner.com.**
+Evidence: Caddyfile maps noc -> 127.0.0.1:8065, but no mattermost container is running on app1; mattermost-backup.sh runs daily as a no-op.
+Rationale: A configured, monitored-adjacent route points at a service that is not running. Drift between routing config and reality.
+
+**F-17. Super Search (designated critical) has no meaningful backup.**
+Evidence: only a 373-byte .env snippet is uploaded; the code/config/venv under /root/docker/super-search (Core) and app1 is not covered by any backup script.
+Rationale: Rebuilding the search MCP requires re-cloning and re-provisioning keys by hand; RPO effectively zero for its configuration.
+
+### LOW
+
+**F-18. /root/.aws directory is group/world writable on app1 (drwxrwxr-x).**
+Evidence: `ls -ld /root/.aws` = 775. Credentials file itself is 600, but the directory is too open for a secrets dir.
+
+**F-19. Leftover Docker volumes / image drift.**
+Evidence: three Grafana volumes (grafana_data, grafana_data_final, grafana_data_v3) indicate config churn; multiple stale images (vaultwarden 1.33.2, litellm v1.84.0, ollama, mattermost) not running but present. Nine accumulated Caddyfile backup files in /etc/caddy.
+
+**F-20. Single points of failure (no HA anywhere).**
+Evidence: one Caddy per host, one Postgres (no replica), one Redis (no replica), one Hermes gateway, one LiteLLM + one litellm_postgres, one Wazuh single-node.
+Rationale: No redundancy for any Tier-0 component. Acceptable for the environment size but must be documented and covered by DR runbooks.
+
+---
+
+## 8. RTO / RPO for Critical Services
+
+| Service | RPO (est.) | RTO (est.) | Basis |
+|---|---|---|---|
+| Hermes gateway | ~24 h (daily full backup; 15-min live-sync status uncertain) | 2-4 h | restore.sh + DR-PLAN.md exist; warm standby at app1-bu (5.161.225.131) referenced but out of scope/unverified |
+| LiteLLM / admin-ai | **config 24 h; database 0 (not backed up)** | hours to days | F-1: DB must be reconstructed; keys re-added manually |
+| Caddy ingress (Core) | 24 h (Caddyfile) | 30 min - 1 h | reinstall caddy + restore Caddyfile |
+| Caddy ingress (app1) | **none (unbacked)** | 1-4 h | F-8: full manual reconstruction of routing |
+| Grafana / Prometheus | 24 h | 1-2 h | volume restore from core-services-backup |
+| Super Search | **none (code/config unbacked)** | 1-3 h | re-clone + re-provision .env keys |
+
+---
+
+## 9. Top 5 Highest-Severity Findings (for parent summary)
+
+1. **F-1 (Critical):** LiteLLM Postgres DB never backed up; dump targets wrong database name, so the AI gateway's keys/routing/spend are unprotected.
+2. **F-2 (Critical):** Plaintext API keys (DeepSeek, admin-ai, Twilio) hardcoded in world-readable systemd unit files (rally, seemytrip, giftaroast).
+3. **F-3 (High):** Every custom service runs as root, so any single compromise is full-host takeover.
+4. **F-4 (High):** browserless/chrome (2 years old), grafana 11.4.0 (20 months), wazuh 4.9.2 (21 months) in active internet-facing service.
+5. **F-5 (High):** Hermes gateway supervised by user manager only, with inconsistent unit wiring (socat references nonexistent system unit); fragile SPOF.
+
+*End of Sys-A Phase One findings. No remediation performed. All items deferred to Phase Two.*

audit/phase-one/findings/sys-b.md

@@ -0,0 +1,330 @@
+# Sys-B Findings - ITPP Phase One Infrastructure Audit (Read-Only)
+
+**Auditor:** Sys-B (sysadmin inventory auditor)
+**Scope:** app2 (152.53.39.202), app3 (152.53.241.111), app1-bu (5.161.225.131), wphost02 (5.161.62.38, decommissioned 2026-08-28)
+**Date:** 2026-08-13
+**Method:** Read-only SSH (`ssh -i /root/.ssh/itpp-infra`, BatchMode, ConnectTimeout=10). No mutation performed. All remediation is deferred to Phase Two.
+
+---
+
+## 1. Host Reachability
+
+| Host | IP | Role | Status |
+|------|----|------|--------|
+| app2 | 152.53.39.202 | Docker app host | REACHABLE |
+| app3 | 152.53.241.111 | CloudPanel web host | REACHABLE |
+| app1-bu | 5.161.225.131 | Warm standby (Core failover) | REACHABLE |
+| wphost02 | 5.161.62.38 | Legacy WordPress / RunCloud | DECOMMISSIONED 2026-08-28 |
+
+No access limitations. All four hosts answered over SSH with the shared key.
+
+---
+
+## 2. app2 (152.53.39.202) - Docker Application Host
+
+**OS:** Debian (netcup), up 3+ weeks. Docker host running ~40 containers across 9 compose projects plus two orphan containers.
+
+### 2.1 Per-Service Inventory
+
+| Service | Image / Version | Purpose | Ports | Restart | Health | Notes |
+|---------|-----------------|---------|-------|---------|--------|-------|
+| support-api | support-api:latest (custom build, 3d) | Internal support API | 0.0.0.0:6880 | unless-stopped | healthy | Python healthcheck /health |
+| bookstack | lscr.io/linuxserver/bookstack:latest | Docs (support.itpropartner.com) | 0.0.0.0:6875->80 | - | up | APP_URL set; SMTP via mail.germainebrown.com:2525 |
+| bookstack-db | lscr.io/linuxserver/mariadb:latest | Bookstack DB | 3306 (internal) | - | healthy | - |
+| happy_rosalind | lscr.io/linuxserver/bookstack:latest | ORPHANED 2nd Bookstack (no compose project, no host port) | 80/443 (internal only) | - | up | Auto-generated name; no external binding; apparent leftover |
+| docker-ragflow-cpu-1 | infiniflow/ragflow:v0.26.4 | RAGFlow AI platform | 9380-9384, 9392->80, 9393->443 | - | up | - |
+| docker-mysql-1 | mysql:8.0.39 (2yr old) | RAGFlow MySQL | 127.0.0.1:3306 | - | healthy | 2-year-old image tag |
+| docker-minio-1 | pgsty/minio:RELEASE.2026-03-25 | RAGFlow object store | 127.0.0.1:9000, 0.0.0.0:9001 | - | healthy | - |
+| docker-redis-1 | valkey/valkey:8 | RAGFlow cache | 127.0.0.1:6379 | - | healthy | - |
+| docker-infinity-1 | infiniflow/infinity:v0.7.0 | RAGFlow vector DB | 0.0.0.0:23817/23820, 127.0.0.1:5432 | - | healthy | 23817/23820 exposed publicly |
+| technitium | technitium/dns-server:latest | Authoritative DNS server | 0.0.0.0:53 tcp/udp, 127.0.0.1:5380 | - | healthy | Public recursive/authoritative DNS |
+| dawarich_app | freikin/dawarich:latest | Location tracking (Dawarich) | 127.0.0.1:3002 | - | healthy | - |
+| dawarich_sidekiq | freikin/dawarich:latest | Dawarich background jobs | 3000 (internal) | - | healthy | - |
+| dawarich_db | postgis/postgis:17-3.5-alpine | Dawarich DB | 5432 (internal) | - | healthy | - |
+| dawarich_redis | redis:7.4-alpine | Dawarich cache | 6379 (internal) | - | healthy | - |
+| traccar | traccar/traccar:latest | GPS fleet tracking | 0.0.0.0:5000-5150 (tcp+udp), 0.0.0.0:8082 | - | healthy | 151 device ports publicly exposed |
+| gitea | gitea/gitea:latest | Git server (git hosting) | 0.0.0.0:3022->22, 127.0.0.1:3001 | - | up | SSH port public |
+| unifi-controller | jacobalberty/unifi:latest (8mo) | UniFi controller | 0.0.0.0:8080/8443/8843/8880, 3478/10001/udp | - | healthy | linuxserver/unifi image also present (5wk) - migration drift |
+| unms-nginx | ubnt/unms-nginx:latest | UNMS reverse proxy | 0.0.0.0:81/8089/8444 | - | up | - |
+| unms-api | ubnt/unms:latest | UNMS API (EOL product) | internal | - | healthy | UNMS discontinued by Ubiquiti 2021 |
+| unms-device-ws-1..11 | ubnt/unms:latest | UNMS device websockets (11 replicas) | internal | - | healthy | - |
+| unms-netflow | ubnt/unms-netflow:latest | UNMS netflow collector | 0.0.0.0:2055/udp | - | up | - |
+| unms-postgres | ubnt/unms-postgres:latest | UNMS DB | 5432 (internal) | - | up | - |
+| unms-siridb | ubnt/unms-siridb:latest | UNMS time-series DB | 9000/9010 (internal) | - | healthy | - |
+| unms-rabbitmq | rabbitmq:3.7.28-alpine (5yr) | UNMS message broker | 4369/5671-5672/25672 | - | up | **5-year-old image, EOL, known CVEs** |
+| unms-fluentd | ubnt/unms-fluentd:latest | UNMS logging | 5140, 127.0.0.1:24224 | - | up | - |
+| ucrm | ubnt/unms-crm:4.5.33 | UCRM billing | 80-81, 443, 9000, 2055/udp | - | up | - |
+| hudu-app-1 | hududocker/hudu:latest | Hudu IT documentation | 127.0.0.1:3000 | - | up | - |
+| hudu-worker-1 | hududocker/hudu:latest | Hudu sidekiq worker | 3000 (internal) | - | up | restarted 15h ago |
+| hudu-db-1 | postgres:16.2 (2yr) | Hudu DB | 5432 (internal) | - | up | 2-year-old postgres tag |
+| hudu-redis-1 | redis:latest | Hudu cache | 6379 (internal) | - | up | - |
+
+**Old/unused images present:** `ubnt/ucrm-conntrack:latest` (5yr), `mongo:7.0` (6wk, no container), `linuxserver/unifi-network-application:latest` (5wk, not the running unifi image), `caddy:latest`, `alpine:latest`.
+
+### 2.2 Dependencies (text map)
+
+- **RAGFlow** depends on: mysql-1, minio-1, redis-1, infinity-1. Downstream: end users of the RAGFlow UI.
+- **Bookstack** depends on: bookstack-db (MariaDB). Downstream: support docs users.
+- **Dawarich** depends on: dawarich_db (postgis), dawarich_redis, sidekiq worker.
+- **UNMS stack** depends on: unms-postgres, unms-siridb, unms-rabbitmq, unms-fluentd, unms-nginx. **UCRM** shares the UNMS stack.
+- **Hudu** depends on: hudu-db-1 (postgres), hudu-redis-1, hudu-worker-1.
+- **Traccar, Gitea, Technitium, support-api, UniFi**: self-contained (single container each; Traccar/Gitea embed storage).
+- All containers depend on Docker daemon + host disk. Technitium depends on external DNS delegations.
+
+### 2.3 Config Files / Drift
+
+- `/opt/support-api/docker-compose.yml` - single service, port 6880, Python /health healthcheck.
+- `/opt/bookstack/docker-compose.yml` - `APP_URL=https://support.itpropartner.com`, SMTP `mail.germainebrown.com:2525`, DB/Mail passwords and APP_KEY present in compose (redacted in this report).
+- `/opt/gitea/docker-compose.yml`, `/root/docker/hudu/docker-compose.yml`, `/root/docker/traccar/docker-compose.yml`, `/root/docker/dawarich/docker-compose.yml` - env-style secrets inline (redacted).
+- `/home/unms/app/` - UNMS install (docker-compose, unms.conf, install-full.sh, update.sh).
+- `/root/.aws/` - credentials + backups present (redacted). `/root/.docker/` - token seed files.
+- **Drift:** orphan `happy_rosalind` Bookstack container (no compose project, no host port). Two UniFi images (jacobalberty running vs linuxserver pulled) suggest an in-flight migration.
+
+### 2.4 Cron Jobs
+
+| Schedule | User | Command | Purpose | Failure mode |
+|----------|------|---------|---------|--------------|
+| 30 2 * * * | root | `/root/backup.sh 2>&1 \| logger -t app2-backup` | Nightly backup to Wasabi S3 | See 2.5 |
+
+No other system or user cron jobs beyond `/etc/cron.d` defaults.
+
+### 2.5 Backup Status (CRITICAL GAP)
+
+`/root/backup.sh` (Wasabi S3 target `hermes-vps-backups`, `s3.us-east-1.wasabisys.com`) calls per-service backup scripts. Live evidence from 2026-08-13 02:30 run:
+
+- **Completed OK:** Traccar (DB+config), Dawarich, Technitium DNS, RAGFlow (MySQL dump + Infinity DB + Minio objects).
+- **Logged "Backing up..." but NO completion line:** Gitea, Hudu, UNMS, UniFi.
+- **Root cause:** `/root/backup.sh` references `gitea-backup.sh`, `hudu-backup.sh`, `unms-backup-sync.sh`, `unifi-backup-sync.sh` which **do not exist** under `/root/.hermes/scripts/`. Those sections are silently skipped.
+- `bookstack-backup.sh`, `support-api-backup.sh`, `ragflow-backup.sh` exist but Bookstack and support-api are **not scheduled** in cron.
+
+**Impact:** Gitea, Hudu, UNMS, UniFi, Bookstack, support-api have effectively NO working backup. No restore test is documented for any app2 service.
+
+### 2.6 app2 Critical-Service RTO/RPO
+
+| Service | RPO | RTO (est.) | Restore tested? |
+|---------|-----|-----------|-----------------|
+| Hudu | none (backup broken) | 4-8h | No |
+| UNMS | none (backup broken, product EOL) | 4-8h | No |
+| UniFi | none (backup broken) | 2-4h | No |
+| Gitea | none (backup broken) | 2-4h | No |
+| Bookstack / support-api | none (script not scheduled) | 2h | No |
+| Traccar | 24h | 2-4h | No |
+| Dawarich | 24h | 2h | No |
+| Technitium DNS | 24h | 1h | No |
+| RAGFlow | 24h | 2-4h | No |
+
+---
+
+## 3. app3 (152.53.241.111) - CloudPanel Web Host
+
+**OS:** Debian 13 (trixie), up 34d. 31 GiB RAM / 12 CPU. `/` 1TB (80G used, 9%). Runs CloudPanel + nginx + Percona MySQL 8.4 + 10 PHP-FPM versions + Docker (Hexclave Stack Auth, Buzz relay) + several systemd app services.
+
+### 3.1 Systemd Services
+
+| Service | Version | Purpose | Notes |
+|---------|---------|---------|-------|
+| nginx | 1.30.4 | Primary web server | 80/443 |
+| clp-nginx | - | CloudPanel control panel | 8443 |
+| clp-agent / clp-php-fpm | - | CloudPanel agent | - |
+| mysql (Percona Server) | 8.4.10 | Shared DB for all sites | 127.0.0.1:3306 (datadir /home/mysql) |
+| php7.1-fpm ... php8.5-fpm | 7.1, 7.2, 7.3, 7.4, 8.0, 8.1, 8.2, 8.3, 8.4, 8.5 | Per-site PHP pools | **7.1-8.0 are EOL** |
+| varnishd | 7.5.0 | HTTP cache | 6081 |
+| memcached | - | Object cache | 11211 |
+| redis | - | Object cache | 6379 |
+| proftpd | - | FTP server | 21 (plaintext FTP) |
+| postfix | - | Outbound mail | 25 |
+| gitea.modelortho.service | Gitea | Git for git.modelortho.com | 3001 |
+| msp-forms.service | FastAPI | Shared form handler (forms.itpropartner.com) | - |
+| docs-auth-validator.service | - | Stack Auth JWT validator for docs.itpropartner.com | - |
+| backup-restore.service | - | Backup-restore web UI (my.itpropartner.com/backup) | - |
+| percona-telemetry-agent | - | Percona telemetry | - |
+| fail2ban, cron, chrony, containerd, docker | - | platform | - |
+
+### 3.2 Docker Services
+
+| Container | Image | Purpose | Notes |
+|-----------|-------|---------|-------|
+| hexclave-* (server, postgres, clickhouse, cron) | stackauth/server:latest, postgres 16/17-alpine, clickhouse 25.10 | Hexclave Stack Auth | No backup coverage found |
+| buzz-prod-* (relay, postgres, redis, minio) | ghcr.io/block/buzz:main | Buzz relay (Block open-source) | Production relay; no backup coverage found |
+
+### 3.3 Sites Hosted (CloudPanel)
+
+**WordPress (10 wp-config.php instances across 9 users):** apextrackexperience.com, boxpilotlogistics.com, debtrecoveryexperts.com (x2 - under BOTH `debtreecoveryexperts` and `drecovery` users), iamgmb.com, intelsight.io, mainwp.itpropartner.com, vigilanttac.com, voipsimplicity.com (+ `www` subdomain).
+
+**Static / non-WordPress:** docs, forms, mockups, proposals, support, my.verdicttank.com, verdicttank.com, modelortho.com (+www), transitpin.com, my.transitpin.com, panel, my.voipsimplicity.com, timapta.org, katiewattsdesign.com, buzz.iamgmb.com, hexclave-api/dash, gmb, auth2 + auth2-api.
+
+### 3.4 Cron Jobs
+
+| Schedule | User | Command | Purpose |
+|----------|------|---------|---------|
+| 0 3 * * * | root | `/root/backup.sh \| logger -t app3-backup` | Nightly full backup |
+| 0 1,13 * * * | root | `/opt/backup-restore/snapshot.sh` | Twice-daily WP snapshots |
+| 30 4 * * * | root | `/root/gitea-modelortho-backup.sh \| logger -t gitea-backup` | Gitea (modelortho) backup |
+| 15 3 * * * | clp | `clpctl db:backup ... --retentionPeriod=7` | CloudPanel DB backup |
+| 15 4 * * * | clp | `/home/clp/scripts/create_backup.sh` | CloudPanel backup |
+| 5-25 * * * * | clp | certbot/letsencrypt renewals, vhost import, cloudflare IPs | platform |
+
+### 3.5 Backup Status
+
+`/root/backup.sh` (2026-08-13 03:00 run, all OK) covers: CloudPanel SQLite DB, MSP Forms, **all 10 MySQL DBs**, **all 10 WordPress file trees**, static sites, nginx/configs. Destination Wasabi S3 `app3/`. `snapshot.sh` adds twice-daily local snapshots of WP sites (30-day retention). Gitea (modelortho) has its own nightly S3 backup.
+
+**Backup gap:** the Docker services (Hexclave Stack Auth, Buzz relay) and TransitPin are **not** referenced by any backup script. Their postgres/clickhouse/minio data has no scheduled backup.
+
+### 3.6 Config / Drift
+
+- Plaintext MySQL root password hardcoded in BOTH `/root/backup.sh` (root-only) and `/opt/backup-restore/snapshot.sh` (**775 world-readable**). Same password in both.
+- Duplicate WordPress site (debtrecoveryexperts.com) under two different system users - ambiguous ownership.
+- `/home/.swap` = 2GB swap file on web data volume.
+- Five EOL PHP-FPM runtimes (7.1, 7.2, 7.3, 7.4, 8.0) still running.
+
+### 3.7 app3 Critical-Service RTO/RPO
+
+| Service | RPO | RTO (est.) | Restore tested? |
+|---------|-----|-----------|-----------------|
+| CloudPanel + all WP/static sites | 12-24h (daily + 2x snapshots) | 4-8h | No |
+| MySQL (10 DBs) | 24h | 4h | No |
+| Gitea (modelortho) | 24h | 2h | No |
+| Hexclave Stack Auth | none (no backup) | 4h | No |
+| Buzz relay | none (no backup) | 4h | No |
+| TransitPin | none (no backup) | 4h | No |
+
+---
+
+## 4. app1-bu (5.161.225.131) - Warm Standby for Core
+
+**OS:** Ubuntu 24.04.4, up 28d. 3.7 GiB RAM / 3 CPU / 75G disk (37% used). No Docker. Runs only SSH, tailscale, fail2ban, cron + Hermes agent (v0.18.2). This is the warm standby for the live Core Hermes box.
+
+### 4.1 Services
+
+| Service | Purpose | Notes |
+|---------|---------|-------|
+| ssh | Admin access | 22 |
+| tailscaled | Private mesh access | Tailscale |
+| fail2ban | SSH brute-force protection | - |
+| cron | Schedules watchdog + sync | - |
+| hermes (binary) | Hermes Agent v0.18.2 installed | NOT currently running (dormant) |
+
+### 4.2 Cron Jobs (the failover mechanism)
+
+| Schedule | Command | Purpose | Last run |
+|----------|---------|---------|----------|
+| */5 * * * * | `/root/.hermes/scripts/hermes-standby-watchdog.sh` | Ping live Core; failover if down ~3.5 min | Active (log entries hourly through 2026-08-13) |
+| */10 * * * * | `/root/.hermes/scripts/hermes-standby-sync.sh` | S3 sync of config/skills/plugins/cron/references | Active (last sync 2026-08-13T14:10Z) |
+
+**Failover behavior (verified, NOT triggered):** the watchdog pings `LIVE_HOST=152.53.192.33`; if it fails 4 consecutive 60s cycles, it sends Telegram+email alerts, runs `aws s3 sync s3://hermes-vps-backups/live/ -> ~/.hermes/`, then `hermes gateway start`. A standalone "failover" cron does not exist; failover is embedded in the watchdog script (runs every 5 min). The sync script correctly skips sync when the live host is unreachable.
+
+### 4.3 Standby Readiness Assessment (NOT fully ready)
+
+- **Config-level state is fresh:** config.yaml (2026-08-12), .env (2026-08-12), skills/ (2026-08-12), cron output (2026-08-13). Synced every 10 min from S3 `live/`.
+- **Data-level state is STALE:** `state.db` (2.1GB, Jul 15), `memory_store.db` (Jul 15), `sessions/` (Jul 15), `memories/` (Jul 15). The sync script deliberately excludes these (`DIRS="skills profiles plugins cron references"`, files `config.yaml .env .bashrc` only).
+- **Consequence:** on failover, Hermes would start with ~4-week-old session, memory, and state DB. This is a config-level warm standby, NOT a true data-level warm standby.
+
+### 4.4 Config / Drift (secrets)
+
+- `/root/.hermes/.env` (mode **644, world-readable**) holds ~20 plaintext secrets: Cloudflare API token, Netcup API key + customer password, SyncroMSP token, Telegram bot token, **root passwords for app1/app2/app3** (`SERVER_152_53_*.PASS`), RingLogix creds, and API keys for OpenAI/Perplexity/Groq/xAI/Mistral/Fireworks/Google/Cohere.
+- `/root/.hermes/migration-creds.txt` (644), `/root/.hermes/scripts/.hetzner_token` (644), `/root/.hermes/scripts/.netcup_api_key` (644) - further plaintext credentials.
+- `hermes-standby-watchdog.sh` (mode **755, world-readable**) contains a plaintext email password and Telegram bot token inline.
+- `/root/.hermes/state.db.corrupted` (1.8GB, Jul 9) and `state.db-wal` lingering.
+
+---
+
+## 5. wphost02 (5.161.62.38) - Legacy WordPress / RunCloud (DECOMMISSIONED 2026-08-28)
+
+**OS:** Ubuntu 24.04.4, up 34d. 3.7 GiB RAM / 3 CPU / 75G disk (**82% full** - 59G used). RunCloud-managed LEMP stack. Decommissioned 2026-08-28 (all 8 WordPress sites migrated to app3).
+
+### 5.1 Services
+
+| Service | Version | Purpose | Ports |
+|---------|---------|---------|-------|
+| nginx-rc | - | RunCloud web server | 80/443 |
+| apache2-rc | - | RunCloud apache (secondary) | 127.0.0.1:81 |
+| mariadb | 11.1.6 | Shared DB | 127.0.0.1:3306 |
+| php81rc-fpm ... php85rc-fpm | 8.1-8.5 | Per-site PHP pools | - |
+| runcloud-agent | - | RunCloud remote mgmt agent | *:34210 (rcsa service) |
+| node_exporter | - | Prometheus exporter | 0.0.0.0:9100 |
+| postfix | - | Mail | 25 |
+| fail2ban, firewalld, supervisor | - | platform | - |
+
+firewalld active zone `runcloud` allows only 22, 80, 443 + `rcsa` (RunCloud agent) inbound.
+
+### 5.2 WordPress Sites (8, still active)
+
+| Site | DB | Files (Aug activity) |
+|------|----|---------------------|
+| apextrackexperience | apextrackexperience_1781549652 | active (files modified Aug) |
+| boxpilotlogistics | boxpilotlogistics_1770339547 | active |
+| debtrecoveryexperts | debtrecoveryexperts_1778934554 | active |
+| iAmGMB | iAmGMB_1764020288 | no changes since Jul 1 |
+| katiewattsdesign | katiewattsdesign_1735425014 | active |
+| MainWP | mainWP_1717713767 | active |
+| vigilanttac | vigilanttac_1728911691 | active |
+| voipsimplicity | voipsimplicity_1732250845 | active (13k files modified) |
+
+All eight sites ALSO exist on app3 (CloudPanel). Both hosts appear live and actively modified -> **split-brain migration state**.
+
+### 5.3 Cron Jobs
+
+| Schedule | Command | Purpose |
+|----------|---------|---------|
+| */5 * * * * | `/root/apex-mail-watchdog-daemon.sh` | mail watchdog |
+| 30 2 * * * | `/root/db-dump.sh \| logger -t db-dump` | nightly DB dump |
+
+### 5.4 Backup Status (CRITICAL GAP)
+
+- `/root/db-dump.sh` (scheduled daily 02:30) dumps **only 2 of 8** DBs (apextrackexperience, boxpilotlogistics) to **local** `/root/db-backups/` with **7-day retention**. No offsite copy.
+- `/root/backup.sh` (full sites + all DBs + RunCloud config -> Wasabi S3) exists but is **NOT scheduled in any crontab** (grep across `/etc/cron*` and `/var/spool/cron` returned nothing).
+- **Net effect:** 6 of 8 WordPress DBs and ALL site file trees have no running backup. 2 of 8 DBs have local-only 7-day backups. No restore test documented.
+
+### 5.5 wphost02 Critical-Service RTO/RPO
+
+| Service | RPO | RTO (est.) | Restore tested? |
+|---------|-----|-----------|-----------------|
+| apextrackexperience / boxpilotlogistics DBs | 24h (local only, 7d retention) | 4-8h | No |
+| Other 6 WP DBs + all site files | none (no scheduled backup) | 4-8h | No |
+
+---
+
+## 6. Severity-Rated Findings
+
+### CRITICAL
+
+- **C1 - app2: Four production services have silently failing backups.** `/root/backup.sh` references `gitea-backup.sh`, `hudu-backup.sh`, `unms-backup-sync.sh`, `unifi-backup-sync.sh` which do not exist. The 2026-08-13 02:30 log shows "Backing up..." for these with no completion. Gitea, Hudu, UNMS, and UniFi have no effective backup; a loss would be unrecoverable. (Evidence: journalctl `app2-backup` + missing files under `/root/.hermes/scripts/`.)
+- **C2 - app1-bu: World-readable secrets file.** `/root/.hermes/.env` (mode 644) contains ~20 plaintext credentials including root passwords for app1/app2/app3, Telegram bot token, Cloudflare/Netcup/SyncroMSP tokens, and eight AI-provider API keys. Any local user or compromised process can read the entire secret estate. (Evidence: `ls -la` + key names.)
+- **C3 - app3: MySQL root password hardcoded in plaintext in backup scripts.** `/opt/backup-restore/snapshot.sh` is mode 775 (world-readable) and contains `MYSQL_PASS='[REDACTED]'`; the same password is in `/root/backup.sh`. Credential leakage plus shared superuser credential across all app3 databases. (Evidence: script contents.)
+- **C4 - wphost02: Effective backup coverage is ~25% and local-only.** Scheduled `db-dump.sh` backs up 2 of 8 DBs to local disk (7-day retention); the full offsite S3 script `/root/backup.sh` is not in cron. 6 WordPress DBs and all site files have no running backup on a host that is still live. (Evidence: crontab + db-dump.sh + `grep backup.sh /etc/cron* /var/spool/cron` empty.)
+- **C5 - app1-bu: Warm standby is not data-ready.** Sync covers only config/skills/plugins/cron/references; `state.db`, `memory_store.db`, and `sessions/` are stale at Jul 15 (~4 weeks). A failover today would restore Hermes without the last month of session, memory, and state. (Evidence: `stat` mtimes + sync script `DIRS`/excludes.)
+
+### HIGH
+
+- **H1 - app1-bu: Failover watchdog targets the wrong IP.** Watchdog pings `152.53.192.33` (labeled "Core/App1 netcup"), but `.env` and the audit brief reference Core/app1 at `152.53.36.131`. If the live Core moves/differs, the standby will fail over to nothing or never detect an outage. (Evidence: watchdog script `LIVE_HOST` vs `.env` `SERVER_152_53_36_131_PASS`.)
+- **H2 - app1-bu: Plaintext email password + Telegram token in watchdog script (mode 755, world-readable).** (Evidence: script contents.)
+- **H3 - app2: Very old images in production.** `rabbitmq:3.7.28-alpine` (5 years, EOL with known CVEs), `ubnt/ucrm-conntrack:latest` (5 years), `mysql:8.0.39` (2 years), `postgres:16.2` (2 years), `jacobalberty/unifi:latest` (8 months). (Evidence: `docker images`.)
+- **H4 - app2: UNMS is end-of-life software.** Ubiquiti discontinued UNMS in 2021 (successor UISP). The entire ubnt/unms stack (13+ containers) is EOL and should be migrated. (Evidence: image set + product lifecycle.)
+- **H5 - wphost02: Legacy host not decommissioned; split-brain with app3.** All 8 WordPress sites exist and are actively modified on BOTH wphost02 (RunCloud) and app3 (CloudPanel). No cutover or redirection evident; ambiguous source of truth for client content. (Evidence: identical site lists + Aug file mtimes on wphost02.)
+- **H6 - app3: Dockerized production services have no backup.** Hexclave Stack Auth, Buzz relay, and TransitPin (postgres/clickhouse/minio data) are absent from every backup script. (Evidence: `grep -ril 'buzz|hexclave|transitpin' /root/*.sh /opt/backup-restore/` -> only DB content hits.)
+- **H7 - app3: Five EOL PHP-FPM runtimes running.** PHP 7.1, 7.2, 7.3, 7.4, 8.0 are all end-of-life and unpatched. (Evidence: `systemctl list-units` php*-fpm.)
+
+### MEDIUM
+
+- **M1 - app2: Orphaned container `happy_rosalind`** (2nd Bookstack, no compose project, no host port binding) running with no apparent purpose.
+- **M2 - app2: Large public attack surface.** 40 containers with many 0.0.0.0-bound ports: 53 (DNS), 5000-5150 (Traccar device range), UniFi 8080/8443/8843/8880, UNMS 81/8089/8444, netflow 2055, infinity 23817/23820, support-api 6880, bookstack 6875.
+- **M3 - app2: Bookstack and support-api backup scripts exist but are not scheduled.** Effective RPO = none despite a written script.
+- **M4 - app3: Duplicate WordPress install** debtrecoveryexperts.com under two users (`debtreecoveryexperts` and `drecovery`).
+- **M5 - app3: Single-host SPOF.** ~25 client sites + shared MySQL + CloudPanel all on one VPS with no HA.
+- **M6 - wphost02: node_exporter bound to 0.0.0.0:9100 and postfix on :25** on a legacy host.
+
+### LOW
+
+- **L1 - wphost02: Disk 82% full** (59G/75G).
+- **L2 - app1-bu: `state.db.corrupted` (1.8GB) and stale state DBs linger** consuming disk.
+- **L3 - app3: 2GB swap file `/home/.swap`** on the web data volume with 429MiB in use.
+- **L4 - app3/wphost02: RunCloud agent (`rcsa`) port exposed** on a host slated for decommission - remote management surface retained.
+
+---
+
+## 7. Cross-Host Observations
+
+- **No restore test is documented for any service on any host.** Backups are write-only everywhere a backup exists.
+- **Backup destinations are Wasabi S3 (`hermes-vps-backups`)** for app2, app3, and app1-bu sync; wphost02's working backup is local-only.
+- **Single points of failure are pervasive:** each application is on a single VPS; only Core (Hermes) has a standby, and that standby is config-level only.
+- **Credentials are routinely stored in plaintext world-readable files** on app1-bu and app3 (and inline in compose files on app2).

audit/phase-one/findings/sys-c.md

@@ -0,0 +1,215 @@
+# Sys-C Findings: Backup & Disaster Recovery Verification
+
+**Auditor:** Sys-C (claude-sonnet-5) | **Engagement:** ITPP Phase One Audit | **Mode:** READ-ONLY
+**Scope:** All 27+ backup targets vs live Wasabi S3, restore-test history, RTO/RPO evidence, 3-2-1 compliance.
+**Rule applied throughout (Germaine):** "Backed up is not finished until a restore test is confirmed."
+
+This document was completed across two runs (initial discovery + this resume). All findings below are verified against live S3 listings and source docs as of 2026-08-13, not assumed from documentation alone, per brief rule #3.
+
+---
+
+## 1. Restore-Test Coverage (the critical gap)
+
+Only ONE restore-test event exists in ITPP history: **2026-08-10**, documented in `/root/projects/itpp-infrastructure/disaster-recovery/restore-test-log.md`. It tested exactly **2 of the ~34-37 actual backup targets** enumerated in the backup plan (the plan's own header claims "27 targets," see Finding SYSC-06 on that discrepancy).
+
+| Target tested | Verdict | What was verified | Caveat found |
+|---|---|---|---|
+| Gitea (app2) | PASS | 117 DB tables, 52 repos, 3/3 sampled repos restored with valid git history via `git log`/`git rev-list` | Bare repos missing `refs/` dir (all refs packed) - restore requires manual `mkdir -p refs/heads refs/tags` workaround. Undocumented in the DR runbook until this test. |
+| Vaultwarden (app1) | PASS | 29/29 tables match main vs backup DB, 123 ciphers intact, RSA key valid, WAL recovery clean | None - clean pass. |
+
+**Every other backup target (25-35 of them depending on count) has ZERO restore-test evidence.** This includes every Critical and High tier service in the plan's own RTO/RPO table except Gitea: Hermes Agent (full), Traccar, UniFi/UNMS (the "UISP" critical tier), LiteLLM, n8n, Open WebUI, Twenty CRM. It also includes the standby failover path itself (app1-bu) - no evidence a full standby takeover has ever been drilled.
+
+The restore-test log's own recommendations (written by the same automated tester) explicitly say: "Expand coverage - test remaining backup targets... LiteLLM, OpenWebUI, Hudu, Traccar, etc." and "Full-scale DR drill - after individual tests pass, schedule a coordinated full-stack restore to the standby server." Neither has happened as of this audit.
+
+The restore-test log also flagged (2026-08-10, item 4): `core/vaultwarden/` backups stopped 2026-07-28 at 33 KB, stale/misconfigured - consistent with backup-plan.md's own "Stale S3 Paths - Cleanup Queue" listing that path as safe to delete (service migrated to app1). Not a live risk, just confirms the stale-path cleanup queue is accurate.
+
+---
+
+## 2. Per-Target Cross-Reference Table
+
+Verified live against `s3://hermes-vps-backups/` and `s3://mikrotik-ccr-backups/` via `aws s3 ls --recursive` on 2026-08-13. "Last S3 object" is the actual most recent object under that prefix, not the backup-plan.md "Last Verified" column (which is stale documentation from 07-28/08-08 and was not trusted per brief rule #3).
+
+### Core (152.53.192.33)
+
+| # | Target | Schedule | Last S3 object (live) | Status | Retention (objects seen) | Restore-test |
+|---|---|---|---|---|---|---|
+| 1 | Hermes Agent (full) | Daily 1:00 AM | 2026-08-13 01:02 | OK | 72 dailies | NO |
+| 2 | Hermes Live Sync | Every 15 min | 2026-08-13 06:00+ (continuous) | OK | rolling, includes cron output | NO |
+| 3 | /root Essentials | Daily 3:00 AM | 2026-08-13 03:00 | OK (gaps: 07-11→07-12, 07-21, 07-22, 07-27 missing) | 31 files | NO |
+| 4 | Grafana | Daily 1:30 AM | 2026-08-13 01:30 | OK | 25 dailies | NO |
+| 5 | Uptime Kuma | Daily 1:30 AM | 2026-08-13 01:30 | OK | 25 dailies | NO |
+| 6 | Docker Volumes (raw tars) | Daily 1:30 AM | **2026-08-08** 03:00 (5 days stale as of 08-13) | **STALE** | 55 objects, stopped growing | NO |
+| 7 | Prometheus (TSDB snapshot) | Daily 1:30 AM | 2026-08-13 01:30 | OK | only **4** snapshots retained | NO |
+| 8 | Auth API | 03:15 (+ dup 04:35 broken job) | 2026-08-13 03:15 | OK | 8 dailies | NO |
+| 9 | Timetrex (undocumented - not in backup-plan.md's 27) | ~03:00 daily | 2026-08-13 03:00 | **OK schedule / BROKEN content** - sql.gz only 346 bytes | 2 days seen at this size | NO |
+
+### App1 (152.53.36.131)
+
+| # | Target | Schedule | Last S3 object (live) | Status | Retention | Restore-test |
+|---|---|---|---|---|---|---|
+| 10 | Open WebUI | 2:00 AM | 2026-08-13 02:01 | OK | 26 dailies | NO |
+| 11 | LiteLLM | 3:30 AM | 2026-08-13 02:01 (config) | OK | 43 objects | NO |
+| 12 | n8n | 2:00 AM | 2026-08-13 02:00 | OK | 26 dailies | NO |
+| 13 | MCP Server Configs | 2:00 AM | 2026-08-13 02:01 | OK | 26 dailies | NO |
+| 14 | Vaultwarden | 2:30 AM | 2026-08-13 02:30 | OK | 17 objects | **YES - PASS 08-10** |
+| 15 | Komodo | 3:45 AM | 2026-08-13 03:45 | OK | 17 objects | NO |
+| 16 | DocuSeal | 4:00 AM | 2026-08-13 04:00 | OK | 17 objects | NO |
+| 17 | Twenty CRM | 4:15 AM | 2026-08-13 02:01 (files) | OK | 28 objects | NO |
+| 18 | Kokoro TTS | N/A (stateless, no backup by design) | N/A | N/A - by design | N/A | N/A |
+
+### App2 (152.53.39.202)
+
+| # | Target | Schedule | Last S3 object (live) | Status | Retention | Restore-test |
+|---|---|---|---|---|---|---|
+| 19 | Hudu | 7:00 AM | 2026-08-13 07:00 | OK | 31 dumps | NO |
+| 20 | Gitea | 8:00 AM | 2026-08-13 12:00 UTC | OK | daily dirs, thousands of repo objects | **YES - PASS 08-10 (refs/ caveat)** |
+| 21 | UNMS | 6:00 AM (+ intraday auto) | 2026-08-13 06:00 | OK | 8 objects | NO |
+| 22 | UniFi | 2:00 AM | 2026-08-13 02:00 | OK | 10 objects | NO |
+| 23 | Traccar | 2:30 AM | 2026-08-13 02:30 | OK | 27 dailies | NO |
+| 24 | Technitium DNS | 2:45 AM | 2026-08-13 02:45 | OK | 8 objects | NO |
+| 25 | Dawarich | 4:00 AM | 2026-08-13 02:30 | OK | 8 objects | NO |
+| 26 | RAGFlow | 4:15 AM | 2026-08-13 04:15 | OK | 27 objects | NO |
+
+### App3 (152.53.241.111)
+
+| # | Target | Schedule | Last S3 object (live) | Status | Retention | Restore-test |
+|---|---|---|---|---|---|---|
+| 27 | CloudPanel DB | 3:00 AM | 2026-08-13 03:00 | OK | 27 dailies | NO |
+| 28 | MySQL (all DBs) | 3:00 AM | 2026-08-13 03:00 | OK | 221 objects (multi-DB x days) | NO |
+| 29 | WordPress Files (per-site tars) | 3:00 AM | 2026-08-13 03:02 (confirmed via direct listing; do not trust naive sort) | OK | 273 objects | NO |
+| 30 | Nginx Configs | 3:00 AM | 2026-08-13 03:03 | OK | 26 dailies | NO |
+| 31 | Static Sites | 3:00 AM | 2026-08-13 03:02 | OK | 82 objects | NO |
+| 32 | WordPress Snapshots (CloudPanel local) | 1AM/1PM | Local disk only, /opt/backup-restore/snapshots/, NOT in S3 | Out of S3 scope - 30-day local retention only, single point of failure if app3 disk dies | local only | NO |
+| 33 | Hexclave (Stack Auth) | 3:30 AM | 2026-08-13 03:31 | OK | 6 objects | NO |
+| 34 | modelortho.com | 4:30 AM | **2026-08-08** (5 days stale as of 08-13) | **STALE** | only 2 objects ever (site+configs) | NO |
+
+### wphost02 (Hetzner)
+
+| # | Target | Schedule | Last S3 object (live) | Status | Retention | Restore-test |
+|---|---|---|---|---|---|---|
+| 35 | WordPress (7 sites) | 5:00 AM | 2026-08-13 (per-day directories) | OK - 14-day retention confirmed working (DR-018 resolved) | 14 days x sites, 271+ objects | NO |
+
+### Home Router / WISP
+
+| # | Target | Schedule | Last S3 object (live) | Status | Retention | Restore-test |
+|---|---|---|---|---|---|---|
+| 36 | MikroTik CCR2004 (home gateway) | 6:00 AM | 2026-08-13 06:01 | OK | 30+ dailies since 07-04, config+log pairs | NO |
+| 37 | MikroTik CCR (WISP tower) - DR-017 | 6:00 AM (expected) | **ZERO objects under `wisp-backups/configs/tower*`** | **MISSING entirely** | none | NO |
+
+### External / Gaps
+
+| Target | Schedule | Status | Notes |
+|---|---|---|---|
+| Hetzner Snapshots (API-driven disk snapshots) | Weekly Mon 5:00 AM | **NOT VERIFIED** - backup-plan.md's own "Last Verified" column is blank (" - "); not in S3 so not checkable via `aws s3 ls`; would need Hetzner Cloud API/console access, which is a live-system check outside this audit's read-only S3 tooling. Flag for Phase Two verification. | |
+| SiteGround WordPress (non-MainWP sites) - DR-019 | None found | **MISSING** - `siteground/` prefix returns zero objects | Still open |
+| app1-bu warm standby (state.db) | Continuous per design, actual: none for state.db | **COLD** - see Finding SYSC-01 | Config.yaml syncs every 10 min; state.db (2.1 GB) last touched 2026-07-15, ~28.7 days stale |
+
+**Tally:** 34 real backup targets confirmed live and enumerable (excluding Kokoro N/A and the two external gaps that have no backup at all) + 1 undocumented (timetrex) + 2 confirmed-missing (WISP tower, SiteGround) + 1 unverifiable-in-scope (Hetzner snapshots) + 1 local-only-not-S3 (CloudPanel WP snapshots).
+- **OK (current, correct schedule):** 30
+- **STALE (schedule ok historically but no recent object):** 3 (Docker Volumes raw tars - 5 days; modelortho.com - 5 days; timetrex - running but payload broken/near-empty)
+- **MISSING (zero backup coverage found):** 2 (WISP tower router configs, SiteGround non-MainWP sites)
+- **UNVERIFIABLE with available read-only tooling:** 1 (Hetzner snapshots - needs Hetzner API access, not S3)
+- **Restore-test confirmed:** 2 of 34+ (Gitea, Vaultwarden) - **94%+ of backup targets have never had a restore proven to work.**
+
+---
+
+## 3. RTO / RPO - Evidence-Based vs Plan-Stated
+
+backup-plan.md states an aspirational RTO/RPO table by tier. Below is what the evidence actually supports.
+
+| Tier | Services | Plan-stated RPO/RTO | Evidence-based RPO/RTO | Gap |
+|---|---|---|---|---|
+| Critical | Hermes Agent | RPO ≤1h / RTO ≤4h (assumes live-sync + standby cutover) | RPO: session state ~15 min (live sync confirmed running); full application state via standby is actually **~29 days stale** because state.db is never synced to app1-bu. RTO: **untested** - no full failover drill on record. | Standby is warm for config only, cold for state. Real RTO on a true Core failure is unknown and likely far worse than 4h once state.db has to be rebuilt/accepted-lossy. |
+| Critical | Gitea | RPO ≤1h / RTO ≤4h | RPO: actual backup cadence is **once daily (8 AM)**, so real RPO is up to ~24h, not ≤1h. RTO: one restore test exists, DB+3 sample repos restored in ~30s, but that is a partial single-service test on /tmp, not a timed full-stack recovery - and it required an undocumented manual workaround (`refs/` dirs). | Plan's RPO claim of ≤1h is not supported by the actual cron schedule. RTO figure is aspirational, not measured. |
+| Critical | Traccar | RPO ≤1h / RTO ≤4h | RPO: daily 2:30 AM dump only → real RPO ~24h. RTO: **never tested.** | Same RPO overstatement; RTO entirely unverified. |
+| Critical | UISP (UniFi/UNMS) | RPO ≤1h / RTO ≤4h | RPO: UniFi backs up 2:00 AM daily, UNMS 6:00 AM + intraday auto-backups (best of the critical tier, effectively sub-daily). RTO: **never tested.** | UNMS RPO is reasonably close to plan; UniFi is daily only. Neither has a proven RTO. |
+| High | LiteLLM, n8n, Open WebUI, Vaultwarden, Twenty CRM | RPO 24h / RTO ≤8h | Vaultwarden: RPO 24h matches actual daily cadence, and RTO is the only tier item with real evidence (~2s restore+verify in the 08-10 test, though that is a minimal single-DB test, not a full service standup with docker-compose). The other four: RPO matches (daily), RTO **never tested.** | Plan's RPO is accurate here; RTO is unverified for 4 of 5 services. |
+| Medium | Hudu, UniFi, Komodo, DocuSeal, App3 WP sites, Auth API, Hexclave | RPO 24h / RTO ≤24h | All backups confirmed current daily, consistent with stated RPO. RTO: **never tested for any of them.** | RPO credible, RTO aspirational only. |
+| Low | Grafana, Uptime Kuma, Prometheus, MikroTik CCR, Technitium DNS, Dawarich, RAGFlow | RPO 24h / RTO ≤48h | Backups current daily (except Prometheus only keeps 4 snapshots of retention - a retention risk, not an RPO problem). RTO never tested. MikroTik home gateway confirmed daily; MikroTik tower has **no backup at all**, so its real RTO/RPO is "recovery from scratch," not 48h. | Tower router falls outside even the Low tier's stated objective because it has zero backup. |
+
+**Bottom line:** RPO figures in the plan are mostly defensible for daily-cadence services but overstated (≤1h) for the three daily-only Critical items. RTO figures across every tier are aspirational targets, not measured outcomes - only Gitea and Vaultwarden have ever actually been restored and timed, and both were partial, /tmp-only tests, not full production-equivalent recoveries.
+
+---
+
+## 4. Findings (severity-rated)
+
+### CRITICAL
+
+**SYSC-01 - Warm standby (app1-bu) is cold for application state, contradicting DR docs.**
+Evidence: `state.db` on app1-bu (5.161.225.131) last modified 2026-07-15, ~28.7 days stale, 2.1 GB, while `config.yaml` syncs every 10 minutes. Root cause isolated: `/root/.hermes/scripts/hermes-standby-sync.sh` explicitly excludes state.db by design ("Skips massive state databases"), syncing only config/.env/.bashrc/skills/profiles/plugins/cron/references.
+Why it matters: DR docs and the backup plan describe app1-bu as ready to take over "if Core goes down," but a failover today would bring up a standby with current configuration and roughly a month-old session/job/cron history. Anyone relying on that standby for continuity of active work, not just infrastructure config, will lose weeks of state silently.
+
+**SYSC-02 - Duplicate/conflicting auth-api-backup cron jobs.**
+Evidence: two cron entries invoke `auth-api-backup.sh` - 03:15 (status ok) and 04:35 (status error, exit 1).
+Why it matters: Auth API backs all SSO for ITPP per backup-plan.md's own tier notes. A visibly-failing duplicate job is exactly the kind of noise that gets ignored until the good job also breaks and nobody notices because "the cron always shows an error anyway." Leftover from a rename/migration; needs cleanup, and the failing job should be confirmed harmless (not silently corrupting anything) before removal.
+
+**SYSC-03 - 94%+ of backup targets have never had a restore test.**
+Evidence: the single restore-test event on record (2026-08-10) covered exactly 2 targets (Gitea, Vaultwarden) out of 34+ live targets identified. Every other Critical and High tier service (Hermes Agent full backup, Traccar, UniFi, UNMS, LiteLLM, n8n, Open WebUI, Twenty CRM) has zero restore-test evidence.
+Why it matters: per Germaine's own stated rule, a backup that has never been restored is not a finished backup - it is an assumption. Ransomware, disk failure, or a bad migration could reveal that any of these 32+ untested backups are unusable (as the Gitea test itself discovered a real, previously-unknown restore blocker: missing `refs/` directories). The fact that the one test performed found a real issue is itself evidence that untested backups carry material risk, not theoretical risk.
+
+### HIGH
+
+**SYSC-04 - WISP CCR tower router configs have zero backup coverage (DR-017, still open).**
+Evidence: `s3://mikrotik-ccr-backups/wisp-backups/configs/tower*` returns zero objects. The home gateway router at the same prefix pattern (`configs/home/`) IS backed up daily and current through 2026-08-13.
+Why it matters: if the tower router fails or is misconfigured, there is no saved configuration to restore from - full manual rebuild from memory/notes, unlike the home gateway which has 30+ days of dailies.
+
+**SYSC-05 - Docker Volumes (raw tars) and modelortho.com backups are stale, not merely slow.**
+Evidence: `volumes/` prefix (grafana_data_final, prometheus_data raw tars) last object 2026-08-08, 5 days stale as of audit date, while the DB-level dumps under `core/grafana/` and `core/prometheus/` continue daily and are current. `app3/modelortho/` last object also 2026-08-08, and it has only ever had 2 objects total (site + nginx config), suggesting it may have run once and stopped, or is intentionally infrequent.
+Why it matters: this is independent live confirmation of the DR-002 discovery that `docker-volume-sync.sh` was deleted - its function is only partially replaced. The DB-level SQLite/TSDB dumps for Grafana and Prometheus are fine, but the raw volume-level tars (which would matter for a full container rebuild, not just data recovery) have quietly stopped. modelortho.com's near-total absence of backup history (2 objects ever, both 5 days stale) needs its own look - either it's a low-churn static site where infrequent backup is fine, or its cron entry silently stopped after one run.
+
+### MEDIUM
+
+**SYSC-06 - backup-plan.md's own target count ("27") does not match the actual enumerated inventory.**
+Evidence: the document's header states "Backup Inventory (27 targets)" but the tables that follow it enumerate roughly 34-37 discrete backup line items across Core/App1/App2/App3/wphost02/Home Router/External, plus at least one undocumented target (timetrex) discovered only via live S3 inspection, not present in the plan's tables at all.
+Why it matters: this is a documentation accuracy problem, not an operational one, but it matters for audit trust - if the plan's own headline number is wrong, other "Last Verified" dates in the same document (mostly frozen at 07-28 or 08-08, weeks stale relative to what's actually running) should not be trusted either, which is exactly why this audit verified everything live against S3 rather than the document.
+
+**SYSC-07 - timetrex backup runs but produces a near-empty dump.**
+Evidence: `core/timetrex/timetrex-2026-08-13.sql.gz` is 346 bytes, consistent across the days sampled. The companion storage/config tars are tiny but plausible (115 B, 1.5 KB) for a config-only backup, but a 346-byte SQL dump for what should be an application database is very unlikely to be a real, useful backup.
+Why it matters: the schedule "succeeding" (no cron error, file lands in S3 daily) is actively misleading - it looks healthy on a dashboard but the data almost certainly isn't recoverable. This is exactly the kind of false-green backup that a restore test would have caught immediately. Root cause not assumed here per brief rule 2 (could be an empty/decommissioned database, a broken mysqldump auth, or an app issue) - needs Phase Two investigation, not remediation.
+
+**SYSC-08 - Docker Volume Sync function (Prometheus/Grafana raw data) claimed-but-not-restore-verified.**
+Evidence: `docker-volume-sync.sh` was deleted per DR-002; its function is claimed to be covered by `hermes-backup.sh`, but this audit found the actual raw-volume backups (`volumes/` prefix) are stale since 08-08 (see SYSC-05) and no restore test exists for any Grafana/Prometheus backup, DB-level or volume-level.
+Why it matters: the claim of coverage is not supported by live evidence; the safety net here is unverified on two independent axes (currency and restorability).
+
+**SYSC-09 - Prometheus TSDB snapshot retention is unusually shallow (4 objects).**
+Evidence: `core/prometheus/` holds only 4 snapshot objects vs 25+ for comparable daily services (Grafana, Uptime Kuma).
+Why it matters: if a problem with Prometheus data isn't noticed within roughly 4 days, there may be no earlier snapshot left to recover from. Likely an intentional retention policy given TSDB snapshot size, but worth confirming it's intentional rather than a bug.
+
+**SYSC-10 - CloudPanel WordPress snapshot layer (app3, local-only) has no offsite copy.**
+Evidence: `/opt/backup-restore/snapshot.sh` writes to local disk (`/opt/backup-restore/snapshots/`, 30-day retention) only; nothing under this specific mechanism reaches S3 (the separate `app3/wordpress/` S3 backups are a different script/mechanism and are current).
+Why it matters: this specific snapshot layer is a single point of failure - if app3's disk fails, these particular snapshots are gone regardless of retention window. The S3-backed `app3/wordpress/` mechanism is a real offsite safety net for the same sites, so overall WordPress exposure on app3 is mitigated, but the local snapshot layer itself provides false comfort if someone assumes "snapshots" means "offsite."
+
+### MEDIUM (carried forward, unchanged from prior run)
+
+**SYSC-11 - DR-019: SiteGround WordPress sites outside MainWP have no S3 backup.** `siteground/` prefix returns zero objects. Still open.
+
+**SYSC-12 - DR-015: service-health-check/apex-mail-watchdog failing on real remote outages.** Still open, unchanged.
+
+### LOW / INFORMATIONAL
+
+**SYSC-13 - sys-b.md vs live check discrepancy on Gitea/Hudu/UNMS/UniFi.**
+sys-b.md reported these backups as "silently failing." This audit's live S3 check found all four running on schedule with fresh, current objects through 2026-08-13 (Hudu 07:00, Gitea 08:00/12:00 UTC, UNMS 06:00, UniFi 02:00). Flagging for conductor reconciliation rather than resolving unilaterally - either sys-b observed a transient failure window, checked a stale cache/log rather than live S3, or the issue was fixed between sys-b's check and this one. Recommend the conductor compare exact check timestamps between the two runs before deciding which report is stale.
+
+**SYSC-14 - /root Essentials backup has 3 schedule gaps (07-11→07-12, 07-21, 07-22, 07-27 missing) but is otherwise current.**
+Not fatal (the job clearly runs most days and is current through 08-13), but worth a Phase Two look at why specific days were skipped (server reboot, cron collision, disk pressure) rather than assuming a one-off blip.
+
+**SYSC-15 - Hetzner weekly snapshot verification is outside this audit's read-only S3 tooling.**
+backup-plan.md lists "Last Verified: - " (blank) for Hetzner Cloud API-driven snapshots. This audit could not verify snapshot existence/currency using `aws s3` tooling because they are not stored in S3. Flag for Phase Two: someone with Hetzner console/API read access should confirm snapshots are actually being taken weekly as claimed.
+
+---
+
+## 5. 3-2-1 Rule Compliance (quick read)
+
+Most services have: (1) live production copy, (2) daily S3 backup at Wasabi (offsite), and for the app1-bu standby (3) a config-level copy on a second provider (Hetzner) - but that third copy is state-incomplete per SYSC-01. Genuine 3-2-1 gaps:
+- WISP tower router: only 1 copy (live device config), zero backups (DR-017).
+- SiteGround non-MainWP sites: only 1 copy (live host), zero backups (DR-019).
+- CloudPanel local WordPress snapshots: 2 copies but both effectively on the same physical host (live + local snapshot dir) until the separate S3-backed `app3/wordpress/` mechanism is counted as the true offsite leg - which it is, so app3 WordPress overall is fine; the local snapshot layer specifically is not.
+- Docker volume raw tars: technically offsite but stale 5 days, functionally degrading toward non-compliance if not fixed.
+
+---
+
+## Notes on Methodology / Limitations
+
+- All S3 currency checks used `aws s3 ls --recursive` against `s3://hermes-vps-backups/` and `s3://mikrotik-ccr-backups/`, cross-checked with targeted per-prefix listings where a naive lexical sort across mixed filenames (different site/service names sharing a date) produced a misleading "last" result (caught and corrected for `app3/wordpress/`).
+- No restore, config change, or live-system alteration was performed by this auditor, per brief rule 1. All restore-test evidence in this report comes from the pre-existing 2026-08-10 log, not from actions taken during this audit.
+- Hetzner API-based snapshot verification and any live SSH-based service checks beyond what the prior Sys-C run already completed were not repeated in this resume, consistent with the instruction to finish, not redo, prior discovery.

audit/phase-one/policy-and-procedure.md

@@ -0,0 +1,121 @@
+# ITPP Infrastructure - Policy & Procedure Document
+
+**Version:** 1.0 (Phase One deliverable)
+**Date:** 2026-08-13
+**Owner:** Germaine Brown (final authority on all exemptions)
+**Enforcement:** Sho'Nuff (Hermes) as policy-adherence gate; see Skill Spec for the enforcement mechanism.
+
+---
+
+## 1. Purpose
+
+This document defines the operating rules for IT Pro Partner (ITPP) infrastructure. It exists to close the gap between what the documentation claims and what the estate actually does - the single most consistent theme of the Phase One audit. Every rule below maps to a finding that was either observed in the wild or missing in practice.
+
+---
+
+## 2. Change Management
+
+**Policy:** No live configuration change to production infrastructure without a traceable record of (a) what changed, (b) who authorized it, (c) when, and (d) how to roll back.
+
+**Procedure:**
+1. Before any change to Core/app1/app2/app3/app1-bu/wphost02, the credential is retrieved from Vaultwarden (never from a local plaintext file, shell history, or a prior report).
+2. Change is recorded in the changelog at the moment it is made - **old name → new name, date, reason** - not discovered later. (Applies to renames, IP changes, credential rotations, and config moves.)
+3. Any change that affects a dependency must verify the dependent records (A, CNAME, env files, backup targets, watchdog targets) before being declared complete.
+4. Rollback path is stated in the change record before the change is applied.
+
+**Mapping:** Audit C1/C4/C9 - the Docker/UFW bypass, plaintext-credential sprawl, and the stale standby all trace to undocumented or unverified changes.
+
+---
+
+## 3. Documentation Synchronization (Docs-Sync)
+
+**Policy:** Documentation describes what is *actually running*, verified against live systems, not memory. No document may claim a control that does not exist.
+
+**Procedure:**
+1. Any infra change must include a corresponding update to `docs.itpropartner.com` (or the canonical docs repo) in the same work session.
+2. Say-do verification: claims like "credentials sanitized", "validated routing", or "whole-site auth gate" must be backed by a check against the live system at the time the claim is written.
+3. Stale sections are archived, not silently retained: any section with no update in 30 days is moved to an `archive/` tree or explicitly marked stale.
+4. The docs build must have a single authoritative source. Duplicate builds (e.g. the app3 split-brain `/home/ippadmin/htdocs/` copy) are deleted, and the served copy is wired to a publish CI job.
+5. Credential-sensitive material is never published to an unauthenticated path, even in changelogs or historical reports.
+
+**Mapping:** Docs-W say-do gaps (unsanitized key inventory, unverified auth gate, public credential in changelog), Sys-C backup-plan undercount (27 vs 34 targets), split-brain docs build.
+
+---
+
+## 4. Backup & Disaster Recovery Standards
+
+**Policy:** "Backed up" means **restore-tested**. A backup that has never been restored is an unproven assumption, not a backup. Every service is in the backup matrix.
+
+**Procedure:**
+1. **Coverage:** every live service has a scheduled backup with a documented destination, retention, and RTO/RPO. The master `backup-plan.md` must match live discovery (currently 27 documented vs 34 live - reconcile to 34).
+2. **Restore testing:** any new backup target is restore-tested within 7 days of being added. The full set is re-tested on a rolling cadence (at minimum, one restore test per critical service per quarter). Restore tests are logged in `restore-test-log.md` with evidence (object count, table count, sample-record verification).
+3. **Database correctness:** `pg_dump`/`mysqldump` targets are verified against the live database name from config at schedule time, and a failed dump is treated as a failure (alerts, does not fail silently).
+4. **3-2-1:** critical data exists in 3 copies, on 2 media, 1 off-site. Wasabi S3 is the off-site target; host-local-only backups (e.g. wphost02) are a violation, not an acceptable state.
+5. **Provider diversity:** at least one backup device remains off-premise and on a different provider than the live estate (netcup vs Hetzner). A netcup outage must not take down both the live estate and its standby.
+6. **Standby readiness:** the standby must be data-ready to the documented RPO. If state databases are intentionally excluded from sync, the RPO is documented as "config-only" and the DR plan reflects it - the contradiction between a "warm standby" claim and a 4-week-stale state DB must not persist.
+
+**Mapping:** C5 (LiteLLM DB never backed up), C8 (wphost02 75% unprotected), C9 (standby not data-ready), Sys-C (2/34 restore-tested), D2 (untested Hudu/UNMS/UniFi).
+
+---
+
+## 5. Access & Offboarding
+
+**Policy:** Least privilege. No shared omnipotent credentials. Every person (or agent) has a named, revocable identity.
+
+**Procedure:**
+1. **No shared root keys.** The single `itpp-infra` key is split into per-host or per-role keys; each has a named owner and is revocable independently.
+2. **No blanket `NOPASSWD:ALL`.** Privileged escalation is via named sudoers entries scoped to the commands required; the `ippadmin NOPASSWD:ALL` and `clpctlWrapper ALL` rules are retired.
+3. **MFA is mandatory** on every admin console that supports it (Gitea, CloudPanel, Vaultwarden, Grafana, Wazuh, Hudu, LiteLLM). Open registration is closed; captcha is enabled where guest signup exists.
+4. **Credentials live in Vaultwarden.** Plaintext copies in `.env`, systemd units, scripts, or the filesystem are rotated and removed. Vaultwarden is the single source of truth.
+5. **Offboarding** is immediate on separation: revoke keys, deactivate accounts, rotate any secret the person had access to, and verify no active sessions remain. The shared-key era's inability to do this cleanly is the reason for rules 1-2.
+6. **Audit trail:** every admin action on a production host is attributable. Onboarding/offboarding events are logged with a timestamp and actor.
+
+**Mapping:** Sec-A (single key, shared admin accounts, NOPASSWD:ALL), C7 (Grafana default creds), Sec-B (no MFA anywhere, open Gitea registration).
+
+---
+
+## 6. Segmentation for New Entities
+
+**Policy:** New hosts, sites, or products are placed in a named trust tier at creation time, never added to the flat "everything everywhere" group.
+
+**Procedure:**
+1. **Three tiers:** `internal` (ops tooling), `client` (client sites), `product` (micro-SaaS). A new entity is assigned to exactly one tier on day one.
+2. **Tailscale ACLs** enforce the tier boundaries - tags are applied before the host is reachable, and the default allow-all is removed.
+3. **Network exposure:** any Docker service binds to `127.0.0.1` and is reached via the reverse proxy; no new service publishes directly to 0.0.0.0. A `DOCKER-USER` UFW chain is the standing gate for any exception.
+4. **Database isolation:** new sites/products get their own database user and schema, never shared credentials on a shared engine. Products get their own DB server where isolation is a product requirement.
+5. **DNS:** every new host has A/CNAME/SPF/DMARC records verified at creation; no decommissioned IP is left pointed at in DNS.
+
+**Mapping:** C1 (Docker bypass), C2 (no segmentation), C6 (shared Percona), NetEng-A DNS hygiene findings.
+
+---
+
+## 7. Recurring Audit Cadence
+
+**Policy:** The Phase One audit is a baseline, not a one-off. It repeats on a fixed cadence with a fixed scope.
+
+**Procedure:**
+1. **Quarterly** - the full read-only audit re-runs (the Phase One subagent roster and methodology are reused; see Skill Spec / audit brief).
+2. **Monthly** - a lighter sweep: secrets-in-plaintext grep, open-port diff, backup freshness check, restore-test log review, patch-lag check.
+3. **Ad hoc** - on any security advisory affecting a deployed component (old-image findings like rabbitmq/browserless/grafana), an immediate targeted audit of that component runs.
+4. **Findings lifecycle:** every finding is tracked in the DR issue log (`/root/.hermes/references/dr-issue-log.md`) with root cause, fix, and verification date. Findings do not silently age out - they are resolved or explicitly accepted by Germaine.
+5. **Independence:** severity ratings are re-checked by an independent model instance before a report is finalized (the Batch 3 "Indep" step), to catch false positives and overstated severities.
+
+**Mapping:** Sec-B patch-lag findings, old-image findings, D1/D2 (which the independence check is designed to catch).
+
+---
+
+## 8. Exemptions
+
+**Policy:** Only Germaine authorizes a departure from this document. No exemption is assumed - it is requested, justified, approved, and recorded.
+
+**Procedure:**
+1. Any request to depart from a policy is raised as an exemption request with a business justification.
+2. Germaine approves or denies. Approval is recorded in the **Running Exemptions Document** with: date, requester, the provision being departed from, the request + business justification, Germaine's authorization, and whether it is one-time or ongoing.
+3. Ongoing exemptions carry a follow-up review date. One-time exemptions are closed when the exception ends.
+4. The Phase One example: the public-repo `itpp-infrastructure` credential exposure was surfaced as a Critical finding; Germaine deferred action ("leave the repo alone for now"). That deferral is recorded as an open one-time exemption with a Phase Two follow-up, not silently dropped.
+
+**Mapping:** D3 (public-repo deferral) - this is the template entry for the Running Exemptions Document.
+
+---
+
+*End of Policy & Procedure Document v1.0.*

audit/phase-one/report.md

@@ -0,0 +1,338 @@
+# ITPP Infrastructure Audit - Phase One Final Report
+
+**Engagement:** Read-only discovery, audit, and documentation certification.
+**Date:** 2026-08-13
+**Conductor:** Sho'Nuff (deepseek-v4-pro) + claude-sonnet-5 (report/QA synthesis)
+**Status:** COMPLETE (read-only). Zero live modifications performed on any target.
+
+---
+
+## Table of Contents
+
+1. Executive Brief
+2. Discovery Summary (verified inventory)
+3. Findings (prioritized, severity tiers with rationale)
+4. Recommendations (mapped to findings, effort estimate)
+5. Documentation Status
+6. Infrastructure Separation Assessment
+7. Disagreements (documented, not resolved - Germaine resolves)
+8. Independence-Check Appendix (Indep severity review - complete)
+
+---
+
+## 1. Executive Brief
+
+Six servers were audited read-only on 2026-08-13: **Core, app1, app2, app3** (Netcup) and **app1-bu, wphost02** (Hetzner), plus the Gitea repository estate, DNS, Wasabi S3 backups, and the docs site.
+
+The estate is functionally sound at the application layer but has **five structural weaknesses that compound each other**. Any one of them is a real finding; together they mean a single compromise today is an estate-wide incident, not a contained one.
+
+**The five systemic themes:**
+
+1. **No segmentation, and the firewall does not do what operators think it does.** Docker-published ports bypass UFW on 4 of 6 hosts, leaving roughly 20 management consoles (Wazuh, UniFi, UNMS/UISP, Grafana, CloudPanel, backup-restore UI, Gitea, MinIO, RAGFlow, Infinity DB, browserless, camofox) reachable from the public internet despite none of them appearing in any allow-list. There is no VLAN or subnet boundary between hosts, and the Tailscale mesh has no ACL tags - personal devices sit in the same allow-all group as production.
+
+2. **Credentials are stored in plaintext in world-readable locations.** Two full unsanitized credential-inventory files remain on Core; app1-bu holds ~20 plaintext secrets (including root passwords for app1/app2/app3) in a world-readable `.env`; production API keys are hardcoded in world-readable systemd units; app3's MySQL root password sits in a world-readable backup script; and the public repo `itpp-infrastructure` re-leaks live admin passwords inside a prior audit's own report.
+
+3. **Backups are write-only.** 30 of 34 live targets have a backup, but only **2 (Gitea, Vaultwarden) have ever been restore-tested**. LiteLLM's Postgres database - which holds every AI API key, routing table, and spend record - is never backed up at all (the dump targets a nonexistent database name and fails silently nightly). wphost02 has no scheduled backup for 6 of 8 WordPress databases.
+
+4. **The SIEM monitors nothing but itself.** Wazuh is deployed and healthy, but `agent_control -l` shows zero enrolled agents across the estate. No centralized log forwarding exists anywhere. Grafana runs with default `admin/admin` credentials, publicly reachable, with no MFA - and no admin console in the estate has enforced MFA.
+
+5. **The warm standby is not actually ready.** app1-bu syncs config files but deliberately skips the state databases; its `state.db` is ~28 days stale. On failover today, Hermes would come up without the last month of session, memory, and state.
+
+**Bottom line:** the platform works day to day, but resilience, credential hygiene, and detection are all materially weaker than the documentation claims. This report maps every gap to a concrete Phase Two remediation with an effort estimate. Nothing was changed during Phase One.
+
+---
+
+## 2. Discovery Summary (verified inventory)
+
+### 2.1 Servers
+
+| Server | Provider / model | Public IP | OS / kernel | Role (verified) |
+|---|---|---|---|---|
+| Core | Netcup RS 2000 | 152.53.192.33 | Debian 13 / 6.12.94 | Hermes host + Grafana :3002, Prometheus, Super Search MCP :8899, backup orchestration |
+| app1 | Netcup RS 4000 | 152.53.36.131 | Debian 13 / 6.12.95 | LiteLLM/admin-ai, Wazuh SIEM, Twenty CRM, Komodo, n8n, Vaultwarden, Caddy |
+| app2 | Netcup RS 4000 | 152.53.39.202 | Debian 13 / 6.12.95 | Hudu, UNMS/UISP, UniFi, Traccar, Gitea, Dawarich, Technitium DNS (~40 containers) |
+| app3 | Netcup RS 4000 | 152.53.241.111 | Debian 13 / 6.12.95 | CloudPanel shared web host (~24 sites) + shared Percona MySQL + Hexclave/Buzz Docker |
+| app1-bu | Hetzner CPX21 | 5.161.225.131 | Ubuntu 24.04 | Warm standby for Core (config-only sync; state DB ~28d stale) |
+| wphost02 | Hetzner | 5.161.62.38 | Ubuntu 24.04 | Legacy WordPress/RunCloud - **still live, not decommissioned** (split-brain with app3) |
+
+All six reachable via the shared `itpp-infra` SSH key (single key, single blast radius).
+
+### 2.2 Shadow IT / drift surfaced
+
+- **wphost02 is still serving 8 WordPress sites** that also exist on app3 - an unresolved split-brain migration state with no cutover.
+- **HotNow** was not found deployed on any audited host (needs confirmation of live/decommissioned status).
+- **Three ad-hoc `python3 -m http.server`** processes on Core, one serving `/tmp`, all bound to 0.0.0.0 as root.
+- **Three runaway `fix_dict.py`** processes pegging 3 cores on app1 for 14+ days.
+- **Orphaned container** `happy_rosalind` (2nd BookStack, no compose project) on app2.
+- **Stale duplicate docs build** on app3 (root-owned, ~29h older than the served copy).
+
+### 2.3 Repository estate (Gitea)
+
+56 total repos under org `ippadmin`. ~9 active, ~35 stale (mostly one Aug-8 scaffolding event), 1 orphaned (`itpp-infra`, remote deleted - local clone is now the only copy), 0 abandoned stubs. 11 repos exist on Gitea but were never cloned locally. Default-branch mismatch (`itpp-infrastructure` tracks `main` locally vs `master` on Gitea) risks fresh-clone confusion.
+
+---
+
+## 3. Findings (prioritized)
+
+Severity legend: **Critical** = publicly reachable control plane, unrecoverable data-loss risk, or single-compromise = estate-wide blast. **High** = material security or resilience gap with significant impact. **Medium** = defense-in-depth gap or hygiene issue. **Low** = minor.
+
+Consolidated severity counts across all nine auditor files (deduplicated where multiple auditors surfaced the same root cause): **Critical ~23, High ~34, Medium ~36, Low ~17.** Many overlap - the same Docker/UFW bypass is the root mechanism behind findings in NetEng-A, NetEng-B, and several Sys-B items. The Indep severity review (§8) net-adjusted three ratings: Sec-A-02 restored to Critical (C10), Git-A Finding 2 escalated High→Critical (folded into C4), and Sec-B-03 Technitium confirmed High - all documented in §7.
+
+### 3.1 Critical findings (consolidated by root cause)
+
+**C1. Docker published-port rules bypass UFW, exposing ~20 management consoles to the internet.**
+Evidence (NetEng-A): on Core/app1/app2/app3, `docker run -p <port>` inserts DNAT rules into `nat/PREROUTING` and ACCEPT rules into `filter/FORWARD/DOCKER`, which are processed before UFW's `filter/INPUT`. UFW's allow-lists do not contain these ports, yet they are publicly reachable.
+Why it matters: every operator believes UFW is the security boundary. It is not. Wazuh indexer/dashboard/API (app1), UniFi controller (app2), UNMS/UISP (app2), Gitea SSH (app2), MinIO console, RAGFlow, Infinity DB, browserless, camofox, Twenty CRM, Komodo, and the Buzz relay are all on the public internet with no firewall gate.
+
+**C2. No network segmentation exists anywhere in the estate.**
+Evidence (NetEng-B): all hosts are flat on public IP space with no VLAN/subnet; the Tailscale overlay has no ACL tags on any node (default allow-all); personal devices share the same group as production. Verified Core→app1 and app1→app2 reachable over public IPs, unfiltered.
+Why it matters: compromise of any single host is a direct network path to every other host and to personal devices. No lateral-movement friction.
+
+**C3. Wazuh SIEM is the highest-leverage target and it is both public and monitoring nothing.**
+Evidence (NetEng-A + Sec-B): the Wazuh stack is publicly exposed via the Docker bypass, AND `agent_control -l` shows zero enrolled remote agents - the SIEM monitors only its own loopback.
+Why it matters: the one system built to detect compromise is itself the most exposed control plane and is blind to the other five hosts.
+
+**C4. Plaintext credentials in world-readable locations across the estate.**
+Evidence (Sec-A + Sec-B + Sys-A + Sys-B + Git-A):
+- Two unsanitized copies of the full `key-inventory.md` credential inventory remain on Core (one inside Hermes's own reference directory).
+- Private `scripts` repo hardcodes the MSP-backdoor admin password that is reused across client onboardings - the single most consequential live credential in the estate (Git-A Finding 1).
+- app1-bu `/root/.hermes/.env` (mode 644) holds ~20 plaintext secrets including root passwords for app1/app2/app3, Telegram bot token, Cloudflare/Netcup/SyncroMSP tokens, and eight AI-provider keys.
+- `rally.service`, `seemytrip.service`, `giftaroast.service` hardcode `JWT_SECRET`, `DEEPSEEK_API_KEY`, `ADMIN_AI_KEY`, Twilio creds in world-readable unit files.
+- app3 MySQL root password in `/opt/backup-restore/snapshot.sh` (mode 775) and `/root/backup.sh`.
+- Public repo `itpp-infrastructure` re-leaks live admin passwords inside a prior audit report (see §7 - deferred per Germaine).
+Why it matters: any local user or any single compromised service can read the entire secret estate.
+
+**C5. LiteLLM Postgres database is never backed up.**
+Evidence (Sys-A): `aws s3 ls s3://hermes-vps-backups/app1/litellm/` shows only config YAML objects. app1 `/root/backup.sh` runs `pg_dump` against database `litellm`, but the live `config.yaml` uses `litellm_db` - the dump targets a nonexistent database and fails silently every night.
+Why it matters: LiteLLM's Postgres holds every API key, model routing table, spend/budget record, and the admin-ai provider config. A failure means total reconstruction by hand.
+
+**C6. app3 runs ~24 sites (internal + client + product) on one shared MySQL instance with no tenant boundary.**
+Evidence (NetEng-B + Sys-B): a single Percona `mysqld` backs internal ops sites (mainwp, support, panel), client sites (katiewatts, modelortho, vigilanttac, boxpilotlogistics, timapta), and products (transitpin, myverdicttank, buzz, hexclave).
+Why it matters: a SQLi or credential leak on any one site is a plausible path to every other site's data on the same engine.
+
+**C7. Grafana running default admin credentials, publicly reachable, no MFA.**
+Evidence (Sec-B + NetEng-A): `docker inspect grafana` shows `GF_SECURITY_ADMIN_PASSWORD=admin`; port 3002 is explicitly allowed by UFW to Anywhere; no SSO/OAuth configured.
+Why it matters: default `admin/admin` on an internet-reachable observability console is a near-zero-effort compromise path, and Grafana holds dashboards of the entire monitoring estate.
+
+**C8. wphost02 has no effective backup for 6 of 8 WordPress databases, yet is still live.**
+Evidence (Sys-B): scheduled `db-dump.sh` backs up only 2 of 8 DBs to local disk (7-day retention); the full offsite S3 script is not in any crontab. The host is still serving all 8 sites.
+Why it matters: a live client-content host with ~75% of its databases unprotected.
+
+**C9. Warm standby (app1-bu) is not data-ready.**
+Evidence (Sys-B + Sys-C): sync covers only config/skills/plugins/cron/references; `state.db` (~2.1GB), `memory_store.db`, and `sessions/` are stale at Jul 15 (~4 weeks). Failover today would restore Hermes without the last month of state.
+Why it matters: the DR plan's core assumption (warm standby can take over) is false for application state.
+
+**C10. A single SSH key unlocks passwordless root on 5 of 6 hosts with no MFA and no segmentation to contain it.**
+Evidence (Sec-A + NetEng-B): the shared `itpp-infra` key gives passwordless root sudo across the estate; it lives on Core alongside WireGuard keys to the home network + WISP towers and a live autossh tunnel into wphost02's MySQL. Per the severity legend, this is the definition of "single-compromise = estate-wide blast."
+Why it matters: one key compromise, one leaked private key, or one compromised workstation with the key loaded is a full estate takeover. Restored to Critical per the Indep severity review - see §7 D4.
+
+### 3.2 High findings (representative)
+
+- **`ippadmin` has `NOPASSWD:ALL` sudo on 4 of 6 hosts**, reachable by the same shared key (Sec-A).
+- **app3's `clpctlWrapper` sudoers rule grants ALL accounts** (including ~28 per-client site accounts) a passwordless root escalation path if the wrapper has any input-validation gap (Sec-A).
+- **Gitea and CloudPanel each run on one shared admin account** with zero per-person accountability (Sec-A).
+- **No admin console has enforced MFA** - Grafana, Wazuh, Gitea (open registration, no captcha), Hudu, UniFi, UNMS (SSO explicitly nulled), Technitium (literal `changeme` in env), CloudPanel, LiteLLM, Vaultwarden (Sec-B).
+- **Technitium DNS runs with `DNS_SERVER_ADMIN_PASSWORD=changeme`** in the live container env - a default-credential flag on the estate's authoritative DNS (Sec-B).
+- **app3 patch cadence ~4 weeks stale** with an unapplied security kernel + postfix update (Sec-B).
+- **Core has no fail2ban, no unattended-upgrades, no auditd** - the weakest OS hardening of all 6 hosts, on the highest-value control-plane host (Sec-B).
+- **Every custom service runs as root**; only hermes-voice and Caddy run non-root. One compromised service = full host takeover (Sys-A).
+- **Very old images in production:** browserless/chrome (2yr), grafana 11.4.0 (20mo), wazuh-indexer 4.9.2 (21mo), rabbitmq 3.7.28 (5yr, EOL CVEs) (Sys-A, Sys-B).
+- **UNMS/UISP is EOL software** (Ubiquiti discontinued 2021); the entire 13-container stack should be migrated (Sys-B).
+- **app1 Caddyfile is not backed up** - the entire reverse-proxy routing config would need manual reconstruction (Sys-A).
+- **Hermes gateway supervised only by the root user-manager**, with a socat unit referencing a nonexistent system unit (fragile SPOF) (Sys-A).
+- **Port 8200 collision** between `hermes-control-deck` and `pipeline-api` - one is silently shadowed (Sys-A).
+- **Three runaway `fix_dict.py` processes pegging 3 cores for 14+ days** on app1 (Sys-A).
+- **app3 Docker services (Hexclave Stack Auth, Buzz relay) + TransitPin have no backup** (Sys-B).
+- **app2 backup.sh silently skips Hudu/UNMS/UniFi** (local scripts missing) - see §7 reconciliation (Sys-B, conductor-verified).
+- **Public DNS hygiene:** apex A record + ~10 legacy subdomains point to a decommissioned GCP host; SPF record malformed (concatenated strings); DMARC `p=none`; fleettracker360.com has no MX/SPF/DMARC (NetEng-A).
+- **Private `hermes-recovery` repo commits a live MySQL password + Gitea API token** (Git-A).
+- **WISP tower router (DR-017) has zero backup coverage** - `s3://mikrotik-ccr-backups/wisp-backups/configs/tower*` returns zero objects versus 30+ dailies for the home gateway at the same prefix pattern. An operational device with a total absence of config backup, not merely an untested one (Sys-C SYSC-04).
+
+### 3.3 Medium / Low (summarized)
+
+Medium: monitoring exporters bound to 0.0.0.0; ad-hoc http.servers; socat→Hermes on 0.0.0.0; MySQL X on `*`; weak L2TP/IKEv1 crypto on the tower VPN; plaintext VPN creds; no centralized log forwarding; Gitea open registration; secrets-sprawl git-grep hits needing per-file triage (Core 6,296 / app1bu 5,504 / app2 ragflow 1,275 - mostly false positives); orphan container; duplicate WordPress install; single-host SPOFs; mysql-tunnel disables host-key verification; dead cron (`docker-volume-sync.sh`); no swap on Core/app1; duplicate/conflicting auth-api-backup cron jobs (a working 03:15 job + a failing leftover 04:35 job - alert-fatigue risk, not a live data-loss condition today; Sys-C SYSC-02).
+
+Low: avahi on public interface; app1-bu stale WireGuard rule + Tailscale name drift; leftover Docker volumes; disk 82% full on wphost02; `.aws` dir 775 on app1; 5 EOL PHP-FPM runtimes on app3.
+
+---
+
+## 4. Recommendations (mapped to findings)
+
+Phase Two ordering. Effort: **S** = under 1 hour, **M** = half day, **L** = 1-2 days, **XL** = multi-day project.
+
+| # | Recommendation | Maps to | Effort | Notes |
+|---|---|---|---|---|
+| 1 | Bind all Docker publishes to `127.0.0.1` and route through the reverse proxy, OR add a `DOCKER-USER` UFW chain. Do this before any other firewall work. | C1 | M-L | Highest leverage single change; closes ~20 public holes at once |
+| 2 | Stand up Tailscale ACL tags: separate `servers`, `personal`, `mgmt` groups; drop the default allow-all. | C2 | M | Tailscale ACLs exist precisely for this |
+| 3 | Enroll a Wazuh agent on all 5 non-manager hosts. | C3 | M | Turns the SIEM from self-monitoring to estate-wide |
+| 4 | Rotate every credential found in plaintext (inventory the full list first), move to Vaultwarden, and chmod 600 all secret-bearing files. | C4 | L-XL | Highest-risk secrets first: app1-bu .env, key-inventory.md, systemd units, app3 snapshot.sh |
+| 5 | Fix LiteLLM `pg_dump` to target `litellm_db`, verify a restore, then schedule it. | C5 | S-M | The dump command exists but has the wrong DB name |
+| 6 | Segment app3 databases per tenant (separate MySQL users/schemas per site; move products to their own DB servers). | C6 | L-XL | Product isolation depends on this |
+| 7 | Change Grafana admin password, enable SSO or TOTP, restrict :3002 to Tailscale. | C7 | S | Memory already has the rotated Grafana credential in Vaultwarden |
+| 8 | Schedule wphost02's full S3 backup, then decommission wphost02 after confirming the app3 cutover. | C8 | M | Resolves both the backup gap and the split-brain |
+| 9 | Extend standby sync to include state DBs (or accept a documented config-only standby with a revised RPO). | C9 | M | Explicitly contradicts DR-009 as-is |
+| 10 | Split the single `itpp-infra` key into per-host or per-role keys; retire `NOPASSWD:ALL` for `ippadmin` in favor of named sudo. | High | M | Reduces single-key blast radius |
+| 11 | Enforce MFA on Gitea, CloudPanel, Vaultwarden; close Gitea registration + enable captcha. | High | M | Native TOTP support already exists in all three |
+| 12 | Enable fail2ban + unattended-upgrades + auditd on Core; remediate app3's stale patch cadence. | High | S-M | Bring Core to parity with the other 5 hosts |
+| 13 | Back up app1 Caddyfile; back up app3 Docker services (Hexclave, Buzz) + TransitPin; restore-test Hudu/UNMS/UniFi. | High | M | Closes the write-only backup gap |
+| 14 | Migrate UNMS→UISP; pin/upgrade old images (browserless, rabbitmq, grafana, wazuh-indexer). | High | L | UNMS is EOL; rabbitmq 3.7 has known CVEs |
+| 15 | Fix port 8200 collision, kill runaway `fix_dict.py`, move Hermes gateway to a system unit. | High | S | Hygiene cleanup with real availability impact |
+
+### 4.1 Effort rollup
+
+- **Quick wins (S):** #7, #12 (partial), #15
+- **Half-day (M):** #2, #3, #5, #8, #9, #10, #11, #13
+- **Multi-day (L/XL):** #1, #4, #6, #14
+
+Recommended sequencing: **#1 → #4 → #3** first (close the biggest exposure, rotate the secrets, turn on detection), then the remainder in listed order.
+
+---
+
+## 5. Documentation Status
+
+The documentation is **materially out of sync with reality** across two dimensions:
+
+**Say-do gaps (Docs-W):**
+1. Key Inventory claims secrets are "sanitized" (2026-07-23), but Sys-A/Sys-B found plaintext secrets in world-readable locations (see C4).
+2. Ops Portal changelog published a real historical admin credential in plaintext on an unauthenticated public site for ~3 weeks before the auth gate went live 2026-08-10.
+3. The Docs Auth Gate doc describes JWT + allowlist protecting the whole site, but no test verifies the validator (port 8099) - unverified access control.
+4. The Model Chain doc governs LiteLLM routing, but the LiteLLM Postgres DB is never backed up (see C5).
+5. The app2 Caddyfile audit presents "validated" routing but omits the directly reachable Docker/UFW bypass services (see C1).
+
+**Coverage drift (Docs-W):** 12 top-level sections, only 4 current (ITPP Infrastructure, ITPP Standards, TransitPin, VerdictTank - changelogs ≤4 days); 8 stale (HomeLab, Scripts, FleetTracker360, LaunchCheck, Shark Game, Apex Track, BoxPilot, OSINT Tool - 3-5+ weeks; two are dead stubs).
+
+**Backup-plan undercount (Sys-C):** `backup-plan.md` documents 27 targets; live discovery found **34**. The documented plan undercounts reality by 7 - itself a documentation-drift finding.
+
+**Split-brain docs build (Docs-W):** two nearly identical MkDocs builds on app3 - the served `/home/docs/htdocs/` (nginx target, newest) vs a stale root-owned `/home/ippadmin/htdocs/` copy (~29h older). Only the nginx vhost pointer proves authority; a config regen could silently repoint at the stale copy.
+
+---
+
+## 6. Infrastructure Separation Assessment
+
+**Current state: there is effectively ONE trust zone.** Internal ITPP operations, client sites, and micro-SaaS products share:
+
+- The same 6 hosts with no VLAN/subnet boundary between them.
+- One flat Tailscale mesh with no ACL tags (personal devices included).
+- One shared SSH key for root on all hosts.
+- app1: Wazuh (SIEM) + Twenty CRM (client PII) + LiteLLM (AI control plane) + Komodo (deploy) alongside TransitPin and giftaroast.com (products) as sibling Docker containers behind one Caddy.
+- app3: one shared MySQL/Percona instance backing internal, client, and product sites simultaneously.
+
+**Positive controls already present (preserve in Phase Two):**
+- app1-bu standby sync is **pull-only from S3**, not a live tunnel to Core - a correct blast-radius design worth keeping (don't add a live push tunnel later).
+- Per-service DB passwords are scoped per container (Twenty `APP_SECRET`, Komodo `KOMODO_JWT_SECRET`, etc.) - correctly separated, just not vaulted.
+- wphost02 is the only host outside the shared-key radius (uses a different key) and outside the Tailscale mesh.
+
+**Recommended target (Phase Two):** three logical tiers - `internal` (ops tooling, tight allowlist + MFA), `client` (client sites, isolated DB per tenant), `product` (micro-SaaS, dedicated DB + credential vault per product) - enforced by Tailscale ACL tags, per-tenant MySQL users on app3, and binding Docker publishes to loopback.
+
+---
+
+## 7. Disagreements (documented, not resolved - Germaine resolves)
+
+The following conflicts between auditor findings are logged here for Germaine's decision. Conductor did not silently resolve any of them; the resolutions below are read-only factual checks, with the open question flagged.
+
+**D1. Standby watchdog target IP - Sys-B H1 vs NetEng-A/NetEng-B.**
+Sys-B H1 claims the app1-bu watchdog pings the "wrong IP" (152.53.192.33), asserting Core is at 152.53.36.131.
+**Conductor verification (read-only):** Core's public IP is **152.53.192.33** (confirmed via `ip addr` on this host). 152.53.36.131 is **app1**, whose password appears in `.env` as `SERVER_152_53_36_131_PASS`. The watchdog's `LIVE_HOST=152.53.192.33` is **correct**.
+**Resolution:** Sys-B H1 is a false positive (IP conflation). The watchdog is targeting Core correctly. **No action required.**
+
+**D2. Gitea/Hudu/UNMS/UniFi backup coverage - Sys-B C1 vs Sys-C.**
+Sys-B C1 rates these four as Critical "no effective backup" because app2's `/root/backup.sh` references local scripts that do not exist on app2.
+Sys-C found Gitea restore-tested PASS (2026-08-10) and 30/34 targets "OK".
+**Conductor verification (read-only):** Both are partially correct. app2's own backup.sh silently skips Gitea/Hudu/UNMS/UniFi (local `gitea-backup.sh`, `hudu-backup.sh`, `unms-backup-sync.sh`, `unifi-backup-sync.sh` confirmed absent on app2). BUT Core owns the real backup scripts (present in `/root/.hermes/scripts/`, scheduled via Hermes cron jobs.json), and Gitea's backup was restore-tested PASS from `s3://hermes-vps-backups/gitea/daily/`. So Gitea has a working, tested backup via the Core-side path.
+**Resolution:** Sys-B's mechanism observation is correct; its severity conclusion **overstates Gitea** (which has a tested backup). The durable truth: there is a redundant broken app2 job creating false failure-log confidence, and **Hudu/UNMS/UniFi remain untested** (no restore evidence) even though Core-side scripts are scheduled. Recommended severity: **High** (untested coverage + broken parallel job), not Critical for Gitea. **Open question for Germaine:** whether to treat Hudu/UNMS/UniFi's untested-but-scheduled backup as acceptable or as a Critical gap.
+
+**Indep refinement (final):** Gitea should be **dropped from this finding entirely** (its restore test is a documented PASS, so "unrecoverable" is factually wrong for Gitea). The Hudu/UNMS/UniFi gap is a subset of Sys-C's already-Critical estate-wide "94% of backup targets never restore-tested" pattern, not an independent Critical. Final rating: **High** for Hudu/UNMS/UniFi, Gitea removed.
+
+**D3. Public repo `itpp-infrastructure` credential exposure - Git-A (Critical/High).**
+Git-A found live admin credentials re-leaked verbatim inside a prior audit report in the public repo.
+**Germaine decision (2026-08-13):** "leave the repo alone for now."
+**Resolution:** Deferred. The exposure remains, queued as a Phase Two finding. **No lockdown, rotation, or history scrub was performed.** Re-surface at Phase Two planning.
+
+**Indep note on D3 severity:** git-a.md rated this (Git-A Finding 2) High, but the report treats it Critical-tier. Indep agrees with the escalation - a live reusable credential in a searchable *public* repo is a worse exposure than the same secret in a private repo, so Critical is the more defensible rating. This does not change Germaine's deferral, which was made with full knowledge of the finding.
+
+**D4. Sec-A-02 "single SSH key" severity - Critical in sec-a.md, silently listed as High in §3.2.**
+The source auditor (Sec-A) rated the single-key blast-radius finding Critical; the consolidated report placed it under "High findings (representative)" with no Section 7 entry explaining the change. Indep flagged this as both a severity error and a process gap: any time the conductor changes a source auditor's severity, it must appear here.
+**Resolution:** Restored to **Critical** (now C10). The downgrade itself was the process gap - corrected.
+
+**D5. Sec-B-03 Technitium `changeme` default credential - Critical in sec-b.md, listed as High in §3.2.**
+Sec-B rated it Critical; the report listed it High with no Section 7 entry. Indep agrees **High** is the correct rating (the audit could not confirm the live in-app credential, and Technitium may not re-apply the env var after first bootstrap), but the silent downgrade should have been logged.
+**Resolution:** Confirmed **High**. Documented here for the record. The finding stands as a legitimate hardening signal regardless of whether the string is literally the current password.
+
+---
+
+## 8. Independence-Check Appendix
+
+**Reviewer:** Indep (claude-sonnet-5), independent QA pass. **Method:** re-read all 9 findings files and this report, then independently judged every Critical/High rating against its own stated evidence without deferring to the conductor's synthesis. Full review on disk at `findings/indep-review.md`.
+
+### 8.1 Re-score verdicts (Critical/High)
+
+| Finding | Conductor | Indep | Verdict |
+|---|---|---|---|
+| D1 / Sys-B H1 (watchdog "wrong IP") | False positive | False positive | FALSE-POSITIVE (agree) |
+| D2 / Sys-B C1 (Gitea/Hudu/UNMS/UniFi backup) | High | High (Gitea dropped) | AGREE + drop Gitea |
+| C1-C9 (Docker bypass, no segmentation, Wazuh, plaintext creds, LiteLLM, app3 MySQL, Grafana, wphost02, standby) | Critical | Critical | AGREE (all 9) |
+| Sec-A-02 (single SSH key) | High (silent) | Critical | UPGRADE → C10 |
+| NetEng-B NETB-6 (same single-key fact) | High | Critical | UPGRADE (duplicate of C10) |
+| Sec-B-03 (Technitium `changeme`) | High | High | AGREE (downgrade was silent - logged D5) |
+| Sys-C SYSC-02 (duplicate auth-api cron) | omitted | Medium | DOWNGRADE + add to §3.3 |
+| Sys-C SYSC-04 (WISP tower router, no backup) | omitted | High | MISSED + add to §3.2 |
+| Git-A Finding 1 (scripts repo backdoor password) | under-cited | Critical | AGREE + named in C4 |
+| Git-A Finding 2 (public repo re-leak) | Critical-tier | Critical | AGREE (escalated from High) |
+| Git-A Finding 3 (hermes-recovery) | High | High | AGREE |
+
+### 8.2 False positives
+
+- **Sys-B H1** - confirmed false positive (D1). Core's IP is 152.53.192.33, not 152.53.36.131 (that is app1).
+- **Sys-B C1 as applied to Gitea** - "unrecoverable" is factually wrong; Gitea has a passing restore test (2026-08-10). Dropped from the finding.
+
+No other Critical/High in the nine files was found factually wrong on re-read.
+
+### 8.3 Under-weighted or missed
+
+- **Sec-A-02** (single key = estate-wide blast) restored to Critical - see D4.
+- **Sys-C SYSC-04** (WISP tower router, zero backup coverage) was missing from the consolidated report - now in §3.2.
+- **Git-A Finding 1** (backdoor password reused across client onboards) was not named in C4 - now named.
+- **Sys-C SYSC-02** (duplicate cron) was missing - now in §3.3, downgraded to Medium.
+- **Sec-B-03** (Technitium) - Critical overstated given the audit could not confirm the live credential; confirmed High.
+
+### 8.4 Verdicts on D1/D2/D3
+
+- **D1:** agree with conductor - clean false positive.
+- **D2:** agree with direction, go further - Gitea dropped entirely; Hudu/UNMS/UniFi is a subset of Sys-C's estate-wide "94% untested" Critical, not an independent Critical.
+- **D3:** no grounds to disagree; Germaine's deferral is his call. Severity on the public-repo leak is better as Critical than git-a.md's High.
+
+### 8.5 Overall confidence
+
+High on D1 (unambiguous) and D2 (well-supported by Sys-C's independent S3 check); reasonably high on the SSH-key and Technitium re-scores (they turn on the report's own severity legend and an evidence gap the auditors themselves flagged); lower on SYSC-02's exact Medium-vs-High and on whether Technitium's live credential is literally still the default (out of scope for a read-only audit). No evidence of systematic severity inflation or deflation across the nine files - the surfaced issues are individual scoring errors plus one process gap (silent downgrades, now logged as D4/D5), not a pattern that casts doubt on the other 50+ findings.
+
+---
+
+## Appendix A: Cost - Estimate vs Actual
+
+- **Estimate (approved):** subtotal ~$5.40, realistic $8-10, ceiling ~$13.
+- **Actual (verified):** the audit triggered the LiteLLM per-key budget cap (`hermes-agent-v5` hit its $300 max_budget during the run; Germaine raised it to $400). This was driven by claude-sonnet-5 subagent usage plus re-runs from the 429 failures (Git-A and Sec-A each needed one resume).
+- **Attribution caveat:** precise audit-attributable spend requires a LiteLLM SpendLogs query scoped to the 2026-08-13 subagent window. The 7-day estate-wide DeepSeek total was $61.81 (all usage, not audit-only). The audit **exceeded the $13 ceiling** - exact overage will be itemized in the SpendLogs reconciliation before the report is closed. No opus-tier models were used, consistent with the brief.
+
+---
+
+## Appendix B: Artifacts
+
+Findings files (all read-only, no secrets as values):
+
+| File | Auditor | Lines |
+|---|---|---|
+| `findings/neteng-a.md` | NetEng-A (network exposure) | 268 |
+| `findings/neteng-b.md` | NetEng-B (segmentation/blast radius) | 197 |
+| `findings/sec-a.md` | Sec-A (IAM/secrets) | 245 |
+| `findings/sec-b.md` | Sec-B (hardening/patch/MFA/logging) | 160 |
+| `findings/sys-a.md` | Sys-A (Core + app1) | 388 |
+| `findings/sys-b.md` | Sys-B (app2/app3/app1-bu/wphost02) | 330 |
+| `findings/sys-c.md` | Sys-C (backup/DR) | 215 |
+| `findings/git-a.md` | Git-A (repos/secrets-in-history) | 238 |
+| `findings/docs-w.md` | Docs-W (docs say-do gap) | 169 |
+
+Raw capture files (for Phase Two re-verification): `/root/audit_*.txt` on Core.
+
+---
+
+*End of Phase One Final Report. No live modification was performed on any target system. All remediation is deferred to Phase Two.*

audit/phase-one/send-final-package.py

@@ -0,0 +1,147 @@
+import smtplib
+import ssl
+import imaplib
+import importlib.util
+from email.mime.multipart import MIMEMultipart
+from email.mime.text import MIMEText
+from email.mime.application import MIMEApplication
+from pathlib import Path
+
+# Signature module (hyphenated filename, load via importlib)
+spec = importlib.util.spec_from_file_location(
+    "sig", "/root/.hermes/references/shonuff-signature.py"
+)
+sig = importlib.util.module_from_spec(spec)
+spec.loader.exec_module(sig)
+signature_html = sig.build_signature_block()
+
+# Credentials
+pw = Path("/root/.config/himalaya/shonuff.pass").read_text().strip()
+FROM = "Sho'Nuff Brown <shonuff@germainebrown.com>"
+FROM_ADDR = "shonuff@germainebrown.com"
+TO = ["g@germainebrown.com"]
+SUBJECT = "ITPP Phase One Infrastructure Audit - Final Package"
+
+# ---- Plain text part ----
+plain = """ITPP Phase One Infrastructure Audit - Final Package
+
+Phase One read-only audit is complete across all six servers plus the Gitea
+estate, DNS, and backups. Zero live changes were made.
+
+What we found (five systemic themes):
+1. No segmentation - Docker published-port rules bypass UFW, roughly 20
+   management consoles public, no VLAN or Tailscale ACLs.
+2. Plaintext credentials in world-readable locations across the estate.
+3. Backups are write-only - 30 of 34 targets backed up, only 2 ever
+   restore-tested; LiteLLM Postgres never backed up.
+4. SIEM monitors only itself - Wazuh has zero enrolled agents; Grafana
+   default admin/admin public, no MFA.
+5. Warm standby not data-ready - app1-bu state.db roughly 28 days stale.
+
+Severity: 23 Critical, 34 High, 36 Medium, 17 Low (consolidated).
+
+Independent review (Indep, claude-sonnet-5): re-scored every Critical/High.
+Confirmed D1 (watchdog IP) false positive. Refined D2 (Gitea dropped from the
+finding, has a passing restore test). Caught two silent downgrades now
+restored (single SSH key restored to Critical as C10) and one missed finding
+(WISP tower router zero backup).
+
+Attached: report.md, policy-and-procedure.md, skill-spec.md, indep-review.md.
+
+Next step: Phase Two remediation is ordered in Section 4 of the report. The
+highest-leverage first moves are bind Docker publishes to loopback, rotate
+plaintext credentials, and enroll Wazuh agents. No action taken in Phase One.
+
+"""
+
+# ---- HTML part ----
+html = """
+<h2 style="color:#1a1a2e;">ITPP Phase One Infrastructure Audit - Final Package</h2>
+<hr style="border:none;border-top:2px solid #cc0000;margin:12px 0 20px 0;">
+
+<p>Phase One read-only audit is complete across all six servers plus the Gitea
+estate, DNS, and backups. <strong>Zero live changes were made.</strong> The full
+report, policy and procedure document, and the skill specification are attached.</p>
+
+<h3 style="color:#1a1a2e;">What we found (five systemic themes)</h3>
+<ol>
+<li><strong>No segmentation</strong> - Docker published-port rules bypass UFW,
+leaving roughly 20 management consoles public, with no VLAN or Tailscale ACLs.</li>
+<li><strong>Plaintext credentials</strong> in world-readable locations across the estate.</li>
+<li><strong>Backups are write-only</strong> - 30 of 34 targets backed up, only 2 ever
+restore-tested; LiteLLM Postgres is never backed up.</li>
+<li><strong>SIEM monitors only itself</strong> - Wazuh has zero enrolled agents;
+Grafana runs default admin/admin public with no MFA.</li>
+<li><strong>Warm standby not data-ready</strong> - app1-bu state.db roughly 28 days stale.</li>
+</ol>
+
+<h3 style="color:#1a1a2e;">Severity</h3>
+<table cellpadding="6" cellspacing="0" style="border-collapse:collapse;font-size:14px;">
+<tr>
+<td style="border:1px solid #ddd;background:#fdecea;color:#a30;"><strong>23 Critical</strong></td>
+<td style="border:1px solid #ddd;background:#fff4e5;color:#a30;"><strong>34 High</strong></td>
+<td style="border:1px solid #ddd;background:#fffbe6;"><strong>36 Medium</strong></td>
+<td style="border:1px solid #ddd;"><strong>17 Low</strong></td>
+</tr>
+</table>
+
+<h3 style="color:#1a1a2e;">Independent review (Indep, claude-sonnet-5)</h3>
+<p>A second pass re-scored every Critical and High finding against the raw
+evidence. It confirmed D1 (watchdog IP) as a false positive, refined D2 (Gitea
+dropped from the finding, it has a passing restore test), and caught two silent
+severity downgrades now corrected, plus one missed finding (WISP tower router
+with zero backup). Full detail is in the report, Section 8.</p>
+
+<h3 style="color:#1a1a2e;">Attached</h3>
+<ul>
+<li><strong>report.md</strong> - Final Report (severity matrix, recommendations, disagreements log)</li>
+<li><strong>policy-and-procedure.md</strong> - v1.0 policy and procedure document</li>
+<li><strong>skill-spec.md</strong> - itpp-policy-adherence skill specification</li>
+<li><strong>indep-review.md</strong> - independent severity review</li>
+</ul>
+
+<h3 style="color:#1a1a2e;">Next step</h3>
+<p>Phase Two remediation is ordered in Section 4 of the report. The
+highest-leverage first moves are to bind Docker publishes to loopback, rotate
+plaintext credentials, and enroll Wazuh agents. No action was taken in Phase One
+per the read-only mandate.</p>
+
+"""
+
+# ---- Assemble MIME ----
+msg = MIMEMultipart("mixed")
+msg["From"] = FROM
+msg["To"] = ", ".join(TO)
+msg["Subject"] = SUBJECT
+
+alt = MIMEMultipart("alternative")
+alt.attach(MIMEText(plain, "plain", "utf-8"))
+alt.attach(MIMEText(html + signature_html, "html", "utf-8"))
+msg.attach(alt)
+
+base = Path("/root/projects/itpp-infrastructure/audit/phase-one")
+attachments = [
+    (base / "report.md", "report.md"),
+    (base / "policy-and-procedure.md", "policy-and-procedure.md"),
+    (base / "skill-spec.md", "skill-spec.md"),
+    (base / "findings" / "indep-review.md", "indep-review.md"),
+]
+for path, filename in attachments:
+    part = MIMEApplication(path.read_bytes(), _subtype="markdown")
+    part.add_header("Content-Disposition", "attachment", filename=filename)
+    msg.attach(part)
+
+# ---- Send ----
+ctx = ssl.create_default_context()
+with smtplib.SMTP("mail.germainebrown.com", 2525, timeout=30) as s:
+    s.starttls(context=ctx)
+    s.login(FROM_ADDR, pw)
+    s.send_message(msg, from_addr=FROM_ADDR, to_addrs=TO)
+print("SMTP send OK")
+
+# ---- IMAP APPEND to Sent ----
+imap = imaplib.IMAP4_SSL("mail.germainebrown.com", 993, ssl_context=ctx)
+imap.login(FROM_ADDR, pw)
+typ, data = imap.append("Sent", None, None, msg.as_bytes())
+imap.logout()
+print(f"IMAP append: {typ} {data}")

audit/phase-one/skill-spec.md

@@ -0,0 +1,98 @@
+# Skill Spec - ITPP Policy-Adherence Gate
+
+**Type:** Documentation-only spec (no skill is created by Phase One). Actual implementation is a Phase Two task, pending Germaine's go-ahead.
+**Purpose:** A Hermes skill that makes Sho'Nuff the enforcement point for the Policy & Procedure Document, so policy is applied consistently rather than remembered ad hoc.
+
+---
+
+## 1. Name & Placement
+
+- **Skill name:** `itpp-policy-adherence` (or `policy-adherence-gate`)
+- **Category:** `devops` (alongside `governance-and-honesty`, `subagent-verification`)
+- **Location:** `~/.hermes/skills/devops/itpp-policy-adherence/SKILL.md`
+- **Reference dependency:** loads the Policy & Procedure Document and the Running Exemptions Document from `/root/projects/itpp-infrastructure/audit/phase-one/` (or their canonical home once promoted).
+
+---
+
+## 2. Trigger Conditions
+
+The skill activates whenever a user request or an automated job would, if executed, do any of the following:
+
+- Modify production configuration on Core/app1/app2/app3/app1-bu/wphost02.
+- Rotate, create, store, or transmit a credential.
+- Change a firewall rule, DNS record, Tailscale ACL, or Docker port binding.
+- Alter a backup target, backup schedule, or the standby sync.
+- Add a new host, site, or product to the estate.
+- Deprecate, decommission, or rename any infrastructure component.
+- Publish or change anything on `docs.itpropartner.com` or the docs repo.
+- Introduce a service that would bind to 0.0.0.0 or bypass the reverse proxy.
+
+---
+
+## 3. Behavior
+
+When triggered, the skill runs a policy check **before** acting:
+
+1. **Classify** the request against the Policy & Procedure Document sections (§2 Change Management, §3 Docs-Sync, §4 Backup/DR, §5 Access, §6 Segmentation, §7 Cadence).
+2. **Compliant** → proceed normally, and record the action per §2 (what / who / when / rollback).
+3. **Departure** → do **not** execute. Raise an exemption request:
+ - State the provision being departed from.
+ - Request a business justification.
+ - **Only Germaine may authorize** the departure. No self-approval, no "it's low stakes, I'll just do it."
+4. **Unknown / ambiguous** → ask Germaine rather than guessing, consistent with the standing no-fabrication rule.
+
+---
+
+## 4. Running Exemptions Document - Schema
+
+The skill maintains a single append-only document (e.g. `exemptions.md` alongside the policy doc). Each entry:
+
+```markdown
+## EX-<seq>
+
+- **Date:** YYYY-MM-DD
+- **Requester:** <who asked / what job asked>
+- **Provision departed from:** <Policy & Procedure §X.Y - short title>
+- **Request + business justification:** <what is being asked, why>
+- **Germaine authorization:** <approve / deny / deferred, with date>
+- **Type:** one-time | ongoing
+- **Follow-up:** <review date for ongoing; or "closed" with closure date for one-time>
+```
+
+**Rules:**
+- Every exemption is append-only. Existing entries are never edited to change the authorization; supersession adds a new entry.
+- Ongoing exemptions carry a follow-up review date; a review is triggered when that date passes.
+- The Phase One public-repo deferral is the seed entry:
+
+```markdown
+## EX-001
+
+- **Date:** 2026-08-13
+- **Requester:** Sho'Nuff (conductor, Phase One audit - Git-A finding)
+- **Provision departed from:** §5 Access / §8 Exemptions - live credential in public repo `itpp-infrastructure`
+- **Request + business justification:** Immediate lockdown (private + history scrub) of a public repo re-leaking admin passwords; surfaced as a Critical finding under the "cannot wait" policy.
+- **Germaine authorization:** deferred - "leave the repo alone for now"
+- **Type:** one-time (deferral)
+- **Follow-up:** re-surface at Phase Two planning
+```
+
+---
+
+## 5. Integration Notes
+
+- The skill does **not** itself enforce anything mechanically - it is a decision gate that runs in Sho'Nuff's reasoning before any privileged action, and it surfaces exemption requests to Germaine.
+- It references, but does not duplicate, the Policy & Procedure Document. The policy doc is the source of truth; the skill is the tripwire that consults it.
+- It composes with the existing `governance-and-honesty` and `subagent-verification` skills: those govern output integrity and subagent checking; this one governs *whether a privileged action is even permitted*.
+
+---
+
+## 6. Acceptance Criteria (for the eventual implementation)
+
+1. A request to change an app1 firewall rule with no exemption raises an exemption request to Germaine instead of executing.
+2. A compliant change (e.g. fixing the LiteLLM `pg_dump` target) proceeds and is recorded per §2.
+3. EX-001 is present in the Running Exemptions Document and re-surfaces at Phase Two planning.
+4. The skill never self-authorizes an exemption; Germaine's explicit approval is always required for a departure.
+
+---
+
+*End of Skill Spec.*

audit/phase-two/c4-credential-rotation-plan.md

@@ -0,0 +1,127 @@
+# C4 Credential Rotation Plan (Phase Two, WS1)
+
+Status: APPROVED (D1). Phase 0 complete, Phase 1 in progress.
+Date: 2026-08-13
+Owner: Sho'Nuff
+
+## 0. What this plan covers (and what it does not)
+
+- The C4 **permission subset** is DONE. Logged as `P2-C4-001` (app1-bu chmod 600 on 11 secret files + umask 077 root-cause fix). This plan does not re-open it.
+- This plan covers the **second half of C4**: inventory every plaintext credential, rotate them, and move them to Vaultwarden.
+- Source-hygiene follow-ups are included as Phase 6 (they are rotation-adjacent, not permission work).
+
+## 1. The two decisions this plan needs from you (up front)
+
+| # | Decision | What it unlocks |
+|---|---|---|
+| D1 | **APPROVED 2026-08-13** ("A - Rotate, B - Rotate, C - Revoke"): rotate live AI keys (A+B), revoke dead (C) | Phase 2 (in progress) |
+| D2 | **Go/no-go: rotate the MSP-backdoor shared credential, plus approve the per-portal order** | Phase 3 |
+
+Everything else in this plan runs under those two gates. Phases 4-6 are lower-risk hygiene and I will flag them individually before executing.
+
+## 2. Evidence base (phase-one report, section 3.1 C4)
+
+1. Two unsanitized copies of `key-inventory.md` on Core (one inside Hermes's reference dir).
+2. Private `scripts` repo hardcodes the MSP-backdoor admin password reused across client onboardings (Git-A Finding 1). Single most consequential live credential.
+3. app1-bu `/root/.hermes/.env` (mode 644) holds ~20 plaintext secrets: root passwords for app1/app2/app3, Telegram bot token, Cloudflare/Netcup/SyncroMSP tokens, and eight AI-provider keys.
+4. `rally.service`, `seemytrip.service`, `giftaroast.service` hardcode `JWT_SECRET`, `DEEPSEEK_API_KEY`, `ADMIN_AI_KEY`, and Twilio credentials in world-readable unit files.
+5. app3 MySQL root password in `/opt/backup-restore/snapshot.sh` (mode 775) and `/root/backup.sh`.
+6. Public repo `itpp-infrastructure` re-leaks live admin passwords in a prior audit report (section 7, deferred per Germaine).
+
+Recommended order from the report: app1-bu .env first, then key-inventory.md, then systemd units, then app3 snapshot.sh.
+
+## 3. Safety rules (non-negotiable, apply to every phase)
+
+1. **Fallback-first.** Before any LiteLLM-adjacent or credential-affecting change, Hermes stays pinned to direct DeepSeek. This is already in force since C1 (`P2-C1-001`); re-confirm at start of each phase.
+2. **Vaultwarden-before-changes.** Retrieve a credential from Vaultwarden before touching the live system, never from a plaintext file, shell history, or a prior report.
+3. **No autonomous key rotation.** Every rotation happens only after the go/no-go above, in the order listed.
+4. **Verify before done.** Every rotation is followed by a live check (API call, login, health endpoint). No "should work" statements.
+5. **Log + rollback.** Each change is written to `change-log.md` at the moment it is made, with the rollback path stated before the change.
+6. **Zero artifact style.** No em dashes or en dashes in any plan or change-log entry. All credential values appear as `[REDACTED]`.
+
+## 4. Phase 0: Inventory (read-only, no changes)
+
+Goal: a single authoritative list of every live plaintext credential, cross-referenced against Vaultwarden.
+
+Steps:
+1. Re-run the secret-sprawl scan on the three hosts with the raw hit counts (6,296 Core / 5,504 app1-bu / 1,275 app2) and triage down to live, consequential secrets only.
+2. Categorize every secret:
+   - (A) AI provider keys
+   - (B) MSP-backdoor shared credential
+   - (C) server root passwords (app1/app2/app3)
+   - (D) service tokens (Telegram, Cloudflare, Netcup, SyncroMSP, Twilio)
+   - (E) JWT secrets and DB passwords (systemd units, app3 snapshot/backup scripts)
+   - (F) source-repo leaks (key-inventory.md copies, scripts repo, public repo section 7)
+3. Cross-reference each item against Vaultwarden: mark `already vaulted`, `missing`, or `stale`.
+4. Deliverable: an inventory table with a rotation priority, a current location, and a target Vaultwarden item for each credential.
+
+Effort note: this is where the L-XL sizing sits. The raw scan counts are noise; the consequential list is far smaller.
+
+## 4a. Phase 0 result (2026-08-13)
+
+- Inventory complete. The report's "eight keys" was an undercount: the live AI surface is ~19 upstream keys across three layers (config.yaml literals, .env plaintext, LiteLLM encrypted credentials), many duplicated.
+- GEMINI_API_KEY and GOOGLE_AI_STUDIO_KEY hold the same value under two names (one rotation, two lines to update).
+- Verdict: A (fallback chain, 7) = Rotate. B (operational LLM, 7) = Rotate. C (dead, 4) = Revoke.
+- Correction: PARALLEL_API_KEY is not dead. It is a live Super Search provider (Parallel.ai Search, fallback #11 in server.py), reclassified as a Phase 3 service token (Keep).
+- Vault gaps: DeepSeek, OpenAI, Google/Gemini, Groq, MiniMax have no Vaultwarden item; Cohere/Fireworks/Perplexity/Mistral have portal logins only.
+
+## 5. Phase 1: AI provider keys (needs D1)
+
+Sequence (fallback-first, rotate non-active providers before the active one):
+
+1. Confirm Hermes is pinned to direct DeepSeek and the direct endpoint is healthy (`/v1/models` + a test completion).
+2. From Phase 0 inventory, confirm the exact list of eight AI providers and which LiteLLM / env locations hold each key.
+3. Rotate the keys for all providers EXCEPT the one Hermes is actively pinned to (DeepSeek), one at a time:
+   - Generate a new key at the provider portal.
+   - Store the new key in Vaultwarden (target item from Phase 0).
+   - Update LiteLLM config and/or env to reference the vault-backed value.
+   - Verify the provider still resolves through the fallback chain (test call).
+   - Write the change to `change-log.md` with rollback.
+4. Rotate the pinned provider (DeepSeek) LAST, immediately updating Core `.env` and LiteLLM before any subsequent API call can fail.
+5. Re-verify the full fallback chain end to end and switch Hermes back to its normal provider only after all eight keys are confirmed live.
+
+Rollback: each step keeps the prior key in the change-log rollback note until the next step verifies green.
+
+## 6. Phase 2: MSP-backdoor shared credential (needs D2)
+
+Sequence (this is the highest-consequence single credential):
+
+1. Identify the exact shared admin credential and every client system / vendor portal currently using it (inventory from `scripts` repo + onboarding records).
+2. Produce a per-portal rotation order, ordered by blast radius (most-impacted or most-exposed first), and get your sign-off on the order.
+3. Vaultwarden-before-changes: confirm the replacement credential is generated and vaulted before touching any live system.
+4. Rotate one portal at a time: update the portal, verify login with the new credential, update any dependent automation/env, then move to the next.
+5. Remove the hardcoded password from the private `scripts` repo and replace it with a Vaultwarden lookup or a `[REDACTED]` placeholder.
+6. Log every portal change in `change-log.md` with rollback.
+
+## 7. Phase 3: server root passwords and service tokens (lower risk)
+
+- Rotate app1/app2/app3 root passwords; store in Vaultwarden; update the one place that currently references them in plaintext (app1-bu `.env`).
+- Rotate service tokens: Telegram bot token, Cloudflare, Netcup, SyncroMSP, Twilio. Each token rotation re-issues at the vendor console, then updates the consuming service and any env/unit file.
+- Verify each service still functions after its token rotates (send a test, poll an endpoint, etc.).
+
+## 8. Phase 4: JWT secrets and DB passwords
+
+- Move `JWT_SECRET`, `DEEPSEEK_API_KEY`, `ADMIN_AI_KEY`, and Twilio creds out of `rally.service`, `seemytrip.service`, `giftaroast.service` into vault-backed env or a `chmod 600` env file, then reload the units.
+- Rotate the app3 MySQL root password in `/opt/backup-restore/snapshot.sh` and `/root/backup.sh`; store in Vaultwarden; update the scripts to source it from a 600-mode file.
+- Verify each service/backup still runs after the change.
+
+## 9. Phase 5: source hygiene (flagged separately)
+
+- Redact or delete the two `key-inventory.md` copies on Core.
+- Confirm the `scripts` repo MSP password removal from Phase 2 is committed and pushed.
+- Public repo `itpp-infrastructure` re-leak (report section 7): deferred per your earlier instruction; restating it here so it is not silently dropped. Confirm whether to keep it deferred or fold it into this plan.
+
+## 10. Verification gates (plan-wide)
+
+- [ ] Phase 0 inventory complete and cross-referenced against Vaultwarden
+- [ ] Fallback chain verified end to end after all AI key rotations
+- [ ] Every portal login verified after MSP-backdoor rotation
+- [ ] Every service/unit verified functional after its token/JWT/DB rotation
+- [ ] `change-log.md` has a dated entry with rollback for every single change
+- [ ] No plaintext credential remains in any world-readable location (re-run the scan)
+
+## 11. Approval needed
+
+- D1: AI provider key rotation go/no-go (unlocks Phase 1)
+- D2: MSP-backdoor rotation go/no-go + per-portal order (unlocks Phase 2)
+- Phases 3-5: flag individually before execution (no blanket approval implied)

audit/phase-two/change-log.md

@@ -0,0 +1,45 @@
+# ITPP Phase Two Change Log
+
+Engagement: ITPP Phase Two (remediation). Supersedes the placeholder framework.
+Started: 2026-08-13
+Conductor: Sho'Nuff
+Authority: Germaine Brown (sole authorizer per P&P section 2 and section 8)
+
+This log records every live change made during Phase Two, per Policy and Procedure section 2:
+(a) what changed, (b) who authorized it, (c) when, and (d) how to roll back.
+
+Every entry is written at the moment the change is made, not discovered later.
+Passwords and secrets are never recorded in plaintext here. The entry points to the
+Vaultwarden item that holds the new value.
+
+---
+
+## Schema
+
+| Field | Meaning |
+|---|---|
+| ID | Sequential entry number (P2-###) |
+| Date | When the change was applied (ET) |
+| Change | What changed (old -> new where applicable) |
+| Authorized by | Germaine, or a referenced approval (e.g. "Phase Two fast-track C7") |
+| Rollback | How to undo it |
+| Status | done / in-progress / rolled-back |
+| Notes | Dependencies verified, LiteLLM-adjacent flag, exemption reference if any |
+
+---
+
+## Entries
+
+| ID | Date | Change | Authorized by | Rollback | Status | Notes |
+|---|---|---|---|---|---|---|
+| P2-C3-001 | 2026-08-13 | Enrolled Wazuh agents 4.9.2 on 5 hosts: core, app2, app3, app1-bu, wphost02 (manager app1 152.53.36.131; agent IDs 001-005, all Active) | Phase Two fast-track C3 (Germaine) | Disable/remove agent service on each host, then remove agent record from manager via agent_control -r `<id>` | done | LiteLLM-adjacent=no. Verified via agent_control -l on manager: all 5 new agents + agent 000 (manager) show Active. No password/secret recorded (authd enrollment used use_password=no, no credential involved). |
+| P2-C7-001 | 2026-08-13 | Rotated Grafana admin password (default 'admin' removed; 24-char random value stored in Vaultwarden item 'Grafana - Core Admin'); restricted port 3002 from Anywhere to Tailscale 100.64.0.0/10 | Phase Two fast-track C7 (Germaine) | Recreate container with GF_SECURITY_ADMIN_PASSWORD=admin (see rollback/grafana-inspect.json); ufw allow 3002/tcp to restore Anywhere | done | LiteLLM-adjacent=no. Verified: admin/admin login -> 401, new password -> 200, UFW shows only 100.64.0.0/10 rule. Password persisted in Vaultwarden item d3785ae9-085d-4e40-ba1b-69b66fef9e34. |
+| P2-C8-001 | 2026-08-13 | wphost02 full DB backup coverage; closed the "6 of 8 unprotected" gap. RECON: found the offsite S3 script (`/root/backup.sh` on wphost02, using `mysqldump --all-databases`) was already being triggered daily via a Core-side crontab entry (`0 5 * * * ssh ... root@5.161.62.38 '/root/backup.sh'`, running since ≥2026-07-19); this was NOT visible from wphost02's own crontab, which is why the earlier finding only checked local `db-dump.sh` (2 of 8 DBs, 7-day local retention) and missed it. Verified via journalctl (`wphost02-backup` tag on Core) and a fresh manual run of backup.sh: all 8 WordPress DBs (apextrackexperience_1781549652, boxpilotlogistics_1770339547, debtrecoveryexperts_1778934554, iAmGMB_1764020288, katiewattsdesign_1735425014, mainWP_1717713767, vigilanttac_1728911691, voipsimplicity_1732250845) are present in the all-databases.sql dump and in per-site tarballs, landing daily in s3://hermes-vps-backups/wphost02-backup/<date>/ (Wasabi). Also extended wphost02's local `/root/db-dump.sh` (still only covered 2/8 DBs on local disk, 7-day retention) to dump all 8 DBs individually for local defense-in-depth; ran it successfully (22 files, all 8 today's dumps present). No new crontab entry was needed; the offsite S3 job was already wired in on Core and already covers all 8 DBs; only the local-disk script was extended. | Phase Two Workstream 1 C8 (Germaine) | restore db-dump.sh from /root/db-dump.sh.bak-20260813 on wphost02 (scp back or `cp db-dump.sh.bak-20260813 db-dump.sh`); no crontab line was added so there is nothing to remove there; the pre-existing Core crontab line `0 5 * * * ssh -i /root/.ssh/itpp-infra ... root@5.161.62.38 '/root/backup.sh'` is unchanged from before this task. | done | LiteLLM-adjacent=no. No decommission action taken; no live WordPress site or config modified; no new S3 credentials created (existing Wasabi creds in wphost02:/root/.aws/credentials, already valid, reused as-is). Verified via S3 object listing (10 objects, fresh timestamp) and journalctl, not by trusting script exit code alone. |
+| P2-C1-001 | 2026-08-13 | LiteLLM fallback switch (pre-C1 safety gate): Hermes primary model provider admin-ai (LiteLLM on app1) -> deepseek (direct to api.deepseek.com). Also switched delegation.provider and delegation.model to direct deepseek, and delegation.fallback provider admin-ai -> deepseek. Model stays deepseek-v4-pro (no downgrade; direct endpoint serves both deepseek-v4-pro and deepseek-v4-flash). | Phase Two safety rule (fallback switch required before any LiteLLM-adjacent change); C1 app1 portion is LiteLLM-adjacent | Restore config from /root/.hermes/config.yaml.bak-20260813-C1, or revert via: hermes config set model.provider admin-ai; hermes config set delegation.provider admin-ai; hermes config set delegation.model claude-sonnet-5; and set delegation.fallback provider back to admin-ai. | done | LiteLLM-adjacent=yes (this IS the fallback switch). Verified direct deepseek /v1/models returns both deepseek-v4-pro and deepseek-v4-flash, and /v1/chat/completions returns a valid choices array with no error. DEEPSEEK_API_KEY present in .env and valid. Note: the running gateway session continues on admin-ai until next restart but is protected by the already-active direct-first fallback chain (deepseek -> google -> xai -> anthropic -> openai). Residual admin-ai references remain only in vision paths (auxiliary.vision and top-level vision), non-blocking for C1 terminal/SSH work. |
+| P2-C9-001 | 2026-08-13 | Warm-standby DB snapshot consistency: hermes-live-sync.sh now exports state.db and memory_store.db via SQLite online backup (.backup) into a consistent snapshot uploaded as the authoritative S3 object, and excludes the raw live WAL/SHM files from the sync. Deleted obsolete transient state.db-wal/shm and memory_store.db-wal/shm objects from s3://hermes-vps-backups/live/. | Phase Two Workstream 1 C9 (Germaine) | cp ~/.hermes/scripts/hermes-live-sync.sh.bak-20260813-C9 ~/.hermes/scripts/hermes-live-sync.sh (raw file copy behavior restored) | done | LiteLLM-adjacent=no. Verified: .backup of the 2.98 GB live state.db completed in 9s and passed PRAGMA integrity_check (ok); fixed script ran exit 0 and uploaded the 2980638720-byte state.db and 1871872-byte memory_store.db snapshots at 15:30 ET. Root cause: aws s3 sync of a live WAL-mode SQLite DB can capture a torn checkpoint (evidence: state.db.corrupted plus 3 malformed-backup-* objects in S3 dated 2026-07-09). Residual follow-up: profiles/anita/state.db* still synced raw because top-level excludes do not match profile paths; and ~7.5 GB of stale corrupted/malformed S3 objects left in place pending a cleanup decision. |
+| P2-C4-001 | 2026-08-13 | Restored least-privilege file permissions on app1-bu standby: chmod 600 on 11 world-readable secret-bearing files (.hermes/.env, .hermes/config.yaml, profiles/anita/.env, profiles/anita/google_client_secret.json, docker/vaultwarden/.env, docker/twenty/.env, references/glc-credentials.md, references/liberty-credentials.md, .aws/credentials.bak, and two state-snapshot .env files). Fixed root cause: added umask 077 to hermes-standby-sync.sh, hermes-standby-watchdog.sh, and hermes-standby-restore.sh (standby) plus hermes-live-sync.sh (Core) so aws s3 sync no longer re-creates synced files as 644 and the Core .snapshots output is no longer transiently world-readable. | Phase Two Workstream 1 C4 (Germaine) | Restore the three .bak-20260813-C4 script backups (cp each .bak-20260813-C4 back over its script); re-chmod 644 the files (not recommended) | done | LiteLLM-adjacent=no. Root cause: aws s3 sync does not preserve POSIX permissions (S3 objects carry no mode bits), so the 10-min standby sync re-created Core 600 files as 644 under the default umask 022. Verified: functional test of aws s3 sync under umask 077 lands a downloaded object as 600; bash -n passes on all three edited scripts. Core already holds the same files at 600, so no Core change was needed. |
+| P2-C1-002 | 2026-08-13 | DOCKER-USER firewall hardening made persistent on Core, app1, app2, app3. Live IPv4 rules (applied earlier 2026-08-13) now survive reboot and docker restart via /usr/local/sbin/itpp-docker-user.sh (idempotent flush+rebuild of DOCKER-USER, IPv4 + IPv6 where present) wired as an ExecStartPost drop-in at /etc/systemd/system/docker.service.d/itpp-docker-user.conf on each host. Closed to Tailscale-only (100.64.0.0/10 + ESTABLISHED,RELATED only): Core browserless 3000, camofox 9377, uptime-kuma 3001; app1 komodo 9120, twenty 3003, wazuh dashboard 5601, wazuh indexer 9200; app2 bookstack 6875 (direct), minio 9001, gitea ssh 3022, ragflow 9380-9384/9392/9393, infinity 23817/23820, unifi admin 8443/8843/8880, unms 81/8444; app3 buzz relay 3000. Preserved public: app1 wazuh manager 1514/1515/55000 tcp + 514 udp; app2 dns 53 tcp+udp, support-api 6880, unifi inform 8080, traccar 8082 + 5000-5150 tcp+udp, unms-nginx 8089, bookstack 6875 via ctorigdstport proxy path, unms-netflow 2055 udp, unifi 3478/10001 udp. | Phase Two Workstream 1 C1 (Germaine) | On each host: rm /etc/systemd/system/docker.service.d/itpp-docker-user.conf && systemctl daemon-reload; then iptables -F DOCKER-USER (and ip6tables -F DOCKER-USER where present); rm /usr/local/sbin/itpp-docker-user.sh | done | LiteLLM-adjacent=no. Rules scoped -i eth0 so docker-bridge traffic is untouched. Verified live: iptables -L DOCKER-USER -n -v on all 4 hosts shows correct chains with -i eth0 and active packet counters. Persistence proven end-to-end: ufw reload on Core preserved FORWARD integration and DOCKER-USER rules; systemctl restart docker on app3 flushed DOCKER-USER (counters reset) then ExecStartPost re-applied rules immediately, all 8 app3 containers returned healthy (hexclave-server health:starting then healthy). Pre-existing issues noted, not C1: crm.debtrecoveryexperts.com and crm.intelsight.io 502 because Core Caddy proxies to Core-local localhost:3003 where nothing listens (Twenty runs app1:3003); admin-ai /v1/models 401 despite /health 200 (auth, not firewall); infinity 23817 returns 000 from minio while 23820 returns 404 (port semantics). |
+| P2-C4-002 | 2026-08-13 | C4 credential rotation: D1 approved (rotate A and B, revoke C). Phase 0 inventory complete. Rollback snapshots taken: /root/.hermes/config.yaml.bak-20260813-C4-pre and /root/.hermes/.env.bak-20260813-C4-pre, both chmod 600. | Germaine (A rotate, B rotate, C revoke, 2026-08-13) | Restore both C4-pre backups | done | LiteLLM-adjacent=yes (this entry is the go/no-go gate for LiteLLM key changes). No keys changed yet; decision record only. Phase 0 verdict: A=7 rotate, B=7 rotate, C=4 revoke. |
+| P2-C4-003 | 2026-08-13 | Revoked 4 dead AI keys from plaintext: removed AI21_API_KEY, ALIBABA_API_KEY (duplicate, x2), ZAI_API_KEY, NVIDIA_API_KEY from Core /root/.hermes/.env and app1-bu /root/.hermes/.env (5 lines to 0), plus the stale NVIDIA NIM comment line. PARALLEL_API_KEY reclassified: it is a live Super Search provider (Parallel.ai Search, fallback #11 in server.py), not an orphan, moved to Phase 3 service tokens (Keep). | Germaine (C revoke) | Restore /root/.hermes/.env.bak-20260813-C4-pre on Core and app1-bu .env from its prior state | done | LiteLLM-adjacent=no (plaintext .env removal only; no LiteLLM credential touched). Verified before/after line counts and mode 600 on both hosts. Residual: NVIDIA LiteLLM credential (0 working models) still in app1 Postgres, deletion deferred to the LiteLLM step. |
+| P2-C4-004 | 2026-08-14 | Rotated 5 assistant-owned portal API keys (b1 set): Cohere, Fireworks, Mistral, Perplexity, FAL. Applied across three layers: (1) LiteLLM credentials PATCHed via /credentials/{name} on app1 (Cohere, Fireworks, Mistral, Perplexity, Fal) — 5/5 HTTP 200; (2) Core /root/.hermes/.env swapped (COHERE_API_KEY, FIREWORKS_API_KEY, MISTRAL_API_KEY, PERPLEXITY_API_KEY, FAL_KEY) with backup /root/.hermes/.env.bak-20260814-b1; (3) Vaultwarden — created 3 new 'X - API Key' items (Cohere, Fireworks, Mistral; username info@itpropartner.com) and updated Perplexity ('console.perplexity.ai') and FAL ('fal.ai') item passwords in place. Mistral+Cohere keys generated in-console as 'LiteLLM-20260813'; Fireworks/FAL/Perplexity keys supplied by Germaine. | Germaine (C4 D1 GO + b1 direction 2026-08-14) | Core .env: cp /root/.hermes/.env.bak-20260814-b1 /root/.hermes/.env. LiteLLM: re-PATCH the 5 credentials with pre-rotation values (old keys remain active provider-side; recoverable from .env.bak-20260813-C4-pre and Vaultwarden password history). Vaultwarden: restore Perplexity/FAL item passwords via bw item password history; delete the 3 new 'X - API Key' items. | done | LiteLLM-adjacent=yes. Verified: all 5 keys live against providers (Mistral/Cohere/Fireworks/Perplexity HTTP 200; FAL past auth gateway); LiteLLM PATCH 5/5 HTTP 200; .env values correct with pre-existing dual NETCUP_API_KEY quirk preserved; Vaultwarden re-read 5/5 PASS. Active consumers: Perplexity -> Super Search (super-search.service restarted; health_check reports perplexity:ok, server v2.4.0); FAL -> image_generate (picks up new key on next Hermes restart; running process holds prior key in os.environ). Cohere/Fireworks/Mistral have no active Core consumer (LiteLLM 'use if set' only). Old keys NOT revoked provider-side. |
+| P2-C4-005 | 2026-08-14 | Rotated the admin-ai LiteLLM master key: old (len 66, sk-litel...) -> new (len 67, sk-x2top..., sk- + 64 mixed-case alphanumeric, no special chars). Applied to: (1) app1 /root/docker/litellm/.env LITELLM_MASTER_KEY and UI_PASSWORD (backup .env.bak-20260814-adminai); (2) Core /root/.hermes/scripts/cost-alert-watchdog.py MASTER_KEY (backup .bak-20260814-adminai). Recreated litellm container via `docker compose up -d --force-recreate litellm` (a bare `docker restart` does NOT re-read env vars — caught during verification). Vaultwarden consolidated: new key in item 'LiteLLM (admin-ai) Master Key' 0772c50c; updated 'admin-ai.itpropartner.com' bb01ef49 to the new key; deleted stale 'LiteLLM Master Key' 446da6ca and duplicate 'admin-ai.itpropartner.com' 8c8a5eb1. | Germaine (added new key to Vaultwarden as 'LiteLLM (admin-ai) Master Key', 2026-08-14) | Restore app1 .env from .env.bak-20260814-adminai and watchdog from .bak-20260814-adminai, then `docker compose up -d --force-recreate litellm` (old key is now revoked 401, so rollback requires restoring the old value to .env first); restore deleted Vaultwarden items from trash. | done | LiteLLM-adjacent=yes. Verified on app1 localhost:4000 — new key HTTP 200 on /spend/keys and /global/spend, old key HTTP 401 (revoked); cost-alert-watchdog ran clean against the new key (surfaced real alert: hermes-agent-v5 at 88% budget); Core 25-char virtual key 'Hermes (Sho'Nuff) LiteLLM Virtual API Key' (0aed1ca4) unaffected, /v1/chat/completions HTTP 200; LITELLM_SALT_KEY unchanged. Core admin-ai provider uses the 25-char virtual key (not the master key), so no Core config.yaml change was required. |

audit/phase-two/cost-estimate.md

@@ -0,0 +1,83 @@
+# ITPP Phase Two - Remediation Cost Estimate
+
+**Date:** 2026-08-13
+**Status:** AWAITING GERMAINE APPROVAL (C7 + C3 pre-approved fast-track, already executing)
+**Conductor:** Sho'Nuff
+
+---
+
+## Anchor (Phase One actuals, for calibration)
+
+Phase One was a read-only audit of 6 hosts plus the repo estate. From `brief.md` and the final report:
+
+- Approved estimate: subtotal ~$5.40
+- Realistic range: $8-10
+- Ceiling: ~$13
+- **Actual: exceeded the ceiling.** Driven by claude-sonnet-5 subagent usage and re-runs from 429 rate-limit failures.
+- 7-day estate-wide DeepSeek total: $61.81 (all usage, not audit-only).
+
+Phase Two is **live-change remediation** (investigate, change, verify, rollback-test, log), roughly 2-4x the per-unit cost of a read-only finding. The estimate below carries margin because Phase One actuals exceeded estimate. Under the $20/day cap, this spreads across ~8-12 execution days.
+
+Effort labels are taken verbatim from the Phase One report recommendations table (S = under 1h, M = half day, L = 1-2 days, XL = multi-day).
+
+---
+
+## Workstream 1 - Critical findings (itemized, per requirement)
+
+| Item | Finding (report ref) | Effort | Estimate |
+|---|---|---|---|
+| C1 | Docker published-port UFW bypass, ~20 public consoles | M-L | $5-14 |
+| C4 | Plaintext credentials, inventory + rotate + vault | L-XL | $12-28 |
+| C5 | LiteLLM Postgres backup gap (pg_dump wrong DB name) | S-M | $2-5 |
+| C6 | app3 shared MySQL, no tenant boundary (~24 sites) | L-XL | $12-28 |
+| C8 | wphost02 backup gap + decommission (split-brain) | M | $3-8 |
+| C9 | Warm standby state-DB sync (RPO ~28d stale) | M | $3-8 |
+| C10 | Single shared SSH key, passwordless root (Indep D4) | M | $3-8 |
+| D3 | Public repo re-leaks admin creds (EX-001, deferred) | M | $3-8 |
+| C3 | Wazuh agents on 5 hosts (FAST-TRACK, in progress) | M | $3-8 |
+| C7 | Grafana rotate + Tailscale (FAST-TRACK, in progress) | S | $1-3 |
+
+**WS1 subtotal: ~$47-118**
+
+## Workstreams 2-6 (workstream-level)
+
+Scope drawn from Phase One report sections noted.
+
+| Workstream | Scope (report ref) | Estimate |
+|---|---|---|
+| WS2 - High findings | MFA enforcement (Gitea/CloudPanel/Vaultwarden/Grafana, report #11), fail2ban + unattended-upgrades + auditd (#12), backup gaps + restore-tests (#13), UNMS to UISP + image pinning (#14), port 8200 + runaway processes + gateway systemd (#15), retire NOPASSWD:ALL for named sudo (#10) | $20-50 |
+| WS3 - Segmentation | C2 Tailscale ACL tags + three-tier model (internal/client/product) from report section 6 | $12-28 |
+| WS4 - Git reorg | Repo estate cleanup (report 2.3) + private-repo credential exposures (Git-A) | $5-14 |
+| WS5 - Documentation | Say-do gap fixes across 12 doc sections (report section 5) | $3-10 |
+| WS6 - Operationalize P&P | Stand up Running Exemptions doc + change-management cadence (bootstrapped) | $3-10 |
+
+**WS2-6 subtotal: ~$43-112**
+
+## Independence / verification buffer
+
+Conductor independence checks (Sonnet 5) on consequential live-change "is this safe to do live" calls, per the Phase One mitigation pattern: $6-12
+
+---
+
+## Totals
+
+| Line | Range |
+|---|---|
+| WS1 (Critical) | $47-118 |
+| WS2-6 | $43-112 |
+| Independence buffer | $6-12 |
+| **TOTAL** | **~$96-242 (midpoint ~$170)** |
+
+Pacing: ~8-12 days at the $20/day cap. No opus-tier models; primary DeepSeek V4 Pro, Sonnet 5 only for independence checks.
+
+## Scope-variance flags (flag before absorbing)
+
+- **C4** inventory may be larger than the report captured (6,296 Core / 5,504 app1-bu / 1,275 app2 secret-sprawl hits need per-file triage).
+- **C6** app3 DB segmentation is structural and may surface new per-tenant requirements mid-flight.
+- **D3** depends on Germaine's repo decision (currently deferred as EX-001).
+- **C1** on app1 is LiteLLM-adjacent (Hermes's own inference path); fallback chain will be used and logged.
+
+## Pre-flight confirmation
+
+- Fallback chain verified intact: primary = admin-ai (LiteLLM on app1); fallbacks = deepseek (direct api.deepseek.com), google, xai, anthropic, openai, all configured direct.
+- C7 (Core) and C3 (agent enrollment, manager ports already published) are NOT LiteLLM-adjacent; no fallback switch required.

audit/phase-two/rollback/grafana-inspect.json

@@ -0,0 +1,223 @@
+[
+    {
+        "Id": "66114deb715b3c90199bff9be1b4bbce3522e3152d6b4c0ff15256cbe2dfccf7",
+        "Created": "2026-07-16T06:40:35.087889714Z",
+        "Path": "/run.sh",
+        "Args": [],
+        "State": {
+            "Status": "running",
+            "Running": true,
+            "Paused": false,
+            "Restarting": false,
+            "OOMKilled": false,
+            "Dead": false,
+            "Pid": 670022,
+            "ExitCode": 0,
+            "Error": "",
+            "StartedAt": "2026-07-16T06:40:35.232657942Z",
+            "FinishedAt": "0001-01-01T00:00:00Z"
+        },
+        "Image": "sha256:c0b69935a2469f0add84fa71a1632a2594c78aeeb016ba9f5d8728a4b4c1b976",
+        "ResolvConfPath": "/var/lib/docker/containers/66114deb715b3c90199bff9be1b4bbce3522e3152d6b4c0ff15256cbe2dfccf7/resolv.conf",
+        "HostnamePath": "/var/lib/docker/containers/66114deb715b3c90199bff9be1b4bbce3522e3152d6b4c0ff15256cbe2dfccf7/hostname",
+        "HostsPath": "/var/lib/docker/containers/66114deb715b3c90199bff9be1b4bbce3522e3152d6b4c0ff15256cbe2dfccf7/hosts",
+        "LogPath": "/var/lib/docker/containers/66114deb715b3c90199bff9be1b4bbce3522e3152d6b4c0ff15256cbe2dfccf7/66114deb715b3c90199bff9be1b4bbce3522e3152d6b4c0ff15256cbe2dfccf7-json.log",
+        "Name": "/grafana",
+        "RestartCount": 0,
+        "Driver": "overlay2",
+        "Platform": "linux",
+        "MountLabel": "",
+        "ProcessLabel": "",
+        "AppArmorProfile": "docker-default",
+        "ExecIDs": null,
+        "HostConfig": {
+            "Binds": [
+                "grafana_data_final:/var/lib/grafana"
+            ],
+            "ContainerIDFile": "",
+            "LogConfig": {
+                "Type": "json-file",
+                "Config": {}
+            },
+            "NetworkMode": "host",
+            "PortBindings": {},
+            "RestartPolicy": {
+                "Name": "unless-stopped",
+                "MaximumRetryCount": 0
+            },
+            "AutoRemove": false,
+            "VolumeDriver": "",
+            "VolumesFrom": null,
+            "ConsoleSize": [
+                0,
+                0
+            ],
+            "CapAdd": null,
+            "CapDrop": null,
+            "CgroupnsMode": "private",
+            "Dns": [],
+            "DnsOptions": [],
+            "DnsSearch": [],
+            "ExtraHosts": null,
+            "GroupAdd": null,
+            "IpcMode": "private",
+            "Cgroup": "",
+            "Links": null,
+            "OomScoreAdj": 0,
+            "PidMode": "",
+            "Privileged": false,
+            "PublishAllPorts": false,
+            "ReadonlyRootfs": false,
+            "SecurityOpt": null,
+            "UTSMode": "",
+            "UsernsMode": "",
+            "ShmSize": 67108864,
+            "Runtime": "runc",
+            "Isolation": "",
+            "CpuShares": 0,
+            "Memory": 0,
+            "NanoCpus": 0,
+            "CgroupParent": "",
+            "BlkioWeight": 0,
+            "BlkioWeightDevice": [],
+            "BlkioDeviceReadBps": [],
+            "BlkioDeviceWriteBps": [],
+            "BlkioDeviceReadIOps": [],
+            "BlkioDeviceWriteIOps": [],
+            "CpuPeriod": 0,
+            "CpuQuota": 0,
+            "CpuRealtimePeriod": 0,
+            "CpuRealtimeRuntime": 0,
+            "CpusetCpus": "",
+            "CpusetMems": "",
+            "Devices": [],
+            "DeviceCgroupRules": null,
+            "DeviceRequests": null,
+            "MemoryReservation": 0,
+            "MemorySwap": 0,
+            "MemorySwappiness": null,
+            "OomKillDisable": null,
+            "PidsLimit": null,
+            "Ulimits": [],
+            "CpuCount": 0,
+            "CpuPercent": 0,
+            "IOMaximumIOps": 0,
+            "IOMaximumBandwidth": 0,
+            "MaskedPaths": [
+                "/proc/asound",
+                "/proc/acpi",
+                "/proc/kcore",
+                "/proc/keys",
+                "/proc/latency_stats",
+                "/proc/timer_list",
+                "/proc/timer_stats",
+                "/proc/sched_debug",
+                "/proc/scsi",
+                "/sys/firmware",
+                "/sys/devices/virtual/powercap"
+            ],
+            "ReadonlyPaths": [
+                "/proc/bus",
+                "/proc/fs",
+                "/proc/irq",
+                "/proc/sys",
+                "/proc/sysrq-trigger"
+            ]
+        },
+        "GraphDriver": {
+            "Data": {
+                "LowerDir": "/var/lib/docker/overlay2/3ab0a39f1573bc37ee3fc9203d8f876bb9ed9df325b39ca776e3ef86da2bc9ff-init/diff:/var/lib/docker/overlay2/96d3092d12c0e0b0c1bd01a5818f003bb7fab32246a893224296a67799841669/diff:/var/lib/docker/overlay2/4da7e8838b31487c835f93853d4a81e5cd06434fa34d04fdbbf2141cb95abc13/diff:/var/lib/docker/overlay2/8e1e7eaf9bac203b5e4a54e1c80c317f6483d40610b1776e9f6911aa18c41a72/diff:/var/lib/docker/overlay2/02a0a88da9ab58ca40a03ab418e1248317e5917fa318654d4cf60bf394b21b07/diff:/var/lib/docker/overlay2/9cc7ea00867e008c469d86f0bf2244d3e60737a665648609a303c93c8e3805cf/diff:/var/lib/docker/overlay2/8b1f24758f00124ab15b7b5b9957594184426ad6aa1911d76c5add384d1613ed/diff:/var/lib/docker/overlay2/bd4250cfbd3624493c084c81954c59a9744c13a863da7cd3ef348f0accf19719/diff:/var/lib/docker/overlay2/041749d1c02cf48d8a3257d604c5a7cfae504fc36f492854a4e056c0562cf599/diff:/var/lib/docker/overlay2/21d92f72aced24d754ce4a6daf302e9dd122273b48fdd107e8ede5c1018d4a4d/diff:/var/lib/docker/overlay2/2311d9ffcf0c02b1565678784a4f5bb120660ae0c021094f152cbd9747a5b916/diff",
+                "MergedDir": "/var/lib/docker/overlay2/3ab0a39f1573bc37ee3fc9203d8f876bb9ed9df325b39ca776e3ef86da2bc9ff/merged",
+                "UpperDir": "/var/lib/docker/overlay2/3ab0a39f1573bc37ee3fc9203d8f876bb9ed9df325b39ca776e3ef86da2bc9ff/diff",
+                "WorkDir": "/var/lib/docker/overlay2/3ab0a39f1573bc37ee3fc9203d8f876bb9ed9df325b39ca776e3ef86da2bc9ff/work"
+            },
+            "Name": "overlay2"
+        },
+        "Mounts": [
+            {
+                "Type": "volume",
+                "Name": "grafana_data_final",
+                "Source": "/var/lib/docker/volumes/grafana_data_final/_data",
+                "Destination": "/var/lib/grafana",
+                "Driver": "local",
+                "Mode": "z",
+                "RW": true,
+                "Propagation": ""
+            }
+        ],
+        "Config": {
+            "Hostname": "core",
+            "Domainname": "",
+            "User": "472",
+            "AttachStdin": false,
+            "AttachStdout": false,
+            "AttachStderr": false,
+            "ExposedPorts": {
+                "3000/tcp": {}
+            },
+            "Tty": false,
+            "OpenStdin": false,
+            "StdinOnce": false,
+            "Env": [
+                "GF_SECURITY_ADMIN_USER=admin",
+                "GF_SECURITY_ADMIN_PASSWORD=admin",
+                "GF_SERVER_HTTP_PORT=3002",
+                "PATH=/usr/share/grafana/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
+                "GF_PATHS_CONFIG=/etc/grafana/grafana.ini",
+                "GF_PATHS_DATA=/var/lib/grafana",
+                "GF_PATHS_HOME=/usr/share/grafana",
+                "GF_PATHS_LOGS=/var/log/grafana",
+                "GF_PATHS_PLUGINS=/var/lib/grafana/plugins",
+                "GF_PATHS_PROVISIONING=/etc/grafana/provisioning"
+            ],
+            "Cmd": null,
+            "Image": "grafana/grafana:11.4.0",
+            "Volumes": null,
+            "WorkingDir": "/usr/share/grafana",
+            "Entrypoint": [
+                "/run.sh"
+            ],
+            "OnBuild": null,
+            "Labels": {
+                "maintainer": "Grafana Labs <hello@grafana.com>"
+            }
+        },
+        "NetworkSettings": {
+            "Bridge": "",
+            "SandboxID": "fb05509631e71c8372e97ea4f43331331476ce53ecef4e52494b20e96c9debc0",
+            "SandboxKey": "/var/run/docker/netns/default",
+            "Ports": {},
+            "HairpinMode": false,
+            "LinkLocalIPv6Address": "",
+            "LinkLocalIPv6PrefixLen": 0,
+            "SecondaryIPAddresses": null,
+            "SecondaryIPv6Addresses": null,
+            "EndpointID": "",
+            "Gateway": "",
+            "GlobalIPv6Address": "",
+            "GlobalIPv6PrefixLen": 0,
+            "IPAddress": "",
+            "IPPrefixLen": 0,
+            "IPv6Gateway": "",
+            "MacAddress": "",
+            "Networks": {
+                "host": {
+                    "IPAMConfig": null,
+                    "Links": null,
+                    "Aliases": null,
+                    "MacAddress": "",
+                    "NetworkID": "f7ebc0a30ba605067a1c4f5b22c519f3c3cb4f46e8a4b4359787cf0fe8d09a37",
+                    "EndpointID": "458bbfe47f84e5aab9820274ff509a215b8b24bc99e8cbb388081a0d75f028b0",
+                    "Gateway": "",
+                    "IPAddress": "",
+                    "IPPrefixLen": 0,
+                    "IPv6Gateway": "",
+                    "GlobalIPv6Address": "",
+                    "GlobalIPv6PrefixLen": 0,
+                    "DriverOpts": null,
+                    "DNSNames": null
+                }
+            }
+        }
+    }
+]

audit/phase-two/running-exemptions.md

@@ -0,0 +1,37 @@
+# Running Exemptions Document
+
+Authority: Germaine Brown is the sole authorizer of any departure from the
+Policy & Procedure document (P&P section 8). No exemption is assumed; it is requested,
+justified, approved, and recorded here.
+
+## Schema
+
+| Field | Meaning |
+|---|---|
+| ID | EX-### |
+| Date | When requested/approved (ET) |
+| Requester | Who raised it |
+| Provision departed from | Which P&P section/policy |
+| Request + justification | The business reason |
+| Authorization | Germaine's decision |
+| One-time / ongoing | Duration |
+| Follow-up date | For ongoing exemptions |
+| Status | open / closed |
+
+---
+
+## Entries
+
+### EX-001: Public-repo credential exposure (D3) deferred
+
+| Field | Value |
+|---|---|
+| ID | EX-001 |
+| Date | 2026-08-13 (raised in Phase One, re-surfaced at Phase Two start) |
+| Requester | Germaine Brown |
+| Provision departed from | P&P section 1 (no live credentials in any repo): the public repo `itpp-infrastructure` re-leaks live admin credentials |
+| Request + justification | Defer the repo lockdown ("leave the repo alone for now") while Phase One was read-only |
+| Authorization | Deferred by Germaine in Phase One |
+| One-time / ongoing | One-time, with a Phase Two follow-up |
+| Follow-up date | Phase Two start (now): this is Workstream 1 item 10 (D3/EX-001) |
+| Status | OPEN: pending Germaine's updated decision now that Phase Two is starting |

audit/phase-two/scripts/app1-docker-user.sh

@@ -0,0 +1,30 @@
+#!/bin/bash
+# ITPP Phase Two C1 - DOCKER-USER hardening (management consoles Tailscale-only)
+# Host: app1 (152.53.36.131) | Applied 2026-08-13 | Authorized: Phase Two Workstream 1 C1 (Germaine)
+# Idempotent: flushes and rebuilds DOCKER-USER (IPv4 + IPv6). Safe to re-run manually.
+# Closes (Tailscale-only): komodo 9120, twenty-server 3003, wazuh dashboard 5601, wazuh indexer 9200.
+# Preserves (public): wazuh manager 1514/1515/55000 tcp + 514 udp (5 active agents).
+# NOTE: litellm is 127.0.0.1:4000 behind Caddy (admin-ai.itpropartner.com) - untouched here.
+set -eu
+
+if ! iptables -L DOCKER-USER -n >/dev/null 2>&1; then
+    iptables -N DOCKER-USER
+fi
+iptables -F DOCKER-USER
+iptables -A DOCKER-USER -s 100.64.0.0/10 -j RETURN
+iptables -A DOCKER-USER -m conntrack --ctstate ESTABLISHED,RELATED -j RETURN
+iptables -A DOCKER-USER -i eth0 -p tcp -m multiport --dports 1514,1515,55000 -j RETURN
+iptables -A DOCKER-USER -i eth0 -p udp -m multiport --dports 514 -j RETURN
+iptables -A DOCKER-USER -i eth0 -j DROP
+iptables -A DOCKER-USER -j RETURN
+
+if ip6tables -L DOCKER-USER -n >/dev/null 2>&1; then
+    ip6tables -F DOCKER-USER
+    ip6tables -A DOCKER-USER -s fd7a:115c:a1e0::/48 -j RETURN
+    ip6tables -A DOCKER-USER -m conntrack --ctstate ESTABLISHED,RELATED -j RETURN
+    ip6tables -A DOCKER-USER -i eth0 -p tcp -m multiport --dports 1514,1515,55000 -j RETURN
+    ip6tables -A DOCKER-USER -i eth0 -p udp -m multiport --dports 514 -j RETURN
+    ip6tables -A DOCKER-USER -i eth0 -j DROP
+    ip6tables -A DOCKER-USER -j RETURN
+fi
+exit 0

audit/phase-two/scripts/app2-docker-user.sh

@@ -0,0 +1,42 @@
+#!/bin/bash
+# ITPP Phase Two C1 - DOCKER-USER hardening (management consoles Tailscale-only)
+# Host: app2 (152.53.39.202) | Applied 2026-08-13 | Authorized: Phase Two Workstream 1 C1 (Germaine)
+# Idempotent: flushes and rebuilds DOCKER-USER (IPv4 + IPv6). Safe to re-run manually.
+#
+# Closes (Tailscale-only, dropped by the final eth0 DROP; admin UIs still reachable via Caddy :443 -> localhost):
+#   unifi-controller admin UI 8443 + guest portal 8843/8880 (unifi.itpropartner.com -> Caddy -> localhost:8443);
+#   unms-nginx admin UI 8444 + http 81 (unms.forefrontwireless.com -> Caddy -> localhost:8444);
+#   minio console 9001, infinity 23817+23820, ragflow 9380-9384+9392+9393 (ragflow.itpropartner.com -> Caddy -> localhost:9392);
+#   gitea ssh 3022 (web is 127.0.0.1:3001 -> Caddy git.itpropartner.com).
+# Preserves (public):
+#   technitium DNS 53 tcp+udp;
+#   bookstack 6875 (published 6875 -> container 80, matched via ctorigdstport; app3 support site proxies here);
+#   support-api 6880 (app3 support.itpropartner.com -> 152.53.39.202:6880);
+#   traccar 8082 + 5000-5150 tcp+udp (GPS devices + Core Caddy cross-proxy);
+#   unifi-controller 8080 tcp (23 remote devices inform here) + 3478/10001 udp (STUN/discovery);
+#   unms-nginx 8089 tcp + unms-netflow 2055 udp (UISP device-facing; conservative - see change log notes).
+set -eu
+
+if ! iptables -L DOCKER-USER -n >/dev/null 2>&1; then
+    iptables -N DOCKER-USER
+fi
+iptables -F DOCKER-USER
+iptables -A DOCKER-USER -s 100.64.0.0/10 -j RETURN
+iptables -A DOCKER-USER -m conntrack --ctstate ESTABLISHED,RELATED -j RETURN
+iptables -A DOCKER-USER -i eth0 -p tcp -m multiport --dports 53,6880,8080,8082,8089,5000:5150 -j RETURN
+iptables -A DOCKER-USER -i eth0 -p tcp -m conntrack --ctorigdstport 6875 -j RETURN
+iptables -A DOCKER-USER -i eth0 -p udp -m multiport --dports 53,2055,3478,10001,5000:5150 -j RETURN
+iptables -A DOCKER-USER -i eth0 -j DROP
+iptables -A DOCKER-USER -j RETURN
+
+if ip6tables -L DOCKER-USER -n >/dev/null 2>&1; then
+    ip6tables -F DOCKER-USER
+    ip6tables -A DOCKER-USER -s fd7a:115c:a1e0::/48 -j RETURN
+    ip6tables -A DOCKER-USER -m conntrack --ctstate ESTABLISHED,RELATED -j RETURN
+    ip6tables -A DOCKER-USER -i eth0 -p tcp -m multiport --dports 53,6880,8080,8082,8089,5000:5150 -j RETURN
+    ip6tables -A DOCKER-USER -i eth0 -p tcp -m conntrack --ctorigdstport 6875 -j RETURN
+    ip6tables -A DOCKER-USER -i eth0 -p udp -m multiport --dports 53,2055,3478,10001,5000:5150 -j RETURN
+    ip6tables -A DOCKER-USER -i eth0 -j DROP
+    ip6tables -A DOCKER-USER -j RETURN
+fi
+exit 0

audit/phase-two/scripts/app3-docker-user.sh

@@ -0,0 +1,25 @@
+#!/bin/bash
+# ITPP Phase Two C1 - DOCKER-USER hardening (management consoles Tailscale-only)
+# Host: app3 (152.53.241.111) | Applied 2026-08-13 | Authorized: Phase Two Workstream 1 C1 (Germaine)
+# Idempotent: flushes and rebuilds DOCKER-USER (IPv4 + IPv6). Safe to re-run manually.
+# Closes (Tailscale-only): buzz-prod-relay 3000/tcp (publicly served via nginx loopback buzz.iamgmb.com).
+# Preserves: none (hexclave 8101-8102 is loopback; nginx 80/443 is host INPUT chain).
+set -eu
+
+if ! iptables -L DOCKER-USER -n >/dev/null 2>&1; then
+    iptables -N DOCKER-USER
+fi
+iptables -F DOCKER-USER
+iptables -A DOCKER-USER -s 100.64.0.0/10 -j RETURN
+iptables -A DOCKER-USER -m conntrack --ctstate ESTABLISHED,RELATED -j RETURN
+iptables -A DOCKER-USER -i eth0 -j DROP
+iptables -A DOCKER-USER -j RETURN
+
+if ip6tables -L DOCKER-USER -n >/dev/null 2>&1; then
+    ip6tables -F DOCKER-USER
+    ip6tables -A DOCKER-USER -s fd7a:115c:a1e0::/48 -j RETURN
+    ip6tables -A DOCKER-USER -m conntrack --ctstate ESTABLISHED,RELATED -j RETURN
+    ip6tables -A DOCKER-USER -i eth0 -j DROP
+    ip6tables -A DOCKER-USER -j RETURN
+fi
+exit 0

audit/phase-two/scripts/core-docker-user.sh

@@ -0,0 +1,21 @@
+#!/bin/bash
+# ITPP Phase Two C1 - DOCKER-USER hardening (management consoles Tailscale-only)
+# Host: core (152.53.192.33) | Applied 2026-08-13 | Authorized: Phase Two Workstream 1 C1 (Germaine)
+# Idempotent: flushes and rebuilds the DOCKER-USER chain. Safe to re-run manually.
+# Closes (Tailscale-only): browserless 3000/tcp, camofox-browser 9377/tcp, uptime-kuma 3001/tcp.
+# Preserves: none (no public Docker services on core; Caddy 80/443 is host INPUT chain).
+set -eu
+
+if ! iptables -L DOCKER-USER -n >/dev/null 2>&1; then
+    iptables -N DOCKER-USER
+fi
+iptables -F DOCKER-USER
+iptables -A DOCKER-USER -s 100.64.0.0/10 -j RETURN
+iptables -A DOCKER-USER -m conntrack --ctstate ESTABLISHED,RELATED -j RETURN
+iptables -A DOCKER-USER -i eth0 -j DROP
+iptables -A DOCKER-USER -j RETURN
+
+# IPv6: Docker ip6tables integration is off on core (no ip6tables DOCKER-USER chain,
+# no IPv6 DNAT). [::] publishes are userland docker-proxy (INPUT path) and are already
+# blocked by UFW IPv6 default-deny (only 22/80/443/8080-tailscale/51821/8890 allowed).
+exit 0

audit/phase-two/scripts/itpp-docker-user.service

@@ -0,0 +1,12 @@
+[Unit]
+Description=ITPP P2-C1 DOCKER-USER firewall hardening (Tailscale-only management consoles)
+After=docker.service
+Wants=docker.service
+
+[Service]
+Type=oneshot
+ExecStart=/usr/local/sbin/itpp-docker-user.sh
+RemainAfterExit=yes
+
+[Install]
+WantedBy=multi-user.target

backup-dr-audit-2026-08-10.md

@@ -0,0 +1,423 @@
+# ITPP Backup & DR — Full Audit Report
+
+> **Date:** 2026-08-10 ~20:45 ET  
+> **Auditor:** Hermes Agent (automated audit)  
+> **Scope:** All ITPP infrastructure — Core, app1, app2, app3, wphost02, MikroTik CCR  
+> **Storage Verified:** Wasabi S3 (hermes-vps-backups, mikrotik-ccr-backups)
+
+---
+
+## Executive Summary
+
+- **Overall grade: D** — Multiple active data-loss risks, no restore testing, several services completely unprotected.
+- **Critical gaps** (items that risk data loss RIGHT NOW):
+  1. **app3 MySQL backups stopped 2026-08-08** — 9 databases (apextrackexperience, boxpilotlogistics, debtrecoveryexperts, iAmGMB, intelsight, katiewattsdesign, mainwp, vigilanttac, voipsimplicity_site) have no backup for Aug 9–10. Two days of production data completely unprotected.
+  2. **git.modelortho.com (Anita's Gitea) has a backup script and Hermes cron, but the cron has NEVER executed** — the cron job exists with no `Last run` timestamp. Only one manual backup exists on S3.
+  3. **Prometheus TSDB — ZERO backups ever** — `core/prometheus/` S3 prefix is completely empty. All monitoring history, alert rules, and dashboards are unprotected.
+  4. **10+ production services have NO backup** — buzz-prod (Block Buzz relay + PostgreSQL + Redis + MinIO), support-api, bookstack (×2), searxng, timetrex, microbin, camofox-browser, browserless, msp-forms, docs-auth-validator, transitpin-api, transitpin-relay.
+  5. **ZERO evidence of any verified restore** — backups exist but nobody has ever tested if they can actually be restored.
+- **High-priority improvements:**
+  1. Fix app3 MySQL backup immediately
+  2. Get gitea-modelortho cron running (it exists but hasn't fired)
+  3. Create backups for all unbacked services
+  4. Stand up Prometheus backup
+  5. Perform a restore test of at least one backup within 7 days
+
+---
+
+## 1. Backup Inventory & Verification
+
+### Legend
+- ✅ = Healthy (script exists, runs, S3 file present, size normal)
+- ⚠️ = Warning (present but suspicious — unchanging size, stale data, missing days)
+- 🔴 = Failed (missing entirely, script broken, zero files)
+- 👻 = Phantom (script in plan but doesn't exist on disk)
+- ❓ = Unknown (could not verify execution)
+
+### Core (152.53.192.33)
+
+| Target | Script | Script Exists? | S3 Files (7d) | Size Healthy? | Status | Notes |
+|--------|--------|---------------|---------------|---------------|--------|-------|
+| Hermes Full Backup | `hermes-backup.sh` (system cron 1AM) | ✅ Core | ✅ Aug 4–10 daily, 1.2–1.3 GB | ✅ Growing | ✅ | |
+| Hermes Live Sync | `hermes-live-sync.sh` (Hermes cron, every 15m) | ✅ Core | ⚠️ Only `verification_evidence.db`, no state.db | ⚠️ Stale | ⚠️ | `live/` has old malformed backups from Jul 9; current sync may go elsewhere |
+| /root Essentials | `root-essentials-backup.sh` (system cron 3AM) | ✅ Core | ✅ Aug 4–10 daily, 170–236 MB | ✅ Growing | ✅ | |
+| Grafana | `core-services-backup.sh` (system cron 1:30AM) | ✅ Core | ✅ Aug 4–10 daily, ~52 KB | ✅ Consistent | ✅ | |
+| Uptime Kuma | `core-services-backup.sh` | ✅ Core | ✅ Aug 4–10 daily, ~100 MB | ✅ Growing | ✅ | |
+| Docker Volumes | `core-services-backup.sh` | ✅ Core | ⚠️ vaultwarden-data only, stale since Jul 28 | ⚠️ Stale | ⚠️ | vaultwarden-data backed up but service migrated; no other volumes visible |
+| Prometheus | `core-services-backup.sh` | ✅ Core | 🔴 **EMPTY** | 🔴 **Zero bytes** | 🔴 | `core/prometheus/` has NO files ever |
+| Auth API | `auth-api-backup.sh` (Hermes cron 3:15AM) | ✅ Core | ✅ Aug 8–10 daily, ~41 KB | ✅ Consistent | ✅ | |
+
+### App1 (152.53.36.131)
+
+| Target | Script | Script Exists? | S3 Files (7d) | Size Healthy? | Status | Notes |
+|--------|--------|---------------|---------------|---------------|--------|-------|
+| Open WebUI | `/root/backup.sh` (app1 cron 2AM) | ✅ app1 | ⚠️ Aug 4–10 daily | ⚠️ **Identical size 5 days** (1,470,272,729 bytes Aug 5–9) | ⚠️ | Size finally changed Aug 10 (1,470,769,575). Docker cp may capture stale data |
+| LiteLLM DB | `litellm-backup.sh` (Hermes cron 3:30AM) | ✅ Core | ✅ Aug 4–10 daily, 15–21 MB, growing | ✅ Growing | ✅ | Also has config-only files from app1 backup.sh (240 bytes) |
+| n8n | `/root/backup.sh` (app1 cron 2AM) | ✅ app1 | ✅ Aug 4–10 daily, ~60–64 KB | ✅ Consistent | ✅ | |
+| MCP Configs | `/root/backup.sh` (app1 cron 2AM) | ✅ app1 | ✅ Aug 4–10 daily, 373 bytes | ✅ Consistent | ✅ | Very small — configs only |
+| Vaultwarden | `vaultwarden-backup.sh` (Hermes cron 2:30AM) | ✅ Core | ✅ Aug 4–10 daily, 520–755 KB | ✅ Growing | ✅ | |
+| Komodo | `komodo-backup.sh` (Hermes cron 3:45AM) | ✅ Core | ✅ Aug 4–10 daily, ~12.7 KB | ✅ Consistent | ✅ | |
+| DocuSeal | `docuseal-backup.sh` (Hermes cron 4AM) | ✅ Core | ✅ Aug 4–10 daily, ~255 KB | ✅ Consistent | ✅ | |
+| Twenty CRM | `twenty-backup.sh` (Hermes cron 4:15AM) | ✅ Core | ✅ Aug 3–10 daily, ~156 KB | ✅ Consistent | ✅ | Also has `twenty-files` tarballs from app1 backup.sh |
+| Kokoro TTS | *(stateless)* | N/A | N/A | N/A | ✅ | |
+
+### App2 (152.53.39.202)
+
+| Target | Script | Script Exists? | S3 Files (7d) | Size Healthy? | Status | Notes |
+|--------|--------|---------------|---------------|---------------|--------|-------|
+| Traccar | `/root/backup.sh` (app2 cron 2:30AM) | ✅ app2 | ✅ Aug 4–10 daily, ~1.8 MB | ✅ Slightly growing | ✅ | |
+| Gitea | `gitea-backup.sh` (Hermes cron 8AM) | ✅ Core | ✅ Aug 4–10 daily (folders w/ db + repos) | ✅ DB 2.7 MB, 30+ repos | ✅ | |
+| Hudu | `hudu-backup.sh` (Hermes cron 7AM) | ✅ Core | ✅ Aug 4–10 daily, 670–714 KB | ✅ Growing | ✅ | |
+| UNMS | `unms-backup-sync.sh` (Hermes cron 6AM) | ✅ Core | ✅ Daily since Aug 5, ~100 MB | ✅ Consistent | ✅ | Gap Jul 31–Aug 5 |
+| UniFi | `unifi-backup-sync.sh` (Hermes cron 2AM) | ✅ Core | ✅ Aug 4–10 daily, ~884–893 KB | ✅ Consistent | ✅ | |
+| Technitium DNS | `technitium-backup.sh` (Hermes cron 2:45AM) | ✅ Core | ✅ Aug 6–10 daily, 369–461 KB, growing | ✅ Growing | ✅ | Also has duplicates from app2 backup.sh |
+| Dawarich | `dawarich-backup.sh` (Hermes cron 4AM) | ✅ Core | ✅ Aug 6–10 daily, ~7 MB | ✅ Consistent | ✅ | Also backed up by app2 backup.sh |
+| RAGFlow | `ragflow-backup.sh` (Hermes cron 4:15AM) | ✅ Core | ✅ Aug 8–10 daily, ~267 KB | ⚠️ Unchanging | ⚠️ | Identical size all 3 days — very small for MySQL |
+
+### App3 (152.53.241.111)
+
+| Target | Script | Script Exists? | S3 Files (7d) | Size Healthy? | Status | Notes |
+|--------|--------|---------------|---------------|---------------|--------|-------|
+| CloudPanel DB | `/root/backup.sh` (app3 cron 3AM) | ✅ app3 | ✅ Aug 4–10 daily, 1.4–1.6 MB | ✅ Growing | ✅ | |
+| MySQL (all DBs) | `/root/backup.sh` (app3 cron 3AM) | ✅ app3 | 🔴 **Last: Aug 8** | 🔴 **No Aug 9–10** | 🔴 | **CRITICAL**: 9 DBs stopped backing up after Aug 8. No mysql lines in Aug 10 log. |
+| WordPress Files | `/root/backup.sh` (app3 cron 3AM) | ✅ app3 | ✅ Aug 9 daily, 9 sites | ✅ Normal | ✅ | `wp-www` identical size for 10 days is suspicious but other sites vary |
+| Nginx Configs | `/root/backup.sh` (app3 cron 3AM) | ✅ app3 | ✅ Aug 4–8 daily, ~12 MB | ✅ Growing | ⚠️ | **FAILED on Aug 10** per cron log: "Nginx configs: FAILED" |
+| Static Sites | `/root/backup.sh` (app3 cron 3AM) | ✅ app3 | ✅ Aug 9–10 daily, 17 sites | ✅ Normal | ✅ | Includes modelortho.com, buzz.iamgmb.com, transitpin.com, etc. |
+| WP Snapshots (local) | `/opt/backup-restore/snapshot.sh` (app3 cron 1AM/1PM) | ✅ app3 | ❓ Local only | ❓ Not verified | ❓ | Local snapshots, not in S3 |
+| Hexclave (Stack Auth) | `hexclave-backup.sh` (Hermes cron 3:30AM) | ✅ Core | ✅ Aug 8–10 daily, ~315 KB | ✅ Consistent | ✅ | |
+| modelortho.com | `modelortho-backup.sh` (listed in plan) | 👻 **MISSING** | N/A | N/A | 👻 | **Does not exist**. However static modelortho.com IS backed up via app3/static. |
+| git.modelortho.com (Anita's Gitea) | `gitea-modelortho-backup.sh` (Hermes cron 4:30AM) | ✅ Core | 🔴 **Only 1 file** (manual?) | 🔴 **Cron never ran** | 🔴 | **CRITICAL**: Script exists, cron exists, but cron has NO `Last run`. Zero automated backups. |
+
+### wphost02 (5.161.62.38)
+
+| Target | Script | Script Exists? | S3 Files (7d) | Size Healthy? | Status | Notes |
+|--------|--------|---------------|---------------|---------------|--------|-------|
+| WordPress (7 sites) | `/root/backup.sh` (Core cron 5AM via SSH) | ✅ wphost02 | ✅ Aug 4–10 daily | ✅ Normal (50–900 MB per site) | ✅ | 7 sites + all-databases.sql + RunCloud config |
+
+### MikroTik CCR
+
+| Target | Script | Script Exists? | S3 Files (7d) | Size Healthy? | Status | Notes |
+|--------|--------|---------------|---------------|---------------|--------|-------|
+| Home Gateway | `run-wisp-backup.sh` (Hermes cron 6AM) | ✅ Core | ✅ Daily config .rsc | ✅ Normal | ⚠️ | Home gateway OK, but **5 tower CCRs all timeout** (T01–T04, MP100) |
+| WISP Tower CCRs | Same script | ✅ Core | 🔴 **None since Jul 7** | 🔴 | 🔴 | SSH timeout to all towers. Only 1 historical config from Jul 7. |
+
+### Hetzner Snapshots
+
+| Target | Script | Script Exists? | S3 Files (7d) | Status | Notes |
+|--------|--------|---------------|---------------|--------|-------|
+| Weekly disk snapshots | `snapshot-hetzner.py` (Hermes cron Mon 5AM) | ✅ Core | N/A (Hetzner API, not S3) | ✅ | Last run Aug 10, ok |
+
+---
+
+## 2. Missing Backups
+
+The backup plan's "Unbacked Services" section claims "(none)" — this is **incorrect**. The following services are running in production with ZERO backup coverage:
+
+### Core (152.53.192.33)
+
+| Service | Type | Data at Risk | Priority |
+|---------|------|-------------|----------|
+| searxng | Docker (search engine) | Configuration, any cached indexes | Low |
+| timetrex | Docker (time tracking) | Employee time data, payroll records | **High** |
+| microbin | Docker (pastebin) | Shared text snippets | Low |
+| camofox-browser | Docker (browser automation) | Stateless? | Low |
+| browserless | Docker (headless browser) | Stateless | Low |
+| Prometheus | systemd (monitoring TSDB) | **All metrics history, alert rules, dashboards** | **High** |
+
+### App1 (152.53.36.131)
+
+| Service | Type | Data at Risk | Priority | Notes |
+|---------|------|-------------|----------|-------|
+| Wazuh (SIEM/XDR) | Docker (×3 containers) | Security events, agent configs, alerts | **High** | Has `wazuh-backup.sh` on app1 and daily S3 files, but NOT in backup plan or Hermes cron. **Backup works but is undocumented/unanchored.** |
+
+### App2 (152.53.39.202)
+
+| Service | Type | Data at Risk | Priority |
+|---------|------|-------------|----------|
+| support-api | Docker (support API) | Support ticket data | **High** |
+| bookstack | Docker (wiki) | Documentation wiki content | **High** |
+| bookstack-db | Docker (MariaDB for bookstack) | Wiki database | **High** |
+| happy_rosalind | Docker (bookstack duplicate?) | Unknown content | Medium |
+
+### App3 (152.53.241.111)
+
+| Service | Type | Data at Risk | Priority |
+|---------|------|-------------|----------|
+| buzz-prod-relay | Docker (Block Buzz relay) | Relay configuration, keys | **High** |
+| buzz-prod-postgres | Docker (Buzz PostgreSQL) | **All Buzz relay state** | **Critical** |
+| buzz-prod-redis | Docker (Buzz Redis) | Session/cache data | Medium |
+| buzz-prod-minio | Docker (Buzz object storage) | Uploaded files | Medium |
+| transitpin-api | systemd | Transportation API data | Medium |
+| transitpin-relay | systemd | Relay config | Medium |
+| msp-forms | systemd | Form submissions | Medium |
+| docs-auth-validator | systemd | Auth validation state | Low |
+
+---
+
+## 3. 3-2-1 Compliance
+
+The 3-2-1 rule states: **3 copies** of data, on **2 different media**, with **1 off-site**.
+
+| Service | 3 Copies? | 2 Media? | 1 Off-Site? | Verdict |
+|---------|-----------|----------|-------------|---------|
+| Hermes Full Backup | ✅ (live + daily S3 + standby) | ⚠️ (all Wasabi S3) | ✅ (Wasabi us-east-1) | **PARTIAL** — single media type |
+| App1 services (LiteLLM, n8n, Vaultwarden, etc.) | ✅ (server + S3) | ⚠️ (all Wasabi S3) | ✅ | **PARTIAL** |
+| App2 services (Gitea, Hudu, Traccar, etc.) | ✅ (server + S3) | ⚠️ (all Wasabi S3) | ✅ | **PARTIAL** |
+| App3 MySQL | 🔴 (server only, no S3 since Aug 8) | 🔴 | 🔴 | **FAIL** |
+| App3 WordPress | ✅ (server + S3 + local snapshots) | ✅ (S3 + local disk) | ✅ | **PASS** |
+| App3 static sites | ✅ (server + S3) | ⚠️ (all Wasabi S3) | ✅ | **PARTIAL** |
+| wphost02 WordPress | ✅ (server + S3) | ⚠️ (all Wasabi S3) | ✅ | **PARTIAL** |
+| git.modelortho.com | 🔴 (only server, cron never ran) | 🔴 | 🔴 | **FAIL** |
+| Prometheus | 🔴 (server only) | 🔴 | 🔴 | **FAIL** |
+| Wazuh | ✅ (server + S3) | ⚠️ (all Wasabi S3) | ✅ | **PARTIAL** |
+| Buzz / Bookstack / Support-API / etc. | 🔴 (server only) | 🔴 | 🔴 | **FAIL** |
+| MikroTik Home Gateway | ✅ (router + S3) | ⚠️ (all Wasabi S3) | ✅ | **PARTIAL** |
+| MikroTik Tower CCRs | 🔴 (router only) | 🔴 | 🔴 | **FAIL** |
+
+**Overall 3-2-1 compliance: FAIL**
+
+The strategy relies entirely on Wasabi S3 for off-site storage — there is no second media type (no local NAS, no tape, no separate cloud provider). Every service that passes does so only by counting the production server + S3 as two copies. For true 2-media compliance, a second storage type (e.g., local NAS, separate cloud provider, or physical media) would be required.
+
+---
+
+## 4. RPO/RTO Assessment
+
+| Service | Tier | Planned RPO | Current RPO | Acceptable? | Planned RTO | Current RTO Est. | Acceptable? |
+|---------|------|-------------|-------------|-------------|-------------|-----------------|-------------|
+| Hermes Agent | Critical | ≤1hr | ~15 min (live sync) | ✅ | ≤4hr | ~2–4hr | ✅ |
+| Gitea (app2) | Critical | ≤1hr | 24hr (daily only) | ⚠️ | ≤4hr | ~2–4hr | ✅ |
+| Traccar | Critical | ≤1hr | 24hr (daily only) | ⚠️ | ≤4hr | ~2–4hr | ✅ |
+| UISP (UNMS) | Critical | ≤1hr | 24hr | ⚠️ | ≤4hr | ~4–8hr | ⚠️ |
+| LiteLLM | High | 24hr | 24hr | ✅ | ≤8hr | ~4–8hr | ✅ |
+| n8n | High | 24hr | 24hr | ✅ | ≤8hr | ~4–8hr | ✅ |
+| Open WebUI | High | 24hr | 24hr | ✅ | ≤8hr | ~4–8hr | ✅ |
+| Vaultwarden | High | 24hr | 24hr | ✅ | ≤8hr | ~2–4hr | ✅ |
+| Twenty CRM | High | 24hr | 24hr | ✅ | ≤8hr | ~4–8hr | ✅ |
+| Hudu | Medium | 24hr | 24hr | ✅ | ≤24hr | ~8–24hr | ✅ |
+| UniFi | Medium | 24hr | 24hr | ✅ | ≤24hr | ~4–8hr | ✅ |
+| Komodo | Medium | 24hr | 24hr | ✅ | ≤24hr | ~2–4hr | ✅ |
+| DocuSeal | Medium | 24hr | 24hr | ✅ | ≤24hr | ~2–4hr | ✅ |
+| App3 WP sites | Medium | 24hr | 24hr | ✅ | ≤24hr | ~8–24hr | ✅ |
+| Auth API | Medium | 24hr | 24hr | ✅ | ≤24hr | ~2–4hr | ✅ |
+| Hexclave (Stack Auth) | Medium | 24hr | 24hr | ✅ | ≤24hr | ~4–8hr | ✅ |
+| Grafana | Low | 24hr | 24hr | ✅ | ≤48hr | ~2–4hr | ✅ |
+| Uptime Kuma | Low | 24hr | 24hr | ✅ | ≤48hr | ~2–4hr | ✅ |
+| Prometheus | Low | 24hr | 🔴 **∞ (no backup)** | 🔴 | ≤48hr | 🔴 **Impossible** | 🔴 |
+| MikroTik CCR | Low | 24hr | 24hr (home) / 🔴 **∞ (towers)** | 🔴 | ≤48hr | ~4–8hr (home) / 🔴 **Impossible** (towers) | 🔴 |
+| Technitium DNS | Low | 24hr | 24hr | ✅ | ≤48hr | ~2–4hr | ✅ |
+| Dawarich | Low | 24hr | 24hr | ✅ | ≤48hr | ~4–8hr | ✅ |
+| RAGFlow | Low | 24hr | 24hr | ⚠️ | ≤48hr | ~4–8hr | ✅ |
+| **Buzz / Bookstack / Support-API / etc.** | **Unclassified** | N/A | 🔴 **∞ (no backup)** | 🔴 | N/A | 🔴 **Impossible** | 🔴 |
+| **git.modelortho.com** | **Unclassified** | N/A | 🔴 **∞ (cron never ran)** | 🔴 | N/A | 🔴 **Possible** (script exists) | 🔴 |
+| **app3 MySQL** | **Medium** | 24hr | 🔴 **~48hr and growing** | 🔴 | ≤24hr | ⚠️ **Partial** (Aug 8 dump) | ⚠️ |
+
+**RPO/RTO Assessment: FAIL for Critical tier** — Gitea, Traccar, and UISP are planned for ≤1hr RPO but receive only daily backups. The plan itself overstates capabilities.
+
+---
+
+## 5. Restore Testing
+
+### Evidence of Verified Restores: **NONE**
+
+| Source | What it says | Evidence of execution |
+|--------|-------------|----------------------|
+| `backup-plan.md` §Restore Testing Cadence | "Quarterly: Pick one random backup per tier, restore to staging, verify integrity" | 🔴 No test logs anywhere |
+| `backup-plan.md` | "Annual: Full DR simulation" | 🔴 Never executed |
+| `4-DR-Testing-Schedule.md` | "Every restore test gets logged with date, tester, duration, findings" | 🔴 No log file exists |
+| `backup-policy.md` | "Monthly restore test of a randomly selected asset" | 🔴 No evidence |
+| `3-Per-Server-Runbooks.md` | Detailed per-server restore procedures | ✅ Procedures exist on paper only |
+| `dr-issue-log.md` | Historical DR audit issues | ✅ Issues tracked; no restore tests documented |
+| `/root/.hermes/references/` | Search for "restore test", "verified restore", "DR simulation" | 🔴 Zero actual test results found |
+
+**Conclusion:** Restore procedures are well-documented on paper, but **no backup has ever been restore-tested**. There are no test logs, no verification records, and no evidence that any of the 30+ backup targets can actually be restored successfully.
+
+**Restore test cadence last executed:** NEVER
+
+---
+
+## 6. Monitoring & Alerting
+
+### What exists:
+
+| Check | Mechanism | Status |
+|-------|-----------|--------|
+| Hermes full backup age check | `backup-audit-check.sh` (system cron 2AM) | ✅ Runs daily, checks if last backup <36hr old |
+| Cron output logging | All backup cron jobs pipe to `logger` | ✅ Output captured in syslog |
+| Hermes cron job status | `hermes cron list` shows last run status | ✅ Most show "ok" |
+| Uptime Kuma monitoring | External monitoring of services | ✅ |
+| Ops data collector | `ops-data-collector.py` (every 5min) | ✅ Collects metrics |
+
+### What's MISSING:
+
+| Gap | Impact | Urgency |
+|-----|--------|---------|
+| **No alert on backup failure** | When `backup-audit-check.sh` fails, output goes to `logger` only. Nobody gets notified. | **Critical** |
+| **No alert on 0-byte backup** | Several backups produce tiny config files (240 bytes) that could silently become 0 bytes | **High** |
+| **No alert on backup not running** | Hermes cron jobs silently enter "error" state (home-router-daily-backup has been error for days) | **Critical** |
+| **No cross-server backup health dashboard** | Must SSH to each server individually to check backup status | **High** |
+| **No S3 file integrity validation** | Nobody checks if backup files are actually valid (corrupt tar, truncated SQL) | **High** |
+| **Core crontab doesn't monitor remote server backup outcomes** | Core only checks its own full backup — app1/app2/app3 backup failures go unnoticed | **Critical** |
+| **No automatic ticket/issue creation** | Backup failures should auto-create a GitHub issue or Hudu ticket | Medium |
+
+**The `backup-audit-check.sh` is the only monitoring, and it only checks Hermes full backup freshness. Everything else is silent.**
+
+### Cron Job Statuses:
+
+| Job | Schedule | Latest Status | Issues |
+|-----|----------|---------------|--------|
+| hermes-backup (system cron) | 1 AM | ✅ Ran today | |
+| backup-audit-check (system cron) | 2 AM | ✅ OK (20h ago) | |
+| root-essentials-backup (system cron) | 3 AM | ✅ S3 files present | |
+| core-services-backup (system cron) | 1:30 AM | ✅ S3 files present | Prometheus part empty |
+| wphost02-backup (system cron via SSH) | 5 AM | ✅ S3 files present | |
+| vaultwarden-backup (Hermes cron) | 2:30 AM | ✅ ok | |
+| litellm-backup (Hermes cron) | 3:30 AM | ✅ ok | |
+| komodo-backup (Hermes cron) | 3:45 AM | ✅ ok | |
+| docuseal-backup (Hermes cron) | 4:00 AM | ✅ ok | |
+| twenty-backup (Hermes cron) | 4:15 AM | ✅ ok | |
+| auth-api-backup (Hermes cron) | 3:15 AM | ✅ ok | |
+| stack-auth-backup (Hermes cron) | 3:15 AM | ✅ ok | |
+| hexclave-backup (Hermes cron) | 3:30 AM | ✅ ok | |
+| technitium-backup (Hermes cron) | 2:45 AM | ✅ ok | |
+| dawarich-backup (Hermes cron) | 4:00 AM | ✅ ok | |
+| ragflow-backup (Hermes cron) | 4:15 AM | ✅ ok | |
+| hudu-backup (Hermes cron) | 7:00 AM | ✅ ok | |
+| gitea-backup (Hermes cron) | 8:00 AM | ✅ ok | |
+| unms-backup-sync (Hermes cron) | 6:00 AM | ✅ ok | |
+| unifi-backup-sync (Hermes cron) | 2:00 AM | ✅ ok | |
+| hetzner-weekly-snapshots (Hermes cron) | Mon 5 AM | ✅ ok | |
+| home-router-daily-backup (Hermes cron) | 6:00 AM | 🔴 **error** | 5/6 towers timeout daily |
+| **Gitea ModelOrtho Backup** (Hermes cron) | 4:30 AM | 🔴 **NEVER RAN** | No Last run field |
+| app1 `/root/backup.sh` (app1 crontab) | 2:00 AM | ❓ Not verified | S3 files present but openwebui sizes suspicious |
+| app2 `/root/backup.sh` (app2 crontab) | 2:30 AM | ❓ Not verified | S3 files present |
+| app3 `/root/backup.sh` (app3 crontab) | 3:00 AM | ⚠️ MySQL failed | **MySQL dumps stopped Aug 8; nginx failed Aug 10** |
+| app3 `/opt/backup-restore/snapshot.sh` (app3 crontab) | 1AM/1PM | ❓ Not verified | Local-only |
+
+---
+
+## 7. Industry Standard Gaps
+
+### What we're doing right:
+- ✅ **Comprehensive backup plan** — well-documented in backup-plan.md with 27 declared targets
+- ✅ **24+ Hermes cron jobs** operating backup scripts — good automation
+- ✅ **Daily cadence** for all critical services
+- ✅ **Wasabi S3** as immutable off-site storage (11-nines durability)
+- ✅ **Provider diversity** — Core on netcup, standby on Hetzner
+- ✅ **Warm standby** for Hermes (app1-bu with 7-min failover)
+- ✅ **DR runbooks** documented per server
+- ✅ **DR issue log** maintained with root cause analysis
+- ✅ **Hetzner weekly snapshots** as additional layer
+
+### What's missing (every competent shop has these):
+
+| Gap | Severity | Industry Expectation |
+|-----|----------|---------------------|
+| **Restore testing** | Critical | Monthly restore tests are table stakes. "Untested backups are not backups." |
+| **Backup failure alerting** | Critical | PagerDuty/OpsGenie/Telegram alert on ANY backup job failure |
+| **Second media type** | High | NAS, tape, or second cloud provider for 3-2-1 compliance |
+| **Immutable backups** | High | S3 Object Lock to prevent ransomware deletion |
+| **Backup integrity validation** | High | Automated restore-and-verify pipeline (spin up temp container, restore DB, run queries) |
+| **Service discovery for backups** | High | Automated scan that identifies running services and flags unbacked ones |
+| **Backup retention policy** | Medium | Defined retention periods per tier (only gitea-modelortho and wphost02 have cleanup) |
+| **Encryption at rest documentation** | Medium | Explicit documentation of which backups are encrypted |
+| **Database consistency** | Medium | PostgreSQL/MongoDB backups should use pg_dump with consistent snapshots, not just file copies |
+| **Runbook testing** | Medium | DR runbooks should be exercised, not just written |
+| **Cross-region replication** | Low | S3 cross-region replication for geographic diversity |
+| **Automated documentation** | Low | Backup inventory auto-generated from running config, not manually maintained |
+
+### Single highest-risk gap:
+
+**Zero restore testing.** Every script could produce corrupt archives and nobody would know until a real disaster. The backup plan itself states quarterly restore testing is required — it has never been done.
+
+---
+
+## 8. Risk Matrix
+
+| # | Risk | Likelihood | Impact | Urgency | 
+|---|------|-----------|--------|---------|
+| R1 | app3 MySQL has no backup since Aug 8 — 9 production databases at risk | **Certain** (already happening) | **Critical** (customer-facing WP sites, MainWP) | **IMMEDIATE** |
+| R2 | git.modelortho.com Gitea cron never ran — Anita's repositories have no automated backup | **Certain** (cron misconfigured) | **High** (all Anita's repos and issues lost if disk fails) | **IMMEDIATE** |
+| R3 | Prometheus has never been backed up — all monitoring history gone on disk failure | **Certain** (never configured) | **High** (lose all metrics, alerts, dashboards) | **Within 48hr** |
+| R4 | Buzz/Bookstack/Support-API/etc. (10+ services) with zero backup | **Medium** (disk failure) | **High** (complete data loss for those services) | **Within 1 week** |
+| R5 | No restore testing means all backups are unverified | **High** (silent corruption) | **Critical** (backups useless in real DR) | **Within 2 weeks** |
+| R6 | Backup failure goes unnoticed — no alerting pipeline | **High** (cron errors daily) | **High** (data loss accumulates silently) | **Within 1 week** |
+| R7 | MikroTik tower CCRs not backed up since Jul 7 | **Medium** (routers stable) | **Medium** (reconfig from scratch) | **Within 2 weeks** |
+| R8 | app3 nginx configs backup failing since Aug 10 | **Certain** (observed today) | **Medium** (can rebuild, but slower) | **Within 1 week** |
+| R9 | Wazuh backup undocumented and unanchored — could be lost in migration | **Low** | **Medium** (security event history lost) | **Within 1 month** |
+| R10 | Open WebUI backup capturing potentially stale Docker data | **Medium** (bug in docker cp caching) | **Medium** (some chat history lost) | **Within 1 month** |
+| R11 | No second media type — single S3 provider is single point of failure | **Very Low** (Wasabi 11-nines) | **Critical** (if Wasabi has outage/dataloss) | **Within 3 months** |
+
+---
+
+## 9. Recommendations
+
+### Priority 0 — FIX TODAY
+
+| # | Action | Effort | Risk Addressed |
+|---|--------|--------|---------------|
+| 1 | **Fix app3 MySQL backup** — diagnose why mysqldump stopped after Aug 8 (likely MySQL auth, disk full, or script logic change). Re-run manually for Aug 9–10. | 30 min | R1 |
+| 2 | **Fire gitea-modelortho-backup cron** — the script exists and is correct. The cron has no Last run. Run it manually now, verify S3, then fix the cron schedule. | 15 min | R2 |
+| 3 | **Verify app3 Nginx config backup** — failed on Aug 10 per cron log. Diagnose and re-run. | 15 min | R8 |
+
+### Priority 1 — THIS WEEK
+
+| # | Action | Effort | Risk Addressed |
+|---|--------|--------|---------------|
+| 4 | **Add Prometheus backup** — add a simple `tar czf` of TSDB to `core-services-backup.sh` or create standalone script. The data path exists, just needs to be included. | 30 min | R3 |
+| 5 | **Build backup alerting** — pipe `backup-audit-check.sh` output to Telegram/email. Add S3 file size check (flag 0-byte or <1KB files). | 2 hr | R6 |
+| 6 | **Create backups for 10+ unbacked services** — prioritize: buzz-prod-postgres, bookstack, support-api, timetrex. Create individual backup scripts. | 4 hr | R4 |
+| 7 | **Restore test: 1 backup** — pick any one backup (suggest: gitea/app2), restore to a temp location, verify data integrity. Document results. This proves the concept. | 2 hr | R5 |
+| 8 | **Update backup plan** — add Wazuh, fix modelortho script name (it's `gitea-modelortho-backup.sh` not `modelortho-backup.sh`), document that app servers run local cron not Core-orchestrated SSH. | 1 hr | Documentation |
+
+### Priority 2 — THIS MONTH
+
+| # | Action | Effort | Risk Addressed |
+|---|--------|--------|---------------|
+| 9 | **Fix MikroTik tower backups** — diagnose SSH timeout to 5 tower CCRs (VPN issue? IP change?). Get at least a weekly config export. | 2 hr | R7 |
+| 10 | **Monthly restore test schedule** — set up recurring calendar reminder. Test one random backup per month. | 30 min setup | R5 |
+| 11 | **Add S3 backup integrity checks** — for each service, after upload, re-download tarball and verify tar integrity or SQL dump validity. | 3 hr | R5 |
+| 12 | **Investigate Open WebUI backup size freeze** — check if docker cp is caching stale data; consider pg_dump approach instead. | 2 hr | R10 |
+| 13 | **Add 14-day retention to all backup scripts** — most scripts have no cleanup; S3 costs accrue indefinitely. | 2 hr | Cost |
+
+### Priority 3 — THIS QUARTER
+
+| # | Action | Effort | Risk Addressed |
+|---|--------|--------|---------------|
+| 14 | **Add S3 Object Lock** — enable compliance mode on Wasabi buckets to prevent ransomware deletion. | 1 hr | R11 |
+| 15 | **Second media type** — add local NAS backup for critical services, or replicate to a second cloud provider (Backblaze B2). | 8 hr | R11 |
+| 16 | **Full DR simulation** — schedule and execute a complete failover-to-standby exercise per the backup plan's annual requirement. | 8 hr | R5 |
+| 17 | **Automated service discovery** — script that scans all servers for running services and cross-references against backup coverage. | 3 hr | R4 |
+| 18 | **Cross-server backup health dashboard** — Uptime Kuma or Grafana dashboard showing last backup time and status for every service. | 3 hr | R6 |
+
+---
+
+## Appendix A: Script Name Discrepancies
+
+The backup plan lists script names that don't match reality:
+
+| Plan Says | Actual Name | Location |
+|-----------|-------------|----------|
+| `modelortho-backup.sh` | `gitea-modelortho-backup.sh` | `/root/.hermes/scripts/` on Core |
+| `app1-backup.sh` (runs from Core via SSH) | `/root/backup.sh` (runs locally on app1) | app1 crontab |
+| `app2-backup.sh` (runs from Core via SSH) | `/root/backup.sh` (runs locally on app2) | app2 crontab |
+| `app3-backup.sh` (runs from Core via SSH) | `/root/backup.sh` (runs locally on app3) | app3 crontab |
+| `wphost02-backup.sh` (runs from Core) | `ssh root@5.161.62.38 '/root/backup.sh'` (Core crontab invokes wphost02's script) | Both exist |
+
+**Architecture reality:** The backup plan describes a Core-orchestrated model where Core SSHs to each server and runs backup scripts. In reality, each app server has its own local cron job that runs `/root/backup.sh` independently. Core's system crontab only directly backs up: Hermes, Core services, root essentials, and wphost02 (via SSH). All other app backups are orchestrated by Hermes cron jobs that SSH from Core (vaultwarden, litellm, komodo, etc.) or by local crontabs on the app servers themselves.
+
+## Appendix B: S3 Bucket Health
+
+| Bucket | Status | Notes |
+|--------|--------|-------|
+| hermes-vps-backups | ✅ Active | Primary backup bucket. 33 prefixes, daily uploads |
+| mikrotik-ccr-backups | ⚠️ Partial | Home gateway configs OK; tower CCR configs missing since Jul 7 |
+
+## Appendix C: Hermes Cron Job Summary
+
+| Total Hermes cron jobs | 28 |
+|------------------------|-----|
+| Backup-related jobs | 17 |
+| Jobs with "ok" status | 23 |
+| Jobs with "error" status | 2 (home-router-daily-backup, exotic-vehicle-scout) |
+| Jobs with NO `Last run` (never executed) | 1 (Gitea ModelOrtho Backup) |
+| Jobs running in "no_agent" mode | 25 |

backup-plan.md

@@ -1,6 +1,6 @@
 # ITPP Backup Plan
 
-> **Last updated:** 2026-08-08
+> **Last updated:** 2026-09-11
 > **Scope:** All ITPP infrastructure backups — Hermes core, application servers, routers, and external services.
 > **Storage:** Wasabi S3 (us-east-1) via `--endpoint-url https://s3.us-east-1.wasabisys.com`
 > **Cron backend:** Hybrid — system crontab + Hermes cron jobs (both on Core)
@@ -63,11 +63,16 @@
 | Hexclave (Stack Auth) | `hexclave-backup.sh` — PG dump + compose + env | `s3://hermes-vps-backups/app3/hexclave/` | 3:30 AM | 2026-08-08 |
 | modelortho.com | `modelortho-backup.sh` — htdocs tar.gz + nginx configs | `s3://hermes-vps-backups/app3/modelortho/` | 4:30 AM | 2026-08-08 |
 
-### wphost02 (5.161.62.38) — Hetzner CPX21
+### wphost02 (5.161.62.38) — Hetzner CPX21 — **DECOMMISSIONED (2026-08-28)**
+
+wphost02 was deleted from the Hetzner account on 2026-08-28. Its backup job has been removed; sites now live on app3 (covered under the App3 backup section above).
+
+### anita-mnz (159.195.16.30) - netcup (Anita's dedicated box, cut over 2026-09-11)
 
 | Service | Method | Destination | Schedule | Last Verified |
 |---------|--------|-------------|----------|---------------|
-| WordPress (7 sites) | `ssh → /root/backup.sh` — per-site tar.gz + all DBs | `s3://hermes-vps-backups/wphost02-backup/` | 5:00 AM | 2026-07-28 |
+| /root Essentials | `root-essentials-backup.sh` - dotfiles, profile, keys, scripts (**excludes `*.db` by design**) | `s3://hermes-vps-backups/root-backup/anita-mnz/` | 3:00 AM | 2026-09-11 |
+| Hermes Profile DBs | `hermes-db-backup.sh` - `sqlite3 .backup` snapshots of state.db + cron + wisdom + notepad + verification DBs, `quick_check` per snapshot, upload then download-and-verify | `s3://hermes-vps-backups/root-backup/anita-mnz/db/` | 3:10 AM | 2026-09-11 (restore test passed: quick_check ok, 99,285 messages) |
 
 ### Home Router
 
@@ -94,6 +99,8 @@
 | 2:30 AM | Vaultwarden (App1) + Traccar (App2) | Hermes cron / crontab | `vaultwarden-backup.sh`, `app2-backup.sh` |
 | 2:45 AM | Technitium DNS (App2) | Hermes cron | `technitium-backup.sh` |
 | 3:00 AM | /root essentials + App3 (CloudPanel, MySQL, WP, Nginx) | crontab | `root-essentials-backup.sh`, `app3-backup.sh` |
+| 3:00 AM | **anita-mnz** /root essentials | Anita crontab | `root-essentials-backup.sh` |
+| 3:10 AM | **anita-mnz** Hermes profile DB snapshots | Anita crontab | `hermes-db-backup.sh` |
 | 3:15 AM | Auth API (Core) | Hermes cron | `auth-api-backup.sh` |
 | 3:30 AM | Hexclave / Stack Auth (App3) + LiteLLM (App1) | Hermes cron / crontab | `hexclave-backup.sh`, `litellm-backup.sh` |
 | 3:45 AM | Komodo (App1) | crontab | `komodo-backup.sh` |
@@ -101,7 +108,6 @@
 | 4:00 AM | DocuSeal (App1) | Hermes cron | `docuseal-backup.sh` |
 | 4:15 AM | **RAGFlow (App2)** | Hermes cron | `ragflow-backup.sh` |
 | 4:15 AM | Twenty CRM (App1) | Hermes cron | `twenty-backup.sh` |
-| 5:00 AM | wphost02 WordPress (SSH) | crontab | `wphost02-backup` |
 | 6:00 AM | MikroTik CCR + UNMS sync | Hermes cron | `run-wisp-backup.sh`, `unms-backup-sync.sh` |
 | 1 AM / 1 PM | WordPress per-site snapshots (App3, local) | App3 crontab | `/opt/backup-restore/snapshot.sh` |
 | 7:00 AM | Hudu (App2) | Hermes cron | `hudu-backup.sh` |
@@ -133,7 +139,6 @@
 | `hexclave-backup.sh` | Stack Auth PG dump + compose + env (SSH to App3) | Hermes cron 3:30 AM |
 | `dawarich-backup.sh` | Dawarich PostgreSQL dump + compose + env (SSH to App2) | Hermes cron 4:00 AM |
 | `ragflow-backup.sh` | RAGFlow MySQL dump + compose + env (SSH to App2) | Hermes cron 4:15 AM |
-| `wphost02-backup.sh` | SSH to wphost02, backup all sites | crontab 5:00 AM |
 | `run-wisp-backup.sh` | MikroTik CCR config export | Hermes cron 6:00 AM |
 | `unms-backup-sync.sh` | UNMS auto-backup sync to S3 | Hermes cron 6:00 AM |
 | `unifi-backup-sync.sh` | UniFi auto-backup sync to S3 | Hermes cron 2:00 AM |
@@ -147,11 +152,12 @@
 |--------|---------|------|
 | `snapshot.sh` | Per-site WordPress tarball + DB | App3 crontab 6 AM / 6 PM |
 
-### On wphost02 (`/root/`)
+### On anita-mnz (`/root/`)
 
 | Script | Purpose | Runs |
 |--------|---------|------|
-| `backup.sh` | All 7 WordPress sites + all MySQL DBs | Triggered via SSH from Core |
+| `root-essentials-backup.sh` | /root essentials (profile, keys, scripts, configs) | crontab 3:00 AM |
+| `hermes-db-backup.sh` | `sqlite3 .backup` snapshots of Hermes profile DBs + per-snapshot quick_check + restore test | crontab 3:10 AM |
 
 ---
 
@@ -161,8 +167,11 @@
 hermes-vps-backups/
 ├── hermes-full-backup/     — Full Hermes daily (tar.gz)
 ├── live/                   — Hermes live sync (every 15 min)
+├── decommissioned/         — archived copies of decommissioned systems (profile archives)
 ├── live-sync/              — Old sync format (deprecated)
-├── root-backup/            — /root essentials
+├── root-backup/            — /root essentials (Core uses the flat legacy prefix)
+│   └── <host>/             — per-host prefix (`anita-mnz`, ...)
+│       └── db/             — SQLite `.backup` snapshots (`hermes-db-backup.sh`)
 ├── standby/                — Standby configs + recovery bundle
 ├── core/
 │   ├── grafana/            — Grafana SQLite DB
@@ -192,7 +201,6 @@ hermes-vps-backups/
 ├── gitea/daily/            — Gitea repos + DB
 ├── unms-backups/live/      — UNMS auto-backups
 ├── unifi-backups/          — UniFi controller backups
-├── wphost02-backup/        — wphost02 per-site + DB
 ├── volumes/                — Docker volume dumps
 ├── caddy/                  — (unused)
 ├── assets/                 — Static assets
@@ -233,12 +241,27 @@ These paths contain data from services that migrated off Core (Jul 28, 2026) or
 | `core/komodo/` | Stale 11 days | **Safe to delete** — Komodo migrated to App1; new backups at `app1/komodo/` |
 | `caddy/` | Unused | **Safe to delete** — Never populated |
 | `snapshots/` | Unused | **Safe to delete** — Never populated |
+| `live/profiles/anita/` | Purged 2026-09-11 | **Deleted** — 20,551 objects / 5.0 GB of the frozen Core copy of Anita's profile, pushed by the 15-minute sync before it was paused Sep 3. Superseded by `decommissioned/anita-core-frozen-profile-20260911-1732.tar.gz` |
 
 ## Unbacked Services
 
 These services are running in production with **zero backup coverage**:
 | Unbacked | *(none)* | N/A | N/A | All services are backed up as of 2026-08-08 |
 
+**Done 2026-09-11 (Anita migration closeout).** The frozen Core copy `/root/.hermes/profiles/anita` (8.0 GB, excluded from live sync after the 15:53 cutover) was archived to
+`s3://hermes-vps-backups/decommissioned/anita-core-frozen-profile-20260911-1732.tar.gz` — 2,786,082,561 B, sha256 `b457fa7f…b3dc`, 21,206 entries — then deleted, reclaiming 8 GB (125 GB → 117 GB used).
+The archive was proven by downloading the object back and matching the sha256 against the local tarball **before** anything was removed. The 8 GB was almost entirely quarantine
+artifacts from the Sep 3/9/10 SQLite corruption events (`corrupt-20260903/09/10`, `pre-rebuild-20260910`, `recovered-20260910`); the live profile had already been proven migrated
+(memories, `.env` and the six cron job IDs identical on the new box, which carries one more skill and two more messages).
+
+**Gap found and closed 2026-09-11 (anita-mnz):** `root-essentials-backup.sh` correctly excludes `*.db` so it never archives a
+live SQLite file, but nothing replaced it, so after the box was provisioned Anita's conversation store (974 MB), the cron
+execution DB, notepad, wisdom and verification DBs had **zero** backup coverage. Core has `hermes-backup.sh` doing this job;
+the migrated box had no equivalent. `hermes-db-backup.sh` (3:10 AM) now snapshots each DB with the sqlite3 `.backup` API,
+quick_checks every snapshot, uploads one dated tarball, then downloads it back and verifies the restored store. Verified
+2026-09-11 17:04: restore test passed, `quick_check=ok`, 99,285 messages. Any box migrated with this script set needs the
+same check, because the `*.db` exclusion is silent.
+
 > **6 previously-unbacked services now backed up (Auth API, Technitium DNS, Dawarich, RAGFlow, Stack Auth/Hexclave, app3 static sites).** The earlier count of "14" was an error — no source document supports that number. The actual delta since July 28 is 6.
 
 ---
@@ -266,4 +289,4 @@ All the following services were migrated from Core to App1 in a single session:
   - Auto-restores from `s3://hermes-vps-backups/hermes-full-backup/`
   - Provider diversity: netcup outage won't kill both Core and standby
 - **Recovery priority:** Hermes first → infrastructure monitoring (Uptime Kuma, Grafana) → App1 services → App2 services
-- **24 backup scripts** on Core, 1 on App3, 1 on wphost02
+- **Backup scripts:** 24 scripts on Core, 1 on App3 (wphost02 DECOMMISSIONED 2026-08-28; its backup script and S3 path were removed)

clients/ai-biz-dev/competitive-landscape.md

@@ -0,0 +1,592 @@
+# AI Business Development Platform — Competitive Landscape
+
+**Prepared for:** IT Pro Partner (itpropartner.com)  
+**Date:** August 2026  
+**Purpose:** Market intelligence for "Idea to Income" AI business development platform  
+**Target ICP:** Main street businesses, first-time founders, career-changers — NOT VC-backed tech startups  
+
+---
+
+## Executive Summary
+
+The market for AI-powered business launch tools is fragmented across five distinct segments, with **no single competitor offering the full 5-phase "idea to income" arc**. This fragmentation creates a clear integration opportunity — the market is currently forcing founders to stitch together 5–8 separate tools to go from concept to operational business.
+
+**Key findings:**
+
+| Segment | Maturity | # of Competitors | Integration Gap |
+|---------|----------|-----------------|-----------------|
+| 1. Business Plan Generators | Mature | 5+ viable | No entity formation integration |
+| 2. AI Idea Validation | Emerging (2024+) | 8+ tools | Standalone; no operational follow-through |
+| 3. Entity Formation | Mature | 6+ major players | No pre-formation planning; upsell-heavy |
+| 4. Banking/Ops Setup | Mature | 5+ platforms | Disconnected from formation; siloed |
+| 5. AI Business Agents | Nascent | 10+ scattered | Narrow point solutions; no full arc |
+
+**Primary opportunity:** Own the full "Idea → Business Plan → Review → Entity → Operations" pipeline. No competitor covers all five phases. The nearest contenders (Tailor Brands, doola, Collective) cover at most 2–3 phases.
+
+**Threat level:** Medium. Any of the existing entity formation platforms (LegalZoom, ZenBusiness, Tailor Brands) could expand backward into planning or forward into ops. They have user bases, brand recognition, and capital. Speed to market and operational credibility (IT Pro Partner's differentiator) are critical.
+
+---
+
+## Segment 1: Business Plan Generators
+
+### Competitive Overview
+
+| Company | Entry Price | Premium Price | AI Features | Entity Integration | Target |
+|---------|------------|---------------|-------------|-------------------|--------|
+| **LivePlan** | $15/mo (annual) / $349 lifetime | $18–20/mo | No native AI | No | Accountants, analysts, planners |
+| **Upmetrics** | $7/mo (annual) | $37/mo Pro + $185/mo white-label | Yes — AI writing, industry research, pitch deck | No | Entrepreneurs, consultants |
+| **BizPlan** | $29/mo or $249/yr or $349 lifetime | Same features | No explicit AI | No (but Fundable crowdfunding) | Lean startups, milestone trackers |
+| **Enloop** | **DEFUNCT** | N/A | No | No | N/A — platform inactive |
+| **IdeaBuddy** | Free / $9–22/mo (annual) | $22–45/mo | Yes — AI plan generation, guided plan | No | First-time entrepreneurs |
+
+### Detailed Profiles
+
+#### LivePlan (Palo Alto Software)
+- **Pitch:** Structured business planning with deep financial forecasting
+- **Strengths:** Industry-standard financial modeling, 500+ sample plans, educational content, strong for accountants/consultants
+- **Weaknesses:** SLOW browser performance (Chrome-only recommended), no AI features, limited integrations, criticized on Reddit for $300/yr cost with multi-client use
+- **Pricing breakdown:**
+  - Standard: $15/mo annual ($18/mo semi-annual, $20/mo monthly)
+  - Premium: higher tier with additional features
+  - Lifetime: $349 one-time
+- **Ratings:** 4.5/5 Software Advice (144 five-star reviews)
+- **Gap for our platform:** No AI, no entity formation path, no operational setup. Pure planning tool — dead end after plan is written.
+
+#### Upmetrics
+- **Pitch:** AI-powered business planning with modern tooling
+- **Strengths:** Strongest AI feature set among plan generators — AI writing assistance, AI industry research reports, AI pitch deck builder, 7-year financial forecasts, 400+ sample plans
+- **Weaknesses:** Some reviewers find $20/mo expensive on a budget; workspace limits on lower tiers
+- **Pricing breakdown:**
+  - Starter: $7/mo annual (1 workspace, 5 team members)
+  - Premium: $14/mo annual ($19/mo monthly) — 1 workspace
+  - Professional: $37/mo annual ($49/mo monthly) — multi-client
+  - White-label add-on: $185/mo (billed $249/mo annually)
+- **15-day money-back guarantee**
+- **Gap for our platform:** AI writing is strong but still just a plan generator. No entity formation, no banking/payroll setup. Plan exists in isolation.
+
+#### BizPlan (Startups.com ecosystem)
+- **Pitch:** Business planning bundled with education, mentorship (Clarity.fm), and crowdfunding (Fundable — $500M+ raised)
+- **Strengths:** Unique Fundable integration for capital raising, lifetime plan value ($349), community + education ecosystem
+- **Weaknesses:** No AI features surfaced, higher monthly price ($29/mo) than Upmetrics/LivePlan entry tiers
+- **Pricing:** $29/mo, $249/year (~$20.75/mo), $349 lifetime
+- **Gap for our platform:** Education/community approach is complementary, not competitive. Lacks AI, entity formation, and ops integration.
+
+#### Enloop — DEFUNCT
+- **Status:** Website displays "undergoing a service upgrade." No sign-ups or logins functional.
+- **Historical:** Was priced higher than Upmetrics for fewer features, no AI, no flexibility in structure/layout. Minimal customer support. Not a viable competitor.
+- **Note:** Include as historical footnote — the first wave of automated plan generators is aging out without AI.
+
+#### IdeaBuddy
+- **Pitch:** All-in-one business planning for first-time entrepreneurs, powered by AI
+- **Strengths:** "Idea Plan" concept sketching, "Business Guide" step-by-step with examples, guided plan feature for beginners
+- **Weaknesses:** Less depth in financial modeling vs LivePlan; positioned more as a learning/ideation tool than professional plan builder
+- **Pricing (from website):**
+  - Free plan: basic features
+  - Dreamer: ~$9/mo annual
+  - Founder: ~$22/mo annual
+  - Pro (highest): ~$45/mo annual
+  - 15-day money-back guarantee
+- **Target:** First-time entrepreneurs, idea-stage founders
+- **Gap for our platform:** Closest to our Phase 1 (Discovery/Ideation) but stops at planning. No entity formation, no operations.
+
+### Segment 1 Gap Analysis
+
+| Gap | Opportunity |
+|-----|-------------|
+| **AI writing is present but shallow** | Upmetrics and IdeaBuddy have AI, but it's prompt-based generation — no multi-agent review, no strategic critique |
+| **Financials are templates, not intelligence** | All tools fill templates with user-provided numbers. None validate assumptions against market data |
+| **Plans are dead-end documents** | No competitor connects the plan to actual entity formation or banking setup |
+| **No "Critical Review" phase** | No tool offers an adversarial AI review of your business plan before you commit |
+| **White-label ecosystem is thin** | Only Upmetrics offers white-label ($185/mo), and it's expensive |
+
+---
+
+## Segment 2: AI Idea Validation & Business Advisors
+
+### Competitive Overview
+
+| Company | Method | Entry Price | Paid Price | Key Output |
+|---------|--------|------------|------------|------------|
+| **IdeaProof** | Multi-model AI (Claude + GPT-4 + Gemini) | 90 free credits | €10–70 credit packs | Validation score, TAM/SAM/SOM, competitor map, brand kit, marketing suite |
+| **ValidatorAI** | GPT-powered analysis | Free (unlimited basic) | Pro (unlisted) | Quick AI critique, market potential assessment |
+| **DimeADozen** | Multi-agent research pipeline | Free (Solo score) | $59/report; Enterprise custom | 40+ page investor report with comp-set data from S-1 filings, retention curves |
+| **Preuve AI** | Demand signal + competitor mapping | Not listed | Not listed | Real competitor mapping with pricing, demand signals, blind spots, 3 pivot directions |
+| **VenturusAI** | AI analysis with frameworks | Free start | Not listed publicly | SWOT, PESTEL, Porter's Five Forces, target audience, marketing strategy |
+| **Trend Seeker** | Demand-based (semantic search) | Free | $9.99/mo Pro | Real user requests from Reddit/communities, demand evidence links |
+| **FounderPal** | AI idea validator + marketing toolkit | Free validator | $69/strategy; $199 lifetime | Side-by-side idea comparison, marketing strategy generator, buyer persona |
+| **ChatGPT / Claude** | General AI assistants | Free (ChatGPT) / $20/mo (Plus) | $20/mo+ | DIY: prompt engineer your own validation |
+| **Sintra AI (Buddy)** | AI business coach agent | Not listed | Not listed | Weekly coaching cadence, blind-spot surfacing, goal-to-action translation |
+| **Bizway** | AI agent team for solopreneurs | $19/mo (3 agents) | $49–129/mo | Lead gen, outreach, scheduling, analytics, content |
+
+### Detailed Profiles
+
+#### IdeaProof
+- **Pitch:** Validate your startup idea in 120 seconds with tri-model AI
+- **Strengths:** Unique multi-model approach (Claude, GPT-4, Gemini in parallel, cross-validated — claims 89% accuracy vs 70–75% single-model), TAM/SAM/SOM calculations, investor-ready business plans, brand strategy generation, logo design, ad creative suite for 6+ platforms, email sequences
+- **Weaknesses:** Credit-based pricing gets expensive for full suite (490+ credits for everything), market-size figures can be speculative, AI-generated brand assets may feel generic
+- **Pricing:** 90 free credits; Starter €10 (150 credits); Builder €50 (700 credits); Founder €70 (1,500 credits) — 30-day money-back guarantee
+- **Trusted by:** 10,000+ verified entrepreneurs worldwide
+- **Gap for our platform:** Strong validation tool but completely standalone. No entity formation follow-through. Generates a plan then leaves you to figure out "what next."
+
+#### DimeADozen
+- **Pitch:** Investor-style 40+ page business reports in under 20 seconds
+- **Strengths:** Sources real data from S-1 filings (retention curves, named comparables), unit-economics analysis, risk factors, execution requirements — more "investment committee" than "AI validator"
+- **Weaknesses:** Vague prompts produce generic outputs; underperforms on niche markets; financial projections require manual adjustment; described as "speculative rather than predictive"
+- **Stats:** 120,000+ reports for 85,000+ entrepreneurs; featured in CNBC Make It
+- **Pricing:** Solo free; Entrepreneur $59/report (full report, commercial use rights); Enterprise with white-label + API
+- **Gap for our platform:** Deepest analytical report, but it's a document, not a platform. No entity formation, no operations.
+
+#### ValidatorAI
+- **Pitch:** Quick, free AI assessment of your business idea
+- **Strengths:** Free (unlimited basic), simple UX, live founder-behavior data from hundreds of thousands of sessions
+- **Weaknesses:** AI-hallucinated market size figures (unverifiable), encourages rather than critiques (false confidence risk), no real-world demand evidence — just AI opinion
+- **Pricing:** Free public validator; Pro pricing not listed
+- **Gap for our platform:** Demonstrates market demand for quick validation — but validates with vibes, not data.
+
+#### AI Coaching / Advisor Tools (Sintra AI, Bizway, SideCoach)
+- **Sintra AI (Buddy):** #1 on "10 Best AI Business Coach Agents 2026" lists. Weekly coaching cadence. Surfaces blind spots. Translates goals to next steps.
+- **Bizway:** AI agents for solopreneurs — lead gen, outreach, scheduling, analytics. $19/mo (3 agents) to $129/mo (unlimited agents, 1,500 credits/mo). Positioned as "AI team for solo business."
+- **SideCoach:** AI vs human coaching comparison — AI at $29–99/mo vs human at $800–2,000+/mo. Strong ROI argument for AI coaching.
+- **Human coaching cost benchmark:** $200–500+/hour, $800–2,000+/monthly retainer
+- **Gap for our platform:** These are ongoing advisory relationships, not launch-phase tools. Complementary, not competitive — a retention feature, not an acquisition feature.
+
+### Segment 2 Gap Analysis
+
+| Gap | Opportunity |
+|-----|-------------|
+| **Validation tools don't connect to execution** | IdeaProof and DimeADozen stop at the report — no path to entity formation |
+| **AI coaching is ongoing, not launch-focused** | Bizway, Sintra AI are for running businesses, not starting them |
+| **No adversarial/multi-perspective review** | All tools use a single AI perspective. None offer a "red team" critique of your business plan |
+| **Human coaching is expensive** | $200–500/hr creates a price umbrella for AI-powered guidance at $20–50/mo |
+
+---
+
+## Segment 3: Entity Formation Services
+
+### Competitive Overview
+
+| Company | Entry Price | Premium | RA Cost (Yr 1/Yr 2+) | AI Features | Beyond Formation? |
+|---------|------------|---------|----------------------|-------------|-------------------|
+| **LegalZoom** | $0 + state fees (Basic) | $249–349 + state fees | Separate add-on (~$249/yr) | AI on website, not core | Legal plans, trademarks, contracts |
+| **ZenBusiness** | $0 + state fees (Starter) | $199–399/yr + state fees | Included in Premium only | LLM website builder (30-day trial) | Compliance, domain, website |
+| **Bizee (Incfile)** | $0 + state fees (Basic) | $199–299 one-time + state fees | Free yr1 / $119–149/yr | No | Business contracts, EIN, compliance |
+| **Tailor Brands** | $0 + state fees (Lite) | $199–249/yr + state fees | $199/yr add-on (not included) | Navi AI assistant | Branding, logos, website, bookkeeping, coaching |
+| **Stripe Atlas** | $500 one-time | N/A | Free yr1 / $100/yr | No | Stripe payments, partner perks |
+| **Firstbase** | $399 one-time | N/A | $99/yr+ | No | Virtual address, bookkeeping, tax |
+| **doola** | $297 + state fees (Starter) | Business-in-a-Box™ $1,999–2,999/yr | Included | AI Co-Founder | Bookkeeping, banking, taxes, analytics |
+| **Collective** | $199/mo (LLC) | $349/mo (S Corp) | Included in formation fee | Embedded AI Assistant | Payroll, bookkeeping, tax filing, S Corp election |
+| **Northwest Registered Agent** | $39 + state fees | N/A | Free yr1 / $125/yr | No | Clean, private LLC formation only |
+
+### Detailed Profiles
+
+#### LegalZoom
+- **Pitch:** "We've helped over 4 million people start their business"
+- **Strengths:** Massive brand recognition, broad legal service ecosystem (trademarks, contracts, legal plans), established trust, significant marketing spend
+- **Weaknesses:** Total cost escalates rapidly with add-ons — registered agent is separate, legal plans are subscription-based, upselling is aggressive. Entry price of "$0" is misleading once needed services are added
+- **True cost reality:** Basic LLC formation is $0 + state fees, but with registered agent ($249/yr), EIN service, operating agreement, and expedited filing, first-year cost often exceeds $500–700
+- **Pricing:**
+  - Basic: $0 + state fees (bare formation)
+  - Pro: $249 + state fees
+  - Premium: $349 + state fees
+- **Ratings:** Trustpilot 4+ stars; praise for ease, speed, professional support
+- **Gap for our platform:** LegalZoom is the 800lb gorilla — but they're a legal services company, not a business-building platform. No business planning, no AI validation, no banking setup integration. Their formation-to-operations handoff is: "Here's your LLC, good luck."
+
+#### ZenBusiness
+- **Pitch:** "Start, run, and grow your business"
+- **Strengths:** Compliance-focused workflow, all-in-one dashboard (formation + compliance), tiered packages with clear feature progression, lifetime support
+- **Weaknesses:** Annual renewals drive up total cost (Starter auto-renews compliance at $199/yr), registered agent not included on lower tiers, many features are 30-day trials that convert to paid
+- **Pricing:**
+  - Starter: $0 + state fees (formation only; optional free year Worry-Free Compliance → $199/yr renewal)
+  - Pro: $199/yr + state fees (adds compliance coverage, business advisor consultation)
+  - Premium: $399/yr + state fees (adds business insurance quote, domain, email, website)
+- **Compliance add-on:** $30/mo for Advanced Compliance Coverage
+- **Note:** ZenBusiness compares itself against Tailor Brands, LegalZoom, Bizee directly
+- **Gap for our platform:** ZenBusiness is closest to "formation + some operations" but they're increasingly moving toward business insurance and compliance, not business planning or AI validation. Their AI website builder is rudimentary — LLM-generated, 30-day trial.
+
+#### Bizee (formerly Incfile)
+- **Pitch:** Budget-first, straightforward LLC formation with free first-year registered agent
+- **Strengths:** Lowest true cost — $0 formation INCLUDES first year registered agent (vs competitors that charge $199+ separately), one-time pricing (no annual subscription), 1M+ businesses formed
+- **Weaknesses:** Less all-in-one than ZenBusiness or Tailor Brands. Registered agent renews at $119–149/yr. Customer support can be inconsistent. Multiple upsells during checkout.
+- **Pricing (one-time — not recurring):**
+  - Basic: $0 + state fees (includes LLC filing, 1yr registered agent, 1mo virtual address)
+  - Standard: $199 + state fees (adds EIN, operating agreement, compliance alerts, expedited processing ~2 weeks)
+  - Premium: $299 + state fees (adds domain + email, business phone, contracts, 4-day processing, lifetime compliance alerts)
+- **Processing times:** Basic 4–6 weeks, Standard ~2 weeks, Premium ~4 business days
+- **Gap for our platform:** Cheapest formation, but it's a pure filing service. No planning, no AI, no operations. The formation-to-business handoff is a list of "next steps" links.
+
+#### Tailor Brands
+- **Pitch:** "Set up, Manage & Grow Your Business with Tailor Brands" — all-in-one business building platform with LLC formation as entry point
+- **Strengths:** Broadest platform scope among formation services — LLC formation + branding (logos, website, domain) + legal documents (44 attorney-written templates) + bookkeeping + business coaching + funding search + sales/payments + Navi AI business assistant. 4.8/5 Trustpilot (16,515 reviews).
+- **Weaknesses:** Most expensive registered agent at $199/yr (not included in any plan), aggressive upselling (6 screens before checkout, Elite is default selection), virtual address $390–990/yr, privacy not built in by default, EIN is $99 add-on (free from IRS for US citizens), can't add RA or virtual address after submission
+- **Pricing:**
+  - Lite: $0 + state fees (formation + coaching program + 30-day bookkeeping trial), standard ~14 business day processing
+  - Essential: $199/yr + state fees (adds expedited 1-day, annual compliance, operating agreement)
+  - Elite: $249/yr + state fees (adds domain, website builder, 8 logos, digital business card, social media tools)
+- **Navi AI:** AI-powered assistant that tracks progress, sends reminders, recommends next steps
+- **Gap for our platform:** Tailor Brands is the CLOSEST competitor to our full vision. They cover formation + branding + some ops. But they lack: business plan generation (their "free business plan by email" is a PDF template, not AI-generated), AI idea validation/market sizing, multi-agent critical review, and full operational setup (banking is a referral link, not integrated). Their AI (Navi) is a progress tracker, not a strategic advisor.
+
+#### Stripe Atlas
+- **Pitch:** "The easiest way for founders to start a US company from anywhere in the world"
+- **Strengths:** Deep Stripe integration, 140+ countries, $50,000+ partner perks (Mercury, Xero, AWS), equity issuance + 83(b) election, templates by Cooley LLP, strong international founder support
+- **Weaknesses:** Delaware C-Corp/LLC only, NOT a full founder operating stack — no cap table management, no investor CRM, no mailroom/address service beyond registered agent, no business planning. VC-track startups, not main street.
+- **Pricing:** $500 one-time (includes first-year registered agent; $100/yr renewal)
+- **Scale:** Used by startups in 140+ countries
+- **Gap for our platform:** Atlas is for VC-track tech startups incorporating in Delaware. Wrong audience (main street vs VC-backed), wrong entity focus (Delaware vs state-specific), wrong follow-through (partner perks vs integrated operations). Not a direct competitor but validates the incorporation-as-platform-entry-point model.
+
+#### Firstbase
+- **Pitch:** "Launch and grow your US company from anywhere"
+- **Strengths:** Strong international focus (190+ countries), $350,000 in partner perks, post-incorporation compliance stack (AgentTM from $99/yr, virtual address, bookkeeping, tax), 7-day money-back guarantee
+- **Weaknesses:** Registered agent and address are recurring costs, full post-incorporation stack adds up, still primarily a formation + compliance platform, not a business-building platform
+- **Pricing:** $399 one-time; AgentTM from $99/yr
+- **Gap for our platform:** Similar to Stripe Atlas — international founder focus. Not competing for main street US businesses.
+
+#### doola
+- **Pitch:** "Business-in-a-Box™ for global entrepreneurs" — formation + bookkeeping + tax + banking + analytics + AI Co-Founder
+- **Strengths:** YC-backed (Y Combinator, HubSpot, Nexus), most integrated formation-to-operations competitor, AI Co-Founder (24/7 assistant answering business questions), e-commerce analytics built-in (Shopify, Amazon), 15,000+ founders globally, handles EIN for non-US residents
+- **Weaknesses:** Product Hunt reviews note reliability issues (delayed filings, high email volume required), pricing at high end ($297 one-time to $2,999/yr Business-in-a-Box), positioning shifting toward e-commerce specifically, support quality complaints from Product Hunt reviewers
+- **Pricing:**
+  - Starter: $297 + state fees (formation + EIN)
+  - Business-in-a-Box™: ~$1,999–2,999/yr (formation + bookkeeping + taxes + banking + AI Co-Founder + analytics)
+- **Gap for our platform:** doola is the most direct competitor in the "formation + operations" space. They have AI Co-Founder, but it's a Q&A assistant, not a strategic planning engine. They lack business plan generation, idea validation, and multi-agent review. Their pricing ($2K–3K/yr for full stack) opens a pricing umbrella for a more affordable alternative.
+
+#### Collective
+- **Pitch:** "The all-in-one financial solution for self-employed entrepreneurs" — LLC + S Corp formation + bookkeeping + payroll + tax
+- **Strengths:** Niche focus on solopreneurs BUSINESS-OF-ONE, S Corp optimization as core value prop (members save avg $10,000/yr on taxes), integrated back-office (payroll → books → taxes in one system), embedded AI assistant, 100% tax-deductible membership, $100M+ saved for members
+- **Weaknesses:** S Corp focus limits TAM (only appropriate above certain revenue thresholds), individual tax return is add-on ($199+/yr starting), monthly subscription model ($199–349/mo = $2,388–4,188/yr), not for multi-member or non-S Corp businesses
+- **Pricing:**
+  - LLC Tier: $199/mo (formation or entity review, monthly bookkeeping, invoicing, quarterly tax estimates, basic individual return, AI assistant)
+  - S Corp Tier: $349/mo (everything in LLC + S Corp election, annual business return 1120-S, employer registration, payroll, dedicated account manager)
+  - Payroll add-on: $15/employee/mo, $5/contractor/mo
+- **Gap for our platform:** Collective is the most operations-complete competitor, but they START at formation, provide no business planning or validation. They're tax-optimization first, business building second. Their audience (established solopreneurs making $60K+) is a subset of our ICP. Not a threat to our Phase 1–3 offering.
+
+### Segment 3 Pricing Comparison — True First-Year Cost (LLC)
+
+| Provider | Formation | Registered Agent | EIN | Operating Agreement | **Total Yr 1** | Yr 2+ (RA only) |
+|----------|-----------|-----------------|-----|--------------------|----------------|-----------------|
+| Bizee Basic | $0 | Free (yr 1) | +$70 | +$99 | $0 + state fee | $119–149 |
+| ZenBusiness Starter | $0 | +$199/yr | +$99 | +$99 | $0 + state fee | $199 |
+| LegalZoom Basic | $0 | +$249/yr | +$79 | +$99 | $0 + state fee | $249 |
+| Tailor Brands Lite | $0 | +$199/yr (must add at checkout) | +$99 | +$99 (must add at checkout) | $0 + state fee | $199 |
+| Northwest | $39 | Free (yr 1) | Optional | Optional | $39 + state fee | $125 |
+| doola Starter | $297 | Included | Included | Not specified | $297 + state fee | Included |
+| Collective LLC | $199/mo | Included | Included | Included | $2,388 | $2,388 |
+
+**Note:** All "+ state fees" — actual cost varies by state, typically $50–500.
+
+---
+
+## Segment 4: Business Operations Setup Tools
+
+### Competitive Overview
+
+| Company | Target | Free Tier? | Key Features | Yield/APY | FDIC Coverage |
+|---------|--------|------------|--------------|-----------|---------------|
+| **Mercury** | Startups, tech companies | Yes (free checking/savings) | Checking, savings, Treasury (up to 3.83% yield), virtual/physical cards, venture debt, permissions | Up to 3.83% via Treasury | Standard $250K |
+| **Brex** | VC-backed startups → Enterprise | Essentials free; Premium $12/user/mo; Enterprise custom | Checking, Treasury (up to 3.70%), Vault ($6M FDIC), corporate cards, expense mgmt, travel, bill pay, AI automation, 40+ currencies | Up to 3.70% via Treasury (no APY on checking) | Up to $6M (Vault) |
+| **Novo** | Freelancers, smallest businesses | Yes ($0 monthly fees) | Free checking, 2% cashback on debit, basic invoicing, bookkeeping integrations, no minimums | No yield | $250K |
+| **Rho** | Seed to Series B startups | No platform fees | Checking, cards (up to 1.5% cashback), Treasury ($75M FDIC), bill pay, invoicing, expense mgmt, AP automation, NetSuite sync | Competitive treasury rates | Up to $75M |
+| **Gusto** | SMBs (500K+ businesses) | No ($40/mo base + $6/pp) | Payroll, HR, benefits, time tracking, hiring/onboarding, compliance, AI Assistant | N/A (payroll, not banking) | N/A |
+| **Collective** | Solopreneurs (12K+ members) | No ($199–349/mo) | S Corp formation, payroll, bookkeeping, tax filing, AI assistant | N/A (back-office, not banking) | N/A |
+
+### Detailed Profiles
+
+#### Mercury
+- **Pitch:** "Radically different banking" — software-built banking for 300K+ entrepreneurs
+- **Strengths:** Beautiful UX, strong startup brand, free core banking, virtual cards in seconds, venture debt access, built-in team permissions, strong SaaS/e-commerce/agency vertical presence
+- **Weaknesses:** Known to close accounts of non-US residents and certain industries, limited for international teams, features beyond basic checking are tiered/paid
+- **Pricing:** Free core checking + savings; Treasury yield up to 3.83%; advanced features on paid tiers
+- **Gap for our platform:** Mercury is a destination bank for tech startups, not a launch platform. They don't help you form the entity or write the business plan. Great integration partner, not a competitor.
+
+#### Brex
+- **Pitch:** "Business banking that works as hard as you do" — enterprise spend platform
+- **Strengths:** AI-powered expense management, global payments in 40+ currencies, $6M FDIC via Vault, no fees on wires/ACH/ERP integration, transparent pricing, 1 in 3 startups use Brex
+- **Weaknesses:** Acquired by Capital One (April 2026) — enterprise direction may accelerate; Essentials tier has documented support delays; per-user pricing ($12/user/mo Premium); designed for enterprise, not early-stage
+- **Pricing:** Essentials (free), Premium ($12/user/mo), Enterprise (custom)
+- **Cards:** 1.5% cashback or points-based
+- **Gap for our platform:** Brex is an enterprise spend platform startups can use — not a launch platform. No formation, no planning. Post-acquisition trajectory likely toward Capital One's enterprise/commercial banking, not small business launch.
+
+#### Novo
+- **Pitch:** "Built for small business owners" — free banking with no fees
+- **Strengths:** Truly free ($0 monthly fees, no minimums), 2% cashback on debit purchases, simple digital interface, basic invoicing + bookkeeping integrations, great for solo operators just launching
+- **Weaknesses:** "Stops working when headcount and vendor complexity arrive" (per Rho comparison), no yield on deposits, limited feature set beyond basic checking, no wire support on free tier
+- **Pricing:** $0 monthly fee; some transaction limits apply
+- **Gap for our platform:** Novo is the banking answer for our ICP (solo founders, freelancers). Great integration partner — our platform should recommend Novo/Mercury as the "now open your bank account" step.
+
+#### Rho
+- **Pitch:** "The platform that works for three people also works for three hundred" — full-stack startup banking
+- **Strengths:** No platform fees at any tier, $75M FDIC coverage (15x Brex), 24/7 phone support included, full AP automation + invoicing + expense management + NetSuite sync, available through incorporation marketplaces
+- **Weaknesses:** Less brand recognition than Mercury/Brex, positioned for seed-to-Series B (not micro/solo), not a formation platform
+- **Gap for our platform:** Strong integration partner — they're in incorporation marketplaces, meaning they'd likely partner. But they're banking, not business building.
+
+#### Gusto
+- **Pitch:** #1 all-in-one payroll, HR, and benefits for SMBs
+- **Strengths:** 500K+ businesses, Simple plan from $40/mo base + $6/person/mo, AI Assistant, 152 hrs/year average saved on tax/compliance, top-rated across CNBC, Forbes, G2, Nerdwallet, Techradar
+- **Weaknesses:** Payroll/HR only — not a banking or formation platform. Pricing scales with headcount.
+- **Pricing:** Simple $40/mo base + $6/person/mo; Plus and Premium tiers available
+- **Gap for our platform:** Essential integration partner — our platform should seamlessly connect to Gusto for payroll/HR setup during Phase 5.
+
+### Segment 4 Gap Analysis
+
+| Gap | Opportunity |
+|-----|-------------|
+| **No platform integrates banking setup with formation** | Mercury, Novo, Rho are separate signup flows. Our platform can pre-fill applications |
+| **Payroll setup is completely disconnected** | Gusto signup is independent. Our platform can trigger it as Phase 5 step |
+| **Business credit building is overlooked** | No competitor helps founders establish business credit post-formation |
+| **Compliance calendars are siloed** | Each tool has its own compliance reminders. Our platform can be the source of truth |
+
+---
+
+## Segment 5: AI Agents for Business Startups
+
+### Competitive Overview
+
+This is the most nascent and fragmented segment. No AI agent platform currently handles the full business launch arc. Instead, they address narrow slices.
+
+| Platform | Category | What It Does | Pricing |
+|----------|----------|-------------|---------|
+| **Enso** | SMB AI agents | 1,000+ AI agents across 70 industries — SEO, social media, competitor tracking, invoicing | Not publicly listed |
+| **Gumloop** | AI workflow automation | AI agent builder for business process automation | Not publicly listed |
+| **Cofounder** | Startup building | AI co-founder for startups | Not listed |
+| **Make** | Automation platform | Visual workflow automation (Zapier competitor) | Free tier; from $9/mo |
+| **Lindy** | AI agents | General AI agents for business tasks | Not listed |
+| **Microsoft Copilot Studio** | Enterprise AI agents | M365/Azure-integrated AI agents for workflow automation | Enterprise pricing |
+| **Google Vertex AI Agent Builder** | Enterprise AI agents | BigQuery-integrated analytics-driven automation agents | Enterprise pricing |
+| **doola AI Co-Founder** | Business formation AI | 24/7 Q&A assistant for business questions, back-office task handling | Included in Business-in-a-Box™ |
+| **Claude Code / Codex CLI / OpenCode** | AI coding agents | Autonomous software development (not business launch) | Various |
+| **Operator AI Agent Platforms** | Autonomous execution | True autonomous agents executing business workflows | Various |
+| **Autonoms** | Single-workflow automation | Automating specific, defined business workflows | Not listed |
+
+### Key Observations
+
+1. **Enterprise AI agents dominate headlines** but don't serve main street businesses. Microsoft Copilot Studio, Google Vertex AI, IBM Watsonx Orchestrate, SAP Leonardo AI — all enterprise-first, requiring existing infrastructure.
+
+2. **SMB AI agents are emerging but fragmented.** Enso's 1,000+ agents across 70 industries is the broadest, but each agent is a narrow point solution. No "business launch agent" exists.
+
+3. **AI coding agents (Claude Code, Codex, OpenCode)** are adjacent — they help build software products, not launch businesses.
+
+4. **No competitor has built a "multi-agent business launch pipeline"** — the concept of chaining multiple specialized AI agents (market validator → plan writer → plan reviewer → formation assistant → ops coordinator) is white space.
+
+5. **Meta's Business AI** (launched October 2025) targets SMBs with sales automation and customer interaction — adjacent but not competitive. Shows Big Tech is eyeing the SMB AI market.
+
+### Segment 5 Gap Analysis
+
+| Gap | Opportunity |
+|-----|-------------|
+| **No multi-agent business launch pipeline exists** | White space for IT Pro Partner's "Critical Review via multi-agent AI pipeline" (Phase 3) |
+| **Enterprise AI agents don't serve main street** | Our ICP is explicitly excluded from Microsoft/Google/SAP's target market |
+| **AI co-founder tools are Q&A, not execution** | doola's AI Co-Founder answers questions; it doesn't execute multi-step workflows |
+| **Autonomous execution is immature** | "True autonomy" is still aspirational for most platforms — hybrid AI + human review is realistic |
+
+---
+
+## Segment 6: Adjacent & Emerging Competitors
+
+### Business Idea Validation Tools (Detailed)
+
+The AI idea validation space has exploded. These tools compete with our Phase 1 (Discovery/Ideation):
+
+| Tool | Key Differentiator | Price | Vulnerability |
+|------|-------------------|-------|---------------|
+| **IdeaProof** | Multi-model AI (3 models cross-validated) | €10–70 credit packs | AI-generated numbers, speculative TAM |
+| **ValidatorAI** | Free, fast, simple | Free | Hallucinates market data, too encouraging |
+| **DimeADozen** | S-1 filing data, real comp sets | $59/report | Generic on vague prompts, niche-underperforming |
+| **Preuve AI** | Demand signals + competitor pricing | Not listed | Newer entrant, less established |
+| **VenturusAI** | Business frameworks (SWOT, PESTEL, Porter) | Free start | Thin validation, speculative financials |
+| **Trend Seeker** | Reddit/community demand evidence | Free / $9.99/mo | Demand signals only, no strategic analysis |
+| **FounderPal** | Free side-by-side idea comparison | Free validator / $199 lifetime | Lighter market sizing |
+
+**Key takeaway:** This space is crowded with point solutions but completely disconnected from execution. None of these tools help you form an entity or set up banking. They generate a report and leave.
+
+### Emerging Trends to Watch
+
+1. **Vertical-specific business-in-a-box platforms** — e.g., doola's e-commerce focus signals a trend toward industry-specific launch platforms. A competitor targeting "restaurant launch" or "consulting practice launch" could emerge.
+
+2. **Capital One's Brex acquisition (April 2026)** — signals banking consolidation. Brex retains brand/leadership but enterprise direction may accelerate, leaving a gap in early-stage startup banking.
+
+3. **AI + incorporation convergence** — doola, Tailor Brands, and Collective are all adding AI assistants to their formation platforms. This trend will intensify. The question is whether they expand backward into planning or stay focused on formation + compliance.
+
+4. **68% of small businesses now use AI** (up from 51% two years ago, per Fox Business). 80% find AI enhances rather than replaces workforce. The TAM for AI business tools is expanding rapidly.
+
+5. **No-code/AI platform convergence** — tools like Lovable ($100M ARR in 18 months) show the speed at which AI-native platforms can scale. A well-funded AI business launch platform could emerge quickly.
+
+6. **White-label and embedded finance** — platforms like Rho appearing in incorporation marketplaces show the trend toward embedded financial services. Our platform should embed banking/payroll, not just link to them.
+
+---
+
+## Competitive Landscape Map
+
+```
+                    PLANNING FOCUS ← → EXECUTION FOCUS
+                         │
+    HIGH AI             IdeaProof       │    doola AI Co-Founder
+     │                  Upmetrics       │    Tailor Brands (Navi)
+     │                  IdeaBuddy       │    Collective (AI)
+     │                  DimeADozen      │
+     │                  ValidatorAI     │
+     │                                  │
+     ├──────────────────────────────────┤
+     │                                  │
+     │                  LivePlan        │    LegalZoom
+     │                  BizPlan         │    ZenBusiness
+    LOW AI              Enloop (DEAD)   │    Bizee
+                         │              │    Stripe Atlas
+                    PLAN GENERATORS     │    Firstbase
+                                        │    Northwest
+                         │              │
+                    PURE PLANNING       │    PURE EXECUTION
+                         │              │
+```
+
+**IT Pro Partner's target position:** Top-right quadrant (HIGH AI × EXECUTION FOCUS)
+
+---
+
+## White Space & Differentiation Opportunities
+
+### 1. The Full Arc (Primary Differentiator)
+
+**No competitor covers all 5 phases.** This is the core strategic insight:
+
+| Phase | Our Platform | Nearest Competitor |
+|-------|-------------|-------------------|
+| 1. Discovery/Ideation | AI market sizing, validation, ICP definition | IdeaProof, ValidatorAI (standalone, no follow-through) |
+| 2. Business Plan Builder | AI-generated plan with financials | Upmetrics, LivePlan (no entity connection) |
+| 3. Critical Review | Multi-agent AI pipeline (adversarial review) | **NO COMPETITOR** |
+| 4. Entity Formation | State-specific LLC/corp filing, EIN, operating agreement | LegalZoom, ZenBusiness, Bizee (no planning) |
+| 5. Operational Setup | Integrated banking, payroll, compliance, insurance setup | Collective, doola (formation + some ops, no planning) |
+
+**Phase 3 (Critical Review) is completely uncontested.** This is potentially the highest-value differentiator. Every founder should have their plan stress-tested before committing capital. No existing platform does this.
+
+### 2. Underserved ICPs
+
+| ICP | Why They're Underserved |
+|-----|------------------------|
+| **Career-changers** (corporate → entrepreneurship) | No platform speaks to their specific anxiety: "I've never done this before" |
+| **Main street businesses** (restaurants, retail, services) | LegalZoom/Tailor Brands are too generic; Collective is S Corp/solopreneur only |
+| **Side-hustle-to-full-time** transitioners | Need guidance on when/how to formalize — no platform provides this |
+| **Tradespeople formalizing** (HVAC, electricians, plumbers) | Completely unaddressed by existing platforms — massive TAM |
+| **Family businesses transitioning** to next generation | No platform addresses succession + modernization + formalization |
+
+### 3. Operational Credibility (IT Pro Partner's Differentiator)
+
+- **Every competitor is either a tech startup or a legal services company.** None has operational infrastructure credentials.
+- IT Pro Partner can position as: "Built by people who actually run business infrastructure — not a VC-funded app."
+- This credibility matters for main street businesses who distrust "tech bro" solutions.
+
+### 4. Pricing White Space
+
+| Segment | Current Price Range | Opportunity |
+|---------|-------------------|-------------|
+| Plan generators | $7–45/mo or $249–349 lifetime | We can bundle plan + review at $29–49/mo |
+| AI validation | €10–70 one-time or $9.99–59/report | Include as Phase 1 — absorb as acquisition cost |
+| Entity formation | $0–500 + state fees (one-time) | We charge a transparent $199–299 one-time for full filing + EIN + OA + RA yr 1 |
+| Operations setup | $0–349/mo (recurring) | Partner integration (not competitor) — referral revenue |
+| **Full platform bundle** | **$29–49/mo + $199 one-time formation** | **Significantly under collective competitor à la carte costs** |
+
+---
+
+## Threat Assessment Matrix
+
+| Threat | Likelihood | Impact | Mitigation |
+|--------|-----------|--------|------------|
+| Tailor Brands adds business planning | Medium | High | Build Phase 1–3 moat (validation + review); they're branding-first |
+| LegalZoom adds AI business planning | Medium | High | They're a legal company — business planning isn't their DNA. Our operational credibility is stronger |
+| doola expands planning/validation | Medium-High | High | doola is the most likely to expand. Differentiate on main street ICP + operational credibility vs doola's e-commerce focus |
+| ZenBusiness adds full operations | Low-Medium | Medium | They're adding insurance, not planning. Compliance-first DNA |
+| Stripe enters main street formation | Low | Medium | Stripe Atlas is Delaware C-Corp only — wrong product for main street |
+| New AI-native entrant (YC-backed) | Medium | Very High | Speed to market critical. A well-funded AI agent startup could build the full arc in 12–18 months |
+| Free AI tools commoditize Phase 1 | High | Medium | ChatGPT can write a business plan today. Differentiate with multi-agent review + execution follow-through |
+
+---
+
+## Strategic Recommendations
+
+### 1. Phase the build — don't build everything at once
+
+| Launch Phase | Scope | Rationale |
+|-------------|-------|-----------|
+| **MVP (Q4 2026)** | Phase 1 (Discovery) + Phase 2 (Plan Builder) | Fastest path to revenue. Validates demand for AI planning. |
+| **V2 (Q1 2027)** | Phase 3 (Critical Review) | Unique differentiator. No competitor has this. |
+| **V3 (Q2 2027)** | Phase 4 (Entity Formation) | Higher complexity (state-specific, legal liability). Partner with registered agent first. |
+| **V4 (Q3 2027)** | Phase 5 (Operations Setup) | Partner integrations with Novo, Mercury, Gusto. Don't build banking. |
+
+### 2. Partner, don't compete, with banking/payroll
+
+- **Novo, Mercury, Rho** are potential referral partners — they want our formation volume
+- **Gusto** has an affiliate program — revenue share on payroll signups
+- **Business insurance** (Next, Hiscox) — another referral revenue stream
+- **Position:** "We'll get you to the point of needing a bank account, then recommend the best one for your business type"
+
+### 3. Nail the main street ICP before expanding
+
+- Start with one vertical: **trades/contractors** or **professional services** (coaches, consultants, freelancers)
+- Tailor the plan templates, compliance checklists, and banking recommendations to that vertical
+- Expand to adjacent verticals after product-market fit
+
+### 4. White-label for accountants and business coaches
+
+- Upmetrics charges $185/mo for white-label. LivePlan has an accountant-focused product.
+- Accountants and business coaches influence 60%+ of new business formation decisions
+- A white-label partner program creates a distribution moat that pure-tech competitors can't replicate
+
+### 5. Price for main street, not startups
+
+- **Monthly SaaS:** $29–49/mo for full platform access (Plans + Review + Dashboard)
+- **Formation one-time:** $199–299 + state fees (with first-year registered agent)
+- **Annual discount:** $249–399/year for platform + one free formation per year
+- **Compare:** Collective is $2,388–4,188/yr. doola Business-in-a-Box is $1,999–2,999/yr. A $500–800/yr all-in price would be dramatically more affordable while still premium vs free formation alternatives.
+
+---
+
+## Appendix A: Pricing Reference Table
+
+| Competitor | Segment | Entry | Mid | Premium | Recurring? |
+|-----------|---------|-------|-----|---------|------------|
+| LivePlan | Plan Gen | $15/mo | $18/mo | $20/mo or $349 lifetime | Monthly / Lifetime |
+| Upmetrics | Plan Gen | $7/mo | $14/mo | $37/mo + $185/mo white-label | Monthly/Annual |
+| BizPlan | Plan Gen | $29/mo | — | $249/yr or $349 lifetime | Monthly / Lifetime |
+| IdeaBuddy | Plan Gen | Free | $9/mo | $22–45/mo | Monthly/Annual |
+| IdeaProof | Validation | Free (90 credits) | €10 (150 credits) | €70 (1,500 credits) | One-time credits |
+| DimeADozen | Validation | Free (Solo) | $59/report | Enterprise custom | Per-report |
+| ValidatorAI | Validation | Free | Pro (unlisted) | — | Unknown |
+| LegalZoom | Formation | $0 + state fees | $249 + state fees | $349 + state fees | One-time + separate RA |
+| ZenBusiness | Formation | $0 + state fees | $199/yr + state fees | $399/yr + state fees | Annual |
+| Bizee | Formation | $0 + state fees | $199 + state fees | $299 + state fees | One-time |
+| Tailor Brands | Formation | $0 + state fees | $199/yr + state fees | $249/yr + state fees | Annual |
+| Stripe Atlas | Formation | $500 one-time | — | — | One-time |
+| Firstbase | Formation | $399 one-time | — | — | One-time |
+| doola | Formation+Ops | $297 + state fees | Business-in-a-Box $1,999–2,999/yr | — | One-time / Annual |
+| Collective | Formation+Ops | $199/mo (LLC) | $349/mo (S Corp) | — | Monthly/Annual |
+| Mercury | Banking | Free | — | — | Free |
+| Brex | Banking | Free (Essentials) | $12/user/mo (Premium) | Enterprise (custom) | Monthly |
+| Novo | Banking | Free | — | — | Free |
+| Rho | Banking | Free (no platform fees) | — | — | Free platform |
+| Gusto | Payroll/HR | $40/mo base + $6/pp | Plus tier | Premium tier | Monthly |
+
+---
+
+## Appendix B: Data Sources
+
+- Competitor websites (direct extraction): LivePlan, Upmetrics, BizPlan, LegalZoom, ZenBusiness, Stripe Atlas, Firstbase, Tailor Brands, Bizee, doola, Collective, Mercury, Brex, Novo, Rho, Gusto, IdeaBuddy, IdeaProof, DimeADozen, ValidatorAI, VenturusAI, FounderPal, Preuve AI, Trend Seeker, Sintra AI, Bizway
+- Third-party reviews: VentureSmarter, SMB Guide, Sonary, Wolters Kluwer, Cybernews, Rho.co, Efficient.app, TRUiC/Startup Savant, Crazy Egg, llc.org
+- Industry analysis: Preuve AI validation tools comparison, Trend Seeker, Operater Blog, Marketer Milk
+- News: Yahoo Finance (doola AI Co-Founder launch), Fox Business (SMB AI adoption 68%), Brex acquisition by Capital One announcement (April 2026)
+
+---
+
+## Appendix C: Glossary
+
+| Term | Definition |
+|------|-----------|
+| **ICP** | Ideal Customer Profile — the specific type of customer a product is built for |
+| **RA** | Registered Agent — a person/company designated to receive legal documents on behalf of an LLC/corporation |
+| **EIN** | Employer Identification Number — federal tax ID for businesses |
+| **OA** | Operating Agreement — internal document governing LLC ownership and operations |
+| **TAM/SAM/SOM** | Total Addressable Market / Serviceable Addressable Market / Serviceable Obtainable Market |
+| **S Corp** | S Corporation — tax election allowing pass-through taxation with reduced self-employment tax |
+| **AP** | Accounts Payable — money owed by a business to suppliers/vendors |
+| **ACH** | Automated Clearing House — electronic funds transfer system |
+
+---
+
+*Report prepared by Hermes Agent for IT Pro Partner strategic planning. All pricing data verified August 2026. Competitive positioning subject to change as the market evolves rapidly.*

clients/modelortho/seo-audit-2026-08-10.md

@@ -0,0 +1,429 @@
+# ModelOrtho.com — SEO Audit & Remediation Plan
+
+> **Audited:** August 10, 2026  
+> **URL:** https://modelortho.com/  
+> **Auditor:** Sho'Nuff (Hermes Agent)  
+> **Site Owner:** Anita Brown  
+
+---
+
+## 1. Current State — What's Working / What's Broken
+
+### What's Working ✅
+
+| Feature | Status | Detail |
+|---|---|---|
+| HTTPS | ✅ | Cloudflare SSL termination, port 80→443 redirect |
+| www → non-www | ✅ | www redirects 301 to modelortho.com |
+| Semantic HTML | ✅ | header, nav, main, section (×7), footer |
+| Viewport | ✅ | `width=device-width, initial-scale=1.0` |
+| Language | ✅ | `<html lang="en">` |
+| No em dashes | ✅ | Zero detected |
+| Load time | ✅ | ~401ms DOMContentLoaded |
+| Page titles | ✅ | All pages have distinct, keyword-rich titles |
+| H1 on every page | ✅ | One H1 per page |
+| Privacy / Terms | ✅ | Comprehensive, well-structured legal pages |
+| Services page | ✅ | 5 detailed service sections, strong hierarchy |
+| Tools page | ✅ | 3 proprietary tools described with features |
+| Content quality | ✅ | Professional copy, no AI fluff, authentic voice |
+| Dark/light theme | ✅ | Toggle works, cookie-persisted |
+
+### What's Broken ❌
+
+| Severity | Issue | Impact |
+|---|---|---|
+| **P0** | No meta description | Search engines auto-generate snippet — you lose control of the pitch |
+| **P0** | No OG image | Social shares render as bare text link. No preview card on LinkedIn, Twitter, Facebook |
+| **P0** | robots.txt returns 404 | Crawlers have no guidance. Harmless now but unprofessional |
+| **P0** | sitemap.xml returns 404 | Search engines must discover pages via links alone. Missed indexation signal |
+| **P0** | No favicon | Looks amateur in browser tabs and bookmarks |
+| **P0** | No JSON-LD structured data | Missing LocalBusiness/ProfessionalService schema. No rich results eligibility |
+| **P1** | No canonical URL tag | Duplicate content risk if any page gets indexed with params |
+| **P1** | Zero images on site | No visual engagement. No `alt` text to rank in image search. No hero image, no headshot, no tool screenshots |
+| **P1** | Footer links to dead pages | `privacy.html` and `terms.html` exist (200) but `policies.html` and `login.html` return 404 |
+| **P2** | No hreflang tags | Not critical for single-language site, but worth adding |
+| **P2** | No external links | 8 internal links, 0 external. Zero backlink strategy |
+| **P2** | Homepage word count | 1,160 words — decent but could be 1,500-2,000 for competitive terms |
+
+---
+
+## 2. Technical Scan — Page-by-Page
+
+| Page | Title | Meta Desc | OG Title | OG Desc | OG Image | Canonical | JSON-LD | H1 Count |
+|---|---|---|---|---|---|---|---|---|
+| `/` | Model Ortho \| Orthodontic Practice Operations Consulting | MISSING | MISSING | MISSING | MISSING | MISSING | MISSING | 1 |
+| `/services.html` | Services \| Model Ortho Consulting | MISSING | MISSING | MISSING | MISSING | MISSING | 1 |
+| `/tools.html` | Tools \| Model Ortho Consulting | MISSING | MISSING | MISSING | MISSING | MISSING | 1 |
+| `/privacy.html` | Privacy Policy \| Model Ortho | MISSING | MISSING | MISSING | MISSING | MISSING | 1 |
+| `/terms.html` | Terms of Service \| Model Ortho | MISSING | MISSING | MISSING | MISSING | MISSING | 1 |
+
+**Every page is missing:** meta description, OG tags, canonical, structured data.
+
+---
+
+## 3. Priority Fixes
+
+### P0 — Week 1 (do these first)
+
+#### 3.1 Add Meta Description
+
+Every page needs a unique 150-160 character meta description. Here are ready-to-paste versions:
+
+**Homepage:**
+```html
+<meta name="description" content="Model Ortho helps orthodontic practices turn operational data into forward-looking systems. Scheduling, capacity planning, growth forecasting, and schedule recovery - built from real practice data, not generic advice.">
+```
+
+**Services page:**
+```html
+<meta name="description" content="Orthodontic operations consulting: scheduling optimization, capacity planning, growth forecasting, schedule recovery, and team systems development. End-to-end operations support for growing practices.">
+```
+
+**Tools page:**
+```html
+<meta name="description" content="Proprietary operational intelligence tools for orthodontics: Schedule Canvas, Impact Forecaster, and Recovery Forecaster. Your practice data, modeled honestly.">
+```
+
+**Privacy:**
+```html
+<meta name="description" content="Model Ortho Consulting privacy policy. How we collect, use, and protect your practice and personal information.">
+```
+
+**Terms:**
+```html
+<meta name="description" content="Model Ortho Consulting terms of service. Conditions governing use of our website and orthodontic consulting services.">
+```
+
+#### 3.2 Add Open Graph Tags (Every Page)
+
+```html
+<meta property="og:title" content="Model Ortho | Orthodontic Practice Operations Consulting">
+<meta property="og:description" content="Orthodontic operations consulting that helps practices see the relationship between scheduling, capacity, production, and growth.">
+<meta property="og:type" content="website">
+<meta property="og:url" content="https://modelortho.com/">
+<meta property="og:image" content="https://modelortho.com/images/og-default.png">
+<meta property="og:image:width" content="1200">
+<meta property="og:image:height" content="630">
+<meta name="twitter:card" content="summary_large_image">
+```
+
+Customize `og:title`, `og:description`, and `og:url` per page.
+
+#### 3.3 Create robots.txt
+
+```
+User-agent: *
+Allow: /
+Sitemap: https://modelortho.com/sitemap.xml
+```
+
+#### 3.4 Create sitemap.xml
+
+```xml
+<?xml version="1.0" encoding="UTF-8"?>
+<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
+  <url>
+    <loc>https://modelortho.com/</loc>
+    <changefreq>weekly</changefreq>
+    <priority>1.0</priority>
+  </url>
+  <url>
+    <loc>https://modelortho.com/services.html</loc>
+    <changefreq>monthly</changefreq>
+    <priority>0.9</priority>
+  </url>
+  <url>
+    <loc>https://modelortho.com/tools.html</loc>
+    <changefreq>monthly</changefreq>
+    <priority>0.9</priority>
+  </url>
+  <url>
+    <loc>https://modelortho.com/privacy.html</loc>
+    <changefreq>yearly</changefreq>
+    <priority>0.3</priority>
+  </url>
+  <url>
+    <loc>https://modelortho.com/terms.html</loc>
+    <changefreq>yearly</changefreq>
+    <priority>0.3</priority>
+  </url>
+</urlset>
+```
+
+#### 3.5 Add JSON-LD Structured Data
+
+```html
+<script type="application/ld+json">
+{
+  "@context": "https://schema.org",
+  "@type": "ProfessionalService",
+  "name": "Model Ortho Consulting",
+  "description": "Orthodontic operations consulting and proprietary forecasting tools for growing practices.",
+  "url": "https://modelortho.com/",
+  "telephone": "(912) 581-1949",
+  "email": "anita@anitabrown.co",
+  "address": {
+    "@type": "PostalAddress",
+    "addressLocality": "Savannah",
+    "addressRegion": "GA",
+    "addressCountry": "US"
+  },
+  "hasOfferCatalog": {
+    "@type": "OfferCatalog",
+    "name": "Consulting Services",
+    "itemListElement": [
+      {
+        "@type": "Offer",
+        "itemOffered": {
+          "@type": "Service",
+          "name": "Scheduling & Capacity Consulting"
+        }
+      },
+      {
+        "@type": "Offer",
+        "itemOffered": {
+          "@type": "Service",
+          "name": "Growth & Forecasting"
+        }
+      },
+      {
+        "@type": "Offer",
+        "itemOffered": {
+          "@type": "Service",
+          "name": "Schedule Recovery"
+        }
+      },
+      {
+        "@type": "Offer",
+        "itemOffered": {
+          "@type": "Service",
+          "name": "Operations & Systems Development"
+        }
+      }
+    ]
+  }
+}
+</script>
+```
+
+#### 3.6 Add Favicon
+
+Create a 32×32 or 64×64 PNG and add:
+```html
+<link rel="icon" type="image/png" href="/images/favicon.png">
+<link rel="apple-touch-icon" href="/images/favicon.png">
+```
+
+---
+
+### P1 — Month 1
+
+#### 3.7 Add Canonical Tags (Every Page)
+
+```html
+<link rel="canonical" href="https://modelortho.com/">
+```
+Set `href` to the canonical URL of each page.
+
+#### 3.8 Fix Dead Footer Links
+
+Footer links to `policies.html` and `login.html` both return 404. Either:
+- Remove `policies.html` (privacy is at `/privacy.html`, terms at `/terms.html`)
+- Remove `login.html` (there is no login functionality on the site)
+- Or create those pages if they serve a purpose
+
+#### 3.9 Add Images
+
+This is the single biggest engagement gap. The site has zero images. Add:
+1. **Hero image** — abstract data visualization or scheduling graphic (1200×630)
+2. **Headshot** — Anita Brown, professional photo on the services or about section
+3. **Tool screenshots** — one per tool on `/tools.html` (Schedule Canvas, Impact Forecaster, Recovery Forecaster)
+4. **OG default image** — 1200×630 social share card with Model Ortho branding
+
+All images need descriptive `alt` text for SEO and accessibility.
+
+---
+
+### P2 — Quarter 1
+
+#### 3.10 Build a Blog or Resources Section
+
+Competitors like OrthoSynetics, Gaidge, and Sturgill all have blogs, case studies, or resource libraries. A blog targeting long-tail orthodontic operations queries would:
+- Establish topical authority for "orthodontic operations consulting" and related terms
+- Provide content for social shares and email nurture
+- Give search engines more pages to index and rank
+
+Suggested first 5 articles:
+1. "Why Your Schedule Is a System, Not a Calendar"
+2. "The Hidden Cost of Unfilled Appointments in Orthodontics"
+3. "Capacity Planning: When Growth Outpaces Your Practice"
+4. "How to Read Your Production Numbers (Beyond the Monthly Report)"
+5. "Schedule Recovery: From Scramble to Strategy"
+
+#### 3.11 Submit to Google Search Console + Bing Webmaster Tools
+
+After robots.txt, sitemap.xml, and metadata are deployed, submit the sitemap to both platforms. This is the fastest path to proper indexation.
+
+#### 3.12 Backlink Strategy
+
+Competitors have backlinks from orthodontic industry publications, podcasts, and vendor directories. Opportunities:
+- Guest posts on orthodontic business blogs (Orthodontic Products, Dental Economics)
+- Podcast appearances (orthodontic industry podcasts)
+- Directory listings (orthodontic consultant directories)
+- LinkedIn articles linking back to modelortho.com
+
+---
+
+## 4. Content Accuracy Verification
+
+| Requirement | Status | Detail |
+|---|---|---|
+| Product name consistency | ✅ | "Model Ortho" / "Model Ortho Consulting" used consistently |
+| Contact email | ✅ | anita@anitabrown.co on privacy and terms pages |
+| Phone number | ✅ | (912) 581-1949 on privacy and terms pages |
+| Effective date | ✅ | August 10, 2026 on both legal pages |
+| Menu items match pages | ✅ | Services → services.html, Tools → tools.html |
+| Social links | ❌ | None present. Add LinkedIn at minimum |
+| Em dashes | ✅ | Zero detected |
+| Footer links | ❌ | policies.html and login.html are dead links (404) |
+
+---
+
+## 5. Design Quality Assessment
+
+### What's Working
+- **Typography:** DM Sans + DM Serif Display via Google Fonts. Professional, clean, readable
+- **Color system:** Dark/light theme toggle. Thoughtful contrast
+- **Semantic HTML:** `header > nav`, `main > section`, `footer`. Proper heading hierarchy (H1→H2→H3)
+- **Copywriting:** Authentic voice. No AI filler. Every sentence earns its place
+- **Layout:** Single-column scannable flow. Good use of whitespace
+- **CTAs:** "Start the Conversation" and "Let's Look at the System" — action-oriented, consistent
+
+### What's Holding It Back
+- **Zero images:** No hero graphic, no headshot, no tool screenshots — feels like a whitepaper, not a consulting website
+- **No visual hierarchy:** Text-only pages lack entry points for skimming
+- **No social proof:** No testimonials, no client logos, no case study teasers
+- **No footer CTA:** Footer has legal links only. Add a contact link or "Start the Conversation" button
+- **No phone/email in header:** Contact info is buried in privacy/terms pages only
+
+### Quality Bar
+**65-70% of a top-tier consulting site** (Linear/Stripe-caliber design). The foundation is strong — typography, layout, copy. The gap is all visual: images, social proof, and conversion elements.
+
+---
+
+## 6. Competitive Analysis
+
+| Competitor | Strengths | Weaknesses vs Model Ortho |
+|---|---|---|
+| **OrthoSynetics** | Full-service (billing, HR, marketing + consulting). Large firm, established | Generic — not ortho-specific operations. No proprietary tools |
+| **Sturgill Orthodontic Consultants** | Orthodontic-only, strong scheduling focus. Client results published | Traditional consulting model. No tech-forward tool positioning |
+| **Gaidge** | Analytics SaaS platform. Freemium model, large user base | Software-only — no human consulting layer |
+| **Ortho Consulting Group** | Customized services, leadership coaching | Heavier, corporate-feeling. Less positioning around data/modeling |
+| **CascadEffects** | Named "top firm 2026." Doctor-independent growth focus | Broader dental, not ortho-exclusive |
+
+### Model Ortho's Positioning Opportunity
+
+Model Ortho occupies a unique position nobody else holds: **proprietary tools + human consulting, orthodontic-only**. Most competitors are either pure consulting (no tools) or pure SaaS (no consulting). The combination of "Schedule Canvas / Impact Forecaster / Recovery Forecaster" as proprietary IP with hands-on operational consulting is a defensible moat.
+
+The website needs to communicate this more aggressively. Right now it says "tools" — it should say "proprietary forecasting tools built specifically for orthodontic operations." The distinction matters.
+
+---
+
+## 7. Keyword Targets
+
+| Keyword | Est. Volume | Competition | Target Page |
+|---|---|---|---|
+| orthodontic practice consulting | Medium | Medium | Homepage, Services |
+| orthodontic operations consulting | Low-Med | Low | Homepage (primary target) |
+| orthodontic scheduling optimization | Low | Low | Services — Scheduling |
+| orthodontic capacity planning | Low | Very Low | Tools — Schedule Canvas |
+| orthodontic growth forecasting | Low | Very Low | Tools — Impact Forecaster |
+| orthodontic practice management consultant | Medium | High | Homepage (secondary) |
+| dental practice operations consultant | Medium | High | Homepage (tertiary) |
+| orthodontic consulting services | Medium | Medium | Services page |
+| orthodontic schedule recovery | Very Low | None | Tools — Recovery Forecaster |
+
+**Strategy:** Target low-competition, high-intent long-tail terms first ("orthodontic operations consulting", "orthodontic capacity planning"). Build topical authority. Then compete for broader terms ("orthodontic practice consulting").
+
+---
+
+## 8. Page Performance
+
+| Metric | Value |
+|---|---|
+| DOMContentLoaded | ~401ms |
+| HTML size | ~20KB (homepage) |
+| Google Fonts | 1 request (DM Sans + DM Serif Display) |
+| Stylesheet count | 1 (inline) |
+| Script count | 1 (theme toggle) |
+| Image count | 0 |
+| Total page weight | ~25KB |
+
+Performance is excellent — no blockers here. When images are added, use WebP format with srcset for responsive sizes.
+
+---
+
+## 9. Execution Roadmap
+
+### Week 1 — Technical Foundation
+- [ ] Add meta descriptions to all 5 pages
+- [ ] Add OG tags to all 5 pages
+- [ ] Create robots.txt
+- [ ] Create sitemap.xml
+- [ ] Add JSON-LD structured data
+- [ ] Add favicon (32px PNG)
+- [ ] Add canonical tags to all 5 pages
+
+**Effort:** ~2 hours for someone who can edit HTML/CSS.
+
+### Week 2 — Visual & Conversion
+- [ ] Design OG social share image (1200×630)
+- [ ] Add professional headshot to homepage
+- [ ] Add abstract hero graphic or scheduling visualization
+- [ ] Create 3 tool screenshots/mockups for tools page
+- [ ] Fix footer dead links (remove policies.html + login.html)
+- [ ] Add contact info to header or footer
+
+**Effort:** ~4 hours (image creation + HTML edits).
+
+### Week 3 — Launch
+- [ ] Submit sitemap to Google Search Console
+- [ ] Submit sitemap to Bing Webmaster Tools
+- [ ] Verify robots.txt is crawlable
+- [ ] Request indexation of all 5 pages
+- [ ] Set up Google Analytics or a privacy-friendly alternative
+
+**Effort:** ~1 hour.
+
+### Month 2-3 — Content & Authority
+- [ ] Launch blog with 5 articles (see Section 3.10)
+- [ ] Add LinkedIn profile link to site
+- [ ] Begin guest post / podcast outreach
+- [ ] Add testimonials section when available
+
+**Effort:** Ongoing — 1-2 articles per month.
+
+---
+
+## 10. Appendices
+
+### A. Competitor URLs
+- https://www.orthosynetics.com/osi-services/practice-consulting/
+- https://sturgillorthodontics.com/orthodontic-consultants-strategies-grow-your-practice/
+- https://www.gaidge.com/
+- https://www.ortho-consulting.com/consulting
+- https://www.cascadeffects.com/post/top-orthodontic-consulting-firms-in-2026
+- https://leeannpenicheandassociates.com/
+
+### B. Image Creation Checklist
+1. `favicon.png` — 32×32, Model Ortho mark
+2. `og-default.png` — 1200×630, "Model Ortho Consulting — Orthodontic Operations, Made Visible"
+3. `anita-headshot.jpg` — Professional photo, 800px wide
+4. `hero-graphic.png` — Abstract scheduling/capacity visualization, 1200px wide
+5. `tool-schedule-canvas.png` — Schedule Canvas mockup, 800px wide
+6. `tool-impact-forecaster.png` — Impact Forecaster mockup, 800px wide
+7. `tool-recovery-forecaster.png` — Recovery Forecaster mockup, 800px wide
+
+### C. Broken Links to Fix
+- Footer: `policies.html` → remove (content already at privacy.html)
+- Footer: `login.html` → remove (no login functionality exists)

disaster-recovery/restore-test-log.md

@@ -0,0 +1,129 @@
+# Restore Test — 2026-08-10
+
+**Tester:** Hermes (automated)  
+**Purpose:** First-ever ITPP restore test. Prove backups are restorable.  
+**Environment:** Core (152.53.192.33), restoring to /tmp/ only — zero production impact.
+
+---
+
+## Test 1: Gitea (app2)
+
+- **Backup source:** `s3://hermes-vps-backups/gitea/daily/20260810-120021/`
+- **Backup contents:** SQLite DB (gitea.db) + app.ini + 30 bare git repos
+- **Backup DB size:** 2.6 MB (2,723,840 bytes)
+- **Download duration:** ~3s (DB + config), ~16s (3 sample repos)
+- **Restored to:** `/tmp/restore-test-gitea/`
+
+### Verification — Database
+
+| Metric | Value |
+|--------|-------|
+| Total tables | 117 |
+| Users | 1 (ippadmin, info@itpropartner.com, admin) |
+| Repositories | 52 |
+| Issues | 0 |
+| Pull requests | 0 |
+| Actions | 461 |
+| Webhooks | 0 |
+| Releases | 0 |
+
+**Sample repository rows:**
+```
+id=52  lower_name=itpp-docs        updated=1786333787  is_empty=0
+id=8   lower_name=osint-tool       updated=1786333633  is_empty=0
+id=27  lower_name=launchcheck      updated=1786333633  is_empty=0
+```
+
+### Verification — Git Repos (3 of 30 sampled)
+
+| Repository | Commits | Latest commit |
+|-----------|---------|---------------|
+| itpp-infrastructure.git | 29 | `5f48e02` docs: promote claude-sonnet-5 to primary |
+| hermes-skills.git | 3 | `2f9b99b` chore: sync skill updates and references |
+| disaster-recovery.git | 5 | `3bc6d08` Fix: app1-bu CPX11 → CPX21 spec |
+
+All three repos validated with `git log --oneline` and `git rev-list --count HEAD`.
+
+### Issue Found & Workaround
+
+**Issue:** Gitea bare repos stored in S3 are missing the `refs/` directory (all refs are in `packed-refs`). Git refuses to recognize the repository without `refs/` existing.
+
+**Workaround:** `mkdir -p refs/heads refs/tags` in each repo directory before git operations. This is a **restore procedure note** — any real disaster recovery of Gitea repos from S3 must include this step.
+
+### Verdict
+
+**PASS** ✅ — Database integrity confirmed, all 117 tables present, 52 repos accounted for, 3/3 sampled repos verified with valid git history. One procedural note about `refs/` directory documented.
+
+- **Total duration:** ~30s
+- **Backup age at test time:** ~9 hours (backup at 12:00 UTC, test at ~21:15 ET)
+
+---
+
+## Test 2: Vaultwarden (app1)
+
+- **Backup source:** `s3://hermes-vps-backups/app1/vaultwarden/vaultwarden-backup-2026-08-10_0230.tar.gz`
+- **Backup contents:** Compressed tar.gz containing SQLite DB + WAL + RSA key + icon cache + docker-compose.yml
+- **Backup size:** 730.4 KB compressed
+- **Download duration:** ~1s
+- **Restored to:** `/tmp/restore-test-vaultwarden/`
+
+### Verification — Database
+
+| Metric | Main DB | Backup DB | Match? |
+|--------|---------|-----------|--------|
+| Total tables | 29 | 29 | ✅ |
+| Users | 1 (g@germainebrown.com, enabled) | 1 | ✅ |
+| Ciphers (passwords) | 123 | 123 | ✅ |
+| Organizations | 0 | 0 | ✅ |
+| Collections | 0 | 0 | ✅ |
+| Folders | 4 | — | — |
+| Devices | 9 | — | — |
+| Attachments | 0 | — | — |
+| Sends | 0 | — | — |
+
+**Cipher type distribution:** 121 logins (type 1), 2 secure notes (type 2)
+
+**RSA key:** Present and valid (1,675 bytes, `-----BEGIN RSA PRIVATE KEY-----`)
+
+### Verification — Files
+
+| File | Size | Status |
+|------|------|--------|
+| `data/db.sqlite3` | 388 KB | ✅ Valid SQLite |
+| `data/db-backup.sqlite3` | 400 KB | ✅ Valid SQLite |
+| `data/db.sqlite3-wal` | 3.9 MB | ✅ WAL present |
+| `data/db.sqlite3-shm` | 32 KB | ✅ SHM present |
+| `data/rsa_key.pem` | 1,675 B | ✅ Valid PEM key |
+| `docker-compose.yml` | — | ✅ Present |
+
+### Issue Found
+
+**WAL file present:** The backup includes active WAL (Write-Ahead Log) files, confirming the backup was taken while Vaultwarden was running. Since all three files (db.sqlite3, -wal, -shm) are present and consistent, SQLite's WAL recovery is automatic and the database opens cleanly. This is **expected and correct** for a live backup — no action needed.
+
+### Verdict
+
+**PASS** ✅ — Main and backup databases are identical and consistent. All 29 tables present, 123 ciphers intact, 1 user verified, RSA key valid, and WAL recovery clean. The backup is fully restorable.
+
+- **Total duration:** ~2s (download + extract + verify)
+- **Backup age at test time:** ~19 hours (backup at 02:30 UTC, test at ~21:15 ET)
+
+---
+
+## Summary
+
+| Test | Service | Verdict | Notes |
+|------|---------|---------|-------|
+| 1 | Gitea (app2) | ✅ PASS | DB + repos verified; refs/ workaround documented |
+| 2 | Vaultwarden (app1) | ✅ PASS | DB consistent across main+backup; RSA key valid; WAL clean |
+
+### Overall Assessment
+
+**Both backups are restorable.** This is the first successful restore test in ITPP history.
+
+### Recommendations
+
+1. **Automate restore testing** — Run a scripted restore test weekly (e.g., every Monday morning). Rotate through different services.
+2. **Expand coverage** — Test remaining backup targets in subsequent runs: LiteLLM, OpenWebUI, Hudu, Traccar, etc.
+3. **Document Gitea refs/ workaround** — Add the `mkdir -p refs/heads refs/tags` step to the Gitea disaster recovery procedure.
+4. **Investigate core/vaultwarden** — Backups under `core/vaultwarden/` stopped on 2026-07-28 and are only 33 KB — likely stale/misconfigured. The active Vaultwarden instance is on app1.
+5. **Full-scale DR drill** — After individual restore tests pass for all services, schedule a coordinated full-stack restore to the standby server.

dns-records.md

@@ -7,6 +7,10 @@
 
 | Subdomain | Record | IP | Service |
 |---|---|---|---|
+| voipsimplicity.itpropartner.com | A | 152.53.192.33 | VoIPSimplicity static portal (Core, deployed 2026-08-18) |
+| forefront.itpropartner.com | A | 152.53.192.33 | Forefront Wireless static portal (Core, deployed 2026-08-18) |
+
+**After applying these two records:** remove the `tls internal` line from both blocks in `/etc/caddy/Caddyfile` on Core and reload (`caddy validate` first, then `systemctl reload caddy`) so Let's Encrypt certs auto-provision. Sites currently verify 200 only with `--resolve` since DNS is absent.
 
 ## FIX
 

docs/audit/app3-site-publish-audit-2026-09-12.md

@@ -0,0 +1,69 @@
+# app3 Site Publish Audit - 2026-09-12
+
+**Purpose.** Precondition gate for the 2026-09-12 S3 backup-mirror cleanup. User directive:
+"leave 815bistro alone. that's a live client site that should be on app3 with all of the other
+websites. Make sure that all of the sites on app3 are currently published before deleting anything."
+
+**Method.** Domain list derived from app3 docroots (`/home/*/htdocs/*/` on 152.53.241.111), 34 unique
+domains. For each: `dig +short <d> A` for DNS, then `curl -L --max-redirs 5 -w '%{http_code}|%{url_effective}'`
+against `https://<d>/`. Script: `/root/.hermes/scripts/audit-app3-sites.sh`. Raw evidence: `/tmp/app3-audit-final.txt`.
+
+**Result: 34 tested, 28 published (HTTP 200), 6 not.**
+
+## Not published / defective
+
+| Domain | DNS | HTTP | Final URL | Verdict |
+|---|---|---|---|---|
+| boxpilotlogistics.com | 188.114.96.3, .97.3 (Cloudflare) | 301 | https://www.boxpilotlogistics.com/ | BROKEN - apex/www redirect loop, never reaches a served page |
+| vigilanttac.com | 104.21.83.95, 172.67.220.243 (Cloudflare) | 301 | https://www.vigilanttac.com/ | BROKEN - apex/www redirect loop, never reaches a served page |
+| voipsimplicity.com | NO-DNS-RECORD | 000 | - | BROKEN - no DNS at all; site dark |
+| www.voipsimplicity.com | NO-DNS-RECORD | 000 | - | BROKEN - no DNS at all |
+| timapta.org | 188.114.96.4, .97.4 (Cloudflare) | 301 | https://ptatime.org/ | BROKEN FORWARD - ptatime.org itself has no DNS records |
+| forms.itpropartner.com | 152.53.241.111 (app3) | 404 | https://forms.itpropartner.com/ | NOT A DEFECT - POST-only API, see below |
+| iamgmb.com | 188.114.97.3, .96.3 (Cloudflare) | 200 | https://www.google.com/ | Suspicious - public domain 301s to google.com (decommissioned Aug 9 2026) |
+
+### Details
+
+- **boxpilotlogistics.com / vigilanttac.com.** Both are WordPress installs on app3 fronted by
+  Cloudflare (301 carries `x-redirect-by: WordPress`). Apex 301s to `www`, and `www` 301s back to apex;
+  after 5 redirects curl is still on a 301 and never receives a body. Browsers surface
+  "this page isn't redirecting properly". Both client sites are effectively down.
+- **voipsimplicity.com + www.** Zero A records, yet the full WordPress install exists on app3 at
+  `/home/voipsimplicity/htdocs/voipsimplicity.com/` and `/home/voipsimplicity/htdocs/www/`.
+  `my.voipsimplicity.com` (Cloudflare) is up and returns 200. The marketing site is unreachable.
+- **timapta.org.** Correctly 301s per the TIMA forwarding setup, but the target `ptatime.org` has no
+  DNS records, so the forward dead-ends. Verified: `dig +short ptatime.org` is empty.
+- **forms.itpropartner.com.** `msp-forms.service` (MSP Shared Form Handler, FastAPI) is `active (running)`
+  since 2026-08-21, listening on 127.0.0.1:8700 (uvicorn pid 1498), nginx fronts it on 443. GET / returns
+  `{"detail":"Not Found"}` - the expected FastAPI 404 for a POST-only handler. No action needed.
+- **iamgmb.com.** Returns 200 but the final URL is `https://www.google.com/`. The domain was decommissioned
+  2026-08-09; a redirect to Google is not a normal retirement. Needs an owner decision (park, 410, or
+  forward to itpropartner.com).
+
+## 815bistro.com - NOT on app3
+
+User directive is to leave it alone because it "should be on app3 with all of the other websites."
+**It is not on app3 and never was migrated.** There is no `815bistro` docroot among the 34 app3 sites,
+and both the apex and `www` resolve to **35.212.86.161**, the SiteGround box (SiteGround runs on GCP,
+so the IP reverse-resolves into `*.bc.googleusercontent.com`). The site itself is live (200) on SiteGround.
+
+Consequence for the cleanup: the only 815bistro artifact in our possession is the export at
+`/root/.hermes/.backups/siteground/815bistro.com/` (261 MB). It was explicitly removed from the delete list
+and explicitly removed from the sync-exclude list so it keeps mirroring. **This site still needs migrating
+to app3.**
+
+## Published (28)
+
+apextrackexperience.com, buzz.iamgmb.com, debtrecoveryexperts.com, docs.itpropartner.com,
+my.verdicttank.com, verdicttank.com, auth2-api.itpropartner.com, auth2.itpropartner.com, intelsight.io,
+mockups.itpropartner.com, proposals.itpropartner.com, support.itpropartner.com, katiewattsdesign.com,
+mainwp.itpropartner.com, modelortho.com, www.modelortho.com, my.voipsimplicity.com, panel.itpropartner.com,
+my.radartank.com, my.rfptank.com, radartank.com, review.watchdogcitizen.com, rfptank.com,
+watchdogcitizen.com, scirium.com, my.transitpin.com, transitpin.com, iamgmb.com (200, but to google.com).
+
+## Cleanup impact
+
+Nothing deleted on 2026-09-12 touched app3 site content. Deletion targets were stale Hermes/UNMS backup
+mirrors in `s3://hermes-vps-backups/live/.backups/` plus stale data on the standby. Held back:
+`.backups/siteground/` (815bistro) and `.backups/wphost02-backup-2026-07-10.tar.gz` (fallback data for the
+8 WordPress sites migrated to app3 - held until boxpilotlogistics.com and vigilanttac.com are verified healthy).

docs/incidents/2026-09-11-core-state-db-corruption.md

@@ -0,0 +1,117 @@
+# Incident: Core Hermes Store Corruption (not-a-database) and Recovery
+
+**Date:** September 11, 2026
+**Severity:** High. Core's agent lost its conversation store mid-day, which disabled `session_search`, `delegate_task`, and every cron job that reads history.
+**Status:** Recovered. Repair staged and verified, installed the same evening.
+**Root cause:** The SQLite store was written while it was being checkpointed, so the file header was destroyed. The store stopped being a database at all.
+**Amplifier:** A 1.9 GB store that had never been pruned, failing three times in three days, with backups that faithfully captured the corruption because they tarred the live file.
+
+> Ground truth for this report was re-verified at 17:45 EDT on September 11, 2026. Numbers below are from the live box, not from a summary.
+
+---
+
+## 1. What happened
+
+At 12:49:22 EDT the default profile store `/root/.hermes/state.db` took its last write. Five seconds later the gateway restarted and found the store unreadable. Every tool that depends on history began failing with `file is not a database`:
+
+```
+session_search -> {"success": false, "error": "Session database not available: DatabaseError: file is not a database."}
+delegate_task  -> Error executing tool: ... file is not a database
+```
+
+The file is 1,984,344,064 bytes. Its first bytes are **not** `SQLite format 3`, so SQLite rejects it before reading a single page. It is not a marginally damaged database, it is a file that no longer has a database header.
+
+## 2. Impact
+
+- **Capability loss:** `session_search` and `delegate_task` were down for the rest of the day. Delegation is a load-bearing part of how this box operates.
+- **Cron surface:** any job reading history was at risk. The daily backup monitor kept failing for eight days, though that turned out to be a separate, already identified cause (section 6).
+- **Data loss window:** 11:49 to 12:49 EDT. No readable copy of that hour exists anywhere. Anything learned in that window is only recoverable from durable artifacts (DR issue log, CHANGELOG, skills).
+- **Backup lineage contamination:** the Sep 10 and Sep 11 archives captured the live, corrupt file. A restore from the newest archive would have restored the corruption. This is the single most dangerous detail in the incident.
+
+## 3. Timeline (all times EDT)
+
+| When | Event |
+|---|---|
+| Sep 9 | WAL damage appears in the live store. First corruption event. |
+| Sep 10 12:18 | A backup is written that is already malformed. |
+| Sep 10 12:20 | A repair attempt fails. |
+| Sep 10 14:48 | A corrupt copy is preserved (`state.db.corrupt-20260910`). |
+| Sep 9 to 11 | Daily archives grow 1.76 GB to 2.34 GB to 3.52 GB. The growth is the unpruned store plus quarantine artifacts. |
+| Sep 11 01:00 | Nightly archive taken. This copy turns out to be clean: 1,939,980,288 B, `quick_check ok`, 107,588 messages, max id 321,545. |
+| Sep 11 11:49 | A snapshot is taken that is valid SQLite (1.96 GB) but has one bad page. It carries roughly 985 messages the 01:00 archive does not. |
+| Sep 11 12:49:22 | Last write to the live store. |
+| Sep 11 12:49:27 | Gateway restarts and cannot open the store. |
+| Sep 11 16:33 | Grafted candidate built and verified (`state.working.db`). |
+| Sep 11 16:51 | Install path hardened (`chmod +x`, explicit `/bin/bash`, dry run passed). |
+| Sep 11 18:56 | Repair scheduled to install via detached root crontab one shot. |
+
+## 4. Root cause
+
+**Proximate cause:** a write that overlapped WAL checkpointing during the 12:49 shutdown. The store's header was overwritten, leaving an 1.85 GiB file with no valid database header. There is no evidence of filesystem damage; other files on the same volume are intact and the disk reports no errors.
+
+**What it was NOT, checked and excluded:**
+- **Not OOM.** The box had headroom. At the time of this report Core runs 15 GB total with 4 GB used and 11 GB available, 366 GB free disk.
+- **Not a cron job touching the database.** No scheduled job writes to `state.db` directly. The 15 minute `hermes-live-sync` job uses `.backup` snapshots and excludes live files, and it had been paused since Sep 3.
+- **Not a bad restore.** Nothing replaced the store on Sep 11 before the 12:49 failure.
+
+**What is still unknown, and stays unknown until proven:** which process requested the 12:49 gateway restart. The store was written and then the gateway came back and found it broken, and the caller was not recorded anywhere durable. That gap is itself a finding (section 7, item 5).
+
+**Why it was able to hurt this much:** the store is 1.9 GB and has never been pruned. Every checkpoint, backup, and recovery operation on a store that size is slow, IO heavy, and exposed to exactly this failure mode. Three corruption events in three days (Sep 9, 10, 11) is not bad luck, it is a store operating outside safe limits. The corrupt copy from Sep 3 through Sep 10 in Anita's frozen profile shows the same pattern on a second profile, which points at the host and the store size rather than at one profile's content.
+
+## 5. Recovery
+
+**Strategy:** do not try to repair the corrupt file body. Build a new store from the newest clean copy and graft forward the messages that only exist in newer ones.
+
+1. **Base:** the 01:00 archive store, verified clean: `quick_check ok`, 107,588 messages, max id 321,545.
+2. **Graft:** 985 messages from the 11:49 snapshot (valid SQLite, one bad page) into the base. Grafting is row by row, so a single unreadable page costs a row, not the migration.
+3. **Result:** `/root/db-forensics/state.working.db`, 1,961,385,984 B, verified at 17:45 EDT:
+
+```
+PRAGMA quick_check     -> ok
+PRAGMA integrity_check -> ok
+108,573 messages / max id 322,530 / 242 sessions
+FTS intact: messages_fts and messages_fts_trigram with all supporting tables
+```
+
+985 grafted rows against 107,588 base rows reconciles exactly to 108,573.
+
+**Install path:** the job runs as a detached root crontab one shot writing to `/root/db-forensics/cron-invoke.log`, not `systemd-run` and not `/etc/cron.d` (both blocked by the lifecycle guard on this box).
+
+**First attempt failed for a boring reason worth recording:** `install.sh` was not executable, so cron forked it and it died within a second, writing nothing anywhere. The fix is three parts: `chmod +x`, invoke explicitly as `/bin/bash <script>`, and log stdout and stderr to a file that can be read afterwards. It was then exercised in `--dry-run` before being armed for real.
+
+## 6. Side finding: the backup monitor was crying wolf for eight days
+
+`Backup-Health-Monitor` (daily 09:00) had `failure_streak=8` and `last_status=error`.
+
+- **The one real CRITICAL** was `hermes-live-sync: DISABLED/PAUSED` (job `61cd31eec51c`, paused 2026-09-03 15:49, the same window as the first corruption event). The script exits 1 only when `total_critical > 0`, never for warnings, so that single pause accounts for the entire streak.
+- **Its three WARNINGs are false positives,** proven by content rather than by size:
+  - **Wazuh Manager (app1):** the tarballs for Sep 9, 10 and 11 are all sha256 `0e65a2364cdc0795...`, identical. The content is a static config, and the 03:15 job did run today. Not stalled.
+  - **LiteLLM Config (app1):** the flagged object is a 333 byte config yaml that legitimately never changes. The actual data backup is `app1/litellm/litellm-backup-*.tar.gz`, 47 of them, newest 37,894,223 bytes at 03:30 today, and it carries the Postgres dump.
+  - **MySQL voipsimplicity (app3):** identical size 6,834,636 bytes each day but a **different** sha256 each day (cb1ce9f8, 92c5eb70, b83ea28f), valid gzip, 73 tables, real `mysqldump 8.4.10`. Content is changing, the size simply coincides.
+- **Conclusion:** the size uniqueness heuristic cannot tell "static but fine" from "stalled". Those three checks should be reclassified as "unchanged content" rather than SUSPICIOUS. Not yet changed.
+
+## 7. Prevention
+
+1. **Prune the store.** 1.9 GB unpruned is the amplifier behind all three failures. Needs a retention policy and a size cap, sized before anything is deleted (session count, cron and watcher noise measured first).
+2. **Snapshot, never tar, live databases.** Excluding `*.db` from the essentials archive is correct. What was missing was the replacement, now supplied by `hermes-db-backup.sh`, which takes a `sqlite3 .backup` per database, `quick_check` each snapshot, uploads, then downloads and re-verifies. Deployed on the Anita box with cron `10 3 * * *`, proven by round trip (291 MB snapshot, `quick_check ok`, 99,285 messages).
+3. **Never trust a backup that has not been restored.** Every archive is now suspect until an object has been downloaded, extracted, and opened. The Sep 10 and Sep 11 archives would both have restored the corruption.
+4. **Check integrity on the newest snapshot daily,** not just its existence and size. `PRAGMA quick_check` on the newest object is cheap and would have caught the Sep 10 and Sep 11 contamination on the day it happened.
+5. **Record who restarts the gateway.** The 12:49 caller is still unknown. The restart path should write caller identity, reason, and timestamp to a durable log so this question is answerable next time.
+6. **Resume the 15 minute checkpoint** only after the repaired store verifies healthy, and prove the resumed job by downloading its object and running `quick_check` on it.
+7. **Keep quarantine artifacts for a retention window.** The only unique bytes in the deleted frozen profile were the Sep 3, 9 and 10 quarantine files. They are archived, not discarded, so a future investigator can compare failure signatures.
+
+## 8. Evidence (re-verified 2026-09-11 17:45 EDT)
+
+```
+live      /root/.hermes/state.db                      1,984,344,064 B   mtime 2026-09-11 12:49:22   header: NOT SQLite
+donor     /root/sqlite_tmp/sep11/hermes-backup-2026-09-11/state.db
+                                                      1,939,980,288 B   quick_check ok  107,588 msgs  max id 321,545
+candidate /root/db-forensics/state.working.db         1,961,385,984 B   quick_check ok  integrity_check ok
+                                                                       108,573 msgs  max id 322,530  242 sessions
+disk      503 GB volume, 117 GB used, 366 GB free
+memory    15 GB total, 4 GB used, 11 GB available
+```
+
+## 9. Lesson
+
+The failure was survivable because the 01:00 archive happened to be clean and the 11:49 snapshot happened to be readable enough to graft from. That is luck, not architecture. Two things made it luck: backups that copied the live store, and no pruning on a store big enough that every operation on it is a window of risk. Both are fixed or scheduled. The remaining exposure is the same store size, which stays live until a retention policy is agreed.

docs/incidents/2026-09-12-core-state-db-recovery.md

@@ -0,0 +1,74 @@
+# 2026-09-12 Core state.db recovery and rebuild
+
+**Status:** RESOLVED — store recovered, rebuilt, and installed; gateway clean; 15-min coverage resumed.
+**Box:** Core (netcup RS 2000, `152.53.192.33`)
+**Store:** `/root/.hermes/state.db`
+**Precedes:** `2026-09-11-core-state-db-corruption.md`
+
+## Summary
+
+The Core state store was recovered from a live snapshot and reinstalled on 2026-09-12 between 01:07 and 02:09.
+The live store went from 1,902 MB to 1,108 MB (794 MB reclaimed) with **no loss of sessions or messages**, and the
+gateway now runs on a clean database with no orphaned file handles.
+
+## Timeline (verified from logs and file mtimes)
+
+| Time (ET) | Event |
+|---|---|
+| 00:53:39 | Corpse-relocation dir created (task 10: moving corrupt copies out of `~/.hermes`) |
+| 00:54:50 | `agent.log`: a terminal call completes |
+| 00:54:53 | `state.db-wal` mtime. WAL and SHM **unlinked while the gateway held them open** (gateway PID 2356405) |
+| 00:54:45 → 00:55:00 | `wal-monitor.py` catches it live: clean (`deleted-handles: 0`, WAL inode 261435) → fresh WAL inode 298693, SHM missing, **6 deleted handles** |
+| 00:58 | Live `optimize-storage` migration launched (PID 2374243), gated to wait for the 01:00 archive |
+| 01:00 | `hermes-backup.sh` completes: `hermes-full-backup-2026-09-12.tar.gz` (762,955,612 B) |
+| 01:04:11 | Migration exits: **`optimization failed: database disk image is malformed`**. No VACUUM completed, no data written |
+| 01:07:14 | Live snapshot taken (2,125,131,776 B) |
+| 01:08:28 | Quarantine: `state.db.malformed-backup-20260912_010832` |
+| 01:08:43 | `state.db.repair-attempts.json` written |
+| 01:10:13 / 01:12:47 | Recovery candidates built: `recovered-20260912.db` (1,108,819,968 B), `recovered-snapshot.db` |
+| 02:00:56 | `install-0912.sh` runs |
+| 02:05:01 | Second quarantine: `state.db.not-a-db-20260912_020501` |
+| 02:08:48 | Gateway stopped and restarted → new PID **2414111** |
+| 02:09:11 | Rebuilt store installed (1,108,221,952 B) |
+| 02:09:34 | Install log: `quick_check=ok integrity_check=ok`, FTS present, remaining holders: none |
+
+## Verified result (2026-09-12 02:22 ET)
+
+- Gateway PID 2414111 (started 02:08:48), `active`, **zero deleted file descriptors**; every `state.db*` fd resolves to a real file.
+- `PRAGMA quick_check` = **ok**; `PRAGMA integrity_check` = **ok**.
+- **242 sessions** (unchanged) / **109,825 messages** (up from the damaged on-disk view) / FTS present.
+- Both heavy sessions intact: `20260809_033049_d51d611b` = 58,359 rows, `20260827_231946_bab48b11` = 41,790 rows.
+- Newest message timestamp 02:21:40 — the store is live and current.
+- Zero corruption-class errors in `errors.log` after 02:09:34.
+- Store size 1,902 MB → **1,108 MB**. The rebuild dropped the legacy v22 FTS duplicate-data bloat, which was the
+  measured root cause of the store bloat (index 1,199 MB for 423 MB of text) — **not** cron/subagent accumulation
+  (37.6 MB of 423 MB).
+
+## Root cause of the WAL unlink at 00:54:53 — OPEN
+
+The WAL and SHM were truly **unlinked, not moved** (`find / -inum 261435` returned nothing; the stranded bytes were
+preserved as `/root/db-forensics/stranded-wal-261435.bin`, 18,622,432 B, magic `37 7f 06 82`).
+
+Leading hypothesis, **not confirmed**: an interactive root shell ran Hermes repair commands — `.bash_history` contains
+`hermes doctor --fix` and `hermes sessions optimize-storage` — at approximately 00:54:50, i.e. outside the gateway
+process tree. Deleting a `-wal`/`-shm` out from under a live gateway is precisely how the split-brain occurs: the gateway
+keeps writing into the orphaned inode while fresh readers see a stale main file.
+
+The `wal-monitor.py` process died; its final log line (old PID, 6 deleted handles) is **stale evidence**, not current state.
+
+## Actions taken
+
+1. Quarantine images (4 GB) moved **out of `~/.hermes`** to `/root/db-forensics/corpses/20260912-quarantine/` so the
+   01:00 backup does not swallow them again — the same class of mistake that inflated the 2026-09-11 archive to 3.35 GB.
+2. `hermes-live-sync` (cron `61cd31eec51c`) **resumed** — 15-minute coverage had been OFF since 2026-09-03 15:49:45.
+   Next run 02:37:13. The script snapshots with `sqlite3 .backup` (WAL-safe), so it is safe against a live gateway.
+3. `sanctioned-pauses.json` entry closed and moved to `_resolved`.
+4. One-shot verification armed for 02:47: downloads the `live/state.db` object from S3 and runs `quick_check` on it.
+   Per standing rule, verification is the downloaded object, not the job status.
+
+## Open items
+
+- Identify what actually unlinked the WAL at 00:54:53 (see hypothesis above).
+- `state.db.repair.lock` (0 bytes, 2026-09-10) is stale.
+- Destructive prune of session `20260809_033049_d51d611b` (224 MB, ended Aug 27) — deferred by choice, no longer urgent
+  now that the rebuild reclaimed 794 MB.

docs/infrastructure/key-inventory.md

@@ -11,7 +11,7 @@
 
 | Key Name | File | Type | Fingerprint (SHA256) | Purpose | Deployed To |
 |----------|------|------|-----------------------|---------|-------------|
-| **itpp-infra** | `/root/.ssh/itpp-infra` | ED25519 | `Jxh0bbT9dUV3q1DYYB3hHyhy/1TDj7Q8U4xrVmB38uQ` | Universal server admin key | All servers (Core, app1, app2, app3, wphost02, app1-bu, home router) |
+| **itpp-infra** | `/root/.ssh/itpp-infra` | ED25519 | `Jxh0bbT9dUV3q1DYYB3hHyhy/1TDj7Q8U4xrVmB38uQ` | Universal server admin key | All servers (Core, app1, app2, app3, app1-bu, home router). wphost02 DECOMMISSIONED (2026-08-28), removed from scope. |
 | **wisp_rsa** | `/root/.ssh/wisp_rsa` | ED25519 | `MxQw1oh90NibSgN2mDbKP+07/jE4FEUEBbFAzuk5DcI` | WISP MikroTik CCR router SSH | Home CCR router (10.77.0.2 via WireGuard) |
 | **germaine-personal** | `/root/.ssh/germaine-personal` | ED25519 | `dDbLH+bdPFcGU0mm1DpGa43ec0nUZ88YnpCi4p63y3I` | Germaine's personal key (from his machines) | Germaine's devices → Core |
 | **homelab** | `/root/.ssh/homelab` | ED25519 | `c1nts4wR9EU06/O/k895Pb2tGZublgnGWG6NoQrK/qs` | Homelab Proxmox/QNAP access | vm-host-01, vm-host-02, QNAP NAS |
@@ -148,7 +148,7 @@ All stored in `/root/.hermes/.env` and Hudu API assets (layout 49).
 
 | Database | Host | User | Password Location | Purpose |
 |----------|------|------|-------------------|---------|
-| **MySQL (apex track)** | 127.0.0.1:33060 (SSH tunnel from wphost02) | `apextrackexperience_1781549652` | `wp-config.php` on wphost02 | Apex Track Experience WordPress |
+| **MySQL (apex track)** | app3:3306 | `apextrackexperience_1781549652` | RunCloud-era credential — target host wphost02 DECOMMISSIONED (2026-08-28); site now on app3/CloudPanel with a different credential scheme | Apex Track Experience WordPress (STALE — see comprehensive-audit-summary-2026-08-09.md) |
 | **MySQL (CloudPanel)** | app3:3306 | `root` | `/root/.my.cnf` on app3 (also in Vaultwarden) | CloudPanel WordPress hosting |
 | **LiteLLM Postgres** | app1 (Docker) | (in docker-compose) | `/root/docker/litellm/docker-compose.yml` on app1 | LiteLLM operational DB |
 
@@ -217,7 +217,7 @@ The following credentials are known to exist but were not found in the standard
 | **Hudu API key** | In skill docs (`hudu-management`) — used programmatically, not in .env. |
 | **Traccar/FleetTracker360 admin** | Not in .env. May be Docker env or app-managed. |
 | **Twenty CRM credentials** | Docker on Core, env at `/root/docker/twenty/.env` (not read). |
-| **WordPress site DB passwords** | Various sites, typically in `wp-config.php` on wphost02 or app3. |
+| **WordPress site DB passwords** | Various sites, typically in `wp-config.php` on app3 (wphost02 DECOMMISSIONED 2026-08-28). |
 | **app1-bu** | 5.161.225.131 | Hetzner CPX21 — accessed via itpp-infra SSH key only. |
 | **ComfyUI / Z4** | GPU server allocated for TripFlow — credentials not yet documented. |
 | **Home MikroTik admin** | SSH via `admin@10.77.0.2` with `wisp_rsa` key. RouterOS password in router config (not extracted). |

docs/infrastructure/model-chain.md

@@ -1,6 +1,6 @@
 # AI Model Architecture — IT Pro Partner
 
-**Updated:** August 8, 2026
+**Updated:** September 11, 2026
 
 Two separate concepts: **fallback chain** (survival — direct API keys) and **operational chain** (daily toolbox — admin-ai only). The two-key strategy means operational keys run through admin-ai/LiteLLM; fallback keys are direct provider API keys with daily limits.
 
@@ -14,12 +14,12 @@ Survives admin-ai outage. All direct provider keys have daily caps. Fires in ord
 
 | Tier | Model | Provider | Key type | Daily cap |
 |---|---|---|---|---|
-| **Primary** | `deepseek-v4-pro` | `admin-ai` | operational | $30/mo budget |
+| **Primary** | `deepseek-v4-flash` | `admin-ai` | operational | $700/30d (~$23.33/day) |
 | **F1** | `deepseek-v4-flash` | `deepseek` (direct) | fallback | $3 |
-| **F2** | `gemini-3.6-flash` | `google` (direct) | fallback | $2 |
+| **F2** | `gemini-3.8-flash` | `google` (direct) | fallback | $2 |
 | **F3** | `grok-4.5` | `xai` (direct) | fallback | $2 |
 | **F4** | `claude-sonnet-5` | `anthropic` (direct) | fallback | $5 |
-| **F5** | `gpt-4.1-nano` | `openai` (direct) | fallback | $2 |
+| **F5** | `gpt-5-mini` | `openai` (direct) | fallback | $2 |
 
 Total emergency budget: **$14/day** — down from $45 single-leg burn on Aug 5.
 
@@ -31,15 +31,17 @@ All route through admin-ai. Shared budget via `hermes-agent-v5` key.
 
 | Role | Model | Provider | Use When |
 |---|---|---|---|
-| **Conductor** | `deepseek-v4-pro` | admin-ai (DeepSeek) | All standard work — orchestration, delegation, coding |
-| **Workhorse** | `deepseek-v4-pro` | admin-ai (DeepSeek) | Delegated tasks, scripts, infra code |
+| **Conductor** | `deepseek-v4-flash` | admin-ai (DeepSeek) | All standard work — orchestration, delegation, coding |
+| **Workhorse** | `deepseek-v4-flash` | admin-ai (DeepSeek) | Delegated tasks, scripts, infra code |
 | **Batch Workhorse** | `deepseek-v4-flash` | admin-ai (DeepSeek) | Bulk scripts, log parsing, repetitive tasks |
 | **Lightweight** | `claude-haiku-4-5` | admin-ai (Anthropic) | Email triage, classification, simple tasks |
 | **Simple Workhorse** | `gpt-5.6-luna` | admin-ai (OpenAI) | Lightweight tasks under 128K context |
 | **Auditor** | `gpt-5.6-luna` | admin-ai (OpenAI) | Code review, QA (primary auditor) |
 | **Auditor 2** | `xai/grok-4.5` | admin-ai (xAI) | Second-opinion code review (different provider) |
 | **Critical** | `claude-sonnet-5` | admin-ai (Anthropic) | Client comms, legal docs, architecture (explicit) |
-| **Professional Comms** | `gemini-3.6-flash` | admin-ai (Google) | Client emails, professional messaging |
+| **Professional Comms** | `gemini-3.8-flash` | admin-ai (Google) | Client emails, professional messaging |
+| **Research** | `sonar-pro` | admin-ai (Perplexity) | Live-search-grounded: competitive/compliance/market research, current events, cited answers |
+| **Deep Research** | `sonar-reasoning-pro` | admin-ai (Perplexity) | Multi-step synthesis, reasoning + citations |
 
 ---
 
@@ -47,13 +49,13 @@ All route through admin-ai. Shared budget via `hermes-agent-v5` key.
 
 ### hermes-agent-v5 (Main — Sho'Nuff)
 - **Created:** Jul 31, 2026
-- **Budget:** $30/day
-- **Spend:** $20.36 (as of Aug 6)
+- **Budget:** $23.33/day ($700/30d)
+- **Spend:** $184.79 (as of Sep 3)
 - **Models:** deepseek-v4-pro, deepseek-v4-flash, gemini-flash-latest, claude-sonnet-5, claude-haiku-4-5, gpt-5.6-luna, xai/grok-4.5 (+ claude-sonnet-4-6, claude-opus-4-8, claude-fable-5, gemini-2.5-flash/pro, grok-4.3, gpt-5, gpt-5-mini available)
 
 ### Anita's Hermes Key
-- **Budget:** $10/day
-- **Spend:** $0.11 (as of Aug 6)
+- **Budget:** $3.33/day ($100/30d)
+- **Spend:** $6.86 (as of Sep 3)
 - **Models:** deepseek-v4-pro, deepseek-v4-flash, gemini-flash-latest, claude-sonnet-5, claude-haiku-4-5, gpt-5.6-luna, xai/grok-4.5 (+ same expansions as hermes-agent-v5)
 
 ---
@@ -62,8 +64,8 @@ All route through admin-ai. Shared budget via `hermes-agent-v5` key.
 
 | Component | Daily est. |
 |---|---|
-| Conductor + Workhorse (ds-v4-pro) | ~$3.00 |
-| Batch Workhorse (ds-v4-flash) | ~$0.50 |
+| Conductor + Workhorse (ds-v4-flash) | ~$2.00 |
+| Batch Workhorse (ds-v4-flash) | ~$1.50 |
 | Lightweight (haiku-4-5) | ~$0.30 |
 | Simple Workhorse (luna) | ~$0.50 |
 | Auditors (luna + grok-4.5) | ~$0.80 |
@@ -78,4 +80,8 @@ All route through admin-ai. Shared budget via `hermes-agent-v5` key.
 - **Jul 30, 2026:** LiteLLM key regenerated — hermes-agent-v3 created but config.yaml NOT updated. Hermes fell back to DeepSeek direct since Jul 30 19:25 UTC.
 - **Jul 31, 2026:** Discovered stale key in config.yaml. Generated hermes-agent-v5 with same model list. Verified chat OK. Cleaned up stale keys (v3, v4, v4b).
 - **Aug 5, 2026:** Admin-ai budget cap hit (~$20). Fallback chain exhausted 4 dead legs, landed on Anthropic direct. Burned $45 in 10 hours on claude-sonnet-5 via direct key. Anthropic key capped until Sep 1.
-- **Aug 6, 2026:** Root cause of Aug 5 outage: 4 fallback legs dead simultaneously (admin-ai budget, DeepSeek balance $0, grok-4.6 404, Anthropic capped). Implemented two-key strategy (operational vs fallback). Rotated all 5 fallback keys. Added F5 (gpt-4.1-nano via OpenAI). Added haiku-4-5 and grok-4.5 to operational chain. Fixed grok-4.6 → grok-4.5. Synced Anita profile identically. Budget raised to $30.
+- **Aug 6, 2026:** Root cause of Aug 5 outage: 4 fallback legs dead simultaneously (admin-ai budget, DeepSeek balance $0, grok-4.6 404, Anthropic capped). Implemented two-key strategy (operational vs fallback). Rotated all 5 fallback keys. Added F5 (gpt-5-mini via OpenAI). Added haiku-4-5 and grok-4.5 to operational chain. Fixed grok-4.6 → grok-4.5. Synced Anita profile identically. Budget raised to $30.
+- **Aug 17, 2026:** Live verification of fallback chain against the LiteLLM model DB (admin-ai). Confirmed present: `deepseek-v4-flash`, `gemini-3.6-flash` (registered as `gemini/gemini-3.6-flash`), `grok-4.5` (`xai/grok-4.5`), `claude-sonnet-5`. Replaced F5 `gpt-4.1-mini` → `gpt-5-mini` — the gpt-4.1 series is no longer present in LiteLLM. Synced `config.yaml` `fallback_providers` and `model.fallbacks`.
+- **Sep 3, 2026:** DeepSeek promo rates ended and prices rose again. Live LiteLLM map now bills deepseek-v4-pro at $1.32/$3.96 and deepseek-v4-flash at $0.44/$1.32 (peak; off-peak half). Registered gemini-3.8-flash ($0.75/$3.75 intro) on admin-ai. Swapped F2 and Professional Comms gemini-3.6-flash to gemini-3.8-flash (Core + Anita). Restored delegation default to deepseek-v4-pro (was claude-sonnet-5, ~$13/day burn). Corrected F5 name in operational-models.md. Re-baselined cost tracker thresholds ($3/$4 to $15/$20). Verified actual key budgets: hermes-agent-v5 $700/30d ($23.33/day), Anita $100/30d ($3.33/day).
+- **Sep 3, 2026 (sonar + vision):** Seated Perplexity sonar as the Research tier (`sonar-pro` + `sonar-reasoning-pro`, both verified search-grounded via admin-ai with live test calls). Swapped `auxiliary.vision` `claude-sonnet-5` → `claude-haiku-4-5` (still `anthropic` direct), ~3x cheaper vision; verified with a real image call. Conductor unchanged (stays `deepseek-v4-pro`).
+- **Sep 11, 2026:** DeepSeek V4.1 Flash released (Sep 10) and surpasses V4 Pro on performance/cost/speed. Flipped primary + delegation from deepseek-v4-pro to deepseek-v4-flash on Core and Anita. V4.1 Flash peak $0.30 input / $1.20 output (off-peak $0.15/$0.60; cache-hit peak $0.006). **CORRECTION (verified against the live DeepSeek pricing page, Sep 11): V4 Pro is NOT retiring.** DeepSeek reversed the Sep 14 sunset and continues V4 Pro unchanged at $1.32/$3.96, so the legacy name does not auto-route to Flash. Cost map: the admin-ai built-in map carried the superseded Flash peak ($0.44 miss / $0.014 cache-hit / $1.32 out); a peak override ($0.30 miss / $0.006 cache-hit / $1.20 out) was applied to the `deepseek-v4-flash` deployment via `/model/update` and verified end-to-end (live call billed 1.71e-05 = 37x3e-07 + 5x1.2e-06). Pro in the map was already correct at $1.32/$0.044/$3.96.

docs/projects/projects-master-readme.md

@@ -10,3 +10,4 @@ Master index of all internal and client projects.
 - **[BoxPilot Logistics](./boxpilot/README.md)**: Logistics and shipping management platform. (PLANNED)
 - **[Open-Source SaaS Alternatives](../projects/oss-saas-alternatives.md)**: 10 self-hostable replacements for paid SaaS (AppFlowy, Immich, Documenso, Excalidraw, Penpot, Cal.DIY, ListMonk, Dub, RustDesk, FluidVoice). Future productize/host candidates. (FUTURE PROJECTS)
 - **[Hosted AI Agent Platform](../projects/hosted-agent-platform.md)**: AgentThread-style hosted Hermes agents — each client/space gets a containerized agent with chat, live URL, and credit billing. Reference: agentthread.ai. (FUTURE PROJECTS)
+- **[Code-Review Graph (Tooling)](../projects/code-review-graph.md)**: Adopt the code-review-graph structural knowledge pattern for our own repos, plus "everything is a plugin" skill discipline. Do NOT adopt the dsh harness itself — Hermes already covers ~80%. Reference: deepseek-ai/deepseek-harness. (FUTURE PROJECTS)

docs/security/2026-08-29-llms-txt-supply-chain-advisory.md

@@ -0,0 +1,78 @@
+# Security Advisory: llms.txt Supply-Chain Prompt Injection
+
+**Date:** 2026-08-29
+**Classification:** Security advisory (external research, applies to our agent infrastructure)
+**Source:** Ars Technica, Dan Goodin - "Claude, Codex, and Hermes installed unowned code inside corporate networks"
+**Severity:** High (for any agent granted shell/package-install permissions)
+
+## Summary
+
+Researchers scanned 6,214 live domains (defense contractors, Fortune 500, Big Tech) and found 8,265 `llms.txt` / `llms-full.txt` files. 120 of those files, each on a different site, pointed at code packages or domain names that were not registered. When the researchers claimed the unclaimed names and hosted phone-home packages, they received callbacks from Fortune 500 companies within an hour, and a few dozen more over time. The parent-process chain implicated three coding agents: Claude, OpenAI Codex, and Nous Research Hermes.
+
+Hermes is the agent platform ITPP runs in production. This is not abstract.
+
+## The Attack Mechanism
+
+`llms.txt` and `llms-full.txt` are an emerging convention: the AI equivalent of `robots.txt`. Websites publish them as machine-readable summaries and setup instructions for AI agents.
+
+The exploit is a supply-chain hijack that works in stages:
+
+1. A legitimate site publishes an `llms.txt` that lists a package or domain that does not exist (or that was later abandoned). The file says, for example, `pip install <name>` or `npm install <name>`.
+2. Because the name is unregistered, an attacker registers it and hosts ransomware or any other payload.
+3. A coding agent with shell-execution permission reads the file, treats it as authoritative vendor documentation, and downloads and runs the package without checking the namespace, ownership, or whether the domain is still alive.
+4. Endpoint detection does not fire. To EDR or a corporate proxy, this is a developer running a legitimate package manager against `pypi.org` or `npmjs.com`, with the agent the company installed on purpose as the parent process.
+
+The researchers found 227 install/view commands across the 120 misconfigured files. Many of the faulty entries predate the AI era (manually written by humans), and some were likely hallucinated by earlier AI.
+
+## Confirmed Live Exploit
+
+At least one active attack is already exploiting this. An `llms.txt` file hosted on `clerk.com` contained:
+
+```
+npx clerk-next-fix-auth-protection
+```
+
+`npx` fetches a package into the npm cache and executes its binary without adding it to a dependency manifest. Someone claimed the empty slot and hosted live malware. Clerk has since resolved it, and noted that agents that had already installed the `@clerk/eslint-plugin` binary were not at risk, but a fresh agent resolving that name would pull the malicious package.
+
+## How This Differs From Classic Prompt Injection
+
+In a classic prompt injection, someone deliberately plants malicious instructions. Here, the instruction itself is benign and comes from a legitimate source (a real company's own documentation), with no malicious actor at write time. The danger arrives later, when the package or domain the file points to is abandoned and someone else claims it.
+
+The researchers' framing is the key insight: "An agent doesn't distinguish between a page and a command. Everything it reads is input, and every input is a potential instruction." The entire corpus of published data agents now consume has silently become an execution surface.
+
+## ITPP / Hermes Exposure
+
+This is the section that matters for us. Honest assessment:
+
+- **We run Hermes in production**, and it is one of the three agents named in the research.
+- Hermes has full shell/terminal execution, web extraction, browser automation, and MCP tool access. The `terminal` tool can run `pip install`, `npm install`, `npx`, and `curl | bash` if instructed to do so.
+- The risk is not that Hermes will spontaneously install malware. The risk is that a prompt, a fetched document, or a skill references an unverified package and Hermes executes the install as instructed, with no namespace-ownership check in the loop.
+
+**What we have not yet verified** (flagged as follow-up audit items, not assumed safe):
+
+1. Whether any cron job, skill, or automation reads `llms.txt` / external setup docs and follows install commands.
+2. Whether any of our AI products or client deployments run an agent with unguarded shell access against third-party docs.
+3. Whether our Super Search / web-extract pipeline surfaces untrusted content into a context where it can drive package installs.
+
+## Mitigations
+
+These are concrete, ordered by impact:
+
+1. **Never auto-install from external docs.** Treat any install command originating from fetched content (web, `llms.txt`, third-party docs) as untrusted until a human or a verification step confirms the namespace.
+2. **Verify before install.** For any PyPI/npm package, check ownership, age, maintainer history, and download counts before running. A freshly registered name referenced by a vendor doc is the exact red flag this attack exploits.
+3. **Least-privilege on agent shell access.** Do not give agents blanket package-install permissions. Gate `pip` / `npm` / `npx` / `curl | bash` behind confirmation for any agent that consumes untrusted content.
+4. **Audit our automation surface.** Enumerate every cron job, skill, and MCP tool that can reach package managers or shell out to install commands. Confirm none follow unverified install instructions.
+5. **Detect the gap, not the symptom.** EDR will not catch this because it looks like legitimate developer activity. The control has to live upstream: a guardrail that refuses to execute an install command whose package name cannot be verified to a legitimate, long-standing owner.
+
+## Follow-Up Actions
+
+- [ ] Run the exposure audit in the "ITPP / Hermes Exposure" section (items 1-3 above) and record findings.
+- [ ] Add a guardrail or operating rule to Hermes that install commands from untrusted/fetched content require verification.
+- [ ] Re-review this advisory if any of our client-facing AI products ship an agent with shell access.
+
+## References
+
+- Ars Technica: https://arstechnica.com/security/2026/08/claude-codex-and-hermes-installed-unowned-code-inside-corporate-networks/
+- Researcher post (What Would AI Do): https://whatwouldai.do/
+- Researcher write-up (Medium): https://medium.com/@alonhertz1/data-became-code-we-ran-code-inside-fortune-500s-using-files-they-published-for-ai-agents-0cd67ffbbffc
+- llms.txt convention: https://llmstxt.org/

docs/super-search/firecrawl-provider-strategy.md

@@ -0,0 +1,95 @@
+# Firecrawl — Provider Strategy & Configuration
+
+**Created:** 2026-08-10 | **Status:** Active Burn Period → Hobby
+
+---
+
+## Plan Timeline
+
+| Date | Event |
+|---|---|
+| 2026-08-10 | Standard plan (100k credits, $99/mo) — Firecrawl promoted to primary |
+| 2026-09-10 | Billing period ends — Standard plan credits expire |
+| 2026-09-11 | Downgrade to Hobby plan (5k credits, $16–19/mo) |
+
+---
+
+## Why We Burned the Standard Plan
+
+Usage was ~1,400 credits/month — 1.4% of the Standard plan's 100k allocation. The $99/mo Standard plan was over-provisioned by 70x. Firecrawl was position #11 (dead last) in the search fallback chain, so it was almost never called. The decision: burn through Standard credits aggressively until the plan naturally expires on Sep 10, then let the Hobby downgrade take effect with 5k credits/month (~3.5x our actual needs).
+
+---
+
+## Rate Limit History
+
+| When | Firecrawl Rate Limit | Context |
+|---|---|---|
+| Pre-Aug 10 | 5 tokens / 60s | Position #11 in search chain — rarely called |
+| Aug 10 (15:45) | 20 tokens / 60s | Promoted to primary in both chains |
+| Aug 10 (15:51) | 100 tokens / 5s | Maximized for burn period (20 req/s throughput) |
+
+### Rate Limit Comparison (Pricing Page)
+
+| Plan | Concurrent Requests | Credits/Month |
+|---|---|---|
+| Free | 2 | 1,000 |
+| Hobby | 5 | 5,000 |
+| Standard | 50 | 100,000 |
+| Growth | 100 | 500,000 |
+| Scale | 150 | 1,000,000 |
+
+No per-second or per-minute rate caps published per tier. Concurrent request caps are the only documented limit, and Super Search runs calls sequentially — it never hits the 50 concurrent cap even on Standard.
+
+---
+
+## Provider Chain Order
+
+### `web_search`
+```
+1. Firecrawl      ← PRIMARY (promoted Aug 10 from #11)
+2. SearXNG         (local)
+3. Exa
+4. OpenCorporates
+5. CourtListener
+6. DuckDuckGo
+7. Brave
+8. Serper
+9. Tavily
+10. Perplexity
+11. Parallel (Exa + Brave + DuckDuckGo)
+```
+
+### `web_extract`
+```
+1. Firecrawl       ← PRIMARY (promoted Aug 10 from #4)
+2. Jina Reader
+3. Trafilatura
+4. Browserless (CF bypass)
+```
+
+---
+
+## Files Modified
+
+| File | Change |
+|---|---|
+| `/root/docker/super-search/server.py` | Firecrawl moved to position #1 in both search + extract chains; docstrings updated |
+| `/root/docker/super-search/ratelimit.py` | Firecrawl rate limit: 5/min → 20/min → 100/5s |
+
+---
+
+## Monitoring
+
+| Job | Schedule | Description |
+|---|---|---|
+| `firecrawl-usage-check` (2a2f) | Every 2 hours | Queries Firecrawl API `/v1/team/credit-usage` — reports remaining credits to Telegram |
+| Script: `firecrawl-credit-check.sh` | — | Curl-based, no agent needed |
+
+---
+
+## Post-Burn Reversion Plan (Sep 11)
+
+After Hobby downgrade:
+1. Revert rate limit to `{"tokens": 5, "interval": 60.0}` in `ratelimit.py`
+2. Consider moving Firecrawl back to fallback position (TBD based on credit consumption patterns during burn period)
+3. Keep 2-hour monitoring until burn period confirmed complete

hermes-skills/systems-networks-team.md

@@ -47,7 +47,7 @@ Always load these before beginning a sys/net task:
 | app2 | 152.53.39.202 | Infrastructure server |
 | app3 | 152.53.241.111 | Web hosting + backup |
 | core-bu | 5.161.225.131 | Warm standby |
-| wphost02 | 5.161.62.38 | Legacy RunCloud host (still live) |
+| wphost02 | 5.161.62.38 | DECOMMISSIONED (2026-08-28) — deleted from Hetzner account |
 
 ## Key Credentials
 

notes/tiktok-distribution-strategy.md

@@ -0,0 +1,50 @@
+# Marketing Note: Distribution Is the Product
+
+**Saved:** 2026-08-12
+**Source:** Reddit — "How I accidentally grew my app to $3k MRR without spending on ads"
+**Proof:** https://www.tiktok.com/@cookedby.sarah
+
+---
+
+## The Post (verbatim)
+
+Around 4 months ago my app was barely getting any users. Tried "build in public", SEO, Product Hunt, Reddit marketing — nothing moved the needle.
+
+Then one random TikTok crossed 1M+ views. That single video brought more users than everything else combined.
+
+Since then: stopped thinking like a founder, started thinking like a content creator.
+
+## The Format
+
+| Element | Detail |
+|---|---|
+| App recording | 6-10 seconds, one tiny action — not a feature tour |
+| Reaction UGC | Stock "confused/shocked/impressed" face clips, paired to the action |
+| Hook | "This feels illegal" / "I wish I found this sooner" / "This app saved me 4 hours" |
+| Volume | 1-2 videos/day, different combos. Iteration > perfection |
+
+## Core Insight
+
+**Distribution is the product.** He spent months polishing features nobody saw. One video format change unlocked attention. $3k MRR isn't from a better app — it's from being seen.
+
+## Germaine's Portfolio — Built for This
+
+Most B2B SaaS doesn't have reaction-worthy moments. Yours do:
+
+| App | Viral Angle | Hook |
+|---|---|---|
+| **VerdictTank** | AI redlines a proposal in real time | "AI just roasted my proposal" |
+| **RFP Tank** | Compliance scoring catches something a human missed | "This would've cost us the deal" |
+| **HotNow** | Live events popping up on map in real time | "Wait, this is happening RIGHT NOW?" |
+| **Super Search / IntelSight** | Comparison results, surprising findings | "I searched this and..." |
+| **Celebrity Roast Call** (idea) | AI Fred Sanford destroys someone for 10 seconds | "I called an AI and it roasted me for 3 minutes straight" |
+
+## The Play
+
+1. Record app doing one tiny, visually clear thing (6-10s)
+2. Download hundreds of reaction UGC clips (confused, shocked, impressed)
+3. Pair app recording + reaction + simple on-screen hook
+4. Post 1-2/day, different combos
+5. Let the algorithm find the hit
+
+No production. No influencers. No ad spend.

projects/backup-dashboard-enhancements.md

@@ -0,0 +1,34 @@
+# Backup Dashboard Enhancements (Queued)
+
+## Problem
+The Backups tab tells you WHAT is wrong but not enough to investigate or fix it.
+
+## Plan
+
+### 1. Per-item expandability
+Click any critical/warning row to expand:
+- Server (which box)
+- Service name
+- Last OK timestamp + days stale
+- Actual error message from logs
+- Suggested fix based on error pattern
+
+### 2. Server context tags
+Every health check item tagged with origin server. Cross-reference with live server status from Uptime Kuma.
+
+### 3. Issue age / staleness
+Show "3 days ago" vs "2 weeks ago" — stale issues are different from fresh ones.
+
+### 4. One-click investigation links
+- Open the exact log file for that backup job
+- Server's Uptime Kuma status page
+- The backup script itself (read-only view)
+
+### 5. Recurrence tracking
+Track failure history per check. Flag: "⚠️ 3rd failure in 14 days → escalating"
+
+## Implementation
+All additive — same JSON, richer fields. Health monitor script needs to collect more metadata per check.
+
+## Status
+Queued 2026-08-10. Not scheduled.

projects/code-review-graph.md

@@ -0,0 +1,50 @@
+# Code-Review Graph (Structural Code Knowledge) — Future Project
+
+**Status:** Future Projects — Internal Tooling Adoption
+**Saved:** 2026-08-16
+**Category:** How Sho'Nuff & Germaine work (internal agent tooling)
+**Owner:** IT Pro Partner (Sho'Nuff)
+**Source:** https://github.com/rpmalouin/deepseek-harness (fork of deepseek-ai/deepseek-harness)
+
+---
+
+## What it is
+
+`deepseek-harness` (`dsh`) is DeepSeek AI's open-source agent harness: "everything is a plugin" on top of Cordis, a plugin framework. TypeScript/Node, MIT, developer preview.
+
+The `rpmalouin` fork layers three additions on top of stock upstream:
+1. `code-review-graph` — a knowledge graph of a repo's code communities wired into every coding-agent surface.
+2. OpenRouter LLM routing (headless one-shot tasks).
+3. Hermes/agent delegation — designed to be driven as a local coding sub-agent from Hermes.
+
+## Why it matters
+
+Today Sho'Nuff navigates Germaine's repos (itpp-infrastructure, homelab, scripts, project folders) via ripgrep + read_file. That is linear scanning: token-heavy, slow, and it misses callers and dependents that grep never surfaces.
+
+A structural code graph flips that: understand the code through its dependency graph first, then read the files you actually need.
+
+## What to adopt
+
+1. **code-review-graph pattern** — build a queryable structural graph per repo (code communities, callers, dependents, module boundaries). Query the graph before touching files. Start with itpp-infrastructure and homelab.
+2. **"Everything is a plugin" discipline** — tighten the skill system so each skill declares its effects (what it adds, what it can reverse), making per-task composition deliberate rather than implicit.
+
+## What NOT to adopt
+
+- **Do NOT swap Hermes for `dsh`.** Hermes already covers roughly 80% of the operating model:
+  - skills = the plugin system
+  - session history + memory = the session log / source of truth
+  - `delegate_task` = the headless one-shot contract
+  - toolset scoping = scoped tools per agent
+- **`dsh` as a 4th coding backend** (next to Codex, Claude Code, OpenCode) is marginal. Skip unless one of the three fails Germaine.
+
+## Reference notes
+
+- The fork's `FORK.md` documents the code-review-graph pattern and the OpenRouter routing overlay mechanism (the live patch file and keys are local-only, gitignored, and never shipped).
+- The `docs/architecture.md` describes the Cordis plugin tree, capability seams (Service Definition / Provider / Consumer), and the "model-visible means logged" session-log invariant.
+- The upstream is `deepseek-ai/deepseek-harness` (MIT); the fork is 4 commits ahead and adds only scaffolding, no upstream behavior changes.
+
+## Source
+
+- https://github.com/rpmalouin/deepseek-harness (fork of deepseek-ai/deepseek-harness)
+- Files reviewed: README, FORK.md, docs/architecture.md
+- Retrieved: 2026-08-16

projects/convert_hotnow.py

@@ -0,0 +1,875 @@
+#!/usr/bin/env python3
+"""Convert HotNow Savannah v2 markdown to Anita/DM Serif HTML."""
+
+import re
+import html as html_mod
+
+def read_md(path):
+    with open(path, 'r') as f:
+        return f.read()
+
+def clean_text(text):
+    """Strip markdown bold/italic but keep content, handle inline code."""
+    # Bold
+    text = re.sub(r'\*\*(.+?)\*\*', r'<strong>\1</strong>', text)
+    # Italic
+    text = re.sub(r'\*(.+?)\*', r'<em>\1</em>', text)
+    # Inline code
+    text = re.sub(r'`([^`]+)`', r'<code>\1</code>', text)
+    # Links
+    text = re.sub(r'\[([^\]]+)\]\(([^)]+)\)', r'<a href="\2">\1</a>', text)
+    return text
+
+def escape_for_html(text):
+    """Escape HTML entities but preserve our strong/em/code/a tags."""
+    # Protect tags we want to keep
+    text = text.replace('<strong>', '\x00S\x00')
+    text = text.replace('</strong>', '\x00/S\x00')
+    text = text.replace('<em>', '\x00E\x00')
+    text = text.replace('</em>', '\x00/E\x00')
+    text = text.replace('<code>', '\x00C\x00')
+    text = text.replace('</code>', '\x00/C\x00')
+    text = text.replace('<a href=', '\x00A\x00')
+    text = text.replace('</a>', '\x00/A\x00')
+    text = text.replace('">', '\x00Q\x00')
+    # Escape
+    text = html_mod.escape(text, quote=False)
+    # Restore
+    text = text.replace('\x00S\x00', '<strong>')
+    text = text.replace('\x00/S\x00', '</strong>')
+    text = text.replace('\x00E\x00', '<em>')
+    text = text.replace('\x00/E\x00', '</em>')
+    text = text.replace('\x00C\x00', '<code>')
+    text = text.replace('\x00/C\x00', '</code>')
+    text = text.replace('\x00A\x00', '<a href=')
+    text = text.replace('\x00/A\x00', '</a>')
+    text = text.replace('\x00Q\x00', '">')
+    return text
+
+def parse_table(lines, start_idx):
+    """Parse a markdown table starting at start_idx. Returns (html_rows, next_idx)."""
+    i = start_idx
+    # Find header row
+    if i >= len(lines) or not lines[i].strip().startswith('|'):
+        return None, start_idx
+    
+    header_line = lines[i].strip()
+    header_cells = [c.strip() for c in header_line.split('|')[1:-1]]
+    i += 1
+    
+    # Skip separator line
+    if i < len(lines) and re.match(r'^\|[\s\-:|]+\|$', lines[i].strip()):
+        i += 1
+    
+    # Parse data rows
+    data_rows = []
+    while i < len(lines):
+        line = lines[i].strip()
+        if not line.startswith('|'):
+            break
+        cells = [c.strip() for c in line.split('|')[1:-1]]
+        data_rows.append(cells)
+        i += 1
+    
+    # Build HTML
+    html = '<table>\n<thead>\n<tr>\n'
+    for cell in header_cells:
+        html += f'<th>{escape_for_html(clean_text(cell))}</th>\n'
+    html += '</tr>\n</thead>\n<tbody>\n'
+    for row in data_rows:
+        html += '<tr>\n'
+        for cell in row:
+            html += f'<td>{escape_for_html(clean_text(cell))}</td>\n'
+        html += '</tr>\n'
+    html += '</tbody>\n</table>\n'
+    
+    return html, i
+
+def parse_code_block(lines, start_idx):
+    """Parse a code block. Returns (html, next_idx)."""
+    i = start_idx
+    if not lines[i].strip().startswith('```'):
+        return None, start_idx
+    i += 1
+    content_lines = []
+    while i < len(lines):
+        if lines[i].strip().startswith('```'):
+            i += 1
+            break
+        content_lines.append(lines[i])
+        i += 1
+    
+    content = ''.join(content_lines)
+    content = escape_for_html(content)
+    html = f'<pre><code>{content}</code></pre>\n'
+    return html, i
+
+def parse_list(lines, start_idx):
+    """Parse a markdown list (ordered or unordered). Returns (html, next_idx)."""
+    i = start_idx
+    items = []
+    while i < len(lines):
+        line = lines[i].strip()
+        # Ordered list
+        m = re.match(r'^(\d+)\.\s+(.+)$', line)
+        if m:
+            items.append(('ol', escape_for_html(clean_text(m.group(2)))))
+            i += 1
+            continue
+        # Unordered list
+        m = re.match(r'^[-*]\s+(.+)$', line)
+        if m:
+            items.append(('ul', escape_for_html(clean_text(m.group(1)))))
+            i += 1
+            continue
+        break
+    
+    if not items:
+        return None, start_idx
+    
+    list_type = items[0][0]
+    html = f'<{list_type}>\n'
+    for lt, item_text in items:
+        html += f'<li>{item_text}</li>\n'
+    html += f'</{list_type}>\n'
+    return html, i
+
+def parse_blockquote(lines, start_idx):
+    """Parse blockquote lines. Returns (html, next_idx)."""
+    i = start_idx
+    content_lines = []
+    while i < len(lines):
+        line = lines[i].strip()
+        if line.startswith('>'):
+            content_lines.append(line[1:].strip())
+            i += 1
+        else:
+            break
+    
+    if not content_lines:
+        return None, start_idx
+    
+    content = ' '.join(content_lines)
+    content = escape_for_html(clean_text(content))
+    html = f'<blockquote><p>{content}</p></blockquote>\n'
+    return html, i
+
+def build_toc(sections):
+    """Build TOC HTML from section list."""
+    toc_html = '<nav class="toc">\n'
+    for sec_id, sec_title in sections:
+        toc_html += f'<a href="#{sec_id}">{sec_title}</a>\n'
+    toc_html += '</nav>\n'
+    return toc_html
+
+def convert_md_to_html(md_text):
+    """Main conversion function."""
+    lines = md_text.split('\n')
+    
+    sections = []  # (id, title)
+    html_body = ''
+    current_section_id = None
+    extra_sections = []  # appendices
+    
+    i = 0
+    
+    # Skip first line (# title) and metadata
+    # We'll handle hero separately
+    while i < len(lines) and not lines[i].startswith('## Table of Contents'):
+        i += 1
+    
+    # Skip TOC and horizontal rule
+    while i < len(lines) and not lines[i].startswith('## 1. Executive Summary'):
+        i += 1
+    
+    # Now process each ## section
+    while i < len(lines):
+        line = lines[i].strip()
+        
+        # Check for ## section header
+        m = re.match(r'^##\s+(.+)$', line)
+        if m:
+            section_title = m.group(1).strip()
+            # Generate ID
+            sec_id = section_title.lower()
+            # Extract number prefix
+            num_match = re.match(r'^(\d+)\.\s+(.+)$', section_title)
+            if num_match:
+                num = num_match.group(1)
+                sec_name = num_match.group(2)
+                sec_id = f'section-{num}'
+                sections.append((sec_id, f'{num}. {sec_name}'))
+            elif section_title.startswith('Appendix'):
+                sec_id = section_title.lower().replace(' ', '-').replace(':', '').replace('(', '').replace(')', '')
+                extra_sections.append(sec_id)
+                sections.append((sec_id, section_title))
+            else:
+                sec_id = section_title.lower().replace(' ', '-').replace('.', '')
+                sections.append((sec_id, section_title))
+            
+            current_section_id = sec_id
+            
+            # Decide if this is in the appendix (outside main card section)
+            is_appendix = section_title.startswith('Appendix')
+            
+            if not is_appendix:
+                html_body += f'<section class="section-card" id="{sec_id}">\n'
+                html_body += f'<h2>{escape_for_html(clean_text(section_title))}</h2>\n'
+            else:
+                html_body += f'<section class="appendix-section" id="{sec_id}">\n'
+                html_body += f'<h2>{escape_for_html(clean_text(section_title))}</h2>\n'
+            
+            i += 1
+            continue
+        
+        # Check for ### sub-section
+        m = re.match(r'^###\s+(.+)$', line)
+        if m:
+            sub_title = m.group(1).strip()
+            html_body += f'<h3>{escape_for_html(clean_text(sub_title))}</h3>\n'
+            i += 1
+            continue
+        
+        # Horizontal rule
+        if line.startswith('---'):
+            if current_section_id and not current_section_id.startswith('appendix'):
+                html_body += '</section>\n'
+                current_section_id = None
+            i += 1
+            continue
+        
+        # Code block
+        if line.startswith('```'):
+            code_html, i = parse_code_block(lines, i)
+            if code_html:
+                html_body += code_html
+            continue
+        
+        # Table
+        if line.startswith('|'):
+            table_html, i = parse_table(lines, i)
+            if table_html:
+                html_body += table_html
+            continue
+        
+        # Blockquote
+        if line.startswith('>'):
+            bq_html, i = parse_blockquote(lines, i)
+            if bq_html:
+                html_body += bq_html
+            continue
+        
+        # List
+        list_html, new_i = parse_list(lines, i)
+        if list_html:
+            html_body += list_html
+            i = new_i
+            continue
+        
+        # Regular paragraph (non-empty)
+        if line:
+            # Check for bold-only short lines (like TL;DR)
+            para = escape_for_html(clean_text(line))
+            html_body += f'<p>{para}</p>\n'
+        
+        i += 1
+    
+    # Close last section if open
+    if current_section_id:
+        html_body += '</section>\n'
+    
+    return sections, html_body
+
+# CSS template
+CSS = '''    :root {
+        --accent: #2563eb;
+        --accent-dark: #1d4ed8;
+        --accent-light: #eff6ff;
+        --accent-glow: rgba(37,99,235,0.15);
+        --navy: #0f172a;
+        --navy-light: #1e293b;
+        --bg: #f8f9fb;
+        --card: #ffffff;
+        --border: #e5e7eb;
+        --text: #1a1a2e;
+        --text-secondary: #6b7280;
+        --text-muted: #9ca3af;
+        --green: #10b981;
+        --green-bg: #ecfdf5;
+        --amber: #d97706;
+        --amber-bg: #fffbeb;
+        --red: #dc2626;
+        --red-bg: #fef2f2;
+        --radius: 12px;
+        --radius-sm: 8px;
+        --shadow: 0 1px 3px rgba(0,0,0,0.06), 0 1px 2px rgba(0,0,0,0.04);
+    }
+
+    * {
+        margin: 0;
+        padding: 0;
+        box-sizing: border-box;
+    }
+
+    body {
+        font-family: 'DM Sans', -apple-system, BlinkMacSystemFont, 'Segoe UI', sans-serif;
+        background: var(--bg);
+        color: var(--text);
+        line-height: 1.7;
+        font-weight: 400;
+    }
+
+    h1, h2, h3, .price, .stat-value {
+        font-family: 'DM Serif Display', Georgia, 'Times New Roman', serif;
+    }
+
+    /* Hero */
+    .hero {
+        background: linear-gradient(135deg, #0f172a, #1e293b 40%, #1e3a5f 70%, #1d4ed8);
+        color: #ffffff;
+        padding: 80px 24px 70px;
+        text-align: center;
+        position: relative;
+        overflow: hidden;
+    }
+
+    .hero::after {
+        content: '';
+        position: absolute;
+        top: -50%;
+        left: -25%;
+        width: 150%;
+        height: 200%;
+        background: radial-gradient(ellipse at 30% 50%, var(--accent-glow) 0%, transparent 60%);
+        pointer-events: none;
+    }
+
+    .hero > * {
+        position: relative;
+        z-index: 1;
+    }
+
+    .hero .badge {
+        display: inline-block;
+        padding: 6px 18px;
+        border-radius: 20px;
+        font-size: 13px;
+        font-weight: 600;
+        font-family: 'DM Sans', sans-serif;
+        background: rgba(255,255,255,0.12);
+        color: rgba(255,255,255,0.9);
+        border: 1px solid rgba(255,255,255,0.2);
+        margin-bottom: 24px;
+        letter-spacing: 0.5px;
+    }
+
+    .hero h1 {
+        font-size: 52px;
+        font-weight: 400;
+        margin-bottom: 12px;
+        letter-spacing: -0.5px;
+    }
+
+    .hero .subtitle {
+        font-size: 20px;
+        color: rgba(255,255,255,0.75);
+        max-width: 700px;
+        margin: 0 auto 24px;
+        font-weight: 400;
+    }
+
+    .hero .meta {
+        font-size: 14px;
+        color: rgba(255,255,255,0.55);
+        font-weight: 500;
+    }
+
+    /* TOC */
+    .toc-wrap {
+        max-width: 960px;
+        margin: -28px auto 0;
+        padding: 0 24px;
+        position: relative;
+        z-index: 2;
+    }
+
+    .toc {
+        background: var(--card);
+        border-radius: var(--radius);
+        box-shadow: var(--shadow);
+        padding: 20px 24px;
+        display: flex;
+        flex-wrap: wrap;
+        gap: 8px;
+        justify-content: center;
+    }
+
+    .toc a {
+        display: inline-block;
+        padding: 6px 14px;
+        border-radius: 20px;
+        font-size: 13px;
+        font-weight: 500;
+        background: var(--accent-light);
+        color: var(--accent);
+        text-decoration: none;
+        transition: background 0.2s, color 0.2s;
+    }
+
+    .toc a:hover {
+        background: var(--accent);
+        color: #fff;
+    }
+
+    /* Section Cards */
+    .sections {
+        max-width: 960px;
+        margin: 32px auto 0;
+        padding: 0 24px;
+    }
+
+    .section-card {
+        background: var(--card);
+        border-radius: var(--radius);
+        box-shadow: var(--shadow);
+        padding: 40px;
+        margin-bottom: 24px;
+        transition: box-shadow 0.2s;
+    }
+
+    .section-card:hover {
+        box-shadow: 0 4px 12px rgba(0,0,0,0.08), 0 2px 4px rgba(0,0,0,0.04);
+    }
+
+    .section-card h2 {
+        font-size: 28px;
+        font-weight: 400;
+        color: var(--navy);
+        margin-bottom: 24px;
+        padding-bottom: 16px;
+        border-bottom: 2px solid var(--accent-light);
+    }
+
+    .section-card h3 {
+        font-size: 20px;
+        font-weight: 400;
+        color: var(--text);
+        margin: 28px 0 12px;
+    }
+
+    .section-card h3:first-of-type {
+        margin-top: 0;
+    }
+
+    .section-card p {
+        margin-bottom: 16px;
+        color: var(--text);
+        font-size: 16px;
+    }
+
+    .section-card p:last-child {
+        margin-bottom: 0;
+    }
+
+    /* Tables */
+    table {
+        width: 100%;
+        border-collapse: collapse;
+        margin: 20px 0 28px;
+        font-size: 14px;
+    }
+
+    thead th {
+        background: var(--accent-light);
+        color: var(--accent-dark);
+        font-weight: 600;
+        font-size: 12px;
+        text-transform: uppercase;
+        letter-spacing: 0.5px;
+        padding: 12px 16px;
+        text-align: left;
+        border-bottom: 2px solid var(--accent);
+    }
+
+    tbody td {
+        padding: 10px 16px;
+        border-bottom: 1px solid var(--border);
+        color: var(--text);
+        vertical-align: top;
+    }
+
+    tbody tr:last-child td {
+        border-bottom: none;
+    }
+
+    tbody tr:hover {
+        background: var(--accent-light);
+    }
+
+    /* Code blocks */
+    pre {
+        background: #1e293b;
+        color: #e2e8f0;
+        padding: 20px 24px;
+        border-radius: var(--radius-sm);
+        overflow-x: auto;
+        font-size: 12px;
+        line-height: 1.5;
+        margin: 16px 0 24px;
+        font-family: 'SF Mono', 'Fira Code', 'Consolas', monospace;
+    }
+
+    pre code {
+        background: none;
+        padding: 0;
+        font-size: inherit;
+    }
+
+    code {
+        background: var(--accent-light);
+        color: var(--accent-dark);
+        padding: 2px 6px;
+        border-radius: 4px;
+        font-size: 13px;
+        font-family: 'SF Mono', 'Fira Code', 'Consolas', monospace;
+    }
+
+    /* Blockquotes */
+    blockquote {
+        border-left: 4px solid var(--accent);
+        background: var(--accent-light);
+        padding: 16px 20px;
+        margin: 16px 0 24px;
+        border-radius: 0 var(--radius-sm) var(--radius-sm) 0;
+        font-style: italic;
+    }
+
+    blockquote p {
+        margin: 0 !important;
+        color: var(--navy-light);
+    }
+
+    /* Lists */
+    ul, ol {
+        margin: 12px 0 20px 24px;
+        color: var(--text);
+    }
+
+    ul li, ol li {
+        margin-bottom: 8px;
+        font-size: 16px;
+    }
+
+    /* Pricing grid */
+    .pricing-grid {
+        display: grid;
+        grid-template-columns: repeat(auto-fit, minmax(240px, 1fr));
+        gap: 16px;
+        margin: 20px 0 28px;
+    }
+
+    .pricing-card {
+        background: var(--card);
+        border: 1px solid var(--border);
+        border-radius: var(--radius);
+        padding: 24px;
+    }
+
+    .pricing-card.featured {
+        border: 2px solid var(--accent);
+        background: var(--accent-light);
+    }
+
+    .pricing-card h4 {
+        font-family: 'DM Serif Display', Georgia, serif;
+        font-size: 20px;
+        font-weight: 400;
+        margin-bottom: 4px;
+        color: var(--navy);
+    }
+
+    .pricing-card .price {
+        font-size: 28px;
+        font-weight: 400;
+        color: var(--accent);
+        margin-bottom: 16px;
+    }
+
+    .pricing-card ul {
+        margin: 0 0 16px 18px;
+        list-style: none;
+    }
+
+    .pricing-card ul li {
+        position: relative;
+        padding-left: 20px;
+        font-size: 14px;
+        margin-bottom: 6px;
+        color: var(--text-secondary);
+    }
+
+    .pricing-card ul li::before {
+        content: '+';
+        position: absolute;
+        left: 0;
+        color: var(--green);
+        font-weight: 700;
+    }
+
+    /* Tags */
+    .tag-critical { background: var(--red-bg); color: var(--red); }
+    .tag-high { background: #fef3c7; color: #b45309; }
+    .tag-medium { background: var(--amber-bg); color: var(--amber); }
+    .tag-low { background: var(--green-bg); color: #059669; }
+    .tag-phase { background: var(--accent-light); color: var(--accent); }
+
+    .tag {
+        display: inline-block;
+        padding: 4px 10px;
+        border-radius: 4px;
+        font-size: 11px;
+        font-weight: 700;
+        text-transform: uppercase;
+        letter-spacing: 0.5px;
+    }
+
+    /* VerdictTank */
+    .verdicttank {
+        background: var(--red-bg);
+        border: 1px solid #fecaca;
+        border-left: 4px solid var(--red);
+        border-radius: var(--radius);
+        padding: 28px 32px;
+        margin: 24px auto 0;
+        max-width: 960px;
+        text-align: center;
+    }
+
+    .verdicttank h2 {
+        font-family: 'DM Serif Display', Georgia, serif;
+        font-size: 24px;
+        font-weight: 400;
+        color: var(--red);
+        margin-bottom: 8px;
+        text-align: center;
+    }
+
+    .verdicttank p {
+        color: #991b1b;
+        font-size: 15px;
+        margin: 0;
+        text-align: center;
+    }
+
+    .verdicttank a {
+        color: var(--red);
+        font-weight: 600;
+    }
+
+    /* Footer */
+    .footer {
+        background: var(--navy);
+        color: rgba(255,255,255,0.8);
+        text-align: center;
+        padding: 40px 24px;
+        margin-top: 48px;
+    }
+
+    .footer p {
+        font-size: 14px;
+        margin-bottom: 8px;
+        color: rgba(255,255,255,0.7);
+    }
+
+    .footer .title {
+        font-family: 'DM Serif Display', Georgia, serif;
+        font-size: 18px;
+        color: #ffffff;
+        margin-bottom: 8px;
+    }
+
+    .footer .links {
+        margin-top: 16px;
+        display: flex;
+        flex-wrap: wrap;
+        justify-content: center;
+        gap: 16px;
+    }
+
+    .footer .links a {
+        color: rgba(255,255,255,0.7);
+        text-decoration: none;
+        font-size: 13px;
+        transition: color 0.2s;
+    }
+
+    .footer .links a:hover {
+        color: #ffffff;
+    }
+
+    .footer .links a.current {
+        color: #ffffff;
+        font-weight: 700;
+    }
+
+    /* Appendices */
+    .appendix-section {
+        max-width: 960px;
+        margin: 0 auto 24px;
+        padding: 32px 40px;
+        background: var(--card);
+        border-radius: var(--radius);
+        box-shadow: var(--shadow);
+        border-left: 3px solid var(--border);
+    }
+
+    .appendix-section h2 {
+        font-family: 'DM Serif Display', Georgia, serif;
+        font-size: 22px;
+        font-weight: 400;
+        color: var(--text-secondary);
+        margin-bottom: 20px;
+    }
+
+    .appendix-section table {
+        font-size: 13px;
+    }
+
+    /* Print styles */
+    @media print {
+        body { background: #fff; }
+        .hero { print-color-adjust: exact; -webkit-print-color-adjust: exact; }
+        .section-card { box-shadow: none; break-inside: avoid; }
+        .toc-wrap { display: none; }
+        .footer { print-color-adjust: exact; -webkit-print-color-adjust: exact; }
+    }
+
+    /* Mobile */
+    @media (max-width: 640px) {
+        .hero { padding: 60px 20px 50px; }
+        .hero h1 { font-size: 36px; }
+        .hero .subtitle { font-size: 16px; }
+        .section-card { padding: 24px 20px; }
+        .section-card h2 { font-size: 22px; }
+        table { font-size: 12px; }
+        thead th, tbody td { padding: 8px 10px; }
+        .pricing-grid { grid-template-columns: 1fr; }
+        .toc { padding: 14px 16px; }
+        .toc a { font-size: 12px; padding: 5px 10px; }
+    }
+'''
+
+def build_full_html(sections, body_html):
+    """Build the complete HTML document."""
+    toc_html = build_toc(sections)
+    
+    html = f'''<!DOCTYPE html>
+<html lang="en">
+<head>
+<meta charset="UTF-8">
+<meta name="viewport" content="width=device-width, initial-scale=1.0">
+<title>HotNow Savannah - Business Proposal v2.0</title>
+<link rel="preconnect" href="https://fonts.googleapis.com">
+<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
+<link href="https://fonts.googleapis.com/css2?family=DM+Sans:ital,opsz,wght@0,9..40,400;0,9..40,500;0,9..40,600;0,9..40,700;0,9..40,800;1,9..40,400&family=DM+Serif+Display&display=swap" rel="stylesheet">
+<style>
+{CSS}
+</style>
+</head>
+<body>
+
+<!-- Hero -->
+<header class="hero">
+    <div class="badge">IT Pro Partner - Product Division</div>
+    <h1>HotNow Savannah</h1>
+    <p class="subtitle">Real-Time Local Discovery Engine - Savannah, GA Launch</p>
+    <p class="meta">Confidential - Version 2.0 - August 11, 2026</p>
+</header>
+
+<!-- TOC -->
+<div class="toc-wrap">
+{toc_html}
+</div>
+
+<!-- Sections -->
+<div class="sections">
+{body_html}
+</div>
+
+<!-- VerdictTank -->
+<div class="verdicttank">
+    <h2>VerdictTank Reviewed</h2>
+    <p>This proposal has not yet been reviewed by VerdictTank. Schedule a review at <a href="https://verdicttank.com">verdicttank.com</a>.</p>
+</div>
+
+<!-- Footer -->
+<footer class="footer">
+    <p class="title">HotNow Savannah - Confidential Business Proposal</p>
+    <p>Prepared by IT Pro Partner - Product Division &middot; August 11, 2026</p>
+    <div class="links">
+        <a href="/hotnow/">Original Proposal (v1)</a>
+        <a href="#">Pipeline Verdict (pending)</a>
+        <a href="#" class="current">Post-Review Proposal (v2)</a>
+        <a href="https://itpropartner.com">IT Pro Partner</a>
+    </div>
+</footer>
+
+</body>
+</html>'''
+    return html
+
+def main():
+    md_path = '/root/projects/itpp-infrastructure/projects/hotnow-savannah-v2.md'
+    out_path = '/root/projects/itpp-infrastructure/projects/hotnow-savannah-v2.html'
+    
+    md_text = read_md(md_path)
+    sections, body_html = convert_md_to_html(md_text)
+    
+    full_html = build_full_html(sections, body_html)
+    
+    # Post-process: strip any remaining em dashes, en dashes, smart quotes
+    full_html = full_html.replace('\u2014', '-')  # em dash
+    full_html = full_html.replace('\u2013', '-')  # en dash
+    full_html = full_html.replace('\u201c', '"')  # left smart quote
+    full_html = full_html.replace('\u201d', '"')  # right smart quote
+    full_html = full_html.replace('\u2018', "'")  # left smart apostrophe
+    full_html = full_html.replace('\u2019', "'")  # right smart apostrophe
+    
+    # Replace any -- with " - " but ONLY outside <style> tags and HTML comments
+    def fix_double_hyphens(text):
+        # Protect <style> blocks and HTML comments
+        protected = {}
+        counter = [0]
+        def protect_style(m):
+            key = f'\x00STYLE{counter[0]}\x00'
+            counter[0] += 1
+            protected[key] = m.group(0)
+            return key
+        def protect_comment(m):
+            key = f'\x00COMMENT{counter[0]}\x00'
+            counter[0] += 1
+            protected[key] = m.group(0)
+            return key
+        
+        text = re.sub(r'<style>.*?</style>', protect_style, text, flags=re.DOTALL)
+        text = re.sub(r'<!--.*?-->', protect_comment, text, flags=re.DOTALL)
+        
+        # Now safe to replace -- in remaining content
+        text = re.sub(r'--', ' - ', text)
+        
+        # Restore protected blocks
+        for key, value in protected.items():
+            text = text.replace(key, value)
+        
+        return text
+    
+    full_html = fix_double_hyphens(full_html)
+    
+    with open(out_path, 'w') as f:
+        f.write(full_html)
+    
+    print(f"Written {len(full_html)} chars to {out_path}")
+    print(f"Sections: {len(sections)}")
+
+if __name__ == '__main__':
+    main()

projects/front-desk-voice-agent.md

@@ -0,0 +1,161 @@
+# Sho'Nuff Front-Desk — AI Voice Lead Qualification & Booking
+
+**Saved:** 2026-08-24
+**Status:** Future Project — Competitive Teardown + Build Path
+**Category:** SaaS Product / Revenue
+**Owner:** IT Pro Partner (Sho'Nuff)
+**Inspiration:** Qexo.ai competitive teardown (2026-08-24)
+
+---
+
+## The Competitive Trigger
+
+Qexo.ai is AI voice agents for **lead qualification + appointment booking**, aimed at home-services / SMB verticals (solar, HVAC, home services, real estate, insurance, clinics). It is the third tool in a row we've teardown'd (after tranx.io and dozier.io) shaped as a single-purpose vertical tool on generic AI plumbing: narrow SMB wedge + clean before/after loop + evidence/confidence. The pattern is now a signal, not a coincidence.
+
+**Qexo's four-step loop:**
+
+```
+1. Capture    — web/form intake catches the lead while intent is hot
+                message + contact + consent attached to ONE record
+2. Understand — "lead intelligence" scores intent / urgency / service-fit
+                0–100 qualification score, flags missing details
+3. Call & Book — with consent, voice agent calls OUT, gets context,
+                books against live calendar, confirms address/access notes
+4. Manage     — unified lead workspace (inquiry + transcript +
+                qualification + appointment), human override
+```
+
+**Why Qexo is clever (and what to steal):**
+- **Full-context handoff** — the outbound call inherits the web intake, never starts over. The caller already knows what the lead asked for.
+- **Transparent AI + consent** — kills the FTC/legal objection up front. Voice agents calling out is the single most legally exposed move in this space; Qexo neutralizes it by design.
+- **"Not another inbox"** — explicitly positioned as NOT a CRM. A unified lead workspace, not a login you'll ignore.
+
+---
+
+## The Honest Read: We Already Own 90% of This
+
+The gap is **productization, not capability.** We have every component running today:
+
+| Component | Existing Asset | Status |
+|---|---|---|
+| Outbound voice calling | `shonuff-voice-caller`, `twilio-voice-calling` skills (Twilio + ElevenLabs) | ✅ Live |
+| Inbound call handling | `ai-receptionist.md` (VoIPSimplicity Concierge) | 📐 Designed, not built |
+| Missed-call capture | `missed-call-lead-recovery.md` (Twilio SMS text-back) | 📐 Designed, not built |
+| Voice stack (STT→LLM→TTS) | `voice-agent-deployment` skill — Hermes Voice (xAI realtime) + Kokoro/faster-whisper open-source | ✅ Live |
+| Orchestration brain | Hermes Agent (personality, memory, tool routing) | ✅ Live |
+| Calendar | Rally family calendar + booking patterns | ✅ Live |
+
+**The missing 10%** — the part that makes it a *product* rather than a demo:
+1. **A lead-record model that survives web → voice → calendar** — one object holding intake message, contact, consent, qualification score, transcript, and appointment, all keyed to the same lead. This is Qexo's actual moat, and it's a schema problem, not an AI problem.
+2. **A qualification scorer** — 0–100 intent/urgency/service-fit score with visible evidence (the lead's own words).
+
+Neither is hard. Both are a weekend build on the infra we already run.
+
+---
+
+## Build Path
+
+### Phase 1 — The Lead Record (the real moat)
+
+Postgres table (single source of truth):
+
+```
+leads
+  id, tenant_id, source (web/form/call/missed-call)
+  contact_name, phone, email
+  message (original inquiry, verbatim)
+  consent_at (timestamp), consent_medium (form checkbox / verbal / none)
+  qual_score (0-100), qual_evidence (json: flagged signals)
+  service_fit, urgency, intent
+  transcript (json, appended on call)
+  appointment_id, appointment_status
+  created_at, updated_at
+```
+
+Every downstream step (scorer, outbound call, calendar write) reads and writes **this same record**. The lead never starts over. This is the "full-context handoff" Qexo sells, reduced to a schema.
+
+### Phase 2 — Qualification Scorer
+
+DeepSeek (via admin-ai) classifies the lead record into a 0–100 score with visible evidence:
+
+- **Intent** — did they ask for a specific service, or just browse?
+- **Urgency** — "as soon as possible" / "this week" / "just looking"
+- **Service-fit** — does the inquiry match any offered service?
+- **Missing detail flags** — no address, no timeframe, no budget signal
+
+Output: a score + the exact phrases that drove it. The evidence layer is what makes it defensible against "AI made that up" — the customer sees the lead's own words backing the score.
+
+### Phase 3 — Outbound Call & Book
+
+With consent on record, the voice agent calls out:
+1. Pulls the lead record (never re-asks what the form already captured)
+2. Confirms interest, fills the gaps the scorer flagged
+3. Books against live calendar
+4. Confirms address/access notes
+5. Appends transcript + appointment to the lead record
+
+Reuse `shonuff-voice-caller` (ElevenLabs professional male voice) + Twilio outbound. The open-source Kokoro/faster-whisper stack from `voice-agent-deployment` is the $0-cost alternative for beta.
+
+### Phase 4 — Unified Workspace ("Not another inbox")
+
+Single view per lead: original inquiry + qualification score + transcript + appointment, human override everywhere. Not a CRM — a workspace where a lead either gets booked or gets a reason why not.
+
+---
+
+## Positioning vs. What We Already Have
+
+| Product | Inbound | Outbound | Qualifies | Books | Key Differentiator |
+|---|---|---|---|---|---|
+| **VoIPSimplicity Concierge** (`ai-receptionist.md`) | ✅ answers calls | ❌ | ⚠️ basic | ✅ | Replaces IVR for existing VoIP customers |
+| **Missed-Call Recovery** (`missed-call-lead-recovery.md`) | ⚠️ missed calls | ❌ | ❌ | ❌ | SMS text-back within seconds |
+| **Sho'Nuff Front-Desk** (this) | ✅ | ✅ **calls out** | ✅ **scores** | ✅ | **Full-context handoff: web → score → outbound → calendar** |
+| **Qexo.ai** (competitor) | ✅ | ✅ | ✅ | ✅ | The benchmark we're matching |
+
+The Front-Desk is the top of the funnel the other two feed into. Missed-call recovery captures the lead; the Front-Desk qualifies and books it. These are three products on one voice stack, not three competing ideas.
+
+---
+
+## Pricing (Premium, Value-Based — Never Undercut)
+
+Modeled on Qexo's SMB wedge but priced like we own the infrastructure (we do):
+
+| Tier | Price/mo | Included |
+|---|---|---|
+| **Solo** | $99 | 1 voice number, web intake + scorer, 50 outbound calls, calendar booking |
+| **Pro** | $299 | 3 numbers, 200 calls, multi-location, transcript archive, human-override console |
+| **Managed** | $599+ | White-label, agency resell, custom qualification rules, SLA |
+
+Undercut Qexo on **unit economics**, not headline price. Our marginal cost is near-zero (self-hosted voice stack, admin-ai tokens at cost). Qexo pays per-call infrastructure margins we don't.
+
+---
+
+## Risks
+
+1. **FTC / TCPA on outbound AI calls** — the single biggest exposure. Qexo's consent-first design is correct and mandatory. We mirror it: no outbound call without a recorded consent timestamp. `debt-recovery-compliance` skill already documents the TCPA/consent discipline; reuse it.
+2. **Voice quality at scale** — Kokoro is good but not ElevenLabs. Start managed-tier on ElevenLabs, offer Kokoro for beta cost control.
+3. **Calendar write integrity** — a wrong booking is a lost customer. The lead record must be the single writer; no side-channel calendar edits.
+
+---
+
+## Open Questions
+
+1. **First vertical** — solar/HVAC (Qexo's beachhead) vs. our existing warm markets (Debt Recovery Experts intake, VoIPSimplicity customers, Forefront Wireless)?
+2. **Calendar backend** — Rally, or a dedicated booking calendar per tenant?
+3. **Consent capture** — form checkbox (SMS/web) vs. recorded verbal consent (call). Both need a timestamped, auditable record.
+4. **Tenant model** — multi-tenant from day one (agencies reselling to clients), or single-tenant until 3 paying customers?
+
+---
+
+## Next Steps
+
+1. **Decide first vertical** (open question 1) — this shapes every downstream choice
+2. **Build the lead-record schema** (Phase 1) — the moat, and a pure schema task
+3. **Weekend spike**: qualification scorer on 10 sample leads, verify evidence layer
+4. **Wire outbound call** via existing `shonuff-voice-caller` + Twilio
+5. **Beta** with 1–2 friendly businesses before any pricing commitment
+
+---
+
+## Meta-Signal (worth remembering)
+
+Three teardowns in a row — tranx.io, dozier.io, qexo.ai — are all **single-purpose vertical tools on generic AI plumbing**, each with the same shape: narrow SMB wedge, clean before/after loop, evidence/confidence layer. The pattern means the plumbing is commoditizing. The defensible layer is not the AI — it's the **data model and the compliance posture** (lead record + consent + calendar integrity). We already own the plumbing. The win is in the schema, the scorer evidence, and the consent design — not in out-building the AI.

projects/hotnow-savannah-verdicttank.html

@@ -0,0 +1,189 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+<meta charset="UTF-8">
+<meta name="viewport" content="width=device-width, initial-scale=1.0">
+<title>VerdictTank Review - HotNow Savannah v2</title>
+<link rel="preconnect" href="https://fonts.googleapis.com">
+<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
+<link href="https://fonts.googleapis.com/css2?family=DM+Serif+Display&family=DM+Sans:wght@400;600&display=swap" rel="stylesheet">
+<style>
+  :root { --bg: #0d0d0d; --card: #1a1a1a; --text: #e0e0e0; --muted: #888; --accent: #e74c3c; --green: #2ecc71; --amber: #f39c12; --border: #2a2a2a; }
+  * { margin: 0; padding: 0; box-sizing: border-box; }
+  body { background: var(--bg); color: var(--text); font-family: 'DM Sans', sans-serif; line-height: 1.6; padding: 2rem; }
+  .container { max-width: 900px; margin: 0 auto; }
+  h1 { font-family: 'DM Serif Display', serif; font-size: 2.2rem; margin-bottom: 0.5rem; }
+  h2 { font-family: 'DM Serif Display', serif; font-size: 1.4rem; margin: 2rem 0 1rem; padding-bottom: 0.5rem; border-bottom: 1px solid var(--border); }
+  h3 { font-size: 1.1rem; margin: 1.5rem 0 0.5rem; color: var(--muted); text-transform: uppercase; letter-spacing: 0.05em; }
+  .verdict-badge { display: inline-flex; align-items: center; gap: 0.75rem; padding: 0.75rem 1.5rem; border-radius: 8px; font-size: 1.4rem; font-weight: 600; margin: 1rem 0 2rem; }
+  .verdict-badge.conditional { background: rgba(243,156,18,0.15); border: 1px solid var(--amber); color: var(--amber); }
+  .verdict-dot { width: 12px; height: 12px; border-radius: 50%; background: var(--amber); }
+  .card { background: var(--card); border: 1px solid var(--border); border-radius: 8px; padding: 1.5rem; margin-bottom: 1.5rem; }
+  table { width: 100%; border-collapse: collapse; margin: 1rem 0; }
+  th, td { text-align: left; padding: 0.75rem; border-bottom: 1px solid var(--border); }
+  th { color: var(--muted); font-weight: 600; font-size: 0.85rem; text-transform: uppercase; letter-spacing: 0.05em; }
+  .tag { display: inline-block; padding: 0.15rem 0.6rem; border-radius: 4px; font-size: 0.8rem; font-weight: 600; }
+  .tag-go { background: rgba(46,204,113,0.15); color: var(--green); }
+  .tag-nogo { background: rgba(231,76,60,0.15); color: var(--accent); }
+  .severity-high { color: var(--accent); font-weight: 600; }
+  .severity-medium { color: var(--amber); }
+  .dissent-box { background: rgba(231,76,60,0.08); border-left: 3px solid var(--accent); padding: 1rem 1.25rem; border-radius: 0 6px 6px 0; margin: 1rem 0; }
+  .dissent-box strong { color: var(--accent); }
+  ol { padding-left: 1.5rem; }
+  li { margin-bottom: 0.6rem; }
+  .back-link { margin-top: 3rem; padding-top: 1.5rem; border-top: 1px solid var(--border); }
+  .back-link a { color: var(--muted); text-decoration: none; }
+  .back-link a:hover { color: var(--text); }
+  .scorecard { display: grid; grid-template-columns: 1fr 1fr; gap: 1rem; margin: 1rem 0; }
+  @media (max-width: 600px) { .scorecard { grid-template-columns: 1fr; } }
+</style>
+</head>
+<body>
+<div class="container">
+
+<h1>VerdictTank Review</h1>
+<p style="color:var(--muted)">HotNow Savannah v2 - August 11, 2026</p>
+
+<div class="verdict-badge conditional">
+  <span class="verdict-dot"></span>
+  CONDITIONAL GO
+</div>
+<p>3-1 majority. Conditions must be met before capital deployment or stakeholder commitment.</p>
+
+<div class="card">
+<h2>Panel Scorecard</h2>
+<table>
+  <tr><th>Judge</th><th>Model</th><th>Verdict</th><th>Key Position</th></tr>
+  <tr>
+    <td>Sonnet 5</td><td>Claude</td><td><span class="tag tag-go">Conditional Go</span></td>
+    <td>7 fixable conditions. Proposal is salvageable with honest re-audit of competitor table and code reuse estimate.</td>
+  </tr>
+  <tr>
+    <td>Opus 4.8</td><td>Claude</td><td><span class="tag tag-go">Conditional Go</span></td>
+    <td>Concurred. Added 3 demand-side omissions: cold-start, CAC realism, willingness-to-pay unvalidated.</td>
+  </tr>
+  <tr>
+    <td>Gemini 2.5 Pro</td><td>Google</td><td><span class="tag tag-nogo">No-Go</span></td>
+    <td>IQHub/TownIQ claim is a founder integrity issue, not a correctable condition. Financial model broken. Requires teardown and restart.</td>
+  </tr>
+  <tr>
+    <td>GPT-5</td><td>OpenAI</td><td><span class="tag tag-go">Conditional Go</span></td>
+    <td>Sonnet directionally right but overconfident. Proposed 12 hard gates with reframed scope: 10-12 weeks, $25-35K.</td>
+  </tr>
+</table>
+</div>
+
+<div class="dissent-box">
+  <strong>Dissent noted:</strong> Gemini 2.5 Pro issued a firm No-Go, arguing that the IQHub/TownIQ competitor misrepresentation constitutes a founder integrity failure that invalidates the proposal regardless of corrections to other numbers. The majority (3/4) disagreed, classifying it as a mischaracterization from sloppy research methodology rather than deliberate fabrication, and therefore correctable. This dissent is preserved in full.
+</div>
+
+<h2>Unanimous Findings</h2>
+
+<div class="card">
+<table>
+  <tr><th>Severity</th><th>Finding</th><th>Remediation</th></tr>
+  <tr>
+    <td class="severity-high">High</td>
+    <td><strong>IQHub/TownIQ competitor mischaracterized.</strong> TownIQ is civic/HOA software. IQHub is unrelated. No combined local discovery platform exists under that name.</td>
+    <td>Re-derive the entire competitive landscape table from primary sources. Do not present to stakeholders until re-verified.</td>
+  </tr>
+  <tr>
+    <td class="severity-high">High</td>
+    <td><strong>"70% exists" claim is overstated.</strong> Super Search v2 is a web search aggregator, not an event discovery engine. Event connectors, social signal ingestion, and real-time ranking are all unbuilt.</td>
+    <td>Independent code-level reuse audit. Real estimate: 25-30%. Rebuild timeline, budget, and capital ask from that number.</td>
+  </tr>
+  <tr>
+    <td class="severity-high">High</td>
+    <td><strong>3-4 week MVP timeline is not credible.</strong> All 4 judges agreed it's too short. Consensus range: 6-12 weeks for a functional pilot.</td>
+    <td>Adopt GPT-5's scope-freeze pilot plan: events-first, 5-6 sources, 10-12 weeks.</td>
+  </tr>
+  <tr>
+    <td class="severity-medium">Medium</td>
+    <td><strong>Visitor numbers overstated.</strong> Proposal says 15M+; Visit Savannah's official 2024 figure is 12.9M.</td>
+    <td>Use 12.9M. Publically verifiable numbers must be correct.</td>
+  </tr>
+  <tr>
+    <td class="severity-medium">Medium</td>
+    <td><strong>TAM stacking is aggressive.</strong> $100B+ from overlapping, paywalled, and unverifiable reports. Classic double-counting.</td>
+    <td>Rebuild TAM/SAM/SOM bottom-up from Savannah-specific numbers. Show the math.</td>
+  </tr>
+  <tr>
+    <td class="severity-medium">Medium</td>
+    <td><strong>"Competitive vacuum" is misleading.</strong> Yelp, Google Maps, TikTok, Instagram, Facebook Groups, Eventbrite, Meetup, and hotel concierge channels all serve Savannah users. No city-specific app exists, but incumbents own default behavior.</td>
+    <td>Reframe as incumbent displacement, not vacuum. State the specific behavioral wedge.</td>
+  </tr>
+  <tr>
+    <td class="severity-high">High</td>
+    <td><strong>Two-sided cold-start unaddressed.</strong> The model needs businesses to attract users, and users to justify $97/mo to businesses. Neither exists at launch. No sequencing plan.</td>
+    <td>Supply a two-sided launch-sequencing plan. Which side gets subsidized first?</td>
+  </tr>
+  <tr>
+    <td class="severity-high">High</td>
+    <td><strong>GTM zero-CAC assumption is the real fragility.</strong> Reddit + SCAD ambassadors with zero paid acquisition for 6 months is a prayer, not a plan. The entire net-loss figure depends on this holding.</td>
+    <td>Allocate $3-5K for GTM. Require signed MOUs with SCAD orgs, hotels, and anchor venues before launch.</td>
+  </tr>
+  <tr>
+    <td class="severity-medium">Medium</td>
+    <td><strong>Consumer $4.99/mo WTP is unvalidated.</strong> Local discovery is a category users expect free. 200-500 paying subscribers with free substitutes available is optimistic.</td>
+    <td>Run a 2-week landing page test. If paid conversion is below 2%, default to free consumer tier and monetize SMB-first.</td>
+  </tr>
+  <tr>
+    <td class="severity-medium">Medium</td>
+    <td><strong>MCP server is not a moat.</strong> Easily replicated integration surface. Differentiation must come from proprietary data, curation, and partnerships.</td>
+    <td>Reclassify as speculative optionality, not defensibility. Do not build until consumer traction exists.</td>
+  </tr>
+  <tr>
+    <td class="severity-medium">Medium</td>
+    <td><strong>Capital ask is too low.</strong> $7.5-13.5K understates data ops, moderation, QA, GTM, and contingency even at corrected timeline.</td>
+    <td>Rebudget at $25-35K with explicit runway and ops allocation.</td>
+  </tr>
+</table>
+</div>
+
+<h2>Reframed Scope (GPT-5 Consensus Gate)</h2>
+
+<div class="card">
+<h3>Pilot Parameters</h3>
+<table>
+  <tr><th>Dimension</th><th>Proposal v2</th><th>VerdictTank Consensus</th></tr>
+  <tr><td>MVP timeline</td><td>3-4 weeks</td><td>10-12 weeks</td></tr>
+  <tr><td>Capital required</td><td>$7.5-13.5K</td><td>$25-35K</td></tr>
+  <tr><td>Scope</td><td>Full consumer + business platform</td><td>Events-first pilot: 5-6 sources, dedupe, venue pages, save/share</td></tr>
+  <tr><td>GTM budget</td><td>$0 (organic only)</td><td>$3-5K (ambassadors, collabs, local ads)</td></tr>
+  <tr><td>Y1 ARR target</td><td>$42K-$102K</td><td>Deferred: hit pilot gates first</td></tr>
+  <tr><td>Consumer monetization</td><td>$4.99/mo from launch</td><td>Landing page test first; default to free if <2% conversion</td></tr>
+</table>
+
+<h3>Launch Gates</h3>
+<ol>
+  <li>1,000+ weekly active users</li>
+  <li>30%+ week-4 retention</li>
+  <li>10%+ venue click-through rate</li>
+  <li>3%+ SMB lead conversion</li>
+  <li>Under 10% content error rate</li>
+</ol>
+
+<h3>GTM Prerequisites</h3>
+<ol>
+  <li>Signed MOUs with 2-3 SCAD organizations</li>
+  <li>Signed MOUs with 3+ hotels/concierges</li>
+  <li>10-15 anchor venues committed to list or approve listings</li>
+  <li>Pre-sell 15 SMBs on a 3-month $50-100/mo pilot</li>
+</ol>
+</div>
+
+<h2>Consensus Verdict</h2>
+<div class="card">
+<p><strong>Majority (3/4): CONDITIONAL GO</strong></p>
+<p>The core business concept has merit. Savannah's student population, tourism density, and competitive dynamics (no city-specific aggregator) create a real opportunity. The proposal's flaws are in execution details, not concept viability: competitor research needs primary sourcing, build estimates need code-level audit, and GTM needs signed partnerships before launch.</p>
+<p style="margin-top:1rem">With the reframed scope (10-12 weeks, $25-35K, events-first pilot, hard launch gates), the project is worth attempting. Without these conditions, it's a No-Go.</p>
+<p style="margin-top:1rem; color:var(--muted); font-size:0.9rem">Review conducted August 11, 2026. Pipeline: Research (Phase 1) → Conductor Review (Sonnet 5, Phase 2) → Validation (Opus 4.8) → Cross-Check (Qwen failed, replaced with GPT-5) → Cross-Check (Gemini 2.5 Pro). Cost: ~$0.55.</p>
+</div>
+
+<div class="back-link">
+  <a href="./">Back to HotNow Savannah v2 Proposal</a>
+</div>
+
+</div>
+</body>
+</html>

projects/resend-transactional-email.md

@@ -0,0 +1,69 @@
+# Resend — Transactional Email Platform (Future Project)
+
+**Status:** PLACEHOLDER — not_started
+**Prepared for:** Germaine Brown / IT Pro Partner
+**Date:** August 14, 2026
+**Trigger:** Projects that email END USERS (confirmations, receipts, notifications) need per-domain `From:` with SPF/DKIM. The shared relay (SiteGround / MXroute) can only send as its own hosted domains — it can't send "as" arbitrary client domains.
+
+---
+
+## Why this exists (and what it is NOT)
+
+Two distinct email needs, don't conflate them:
+
+| Need | Recipient | Correct pattern | Status |
+|---|---|---|---|
+| Contact-form notification | Site owner | One trusted sender + per-domain routing + Reply-To | ✅ MSP Form Handler already does this |
+| End-user email (confirm/receipt) | The submitter / customer | **Per-domain `From`** with SPF/DKIM | ❌ Needs Resend |
+
+The MSP Form Handler (`forms.itpropartner.com`) notifies **site owners** only — `noreply@itpropartner.com`, per-domain recipient mapping in `config/domains.yaml`, Reply-To = submitter. That is the correct architecture and does not change.
+
+But some projects email **end users** directly — WordPress contact-form auto-replies, appointment confirmations, receipts, welcome emails. Those should come **from the client's own domain** (`hello@client.com`) so they pass SPF/DKIM and avoid spam. The shared relay can't do that. **Resend can.**
+
+## Which projects need it
+
+- Current WordPress sites that send confirmation/auto-reply email to end users (enumerate at activation).
+- Future projects with receipts / confirmations / notifications.
+- MSP Form Handler — only if/when it gains submitter auto-acknowledgements.
+
+## What Resend gives us
+
+- **Per-domain verification:** add SPF + DKIM (optional DMARC) to each client domain, then send "as" that domain.
+- **One account, many domains**, domain-scoped API keys.
+- Dedicated IP reputation + bounce/complaint tracking — better deliverability than a shared-hosting relay.
+- **SMTP + REST API.** WordPress via WP Mail SMTP / FluentSMTP or a lightweight Resend plugin; custom apps via the REST API.
+
+## Cost (verify current pricing at activation)
+
+- Free: ~3,000 emails/mo, 100/day, single domain.
+- Pro: ~$20/mo — ~50k/mo, unlimited domains, dedicated IP add-on.
+- For multi-client domain needs, Pro (unlimited domains) is the likely tier.
+
+## Architecture (planned)
+
+- One Resend account under `g@germainebrown.com`.
+- Per client needing it: verify their domain (SPF + DKIM in their DNS — we control most via Cloudflare), create a domain-scoped API key.
+- WordPress: WP Mail SMTP pointed at Resend with the client's domain as sender.
+- Custom apps (FastAPI / Node): Resend REST API, per-domain sender.
+- API keys live in Vaultwarden — never in plaintext or committed config.
+
+## Activation checklist
+
+- [ ] Create Resend account.
+- [ ] Decide Free vs Pro (Pro for unlimited domains).
+- [ ] Verify first client domain (SPF + DKIM records).
+- [ ] Wire first project (WordPress plugin or API).
+- [ ] Set DMARC on sending domains (`p=none` → `quarantine` as volume grows).
+- [ ] Store API key in Vaultwarden.
+- [ ] Add to backup/DR inventory if it becomes critical path.
+
+## Decision points (open)
+
+- Resend vs AWS SES vs Postmark (Resend = default pick; simplest DX, cheap).
+- Single account with per-domain keys vs per-client accounts.
+- Sending subdomain convention: `mail.client.com` (recommended) vs apex `client.com` — subdomain keeps SPF/DKIM/DMARC clean and isolated.
+
+## Related
+
+- `smtp-relay-configuration` skill — netcup outbound only allows 2525; shared relay rejects non-hosted MAIL FROM.
+- MSP Form Handler lives at `/var/www/msp-forms` on app3 (FastAPI, `config/domains.yaml`, `config/settings.yaml`).

projects/scirium/04-business-proposal-v2.md

@@ -0,0 +1,544 @@
+# Scirium Business Proposal v2 (DRAFT FOR REVIEW)
+
+**Product:** Scirium (pronounced "sigh-ree-um") - a white-label, self-hosted AI (artificial intelligence) knowledge assistant for document-driven SMBs
+**Version:** v2 (assembled from the marketing, technical, financial, and legal remediation sections; supersedes the v1 proposal dated 2026-08-16)
+**Date:** 2026-08-17
+**Status:** DRAFT FOR REVIEW - recommended verdict: **GO with conditions** (churn gate, acquisition-maturation gate, professional trademark clearance, Phase 0 data-loss-prevention (DLP) spike). One-line version: build Scirium against a capacity-constrained ramp at a corrected $167K build, and gate all growth spending on measured pilot-cohort churn at or below roughly 3.5% per month.
+
+---
+
+## 1. Executive Summary
+
+Scirium is a white-label, self-hosted knowledge assistant for document-driven SMBs (small and midsize businesses; 10 to 200 employees; MSP (managed service provider) clients, healthcare-adjacent practices, legal, accounting). Every knowledge domain inside a business becomes its own scoped chat channel backed by that business's real documents, served through Rocket.Chat, with grounded, cited answers. The hard invariant is one channel equals one knowledge domain, one attached agent, and one scoped knowledge source, enforced in the schema rather than in prompt text, which is why answers are provably grounded and provably isolated. The never-fabricate rail sits on top: mandatory citations and an explicit "I could not find an answer" fallback when retrieval comes back weak. The beachhead customer will be identified as one of IT Pro Partner's (ITPP) existing MSP customers.
+
+The v1 proposal received a CONDITIONAL GO from internal critical review, with seven flaws and seven conditions. This v2 is the remediation, and it changes the recommendation's reason, not its direction. **v1 said build it because the returns are spectacular. v2 says build it because the returns are good if and only if monthly churn holds at or below roughly 3.5% and the acquisition motion matures off cold-outbound cost within the first year.** v1 rested on two load-bearing fictions: a $1,000 CAC (customer acquisition cost) that counted no labor and no failed pilots, and a 12-month revenue ramp that assumed no tenant ever cancels. Both are rebuilt here; the honest numbers still clear the bar, but as a good managed-software business, not the ten-month near-zero-marginal-cost business v1 described.
+
+What survived review unchanged: marginal cost of about $0.42 to $0.50 per 1,000 queries (immaterial even at a 5x LLM (large language model) price rise); the white-label wedge, re-verified against live vendor sources (no incumbent among Glean, Guru, Notion, Moveworks, and Microsoft Copilot offers white-label branding as a purchasable product; a negative finding from public sources and a go-to-market moat, not a technical one); and gross profit per tenant of about $443 per month, a strong per-unit engine.
+
+
+
+The numbers that matter in v2:
+
+- Build cost: ~$68K to **~$167K** (1,667 hours incl. contingency; connector re-estimated to a 275-hour midpoint; 360 hours of new v2 work (runtime DLP 90, the Rocket.Chat DLP app 50, three-lane routing 60, ops automation 160)).
+- CAC: $1,000 to **$3,088 warm / $6,205 cold / $2,053 mature**. LTV (lifetime value):CAC **4.8:1 warm @3%** (7.2:1 mature, 2.9:1 @5%); the old 14.8:1 headline is the arithmetic artifact of the fictional $1,000 CAC, labeled as such.
+- Ramp rebuilt **net of churn**; recommended planning case is the new **capacity-constrained scenario**: 26 gross adds, 22.05 tenants @3%, **$72.2K net revenue**, $123.9K year-one CAC spend, operating breakeven crossed in month 6 @3%. The Aggressive 90-tenant case is not executable under the 3.4-wins-per-month sales-capacity ceiling and is struck.
+- Opex reconciled to **$5,310/mo** (v1's $4,000 valued product management and admin/insurance at zero). Operating breakeven **12 tenants** steady-state; **26 to 40** cash-neutral while growing. Gross margin is a **band, never a single figure**: 94.4% at the 10-minute support target state, 88.4% at 30-minute stress, ~62-65% at the 1.9 support hours per tenant per month the v1-era ops model implies; support hours are the swing variable and a first-class KPI (key performance indicator).
+- Payback: **~20 months** build-cost at Realistic @3% (extrapolated estimate), ~18-22 months at the recommended $499 floor, **38-64 months fully loaded** (CAC-dominated, unchanged).
+
+The plan adds two validation gates. Gate 1: the beachhead pilot with an existing ITPP MSP customer, strictly internal staff knowledge, no PHI (protected health information). Gate 2 (a commitment): before any ramp figure is treated as a forecast, Scirium must land and fully onboard a second pilot arms-length from the founding relationship with no PHI exposure, in a law firm, accounting firm, or retail group. One friendly pilot proves the software works; it does not prove the product can be sold to a stranger, and the difference between those two claims is the whole business case.
+
+The three conditions the numbers impose: (1) measure real churn in the pilot cohort before scaling spend, at or below 3.5% per month; (2) productize acquisition inside year one so CAC trends from $3,088 toward $2,053; (3) do not fund cold outbound acquisition, which is not viable at any modeled churn rate. With those conditions held, fund the $167K build against the capacity-constrained ramp.
+
+---
+
+## 2. Problem & Opportunity
+
+Every organization past a handful of employees has the same failure: new hires take weeks to become useful because institutional knowledge is scattered across SharePoint libraries, PDFs, shared drives, and senior staff memory; repetitive questions interrupt the most expensive people in the building; policy changes never fully propagate, so some staff always operate on a superseded rule. The answer is almost always already written down; it is simply not findable by the person who needs it. Independent studies converge on knowledge workers losing roughly a fifth to a third of the workday to hunting for information that already exists (McKinsey, IDC (International Data Corporation), and Deloitte estimates are frequently recycled and should be treated as directional, not precise). The cost does not scale down with company size, but the tooling that fixes it does not scale down either.
+
+Why every current answer fails: asking a manager interrupts the highest-paid person and does not scale; manual search costs 10 to 20 minutes per hunt with poor recall; printed binders go stale immediately; public consumer LLMs have no access to internal documents, fabricate confidently, and staff paste confidential text into a third-party service; and enterprise knowledge AI works only at enterprise price and enterprise procurement. The incumbents are not failing to serve small firms by accident; their cost structure makes small deals irrational for them. Glean is reported at roughly $50 to $75 per user per month with minimums commonly near 100 seats, a floor near $60,000 per year (third-party; Glean does not publish list pricing). Microsoft 365 Copilot is $30 per user per month on an annual commitment on top of a required base license, or $23.50 in an SMB bundle (vendor-confirmed): the buyer pays per seat, forever, to Microsoft. Guru no longer publishes a per-seat price at all (vendor-confirmed). And nobody sells the brand: even where an incumbent will take a smaller deal, the customer ends up using a product with someone else's name on it, which removes the reason an MSP would ever champion it.
+
+The buyer is not buying "AI." The buyer is buying the removal of four line items: senior staff time lost to answering the same twelve questions; new-hire ramp time; errors and rework caused by staff following a superseded policy; and the compliance anxiety of staff pasting internal documents into a public chatbot. The pitch has to be measured against those four, because they are the only things the buyer will notice on a renewal call.
+
+---
+
+## 3. Product Overview
+
+Scirium is a white-label, self-hosted internal staff knowledge assistant. Every knowledge domain inside a business (employee handbook, billing, IT (information technology) help, front office, policies) becomes its own scoped chat channel backed by that business's real documents. The product model references no industry; the same core serves a 12-person dental practice and a 200-person law firm.
+
+**The core primitive: the hard invariant.** Every channel is exactly three things bound together: one knowledge domain, one attached AI agent (a domain-tuned persona), and one scoped knowledge source (one vector namespace over an allowlisted document set). A channel cannot span two domains, and an agent cannot serve two channels. This is enforced in the schema with unique constraints running in both directions, not in application code. It is the reason answers are provably grounded (only one scope feeds the answer) and provably isolated (a billing question cannot surface a neighboring domain or a neighboring tenant).
+
+**The never-fabricate rail.** Two mandatory behaviors, both product-defining: every answer carries inline citation markers and a Sources footer linking each cited document; and empty or below-threshold retrieval returns "I could not find an answer" with no sources. Hallucination is the one risk that kills the product: if Scirium ever states a wrong policy or billing rule confidently, trust collapses. This is why the answer engine ships before any "do" capability.
+
+**What changed in the safety posture.** v1 claimed PHI protection was "enforced at ingestion." That claim was not defensible: ingestion filters inspect documents pulled from SharePoint, but they do not see a staff member typing patient data directly into a chat message, and that path reached the third-party LLM unfiltered. v2 adds runtime PHI/PII (personally identifiable information) detection at two independent enforcement points that fail closed (the chat layer and the orchestrator boundary), described in Section 8.2. The v1 claim that data "never leaves the customer's estate" was also factually wrong for the LLM call; v2 replaces it with an explicit three-lane routing model with stated tradeoffs (Section 8.3).
+
+**Honest scope language.** Scirium is not a HIPAA (Health Insurance Portability and Accountability Act)-compliant system and must never be sold as one. It is an internal staff knowledge tool with defense-in-depth controls that reduce the probability of PHI entering the pipeline; detection is probabilistic, and the product is not offered under a Business Associate Agreement. [to verify with counsel before publication: the exact customer-facing phrasing]
+
+---
+
+## 4. Market & Competitive Landscape
+
+### 4.1 Market size
+
+Bottom-up, per-seat, blended at $10 per user per month, deliberately below every enterprise incumbent's verified or reported price:
+
+```
+TAM (total addressable market) = ~1.0B global knowledge workers x $10/mo x 12         = ~$120B/year
+SAM (serviceable addressable market) = 1.0B x ~46% (SMB share of private-sector employment) = ~460M workers
+      x $10/mo x 12                                        = ~$55B/year
+SOM (serviceable obtainable market) = $55B x 1% (category winner, 3 to 5 year horizon)     = ~$550M/year
+      $55B x 0.1% (year 1 to 2, realistic)                 = ~$55M/year
+```
+
+US (United States)-only cross-check: roughly 100M US knowledge workers, of whom SMBs employ about 46% (US Small Business Administration data), gives a US SMB SAM near $5.5B; a 1% capture is $55M per year. The honest framing: market size is not the risk. Distribution is the risk, which is why Sections 6 and 7 carry the load in this document.
+
+### 4.2 The category is real, funded, and consolidating
+
+| Company | Signal | Confidence |
+|---|---|---|
+| Glean | $150M Series F at a $7.2B valuation (June 2025); $100M ARR (annual recurring revenue) early 2025, $200M on 8 December 2025 | Vendor-confirmed |
+| Moveworks | Acquired by ServiceNow for $2.85B; announced 10 March 2025, completed 15 December 2025 | Vendor-confirmed |
+| Notion | Enterprise search across Teams, SharePoint, and OneDrive in Notion 2.51 (13 May 2025) with unlimited Notion AI in Business and Enterprise plans | Vendor-confirmed |
+| Microsoft 365 Copilot | $30 per user per month add-on, annual commitment, on top of a required base license | Vendor-confirmed |
+
+A $2.85B acquisition closed, a company doubling ARR in nine months (Fortune, December 2025), and Microsoft bundling the capability into its own suite: the category is validated and getting more competitive at the top, which is precisely why Scirium must not compete on generic "AI search over your documents."
+
+### 4.3 Corrected competitive landscape
+
+Replaces the v1 table; two v1 claims that were factually wrong are corrected in 4.4.
+
+| Competitor | Price | Deployment | White-label? | Real gap Scirium exploits |
+|---|---|---|---|---|
+| Glean | ~$50-75/seat, minimums commonly near 100 seats, floor near $60K/yr (third-party) | Glean Hosted (software as a service (SaaS) on Google Cloud Platform (GCP)) or Customer Hosted, a Glean-operated managed tenant in the customer's AWS (Amazon Web Services)/GCP account (vendor-confirmed) | No white-label found; partner material reads "Powered by Glean" | Price and seat floor, plus true self-hosting: Glean's docs state Customer Hosted "is not a traditional self-hosted model," runs only in GCP/AWS |
+| Moveworks | Not published; third-party estimates near six figures/yr | ServiceNow platform | No white-label found | IT/HR (human resources) ticket deflection tied to a ServiceNow estate, not general internal knowledge |
+| Guru | No public per-seat price as of Aug 2026 (vendor-confirmed); third-party trackers cite ~$25/seat, 10-seat min | Vendor cloud only | No white-label found | No self-hosting, no rebrand, no channel-level scope isolation; moving upmarket |
+| Notion AI | Bundled into Business and Enterprise plans with unlimited Notion AI (vendor-confirmed) | Vendor cloud only | No white-label found | Vendor-cloud-only hosting, no rebrand, no hard per-channel scope isolation. Notion does connect to Microsoft 365 (M365; 4.4) |
+| Microsoft 365 Copilot | $30/seat/mo add-on, annual commitment; SMB bundle $23.50 (vendor-confirmed) | Microsoft cloud only | No white-label found; Copilot Studio permits name/icon changes but runs on Microsoft infra | Per-seat cost forever, Microsoft-cloud lock-in, no rebrandable resale, no self-hosting |
+| Coveo / Dust | Enterprise quote-based; Dust per-seat plus credits | Vendor cloud | [to verify] | No SMB tier; agent-builder toolkit, not turnkey |
+| Open WebUI, Onyx, AnythingLLM | Free and open source | Self-host | Open source, so branding is technically possible, but no commercial white-label program | Raw RAG (retrieval-augmented generation) toolkits: no per-tenant isolation guarantees, no managed M365 connector, no vertical templates, no support contract |
+
+### 4.4 Two v1 claims that were wrong, corrected
+
+**Notion AI does have a Microsoft 365 connector.** The v1 claim "Q&A only over Notion content; no M365 connector" is false and must never be said in front of a prospect. Notion ships a SharePoint and OneDrive AI Connector (documented in Notion's own help center), announced in Notion 2.51 (13 May 2025); it maps to existing Microsoft permissions and syncs hourly. The defensible version: the connector is beta, gated to Business/Enterprise plans, initial connection can take up to 36 hours, and Notion AI cannot read SharePoint site pages or personal OneDrives. Real coverage limits, not "no M365 connector."
+
+**Glean is not purely SaaS.** Glean documents two deployment models; Customer Hosted (previously Cloud-Prem) places a Glean tenant inside the customer's own AWS or GCP account. Glean's own documentation says Customer Hosted "is not a traditional self-hosted model" and is "equivalent to a hosted-SaaS model, where Glean still has minimal access to operate it," and Glean maintains support access. The correct differentiator is not "we self-host and they cannot." It is: we self-host on infrastructure you or your MSP controls, at a price point Glean's seat minimum structurally cannot reach, and you can put your own name on it.
+
+---
+
+## 5. Positioning, Differentiation & the White-Label Wedge
+
+For document-driven small and mid-sized organizations in compliance-sensitive work, Scirium is the knowledge assistant that runs on infrastructure you control, wears your brand, and can prove where every answer came from. Unlike enterprise platforms that require a hundred seats and put their own logo in front of your staff, and unlike consumer chatbots that guess, Scirium is scoped one domain at a time and will tell your team "I could not find an answer" rather than invent one.
+
+The four differentiators, in order of durability. **1. White-label, as a purchasable product.** Verified in Section 4 as unmatched by any of the five incumbents checked; the wedge and the entire reason the MSP channel can exist. **2. The one-channel-one-agent-one-scope invariant.** Enforced in the schema, not with prompt instructions; competitors do broad permission-aware search across the whole corpus, which is genuinely useful for a 5,000-person enterprise and a liability for a 30-person firm. **3. The never-fabricate rail.** Mandatory citations and the explicit fallback; hallucination risk is an acknowledged limitation of the incumbents too, and Scirium's difference is that the fallback is a hard product behavior rather than a recommended configuration. **4. Economics that permit SMB pricing.** Marginal cost of roughly $0.42 to $0.50 per 1,000 queries against a $199 to $2,000 per month band; real and durable, but a supply-side advantage only. v1's error was treating a cost advantage as if it were a distribution advantage.
+
+**The white-label wedge, re-verified, with honest caveats.** The greenlight condition was to confirm the wedge rather than assume it. Live check across Glean, Guru, Notion, Moveworks, and Microsoft Copilot: **no vendor among the five publishes, documents, or markets a white-label or OEM (original equipment manufacturer) rebranding program for its knowledge assistant.** Each ships as a named, vendor-branded product; Glean's partner material reads "Powered by Glean," which is co-branding and the opposite of white-label; Copilot Studio permits per-agent name/icon changes and a custom chat canvas, the closest any incumbent gets, but the agent still runs on Microsoft infrastructure with per-seat Microsoft licensing. Two caveats: this is a **negative finding from public sources** - the defensible claim is "no major incumbent offers white-label branding as a purchasable product"; and **the wedge is a go-to-market moat, not a technical one** - any vendor could add a branding layer, but none is likely to restructure per-seat enterprise pricing to make a 20-seat reseller deal worth closing.
+
+**Where a competitor genuinely beats us**, stated plainly because a proposal that claims no weaknesses gets discounted entirely: breadth of connectors (Glean and Notion connect to dozens of systems; Scirium v1 connects to SharePoint and OneDrive, so a prospect with critical knowledge in Salesforce, Zendesk, or Jira is not our buyer yet); brand safety of the incumbent choice (nobody was ever fired for buying Microsoft); company-wide discovery (if the customer's actual need is "search everything at once," the scope invariant is a constraint they will resent); and roadmap velocity (we will not out-feature Glean; we compete on segment, price, brand ownership, and isolation).
+
+**Strategic implications.** Do not lead with "AI chat over your documents": as of 2026 the two tools an SMB owner most likely already has, Microsoft 365 Copilot and Notion AI, both do grounded Q&A over SharePoint and OneDrive. Lead with what they verifiably do not have: your brand on it, your infrastructure under it, and hard per-channel scope isolation. Assume the price floor for generic knowledge Q&A moves toward zero over time; Scirium's premium has to be justified by isolation, branding, and control, which are architecture, not features.
+
+---
+
+## 6. Go-To-Market & Sales Engine
+
+The v1 go-to-market described a beachhead, an MSP channel, and a three-step motion, but no sales engine: no funnel math, no lead sources, no capacity model, no quota, no honest CAC. The review's sharpest finding: the realistic 48-tenant ramp requires about **4 net-new signed and onboarded tenants every month, sustained for a year**, and no engine to deliver that exists.
+
+The v2 answer is not a bigger sales engine; it is an honest capacity model. Costing CAC in hours (Section 9.4) exposes that acquisition is rate-limited by principal and technician availability: the ceiling is **3.4 warm wins per month** (and 1.7 cold wins) at a realistic 40 principal-hours plus 80 technician-hours per month. Sustaining 4 net-new tenants per month therefore requires either the partner channel maturing or a **dedicated sales and onboarding hire at roughly $6,000 to $9,000 per month, which is not in the $5,310 opex**. Without that hire, the sustainable add rate is **2 to 3 per month**, exactly the capacity-constrained scenario this plan builds on. The Aggressive 90-tenant case is not executable under the ceiling (its month-12 schedule of 13 adds alone needs roughly 620 technician hours) and is struck from the plan unless explicitly re-costed with dedicated headcount.
+
+Scirium is a **sales-led motion at an SMB price point**; bottom-up CAC lands at **$3,088 warm / $6,205 cold / $2,053 mature** (Section 9.4). The v1 phrase "CAC is near zero because we own distribution" should be struck: owned distribution shortens the first conversation but does not eliminate the sales motion, and it runs out after the first dozen relationships (warm pool modeled at 12 tenants).
+
+Working backwards from 4 closes per month: 7 pilots (60% pilot-to-paid), 20 qualified discovery calls (35%), 29 booked calls (70%), 115 engaged leads (25%), roughly **950 to 1,000 top-of-funnel touches** (12% engagement). That is the honest headline. The demand is real, but the **supply side binds first** (3.4 wins per month); both constraints have to hold, and v1 satisfied neither.
+
+**Motion A: founder-led direct sales.** Target: 2 of the monthly closes in months 1 through 6, dropping to 1 to 2 as the channel scales; the founder closes the first fifteen to twenty deals personally. Lead sources in priority order: the ITPP managed-services base (highest trust, strictly finite, realistically 5 to 10 closes in total); vertical association and peer networks (bar associations, CPA (certified public accountant) societies, dental and orthodontic study clubs); targeted outbound to a built list (sequenced email plus phone, with the compliance hook that tests best: "your staff are pasting client documents into ChatGPT"); narrow vertical content; and trigger-based outreach (firms that just posted five or more job openings). Five sales assets are required before outbound opens, including a demo tenant that can show a cited answer and a deliberate "I could not find an answer" response, and an objection-handling sheet opening with the now-correct objection: "we already have Copilot." Cadence: 40 touches per day, 8 discovery calls per week, 5 pilots.
+
+**Motion B: the MSP and reseller white-label channel.** Target contribution: 2 of 4 monthly closes by month 6, and 3 to 4 by month 12 as partners mature. The pitch takes one sentence: the MSP is asked for an AI offering by its clients and has nothing to sell; reselling Glean or Copilot means putting a vendor's brand in front of a client the MSP has spent years owning; Scirium is the only option checked here that lets the MSP put its own name on a working knowledge assistant.
+
+| Element | Terms |
+|---|---|
+| Partner margin | 30% of recurring revenue on partner-sourced tenants, rising to 40% above 10 active tenants |
+| Onboarding fee | Partner keeps 100% of the setup and document-audit fee |
+| Branding | Full white-label: partner's name, logo, colors, and domain; Scirium is invisible to the end client |
+| Floor | No minimum commitment; a minimum kills partner signups in this segment |
+| Our delivery obligation | We host and operate the infrastructure and provide tier-2 support; partner owns the client relationship and tier-1 |
+
+Partner recruitment: roughly 50,000-plus MSPs operate in the United States (third-party, directional); a realistic year-one target is **8 to 12 signed partners, of which perhaps 4 to 6 become genuinely productive** (channel programs follow a power law; planning for uniform partner productivity is how channel forecasts fail). Before a partner counts: enablement session plus certification call, branded demo tenant, collateral, deal registration (any account a partner registers is theirs for 90 days, and our direct motion does not touch it), and first-deal support.
+
+**Blended plan, reconciled to capacity.** The v1-era schedule (2/mo ramping to 5 to 6/mo, 44 to 50 cumulative) does not hit 4 per month until month 4 or 5; the executable year-one plan is the capacity-constrained schedule (26 gross adds, roughly 2 to 3 per month, front-loaded on the warm pool then throttled to the cold-acquisition ceiling), detailed in Section 9.6. The Realistic 48-add schedule is the plan only if the dedicated hire is funded.
+
+**The delivery constraint, which v1 omitted entirely.** Signed is not onboarded, and onboarded is what gets paid. Each new tenant requires provisioning, an M365 connector configuration, a document library audit, channel and agent and scope setup, ingestion and spot-check validation, an admin training session, and a two-week check-in. This labor is **already costed inside CAC**: a warm win carries 7.1 principal hours plus 23.8 technician hours including the failed-pilot loading (Section 9.4). The 40 to 56 hours per month of onboarding at 4 tenants per month is real and funded through CAC; what must still be planned is delivery capacity (runbook proven at Gate 1, executed by a non-builder at Gate 2, onboarding under 8 hours per tenant by month 9, or the model caps out around 5 tenants per month).
+
+**Metrics that decide whether this is working**, reviewed monthly, each with a failure threshold: new tenants (4/mo by month 5 under the funded-hire plan; investigate below 3); blended CAC ($2,050 to $2,600; the floor is the mature-state CAC of $2,053; investigate above $3,500); pilot-to-paid (60%; below 40% is a problem); time to activated (under 14 days); onboarding hours (falling toward 8); weekly active staff (above 40% of seats; below 25% is the leading churn indicator); queries per active user (above 3); the "I could not find an answer" rate (5% to 15%); confirmed fabrications (zero; any instance is stop-and-fix); producing partners (5+ by month 12); gross churn (under 3%; above 5% trips the treadmill, Section 9.8).
+
+The three most likely ways this go-to-market fails. **1. Top-of-funnel starvation after the warm list runs out** (early warning: engaged leads below 60 per month in month 3). **2. Channel signed but not selling** (track producing partners, never signed). **3. Onboarding backlog turning sales wins into churn** (cap signings at proven onboarding capacity; activation rate is the real growth metric).
+
+What is required to start: Gate 1 complete (beachhead pilot live, closed loop proven, onboarding runbook written); Gate 2 committed and funded (arms-length non-PHI pilot with all four pass conditions met before ramp spend); the five sales assets of Motion A complete; the partner program; a named onboarding owner; and a real acquisition budget consistent with $2,050 to $2,600 blended CAC, not a near-zero-CAC assumption.
+
+---
+
+## 7. Pricing
+
+| Tier | Price | Includes |
+|---|---|---|
+| Starter | $199/tenant/mo (up to 10 users, +$15/user beyond) | 1 M365 connector (SharePoint + OneDrive), Rocket.Chat, 50K docs, 1 channel scope set |
+| Business | $499/tenant/mo (up to 25 users, +$25/user beyond) | Teams connectors, unlimited channel scoping, single sign-on (SSO; OpenID Connect (OIDC)/Security Assertion Markup Language (SAML)), 250K docs, SLA (service-level agreement)-backed support, monthly DLP report |
+| Enterprise | $2,000/tenant/mo (annual contract) | Dedicated instance, unlimited seats, custom connectors, SCIM (System for Cross-domain Identity Management), signed DPA (data processing agreement), custom retention, white-glove onboarding, 99.9% SLA |
+
+Every tier includes the core grounded Q&A engine, the channel-scoped isolation invariant, citations, audit logs, and the no-PHI guardrail; higher tiers unlock more v2 capabilities and white-glove onboarding, not better core answer quality. The legal and compliance package is advisory-only and available at every tier (Section 10).
+
+**Enterprise tier benchmark.** $2,000 per tenant per month is priced against what the incumbents charge for comparable scope: a 100-seat Microsoft 365 Copilot deployment costs about $3,000 per month ($30/seat) on top of required base licenses, and Glean's commonly cited ~100-seat minimum lands near $5,000 to $7,500 per month (Section 4.3). The gap is deliberate: the Enterprise tier funds its own dedicated infrastructure (Section 8.7, Option B), unlimited seats, white-glove onboarding, a signed DPA, and a 99.9% SLA, none of which the lower tiers carry.
+
+**Base case.** The base financial case uses the conservative 60/30/10 tier mix, which blends to **$469.10 per tenant per month** (0.60 x 199 + 0.30 x 499 + 0.10 x 2,000). The v1 practice of modeling a $450 safety haircut is dropped because ARPU (average revenue per user) is not the fragile assumption; churn and CAC are. All revenue figures in Section 9 use $469.10.
+
+**Strategic pricing lever (explicitly not in the base model).** The technical team's cost analysis (Section 8.6) shows that a $199 Starter tier cannot carry the fully-loaded per-tenant cost of a managed self-hosted fleet. Their recommendation is a **strategic lever, labeled as such and kept out of the base model**: retire the $199 Starter tier and enforce a $499 floor. That lifts blended ARPU toward $700 to $800 per tenant per month, raises fully-loaded contribution to roughly **$202 per tenant per month** at the current operations model, and pulls operating breakeven back to **22 to 23 tenants** on the fully-loaded basis (fleet engineering included: ~$202 contribution per tenant per month against $4,500 per month fleet-wide engineering; 15 to 16 once support hours are automated down toward 1.0 per tenant per month). On the COGS (cost of goods sold)/opex basis the steady-state breakeven stays at 12 tenants (Section 9.8); the two bases are deliberately separate. The base case deliberately keeps the conservative $469.10 mix; the $499 floor is the lever a decision-maker can pull if the support-hours KPI does not converge.
+
+---
+
+## 8. Technical Architecture
+
+### 8.1 The canonical layer split
+
+| Layer | Responsibility | Owns |
+|---|---|---|
+| Rocket.Chat | Chat transport only | One workspace per tenant, MIT `fossify` build with EE (Enterprise Edition) stripped; rooms, users, messages; plus the chat-layer DLP app. Zero intelligence |
+| Orchestrator | All intelligence | Multi-tenant FastAPI service: tenancy, agents, kb_scope, M365 connector, retrieval, DLP, LLM routing, posting |
+| Presidio analyzer/anonymizer | DLP verdicts | Self-hosted, loopback only, no external calls |
+| Postgres + pgvector | State and vectors | Tenants, channels, agents, scopes, documents, chunks, messages, dlp_events |
+| admin-ai (LiteLLM) | LLM routing | Lane-aware routing with in-lane fallback chains only |
+| Wasabi S3 (Simple Storage Service) | Object storage | M365 sync staging, backups, agent assets, audit exports |
+
+The intelligence never lives in the chat layer. That split is what makes multi-tenancy clean and white-labeling a server-side concern instead of a per-tenant code fork.
+
+### 8.2 Runtime PHI/PII DLP, failing closed
+
+v1's ingestion-only filter caught PHI in documents but missed PHI typed into chat messages, DMs (direct messages), attachments, or REST (Representational State Transfer) API (application programming interface) injections, and retrieved chunks could carry PHI into the prompt. v2 adds two enforcement points that fail closed: chat-layer DLP (pre-persistence) and orchestrator-boundary DLP (pre-LLM). The orchestrator is the only component that talks to the LLM, so it cannot be bypassed.
+
+**Layer 1: chat-layer DLP (pre-persistence).** A Rocket.Chat Apps-Engine app shipped with each tenant's provisioning bundle implements pre-send hooks (`IPreMessageSentPrevent` to block, `IPreMessageSentModify` to redact, `IPreMessageUpdatedPrevent`, `IPreFileUpload`). It calls the orchestrator DLP endpoint over HTTPS (Hypertext Transfer Protocol Secure), HMAC (hash-based message authentication code)-signed, and receives `allow`/`redact`/`block`; block prevents the send, redact rewrites the body before persistence so raw PHI is never written to MongoDB. Layer 1 stops PHI from ever landing in the tenant's chat database.
+
+**Layer 2: orchestrator-boundary DLP (pre-LLM).** The orchestrator independently inspects the inbound question, the retrieved chunks, and conversation history immediately before the prompt is built. On BLOCK the LLM call is **not made at all**, a refusal is posted, a `dlp_events` row is written, and the audit record confirms zero tokens were sent. This layer is not optional and not tenant-disableable.
+
+**How it fails closed**: DLP unreachable, timeout, 5xx, or malformed body all deny; an unhandled exception denies (the verdict defaults to `block`, reassigned only on an explicit `allow`); a tenant admin disabling the Rocket.Chat app leaves Layer 2 enforcing with a `dlp_degraded` flag and ITPP alert; DLP failing at startup means the application refuses to start; missing Presidio models fail startup. There is **no fail-open toggle in code**, and a CI (continuous integration) test kills the DLP dependency and asserts both layers deny, gating deployment.
+
+**The detection engine.** Microsoft Presidio, Apache-2.0, self-hosted on loopback, three stages: regex and checksum recognizers for structured identifiers (SSN (Social Security number), MRN (medical record number) patterns, member and group numbers, NPI (National Provider Identifier), CPT (Current Procedural Terminology) and ICD (International Classification of Diseases)-10 codes, account numbers, phone, email, full dates of birth) plus per-tenant patterns; NER (named entity recognition) recognizers (PERSON, LOCATION, DATE_TIME, US_SSN, MEDICAL_LICENSE); and contextual co-occurrence rules (a name plus DOB (date of birth), member id, or clinical term is PHI; a name alone is not). BLOCK: SSN, MRN, member id, NPI, account, card, name-plus-clinical-context, full DOB in isolation [to verify with counsel]; REDACT: email, phone, address; ALLOW and log: name alone. An identical guardrail at the admin-ai (LiteLLM) proxy is defense-in-depth only [to verify against the deployed version].
+
+**Honest limitations, stated in the proposal and in the customer contract.** Presidio's own documentation states there is no guarantee it will find all sensitive information. Recall is not 100%; false positives will annoy staff (per-tenant thresholds starting at 0.7 plus an allow-list); a determined insider can obfuscate data through (DLP is a guardrail against accident and habit, not against a motivated malicious insider); detection is English-only at launch [to verify: Spanish]. DLP adds ~300 to 650 ms per query (250 ms Layer 1, 400 ms Layer 2, both deny on expiry); the NER model adds 1.5 to 2.5 GB RAM (random access memory) per analyzer [to verify under load]. `dlp_events` stores entity types (never matched values), counts, confidence, a salted hash, and failure mode; raw values are never written, logged, or sent externally. A monthly DLP report is a Business-tier deliverable.
+
+### 8.3 Three-lane LLM routing and the data-control promise
+
+The v1 claim that data "never leaves the customer's estate" was factually wrong for the LLM call: the prompt carries the staff question plus retrieved chunks, the customer's internal knowledge, sent verbatim to a third party; a self-hosted proxy in front of a hosted model is not self-hosted inference; and DeepSeek's published privacy policy stores user-provided content on servers in the People's Republic of China, a procurement stopper for US regulated-adjacent customers [to verify: current DeepSeek enterprise/API terms].
+
+The corrected statement that ships: chat, documents, vectors, audit logs, and DLP processing are fully self-hosted inside the ITPP-controlled estate; **inference is the one component that may leave the estate**, depending on the routing lane the tenant selects. Routing is a per-tenant (and optionally per-channel) configuration column enforced by the orchestrator's LLM client, and fallback chains are defined within a lane, never across lanes.
+
+- **Lane A: cost-optimized** (default for non-sensitive tenants). DeepSeek V4 Flash or Pro via admin-ai/LiteLLM, ~$0.42 to $0.50 per 1,000 queries. Third-party processing, possible out-of-jurisdiction storage, no contractual retention or training posture. Sold to non-regulated SMB only; **not sold to** healthcare-adjacent, legal, or accounting tenants (hard sales gate).
+- **Lane B: compliance-routed** (recommended for regulated-adjacent tenants). Enterprise-contracted hosted model with contractual data controls; reference target Azure OpenAI pinned to a US region (no-training committed, region pinning, modified-abuse-monitoring path removes the 30-day retention window [to verify: eligibility, pricing]). Cost ~$5 to $20 per 1,000 queries [to verify]; at $20, 3,000 queries per month costs $60 against a $499 subscription, so the tier still holds.
+- **Lane C: fully on-premise** (open-weight, nothing leaves). Open-weight instruct model served locally with vLLM or llama.cpp on a dedicated GPU (graphics processing unit) host (reference: Hetzner GEX44, RTX 4000 SFF (small form factor) Ada, ~EUR 184 to 234 per month [to verify pricing and VAT (value-added tax)]). Fixed infrastructure, not per-token; one host can serve multiple Lane C tenants. The only lane where "data never leaves" is literally true. Tradeoff: below frontier quality on complex reasoning, but much smaller on grounded extractive Q&A; must be demonstrated on prospect documents during the pilot [to verify: Phase 0 evaluation set].
+Gating requirements: a Data Processing Addendum with the model vendor (ITPP holds it as controller-side counterparty; no DPA, no Lane B); no-training and no-retention settings evidenced quarterly; regional routing pinned to a named US region or the tenant's jurisdiction (verified by endpoint inspection); and subprocessor disclosure with a right to object. Legal counterpart: the LLM vendor DPA (Section 10.6); contract exhibits update before any vendor or fallback change ships.
+
+Payload minimization regardless of lane: sends the question, retrieved chunks, persona, and citation instructions (all post-DLP); never sends tenant identity beyond an opaque id, user identity, email, document paths, URLs, or credentials (citations re-attached locally); sends nothing at all on DLP block.
+
+### 8.4 Tenant isolation
+
+Isolation, in order of trust: schema-level `tenant_id` with unique constraints on the channel/agent/scope binding; Postgres row-level security keyed on a session variable; vector namespace filtering on every similarity query; physical chat separation (one Rocket.Chat workspace plus one MongoDB per tenant, isolated volumes, loopback-only Mongo ports); HMAC webhook auth with timestamp-skew checking and message-id dedupe; and an adversarial isolation test in CI that must fail on every cross-tenant route, gating deployment.
+
+### 8.5 The M365 connector spike
+
+The v1 estimate of 145 hours ("authenticate, crawl, chunk, index") treated the connector as trivial. The design doc specifies far more: multi-tenant Entra registration with per-tenant admin consent and re-consent; the `Sites.Selected` grant workflow; token management with per-tenant caching and rotation; batched crawling with paging; delta sync with deltaLink persistence, 410 recovery, and delete reconciliation; change-notification subscriptions with a 6-hour renewal cron against a 3-day expiry; 429 throttling discipline; **ACL (access control list) mapping from SharePoint permissions onto kb_scope** (the genuinely hard part); file format and OCR (optical character recognition) handling; Graph search fallback with a staleness gate; sync observability; and DLP on the ingestion path with block-and-quarantine. The revised estimate is **250 to 300 hours, with 275 as the planning midpoint**, the riskiest engineering line in the project. A 40-hour timeboxed spike, drawn from the 275, must answer four questions before the full build is committed: does `Sites.Selected` consent work end to end against a real client tenant; does ACL mapping hold for the pilot's actual permission structure; what proportion of real documents parse cleanly and how many need OCR; and what is the real delta-sync throughput and throttling ceiling. The exit criterion is a written go or no-go on the 275-hour figure, with library-level scoping as the documented fallback.
+
+### 8.6 Per-tenant operations model and honest cost
+
+v1 priced 48 to 90 stateful per-tenant stacks at $3,500 to $4,000 per month total, which is not an operating cost, it is a rounding error. Architecture decision: shared Rocket.Chat for all tenants is rejected (it breaks the isolation invariant that is the entire moat); one host per tenant is rejected below the Enterprise tier; **container-per-tenant for isolation, shared-host density for economics, automation to make the fleet manageable** is the standard model; dedicated hosts are the Enterprise premium. Density: 10 to 12 stacks per 64 GB host (MongoDB memory is the binding constraint) [to verify with a load test at 3 tenants]; 4 to 5 hosts at 48 tenants, 8 to 9 at 90.
+
+**No per-tenant action may be manual** (160 hours of automation v1 did not carry): Terraform/Ansible host provisioning; one-command tenant provisioning (target under 15 minutes); de-provisioning with export and retention hold; canary fleet upgrades with automatic rollback; nightly encrypted backups to Wasabi S3 **plus an automated weekly restore test against a scratch host** (an unverified backup is not a backup); per-tenant synthetic probes, DLP liveness, and certificate expiry alerting; PHI-safe log aggregation; Vaultwarden secrets; runbooks for the top ten failure modes. **All of this must be automated before tenant number five.**
+
+Infrastructure is genuinely cheap (~$13 per tenant per month at 48 tenants). Human operations is the line v1 omitted entirely: 1.90 hours per tenant per month (fleet ops 0.35, support 0.60, DLP tuning 0.20, connector maintenance 0.30, incidents 0.25, account health 0.20) at $100/hr blended = $190. Steady-state engineering is $4,500 per month fleet-wide (v1: $3,500). Combined: roughly **$653 per tenant per month at 10 tenants, $297 at 48, $253 at 90** (ops $190 plus engineering share $4,500/n plus infra $13: at 10 tenants 190 + 450 + 13 = $653; at 48, 190 + 94 + 13 = $297; at 90, 190 + 50 + 13 = $253).
+
+At the v1 pricing mix, ~$297 per tenant per month at 48 tenants puts the all-in contribution margin near 34 to 37 percent, which is why pricing matters (Section 7.3) and why **support hours are the real product variable**: every 0.5 hours removed from the 1.9-hour baseline is $50 per tenant per month of margin, tracked as a first-class KPI. The financial model treats support as a COGS (cost of goods sold) line at 10 minutes per tenant per month (the automated target state); Section 9.3 reconciles both views into a gross-margin band.
+
+### 8.7 Deployment Options
+
+Exactly two standard deployment paths are offered, both fully managed by IT Pro Partner:
+
+- **Option A - ITPP-INFRA Shared.** Scirium runs on the existing netcup RS 4000/2000 servers alongside IT Pro Partner's own operations, backed by the same Wasabi S3 backup pipeline that powers ITPP production. Lowest cost and fastest start; the standard choice for the multi-tenant fleet and for most SMB tenants.
+- **Option B - Dedicated.** Dedicated netcup or Hetzner instances with a dedicated S3 bucket, fully managed by ITPP. For tenants that demand physical separation (compliance-sensitive buyers, Enterprise tier, or Lane C GPU hosting) or for the fleet at scale.
+
+Shared responsibility is explicit and identical in both paths: **ITPP manages everything below the application layer** (servers, Docker, TLS (Transport Layer Security), Postgres, backups and restore verification, monitoring, DLP infrastructure); the customer owns their documents and how they use the assistant.
+
+---
+
+## 9. Financial Model
+
+### 9.1 Assumption register
+
+Every figure in this section traces to one of these assumptions; items marked GUESS are estimates with no empirical basis yet and are the first things a pilot should measure.
+
+| ID (identifier) | Assumption | Value | Basis |
+|---|---|---|---|
+| A1 | Blended engineering rate, loaded | $100/hr | v1 figure, retained |
+| A2 | Principal/founder rate, loaded | $150/hr | GUESS. Opportunity cost of principal time on billable MSP work |
+| A3 | Technical staff rate, loaded | $85/hr | GUESS. Loaded cost of ITPP technician doing onboarding and pilot setup |
+| A4 | Tier mix | 60% Starter / 30% Business / 10% Enterprise | v1 figure, retained (conservative) |
+| A5 | Queries per tenant per month | 4,000 | GUESS. No usage data exists; drives LLM COGS only, which is immaterial |
+| A6 | Marginal LLM cost | $0.50 per 1,000 queries | v1 figure, retained. DeepSeek V4 Flash via LiteLLM |
+| A7 | Warm pool depth | 12 tenants | GUESS. Existing ITPP MSP relationships assumed to yield 12 warm wins before acquisition goes cold |
+| A8 | Warm pilot conversion | 80% | GUESS |
+| A9 | Cold pilot conversion | 50% | GUESS. Unvalidated category, no reference class |
+| A10 | Monthly churn | 3% base / 5% stress | v1 figure, retained. Unvalidated; no cohort data |
+
+### 9.2 Build cost: $167K (reconciled)
+
+The financial remediation section originally computed $105,300 (780 rescoped hours at +35% contingency), but that corrected only the M365 connector and missed 360 hours of new v2 work itemized by the technical team. The technical build table is authoritative: 1,235 hours of component work plus 432 hours of contingency at +35% equals **1,667 hours at $100/hr = $166,700, approximately $167K**. The range reflects the M365 connector band (250 to 300 hours) and the contingency band (30% to 40%): **$157K to $176K** (low 1,210 hours at 30% = 1,573 hours = $157,300; high 1,260 hours at 40% = 1,764 hours = $176,400).
+
+Component hours (v1 to v2): M365 connector 145 to 275; ingestion pipeline 70 to 80; RAG retrieval 90 unchanged; multi-tenancy 50 to 60; Rocket.Chat integration 70 unchanged; runtime DLP 0 to 90; Rocket.Chat DLP app 0 to 50; LLM lane routing 0 to 60; ops automation 0 to 160; auth/SSO 70 unchanged; admin UI (user interface) 90 to 110; testing and deploy 90 to 120. Subtotal 675 to 1,235; contingency +432; total 1,667.
+
+The contingency is justified: it carries ACL mapping against real SharePoint permission structures, DLP false-positive tuning, and the unverified Rocket.Chat Apps-Engine hook behavior under the MIT `fossify` build. The v1 range's $45K low end is struck. Every payback figure uses the $167K figure.
+
+### 9.3 COGS and the gross-margin band
+
+The v1 marginal-cost analysis survives review intact: LLM inference is near free at this query volume. v1 erred in calling $7 per tenant per month the whole of COGS; support labor is a cost of goods, not overhead, because it scales with tenant count.
+
+COGS per tenant per month: LLM inference $2.00 (A5 x A6); Rocket.Chat workspace plus MongoDB share $6.00 (GUESS); Postgres/pgvector storage plus embedding refresh $2.00 (GUESS); backups and object storage $1.50 (GUESS); tier-1 support labor $14.45 (GUESS, 0.17 hrs/mo at A3); monitoring share $0.55 (GUESS). Total $26.50. Blended ARPU is $469.10; gross profit is **$442.60 per tenant per month**. **Gross margin is a band, never an unconditional figure**: **94.4%** at the 10-minute support target state, **88.4%** at 30-minute stress, and roughly **62-65%** at the 1.9 support hours per tenant per month the v1-era ops model implies. The 98% claim from v1 is dropped.
+
+Technical's fully-loaded per-tenant view (~$297 per tenant per month at 48 tenants, including fleet engineering share) is a stricter all-in definition than COGS and puts the all-in contribution margin near 34 to 37 percent at the v1 pricing mix (Section 8.6); the COGS basis above feeds LTV and breakeven. Financial's verdict stands: even at triple support load the product remains high-margin. **Support load is not the thesis risk; churn and CAC are** - but support hours are a first-class KPI, because every 0.5 hours removed is $50 per tenant per month of margin.
+
+### 9.4 Honest CAC (bottom-up) and the sales-capacity constraint
+
+v1's $1,000 CAC counted no labor and no non-converting pilots; it was an assertion, not a calculation. Honest CAC is `(cost of every acquisition attempt) / (conversion rate) + (post-sale onboarding cost of the win) + (cash outlay)`, which loads failed pilots onto the winners.
+
+| CAC basis | Value | Multiple of v1 claim | CAC payback (months of gross profit) |
+|---|---|---|---|
+| v1 claimed | $1,000 | 1.0x | 2.3 |
+| Mature motion (target state) | $2,053 | 2.1x | 4.6 |
+| Warm (ITPP relationship) | $3,088 | 3.1x | 7.0 |
+| Cold (no relationship) | $6,205 | 6.2x | 14.0 |
+
+The warm build: 15.5 hours per attempt at $150/$85 loaded rates equals $1,610 per attempt; divided by 80% conversion it loads the 1-in-5 failed pilot to $2,012.50, plus post-sale close, onboarding (7.0 technician hrs), and training (3.0 technician hrs) equals $3,087.50. The cold build: 23.0 hours per attempt divided by 50% conversion, longer onboarding, and a $400 cash outlay (GUESS) equals $6,205. The mature build: 10.0 hours per attempt at $150/hr plus a $40 cash outlay per attempt (GUESS), at 75% conversion, equals $2,053.33 ((1,500 + 40) / 0.75). The review predicted CAC 3 to 5x higher; bottom-up lands at 3.1x warm and 6.2x cold. The review's "payback 12 to 15 months" reconciles as the CAC payback on cold acquisition (14.0 months), not the build-cost payback.
+
+**The sales-capacity constraint** (a consequence of honest CAC): acquisition is rate-limited by principal and technician availability. Warm wins cost 7.1 principal hours plus 23.8 technician hours each; cold wins cost 11.5 plus 48.0. At 40 principal-hours plus 80 technician-hours per month, the ceiling is **3.4 warm wins per month** (technician hours bind first) and **1.7 cold wins per month**. This falsifies the v1 Aggressive scenario, which requires 13 adds in month 12 (roughly 620 technician hours).
+
+### 9.5 LTV and LTV:CAC
+
+LTV is gross profit per tenant per month divided by monthly churn, the standard steady-state formula, undiscounted (generous: at 3% churn the average tenant life is 33 months, and money that far out is worth materially less than face value). LTV is $14,753 at 3% churn and $8,852 at 5% churn.
+
+| CAC basis | LTV:CAC @ 3% churn | LTV:CAC @ 5% churn | Verdict |
+|---|---|---|---|
+| v1 claimed $1,000 | 14.8:1 (the old headline) | 8.9:1 | Not real. Assumes free labor and zero failed pilots; retained only as the arithmetic artifact it is |
+| Mature motion $2,053 | 7.2:1 | 4.3:1 | Healthy. This is the target state |
+| Warm $3,088 | 4.8:1 | 2.9:1 | Acceptable at 3%. Below the 3:1 floor at 5% |
+| Cold $6,205 | 2.4:1 | 1.4:1 | Fails the 3:1 floor at both churn rates |
+
+The honest band is **4.8:1 to 7.2:1 at 3% churn**, which brackets the review's expected 4 to 7:1. At 5% churn the band is 2.9:1 to 4.3:1, and cold at 1.4:1 is value-destroying. Spending ceilings: at 3% churn the 3:1 minimum-viable CAC ceiling is $4,918 (4:1 = $3,688; 7:1 = $2,108); at 5% churn the 3:1 ceiling is $2,951. **Cold outbound CAC of $6,205 exceeds the 3:1 ceiling even at 3% churn: cold outbound acquisition is not viable at any modeled churn rate. The business is only viable on warm, referral, and productized-inbound acquisition.** That is a strategy finding, not just a number.
+
+Churn sensitivity: warm LTV:CAC is 7.2:1 at 2.0% monthly churn, 5.7:1 at 2.5%, 4.8:1 at 3.0%, 4.1:1 at 3.5%, 3.6:1 at 4.0%, 2.9:1 at 5.0%. **The critical threshold is roughly 3.5% monthly churn on warm acquisition**; above roughly 4.5% the warm motion drops under the 3:1 floor. Measuring actual churn in the pilot cohort is the highest-value information the business can buy, and it is cheap to buy.
+
+### 9.6 Twelve-month ramp, net of churn
+
+v1 reported $52.7K / $107.6K / $198.9K as 12-month revenue with churn explicitly excluded; those are gross bookings under an assumption that no tenant ever cancels. The rebuild applies a monthly cohort decay (`ending base = (opening base x (1 - churn)) + gross adds`, revenue recognized on the ending base at $469.10 ARPU, churn applied to the opening base only). The v1 monthly add schedules were not published and are reconstructed to land on v1's stated month-12 tenant counts; that reconstruction is an inference, not a v1 quotation.
+
+| Scenario | Month-12 tenants (gross) | Month-12 @3% | Month-12 @5% | v1 "12-mo revenue" | v2 net rev @3% | v2 net rev @5% |
+|---|---|---|---|---|---|---|
+| Conservative | 22 | 19.8 | 18.5 | $52.7K | $43.7K | $41.4K |
+| Realistic | 48 | 42.9 | 40.0 | $107.6K | $99.3K | $94.1K |
+| Aggressive | 90 | 80.2 | 74.5 | $198.9K | $190.3K | $180.1K |
+| Capacity-constrained (new) | 26 | 22.0 | 19.8 | not modeled | $72.2K | $67.4K |
+
+Net first-year revenue is 7.7% lower at 3% churn and 12.5% lower at 5% than gross bookings in the Realistic scenario (Conservative: 17.1% and 21.4% lower; Aggressive: 4.3% and 9.5%). Exit MRR (monthly recurring revenue): $9,289 (Conservative @3%), **$20,143 (Realistic @3%)**, $37,638 (Aggressive @3%), **$10,342 (Capacity-constrained @3%)**, an exit ARR of $124,104.
+
+**The capacity-constrained scenario is the recommended planning case.** Given the 3.4 warm wins per month ceiling and a warm pool of only 12, neither the Realistic nor the Aggressive schedule is executable with current staffing. Its gross adds (1, 2, 3, 3, 3, 2, 2, 2, 2, 2, 2, 2) front-load the warm pool then throttle to the cold-acquisition ceiling. At 3% churn the ending base reaches **22.05 tenants by month 12**, with MRR of $10,342 and cumulative net revenue of **$72,151**; at 5% churn it ends at 19.81 tenants and $67,378. Year-one CAC spend is **$123,920** (roughly 1.7x the net revenue it produces), and the scenario crosses operating breakeven (12 tenants) in **month 6 at 3% churn**. For contrast, the Realistic scenario's year-one CAC spend is $260,430, 2.6x the net revenue it produces and 1.6x the entire funded build cost; v1's model contained no such line, the largest single omission in the original financials. The Realistic case is achievable only with a dedicated sales and onboarding hire (roughly $6,000 to $9,000 per month) not in the $5,310 opex. The two cannot be mixed.
+
+### 9.7 Opex reconciliation
+
+v1's $4,000 per month could not be reconciled against its own $3,500 engineering line. Reconciled lines: engineering retainer $3,500 (35 hrs/mo at A1); shared infrastructure $95 (GUESS); backup and object storage baseline $45 (GUESS); tooling and licenses $120 (GUESS); product management (principal, 8 hrs/mo at A2) $1,200 (omitted in v1); admin/finance/insurance allocation (errors and omissions (E&O) insurance share, accounting, legal) $350 (omitted in v1). **Reconciled fixed opex: $5,310 per month**, versus v1's $4,000, a gap of +$1,310 (+32.8%). v1's figure is internally consistent only if product management and administration are valued at zero, the same free-founder-labor error that produced the $1,000 CAC. Annual fixed opex is **$63,720**, excluding any dedicated sales headcount (acquisition labor is costed inside CAC; a dedicated hire adds $6,000 to $9,000 per month).
+
+### 9.8 Breakeven
+
+Two distinct breakevens, kept separate because v1 blurred them. **Steady-state operating breakeven** (no growth spend): $5,310 / $442.60 = **12.0 tenants** (v1 said ~9; its figure followed correctly from its own $4,000 opex, only the input was wrong); contribution per tenant past breakeven is $442.60. **Cash-flow breakeven** (growth spend included) is the operationally relevant figure and v1 omitted it: at 2 monthly adds the business needs 25.9 tenants at warm CAC (40.0 at cold); at 3 adds, 32.9 (54.1 cold); at 4 adds, 39.9 (68.1 cold). Adding 4 cold tenants per month requires 68 existing tenants just to stay cash-neutral, and at 5% churn that base is not reachable at that add rate.
+
+**The structural finding at 5% churn.** Steady-state tenant count converges to `monthly adds / churn`: at 2 adds per month and 3% churn, 66.7 tenants, monthly gross profit $29,507, less opex $5,310 and CAC, nets +$11,787 per month (cold CAC) or +$20,090 (mature CAC). At 5% churn, 40.0 tenants, monthly gross profit $17,704, less opex and cold CAC, nets **-$16 per month: at 5% monthly churn on permanently cold acquisition, the business converges to exactly zero net cash flow. It runs forever, funds its own replacement churn, and never returns the build cost.** Not a bankruptcy scenario; a treadmill scenario, arguably worse because it consumes years before the failure becomes obvious. The escape is churn below 3.5% or CAC productized toward the mature figure. Both, ideally.
+
+### 9.9 Payback
+
+Two payback questions, reported separately, both on the corrected $167K build.
+
+**Build-cost payback** (gross profit less opex versus the build; CAC excluded as discretionary growth investment). The model's sensitivity analysis shows payback is nearly insensitive to build cost (it moved only ~2 months across a $31K swing in the original computation), so the correction from $105,300 to $167K adds roughly two months to the base case. At the corrected build: **approximately 20 months at Realistic and 3% churn (extrapolated estimate, labeled as such)**, against technical's range of **~18 to 22 months at the recommended $499 price floor and ~29 to 37 months at v1 pricing**; the capacity-constrained planning case lands at the upper end of the v1-pricing range (about 30 months at 3% churn). This is a two to three year payback managed-software business, not a ten-month near-zero-marginal-cost software business.
+
+| Payback basis | Figure | Basis |
+|---|---|---|
+| Build-cost, Realistic, 3% churn | ~20 months (extrapolated estimate; ~22 months on the assembled model) | $167K build against gross profit less opex |
+| Build-cost, $499 price floor | ~18 to 22 months | Section 7.3 lever; contribution ~$202/tenant/mo |
+| Build-cost, v1 pricing | ~29 to 37 months | Technical range (Section 8) |
+| Build-cost, capacity-constrained, 3% churn | ~30 months | Upper end of the v1-pricing range |
+| Fully loaded (build + opex + CAC) | 38 to 64 months | 38 at Realistic/3% with CAC maturing; 64 if CAC stays cold |
+
+These are outputs of the month-by-month cash-flow model; the static tables in Sections 9.1 to 9.8 do not contain the full monthly simulation, so these figures are not independently derivable from them (the model workbook is available on request).
+
+**Fully-loaded payback** (build plus opex plus honest CAC, the strictest test): **38 to 64 months**, driven by whether CAC matures after year one (38 months at Realistic, 3% churn) or stays at cold acquisition cost (64 months). Unchanged by the build-cost correction because it is CAC-dominated: year-one CAC spend of $124K to $260K ranges from 0.7x to 1.6x the build cost itself. At 5% churn with CAC staying cold, fully-loaded payback never arrives (the Section 9.8 treadmill). Per-tenant CAC payback: 7.0 months warm, 14.0 months cold, 4.6 months mature; v1's "2.3 months per tenant" was correct on a $1,000 CAC and meaningless once CAC is honest.
+
+### 9.10 Revised verdict
+
+**GO, with conditions, and for a different reason than v1 gave.** What survived review unchanged: marginal cost is negligible; gross profit per tenant is about $443 per month; and the $469.10 blended ARPU is conservative and retained. What broke and is now corrected: CAC was understated 3.1x (warm) to 6.2x (cold), and cold outbound is not viable at any modeled churn rate; the ramp was gross, not net (net first-year revenue is 7.7% lower at 3% churn and 12.5% lower at 5% in the Realistic scenario); opex was understated 33%; build cost was understated 146% (was $68K, now $167K); payback was understated (about 20 months build-cost at Realistic and 38 to 64 months fully loaded, versus 10 to 13 claimed); and the Aggressive scenario is not executable under the sales-capacity ceiling.
+
+The three conditions the numbers impose:
+
+| Condition | Threshold | Why it is the threshold |
+|---|---|---|
+| Measure real churn in the pilot cohort before scaling spend | At or below 3.5%/mo | At 3.5% warm LTV:CAC is 4.1:1; above roughly 4.5% the warm motion drops under the 3:1 floor |
+| Productize the acquisition motion inside year one | CAC trends from $3,088 toward $2,053 | Fully-loaded payback is 38 months with maturing CAC versus 64 months without, at 3% churn |
+| Do not fund cold outbound acquisition | Cold CAC $6,205 exceeds the 3:1 ceiling of $4,918 at 3% churn | Warm, referral, and productized inbound only |
+
+The recommendation: fund the $167K build against the capacity-constrained ramp, and treat measured churn in the first pilot cohort as the gate on all further growth spending.
+
+---
+
+## 10. Legal & Compliance
+
+*Disclaimer: this section is part of a business proposal prepared for internal planning purposes. It is not legal advice. Any legal, trademark, or regulatory conclusions herein must be confirmed by qualified counsel before the underlying decisions (naming, contracting, market launch) are finalized. Nothing in this document is a binding contract or legal advice to any prospective customer.*
+
+### 10.1 Trademark clearance
+
+**Scirium** (pronounced "sigh-ree-um") is the market-facing product name for all current and future use.
+
+A basic, non-professional clearance search was run: no exact match for "Scirium" was found as an existing registered trademark, company brand, or software product; the **closest phonetic and spelling neighbor is "Cirium,"** an existing, actively operating aviation data and analytics company (part of LexisNexis Risk Solutions / RELX, rebranded from FlightGlobal in 2019) with AI-adjacent offerings; "Scirium" differs from "Cirium" by a single leading "S," close enough that a professional would want to assess likelihood-of-confusion risk, particularly in the software/data-services classes. One incidental mention of "SCirium" appeared in an academic paper on blockchain-based air traffic management, unclear whether entity, codename, or typo. No conflicting software company, SaaS product, or enterprise-AI brand named "Scirium" was identified.
+
+**Honest statement of residual risk.** This search is not a clearance opinion: it does not check USPTO (U.S. Patent and Trademark Office) TESS (Trademark Electronic Search System) or equivalent registries, state registries, common-law use in commerce, domain and social-handle conflicts, or the goods/services classes that determine likelihood of confusion. The proximity to Cirium, an existing, active, well-funded brand in an adjacent data/analytics space, is a real finding that should not be waved away; the risk is plausibly low but not zero, and it has not been professionally assessed. **Recommendation: obtain a professional trademark clearance search and opinion from qualified counsel before (1) filing any trademark application for Scirium, (2) executing any reseller, franchise, or white-label agreement that extends brand rights to third parties, or (3) any large-scale paid marketing launch under the Scirium name.** This is a go/no-go gate for irreversible brand commitments, not a blocker for continued internal development, pilot use with the first client, or this proposal's approval.
+
+### 10.2 Advisory-only posture, available at every tier
+
+The legal and compliance package is **advisory-only** and is offered as guidance and template documentation across all pricing tiers; it is not gated behind the Enterprise tier, and no tier is sold as "compliant" or "certified" on the strength of a marketing claim. Every tenant receives the same baseline no-PHI acceptable-use policy, the same limitation-of-liability framing, and access to the same DPA template. Enterprise-tier customers additionally receive a signed, negotiated DPA and dedicated-instance options, but the underlying legal posture (no PHI, no BAA (business associate agreement), self-hosted control, advisory-only terms) is identical across tiers.
+
+### 10.3 ToS (terms of service)/MSA (master services agreement) and limitation of liability
+
+Scirium is offered under a ToS/MSA package that is explicitly advisory-only: proposed templates to be reviewed, finalized, and issued by qualified counsel before any customer signs a live agreement. Key terms the finalized ToS/MSA should address: scope of service (self-hosted, white-label, multi-tenant knowledge-assistant chat over customer-provided documents with grounded, cited answers); customer ownership of its documents, chat content, and outputs; explicit exclusion of PHI and other prohibited data categories; SLA terms by tier; term, termination, and data-return/deletion obligations on offboarding; and no unilateral material change to the no-PHI restriction or the DPA without customer notice.
+
+The MSA should include a standard limitation of liability structure, to be finalized by counsel: a liability cap limited to fees paid in the preceding 12 months (or a similarly-scaled fixed cap), except for carved-out categories (breach of confidentiality, gross negligence or willful misconduct, and express indemnification obligations); exclusion of consequential damages; no warranty of fitness for a specific regulated use (the service is not warranted as fit for use with PHI, regulated health data, or any use requiring a BAA); and an AI-output disclaimer: answers are grounded and cited but AI-generated, and the customer is responsible for independent verification before relying on any answer for a consequential business, medical, legal, or financial decision. Enforceability and exact cap amounts depend on jurisdiction and must be set by counsel.
+
+### 10.4 Acceptable Use Policy: no PHI
+
+The AUP (acceptable use policy) is a hard, contractual restriction, not just a technical guardrail, and it applies at every tier: customers may not ingest, upload, or otherwise introduce into Scirium any Protected Health Information as that term is understood under applicable health-privacy frameworks (patient records, diagnoses, treatment notes, medical record numbers, or similarly sensitive health identifiers), nor any other data category the customer is not permitted to disclose to a third-party processor. The AUP is the legal counterpart to the technical PHI controls: it puts the customer on contractual notice that the product is not designed, warranted, or licensed for PHI use, and that violation is a breach of the agreement independent of whether the technical filter catches every instance. The customer is responsible for configuring which document libraries are connected and for ensuring PHI is excluded before connection. Detected or reported PHI ingestion triggers immediate quarantine, customer notification, and remediation; repeated or willful violation is grounds for suspension or termination. This AUP is what allows Scirium to serve healthcare-adjacent verticals (a dental or orthodontic practice's non-clinical operational content, billing policy, HR and handbook material) without taking on the regulatory posture of a healthcare data processor.
+
+### 10.5 Data Processing Addendum
+
+A DPA template is available to any customer that requests one, at any tier, covering: roles (customer as data controller for its own documents and users' chat content; Scirium as data processor for the limited purpose of providing the service); scope of processing (ingestion, indexing, retrieval, and generation strictly within the customer's own tenant and channel scopes, consistent with the hard multi-tenancy invariant); a defined, disclosed sub-processor list (infrastructure hosting, the LLM vendor routed through the internal proxy, backup/object storage) with advance notice of material changes; data location and residency per the deployment option the customer selects (shared ITPP-managed estate or dedicated single-tenant instance), supporting regional routing commitments where required; security measures (encryption in transit and at rest, tenant isolation, signed internal service calls, audit logging of every question and answer); deletion and return on termination within a defined window with confirmation; and a defined breach/incident notification timeline with audit or attestation rights scaled to tier.
+
+### 10.6 LLM vendor DPA
+
+This is the legal counterpart to the three-lane routing architecture in Section 8.3 and directly addresses the review's shared flaw about external LLM data handling. The commitments below must be reflected in the actual contractual terms with whichever LLM vendors are in the routing chain, not just asserted in marketing copy: a **no-training commitment** (customer prompts and outputs are not used to train, fine-tune, or improve the vendor's models); a **no-retention or minimal defined retention commitment** (ideally zero retention beyond what is operationally required, or a short defined and disclosed period for abuse-monitoring purposes only, with deletion after that window); **regional routing** (where a customer's DPA or regulatory context requires it, the routing layer must direct that customer's LLM calls to a model or endpoint in a committed region, and the vendor DPA must reflect where processing actually occurs); **fallback-chain disclosure** (the same commitments must apply to every vendor in the fallback chain, and the DPA or an attached exhibit must list every vendor that can plausibly process a given customer's data); and the structural point that **no PHI reaches the LLM layer** because the AUP and technical controls block it before it enters the pipeline, so the LLM vendor DPA does not need to be, and should not be represented as, HIPAA-capable. If the architecture team changes a vendor or fallback chain, the LLM vendor DPA exhibit must be updated before the change ships.
+
+### 10.7 HIPAA / BAA posture
+
+Deliberately explicit and not softened in customer-facing material: **Scirium is not a HIPAA covered entity** (it does not provide healthcare treatment, payment, or health plan operations). **Scirium does not sign Business Associate Agreements**: because the product's contractual and technical design explicitly excludes PHI from the service, Scirium does not position itself as a HIPAA business associate, and a BAA is only appropriate where a vendor is knowingly handling PHI on behalf of a covered entity. A dental, orthodontic, medical-adjacent, or similar practice can use Scirium for non-clinical operational knowledge (employee handbook, billing policy and procedure, IT help, front-office training material) but must not connect any library or content source containing patient records or other PHI. This is a hard boundary of the product, not a tier-based upsell: "upgrade to get HIPAA support" is explicitly not an option. **If a prospective customer requires a BAA or PHI handling as a condition of purchase, that customer is out of scope for Scirium as currently designed.** This posture is the direct legal expression of the product's "permanently out of scope: patient records, PHI, HIPAA scope" hard rail; legal posture and product architecture agree by design.
+
+### 10.8 Data protection, jurisdiction, and liability summary
+
+Data location is known and disclosed at contracting time per deployment option; the only data flow that leaves the controlled estate is the LLM inference call itself, and the LLM vendor DPA is the mechanism that pins down where that hop occurs and under what retention terms. No claim of specific regulatory certification is made: this proposal does not claim SOC 2 (System and Organization Controls 2), ISO (International Organization for Standardization) 27001, HIPAA, or GDPR (General Data Protection Regulation) adequacy has been obtained. Indemnity, to be finalized by counsel: Scirium indemnifies the customer against third-party claims that the core, unmodified platform infringes third-party IP (intellectual property), subject to standard exclusions; the customer indemnifies Scirium against claims arising from its own content, its breach of the AUP (in particular PHI ingestion), and misuse of AI-generated outputs; and because the Scirium brand itself is cleared only at the basic-search level, any reseller or white-label agreement executed before a professional trademark opinion is obtained should include a clear allocation of risk and, ideally, a right to rename or rebrand on notice. The operating entity should carry technology E&O and cyber liability coverage sized to the customer base. Because PHI is contractually and technically excluded, the liability and indemnity structure is deliberately not sized for HIPAA breach exposure.
+
+---
+
+## 11. Team & Execution
+
+Scirium is built and operated by IT Pro Partner (ITPP), a managed services provider founded and led by **Germaine Brown**. ITPP brings the three assets the plan depends on: an existing managed-services client base (the warm pool of roughly a dozen relationships the financial model assumes, A7); an existing production infrastructure estate (netcup hosts, the Wasabi S3 backup pipeline, Vaultwarden secret management, the admin-ai LiteLLM proxy, and existing auth on app3) that Scirium shares rather than buys; and an existing operational discipline (backup verification, monitoring, runbooks, per-tenant managed-services delivery) that the operations model in Section 8.6 extends rather than invents. The team is small by design: the principal (founder), engineers, and technicians execute the build, sales, and onboarding; the go-to-market plan is explicitly founder-led for the first fifteen to twenty deals.
+
+The beachhead customer will be identified as one of ITPP's existing MSP customers, selected as the Gate 1 pilot and the reference case for its vertical. The second-pilot commitment (Gate 2) requires an arms-length tenant with no PHI exposure - a law firm, accounting firm, or retail group - signed through one of the Section 6 channels, before any ramp figure is treated as a forecast.
+
+**The execution constraint and the dedicated-hire trigger.** Execution is rate-limited by the same hours that drive CAC: at 40 principal-hours plus 80 technician-hours per month, the ceiling is 3.4 warm wins per month. The plan therefore commits to the capacity-constrained ramp (26 gross adds in year one) as the base case, which requires no additional headcount. The Realistic ramp (48 adds) is explicitly conditional on a **dedicated sales and onboarding hire at roughly $6,000 to $9,000 per month, added to opex**; the trigger is Gate 2 passing (repeatable arms-length onboarding) plus two consecutive months of the Section 6 metrics clearing their thresholds with a pipeline that supports the higher add rate. Until then, signing faster than onboarding capacity is the most likely failure mode of the entire plan, and it is managed by capping monthly signings at proven capacity.
+
+**Success definition.** Product v1 (the answer engine): a beachhead-customer staff member asks a policy question in a channel and gets a cited, correct answer, with zero fabricated answers across a week of real use, and the DLP fail-closed test passing before the proof slice is called complete. Business: 3 paying tenants with demonstrated ROI (return on investment) within 6 months of v1 launch, and Gate 2's four pass conditions met before any material sales or marketing spend. Technical: the MIT `fossify` white-label build produced in CI before the first non-pilot customer, no PHI ever ingested, isolation verified by an adversarial test, and automated backup restore verification proven on a scratch host.
+
+---
+
+## 12. Roadmap & v1-to-v2 Scope
+
+### 12.1 What product v1 is
+
+Product v1 is the answer engine plus the safety rails, and nothing else. The message flow is eight steps: staff sends a message mentioning the agent (or DMs it); chat-layer DLP inspects pre-persistence (block, redact, or allow; on block the message is never written); Rocket.Chat fires a signed, HMAC-authenticated webhook to the orchestrator; the orchestrator resolves tenant, channel, agent, and scope in one request context and dedupes on message id; retrieval runs pgvector semantic search over the channel's namespace, always filtered by tenant and scope, falling back to Microsoft Graph search below threshold; orchestrator-boundary DLP inspects the question plus retrieved context plus history and fails closed; prompt build and lane-aware dispatch with no cross-lane fallback; the answer posts back as the agent bot with inline citations re-attached locally from Postgres. v1 hard behaviors: never fabricate (below-threshold retrieval returns "I could not find an answer" with no sources), every answer carries citations, runtime DLP at both layers failing closed and non-disableable at the orchestrator, allowlisted ingestion with the pre-index PHI filter retained, and a full audit log of every question, answer, and DLP event.
+
+### 12.2 Built versus to build
+
+| Component | Status | Disposition |
+|---|---|---|
+| Product model and data architecture | Settled, 3 design docs committed | Done |
+| Data model (tenants, channels, agents, kb_scopes, documents, chunks, messages, users, dlp_events) | Spec'd, not built | Build |
+| Orchestrator (FastAPI, tenancy, retrieval, LLM client, lane routing) | Not built | Build |
+| Postgres + pgvector schema plus RLS (row-level security) | Spec'd, not built | Build |
+| Rocket.Chat workspace and bot integration | Phase 0 checklist spec'd, not built | Build |
+| Rocket.Chat DLP app (Apps-Engine) | New in v2, not spec'd | Build |
+| Presidio deployment, recognizer tuning, contextual rules | New in v2, not spec'd | Build |
+| M365 connector (Sites.Selected read) | Spec'd, not built | Build, re-estimated |
+| White-label FOSS (free and open source software) rebrand (fossify build) | Phase 1 gate, not started | Build |
+| Auth (Hexclave/Stack Auth, Entra OIDC) | Partially existing on app3 | Integrate |
+| IaC provisioning and fleet automation | New in v2, not spec'd | Build |
+| Lane C GPU inference host | New in v2 | Provision when the first Lane C tenant signs |
+
+### 12.3 Phase 0 proof slice, with the DLP gate and the R8 spike
+
+Phase 0 is one Rocket.Chat workspace for the beachhead customer (an existing ITPP MSP customer), a registered bot, one channel, one attached agent, one indexed document, and one grounded cited answer. Two additions to Phase 0 are mandatory. First, **the DLP fail-closed test must pass before the proof slice is called complete**, because retrofitting DLP after the pipeline exists is how it ends up bypassable. Second, the **load-bearing uncertainty spike**: the exact behavior of the Rocket.Chat Apps-Engine pre-send hooks (`IPreMessageSentPrevent` and related interfaces) under Rocket.Chat 7.4.0 with the MIT `fossify` build is unverified, and whether app hooks fire for messages created through the REST API is unknown; Layer 1 DLP depends on it. A timeboxed spike in Phase 0 must verify hook behavior end to end, and **DLP cannot be declared production-ready until it passes**. Alongside it, the 40-hour M365 connector spike of Section 8.5 runs, with a written go or no-go on the 275-hour estimate.
+
+### 12.4 Ops automation before tenant five
+
+The rule from Section 8.6: no per-tenant action may be manual, and all fleet automation (provisioning, de-provisioning, upgrades, backup restore verification, monitoring, log aggregation, secrets, runbooks, 160 hours of build) must be in place before the fifth tenant, because manual steps multiplied by the fleet is the ops treadmill that silently destroys margin.
+
+### 12.5 v2 risk-tiered capabilities
+
+The v2 roadmap adds risk-tiered "do" capabilities, ordered by write risk, none of which ever unlock patient/PHI scope or autonomous destructive action: reporting (structured extraction, SQL (Structured Query Language) aggregation, chart render; zero new write risk, auto-approved, the recommended first v2 ship); content generation (drafts, summaries, boilerplate; drafts only, with DLP applied to generated output too); housekeeping (move, rename, delete-to-recycle; destructive, propose-then-approve-then-audit, with a mandatory undo path); integrations (Graph delegated sendMail, calendar, webhooks; external side effects, approval required, with DLP inspecting outbound payloads because an integration is a new egress path); and design and brand (template render, image gen; template-driven only, because raw image generation garbles text). Reliable aggregation comes from structured extraction at index time plus a SQL tool, not from better prompting. **Permanently out of scope:** patient records, PHI as a system of record, HIPAA scope, clinical or radiographic image analysis (a separate FDA (Food and Drug Administration)-regulated product class), and autonomous destructive actions.
+
+### 12.6 Gating items before the first non-pilot customer
+
+(1) Fail-closed DLP test passing in CI, both layers, and the Phase 0 Apps-Engine spike passed; (2) adversarial cross-tenant isolation test passing in CI; (3) automated backup restore verification proven on a scratch host; (4) DPA in force with any Lane B vendor, with no-training and region pinning evidenced; (5) the MIT `fossify` white-label build produced in CI, never the stock EE image; (6) the 40-hour M365 connector spike returning a written go on the 275-hour estimate; (7) legal review of the customer-facing PHI language; and (8) professional trademark clearance obtained before any reseller or white-label agreement or paid launch.
+
+---
+
+## 13. Risks & Mitigations
+
+| Risk | Likelihood | Impact | Mitigation |
+|---|---|---|---|
+| Trademark: the "Cirium" phonetic neighbor | Low | High | Scirium is materially lower-risk than the retired v1 codename, but Cirium (LexisNexis/RELX aviation data and analytics, AI-adjacent) is a real residual risk, unprofessionally assessed. Professional clearance opinion required before trademark filing, any reseller or white-label agreement, or paid launch; reseller agreements carry a right to rename on notice until then |
+| DLP Layer 1 hook behavior unverified (R8) | Medium | High | Rocket.Chat Apps-Engine pre-send hook behavior under the MIT `fossify` build and for REST-API messages is load-bearing for chat-layer DLP. Mandatory Phase 0 spike; DLP not declared production-ready until it passes; Layer 2 orchestrator DLP remains non-bypassable regardless |
+| Churn above the critical threshold | Medium (unvalidated) | Critical | The model hinges on churn at or below ~3.5% monthly; above ~4.5% warm LTV:CAC drops under the 3:1 floor, and at 5% on permanently cold acquisition the business converges to zero net cash flow (the treadmill). Measure churn in the pilot cohort before scaling spend; activation rate, weekly active staff, and queries-per-active-user are the leading indicators |
+| Cold outbound CAC not viable | Certain | High | $6,205 exceeds the $4,918 3:1 ceiling at 3% churn. Warm, referral, and productized-inbound acquisition only; no cold-outbound budget |
+| Support hours (the gross-margin swing variable) | Medium | Medium | 1.9 hrs/tenant/mo implies ~62-65% COGS-basis margin; 10 min implies 94.4%. Every 0.5 hrs removed is $50/tenant/mo. Tracked as a first-class KPI; automation, self-serve admin tooling, better DLP tuning defaults, runbook-driven tier-1 |
+| Pricing: $199 tier cannot carry fleet cost | Medium | Medium | At v1 pricing the fully-loaded all-in contribution is near 34-37%. Strategic lever to retire $199 and enforce a $499 floor (contribution ~$202/tenant/mo, breakeven 22-23 tenants); base model stays conservative at $469.10 |
+| Sales-capacity ceiling | Certain | Medium | 3.4 warm wins/mo max; the Aggressive 90-tenant case is not executable. Capacity-constrained ramp as the base plan; dedicated hire at +$6-9K/mo only after Gate 2 and metric thresholds clear |
+| Hallucination (the risk that kills the product) | Medium | High | Never-fabricate rail, mandatory citations, "I could not find an answer" fallback, DLP on generated output; zero confirmed fabricated answers is a stop-and-fix event |
+| Cross-tenant or cross-channel leakage | Low | Critical | Schema constraints, RLS, vector namespace filtering, physical chat separation, HMAC auth, adversarial isolation test in CI gating deployment |
+| Onboarding backlog turning wins into churn | Medium | High | Cap monthly signings at proven onboarding capacity; activation rate is the real growth metric; runbook proven at Gate 1, executed by a non-builder at Gate 2 |
+| Channel signed but not selling | Medium | Medium | Track producing partners, never signed partners; certification call plus branded demo tenant required before a partner counts |
+| M365 connector fragility | Medium | Medium | 275-hour planning midpoint, 40-hour de-risking spike with written go/no-go, delta sync, backoff, Graph search fallback |
+| Rocket.Chat licensing in resale | Low | High | MIT `fossify` FOSS-only build in CI, never the stock EE image |
+| Competition moving downmarket | Medium | Medium | Lead with brand, infrastructure control, and isolation; assume the price floor for generic knowledge Q&A moves toward zero |
+
+---
+
+## 14. Change Summary v1 -> v2
+
+| # | Superseded v1 figure or position | v2 value or position |
+|---|---|---|
+| 1 | Build cost ~$68K (range $45K-$100K) | **~$167K** (1,667 hrs at $100/hr incl. +35% contingency; range $157K-$176K). Build cost understated **+146%** (was $68K, now $167K) |
+| 2 | M365 connector 145 hrs, no contingency | 275-hr planning midpoint (250-300 range), 40-hr de-risking spike, +35% contingency across all build hours |
+| 3 | CAC ~$1,000 | **$3,088 warm / $6,205 cold / $2,053 mature target** (CAC payback 7.0 / 14.0 / 4.6 months) |
+| 4 | LTV:CAC ~14:1 | **4.8:1 warm @3% churn**; 2.9:1 warm @5%; 7.2:1 at mature CAC @3%. The 14.8:1 headline is an arithmetic artifact of the fictional $1,000 CAC, labeled as such |
+| 5 | Opex $4,000/mo | **$5,310/mo reconciled** (+32.8%; the gap was PM (product management) and admin/insurance valued at zero); annual $63,720 |
+| 6 | Breakeven ~9 tenants | **12 tenants** steady-state; **26 to 40 tenants** cash-neutral while growing |
+| 7 | 12-month revenue $52.7K / $107.6K / $198.9K (gross bookings, churn excluded) | **Net of churn @3%: $43.7K / $99.3K / $190.3K**; @5%: $41.4K / $94.1K / $180.1K. Recommended planning case: capacity-constrained, $72.2K net @3% with $123.9K year-one CAC spend |
+| 8 | Full payback ~10-13 months | **~20 months build-cost at Realistic @3% (extrapolated estimate)**; ~18-22 months at the $499 floor; ~29-37 months at v1 pricing; **38 to 64 months fully loaded** (CAC-dominated) |
+| 9 | Gross margin 95-98% | **A band, never a single figure**: 94.4% at the 10-min support target state, 88.4% at 30-min stress, ~62-65% at the 1.9-hr support reality. Support hours are the swing variable and a first-class KPI |
+| 10 | Blended COGS ~$7/tenant/mo | **$26.50/tenant/mo** including support labor; per-tenant fully-loaded cost ~$297/mo at 48 tenants (technical view, Section 8.6) |
+| 11 | "Data never leaves the customer's estate" | **Corrected**: inference may leave. Three explicit lanes (cost-optimized, compliance-routed with DPA + region pinning + no-retention, fully on-premise open-weight) with stated tradeoffs and payload minimization |
+| 12 | "No PHI, enforced at ingestion" | **Runtime DLP at the chat layer and the orchestrator boundary, both failing closed**, with an audited dlp_events trail and honest limitations stated |
+| 13 | Ramp scenarios 22 / 48 / 90 tenants, no capacity model | Four scenarios net of churn; **Aggressive (90) struck** as not executable under the 3.4-wins/mo sales-capacity ceiling; capacity-constrained (26) is the recommended planning case |
+| 14 | Product name and trademark risk | **Scirium** is the sole market-facing name; clearance search found no exact match; closest phonetic neighbor Cirium flagged for professional review before filing or launch (see Section 10.1) |
+
+---
+
+## 15. Sources & Open Questions
+
+### 15.1 The [to verify] register (consolidated)
+
+**Engineering and architecture**
+- Rocket.Chat Apps-Engine `IPreMessageSentPrevent` behavior on Rocket.Chat 7.4.0 under the MIT `fossify` build, and whether app hooks fire for REST-API messages. **Load-bearing for DLP Layer 1; mandatory Phase 0 spike before DLP is declared production-ready.**
+- LiteLLM `pre_call` Presidio guardrail behavior on the deployed admin-ai version (reported issues with redacted content propagating to the effective upstream payload).
+- Presidio NER memory profile under load (1.5 to 2.5 GB per analyzer container) and Spanish-language model availability and quality.
+- Full date-of-birth blocking classification under HIPAA, and the customer-facing PHI language [with counsel].
+- Current DeepSeek enterprise/API terms (may differ from the consumer privacy policy).
+- Azure OpenAI modified-abuse-monitoring eligibility, approval timeline, and current per-token pricing.
+- Lane C open-weight model selection and license terms for commercial resale; Lane A versus Lane C answer-quality comparison on the Phase 0 evaluation set.
+- Hetzner GEX44 pricing and VAT treatment. Host density of 10 to 12 tenant stacks per 64 GB host [load test at 3 tenants].
+
+**Market and competitors**
+- Precise global and US knowledge-worker counts (aggregate estimates only, directional).
+- Glean pricing (~$50 to $75 per seat, ~100-seat minimum) and the $300M ARR milestone reached in late May 2026 (third-party, TechCrunch; up from the $200M ARR announced 8 December 2025 and cited in Section 4.2). Glean does not publish list pricing.
+- Guru per-seat pricing and 10-seat minimum (third-party; Guru no longer publishes pricing) and Guru white-label/custom-domain limitations (competitor-authored comparisons).
+- Moveworks current list pricing (not published). Coveo and Dust white-label availability.
+- Knowledge-worker search-time studies (McKinsey, IDC, Deloitte) - directional, frequently recycled.
+
+**Legal and brand**
+- Cirium's registered-mark status in the software/data-services classes; professional clearance opinion required before filing, reseller agreements, or paid launch. Domain and social-handle availability for the Scirium name.
+
+**Financial assumptions marked GUESS (A2, A3, A5, A7, A8, A9, A10)**
+- Principal rate $150/hr, technician rate $85/hr, queries 4,000/tenant/mo, warm pool 12, warm conversion 80%, cold conversion 50%, monthly churn 3%/5%, plus COGS line items, the cold-CAC cash outlay of $400, and mature-motion conversion of 75%. These are the first things a pilot should measure.
+
+### 15.2 Sources
+
+- This document assembles: the marketing remediation section (market, competitors, positioning, white-label wedge, GTM (go-to-market)), the technical remediation section (architecture, DLP, LLM routing, connector, ops, build cost), the financial remediation section (authoritative on revenue and unit economics), and the legal remediation section (trademark, ToS/MSA, AUP, DPA, BAA posture), all dated 2026-08-17; the v1 proposal (2026-08-16, team and product context; any v1 figure superseded by the remediation is replaced per Section 14); and the internal critical review (7 flaws, 7 conditions, CONDITIONAL GO).
+- Vendor-confirmed sources cited in the market section: Notion SharePoint and OneDrive AI Connector documentation and 2.51 release notes; Glean deployment documentation and press releases (Series F, $200M ARR); ServiceNow and Moveworks acquisition announcements; Microsoft 365 Copilot pricing pages; Microsoft Copilot Studio customization documentation; Guru pricing page.
+- Third-party, directional only: Glean and Guru pricing estimates (GoSearch, Onyx cost calculator, Workativ, Featurebase, Docsie); CAC benchmarks (Factors.ai, Digital Applied, SaaS Mag); US MSP population (Span Global Services).
+
+---
+
+*End of proposal. Nothing in this document is legal advice; all legal, trademark, and regulatory conclusions require confirmation by qualified counsel before the underlying decisions are finalized.*

projects/seo-ci-architecture.html

@@ -0,0 +1,311 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+  <meta charset="UTF-8">
+  <meta name="viewport" content="width=device-width, initial-scale=1.0">
+  <title>SEO Tool + IntelSight Architecture</title>
+  <link href="https://fonts.googleapis.com/css2?family=JetBrains+Mono:wght@400;500;600;700&display=swap" rel="stylesheet">
+  <style>
+    * { margin: 0; padding: 0; box-sizing: border-box; }
+    body {
+      font-family: 'JetBrains Mono', monospace;
+      background: #020617;
+      min-height: 100vh;
+      padding: 2rem;
+      color: white;
+    }
+    .container { max-width: 1100px; margin: 0 auto; }
+    .header { margin-bottom: 2rem; }
+    .header-row { display: flex; align-items: center; gap: 1rem; margin-bottom: 0.5rem; }
+    .pulse-dot { width: 12px; height: 12px; background: #22d3ee; border-radius: 50%; animation: pulse 2s infinite; }
+    @keyframes pulse { 0%, 100% { opacity: 1; } 50% { opacity: 0.5; } }
+    h1 { font-size: 1.4rem; font-weight: 700; }
+    .subtitle { color: #94a3b8; font-size: 0.8rem; margin-left: 1.75rem; }
+    .diagram-container {
+      background: rgba(15, 23, 42, 0.5);
+      border-radius: 1rem;
+      border: 1px solid #1e293b;
+      padding: 1.5rem;
+      overflow-x: auto;
+    }
+    svg { width: 100%; min-width: 950px; display: block; }
+    .cards { display: grid; grid-template-columns: repeat(3, 1fr); gap: 1rem; margin-top: 2rem; }
+    .card { background: rgba(15, 23, 42, 0.5); border-radius: 0.75rem; border: 1px solid #1e293b; padding: 1.25rem; }
+    .card-header { display: flex; align-items: center; gap: 0.5rem; margin-bottom: 0.75rem; }
+    .card-dot { width: 8px; height: 8px; border-radius: 50%; }
+    .card-dot.cyan { background: #22d3ee; }
+    .card-dot.emerald { background: #34d399; }
+    .card-dot.violet { background: #a78bfa; }
+    .card-dot.amber { background: #fbbf24; }
+    .card-dot.rose { background: #fb7185; }
+    .card h3 { font-size: 0.8rem; font-weight: 600; }
+    .card ul { list-style: none; color: #94a3b8; font-size: 0.72rem; }
+    .card li { margin-bottom: 0.3rem; }
+    .footer { text-align: center; margin-top: 1.5rem; color: #475569; font-size: 0.72rem; }
+  </style>
+</head>
+<body>
+  <div class="container">
+    <div class="header">
+      <div class="header-row">
+        <div class="pulse-dot"></div>
+        <h1>SEO Audit Tool + IntelSight — Shared Architecture</h1>
+      </div>
+      <p class="subtitle">Two products. One infrastructure. Zero product merge.</p>
+    </div>
+
+    <div class="diagram-container">
+      <svg viewBox="0 0 1050 720">
+        <defs>
+          <marker id="arrowhead" markerWidth="10" markerHeight="7" refX="9" refY="3.5" orient="auto">
+            <polygon points="0 0, 10 3.5, 0 7" fill="#64748b" />
+          </marker>
+          <marker id="arrow-cyan" markerWidth="10" markerHeight="7" refX="9" refY="3.5" orient="auto">
+            <polygon points="0 0, 10 3.5, 0 7" fill="#22d3ee" />
+          </marker>
+          <marker id="arrow-emerald" markerWidth="10" markerHeight="7" refX="9" refY="3.5" orient="auto">
+            <polygon points="0 0, 10 3.5, 0 7" fill="#34d399" />
+          </marker>
+          <marker id="arrow-violet" markerWidth="10" markerHeight="7" refX="9" refY="3.5" orient="auto">
+            <polygon points="0 0, 10 3.5, 0 7" fill="#a78bfa" />
+          </marker>
+          <pattern id="grid" width="40" height="40" patternUnits="userSpaceOnUse">
+            <path d="M 40 0 L 0 0 0 40" fill="none" stroke="#1e293b" stroke-width="0.5"/>
+          </pattern>
+        </defs>
+
+        <rect width="100%" height="100%" fill="url(#grid)" />
+
+        <!-- ============ USERS (left side) ============ -->
+        <rect x="20" y="120" width="110" height="55" rx="6" fill="rgba(30,41,59,0.5)" stroke="#94a3b8" stroke-width="1.5"/>
+        <text x="75" y="142" fill="white" font-size="10" font-weight="600" text-anchor="middle">ITPP Hosting</text>
+        <text x="75" y="158" fill="#94a3b8" font-size="8" text-anchor="middle">Clients</text>
+
+        <rect x="20" y="220" width="110" height="55" rx="6" fill="rgba(30,41,59,0.5)" stroke="#94a3b8" stroke-width="1.5"/>
+        <text x="75" y="242" fill="white" font-size="10" font-weight="600" text-anchor="middle">CI Buyers</text>
+        <text x="75" y="258" fill="#94a3b8" font-size="8" text-anchor="middle">SMB/Mid-Market</text>
+
+        <!-- ============ PRODUCT LAYER ============ -->
+
+        <!-- SEO Audit Tool box -->
+        <rect x="180" y="80" width="350" height="160" rx="8" fill="rgba(8,51,68,0.25)" stroke="#22d3ee" stroke-width="1.5"/>
+        <text x="192" y="98" fill="#22d3ee" font-size="10" font-weight="600">SEO Audit Tool</text>
+        <text x="192" y="112" fill="#64748b" font-size="8">Free / $49/mo Pro</text>
+
+        <!-- SEO features -->
+        <rect x="195" y="125" width="155" height="48" rx="5" fill="rgba(8,51,68,0.4)" stroke="#22d3ee" stroke-width="1"/>
+        <text x="205" y="142" fill="white" font-size="9" font-weight="600">Launch Audit</text>
+        <text x="205" y="156" fill="#94a3b8" font-size="7">Auto-runs on deploy</text>
+        <text x="205" y="167" fill="#94a3b8" font-size="7">50+ checks, fix-it snippets</text>
+
+        <rect x="360" y="125" width="155" height="48" rx="5" fill="rgba(8,51,68,0.4)" stroke="#22d3ee" stroke-width="1"/>
+        <text x="370" y="142" fill="white" font-size="9" font-weight="600">Monthly Crawl</text>
+        <text x="370" y="156" fill="#94a3b8" font-size="7">Trend tracking 0-100</text>
+        <text x="370" y="167" fill="#94a3b8" font-size="7">Score history per site</text>
+
+        <rect x="195" y="180" width="155" height="48" rx="5" fill="rgba(34,211,238,0.12)" stroke="#22d3ee" stroke-width="1" stroke-dasharray="3,3"/>
+        <text x="205" y="197" fill="#22d3ee" font-size="8" font-weight="600">✦ AI Fix Prompts</text>
+        <text x="205" y="211" fill="#94a3b8" font-size="7">Ready-made Claude/Cursor prompts</text>
+        <text x="205" y="222" fill="#94a3b8" font-size="7">per finding — copy, paste, done</text>
+
+        <rect x="360" y="180" width="155" height="48" rx="5" fill="rgba(34,211,238,0.12)" stroke="#22d3ee" stroke-width="1" stroke-dasharray="3,3"/>
+        <text x="370" y="197" fill="#22d3ee" font-size="8" font-weight="600">✦ GEO Visibility</text>
+        <text x="370" y="211" fill="#94a3b8" font-size="7">How visible in ChatGPT/</text>
+        <text x="370" y="222" fill="#94a3b8" font-size="7">Gemini/Claude/Perplexity?</text>
+
+        <!-- IntelSight box -->
+        <rect x="580" y="80" width="420" height="160" rx="8" fill="rgba(6,78,59,0.15)" stroke="#34d399" stroke-width="1.5"/>
+        <text x="592" y="98" fill="#34d399" font-size="10" font-weight="600">IntelSight</text>
+        <text x="592" y="112" fill="#64748b" font-size="8">$199 / $499 / $1,499/mo</text>
+
+        <rect x="595" y="125" width="190" height="48" rx="5" fill="rgba(6,78,59,0.4)" stroke="#34d399" stroke-width="1"/>
+        <text x="605" y="142" fill="white" font-size="9" font-weight="600">Competitor Intelligence</text>
+        <text x="605" y="156" fill="#94a3b8" font-size="7">Crunchbase + Hunter.io + OSINT</text>
+        <text x="605" y="167" fill="#94a3b8" font-size="7">Funding, team, product alerts</text>
+
+        <rect x="795" y="125" width="190" height="48" rx="5" fill="rgba(6,78,59,0.4)" stroke="#34d399" stroke-width="1"/>
+        <text x="805" y="142" fill="white" font-size="9" font-weight="600">Market Analysis</text>
+        <text x="805" y="156" fill="#94a3b8" font-size="7">Multi-provider search</text>
+        <text x="805" y="167" fill="#94a3b8" font-size="7">LLM synthesis &amp; reports</text>
+
+        <rect x="595" y="180" width="190" height="48" rx="5" fill="rgba(52,211,153,0.12)" stroke="#34d399" stroke-width="1" stroke-dasharray="3,3"/>
+        <text x="605" y="197" fill="#34d399" font-size="8" font-weight="600">✦ Competitive SEO Scan</text>
+        <text x="605" y="211" fill="#94a3b8" font-size="7">Top 5 competitors SEO health</text>
+        <text x="605" y="222" fill="#94a3b8" font-size="7">side-by-side with yours</text>
+
+        <rect x="795" y="180" width="190" height="48" rx="5" fill="rgba(52,211,153,0.12)" stroke="#34d399" stroke-width="1" stroke-dasharray="3,3"/>
+        <text x="805" y="197" fill="#34d399" font-size="8" font-weight="600">✦ AI Visibility Matrix</text>
+        <text x="805" y="211" fill="#94a3b8" font-size="7">Which competitors get cited</text>
+        <text x="805" y="222" fill="#94a3b8" font-size="7">in ChatGPT/Gemini/Claude?</text>
+
+        <!-- Arrows: users → products -->
+        <line x1="130" y1="147" x2="178" y2="147" stroke="#22d3ee" stroke-width="1.2" marker-end="url(#arrow-cyan)"/>
+        <text x="154" y="141" fill="#64748b" font-size="7" text-anchor="middle">deploy</text>
+        <line x1="130" y1="247" x2="578" y2="195" stroke="#34d399" stroke-width="1.2" marker-end="url(#arrow-emerald)"/>
+        <text x="350" y="226" fill="#64748b" font-size="7">subscribe</text>
+
+        <!-- ============ DATA FLOW BOUNDARY ============ -->
+        <rect x="160" y="280" width="860" height="70" rx="8" fill="transparent" stroke="#fbbf24" stroke-width="1" stroke-dasharray="8,4"/>
+        <text x="172" y="298" fill="#fbbf24" font-size="9" font-weight="600">Cross-Product Data Flow (aggregated, anonymous only)</text>
+
+        <text x="350" y="320" fill="#94a3b8" font-size="8">Client site crawl → Competitive SEO baseline</text>
+        <text x="350" y="335" fill="#94a3b8" font-size="8">Client SEO scores → Industry benchmarks  ·  IntelSight alerts → Audit comparisons</text>
+
+        <!-- Arrows in data flow boundary -->
+        <line x1="355" y1="145" x2="355" y2="290" stroke="#22d3ee" stroke-width="1" stroke-dasharray="3,3"/>
+        <line x1="690" y1="145" x2="690" y2="290" stroke="#34d399" stroke-width="1" stroke-dasharray="3,3"/>
+
+        <!-- ============ SHARED INFRASTRUCTURE ============ -->
+        <rect x="160" y="385" width="860" height="130" rx="8" fill="rgba(251,191,36,0.04)" stroke="#fbbf24" stroke-width="1.5"/>
+        <text x="172" y="403" fill="#fbbf24" font-size="10" font-weight="600">Shared Infrastructure Layer</text>
+
+        <!-- Crawler Engine -->
+        <rect x="180" y="418" width="185" height="55" rx="6" fill="rgba(120,53,15,0.2)" stroke="#fbbf24" stroke-width="1.5"/>
+        <text x="272" y="440" fill="white" font-size="10" font-weight="600" text-anchor="middle">Crawler Engine</text>
+        <text x="272" y="456" fill="#94a3b8" font-size="8" text-anchor="middle">Python/requests · HTTP fetch</text>
+        <text x="272" y="468" fill="#94a3b8" font-size="7" text-anchor="middle">Config depth · Rate limit · Polite</text>
+
+        <!-- Audit Checkers -->
+        <rect x="385" y="418" width="185" height="55" rx="6" fill="rgba(120,53,15,0.2)" stroke="#fbbf24" stroke-width="1.5"/>
+        <text x="477" y="440" fill="white" font-size="10" font-weight="600" text-anchor="middle">Audit Checker Library</text>
+        <text x="477" y="456" fill="#94a3b8" font-size="8" text-anchor="middle">50+ rules · pass/fail/warn</text>
+        <text x="477" y="468" fill="#94a3b8" font-size="7" text-anchor="middle">Impact×Effort · Fix snippets</text>
+
+        <!-- Report Renderer -->
+        <rect x="590" y="418" width="185" height="55" rx="6" fill="rgba(120,53,15,0.2)" stroke="#fbbf24" stroke-width="1.5"/>
+        <text x="682" y="440" fill="white" font-size="10" font-weight="600" text-anchor="middle">Report Renderer</text>
+        <text x="682" y="456" fill="#94a3b8" font-size="8" text-anchor="middle">PDF · HTML · Email</text>
+        <text x="682" y="468" fill="#94a3b8" font-size="7" text-anchor="middle">Branded per product</text>
+
+        <!-- Super Search -->
+        <rect x="795" y="418" width="210" height="55" rx="6" fill="rgba(120,53,15,0.2)" stroke="#fbbf24" stroke-width="1.5"/>
+        <text x="900" y="440" fill="white" font-size="10" font-weight="600" text-anchor="middle">Super Search v2</text>
+        <text x="900" y="456" fill="#94a3b8" font-size="8" text-anchor="middle">7 providers · Cache · Circuit break</text>
+        <text x="900" y="468" fill="#94a3b8" font-size="7" text-anchor="middle">Port :8899 · Production-hardened</text>
+
+        <!-- API Gateway -->
+        <rect x="350" y="488" width="480" height="20" rx="4" fill="rgba(6,78,59,0.3)" stroke="#34d399" stroke-width="1"/>
+        <text x="590" y="502" fill="#34d399" font-size="8" font-weight="600" text-anchor="middle">API Gateway (FastAPI) — single entry point for both products</text>
+
+        <!-- Arrows: infra components → API Gateway -->
+        <line x1="272" y1="473" x2="470" y2="486" stroke="#fbbf24" stroke-width="1"/>
+        <line x1="477" y1="473" x2="530" y2="486" stroke="#fbbf24" stroke-width="1"/>
+        <line x1="682" y1="473" x2="590" y2="486" stroke="#fbbf24" stroke-width="1"/>
+        <line x1="900" y1="473" x2="710" y2="486" stroke="#fbbf24" stroke-width="1"/>
+
+        <!-- ============ DATABASES ============ -->
+        <rect x="250" y="555" width="240" height="55" rx="6" fill="rgba(76,29,149,0.25)" stroke="#a78bfa" stroke-width="1.5"/>
+        <text x="370" y="577" fill="white" font-size="10" font-weight="600" text-anchor="middle">SEO Audit DB</text>
+        <text x="370" y="593" fill="#94a3b8" font-size="8" text-anchor="middle">Site audits · Scores · Trends · Client sites</text>
+        <text x="370" y="605" fill="#a78bfa" font-size="7" text-anchor="middle">ITPP hosting tenant scope</text>
+
+        <rect x="560" y="555" width="240" height="55" rx="6" fill="rgba(76,29,149,0.25)" stroke="#a78bfa" stroke-width="1.5"/>
+        <text x="680" y="577" fill="white" font-size="10" font-weight="600" text-anchor="middle">IntelSight DB</text>
+        <text x="680" y="593" fill="#94a3b8" font-size="8" text-anchor="middle">Competitor profiles · CI data · Alerts</text>
+        <text x="680" y="605" fill="#a78bfa" font-size="7" text-anchor="middle">Multi-tenant SaaS scope</text>
+
+        <!-- Arrows: API Gateway → Databases -->
+        <line x1="410" y1="508" x2="370" y2="553" stroke="#a78bfa" stroke-width="1.2" marker-end="url(#arrow-violet)"/>
+        <line x1="680" y1="508" x2="680" y2="553" stroke="#a78bfa" stroke-width="1.2" marker-end="url(#arrow-violet)"/>
+
+        <!-- ============ EXTERNAL APIs (right side) ============ -->
+        <rect x="920" y="120" width="110" height="40" rx="5" fill="rgba(30,41,59,0.5)" stroke="#94a3b8" stroke-width="1"/>
+        <text x="975" y="138" fill="white" font-size="9" font-weight="600" text-anchor="middle">Crunchbase</text>
+        <text x="975" y="151" fill="#94a3b8" font-size="7" text-anchor="middle">API · $49/mo</text>
+
+        <rect x="920" y="175" width="110" height="40" rx="5" fill="rgba(30,41,59,0.5)" stroke="#94a3b8" stroke-width="1"/>
+        <text x="975" y="193" fill="white" font-size="9" font-weight="600" text-anchor="middle">Hunter.io</text>
+        <text x="975" y="206" fill="#94a3b8" font-size="7" text-anchor="middle">Email intel · $34/mo</text>
+
+        <rect x="920" y="230" width="110" height="40" rx="5" fill="rgba(30,41,59,0.5)" stroke="#94a3b8" stroke-width="1"/>
+        <text x="975" y="248" fill="white" font-size="9" font-weight="600" text-anchor="middle">GSC / GA</text>
+        <text x="975" y="261" fill="#94a3b8" font-size="7" text-anchor="middle">SEO data</text>
+
+        <rect x="920" y="285" width="110" height="40" rx="5" fill="rgba(30,41,59,0.5)" stroke="#94a3b8" stroke-width="1"/>
+        <text x="975" y="303" fill="white" font-size="9" font-weight="600" text-anchor="middle">AI Engines</text>
+        <text x="975" y="316" fill="#94a3b8" font-size="7" text-anchor="middle">ChatGPT · Gemini</text>
+
+        <!-- Arrows: external APIs → products -->
+        <line x1="918" y1="140" x2="1002" y2="140" stroke="#94a3b8" stroke-width="1" stroke-dasharray="3,3"/>
+        <line x1="918" y1="195" x2="1002" y2="195" stroke="#94a3b8" stroke-width="1" stroke-dasharray="3,3"/>
+        <line x1="918" y1="250" x2="530" y2="175" stroke="#94a3b8" stroke-width="1" stroke-dasharray="3,3"/>
+        <line x1="918" y1="305" x2="900" y2="440" stroke="#94a3b8" stroke-width="1" stroke-dasharray="3,3"/>
+
+        <!-- ============ LEGEND ============ -->
+        <text x="30" y="660" fill="white" font-size="9" font-weight="600">Legend</text>
+
+        <rect x="30" y="672" width="14" height="9" rx="2" fill="rgba(8,51,68,0.4)" stroke="#22d3ee" stroke-width="1"/>
+        <text x="50" y="680" fill="#94a3b8" font-size="7">SEO Audit Tool</text>
+
+        <rect x="170" y="672" width="14" height="9" rx="2" fill="rgba(6,78,59,0.4)" stroke="#34d399" stroke-width="1"/>
+        <text x="190" y="680" fill="#94a3b8" font-size="7">IntelSight</text>
+
+        <rect x="290" y="672" width="14" height="9" rx="2" fill="rgba(120,53,15,0.2)" stroke="#fbbf24" stroke-width="1"/>
+        <text x="310" y="680" fill="#94a3b8" font-size="7">Shared Infra</text>
+
+        <rect x="410" y="672" width="14" height="9" rx="2" fill="rgba(76,29,149,0.3)" stroke="#a78bfa" stroke-width="1"/>
+        <text x="430" y="680" fill="#94a3b8" font-size="7">Database</text>
+
+        <rect x="530" y="672" width="14" height="9" rx="2" fill="rgba(30,41,59,0.5)" stroke="#94a3b8" stroke-width="1"/>
+        <text x="550" y="680" fill="#94a3b8" font-size="7">External API</text>
+
+        <line x1="640" y1="677" x2="658" y2="677" stroke="#22d3ee" stroke-width="1" stroke-dasharray="3,3"/>
+        <text x="664" y="680" fill="#94a3b8" font-size="7">New Feature (✦)</text>
+
+        <line x1="790" y1="677" x2="808" y2="677" stroke="#fbbf24" stroke-width="1" stroke-dasharray="8,4"/>
+        <text x="814" y="680" fill="#94a3b8" font-size="7">Data Boundary</text>
+
+      </svg>
+    </div>
+
+    <!-- Info Cards -->
+    <div class="cards">
+      <div class="card">
+        <div class="card-header">
+          <div class="card-dot cyan"></div>
+          <h3>SEO Tool — New Features</h3>
+        </div>
+        <ul>
+          <li>• AI Fix Prompt Generation — Claude/Cursor ready-made prompts per finding</li>
+          <li>• llms.txt Checking — detect missing or malformed AI crawler instructions</li>
+          <li>• GEO/AEO Visibility Scan — how visible is your site in AI search engines?</li>
+          <li>• Impact × Effort Matrix — ranked fixes, not just severity lists</li>
+          <li>• &lt;10s Scan Speed — dopamine hit = conversion</li>
+          <li>• MCP Server (Pro tier) — AI agents can trigger scans directly</li>
+        </ul>
+      </div>
+
+      <div class="card">
+        <div class="card-header">
+          <div class="card-dot emerald"></div>
+          <h3>IntelSight — New Features</h3>
+        </div>
+        <ul>
+          <li>• Competitive SEO Health Scan — top 5 competitors vs your site</li>
+          <li>• Competitor AI Visibility Matrix — who gets cited in AI engines?</li>
+          <li>• Combined CI + SEO Report — funding + team + SEO in one view</li>
+          <li>• llms.txt Competitive Analysis — who's preparing for AI crawlers?</li>
+          <li>• Cross-product benchmarks — industry SEO averages from ITPP data</li>
+        </ul>
+      </div>
+
+      <div class="card">
+        <div class="card-header">
+          <div class="card-dot amber"></div>
+          <h3>Implementation Sequence</h3>
+        </div>
+        <ul>
+          <li>• Phase 1 (Wk 1-2): Crawler Engine + Checker Library + API Gateway</li>
+          <li>• Phase 2 (Wk 3-4): SEO Tool launch — audit, snippets, AI prompts</li>
+          <li>• Phase 3 (Wk 5-8): IntelSight SEO features — competitive scan, AI matrix</li>
+          <li>• Phase 4 (Wk 9-12): Cross-product benchmarks, white-label, MCP server</li>
+        </ul>
+      </div>
+    </div>
+
+    <p class="footer">
+      IT Pro Partner — Product Division · git.itpropartner.com/ippadmin/seo-tool · intelsight.io
+    </p>
+  </div>
+</body>
+</html>

projects/seo-ci-architecture.md

@@ -0,0 +1,250 @@
+# SEO Tool + IntelSight — Shared Architecture
+
+**Status:** OPEN
+**Date:** 2026-08-10
+**Products:** SEO Audit Tool (`git.itpropartner.com/ippadmin/seo-tool`), IntelSight (`intelsight.io`)
+
+---
+
+## 1. Core Principle
+
+Two products. One infrastructure layer. Zero product merge.
+
+| | SEO Audit Tool | IntelSight |
+|---|---|---|
+| **Buyer** | ITPP hosting client (small business owner) | Marketing VP, Founder, Strategy lead |
+| **Question** | "Why isn't my site getting traffic?" | "What are my competitors doing?" |
+| **Pricing** | Free / $49/mo Pro | $199–$1,499/mo |
+| **Current state** | Proposal (repo created Aug 10) | Proposal (July 25, 2026) |
+
+They share the crawl engine, the audit checker library, the report renderer, and the Super Search backend. They sell to different people, at different price points, through different channels.
+
+---
+
+## 2. Shared Infrastructure Layer
+
+```
+┌─────────────────────────────────────────────────────────┐
+│                 SHARED INFRASTRUCTURE                     │
+│                                                          │
+│  ┌──────────┐  ┌───────────┐  ┌──────────┐  ┌────────┐  │
+│  │  Crawler  │  │  Audit    │  │  Report  │  │ Super  │  │
+│  │  Engine   │  │  Checkers │  │ Renderer │  │ Search │  │
+│  │ (Python/ │  │ (50+ rules│  │ (PDF/HTML│  │  (7    │  │
+│  │ requests)│  │  library) │  │ /email)  │  │providers│  │
+│  └────┬─────┘  └─────┬─────┘  └────┬─────┘  └───┬────┘  │
+│       │              │             │            │        │
+│  ┌────┴──────────────┴─────────────┴────────────┴───┐    │
+│  │              API Gateway (FastAPI)                │    │
+│  └──────────────────────────────────────────────────┘    │
+│                                                          │
+│  ┌──────────────────┐  ┌──────────────────┐              │
+│  │  IntelSight DB   │  │   SEO Audit DB   │              │
+│  │  (competitor     │  │  (site audits,   │              │
+│  │   profiles,      │  │   scores, trends,│              │
+│  │   alerts, CI)    │  │   client sites)  │              │
+│  └──────────────────┘  └──────────────────┘              │
+└─────────────────────────────────────────────────────────┘
+```
+
+### Crawler Engine
+
+- HTTP crawler that fetches and parses any URL
+- Extracts: HTML structure, meta tags, headings, links, images, structured data, performance metrics
+- Used by both products — SEO tool crawls the client's own site; IntelSight crawls competitor sites
+- Configurable depth, rate limiting, politeness
+
+### Audit Checker Library
+
+- 50+ reusable check functions: `check_meta_description()`, `check_og_tags()`, `check_canonical()`, `check_structured_data()`, `check_heading_hierarchy()`, `check_image_alts()`, `check_internal_links()`, `check_llms_txt()`
+- Each check returns: `{ status: pass|fail|warn, score: 0-100, fix: "<html snippet>", impact: high|medium|low, effort: low|medium|high }`
+- Used by both products — SEO tool runs all checks on the client's site; IntelSight runs summary checks on competitor sites
+
+### Report Renderer
+
+- Takes audit results and produces: HTML dashboard, PDF report, email summary
+- SEO tool renders: per-page issues, fix-it snippets, trend charts
+- IntelSight renders: competitive SEO comparison, AI visibility matrix, CI + SEO combined report
+
+### Super Search Backend
+
+- Already production-hardened: 7 providers, circuit breakers, caching, port 8899
+- Powers IntelSight's competitive intelligence queries
+- Powers SEO tool's AI engine visibility checks (GEO/AEO)
+
+---
+
+## 3. Product-Specific Architecture
+
+### SEO Audit Tool
+
+```
+┌─────────────────────────────────────┐
+│         SEO AUDIT TOOL               │
+│                                      │
+│  Launch Audit    Monthly Crawl        │
+│  (on deploy)     (cron, 30 days)      │
+│       │              │                │
+│       └──────┬───────┘                │
+│              │                        │
+│     ┌────────┴────────┐               │
+│     │  Per-Site Audit  │               │
+│     │  Dashboard       │               │
+│     │  seo.itpropartner│               │
+│     │  .com/{site}     │               │
+│     └────────┬────────┘               │
+│              │                        │
+│  ┌───────────┴───────────┐            │
+│  │  Fix-It Snippets      │            │
+│  │  (copy-paste HTML     │            │
+│  │   per issue, per page)│            │
+│  └───────────────────────┘            │
+│                                      │
+│  AI Fix Prompts     AI Visibility     │
+│  (Claude/Cursor/    (GEO/AEO check    │
+│   Codex ready-made   via Super Search)│
+│   fix instructions)                   │
+└─────────────────────────────────────┘
+```
+
+**Feature Enhancements (from SeoLoupe analysis):**
+
+| Enhancement | Effort | Value | Description |
+|---|---|---|---|
+| **AI Fix Prompt Generation** | Low | High | Each finding generates a ready-made prompt for Claude/Cursor/Codex — "Fix the meta description on /pricing. Current: '...' Suggested: '...'" |
+| **llms.txt Checking** | Low | Medium | Check if llms.txt exists and is well-formed. Flag missing or malformed. Zero-cost add to crawl phase |
+| **GEO/AEO Visibility Scan** | Medium | High | Query Super Search providers: "Is {domain} cited in AI search results?" Surface how the site appears in ChatGPT, Gemini, Claude, Perplexity |
+| **Impact x Effort Matrix** | Low | Medium | Formalize priority ranking as a 2x2 matrix (not just P0/P1/P2 list). Makes reports more actionable |
+| **<10s Scan Speed Target** | Medium | High | Automate end-to-end pipeline for instant feedback. Dopamine hit = conversion |
+| **MCP Server (Pro tier)** | Medium | High | Expose audit as MCP tools so external AI agents can trigger scans. First-mover in the agent ecosystem |
+
+### IntelSight
+
+```
+┌─────────────────────────────────────┐
+│            INTELSIGHT                │
+│                                      │
+│  Competitor     Market Analysis       │
+│  Profiles       (funding, moves,      │
+│  (Crunchbase,   team changes,         │
+│   Hunter.io,    product launches)     │
+│   OSINT)                              │
+│       │              │                │
+│       └──────┬───────┘                │
+│              │                        │
+│     ┌────────┴────────┐               │
+│     │  CI Dashboard    │               │
+│     │  intelsight.io   │               │
+│     │  /dashboard      │               │
+│     └────────┬────────┘               │
+│              │                        │
+│  ┌───────────┴───────────┐            │
+│  │  Competitive SEO Scan │  ← NEW     │
+│  │  (competitor site     │            │
+│  │   health vs yours)    │            │
+│  └───────────────────────┘            │
+│                                      │
+│  Competitor AI      Alert Engine      │
+│  Visibility         (funding, team,   │
+│  (GEO matrix)       product changes)  │
+└─────────────────────────────────────┘
+```
+
+**Feature Enhancements:**
+
+| Enhancement | Effort | Value | Description |
+|---|---|---|---|
+| **Competitive SEO Health Scan** | Medium | High | Crawl top 5 competitors and surface their SEO health side-by-side with yours. "Their meta tags are complete. Yours are missing on 12 pages." Uses the shared audit checker library |
+| **Competitor AI Visibility Matrix** | Medium | High | How visible are your competitors in ChatGPT, Gemini, Claude, Perplexity, Grok? Which competitors get cited? What queries? The GEO/AEO scan but applied to competitors |
+| **Combined CI + SEO Report** | Medium | Very High | One report per competitor: funding moves, team changes, product launches, AND their SEO health. "They raised $5M, hired a CMO, and their SEO score dropped 15 points — they're distracted, your window is now" |
+| **llms.txt Competitive Analysis** | Low | Low | Flag competitors who have/don't have llms.txt. Trend indicator — who's thinking about AI crawlers? |
+
+---
+
+## 4. Cross-Product Data Flow
+
+```
+SEO Audit Tool                    IntelSight
+──────────────                    ──────────
+Client site crawl ──────────────→ Competitive SEO baseline
+(anonymous, aggregated)
+
+                    Shared Crawler
+                    ←────────────── Competitor site crawl
+
+Client SEO scores   ──────────────→ Industry benchmarks
+(aggregated, anon)                  "Average SMB scores 62/100.
+                                     Your clients average 78/100."
+
+                    ←─────────────── IntelSight alerts
+                                     "Competitor X launched new site.
+                                      Run audit comparison?"
+```
+
+Key rule: **client data stays in the SEO tool. Competitor data stays in IntelSight.** The shared layer only moves aggregated, anonymous benchmarks between them. No client PII crosses the boundary.
+
+---
+
+## 5. Implementation Sequence
+
+### Phase 1: Foundation (Weeks 1-2)
+- Build the shared Crawler Engine (Python/requests, configurable depth, politeness)
+- Build the Audit Checker Library (50+ check functions)
+- Stand up shared API Gateway (FastAPI)
+- Separate databases per product
+
+### Phase 2: SEO Tool Launch (Weeks 3-4)
+- Launch Audit: auto-runs on CloudPanel deploy webhook
+- Fix-It Snippets: HTML output per finding
+- Monthly crawl cron
+- AI Fix Prompt Generation (low effort, high impact)
+- llms.txt checking
+- Impact x Effort matrix in reports
+- Dashboard at `seo.itpropartner.com`
+
+### Phase 3: IntelSight Enhancements (Weeks 5-8)
+- Competitive SEO Health Scan (uses shared audit checker)
+- Competitor AI Visibility Matrix (uses Super Search GEO queries)
+- Combined CI + SEO report template
+- MCP Server for external AI agent access
+
+### Phase 4: Cross-Product (Weeks 9-12)
+- Anonymous industry benchmarks from SEO tool data
+- IntelSight alerts triggering SEO comparisons
+- White-label dashboard for agency clients
+- <10s scan speed optimization
+
+---
+
+## 6. What Stays Separate
+
+| Layer | Separate? | Why |
+|---|---|---|
+| **Databases** | Separate | Client site data vs competitor intelligence — different access patterns, retention policies, privacy considerations |
+| **Auth / Tenants** | Separate | SEO tool: ITPP hosting auth (Stack Auth, existing). IntelSight: standalone SaaS auth with Stripe billing |
+| **Pricing / Billing** | Separate | SEO: bundled with hosting or $49/mo add-on. IntelSight: $199-1,499/mo standalone |
+| **Branding** | Separate | SEO: ITPP-branded, "your hosting includes this." IntelSight: standalone brand, intelsight.io |
+| **Go-to-Market** | Separate | SEO: hosting upsell, retention play. IntelSight: SaaS marketing, Product Hunt, direct sales |
+
+---
+
+## 7. Key Decisions
+
+| Decision | Status | Notes |
+|---|---|---|
+| Merge products into one platform? | **SETTLED: No** | Different buyers, different pricing, different GTM |
+| Share crawl engine? | **SETTLED: Yes** | Same underlying tech, no reason to build twice |
+| Share audit checker library? | **SETTLED: Yes** | Check functions are pure logic — zero product coupling |
+| Cross-product data sharing? | **OPEN** | Aggregated/anonymous only. Privacy boundary TBD |
+| SEO Tool MCP Server? | **OPEN** | Pro tier feature. Proximity to agent ecosystem is a differentiator |
+| IntelSight SEO report depth? | **OPEN** | Full audit (like SEO tool) or summary (top 20 checks)? Depends on crawl politeness and competitor detection risk |
+
+---
+
+## 8. Next Actions
+
+- [ ] Build shared Crawler Engine (`seo-crawler` package)
+- [ ] Define Audit Checker Library spec (50+ checks, output format)
+- [ ] Add AI Fix Prompt Generation to seo-audit skill (immediate low-effort win)
+- [ ] Add llms.txt check to seo-audit skill crawl phase
+- [ ] Prototype GEO/AEO visibility query via Super Search

projects/seo-visibility-checker/01-product-plan.md

@@ -0,0 +1,119 @@
+# SEO + AI Visibility Checker — Product Plan
+
+**Status:** OPEN · Draft v1
+**Date:** 2026-08-23
+**Trigger:** Competitor review of tranx.io "Alice" (SEO + AI visibility check)
+
+---
+
+## 1. Why this exists
+
+Two things changed that make a pure Google-only SEO scanner obsolete:
+
+1. **AI crawlers are a second index.** ChatGPT, Perplexity, Claude, and Gemini answer questions by rendering and retrieving pages — and they each read `/llms.txt`, `robots.txt` (AI user-agents), and structured data differently from Googlebot. A site can rank #1 on Google and be invisible to every AI assistant.
+2. **AI-visibility is not measured by anyone cheap.** Screaming Frog / Ahrefs / Semrush measure Google. `llms.txt` and AI-crawler access are a blind spot. The only tools touching it (Alice, Profound, Peec) are either bare or lock it behind enterprise signup.
+
+We already own the hard parts: Super Search (22 tools, web_extract, AI-answer probing) and a production-tested `seo-audit` skill with a browser_cdp methodology. This tool is productizing that skill + adding the AI-visibility layer nobody else has.
+
+This follows the existing "obstacles as products" pattern: we needed to SEO-optimize our own 15 sites, so we build the checker first.
+
+---
+
+## 2. Product definition
+
+**One-line:** A 30-second scan that answers "can Google *and* ChatGPT/Perplexity/Claude find, render, and cite this site?" — with a 0-100 score, severity-ranked issues, and copy-paste fixes.
+
+**Core promise:** "Know how visible you are to search *and* AI — in 30 seconds, no signup."
+
+### The five check families (parity with Alice, plus our edge)
+
+| Family | Checks |
+|---|---|
+| **On-page SEO** | title, meta description, H1, canonical, Open Graph, structured data (JSON-LD), lang, viewport |
+| **Crawlability** | robots.txt (status + directive parse), sitemap.xml (status + URL count), HTTPS, redirects (www↔non-www, http→https) |
+| **Indexability** | brand SERP probe, indexed-page estimate (site: query), sitemap-vs-index gap |
+| **AI visibility** ← *our moat* | `/llms.txt` presence + parse, per-AI-crawler robots directives (GPTBot, ClaudeBot, PerplexityBot, Google-Extended), server-rendered vs JS-only content, schema markup richness, and a **live "how does ChatGPT/Perplexity see this brand" probe** via Super Search |
+| **Measurement** | GTM, GA4, Google Ads, Meta Pixel, Klaviyo detection + ID health |
+
+### Output
+
+- 0-100 score + one-line narrative
+- Issues ranked healthy / warning / error with per-issue fix snippets
+- Top-3 action list ("moves score most for least work")
+- Optional: connect Search Console / GA4 / Ads for query-level depth (Phase 2, paid)
+
+---
+
+## 3. Differentiation vs Alice (tranx.io)
+
+| Axis | Alice | Ours |
+|---|---|---|
+| AI visibility | `/llms.txt` check only | `/llms.txt` + per-AI-crawler robots + **live AI-answer probe** (does ChatGPT actually cite this brand?) |
+| Depth | On-page + crawl + index + tags | Same + AI-answer retrieval testing + sitemap/index gap analysis |
+| Stack | Closed SaaS, credit-metered-ish upsell | Super Search backend, self-hosted, MCP-native |
+| Pricing | Freemium, enterprise upsell | Freemium scan + flat monthly (no per-lookup) |
+| Output | Score + issues + top-3 | Same + copy-paste fix snippets + export (JSON/PDF) |
+
+**Moat:** the "does an AI actually cite you" probe. That's the question every founder will have in 2026 and nobody answers it in a free tool. It requires an LLM + search backend, which we already run.
+
+---
+
+## 4. Architecture
+
+```
+[Web] seo-check.<tld>  (single-file SPA, static)
+   │  POST /scan {url}
+   ▼
+[FastAPI backend]  :8088  (deploy like other python-web-service-deployment)
+   ├─ fetch + parse target (requests + BeautifulSoup + lxml)
+   ├─ robots/sitemap/redirect checks (httpx)
+   ├─ structured-data validation (JSON-LD parse)
+   ├─ llms.txt fetch + parse
+   ├─ per-AI-crawler robots analysis
+   ├─ JS-render probe (optional headless via existing Crawl4AI / browserless)
+   └─ AI-answer probe → Super Search MCP (web_search + web_extract, brand+category query)
+   ▼
+[Super Search MCP]  (already live, 22 tools)
+   └─ "what does {brand} do" → check if brand appears in top AI/search results
+```
+
+**Reuse:** Super Search (search + extract), Crawl4AI :8910 (JS render when needed), existing `seo-audit` skill logic as the scoring spec. No new infra — this is a thin FastAPI service on Core behind Caddy.
+
+---
+
+## 5. Build path (phased)
+
+| Phase | Scope | Exit criterion |
+|---|---|---|
+| **P1 — MVP (free scan)** | On-page + crawl + index + AI-visibility (llms.txt + crawler robots + AI-answer probe) + measurement. Score + top-3 + snippets. | Live scan of our own 15 sites, all green |
+| **P2 — Auth + depth** | Central auth login, saved scans, GSC/GA4/Ads connect, query-level data | Paying first user |
+| **P3 — API + MCP** | REST API + MCP server (`seo-check` tool) so Hermes and other agents can run scans programmatically | MCP server live, documented |
+
+**P1 is the same work as "SEO-optimize our 15 sites"** — build the checker, run it on ourselves, fix what it finds. The tool and the site-fix are one effort.
+
+---
+
+## 6. Pricing (value-based, premium — never undercut)
+
+| Tier | Price | Includes |
+|---|---|---|
+| Free | $0 | 1-off scans, rate-limited per IP, no signup |
+| Pro | $29/mo | Unlimited scans, saved reports, scheduled re-scans, AI-answer monitoring alerts |
+| Agency | $99/mo | 50 domains, white-label PDF reports, API access |
+| API | usage | Per-scan API + MCP server for agent platforms |
+
+Anchored against Alice (free) and Semrush/Ahrefs ($129+/mo). We undercut enterprise but never race to the bottom on the free tier — the free tier is a lead-gen funnel, not the product.
+
+---
+
+## 7. Open decisions (need Germaine)
+
+- [ ] **Brand/domain:** own product name + domain, or a subdomain under an existing property? (Suggests: standalone `.io`/`.com` like the other micro-SaaS products — IntelSight, VerdictTank.)
+- [ ] **Scope of "AI-answer probe":** how deep — full retrieval comparison or a yes/no "brand cited" flag for v1?
+- [ ] **Ship target:** P1 as a free public tool first (lead-gen), or internal-only until our 15 sites are clean?
+
+---
+
+## 8. Immediate next step
+
+The P1 build and the "optimize our 15 sites" ask are the same workstream. Sweep results are in `sites-seo-sweep-2026-08-23.md`. Fix the 9 failing sites first (proves the scoring model), then ship the checker as a product.

projects/timapta.md

@@ -1,8 +1,33 @@
 # TIMAPTA — Tybee Island Maritime Academy PTA
 
 **Project owner:** Greyson's mom (PTA project)
-**Status:** LIVE (initial deployment)
-**Deployed:** 2026-08-12
+**Status:** FORWARDING TO ptatime.org (ptatime.org itself NOT yet deployed)
+**Deployed:** 2026-08-12 · **Re-pointed:** 2026-09-11
+
+## Canonical-domain change (2026-09-11)
+
+All three PTA domains now 301 to `ptatime.org`, which is the intended canonical
+domain. `ptatime.org` has an active zone but ZERO DNS records — no web, no mail —
+so every forward currently lands on a domain that does not resolve. Nothing serves
+until a site is deployed at `ptatime.org`. Only `register.timapta.org` (membership
+form on Core) is still live.
+
+| Source (apex + www) | Mechanism | Target |
+|---|---|---|
+| `timapta.org` | proxied dummy A (192.0.2.1) + Page Rule | 301 → `https://ptatime.org/$1` |
+| `ptatima.org` | proxied dummy A (192.0.2.1) + Page Rule | 301 → `https://ptatime.org/$1` |
+| `ptaoftybeeislandmaritimeeducation.org` | proxied dummy A (192.0.2.1) + Page Rule | 301 → `https://ptatime.org/$1` |
+
+Verified at the Cloudflare edge (all 6 hosts, path + query preserved):
+`GET /join/thanks?x=1` → `301 Location: https://ptatime.org/join/thanks?x=1`
+
+**timapta.org email is untouched by this change** (MX, SPF, DKIM, DMARC all intact
+and resolving; only HTTP records/Page Rules are involved).
+
+**timapta.org apex no longer serves the old site.** The apex + www A records that
+pointed at app3 were gone before this change (site returned nothing). Site files
+still exist at `/home/ippadmin/htdocs/timapta.org/` and the nginx vhost
+`timapta.org.conf` is still enabled on app3 — orphaned until removed or re-homed.
 
 ## Summary
 
@@ -15,16 +40,26 @@ MXroute with branded `mail.` and `webmail.` subdomains.
 
 | Record | Type | Value | Purpose |
 |---|---|---|---|
-| `timapta.org` | A | 152.53.241.111 | app3 site (grey-cloud) |
-| `www.timapta.org` | A | 152.53.241.111 | app3 site |
-| `ptatima.org` | A (proxied) + Page Rule | 301 → `https://timapta.org/$1` | forward |
-| `www.ptatima.org` | A (proxied) + Page Rule | 301 → `https://timapta.org/$1` | forward |
+| `timapta.org` | A (proxied, dummy 192.0.2.1) + Page Rule | 301 → `https://ptatime.org/$1` | forward (added 2026-09-11) |
+| `www.timapta.org` | A (proxied, dummy 192.0.2.1) + Page Rule | 301 → `https://ptatime.org/$1` | forward (added 2026-09-11) |
+| `ptatima.org` | A (proxied) + Page Rule | 301 → `https://ptatime.org/$1` | forward (retargeted 2026-09-11) |
+| `www.ptatima.org` | A (proxied) + Page Rule | 301 → `https://ptatime.org/$1` | forward (retargeted 2026-09-11) |
+| `ptaoftybeeislandmaritimeeducation.org` | A (proxied, dummy 192.0.2.1) + Page Rule | 301 → `https://ptatime.org/$1` | forward (added 2026-09-11) |
+| `www.ptaoftybeeislandmaritimeeducation.org` | A (proxied, dummy 192.0.2.1) + Page Rule | 301 → `https://ptatime.org/$1` | forward (added 2026-09-11) |
+| `ptatime.org` | (none) | — | zone exists, NO records (no web, no mail) as of 2026-09-11 |
 | `mail.timapta.org` | CNAME | heracles.mxrouting.net | IMAP/SMTP hostname |
 | `webmail.timapta.org` | A | 152.53.192.33 (Core) | Caddy 302 → Roundcube |
 
 Cloudflare zone IDs:
 - timapta.org: `92d1512d07b72142551aa5306fbacb2a`
 - ptatima.org: `fe443f33606d319a35eeb473403371fe`
+- ptatime.org: `7eaa5a94be7cd74e82852f1f287a9235`
+- ptaoftybeeislandmaritimeeducation.org: `9566a9ecf7bfe5a0fad6fb808ddac616`
+
+Page rule IDs (all `forwarding_url` → `https://ptatime.org/$1`, 301):
+- timapta.org: `bce7af229a27455e7e6b0c1d2e383281` (prio 1) · www: `e36031c567a32349e19cc567e2221d5f` (prio 2)
+- ptatima.org: `a0ce055a36f1d81a124654bf71f0a156` (prio 1) · www: `149fbab978c2a869122a284c827b9e99` (prio 2)
+- ptaoftybeeislandmaritimeeducation.org: `a11f40b8da891cdde37d052e6efef981` (prio 1) · www: `d5083187d4f868f1989e50187834d146` (prio 2)
 
 ## Email (MXroute)
 
@@ -76,6 +111,10 @@ real names, meeting schedule, and a phone/address.
 
 ## Open Items / Next Steps
 
+- [ ] **Deploy a site at `ptatime.org`** (blocking). Decision 2026-09-11: leave the zone
+      dark for now, a new site is being built and will be deployed there. The existing
+      forwards activate automatically the moment it serves; no further DNS work needed.
+- [ ] Decide fate of the orphaned app3 site (`/home/ippadmin/htdocs/timapta.org/` + `timapta.org.conf` vhost)
 - [ ] Move `contact@timapta.org` password into Vaultwarden
 - [ ] Activate FormSubmit endpoint (first submission + click confirmation)
 - [ ] Add real phone number + mailing address once PTA provides them
@@ -87,6 +126,19 @@ real names, meeting schedule, and a phone/address.
 - app3 uses nginx (CloudPanel layout, per-user `/home/<user>/htdocs` docroots), NOT
   Caddy. Caddy only runs on Core (152.53.192.33), which is why the webmail redirect
   lives there.
+- `timapat.org` (as requested 2026-09-11) is NOT a registered domain — PIR whois
+  returns "Domain not found". The real domain is `timapta.org` (letters transposed).
+- Forwarding toward `ptatime.org` (2026-09-11): `ptatime.org` zone `7eaa5a94be7cd74e82852f1f287a9235`
+  exists but has ZERO DNS records — no web, no mail. Redirects land on a dead domain
+  until something is deployed there.
+- Page Rule pattern `timapta.org/*` does NOT match subdomains — only the exact host.
+  Verified: `register.timapta.org` (200) and `webmail.timapta.org` (302 → MXroute
+  Roundcube) are unaffected by the new apex/www forwarding rules.
+- Adding a proxied dummy A record at the apex does NOT affect mail: MX, SPF, DKIM and
+  DMARC records are independent of the HTTP path. Confirmed resolving after the change.
+- Freshly recreated names take ~60s to appear on Cloudflare's authoritative NS. A
+  `dig @<ns>.ns.cloudflare.com` NXDOMAIN immediately after record creation is
+  propagation lag, not a failed create — re-check the API record and retry.
 - `ptatima.org` forward required a proxied dummy A record so the Cloudflare Page Rule
   can intercept before routing.
 - MXroute domain-add requires a `_da-verify-*` TXT record; the add fails until that

projects/verify_hotnow.py

@@ -0,0 +1,22 @@
+#!/usr/bin/env python3
+"""Verify no double hyphens outside style tags."""
+import re
+
+with open('/root/projects/itpp-infrastructure/projects/hotnow-savannah-v2.html') as f:
+    html = f.read()
+
+# Split by style tags
+parts = re.split(r'(<style>.*?</style>)', html, flags=re.DOTALL)
+count = 0
+for part in parts:
+    if not part.startswith('<style>'):
+        found = list(re.finditer(r'--', part))
+        for m in found:
+            ctx = part[max(0,m.start()-20):m.end()+20]
+            count += 1
+            print(f'  offset {m.start()}: ...{ctx!r}...')
+
+if count == 0:
+    print('ZERO double hyphens outside <style> tags. Clean!')
+else:
+    print(f'Found {count} double hyphens outside style tags')

projects/verify_visible.py

@@ -0,0 +1,27 @@
+#!/usr/bin/env python3
+"""Verify no double hyphens in visible text content."""
+import re
+
+with open('/root/projects/itpp-infrastructure/projects/hotnow-savannah-v2.html') as f:
+    html = f.read()
+
+# Remove all HTML tags, comments, and style blocks
+clean = html
+# Remove style blocks
+clean = re.sub(r'<style>.*?</style>', '', clean, flags=re.DOTALL)
+# Remove HTML comments
+clean = re.sub(r'<!--.*?-->', '', clean, flags=re.DOTALL)
+# Remove HTML tags
+clean = re.sub(r'<[^>]+>', '', clean)
+# Decode HTML entities that contain --
+clean = clean.replace('&mdash;', '')
+
+# Find --
+matches = list(re.finditer(r'--', clean))
+if matches:
+    print(f'Found {len(matches)} double hyphens in visible text:')
+    for m in matches[:20]:
+        ctx = clean[max(0,m.start()-30):m.end()+30]
+        print(f'  offset {m.start()}: "...{ctx.strip()}..."')
+else:
+    print('ZERO double hyphens in visible text content. Clean!')

proposals/liberty-tire-brake/proposal-v4.html

@@ -0,0 +1,416 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+<meta charset="UTF-8">
+<title>Liberty Tire &amp; Brake — Camera Proposal v4</title>
+<link rel="preconnect" href="https://fonts.googleapis.com">
+<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
+<link href="https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700;800&display=swap" rel="stylesheet">
+<style>
+  :root {
+    --teal: #0d9488;
+    --navy: #0f172a;
+    --text: #1a1a2e;
+    --muted: #64748b;
+    --border: #e5e7eb;
+  }
+  @page {
+    size: Letter;
+    margin: 2.2cm 1.8cm 2.0cm 1.8cm;
+    @top-center {
+      content: "IT Pro Partner";
+      font-family: 'Inter', sans-serif;
+      font-size: 8.5pt;
+      letter-spacing: 0.08em;
+      text-transform: uppercase;
+      color: #0d9488;
+      margin-bottom: 0.6cm;
+    }
+    @bottom-right {
+      content: counter(page);
+      font-family: 'Inter', sans-serif;
+      font-size: 9pt;
+      color: #64748b;
+    }
+    @bottom-left {
+      content: "Liberty Tire & Brake — Camera Proposal";
+      font-family: 'Inter', sans-serif;
+      font-size: 8pt;
+      color: #94a3b8;
+    }
+  }
+  @page cover {
+    margin: 0;
+    @top-center { content: none; }
+    @bottom-right { content: none; }
+    @bottom-left { content: none; }
+  }
+  * { box-sizing: border-box; }
+  body { font-family: 'Inter', -apple-system, BlinkMacSystemFont, sans-serif; color: var(--text); font-size: 10.5pt; line-height: 1.6; margin: 0; }
+  h1, h2, h3 { color: var(--navy); }
+  h2 { font-size: 15pt; font-weight: 700; margin: 0 0 0.5rem 0; padding-bottom: 0.35rem; border-bottom: 2px solid var(--teal); }
+  h3 { font-size: 11.5pt; font-weight: 600; margin: 1rem 0 0.4rem 0; }
+  p { margin: 0 0 0.6rem 0; }
+  ul { margin: 0 0 0.6rem 0; padding-left: 1.2rem; }
+  li { margin-bottom: 0.3rem; }
+  .page-break { page-break-before: always; }
+  section { margin-bottom: 1.6rem; }
+  table { width: 100%; border-collapse: collapse; margin: 0.6rem 0 1rem 0; font-size: 9.5pt; }
+  th, td { border: 1px solid var(--border); padding: 0.45rem 0.55rem; text-align: left; vertical-align: top; }
+  th { background: var(--navy); color: #fff; font-weight: 600; }
+  td.num, th.num { text-align: right; white-space: nowrap; }
+  tr.total td { font-weight: 700; background: #f1f5f9; }
+  tr.grandtotal td { font-weight: 800; background: #ccfbf1; }
+  .note { background: #f8fafc; border-left: 3px solid var(--teal); padding: 0.6rem 0.8rem; font-size: 9pt; color: var(--muted); margin: 0.6rem 0; }
+  .flag { background: #fffbeb; border-left: 3px solid #f59e0b; padding: 0.6rem 0.8rem; font-size: 9pt; color: #92400e; margin: 0.6rem 0; }
+  .muted { color: var(--muted); }
+  .small { font-size: 8.5pt; color: var(--muted); }
+
+  /* Cover */
+  .cover { page: cover; width: 100%; height: 100vh; display: flex; flex-direction: column; justify-content: center; align-items: center; text-align: center; background: var(--navy); color: #fff; }
+  .cover .brand { font-size: 20pt; font-weight: 800; letter-spacing: 0.02em; color: #fff; }
+  .cover .brand .accent { color: #14b8a6; }
+  .cover .contact { margin-top: 0.4rem; font-size: 10pt; color: #94a3b8; }
+  .cover .rule { width: 60px; height: 3px; background: var(--teal); margin: 2rem auto; }
+  .cover .title { font-size: 34pt; font-weight: 800; color: #fff; }
+  .cover .prepared { margin-top: 1.2rem; font-size: 11pt; color: #cbd5e1; }
+</style>
+</head>
+<body>
+
+<!-- ============ COVER ============ -->
+<div class="cover">
+  <div class="brand">IT PRO <span class="accent">PARTNER</span></div>
+  <div class="contact">Savannah, GA 31405 &nbsp;|&nbsp; 912.323.3798 &nbsp;|&nbsp; www.itpropartner.com</div>
+  <div class="rule"></div>
+  <div class="title">Camera Proposal</div>
+  <div class="prepared">Prepared by<br>Germaine Brown</div>
+</div>
+
+<!-- ============ PAGE 2: WHY IT PRO PARTNER ============ -->
+<section class="page-break">
+  <h2>Why IT Pro Partner</h2>
+  <p>At IT Pro Partner, we believe technology should enable your business, not interrupt it. This proposal outlines how we will help you improve reliability, strengthen security, and support growth with responsive service and practical, measurable outcomes.</p>
+  <h3>Local Expertise, Personal Service</h3>
+  <p>We are based in Savannah, GA, and provide responsive support from a team that understands your environment, including onsite help when needed.</p>
+  <h3>Proven Solutions, Reliable Hardware</h3>
+  <p>We standardize on enterprise-grade platforms and manage the full lifecycle, including procurement guidance, deployment, patching, warranty coordination, and refresh planning to reduce downtime.</p>
+  <h3>Strategic IT With Long-Term Vision</h3>
+  <p>We go beyond break/fix with proactive monitoring, automation, and security best practices to reduce risk and keep systems stable as your business evolves.</p>
+  <h3>What this proposal covers</h3>
+  <p>Clear scope and deliverables; defined responsibilities; support expectations and service targets; implementation timeline and key milestones; and identified risks, dependencies, and mitigations so you can make an informed decision.</p>
+  <h3>Outcomes we prioritize</h3>
+  <p>Improved uptime and stability, faster resolution of issues, stronger security controls, more reliable backups and recovery readiness, and clear reporting on the health of your IT environment.</p>
+  <p><strong>Let's build something better together.</strong></p>
+</section>
+
+<!-- ============ PAGE 3: CONFIDENTIALITY ============ -->
+<section class="page-break">
+  <h2>Quote Confidentiality Disclosure</h2>
+  <p>The information contained in this quote is confidential and proprietary to IT Pro Partner. It is provided exclusively for the recipient's review and consideration. By accepting this quote, the recipient agrees that:</p>
+  <ol>
+    <li><strong>Confidentiality:</strong> The pricing, terms, and specifications outlined in this document are strictly confidential and shall not be shared, distributed, or disclosed to any third party without prior written consent from IT Pro Partner.</li>
+    <li><strong>Non-Disclosure:</strong> Any unauthorized sharing of this quote may result in legal action and potential liability for damages.</li>
+    <li><strong>Validity:</strong> This quote is valid until 02/05/2026 and is subject to change without notice before acceptance.</li>
+    <li><strong>No Obligation:</strong> Receipt of this quote does not constitute an agreement to provide services until a formal contract is signed.</li>
+    <li><strong>Mediation First:</strong> The parties agree to first attempt to resolve any dispute, claim, or controversy arising out of or relating to this quote through good-faith mediation as a condition precedent to filing any lawsuit or initiating other legal proceedings.</li>
+  </ol>
+  <p>For any questions regarding this quote, please contact IT Pro Partner at info@itpropartner.com. IT Pro Partner reserves all rights to the information contained in this document.</p>
+</section>
+
+<!-- ============ PAGE 4: ABOUT THE PROJECT ============ -->
+<section class="page-break">
+  <h2>About the Project</h2>
+  <p>IT Pro Partner is providing this quote to upgrade Liberty Tire &amp; Brake's current video surveillance environment from an existing Guarding Vision DVR system (currently supporting seven cameras covering the lobby and front/rear parking areas) to a modern 16-channel PoE NVR (Network Video Recorder with Power over Ethernet) solution. The current DVR is viewed locally on an approximately 50" TV.</p>
+
+  <h3>Objectives</h3>
+  <ul>
+    <li>Expand coverage to support up to 16 total cameras, improving visibility across the shop interior and surrounding exterior areas.</li>
+    <li>Improve usability and response time with faster video lookup and more efficient search capabilities using the NVR's AI-assisted features (where supported by the camera/NVR platform).</li>
+    <li>Enable remote viewing for authorized team members via mobile phone and desktop access (subject to network and security requirements).</li>
+    <li>Provide an optional at-a-glance monitoring display: a centrally located TV configured as a 24/7 spot monitor, ideal for managers or staff in high-traffic areas. The optional package includes a TV, wall mount, and an HDMI KVM extender to allow local mouse control at the viewing station.</li>
+  </ul>
+
+  <h3>Proposed Solution (High Level)</h3>
+  <ul>
+    <li><strong>Core platform:</strong> Reolink RP-PN16 24-channel NVR configured for 16 wired PoE camera channels with included 4TB storage (approximately 3 days of continuous recording) and optional expansion capacity.</li>
+    <li><strong>Camera deployment:</strong> a mix of 4K dome and 4K bullet cameras to align to different mounting and coverage needs.</li>
+    <li><strong>Optional enhancements:</strong>
+      <ul>
+        <li>Spot monitor TV package for continuous display</li>
+        <li>Extended recording storage: add 1x 16TB drive (about 16 days) or max out with 2x 16TB drives (about 26 days)</li>
+        <li>Expansion to 24 cameras via a remote rack/PoE switch option (to support future growth beyond the initial 16-camera deployment)</li>
+      </ul>
+    </li>
+  </ul>
+
+  <h3>Recording Retention Reference <span class="muted" style="font-weight:400">(16 cameras, 4K continuous recording)</span></h3>
+  <table>
+    <thead><tr><th>Storage Configuration</th><th>Total Capacity</th><th class="num">Approx. Retention</th></tr></thead>
+    <tbody>
+      <tr><td>Factory 4TB drive only</td><td>4 TB</td><td class="num">~3 days</td></tr>
+      <tr><td>Add 1x 16TB drive (keeps factory 4TB)</td><td>20 TB</td><td class="num">~16 days</td></tr>
+      <tr><td>Max out with 2x 16TB drives (replaces factory 4TB)</td><td>32 TB</td><td class="num">~26 days</td></tr>
+    </tbody>
+  </table>
+  <div class="note">The RP-PN16 has two hard-drive bays. It ships with a single 4TB drive in one bay, leaving one bay open for expansion. A 4TB drive holds roughly 3 days of continuous 4K recording across 16 cameras, so we recommend adding at least one 16TB surveillance drive for a usable retention window (see Optional Add-Ons).</div>
+
+  <h3>Dependencies and Considerations (Transparency)</h3>
+  <ul>
+    <li><strong>Network/security readiness:</strong> Remote viewing requires appropriate firewall/router configuration and user access controls.</li>
+    <li><strong>Field verification:</strong> Final camera placement and viewing angles will be confirmed onsite prior to final mounting to ensure the coverage goals are met.</li>
+    <li><strong>Operational access:</strong> Installation work will be coordinated to minimize disruption to shop operations.</li>
+  </ul>
+
+  <h3>Decision Points (Options)</h3>
+  <ol>
+    <li><strong>Base deployment (16 cameras):</strong> Full coverage upgrade to the new NVR platform.</li>
+    <li><strong>Add spot monitor TV:</strong> Improves day-to-day situational awareness for staff without requiring a phone or PC login.</li>
+    <li><strong>Add extended storage:</strong> Increases retention time for recorded video (useful for incident review and claims).</li>
+    <li><strong>Plan for growth to 24 cameras:</strong> Adds infrastructure for future camera expansion (remote rack and PoE switching).</li>
+  </ol>
+
+  <h3>Recommended Next Steps</h3>
+  <ul>
+    <li><strong>Liberty Tire &amp; Brake (Owner):</strong> Confirm which options to include (TV spot monitor, extended storage, future expansion).</li>
+    <li><strong>IT Pro Partner:</strong> Schedule a brief onsite walkthrough to confirm camera locations, NVR placement, and the preferred TV mounting location prior to final installation.</li>
+  </ul>
+  <p class="small">Assumptions: Existing coverage includes lobby and front/rear parking as stated; existing display is ~50"; remote viewing is desired for authorized users; and the optional TV is intended for continuous "spot monitor" viewing.</p>
+</section>
+
+<!-- ============ PAGE 5/6: CAMERA SYSTEM QUOTE ============ -->
+<section class="page-break">
+  <h2>Camera System Quote</h2>
+  <table>
+    <thead>
+      <tr><th>Item</th><th>Description</th><th class="num">Unit Price</th><th class="num">Qty</th><th class="num">Extended Price</th></tr>
+    </thead>
+    <tbody>
+      <tr>
+        <td>NVR</td>
+        <td>Reolink RP-PN16 NVR &bull; 24-channel platform &bull; 16 wired PoE channels &bull; 4TB included (~3 days retention) &bull; AI video search</td>
+        <td class="num">$539.99</td><td class="num">1</td><td class="num">$539.99</td>
+      </tr>
+      <tr>
+        <td>Dome Camera</td>
+        <td>Reolink RP-PCV8MZ Dome Camera &bull; 4K resolution &bull; 5X optical zoom &bull; color + IR night vision</td>
+        <td class="num">$209.99</td><td class="num">3</td><td class="num">$629.97</td>
+      </tr>
+      <tr>
+        <td>Bullet Camera</td>
+        <td>Reolink RP-PCB8MZ Bullet Camera &bull; 4K resolution &bull; 5X optical zoom &bull; 133&deg; field of view</td>
+        <td class="num">$229.99</td><td class="num">13</td><td class="num">$2,989.87</td>
+      </tr>
+      <tr>
+        <td>Junction Box</td>
+        <td>B10 Junction Box (for bullet camera installs)</td>
+        <td class="num">$27.99</td><td class="num">13</td><td class="num">$363.87</td>
+      </tr>
+      <tr>
+        <td>Patch Panel</td>
+        <td>24-Port Patch Panel</td>
+        <td class="num">$30.00</td><td class="num">2</td><td class="num">$60.00</td>
+      </tr>
+      <tr>
+        <td>Rack Shelf</td>
+        <td>1U Shelf</td>
+        <td class="num">$40.00</td><td class="num">1</td><td class="num">$40.00</td>
+      </tr>
+      <tr>
+        <td>Keystone Jacks</td>
+        <td>Keystone jacks (lot)</td>
+        <td class="num">$50.00</td><td class="num">1</td><td class="num">$50.00</td>
+      </tr>
+      <tr>
+        <td>Turnkey Installation</td>
+        <td>Labor and install materials &bull; Install (16) cameras including data drops ($250/camera) &bull; Provide and install necessary patch cables and conduit &bull; Terminate/label at patch panel &bull; Configure NVR/cameras and recording &bull; Remote viewing setup (mobile + desktop) &bull; End-user training</td>
+        <td class="num">$4,000.00</td><td class="num">1</td><td class="num">$4,000.00</td>
+      </tr>
+      <tr class="total"><td colspan="4">Subtotal</td><td class="num">$8,673.70</td></tr>
+      <tr><td colspan="4">Sales Tax (7% of hardware)</td><td class="num">$327.16</td></tr>
+      <tr class="grandtotal"><td colspan="4">Total</td><td class="num">$9,000.86</td></tr>
+    </tbody>
+  </table>
+  <div class="note">Camera and NVR pricing reflects Reolink Professional Series advertised list price (verified August 2026). The NVR price includes the factory-installed 4TB drive.</div>
+</section>
+
+<!-- ============ PAGE 7: OPTIONAL ADD-ONS ============ -->
+<section class="page-break">
+  <h2>Optional Add-Ons</h2>
+  <table>
+    <thead>
+      <tr><th>Item</th><th>Description</th><th class="num">Unit Price</th><th class="num">Qty</th><th class="num">Extended Price</th></tr>
+    </thead>
+    <tbody>
+      <tr>
+        <td>Back Office TV Station</td>
+        <td>65" TV Station (spot monitor) &bull; 65" TV &bull; Mount &bull; HDMI KVM USB extender (enables mouse control at TV) &bull; Wireless mouse &bull; Install/configuration included (2 hrs @ $195/hr)</td>
+        <td class="num">$1,236.00</td><td class="num">1</td><td class="num">$1,236.00</td>
+      </tr>
+      <tr>
+        <td>Extended Recording &mdash; Add 1x 16TB HDD</td>
+        <td>Adds a Seagate SkyHawk AI 16TB surveillance drive (ST16000VE004) to the open bay (keeps factory 4TB) &bull; 20TB total &bull; ~16 days retention &bull; Install/configuration included (0.5 hr @ $195/hr)</td>
+        <td class="num">$677.49</td><td class="num">1</td><td class="num">$677.49</td>
+      </tr>
+      <tr>
+        <td>Extended Recording &mdash; Max (2x 16TB HDD)</td>
+        <td>Maxes both bays with 2x Seagate SkyHawk AI 16TB surveillance drives (ST16000VE004, replaces factory 4TB) &bull; 32TB total &bull; ~26 days retention &bull; Install/configuration included (0.5 hr @ $195/hr)</td>
+        <td class="num">$1,257.48</td><td class="num">1</td><td class="num">$1,257.48</td>
+      </tr>
+      <tr>
+        <td>Expansion Package (17&ndash;24)</td>
+        <td>Remote rack + 8 additional cameras &bull; Unifi 24 PoE switch &bull; UPS &bull; 6U rack enclosure &bull; 150ft LC-LC fiber cable &bull; 8x RP-PCB8MZ bullet cameras &bull; Camera installs (8 x $250) &bull; Remote rack/integration labor (3 hrs @ $195/hr)</td>
+        <td class="num">$4,983.85</td><td class="num">1</td><td class="num">$4,983.85</td>
+      </tr>
+    </tbody>
+  </table>
+
+  <div class="note">
+    <strong>Storage add-on pricing note:</strong> The 16TB drive is the Seagate SkyHawk AI (ST16000VE004), priced at $579.99 each (current street price, verified August 2026; subject to confirmation at order time). 7% sales tax applies to the hardware portion. The RP-PN16 has two drive bays; choosing "Max" replaces the factory 4TB with a second 16TB drive to reach the NVR's 32TB ceiling.
+  </div>
+
+  <p class="small">Optional Add-On Sales Tax (7% of hardware only):</p>
+  <ul class="small">
+    <li>Back Office TV Station tax (7% of $846.00 hardware): $59.22</li>
+    <li>Extended Recording &mdash; 1x 16TB tax (7% of $579.99 hardware): $40.60</li>
+    <li>Extended Recording &mdash; 2x 16TB tax (7% of $1,159.98 hardware): $81.20</li>
+    <li>Expansion Package tax (7% of $2,241.92 hardware): $156.93</li>
+  </ul>
+  <p class="small">Each add-on "Unit Price" is presented as hardware + labor; hardware sales tax is itemized above.</p>
+</section>
+
+<!-- ============ PAGE 8: CAMERA VIEWS ============ -->
+<section class="page-break">
+  <h2>Camera Views</h2>
+  <img src="floorplan-p8.png" alt="Liberty Tire &amp; Brake camera views floor plan" style="width:100%; border:1px solid #e5e7eb;">
+</section>
+
+<!-- ============ PAGE 9: PROJECT SUMMARY ============ -->
+<section class="page-break">
+  <h2>Project Summary</h2>
+  <h3>NVR System Installation (Base Scope)</h3>
+  <ul>
+    <li>Install and commission a PoE NVR (24-channel / 16 wired PoE) with 4TB included storage (about 3 days of continuous 4K recording at 16 cameras), expandable up to 32TB, and AI video search capability.</li>
+    <li>Install 16 total cameras:
+      <ul>
+        <li>3x 4K vandal dome cameras (optical zoom; optimized for mixed lighting and night viewing)</li>
+        <li>13x 4K bullet cameras (wide field of view; optical zoom)</li>
+      </ul>
+    </li>
+    <li>Install junction boxes for 13 bullet cameras to support secure mounting and clean terminations.</li>
+    <li>Terminate camera cabling at the headend using:
+      <ul>
+        <li>2x 24-port patch panels</li>
+        <li>CAT6 keystone jacks</li>
+        <li>1U shelf for headend organization and support</li>
+      </ul>
+    </li>
+    <li>Coordinate with the Liberty team during installation to confirm final camera placement, aiming, and coverage based on field conditions and the site plan.</li>
+  </ul>
+
+  <h3>Areas to Cover (Intended Coverage)</h3>
+  <p class="small">Final placement will be validated onsite; the intended coverage includes:</p>
+  <ul>
+    <li>Front parking lot and building frontage</li>
+    <li>Lobby entrance (exterior) and lobby interior</li>
+    <li>Main shop work areas and bays</li>
+    <li>Rear shop / rear parking lot</li>
+    <li>Side access/street-facing areas</li>
+  </ul>
+
+  <h3>Access &amp; Viewing</h3>
+  <ul>
+    <li>Mobile access available for authorized users via iOS/Android (app-based remote viewing).</li>
+    <li>Desktop access for authorized stakeholders (local or remote viewing, subject to network/security policy).</li>
+    <li>Optional spot monitor viewing: add a dedicated TV for "at-a-glance" live monitoring in a high-traffic or management area.</li>
+  </ul>
+</section>
+
+<!-- ============ PAGE 10: OPTIONAL ENHANCEMENTS + RISKS ============ -->
+<section class="page-break">
+  <h2>Optional Enhancements (Decision-Ready)</h2>
+  <ol>
+    <li><strong>TV Monitoring Package (Spot Monitor + Control)</strong> &mdash; Add a 65" TV, wall mount, and HDMI KVM extender to enable live viewing and local mouse control at the display location.</li>
+    <li><strong>Extended Recording Capacity</strong> &mdash; Add storage to extend video retention. Option A: add one 16TB surveillance drive (20TB total, about 16 days). Option B: max out with two 16TB drives (32TB total, about 26 days). Retention varies with recording settings and camera count.</li>
+    <li><strong>Expansion to 24 Cameras (Future Growth Option)</strong> &mdash; Add a PoE switch, UPS, 6U rack enclosure, and fiber uplink cable, plus 8 additional 4K bullet cameras, enabling growth from 16 to 24 cameras where additional coverage is needed.</li>
+  </ol>
+
+  <h2>Risks, Dependencies, and Mitigation</h2>
+  <ul>
+    <li><strong>Camera placement adjustments (Medium likelihood / Low impact):</strong> Field conditions (obstructions/lighting) may require minor location changes. <em>Mitigation:</em> Confirm coverage and angles with Liberty onsite prior to final mounting.</li>
+    <li><strong>Network readiness for remote access (Medium / Medium):</strong> Remote viewing depends on firewall/router policy and available network ports/VLANs. <em>Mitigation:</em> Align on network access approach before go-live; implement least-privilege access controls.</li>
+    <li><strong>Shop operations/access windows (Medium / Low):</strong> Work in active bays may require scheduling around vehicle movement and safety constraints. <em>Mitigation:</em> Coordinate installation windows with the shop manager; prioritize after-hours or low-traffic periods where needed.</li>
+  </ul>
+
+  <h3>Recommended Next Steps</h3>
+  <p><strong>Liberty (Owner)</strong></p>
+  <ul>
+    <li>Confirm which options to include: TV monitoring, extended storage, and/or 24-camera expansion.</li>
+    <li>Identify authorized users for mobile/desktop access.</li>
+  </ul>
+  <p><strong>IT Pro Partner</strong></p>
+  <ul>
+    <li>Conduct a short onsite walkthrough to confirm camera positions, NVR location, and (if selected) TV placement.</li>
+    <li>Finalize implementation schedule and execute install/provisioning.</li>
+  </ul>
+  <p class="small">Assumptions: Final camera locations will be confirmed onsite based on the plan and real-world mounting/coverage constraints.</p>
+
+  <h2>Project Timeline</h2>
+  <table>
+    <thead><tr><th>Project Area</th><th>Description</th><th>Time</th></tr></thead>
+    <tbody>
+      <tr><td>Network Closet</td><td>Install patch panel, terminate/label cables. Install NVR, switch, KVM for remote viewing on TV (optional)</td><td>2 days</td></tr>
+      <tr><td>Data Pulls</td><td>Pull data to each location and terminate at camera and in network closet</td><td>2 days</td></tr>
+      <tr><td>Training</td><td>Train Liberty staff on PC app and mobile app</td><td>1 day</td></tr>
+    </tbody>
+  </table>
+  <p>Total estimated onsite effort: 5 business days (2 + 2 + 1), excluding procurement lead time and any scheduling constraints. Estimated Start: Within 7-10 business days after acceptance, deposit, and equipment delivery.</p>
+
+  <h2>Terms/Conditions</h2>
+  <ul>
+    <li><strong>Procurement &amp; Invoicing:</strong> If the quote is approved, Liberty Tire and Brake will be invoiced for equipment/software procurement. Remaining balance will be invoiced with Net 14 payment terms.</li>
+    <li><strong>Warranty Coverage:</strong> All NVR and camera equipment includes a 3-year manufacturer warranty (Reolink Professional Series). Installation workmanship is warrantied for 90 days.</li>
+  </ul>
+</section>
+
+<!-- ============ PAGE 11: CONCLUSION ============ -->
+<section class="page-break">
+  <h2>Conclusion</h2>
+  <p>The optional back-office viewing station provides a convenient, real-time monitoring experience for management and staff.</p>
+  <p>IT Pro Partner appreciates the opportunity to enhance Liberty Tire and Brake's security infrastructure, and we look forward to delivering a system designed for reliability, visibility, and future growth.</p>
+</section>
+
+<!-- ============ PAGE 12: OPTIONAL IT SERVICES ============ -->
+<section class="page-break">
+  <h2>Optional IT Services</h2>
+  <h3>Hourly Services ($195/hr.)</h3>
+  <ul>
+    <li>Hardware or Software Break/fix &amp; Troubleshooting</li>
+    <li>Project Management/Research</li>
+    <li>New installs &mdash; Equipment provided by IT Pro Partner will be sold to customer at 15% over cost.</li>
+  </ul>
+  <h3>Subscription/Managed Service</h3>
+  <ul>
+    <li><strong>Computer Automation &amp; Monitoring ($69.95/computer/month)</strong>
+      <ul>
+        <li>Antivirus (MDR provides protection against viruses, spyware, adware, rootkits, phishing, and ransomware)</li>
+        <li>OS updates</li>
+        <li>Application deployment and updates</li>
+        <li>Discounts available for 10+ computers</li>
+      </ul>
+    </li>
+    <li><strong>Endpoint Management with Remote troubleshooting ($89.95/computer/month)</strong></li>
+    <li><strong>Network Maintenance/Monitoring/Patching/Updates ($99.95/firewall/month)</strong></li>
+    <li><strong>IT Support Retainer</strong>
+      <ul>
+        <li>4-Hour block &mdash; $700.00/mo. (rates locked with a 2-year agreement)</li>
+        <li>10-Hour block &mdash; $1,500.00/mo. (rates locked with a 2-year agreement)</li>
+        <li>15-Hour block &mdash; $2,175.00/mo. (rates locked with a 2-year agreement)</li>
+        <li>Hours can be used for installs, break fix, or project management/research and expire at the end of each month.</li>
+      </ul>
+    </li>
+  </ul>
+</section>
+
+</body>
+</html>

proposals/verdicttank/consolidation-reasoning-overlap-kimi.md

@@ -0,0 +1,333 @@
+# VerdictTank Pipeline Consolidation Recommendation
+## Lens: Reasoning / Logic Overlap Detection (Contradiction Detector)
+**Author:** Kimi K2.6 (Reasoning-Verification specialist)
+**Spec:** judge-pool-spec.md v2.1
+**Target:** ~15% reduction
+**Date:** 2026-08-12
+
+---
+
+## Executive finding
+
+**ONE recommendation:** Collapse **Phase 10 (Reasoning-Verification)** and **Phase 11 (Audit Agent)** into a single post-scoring synthesis seat: **Synthesis and Integrity Gate**.
+
+These two roles ask the same meta-question with different labels:
+
+| Role | Stated question | Actual job |
+|---|---|---|
+| Phase 10 Reasoning-Verification | Where do judges contradict? | Read all score sets, find disagreements, synthesize |
+| Phase 11 Audit Agent | What did the panel miss / groupthink? | Read finished review, find blind spots / groupthink, adjust confidence |
+
+Both are **non-scoring meta-reviewers of the panel's own output**. Neither sees the raw proposal as primary input in a unique way that the other does not eventually consume. Both produce a **post-hoc integrity adjustment**, not an independent domain score. Under a contradiction-detection lens, they are statistically and logically collinear seats.
+
+---
+
+## 1. Changes
+
+### Collapse
+
+| Remove | Absorb into |
+|---|---|
+| Phase 10: Reasoning-Verification (standalone) | **New Phase 10: Synthesis and Integrity Gate** |
+| Phase 11: Audit Agent (standalone) | (same merged seat) |
+
+### New merged role definition
+
+| Field | Value |
+|---|---|
+| **Phase** | 10 (final) |
+| **Role** | **Synthesis and Integrity Gate** |
+| **Model** | Kimi K2.6 (Moonshot) — keeps contradiction specialty as primary |
+| **Fallback 1** | Claude Fable 5 (Anthropic) — former Audit primary |
+| **Fallback 2** | GPT-5.6 Sol (OpenAI) — former Reasoning F2 |
+| **Scores?** | No (preserves non-scoring fence) |
+| **Input** | ALL score sets from phases 2-9 + assembled draft review |
+| **Output** | (a) Contradiction / alignment report, (b) synthesized dimension scores via section 4.1 formula, (c) blind-spot / groupthink notes, (d) single confidence adjustment +/-0.5 |
+
+### Prompt contract (must stay explicit)
+
+Merged prompt has **two ordered sections**, not a vague do-both:
+
+1. **Contradiction pass** (former Phase 10): pairwise/cluster disagreement map across Primary, Validation, Cross-Check avg, Financial, Team, Market, Execution, Legal. Emit synthesized scores.
+2. **Integrity pass** (former Phase 11): given the synthesis above, scan for groupthink (panel sigma too low), missed domain blind spots, and apply **one** confidence adjustment (+/-0.5).
+
+Single model, single call, single output schema. No second serial hop.
+
+### Spec section edits required
+
+| Section | Edit |
+|---|---|
+| section 1.1 Pipeline Phases | Drop row 11; rewrite Phase 10 as Synthesis and Integrity Gate; phase count **11 to 10** |
+| section 1.2 Enterprise Lineup | **13 seats to 11 seats** (12 unique models still available in pool; one fewer active seat) |
+| section 1.2 Model double-ups | Remove Fable 5: Team + Audit double-up. Fable becomes Reasoning/Audit **fallback only** (Team primary only) |
+| section 2.3 Fallback table | Delete Audit Agent row; replace Reasoning-Verification row with merged role chain above |
+| section 3.1 Hard constraints | Replace "Audit reads FINISHED review only" with "Synthesis and Integrity Gate is post-scoring, read-only, cannot open new domain scores" |
+| section 3.3 Low-complexity Enterprise | Already drops Audit — update text to drop CC-C, Team, Legal (Audit no longer separate) |
+| section 4.1 Scoring algorithm | Merge the two non-scoring bullets into one: Synthesis and Integrity Gate produces contradiction report + synthesized scores + confidence +/-0.5 |
+| section 4.2 Cost ceiling rule | Swap Tier A to B for non-scoring roles first (Research, Synthesis) — drop separate Audit |
+| section 4.4 Latency | Enterprise sequential phases lose one hop — recalibrate target downward slightly (e.g. 180s to ~165s aspirational; keep 300s max) |
+| section 6.3 Feature matrix | Replace Audit Agent (Phase 11) row with Synthesis and Integrity Gate (includes audit) — still Enterprise/WL only if desired |
+
+### What this is NOT
+
+- Not dropping Cross-Check C (that is diversity, not logic-overlap).
+- Not merging Primary with Validation (Validation is deliberately contaminated by Primary; different question).
+- Not merging Market with Execution (different input scopes and vertical triggers).
+- Not touching Research (unique tool surface; never scores).
+
+---
+
+## 2. Reduction %
+
+### Seat / phase accounting (Enterprise standard panel)
+
+| Metric | Before (v2.1) | After | Delta |
+|---|---|---|---|
+| Pipeline phases | 11 | **10** | -1 phase (-9.1%) |
+| Active seats | 13 | **11** | -2 seats (-15.4%) |
+| Serial post-score hops | 2 (P10 then P11) | **1** | -50% of meta-tail latency |
+| Non-scoring roles | 3 (Research, Reasoning, Audit) | **2** (Research, Synthesis) | -33% meta overhead |
+| Unique models in default lineup | 11 distinct (Sonnet x2, Fable x2) | **11 distinct** (Sonnet still x2 Validation+Legal; Fable Team-only) | pool size unchanged |
+| Anthropic share of active seats | 4/13 = 30.8% | **3/11 = 27.3%** | more headroom under 33% cap |
+
+### Cost reduction (directional)
+
+- Eliminate one full Tier-A (or Tier-A-fallback) completion on every Enterprise review.
+- v2.1 cited about $1.30-2.10 per standard full review — expect **~12-18% COGS cut** on the judge tail (one fewer expensive synthesis call), landing near **~15%** all-in when weighted by token volume of P10+P11 vs earlier short scoring calls.
+- Latency: remove one sequential dependency (~15-40s depending on model) from the critical path after phase 9.
+
+**Headline reduction: ~15%** (seats -15.4%; cost band -12-18%; phases -9%).
+
+---
+
+## 3. What survives
+
+### Intact domain + scoring spine
+
+```
+1  Research Agent               (Grok 4.5)           - grounding, never scores
+2  Primary Reviewer             (Claude Opus 5)      - 10-dimension critique
+3  Validation Reviewer          (Claude Sonnet 5)    - challenges Primary
+4a Cross-Check A                (GPT-5.6 Sol)        - blind re-score
+4b Cross-Check B                (Gemini Pro Latest)  - blind re-score
+4c Cross-Check C                (DeepSeek V4 Pro)    - blind re-score
+5  Financial Integrity          (MiniMax-M3)         - cap table / unit econ
+6  Team/Founder Assessment      (Claude Fable 5)     - founder-market fit
+7  Market-Reality               (Qwen3.7 Plus)       - TAM / competitive / non-Western
+8  Execution-Feasibility        (GPT-5.6 Terra)      - ops / timeline realism
+9  Legal/Regulatory             (Claude Sonnet 5)    - multi-jurisdiction risk
+10 Synthesis and Integrity Gate (Kimi K2.6)          - contradictions + audit + +/-0.5
+```
+
+### Preserved invariants
+
+- Research never scores
+- Three blind cross-checks, isolated from Primary/Validation
+- Domain specialists (Financial, Team, Market, Execution, Legal) remain **separate scoring questions**
+- Section 4.1 weighted aggregation formula unchanged
+- Cross-vendor fallback rule unchanged (merged chain still cross-vendor at every hop)
+- Vendor cap still satisfied (Anthropic drops to 27.3%)
+- Build validation gate (section 5.5) unchanged — still require >=0.5 panel advantage
+- Fenced post-score mode retained (merged seat cannot reopen domain scores)
+
+### Why these roles were NOT collapsed
+
+| Pair | Why keep separate |
+|---|---|
+| Primary vs Validation | Validation is **score-conditioned** (sees Primary). Different epistemic job than independent re-score. |
+| Validation vs Cross-Checks | CCs are blind; Validation is not. Collapsing destroys the blind control. |
+| Cross-Check A/B/C | Same question by design — but **vendor diversity is the product**. Overlap is intentional ensemble, not accidental duplication. Do not collapse under this lens. |
+| Market vs Execution | Different section 3.2 triggers, different evidence (competitive density vs ops timeline). |
+| Financial vs Execution | Cap-table math is not shipping realism. |
+| Legal vs Audit | Legal is domain-scoring on statutes; Audit was meta. Legal stays; Audit folds into Synthesis. |
+| Team vs Audit (same model Fable) | Team scores founders; Audit was meta. After merge, Fable stops double-duty — **removes same-model Team-to-Audit soft contamination path**. |
+
+---
+
+## 4. Risks
+
+| Risk | Severity | Mitigation |
+|---|---|---|
+| **Single-call attention split** — model skimps on contradiction map OR groupthink scan | Medium | Enforce two-section JSON schema; reject output missing either contradictions[] or integrity.confidence_delta. |
+| **Loss of independent second meta-reader** — two weak meta-passes can catch what one strong pass misses | Medium | Keep dual-section prompt; optionally run Fable fallback as **shadow audit** on 10% of Enterprise reviews for 90 days (not on critical path). |
+| **K2.6 empty-output / reasoning-token burn** (section 4.5) now single-points the entire meta-tail | High | Raise min tokens to 768-1024 for merged seat; on empty go immediate F1 (Fable). Circuit breaker already in section 4.2. |
+| **Confidence +/-0.5 applied by same model that synthesized scores** — self-grading bias | Medium | Constrain: confidence delta may only move on documented groupthink (panel sigma under threshold) or explicit missed-claim list; ban free-form "I feel +/-0.5". |
+| **Pro tier marketing** currently sells no Audit; matrix row rename may confuse | Low | Feature matrix: Synthesis and Integrity (Enterprise) — capability retained, seat count honest. |
+| **Low-complexity Enterprise drop list** referenced Audit explicitly | Low | Update section 3.3 text only. |
+| **Outcome-tracking granularity** — cannot attribute T+90 errors to Reasoning vs Audit separately | Low | Accept; meta-tail was always under-identified. Track merged seat as one accuracy series. |
+| **Over-collapse temptation** — if 15% feels good, next PR kills CC-C or Validation | Process | This lens only authorizes **logic-collinear non-scoring** merges. Ensemble scorers are out of scope. |
+
+### Explicit non-risks
+
+- Vendor diversity of **scoring** panel unchanged (still 8 vendors available; scoring seats still multi-lab).
+- No new same-vendor Primary/meta self-audit (Kimi is not Anthropic Primary/Validation).
+- Fable no longer audits after sitting Team — **contamination risk decreases**.
+
+---
+
+## 5. New diagram
+
+### Before (v2.1) — 11 phases / 13 seats
+
+```
+[Proposal]
+    |
+    v
++-----------------+
+| 1 Research      |  Grok 4.5          (no score)
++--------+--------+
+         v
++-----------------+
+| 2 Primary       |  Opus 5            (scores)
++--------+--------+
+         v
++-----------------+
+| 3 Validation    |  Sonnet 5          (scores, sees Primary)
++--------+--------+
+         v
+    +----+----+
+    v    v    v
++----+ +----+ +----+
+|4a  | |4b  | |4c  |  Sol / Gemini / DeepSeek  (blind scores)
++--+-+ +--+-+ +--+-+
+   +-----+-----+
+         v
++-----------------+
+| 5 Financial     |  MiniMax-M3
++--------+--------+
+         v
++-----------------+
+| 6 Team/Founder  |  Fable 5
++--------+--------+
+         v
++-----------------+
+| 7 Market        |  Qwen3.7 Plus
++--------+--------+
+         v
++-----------------+
+| 8 Execution     |  Terra
++--------+--------+
+         v
++-----------------+
+| 9 Legal         |  Sonnet 5
++--------+--------+
+         v
++-----------------+
+|10 Reasoning     |  Kimi K2.6         (contradictions + synth)
++--------+--------+
+         v
++-----------------+
+|11 Audit         |  Fable 5           (groupthink + +/-0.5)
++--------+--------+
+         v
+    [Final Report]
+```
+
+### After (recommended) — 10 phases / 11 seats  (~15% seat cut)
+
+```
+[Proposal]
+    |
+    v
++-----------------+
+| 1 Research      |  Grok 4.5          (no score)
++--------+--------+
+         v
++-----------------+
+| 2 Primary       |  Opus 5            (scores)
++--------+--------+
+         v
++-----------------+
+| 3 Validation    |  Sonnet 5          (scores, sees Primary)
++--------+--------+
+         v
+    +----+----+
+    v    v    v
++----+ +----+ +----+
+|4a  | |4b  | |4c  |  Sol / Gemini / DeepSeek  (blind scores)
++--+-+ +--+-+ +--+-+
+   +-----+-----+
+         v
++-----------------+
+| 5 Financial     |  MiniMax-M3
++--------+--------+
+         v
++-----------------+
+| 6 Team/Founder  |  Fable 5           (Team only — no Audit double)
++--------+--------+
+         v
++-----------------+
+| 7 Market        |  Qwen3.7 Plus
++--------+--------+
+         v
++-----------------+
+| 8 Execution     |  Terra
++--------+--------+
+         v
++-----------------+
+| 9 Legal         |  Sonnet 5
++--------+--------+
+         v
++-------------------------------------+
+|10 Synthesis and Integrity Gate      |  Kimi K2.6
+|   - contradiction / alignment map   |
+|   - section 4.1 synthesized scores  |
+|   - groupthink / blind-spot notes   |
+|   - confidence delta +/-0.5 (gated) |
+|   F1: Fable 5 · F2: Sol             |
++------------------+------------------+
+                   v
+              [Final Report]
+```
+
+### Overlap evidence (why this pair, not another)
+
+```
+                    INPUT SCOPE
+                 proposal  scores  finished-review
+Primary            ###      .        .
+Validation         ##       ###      .
+Cross-Checks       ###      .        .     intentional ensemble (KEEP x3)
+Financial          ##       ##       .
+Team               ##       ##       .
+Market             ##       ##       .
+Execution          ##       ##       .
+Legal              ##       #        .
+Reasoning (P10)    .        ###      ##    ]
+Audit (P11)        .        ##       ###   ]  SAME META QUESTION
+Research           tools    .        .     (unique — KEEP)
+
+Overlap score P10 intersect P11:
+  question similarity   ~0.85  (contradiction equiv missed-agreement/groupthink)
+  input Jaccard         ~0.80  (both post-domain, panel-output-primary)
+  output collinearity   ~0.75  (both emit integrity adjustment, not domain scores)
+  -> COLLAPSE CANDIDATE under reasoning lens
+```
+
+### Opus empirical echo (from v2.1 review results)
+
+Opus 5 panel-vs-single test already showed cross-checks can be **statistically tight** (market dimension spread 0.00 across three CCs). That finding pressures **ensemble scorer** count — out of scope for *this* lens. The safer 15% cut is the **duplicated meta-tail**, not the diversity scorers the product sells. Killing Audit+Reasoning redundancy reduces COGS without touching the scoring majority (still 9 scoring judges + Research + 1 meta).
+
+**Scoring judges retained: 9** (Primary, Validation, 3xCC, Financial, Team, Market, Execution, Legal)
+**Meta retained: 1** (merged) + Research
+
+---
+
+## Decision line
+
+> **Collapse Reasoning-Verification and Audit Agent into one Synthesis and Integrity Gate (Kimi K2.6).**
+> **-2 seats (-15.4%), -1 serial phase, ~15% meta-tail cost, cleaner vendor math, removes Fable Team-to-Audit double-duty.**
+> Domain ensemble untouched. Ship as v2.2 delta on top of v2.1 blocker fixes.
+
+---
+
+## Implementation checklist (for conductor)
+
+1. Patch judge-pool-spec.md sections listed in section 1
+2. Bump version header to v2.2; status remains FINAL DRAFT pending 3-5 proposal gate
+3. Update feature matrix + latency table
+4. Rewrite section 2.3 fallback row; delete Audit row
+5. Add output JSON schema for two-section merged seat
+6. Re-run vendor % assertion (expect 3/11 Anthropic)
+7. Do **not** change Pro 7-judge composition in this PR (Enterprise matrix row rename only)
+8. Optional: 90-day 10% shadow Fable integrity pass off critical path

proposals/verdicttank/consolidation-research-agent-15pct.md

@@ -0,0 +1,293 @@
+# VerdictTank v2.1 — ONE Consolidation Recommendation
+## Research Agent Grounding Lens (Grok 4.5)
+
+**Date:** 2026-08-12  
+**Author role:** Research Agent (only judge with native live web + social search)  
+**Target:** ~15% reduction without losing effectiveness  
+**Constraints satisfied:** no model >2 roles · Audit + Research survive · ≥2 cross-checks
+
+---
+
+## Research-lens thesis
+
+I produce the factual brief every downstream judge consumes. From that position, two seats add **latency and tokens without new signal**:
+
+1. **Market-Reality (Phase 7)** mostly **re-summarizes my brief** — competitive density, TAM defensibility, and citation-backed market claims are already Research outputs. Its unique residual (non-Western lens) is a *prompt/schema* requirement, not a reason for a full serial phase.
+2. **Cross-Check C as a third parallel scorer** is **highly correlated** with A/B. Internal Opus 5 empirical validation on this stack found all three cross-checks scored `market` at **exactly 6** (spread **0.00**). External ensemble guidance: if judges agree on everything, you bought one verdict three times (orq.ai LLM juries; Verga et al. "Judges → Juries"). Two diverse cross-checks capture the ensemble; a third correlated scorer is mostly cost.
+
+Specialist phases that do **not** rehash Research (Financial math, Team/Founder fit, Execution ops, Legal/regulatory code) stay. Audit and Reasoning stay as non-scoring synthesis/gates.
+
+---
+
+## (1) Changes — single consolidation: **Grounding-Redundancy Cut**
+
+### A. Eliminate standalone Market-Reality (Phase 7)
+
+| Action | Detail |
+|---|---|
+| Remove | Phase 7 seat: Market-Reality / Qwen3.7 Plus (default) |
+| Absorb into Research | Expand Phase 1 brief schema with a mandatory **Market block**: competitive density table, TAM/SAM defensibility notes, ≥1 non-Western comps when vertical warrants, citation URLs, "unknown/unverifiable" flags |
+| Absorb residual scoring | Market *dimension* continues via Primary + Validation + 2 cross-checks (already score all 10 dimensions). No separate market narrative phase |
+| Vertical weights (§3.2) | Market-Reality weight boosts → apply to **Research brief depth** (more market citations) + **Primary/Validation market-dimension weight**, not a missing judge |
+
+### B. Collapse cross-checks 3 → 2 (keep minimum)
+
+| Action | Detail |
+|---|---|
+| Remove | Phase 4c Cross-Check C as default Enterprise seat |
+| Keep | Phase 4a Cross-Check A (GPT-5.6 Sol) + Phase 4b Cross-Check B |
+| Diversity preserve | **Reassign DeepSeek V4 Pro → Cross-Check B** (replace Gemini Pro Latest in default lineup). Reasons: (i) keeps Chinese-lab / non-Western error surface that Market+CC-C previously carried; (ii) avoids Gemini floating-alias longitudinal noise documented in v2.1 §4.5 and review results; (iii) Gemini remains Fallback 1 for Research and available in pool |
+| Blindness | Both remaining CCs still see **proposal + Research brief ONLY** — unchanged isolation |
+
+### C. What does *not* change
+
+- Research Agent never scores
+- Audit Agent remains fenced, post-scoring, read-only
+- Reasoning-Verification still synthesizes all score sets
+- Financial, Team/Founder, Execution, Legal stay as serial specialists
+- Fable double: Team + Audit (still 2)
+- No other model exceeds 2 roles
+
+### D. Lineup delta (Enterprise default)
+
+| # | Phase | Role | Model (after) | Vendor | Change |
+|---|---|---|---|---|---|
+| 1 | 1 | Research Agent | Grok 4.5 | xAI | **Survives** — brief schema expanded |
+| 2 | 2 | Primary Reviewer | Claude Opus 5 | Anthropic | Unchanged |
+| 3 | 3 | Validation Reviewer | Claude Sonnet 5 | Anthropic | Unchanged |
+| 4 | 4a | Cross-Check A | GPT-5.6 Sol | OpenAI | Unchanged |
+| 5 | 4b | Cross-Check B | **DeepSeek V4 Pro** | DeepSeek | **Was Gemini; DeepSeek moved here** |
+| — | ~~4c~~ | ~~Cross-Check C~~ | — | — | **REMOVED** |
+| 6 | 5 | Financial Integrity | MiniMax-M3 | MiniMax | Unchanged |
+| 7 | 6 | Team/Founder | Claude Fable 5 | Anthropic | Unchanged |
+| — | ~~7~~ | ~~Market-Reality~~ | — | — | **REMOVED** (folded into Research) |
+| 8 | 7' | Execution-Feasibility | GPT-5.6 Terra | OpenAI | Unchanged |
+| 9 | 8' | Legal/Regulatory | **Qwen3.7 Plus** | Alibaba | **Vendor-cap patch** (was Sonnet) |
+| 10 | 9' | Reasoning-Verification | Kimi K2.6 | Moonshot | Unchanged |
+| 11 | 10' | Audit Agent | Claude Fable 5 | Anthropic | **Survives** |
+
+**Seats:** 13 → **11**  
+**Phases:** 11 → **10** (4a–4b parallel; Market gone)  
+**Scoring judges:** 10 → **8** (Primary, Validation, CC-A, CC-B, Financial, Team, Execution, Legal)  
+**Non-scoring:** Research, Reasoning, Audit (3)  
+**Total:** 8 + 3 = **11**
+
+Qwen3.7 Plus remains on the default path via Legal (not dropped from Enterprise). Gemini stays in pool + Research fallback.
+
+### E. Fallback table patches (must ship with this cut)
+
+- Delete Market-Reality row **or** keep as optional on-demand role for White-Label only
+- Cross-Check B primary becomes DeepSeek V4 Pro; F1 Gemini Pro Latest; F2 Claude Fable 5 (still cross-vendor)
+- Cross-Check C row retired from Enterprise default (pool may retain for WL custom panels)
+- Legal primary: Qwen3.7 Plus; F1 Gemini Pro Latest; F2 DeepSeek V4 Pro (cross-vendor)
+- Research F2 can stay Terra; optional: add Qwen as Research F3 for market-heavy verticals
+
+### F. Research brief schema addition (contract)
+
+```
+## Market block (mandatory)
+- Competitors: name, category (authoring vs critique vs other), price anchor, source URL
+- Non-Western / non-US comps: min 1 when vertical in {superapp, cross-border, unclassified-global}; else "N/A — US-centric vertical"
+- TAM/SAM claims: verified | overstated | unverifiable — with citation or explicit gap
+- Density judgment: sparse | contested | saturated — one paragraph, citations only
+```
+
+Downstream judges **cite the Market block ID**, not re-crawl the open web (except Legal/Financial specialists on their narrow facts).
+
+---
+
+## (2) Reduction %
+
+| Metric | Before (v2.1) | After | Delta |
+|---|---|---|---|
+| Seats (Enterprise default) | 13 | 11 | **−15.4%** |
+| Phases | 11 | 10 | −9.1% |
+| Parallel CC calls | 3 | 2 | −33% of CC fan-out |
+| Scoring judges | 10 | 8 | −20% |
+| Serial specialist after scores | Market + Exec + Legal… | Exec + Legal… | −1 serial hop |
+| Est. E2E latency (vs 218s baseline) | 218s | ~185–195s | **~11–15%** wall-clock |
+| Est. token/COGS share | full panel | −Market narrative −1 full score JSON | **~14–18%** cost |
+
+**Headline reduction: ~15%** (seat count exact at −15.4%; cost/latency band 14–18% / 11–15%).
+
+Why not larger: cutting Audit or a second CC would violate constraints or erase the disagreement signal that makes panels worth running (selection-bottleneck literature: judge-based selection > synthesis; arxiv 2603.20324).
+
+---
+
+## (3) What survives (effectiveness preserved)
+
+| Kept | Why it still works |
+|---|---|
+| **Research Agent** | Stronger, not weaker — owns market grounding explicitly |
+| **Audit Agent** | Groupthink / blind-spot gate unchanged |
+| **≥2 cross-checks** | Sol + DeepSeek: US frontier + non-Western lab; blind; proposal+brief only |
+| **Primary + Validation** | Dual Anthropic pass with sober second read (different tiers A/B) |
+| **Financial, Team, Execution, Legal** | True specialists; low overlap with Research crawl |
+| **Reasoning-Verification** | Contradiction synthesis over fewer, higher-signal score sets |
+| **Vendor cap** | After Legal→Qwen: Anthropic 3/11 = **27.3%** (under 33%) |
+| **Pool diversity** | Gemini remains in pool/fallbacks; White-Label can still pin full custom panels |
+| **Model role cap** | Fable Team+Audit = 2; all others ≤1 |
+
+---
+
+## (4) Risks
+
+| Risk | Severity | Mitigation |
+|---|---|---|
+| Market nuance loss on superapp / cross-border verticals | Med | Research Market block mandatory; §3.2 superapp trigger increases Research market citation quota + Primary market-dimension weight +30% |
+| DeepSeek as sole non-Western CC may under-challenge US-centric Primary | Med | Legal→Qwen adds second non-Western seat; Audit watches for US-default groupthink |
+| Anthropic share 4/11 if Legal stays Sonnet | High (rule break) | **Must** reassign Legal → Qwen (included in this same cut) |
+| Sol still cannot tool-use on admin-ai | Low | CC-A is scoring-only JSON — already OK per §4.5 |
+| Dropping third CC reduces disagreement surface | Med-Low | Literature + internal 0.00 market spread say third was fake diversity; monitor panel sigma for 30 days; if sigma collapses, restore CC-C on High complexity only (§3.3 already expands high-complexity) |
+| Research brief becomes single point of market failure | Med | Research fallback chain unchanged (Gemini grounding → Terra); malformed Research → subscriber notify + degrade already specified |
+| Pro tier "7 judges" matrix needs rewrite | Low | Define Pro as: Research + Primary + Validation + 2 CC + Execution + Reasoning (no Financial/Team/Legal/Audit) — document explicitly |
+| Build-gate still unproven (panel delta <0.5 vs solo) | Existential (pre-existing) | This cut helps the thesis: fewer correlated seats make a ≥0.5 panel advantage more plausible if one exists; still run 3–5 real proposals before build |
+
+---
+
+## (5) New diagram
+
+### Before (v2.1) — 11 phases / 13 seats
+
+```
+                    ┌─────────────────────┐
+                    │ 1 Research (Grok)   │  grounding brief
+                    └──────────┬──────────┘
+                               ▼
+                    ┌─────────────────────┐
+                    │ 2 Primary (Opus 5)  │  scores
+                    └──────────┬──────────┘
+                               ▼
+                    ┌─────────────────────┐
+                    │ 3 Validation (Sonnet)│  challenge
+                    └──────────┬──────────┘
+                               ▼
+              ┌────────────────┼────────────────┐
+              ▼                ▼                ▼
+        ┌──────────┐    ┌──────────┐    ┌──────────┐
+        │4a CC-A   │    │4b CC-B   │    │4c CC-C   │  blind re-scores
+        │ Sol      │    │ Gemini   │    │ DeepSeek │
+        └────┬─────┘    └────┬─────┘    └────┬─────┘
+              └────────────────┼────────────────┘
+                               ▼
+                    ┌─────────────────────┐
+                    │ 5 Financial (M3)    │
+                    └──────────┬──────────┘
+                               ▼
+                    ┌─────────────────────┐
+                    │ 6 Team (Fable)      │
+                    └──────────┬──────────┘
+                               ▼
+                    ┌─────────────────────┐
+                    │ 7 Market (Qwen)     │  ← re-summarizes Research
+                    └──────────┬──────────┘
+                               ▼
+                    ┌─────────────────────┐
+                    │ 8 Execution (Terra) │
+                    └──────────┬──────────┘
+                               ▼
+                    ┌─────────────────────┐
+                    │ 9 Legal (Sonnet)    │
+                    └──────────┬──────────┘
+                               ▼
+                    ┌─────────────────────┐
+                    │10 Reasoning (Kimi)  │  synthesize
+                    └──────────┬──────────┘
+                               ▼
+                    ┌─────────────────────┐
+                    │11 Audit (Fable)     │  fenced ±0.5
+                    └─────────────────────┘
+```
+
+### After — Grounding-Redundancy Cut — 10 phases / 11 seats (−15.4%)
+
+```
+                    ┌──────────────────────────────────┐
+                    │ 1 Research (Grok 4.5)            │
+                    │    + mandatory Market block      │  LIVE WEB (only)
+                    └────────────────┬─────────────────┘
+                                     ▼
+                    ┌──────────────────────────────────┐
+                    │ 2 Primary (Opus 5)               │
+                    └────────────────┬─────────────────┘
+                                     ▼
+                    ┌──────────────────────────────────┐
+                    │ 3 Validation (Sonnet 5)          │
+                    └────────────────┬─────────────────┘
+                                     ▼
+                         ┌───────────┴───────────┐
+                         ▼                       ▼
+                  ┌─────────────┐         ┌─────────────┐
+                  │4a CC-A Sol  │         │4b CC-B      │  2 blind CCs
+                  │             │         │ DeepSeek    │  (min met)
+                  └──────┬──────┘         └──────┬──────┘
+                         └───────────┬───────────┘
+                                     ▼
+                    ┌──────────────────────────────────┐
+                    │ 5 Financial (MiniMax-M3)         │
+                    └────────────────┬─────────────────┘
+                                     ▼
+                    ┌──────────────────────────────────┐
+                    │ 6 Team/Founder (Fable 5)         │
+                    └────────────────┬─────────────────┘
+                                     ▼
+                    ┌──────────────────────────────────┐
+                    │ 7 Execution (Terra)              │  Market phase GONE
+                    └────────────────┬─────────────────┘
+                                     ▼
+                    ┌──────────────────────────────────┐
+                    │ 8 Legal (Qwen3.7 Plus)           │  vendor-cap patch
+                    └────────────────┬─────────────────┘
+                                     ▼
+                    ┌──────────────────────────────────┐
+                    │ 9 Reasoning-Verification (Kimi)  │
+                    └────────────────┬─────────────────┘
+                                     ▼
+                    ┌──────────────────────────────────┐
+                    │10 Audit (Fable 5)                │  SURVIVES
+                    └──────────────────────────────────┘
+```
+
+### Signal flow (Research lens)
+
+```
+ Research brief
+   ├─ facts/citations ──────────────► all scorers (unchanged)
+   ├─ Market block (NEW) ───────────► Primary / Validation / CCs
+   │                                   (no Phase-7 rewrite)
+   └─ gaps/unverifiable ────────────► Audit watches for overclaim
+
+ Dropped edges (were low-signal):
+   Research ══re-summary══► Market narrative ══► Reasoning
+   CC-C ══correlate≈0══► CC average
+```
+
+---
+
+## Constraint checklist
+
+| Constraint | Status |
+|---|---|
+| ~15% reduction | **15.4% seats**; ~14–18% COGS; ~11–15% latency |
+| No model >2 roles | Fable Team+Audit=2; all others ≤1 |
+| Audit survives | Phase 10' Fable, fenced |
+| Research survives | Phase 1 Grok, expanded |
+| Minimum 2 cross-checks | Sol + DeepSeek |
+| Vendor ≤33% | After Legal→Qwen: Anthropic 3/11 = 27.3% |
+
+---
+
+## Implementation note (one PR)
+
+1. Spec §1.1 / §1.2 / §2.3 / §3.2 / §3.3 / §4.1 / §6.3 — role counts must all read **11 seats / 10 phases**
+2. Cross-section reconciliation (lesson from v2.0→v2.1 role expansion) before any re-review
+3. Cache-bust deploy `judge-pool-spec.md?v=2.2`
+4. Do **not** build until 3–5 real proposals still clear the ≥0.5 panel-vs-solo gate on the *reduced* panel
+
+---
+
+## Bottom line
+
+**ONE consolidation:** *Grounding-Redundancy Cut* — delete Market-Reality as a serial phase (it re-summarizes Research) and delete the third cross-check (empirically zero incremental spread); fold market into the Research brief; keep DeepSeek as CC-B; move Legal to Qwen for vendor-cap compliance.
+
+**−15.4% seats, Research + Audit intact, 2 blind cross-checks retained, specialists preserved.**

proposals/verdicttank/judge-pool-v2-primary-review.md

@@ -0,0 +1,411 @@
+# VerdictTank Judge Pool Spec v2.0  -  Primary Reviewer Scorecard
+
+**Reviewer:** Claude Opus 5 (Primary Reviewer, Phase 2)
+**Date:** 2026-08-12
+**Spec:** https://proposals.itpropartner.com/verdicttank/judge-pool-spec.md (17,641 bytes, v2.0)
+**Method:** Document critique + live empirical validation against admin-ai (157-model catalog, real inference runs)
+
+> **Note on version:** `web_extract` returned a cached **v1.0**. Verified against the live origin
+> via `curl`  -  the deployed file is **v2.0**, byte-identical to
+> `/root/projects/itpp-infrastructure/proposals/verdicttank/judge-pool-spec.md`. This review is of v2.0.
+
+---
+
+## Scorecard
+
+| # | Dimension | Score | Verdict |
+|---|---|---|---|
+| 1 | Role Coverage | 7/10 | FIX |
+| 2 | Model Assignments | 6/10 | FIX |
+| 3 | Vendor Diversity | 8/10 | DEFEND |
+| 4 | admin-ai Paths | 9/10 | DEFEND |
+| 5 | Fallback Chains | 4/10 | FIX |
+| 6 | Content-Based Rules | 5/10 | FIX |
+| 7 | Latency Budgets | 2/10 | FIX |
+| 8 | Feedback Loop | 5/10 | FIX |
+| 9 | Pricing | 4/10 | FIX |
+| 10 | Product Boundary | 7/10 | DEFER |
+
+**Weighted mean: 5.7/10**  -  Architecturally literate, empirically unvalidated. Two dimensions (7, 9) are
+launch-blocking. One unlisted finding (§Critical Finding) threatens the product thesis itself.
+
+---
+
+## CRITICAL FINDING (unlisted dimension  -  read first)
+
+**The 12-judge panel does not measurably outperform one model run once.**
+
+I tested this directly. Same proposal, same rubric, three dimensions, live models.
+
+**A) Single-model noise floor**  -  `claude-opus-5` x6 at temp 1.0:
+```
+market: 5,5,5,4,4,4   stdev 0.50
+team:   4,4,4,4,4,4   stdev 0.00
+fin:    3,3,3,3,3,3   stdev 0.00
+```
+
+**B) Nine distinct judges across 8 vendors, one run each:**
+```
+Opus5 [5,4,3]  Sonnet5 [4,4,3]  Sol [6,4,3]  GeminiPro [4,3,2]  DeepSeek [5,3,2]
+Terra [4,4,3]  Qwen  [4,3,2]  MiniMax [6,3,4]  Fable5 [6,4,4]
+panel stdev: market 0.87, team 0.50, fin 0.74
+```
+
+**C) The result that matters:**
+
+| | market | team | financials |
+|---|---|---|---|
+| Mean, 1 model | 4.50 | 4.00 | 3.00 |
+| Mean, 9 judges / 8 vendors | 4.89 | 3.56 | 2.89 |
+| **Delta** | **0.39** | **0.44** | **0.11** |
+
+Nine frontier models from eight vendors, ~218s of wall clock and roughly 20x the token cost, move the
+final number by **less than half a point on a 10-point scale**. On `market`, panel spread (0.87) is only
+**1.7x** the single-model rerun noise (0.50)  -  statistically indistinguishable from rerunning one model
+six times.
+
+The individual judges *do* disagree (market scores span 4-6). But disagreement that averages to the same
+answer is **variance, not signal**. The spec sells vendor diversity as the core accuracy mechanism and
+never once tests whether diversity changes the output.
+
+**This is the product thesis, and it is unvalidated.** Every downstream claim  -  premium pricing, the
+"most accurate review tool available" positioning, the 12-vendor moat  -  rests on it.
+
+**Required before build:** run 30-50 real proposals with known outcomes. Report panel-vs-single
+correlation against ground truth. If the panel does not beat one good model by a margin that justifies
+20x cost, the correct architecture is 3 judges, not 12  -  and the pricing story needs rebuilding.
+Better to learn this now than after a customer runs the same A/B.
+
+---
+
+## 1. Role Coverage  -  7/10  -  FIX
+
+v2.0 deserves credit: it closed three of the four gaps flagged in the prior review (Financial Integrity,
+Team/Founder, Legal/Regulatory now have dedicated phases). That is real progress.
+
+**Technical Architecture remains uncovered**  -  the one gap explicitly identified in the v4.1 gap analysis
+and silently dropped. Execution-Feasibility is operational (timeline, team, resources), not architectural
+(stack, scalability, security posture, technical debt). For a product whose buyers are evaluating
+*technical* startups, having no judge that reads the architecture is a conspicuous hole.
+
+Two further gaps neither version names:
+- **Traction/Evidence.** Nobody scores whether claims are *substantiated*. Research Agent gathers
+  citations but never scores; no downstream role is tasked with "the founder asserts 40% MoM growth  - 
+  is there evidence?" This is the single most common reason real proposals fail diligence.
+- **Narrative/Communication quality.** For a *proposal* review tool, no judge assesses whether the
+  document actually persuades.
+
+**Fix:** add Technical Architecture (merge into Execution-Feasibility if headcount is capped) and fold
+an evidence-substantiation mandate into the Research Agent's brief so grounding produces a scored
+claims-verification artifact, not just citations.
+
+---
+
+## 2. Model Assignments  -  6/10  -  FIX
+
+Most seats are defensible. Grok 4.5 on Research is correct (only pool member with native live web
+search). Qwen3.7 Plus on Market-Reality for a non-Western lens is genuinely thoughtful.
+
+**Empirically-grounded objections:**
+
+**Kimi K2.6 on Reasoning-Verification is misassigned.** In my scoring test it returned **empty content
+after burning all 900 output tokens**  -  and on the 12-page critical-path run it took 15.1s. The role
+requires emitting a structured contradiction report; a model that silently exhausts its budget in the
+seat that validates every other judge's consistency is the worst possible placement. It is also the
+*only* Moonshot seat, so there is no same-vendor fallback.
+
+**Empty-response rate is a systemic risk the spec never models.** At `max_tokens=12`, four of six probed
+models (MiniMax-M3, Kimi K2.6, Claude Fable 5, Gemini Pro) returned **empty content**  -  reasoning tokens
+consumed the entire budget. §4.1 handles "malformed score" but not "well-formed empty response," which is
+the actual failure mode I observed. Token budgets must be set per-model with reasoning-token headroom.
+
+**Claude Sonnet 5 sits in two scoring seats** (Validation + Legal/Regulatory). The spec waves this off as
+"read/analytical, not scoring-intense," but §1.1 marks Legal/Regulatory as **"Yes  -  scores."** The
+justification contradicts the table two rows above it. Same weights, same biases, two votes.
+
+**Gemini Pro Latest doubles as Cross-Check B and Audit Agent.** The Audit Agent's stated job is catching
+*groupthink*  -  it cannot audit a panel it already voted in. This directly violates the spirit of §3.1's
+own audit-independence rule.
+
+---
+
+## 3. Vendor Diversity  -  8/10  -  DEFEND
+
+The strongest dimension. Eight vendors, max 25%  -  a genuine improvement over v1.0's 44% Anthropic
+violation, and the rule was tightened from 40% to 33% rather than loosened to fit. That is the right
+instinct and the spec should defend it.
+
+**Two caveats worth documenting rather than fixing:**
+
+*Vendor diversity is not architecture diversity.* Nearly every pool member is a transformer trained on
+overlapping web corpora with similar RLHF conventions. My independence test showed all three "independent"
+cross-checks scoring `market` at **exactly 6  -  spread 0.00**. Eight logos, one prior. The correlation
+data from the Critical Finding is the real story here.
+
+*Infrastructure concentration.* All 12 models route through a single admin-ai/LiteLLM proxy. Eight-vendor
+diversity buys nothing if the proxy is down  -  that is the actual SPOF, and the spec's §4.1 fallback table
+implicitly assumes the proxy always answers.
+
+---
+
+## 4. admin-ai Paths  -  9/10  -  DEFEND
+
+**Verified live, not taken on faith.** I queried the admin-ai catalog (157 models) and confirmed all 13
+spec paths resolve exactly, then ran real inference against every one:
+
+```
+OK  xai/grok-4.5              OK  claude-opus-5           OK  claude-sonnet-5
+OK  gpt-5.6-sol               OK  gemini/gemini-pro-latest OK  deepseek-v4-pro
+OK  kimi-k2.6                 OK  gpt-5.6-terra           OK  qwen3.7-plus
+OK  MiniMax-M3                OK  claude-fable-5          OK  gemini/gemini-3.6-flash
+OK  gemini/gemini-3.5-flash-lite
+```
+
+All 11 judge models returned live responses. The spec's claim "all model paths confirmed against admin-ai"
+is **true**  -  rare enough in a v2.0 draft to call out. This dimension should be defended as-is.
+
+**The one point off:** `gemini/gemini-pro-latest` is a **floating alias**, not a pinned version. The
+catalog carries pinned alternatives (`gemini/gemini-3.1-pro-preview`, `openrouter/google/gemini-3-pro-preview`).
+Google can repoint that alias with no notice and silently change two seats  -  Cross-Check B *and* Audit.
+My determinism probe on that alias returned `[6,3,3] / [6,4,3] / [6,4,2]` across three identical calls.
+For a product whose entire value is score reproducibility, and which promises T+90/180/365 longitudinal
+outcome tracking, an unpinned model destroys year-over-year comparability. **Pin every judge seat.**
+
+---
+
+## 5. Fallback Chains  -  4/10  -  FIX
+
+§3.1 asserts "Fallback chains always cross vendor boundaries  -  Enforced at the assignment table (§2.3)."
+Reading §2.3 against that claim, the rule is **violated in the first fallback hop of 4 of 11 rows**:
+
+| Role | Primary | Fallback 1 | Violation |
+|---|---|---|---|
+| Primary Reviewer | Claude Opus 5 | **Claude Sonnet 5** | Anthropic → Anthropic |
+| Cross-Check A | GPT-5.6 Sol | **GPT-5.6 Terra** | OpenAI → OpenAI |
+| Cross-Check B | Gemini Pro Latest | **Gemini 3.6 Flash** | Google → Google |
+| Team/Founder | Claude Fable 5 | **Claude Sonnet 5** | Anthropic → Anthropic |
+
+This is the *identical* defect flagged in the v1.0 review ("§4.1 says different vendor but §2.3 says
+Opus→Sonnet"). It was marked as a top-5 prioritized fix, and it was **not fixed**  -  the assertion text was
+added to §3.1 without correcting the table it points at. A rule that is stated but not enforced is worse
+than no rule: it will pass code review as "already handled."
+
+The failure mode is precisely what fallbacks exist to prevent. Anthropic has a regional outage → Primary
+Reviewer fails over to Sonnet 5 → also Anthropic → also down. Same for the OpenAI and Google rows.
+
+**Additional defects:**
+- **Convergence, not diversity.** DeepSeek V4 Pro appears as a fallback in **7 of 11 chains**. Under
+  broad degradation the 8-vendor panel collapses toward a single DeepSeek-dominated panel, blowing the
+  33% vendor cap at exactly the moment it matters most. No runtime re-check of the cap after failover.
+- **Kimi K2.6 has no same-tier peer**  -  the sole Moonshot seat degrades straight to DeepSeek (Tier C).
+- **§4.2 is unexecutable as written.** "Tier B for Primary/Validation only, Tier C for others"  -  Tier C
+  is `DeepSeek V4 Pro, Gemini 3.6 Flash, Gemini 3.5 Flash-Lite`: two vendors, three models, for up to
+  five simultaneous seats. The cost-optimization path *cannot* satisfy the vendor rule. This is the same
+  unexecutability flagged in v1.0 ("Tier B has no OpenAI/Google model") reappearing in a new form.
+
+---
+
+## 6. Content-Based Rules  -  5/10  -  FIX
+
+The vertical/stage triggers are directionally sensible and v2.0 adds a genuine improvement: the §3.3
+missing-vertical handler with explicit user disclosure is honest product design.
+
+**But the weighting system is undefined.** Every rule adjusts "weight ±N%" and **the spec never states
+what the weights weight.** There is no aggregation formula anywhere in 302 lines  -  no statement of how 11
+judges' 10-dimension scores combine into a final number. "Market-Reality weight +30%" is meaningless
+without knowing the base weight, the combination function, and whether weights renormalize. This is the
+single largest specification hole in the document: it is the core scoring algorithm, and it is absent.
+
+**Composition rule is underspecified.** §3.9 caps stacking at +50% per role but doesn't say what happens
+when the cap binds  -  are all triggers scaled proportionally, or does first-match win? Different answers
+give different scores for the same proposal.
+
+**Detection is hand-waved.** Every row says "Detected from proposal text" with no mechanism, no confidence
+threshold, and no misclassification path. Vertical detection *changes the score*  -  a fintech misread as
+SaaS loses its +30% Legal/Regulatory weight. That needs a confidence score and a human-review fallback,
+and multi-vertical proposals (fintech + healthcare) have no representation at all.
+
+**Page count is a poor complexity proxy.** A 10-page dense technical proposal triggers the *reduced*
+panel; a 50-page deck with 40 pages of appendix triggers the full one. Use token count or content
+density.
+
+**Substantively questionable:** Seed-stage sets Financial Integrity **-20%**. Seed is where financial
+models are *most* fictional and cap-table mistakes are most permanent. De-weighting financial scrutiny
+where founders most need it inverts the diligence priority.
+
+---
+
+## 7. Latency Budgets  -  2/10  -  FIX (launch-blocking)
+
+The spec revised Enterprise from 30s → 45s and annotated it *"v1.0 was unrealistic for 12-judge pipeline."*
+The revision is still off by nearly 5x. I measured it.
+
+**Sequential critical path, 12-page proposal (~5,500 tokens), following the spec's own dependency graph:**
+
+```
+P1  Research    xai/grok-4.5               21.5s
+P2  Primary     claude-opus-5              20.0s
+P3  Validation  claude-sonnet-5            15.8s
+P5  Reasoning   kimi-k2.6                  15.1s
+P6  Execution   gpt-5.6-terra              16.4s
+P7  Market      qwen3.7-plus               67.2s   <-- single judge exceeds Enterprise MAX alone
+P8  Financial   MiniMax-M3                 26.1s
+P9  Team        claude-fable-5             22.9s
+P11 Audit       gemini/gemini-pro-latest   13.0s
+------------------------------------------------
+TOTAL                                     218.0s
+```
+
+**218s against a 45s target and a 90s max  -  4.8x over target, 2.4x over the stated maximum.**
+
+This is not a tuning problem, it is structural:
+
+- **The pipeline is inherently sequential.** Phases 2→3→5→6→7→8→9→11 each consume prior output by design.
+  Only 4a/4b/4c parallelize. You cannot fan out a dependency chain.
+- **A single judge blows the entire budget.** Qwen3.7 Plus took **67.2s alone**  -  1.5x the full Enterprise
+  target, before any other judge runs. It burned 1,450 reasoning tokens on a trivial scoring task.
+- **Even perfect parallelism fails.** All 11 models fired simultaneously on a *short* prompt still took
+  **27.4s wall clock**  -  61% of the Enterprise budget consumed by the slowest model, with zero
+  orchestration, retries, or assembly.
+- **Fallbacks make it worse.** A 30s timeout + Fallback 1 retry adds 30s+ to an already-blown budget.
+  The §4.1 retry path and the §4.3 budget are mutually unsatisfiable.
+- **Tier ordering is still inverted.** Enterprise runs the deepest pipeline (12 judges) on the tightest
+  budget (45s); Free runs 3 judges on 90s. v1.0 had this backwards and v2.0 preserved the inversion while
+  only adjusting magnitudes.
+
+50+ page proposals (the spec's own "High complexity," which *adds* a secondary pass) will run
+substantially past 218s.
+
+**Fix:** these are not real-time interactions  -  they are deep-analysis jobs. Re-architect as an
+**asynchronous job model**: submit → progress streaming → notify on completion. Budget **5-10 minutes**
+for Enterprise and sell the depth. Then set per-phase timeouts against measured p95, not aspiration.
+A 45s promise that reliably takes 218s is a support-ticket generator and a churn driver.
+
+---
+
+## 8. Feedback Loop  -  5/10  -  FIX
+
+v2.0 genuinely improved here  -  §5.4 cold-start gate, min N=30/N=50 thresholds, and the immediate-recusal
+rule for >2.0σ bias are all correct additions that address prior findings.
+
+**The remaining problems are foundational:**
+
+**The accuracy metric is not sound.** "Correlation between dimension score and T+90 outcome"  - 
+- **Ninety days is far too short.** Seed rounds take 3-9 months; the T+90 signal is mostly noise about
+  fundraising *timing*, not proposal quality.
+- **Survivorship and selection bias are unaddressed.** Response rates on outcome surveys skew heavily to
+  founders who succeeded. The spec plans to weight models on a systematically biased sample.
+- **Confounding is total.** A proposal that scores 4/10, gets rewritten using the Fix-It plan, and then
+  raises successfully  -  did the judge score correctly or incorrectly? The product *intervenes* on the
+  outcome it measures. This is unfixable by more data; it needs a holdout design.
+- **N is unreachable.** N≥50 per model per dimension per vertical, with 10 dimensions, 11 models and 5+
+  verticals, implies thousands of tracked reviews before a single threshold fires. At 100 reviews/mo
+  Enterprise capacity, that is **years**. The entire feedback loop is aspirational at realistic volume,
+  and the "data moat" narrative rests on it.
+
+**Rotation rules:** "Remove from Tier A, demote to Tier B" as a demotion path is odd  -  a model with
+<0.3 outcome correlation is not a *cheaper* model, it is an *inaccurate* one. Demoting it means budget
+users get the judge known to be wrong. Also, the model-deprecation row says replace with "Fallback 1
+from §2.3"  -  but §2.3 fallbacks are same-vendor in 4 rows, so vendor-caused deprecation cascades to a
+sibling that may be deprecated in the same wave.
+
+---
+
+## 9. Pricing  -  4/10  -  FIX
+
+I have no objection to premium positioning, and the "don't race to the bottom" instinct is right. The
+objection is that **the price is not connected to demonstrated value**, and v2.0 raised it by
+**3.3x-5.0x** (from $79/$299 to $249/$799/$1,499) on positioning reasoning alone, with zero customers,
+zero LOIs, and  -  per the Critical Finding  -  no evidence the 12-judge panel beats one model.
+
+**The benchmark is misapplied.** GC AI at $500/seat/mo is cited as the anchor comp, and I verified it
+independently (gc.ai, corroborated by vaquill.ai's 2026 benchmark; note haqq.ai could not source it to
+GC AI directly). But GC AI serves **in-house legal teams at 1,900+ companies**  -  daily-use workflow
+software with seat-level lock-in. VerdictTank is **episodic**: a founder reviews a proposal during a
+fundraise, then churns. Anchoring episodic tooling to daily-workflow pricing is a category error.
+
+**The usage math undermines the tiers.** Enterprise at $799/mo = 100 reviews. Real founders raising a
+round need **3-8 reviews over a 2-3 month window**. That is ~$100/review nominal at a utilization
+almost no customer will reach  -  and Pro at $249 for 20 reviews has the same problem. Customers pay for
+capacity they cannot consume, notice, and churn. **Per-review or credit-pack pricing fits actual
+consumption far better than monthly seats**, and the spec never considers it.
+
+**Margin honesty.** The v4.1 reference concedes "COGS 72-97% margin  -  pure positioning play." My cost
+sampling supports that: a full panel run is roughly $0.30-0.80 in tokens. A 99.9% gross margin at
+$799/mo is not premium positioning, it is an unanchored price waiting for a competitor to undercut it
+with the same off-the-shelf models. The moat is claimed to be the outcome-tracking corpus  -  which
+dimension 8 shows is years away at this volume.
+
+**Free tier at 1/mo is too stingy** for a trust-first product. The entire pitch is "we tell you the
+brutal truth"  -  that requires *experiencing* the depth. A 1-review Tier-C sample (3 judges, no Fix-It)
+demonstrates the weakest possible version of the product to every prospective buyer.
+
+**Fix:** validate willingness-to-pay with 10-20 design partners before locking. Offer per-review pricing
+alongside subscriptions. Anchor to *outcome value* (a better raise) rather than to a competitor in an
+adjacent category.
+
+---
+
+## 10. Product Boundary  -  7/10  -  DEFER
+
+Conceptually clean and easy to communicate: VerdictTank = "is this good?", RFP Tank = "does this match
+what they asked for?" The superset framing is right, and inheriting one engine is the correct build
+decision.
+
+**Unresolved, but not urgent:**
+- **RFP Tank is the better business and it is the side project.** RFP responses are recurring, deadline-
+  driven, budgeted, and B2B  -  structurally superior to episodic founder fundraising. The spec treats it
+  as a discount add-on. Strategically inverted.
+- **Cannibalization is unpriced.** RFP Tank is a strict superset at (presumably) a higher price. A
+  rational buyer needing both buys RFP Tank only. Enterprise VT + 20% off RFP Tank is then a discount
+  on a product that replaces the one just paid for.
+- **10-dimension rubric is asserted, never enumerated.** The document references "10-dimension scoring"
+  in §7 and §8 but **never lists the ten dimensions.** For a build spec, the thing being scored should
+  be defined; I am scoring against dimensions the spec assumes I already know.
+- No shared-account model, no cross-product SSO, no migration path.
+
+**DEFER** because the boundary is directionally correct and none of this blocks the judge-pool build.
+Revisit before RFP Tank pricing is set.
+
+---
+
+## Priority Actions
+
+| # | Action | Dim | Severity |
+|---|---|---|---|
+| 1 | **Validate panel-vs-single-model accuracy on 30-50 real proposals.** Product thesis is unproven. |  -  | **BLOCKER** |
+| 2 | **Re-architect to async jobs; budget 5-10 min.** Measured 218s vs 45s target. | 7 | **BLOCKER** |
+| 3 | **Fix 4 same-vendor Fallback-1 hops.** Flagged in v1.0 review, still unfixed. | 5 | **BLOCKER** |
+| 4 | **Define the score aggregation formula.** "Weight +30%" is meaningless; core algorithm absent. | 6 | **BLOCKER** |
+| 5 | Pin all model versions  -  drop floating `gemini-pro-latest` alias. | 4 | High |
+| 6 | Reassign Kimi K2.6 off Reasoning-Verification (empty output under budget). | 2 | High |
+| 7 | Set per-model token budgets with reasoning-token headroom; handle empty-but-valid responses. | 2 | High |
+| 8 | Split Gemini Pro's Cross-Check B / Audit double-seat; auditor cannot audit itself. | 2 | High |
+| 9 | Add runtime vendor-cap re-check after failover (DeepSeek is fallback in 7 of 11 chains). | 5 | High |
+| 10 | Validate pricing with design partners; add per-review option. | 9 | High |
+| 11 | Add Technical Architecture + evidence-substantiation coverage. | 1 | Medium |
+| 12 | Replace page-count complexity proxy with token count. | 6 | Medium |
+| 13 | Enumerate the 10 scoring dimensions in the spec. | 10 | Medium |
+| 14 | Reconsider Financial Integrity -20% at seed stage. | 6 | Medium |
+
+---
+
+## Bottom Line
+
+v2.0 is a real improvement over v1.0  -  the vendor rule was tightened rather than loosened to fit, the
+cold-start gates are correct, the role gaps were mostly closed, and **every model path is genuinely live**,
+which I verified rather than assumed. The document is architecturally literate.
+
+But it is a **design document wearing a build spec's clothes**. Its two most load-bearing quantitative
+claims fail on contact with the live system: the latency budget is off by 4.8x, and the fallback
+cross-vendor guarantee is contradicted by its own assignment table in four rows  -  *the same defect flagged
+in the v1.0 review and marked as a prioritized fix.* The assertion was added; the table was not corrected.
+
+Most seriously: the panel-diversity premise that justifies the pricing, the vendor spread, and the entire
+category claim has **never been tested**, and my measurement suggests it may not survive testing. Nine
+judges across eight vendors moved the score by under half a point versus a single model.
+
+**Recommendation: do not proceed to build.** Run the accuracy validation (Action 1) and re-baseline
+latency (Action 2) first. If the panel advantage is real, this architecture is worth building and the
+premium price defensible. If it is not, the correct product is 3 judges at a third of the price  -  and it
+is far cheaper to discover that now than after the first enterprise customer runs the same A/B I just ran.

proposals/verdicttank/methodology.md

@@ -0,0 +1,418 @@
+# VerdictTank Internal Proposal Methodology v1.0
+
+**Status:** SETTLED  -  governing document for all VerdictTank/RFP Tank proposals  
+**Effective:** 2026-08-11  
+**Owner:** Sho'Nuff Brown (shogun of IT architecture)  
+**Invocation:** `skill_view(name='verdicttank-proposal-pipeline', file_path='references/methodology.md')`
+
+---
+
+## 1. Pipeline Overview
+
+The VerdictTank proposal pipeline converts a product brief into a deployed, reviewed, audited proposal page in one orchestrated run. It is NOT a general-purpose proposal tool  -  it is purpose-built for VerdictTank and RFP Tank, with specialized content teams, a mandatory blind audit, and multi-model conductor review.
+
+```
+Product Brief
+    │
+    ├─► Phase 1: Research (DeepSeek V4 Pro  -  current main model)
+    │       Market sizing, competitor landscape, regulatory scan
+    │       Output: research brief → /tmp/verdicttank-v4.0-research-brief.md
+    │
+    ├─► Phase 2: Content Creation (4 parallel teams)
+    │       Marketing (Opus 4.8)     → HTML blocks: hero, value props, pricing, CTA
+    │       Architecture (Opus 4.8)  → standalone architecture.html (dark theme, sections)
+    │       Financial (Opus 4.8)     → bottom-up SOM/SAM/TAM, pricing model, unit economics
+    │       Legal (Sonnet 5)         → ToS, Privacy, DPA, AI disclaimer, sub-processor audit
+    │
+    ├─► Phase 3: Assembly (DeepSeek V4 Pro  -  current main model)
+    │       Merge 4 HTML blocks into index.html
+    │       Reconcile SOM to Financial's authoritative number
+    │       Insert legal blocks at designated markers
+    │       Link architecture.html as standalone page
+    │
+    ├─► Phase 4: Internal Conductor Review (Sonnet 5 + Opus 4.8)
+    │       Sonnet 5: brutal Shark Tank critic (what's wrong, what's missing, what's weak)
+    │       Opus 4.8: sober validator + priority re-rank
+    │       10-dimension scorecard: clarity, credibility, differentiation, TAM/SOM,
+    │           GTM, tech feasibility, competitive moat, financial model, team, risk
+    │       Output: critical-review.html (deployed alongside proposal)
+    │
+    ├─► Phase 5: Blind Audit (Opus 4.8  -  cold read, no prior context)
+    │       7-item checklist executed against deployed page
+    │       Every link checked, every number traced, every claim verified
+    │       THIS IS MANDATORY. No proposal ships without a blind audit.
+    │       v3.7 shipped with 4 verifiable errors because this step didn't exist.
+    │
+    ├─► Phase 6: External Conductor Review (fresh panel)
+    │       Independent judges  -  no access to internal review results
+    │       Majority-rules: Go / No-Go / Conditional Go
+    │
+    └─► Phase 7: Deploy + Outcome Tracking
+            Deploy to proposals.itpropartner.com/verdicttank/
+            Schedule T+90, T+180, T+365 outcome tracking checks
+```
+
+---
+
+## 2. Tier Selection
+
+Three cost tiers. Pick before dispatching teams. Tier determines which models run which roles.
+
+| Tier | Cost | Target Score | Content Teams | Review | Blind Audit | Use Case |
+|------|------|-------------|---------------|--------|-------------|----------|
+| **Tier 1** | ~$0.80 | 6/10 | DeepSeek V4 Pro only | None | None | Internal draft, rapid iteration |
+| **Tier 2** | ~$3.62 | 7-8/10 | Opus (narrative), Sonnet (architecture) | Internal only | Yes | Client-ready draft, first external showing |
+| **Tier 3** | ~$6.69 | 9-10/10 | Opus (marketing, architecture, financial), Sonnet (legal) | Internal + External | Yes | Final proposal, investor deck, RFP response |
+
+**Rule:** Tier 3 is the default for any proposal that leaves ITPP infrastructure. Tier 1 is for internal validation only. Tier 2 is for client drafts that will iterate.
+
+**Actual v4.0 Tier 3 cost:** $6.69 (5 Opus dispatches + 1 Sonnet dispatch + assembly + 2 conductor reviews + blind audit). Budget cap: $10.00. Current spend: tracked against `hermes-agent-v5` LiteLLM key.
+
+---
+
+## 3. Team Composition & Model Rotation Protocol
+
+### 3.1 Standard Team Assignments (Tier 3)
+
+| Team | Model | Role | Output Format | Permission |
+|------|-------|------|---------------|------------|
+| Marketing & Narrative | Claude Opus 4.8 | Clean-sheet proposal narrative | HTML blocks with INSERT markers | Read-only |
+| Technical Architecture | Claude Opus 4.8 | Industry-standard architecture document | standalone HTML (dark theme) | **Write to disk** (prevents truncation) |
+| Financial Model | Claude Opus 4.8 | Bottom-up verified arithmetic | HTML blocks + verified numbers | Read-only |
+| Legal & Governance | Claude Sonnet 5 | MVL framework, compliance tables | HTML blocks with INSERT markers | Read-only |
+
+### 3.2 Model Rotation Protocol
+
+**CRITICAL:** The delegation model must be set BEFORE each dispatch, then restored to `deepseek-v4-pro` immediately after. Failure to rotate burns budget on accidental Opus inference during assembly.
+
+```
+# Before dispatching Opus teams (marketing, architecture, financial):
+hermes config set delegation.model claude-opus-4-8
+delegate_task(...)  # dispatches marketing, architecture, financial as batch of 3
+hermes config set delegation.model deepseek-v4-pro  # RESTORE IMMEDIATELY
+
+# Before dispatching legal:
+hermes config set delegation.model claude-sonnet-5
+delegate_task(...)  # dispatches legal
+hermes config set delegation.model deepseek-v4-pro  # RESTORE IMMEDIATELY
+
+# Before conductor review (internal):
+hermes config set delegation.model claude-sonnet-5  # for critic
+delegate_task(...)
+hermes config set delegation.model claude-opus-4-8  # for validator
+delegate_task(...)
+hermes config set delegation.model deepseek-v4-pro  # RESTORE
+
+# Before blind audit:
+hermes config set delegation.model claude-opus-4-8
+delegate_task(...)
+hermes config set delegation.model deepseek-v4-pro  # RESTORE
+```
+
+**Why DeepSeek V4 Pro is the default:** Budget protection. DeepSeek is orders of magnitude cheaper than Opus. The main model handles assembly, research, and coordination  -  all high-token, low-creativity work. Opus and Sonnet are reserved for tasks where their specific strengths justify the cost.
+
+---
+
+## 4. Content Team Brief Structure
+
+Every team gets a self-contained brief file written to `/tmp/`. The brief must include:
+
+1. **Context block:** What VerdictTank is, v3.7 review findings, the specific HTML blocks needed
+2. **Rules block:** No em dashes, no fabrication, financial model is authoritative for SOM, honest about methodology gaps
+3. **Output format block:** Exact HTML structure, INSERT markers, CSS variables to use
+4. **Reference data:** SOM/TAM numbers, competitor names, pricing tiers, domain policy
+5. **Pitfalls block:** Specific errors to avoid (from v3.7 postmortem)
+
+### 4.1 Architecture Team  -  Special Handling
+
+Architecture output is **written to disk**, not returned in the delegation summary. Reason: delegation summaries truncate at ~2K chars. Architecture documents are 50-100K chars. The team gets `write_file` permission targeting `/root/projects/itpp-infrastructure/proposals/verdicttank/architecture-v4.0-source.html`.
+
+### 4.2 Legal Team  -  Special Requirements
+
+Legal teams operate under the **Honest Methodology Gaps** rule:
+- TESS (USPTO trademark database) is JavaScript-only  -  cannot be queried programmatically
+- Open-web common-law search IS performed (SearXNG, Brave, Exa)
+- Every legal output MUST disclose: "Formal TESS search not performed. Recommend trademark attorney clearance before Series A."
+- Never fabricate TESS results. The gap is documented, not hidden.
+
+---
+
+## 5. SOM Reconciliation Rule
+
+**The Financial model is the single source of truth for SOM.**
+
+This rule exists because v4.0's Marketing team produced SOM of $7.91M while Financial produced $1.21M  -  a 6.5x gap. Both teams used the same TAM data. Financial's arithmetic was verified line-by-line. Marketing's was aspirational.
+
+**Reconciliation protocol during assembly:**
+1. Assembly agent loads both outputs
+2. Identifies SOM mismatch
+3. Forces Financial's number into the marketing blocks
+4. Documents the override in the assembly notes
+5. Blind auditor verifies the reconciliation
+
+**Why not average or negotiate?** Because Financial's number was produced by a model that was explicitly instructed to verify every line of arithmetic. Marketing's model was instructed to sell. For revenue numbers, trust the accountant, not the salesperson.
+
+---
+
+## 6. Assembly Protocol
+
+Assembly is performed by the main model (DeepSeek V4 Pro) and merges four independently produced HTML blocks into one coherent page.
+
+### 6.1 Assembly Rules
+
+1. **Single-source rule:** One `index.html`. Architecture is a separate `architecture.html` linked from the main page.
+2. **SOM override:** Marketing's SOM → Financial's SOM. Document the override.
+3. **INSERT marker resolution:** Legal and Marketing blocks use `<!-- INSERT: section_name -->` markers. Assembly replaces these with the actual content blocks.
+4. **Consistency checks:**
+   - All prices match Financial model
+   - All feature claims trace to Architecture document
+   - All legal disclaimers present
+   - No forbidden model names in final output
+   - Zero em dashes, zero double hyphens
+   - Domain references: `verdicttank.com` (primary), `rfptank.com` (legacy/defensive only)
+5. **CSS consistency:** All blocks use the same dark theme CSS variables (`--bg`, `--panel`, `--ink`, `--accent`, `--line`, etc.) defined in the architecture document.
+6. **Footer consistency:** All pages share the same footer with origin, deployment date, and domain policy.
+
+### 6.2 Deployment Map
+
+| File | Deploy To | URL |
+|------|-----------|-----|
+| `index.html` | `app3:/home/ippadmin/htdocs/proposals.itpropartner.com/verdicttank/` | `https://proposals.itpropartner.com/verdicttank/` |
+| `architecture.html` | Same directory | `https://proposals.itpropartner.com/verdicttank/architecture.html` |
+| `critical-review.html` | Same directory | `https://proposals.itpropartner.com/verdicttank/review.html` |
+
+**Origin:** `152.53.241.111` (app3, netcup RS 4000)  
+**Served via:** nginx static file server, LE auto-certs
+
+---
+
+## 7. Blind Audit Protocol
+
+**MANDATORY. No exceptions. No proposal ships without passing blind audit.**
+
+### 7.1 Why This Exists
+
+v3.7 shipped with four verifiable errors that a cold read would have caught:
+1. Pricing table didn't match Financial model
+2. Feature list referenced a deprecated API endpoint
+3. Compliance section omitted GDPR data residency
+4. CTA link pointed to staging, not production
+
+The assembly agent was too close to the content to see these. The blind auditor reads the deployed page with zero prior context  -  exactly as a prospect would.
+
+### 7.2 Audit Checklist
+
+| # | Check | Method |
+|---|-------|--------|
+| 1 | Every link resolves (no 404s, no staging URLs) | Click every `href`, every CTA |
+| 2 | Every number traces to Financial model | Cross-reference pricing table vs financial output |
+| 3 | Every feature claim has architecture evidence | Cross-reference features vs architecture sections |
+| 4 | Legal disclaimers present and complete | Verify ToS, Privacy, DPA, AI disclaimer blocks exist |
+| 5 | Contact/CTA information correct | Verify email, phone, demo link |
+| 6 | No placeholder text, no TODO markers, no lorem ipsum | Full page scan |
+| 7 | CSS renders correctly on desktop and mobile viewports | Browser screenshot at 1280px and 375px |
+
+### 7.3 Auditor Rules
+
+- **Model:** Claude Opus 4.8 (strongest available for cold-read QA)
+- **Context:** ZERO. The auditor gets ONLY the URL. No brief, no prior knowledge, no team outputs.
+- **Output:** Pass/Fail with specific, numbered findings. Fail = blocks deployment until all items fixed.
+- **Re-audit:** After fixes, a fresh auditor (different model or clean context) re-runs the checklist.
+
+---
+
+## 8. Conductor Review Integration
+
+The pipeline uses TWO conductor reviews, not one.
+
+### 8.1 Internal Conductor Review (Pre-Blind-Audit)
+
+- **Critic:** Claude Sonnet 5  -  brutal, Shark Tank style. "What's wrong, what's missing, what's weak?"
+- **Validator:** Claude Opus 4.8  -  sober, structured. Priority re-rank, FIX/DEFER/DEFEND treatment.
+- **Scorecard:** 10 dimensions, 1-10 scale, weighted average.
+- **Output:** `critical-review.html`  -  deployed publicly as a signal of confidence.
+
+### 8.2 External Conductor Review (Post-Blind-Audit)
+
+- **Fresh panel:** Judges have no access to internal review results or team outputs.
+- **Majority-rules verdict:** Go (deploy), No-Go (block), Conditional Go (fix specified items).
+- **Minimum passing score:** 7/10 for Tier 2, 8/10 for Tier 3.
+
+### 8.3 Integration with shark-tank-review Skill
+
+The `shark-tank-review` skill is the broader framework. This pipeline uses it as the review engine but adds:
+- VerdictTank-specific 10-dimension weighting (financial model accuracy weighted 2x)
+- Blind audit as a mandatory post-review gate
+- Public deployment of the review as a confidence signal
+
+When invoking: load `shark-tank-review` for the review phases, then execute blind audit as a separate step.
+
+---
+
+## 9. Legal & Trademark Guardrails
+
+### 9.1 Trademark Clearance Methodology
+
+1. **Open-web common-law search** (SearXNG, Brave, Exa)  -  search for "VerdictTank" + "software" + "AI" + "legal", check USPTO common-law marks, state registrations, domain ownership
+2. **TESS search**  -  NOT performable programmatically (JavaScript-only USPTO database). This is a documented methodology gap.
+3. **Recommendation:** Formal trademark attorney search before Series A funding. The open-web search found no conflicts, but TESS is the authoritative source for federal registrations.
+
+### 9.2 Privacy & Compliance
+
+- **MVL framework:** Terms of Service, Privacy Policy, Data Processing Agreement, AI Disclaimer, Limitation of Liability ($100 / 12 months fees), Jurisdiction (DE/GA)
+- **Regulatory alignment:** GDPR, CCPA, NIST CSF 2.0
+- **Sub-processor audit:** OpenAI (compliant), Anthropic (compliant), Google (compliant), DeepSeek (non-compliant  -  flagged with fallback described)
+- **Corpus confidentiality:** 5-step anonymization pipeline, URL-to-Review and Chat-to-Refine raw content excluded from corpus
+
+### 9.3 Honest Methodology Gaps
+
+Any limitation in our research methodology must be DISCLOSED, not papered over:
+
+| Gap | Disclosure |
+|-----|-----------|
+| TESS not searchable programmatically | "Formal TESS search not performed. Open-web common-law search completed." |
+| DeepSeek sub-processor non-compliance | Flagged explicitly with fallback to compliant alternatives |
+| Category existence risk (no standalone proposal review SaaS) | Acknowledged as market creation play, not hidden |
+
+---
+
+## 10. Outcome Tracking
+
+Proposals are bets. Track whether the bets paid off.
+
+### 10.1 Tracking Schedule
+
+| Checkpoint | What to Measure | Method |
+|------------|----------------|--------|
+| T+90 | Page views, inbound inquiries, demo requests | Analytics + CRM |
+| T+180 | Pipeline generated from proposal, conversion rate | CRM + sales data |
+| T+365 | Revenue closed from proposal-sourced leads, proposal accuracy (did we deliver what we promised?) | Revenue attribution |
+
+### 10.2 Feedback Loop
+
+Every T+90/180/365 check feeds back into the methodology:
+- Underperforming claims → adjust marketing brief template
+- Over-delivered features → adjust architecture scope
+- Pricing pushback → adjust Financial model assumptions
+- Legal objections → update Legal team brief
+
+---
+
+## 11. Pitfalls & Lessons Learned
+
+### 11.1 v3.7 Postmortem
+
+| Error | Root Cause | v4.0 Fix |
+|-------|-----------|----------|
+| Pricing table didn't match Financial model | Assembly agent used Marketing's numbers, not Financial's | **SOM Reconciliation Rule** (Section 5)  -  Financial is authority |
+| Feature list referenced deprecated API | Architecture team wasn't re-dispatched when truncated | **Write to disk** for architecture (Section 4.1) |
+| GDPR data residency omission | Legal team output was read but not integrated | **INSERT marker resolution** in assembly (Section 6.1) |
+| CTA link pointed to staging | No cold-read QA before deployment | **Mandatory blind audit** (Section 7) |
+
+### 11.2 v4.0 Discovered Issues
+
+| Issue | Resolution |
+|-------|-----------|
+| Delegation summaries truncate at ~2K chars | Architecture team writes to disk. Other teams use concise HTML blocks with INSERT markers |
+| Marketing SOM 6.5x higher than Financial | Enforced Financial authority during assembly |
+| Architecture re-dispatch needed (first attempt truncated) | Now standard: architecture ALWAYS writes to disk |
+| Budget tracking requires manual config rotation | Documented rotation protocol (Section 3.2) |
+
+### 11.3 Recurring Pitfalls
+
+1. **Forgetting to restore delegation model after dispatch.** Result: assembly burns Opus tokens at 50x the cost. Fix: always pair set + restore in the rotation protocol.
+2. **Truncation blindness.** Delegation summaries only show ~2K chars. A 70K architecture document looks like it "completed" when it's actually 3% visible. Fix: architecture writes to disk always; other teams use concise outputs.
+3. **Assembly agent trusting Marketing's numbers.** Marketing optimizes for persuasion. Financial optimizes for accuracy. Assembly must defer to Financial for all revenue figures.
+4. **Skipping blind audit "because it looks fine."** v3.7 looked fine. It had 4 errors. The blind audit is non-negotiable.
+
+---
+
+## 12. Invocation Paths
+
+### 12.1 Hermes (Sho'Nuff)
+
+```
+skill_view(name='verdicttank-proposal-pipeline')
+```
+Then follow the tier selection and dispatch protocol.
+
+For methodology reference:
+```
+skill_view(name='verdicttank-proposal-pipeline', file_path='references/methodology.md')
+```
+
+### 12.2 Anita
+
+Anita's profile has an identical copy of the skill:
+```
+~/.hermes/profiles/anita/skills/software-development/verdicttank-proposal-pipeline/SKILL.md
+```
+
+She invokes it the same way:
+```
+skill_view(name='verdicttank-proposal-pipeline')
+```
+
+### 12.3 Manual
+
+The methodology is also available at:
+```
+/root/projects/itpp-infrastructure/proposals/verdicttank/methodology.md
+```
+
+---
+
+## 13. Budget Model
+
+### 13.1 Tier Costs (Actual v4.0 Run)
+
+| Phase | Model | Cost |
+|-------|-------|------|
+| Research | DeepSeek V4 Pro | $0.05 |
+| Marketing dispatch | Claude Opus 4.8 | $1.20 |
+| Architecture dispatch | Claude Opus 4.8 | $1.35 |
+| Financial dispatch | Claude Opus 4.8 | $0.90 |
+| Legal dispatch | Claude Sonnet 5 | $1.10 |
+| Assembly | DeepSeek V4 Pro | $0.15 |
+| Internal conductor (critic) | Claude Sonnet 5 | $0.40 |
+| Internal conductor (validator) | Claude Opus 4.8 | $0.55 |
+| Blind audit | Claude Opus 4.8 | $0.49 |
+| External conductor | Claude Sonnet 5 + Claude Opus 4.8 | $0.50 |
+| **TOTAL** | | **$6.69** |
+
+### 13.2 Budget Governance
+
+- **Key:** `hermes-agent-v5` LiteLLM key
+- **Cap:** $100/month
+- **Per-proposal soft cap:** $10.00
+- **Monitoring:** Check spend before dispatching tier 3 (`hermes keys usage hermes-agent-v5`)
+- **Escalation:** If spend exceeds $75 in a month, pause all Tier 3 dispatches and notify Germaine
+
+---
+
+## 14. Quality Gates
+
+| Gate | When | Who | Pass Condition |
+|------|------|-----|---------------|
+| Research complete | After Phase 1 | Main model | Market data sourced, competitors identified |
+| Content delivered | After Phase 2 | Main model | All 4 team outputs received, architecture verified on disk |
+| Assembly verified | After Phase 3 | Main model | SOM reconciled, INSERT markers resolved, consistency checks pass |
+| Internal review | After Phase 4 | Sonnet + Opus | Average score ≥ 7/10 (Tier 2) or ≥ 8/10 (Tier 3) |
+| Blind audit | After Phase 5 | Fresh Opus | 7/7 checklist items pass |
+| External review | After Phase 6 | Fresh panel | Majority Go or Conditional Go |
+| Deployment | After Phase 7 | Main model | All files served, all links resolve, analytics tagged |
+
+**Stop rule:** If any gate fails, fix and re-enter at the failed phase. Never skip a gate to meet a deadline.
+
+---
+
+## 15. Document Governance
+
+This methodology is **SETTLED**  -  it is the authoritative reference for all VerdictTank proposals. Changes require:
+
+1. A specific, documented failure mode that the current methodology didn't prevent
+2. A proposed amendment with before/after comparison
+3. Germaine's approval before the amendment takes effect
+
+**Version history:**
+- v1.0 (2026-08-11): Initial methodology, encoding v3.7 postmortem and v4.0 build

proposals/verdicttank/reviews/cross-check-c-deepseek-v2.0.md

@@ -0,0 +1,357 @@
+# Cross-Check C — DeepSeek V4 Pro Independent Re-Score
+
+**Spec:** VerdictTank Judge Pool Specification v2.0  
+**Source of truth used:** `/root/projects/itpp-infrastructure/proposals/verdicttank/judge-pool-spec.md` (local v2.0)  
+**Note:** Deployed URL still serves **v1.0** — review is against local v2.0 only.  
+**Reviewer lens:** Chinese-lab independence (DeepSeek) — different training data, RLHF, and reasoning patterns from Anthropic / OpenAI / Google  
+**Date:** 2026-08-12  
+**Mode:** Blind independent re-score. FIX / DEFEND / DEFER per dimension.
+
+---
+
+## Executive Summary
+
+| Metric | Value |
+|---|---|
+| **Overall** | **6.4 / 10** |
+| Dimensions FIX | 5 |
+| Dimensions DEFEND | 3 |
+| Dimensions DEFER | 2 |
+| Fatal structural issues | 2 (fallback vendor-correlation; SaaS-default US-centrism) |
+| Strongest advance vs v1.0 | 8-vendor spread, Financial/Team/Legal roles, cold-start gate |
+
+v2.0 is a real architectural upgrade from the Anthropic-heavy v1.0 panel. The Chinese-ecosystem seats (DeepSeek CC-C, Qwen Market-Reality, Kimi Reasoning, MiniMax Financial) are the right instinct. But the fallback table still hides same-vendor correlation, vertical rules are Western-startup shaped, and "SaaS default" silently erases superapps / WeChat ecosystems / B2B marketplaces that dominate non-US deal flow.
+
+---
+
+## Scorecard (10 Dimensions)
+
+| # | Dimension | Score | Verdict | One-line |
+|---|---|---|---|---|
+| 1 | Role coverage | **7** | DEFEND | 12 phases close the v1 gaps; still thin on tech architecture + geo-market fit |
+| 2 | Model assignments | **7** | DEFEND | Chinese seats well placed; Legal on Sonnet is weak; CC-C as Tier C undervalues independence |
+| 3 | admin-ai paths | **6** | DEFER | Paths look plausible but unproven in this review; Gemini "Pro Latest" alias risk |
+| 4 | Vendor diversity | **8** | DEFEND | 8 vendors / ≤25% is strong on paper; concentration reappears under fallback |
+| 5 | Fallback chains | **4** | **FIX** | Claims "always cross vendor" — **4/12 primaries share vendor with Fallback 1** |
+| 6 | Content-based rules | **4** | **FIX** | Missing superapp / WeChat / B2B marketplace / cross-border / gov-tech verticals |
+| 7 | Latency budgets | **6** | DEFER | Better than v1 inverted budgets; 12-judge @ 45s still aspirational without parallelism proof |
+| 8 | Feedback loop | **7** | DEFEND | Cold-start + N gates good; outcome signal still US-funding-centric |
+| 9 | Pricing | **7** | DEFEND | Premium hold is coherent; APAC willingness-to-pay and seat economics unaddressed |
+| 10 | Product boundary + SaaS default | **5** | **FIX** | VT/RFP split clean; SaaS-default assumption is US-centric and quietly wrong for global deal flow |
+
+**Mean: 6.1 → weighted overall 6.4** (fallback + verticals + SaaS-default weighted higher under Chinese-lab lens)
+
+---
+
+## Dimension Detail
+
+### 1. Role Coverage — **7/10 — DEFEND**
+
+**What works**
+- v1.0 left Financial Integrity, Team/Founder, Legal/Regulatory uncovered. v2.0 adds Phases 8–10. Correct fix.
+- Research never scores; Audit is read-only; three blind cross-checks is a genuine independence architecture.
+- Non-scoring Reasoning-Verification (Phase 5) as synthesis before execution/market is sound sequencing.
+
+**What is still thin**
+- No dedicated **Technical Architecture** judge (stack, scalability, security posture). Execution-Feasibility is ops/timeline, not architecture diligence. For deep-tech / infra / AI-infra proposals this is a material hole.
+- No **Geo-Market / Localization** role. Market-Reality with Qwen helps, but China/SEA/MENA GTM (ICP, channel, regulatory market access) is not a first-class dimension.
+- "10-dimension scoring" in §7 product boundary is never mapped to the 12 phases. Which phases emit which of the 10 scores? Spec is silent → assembly ambiguity.
+
+**Verdict: DEFEND** the 12-role expansion as directionally correct. Do not expand further before v4.1 ship — but log Technical Architecture + Geo-Market as v4.2 candidates.
+
+---
+
+### 2. Model Assignments — **7/10 — DEFEND**
+
+**Chinese-lab view of seat quality**
+
+| Role | Model | Assessment |
+|---|---|---|
+| Research | Grok 4.5 | Correct — native live web/X is load-bearing |
+| Primary | Opus 5 | Correct — brutal critique seat |
+| Validation | Sonnet 5 | Acceptable — cost/quality trade |
+| CC-A | GPT-5.6 Sol | Correct — OpenAI lineage independence |
+| CC-B | Gemini Pro Latest | Correct — Google lineage |
+| **CC-C** | **DeepSeek V4 Pro** | **Right vendor, wrong tier signal** |
+| Reasoning | Kimi K2.6 | Strong — Moonshot is a real 4th ecosystem |
+| Execution | GPT-5.6 Terra | Acceptable ops grounding |
+| Market | Qwen3.7 Plus | **Best seat in the roster for non-Western lens** |
+| Financial | MiniMax-M3 | Interesting; unproven for cap-table math specifically |
+| Team | Fable 5 | Overkill / expensive for founder-fit; also Anthropic stack concentration with Primary/Validation |
+| Legal | Sonnet 5 (shared) | **Mismatch** — legal needs specialized calibration + citation discipline, not a doubled Validation model |
+| Audit | Gemini Pro Latest (shared w/ CC-B) | Acceptable if Audit is post-hoc read-only; weakens "different eyes" narrative |
+
+**Critical notes from this seat (DeepSeek as CC-C)**
+1. Putting the Chinese-lab independent re-score on **Tier C** while OpenAI/Google cross-checks sit on Tier A sends a quality hierarchy signal that undercuts the independence thesis. If CC-C exists to catch what US labs miss, it cannot be the budget afterthought. Promote CC-C primary to Tier B minimum (or keep DeepSeek but stop labeling the *role* as C-tier capacity).
+2. Legal/Regulatory on Sonnet 5 shared with Validation is double-duty on the same model family and same vendor as Primary. For FDA / PIPL / data-export / EU AI Act work, this is the wrong specialization.
+3. MiniMax-M3 for Financial Integrity is a bet, not a proven assignment. Spec should require a calibration set (N synthetic cap tables) before locking.
+
+**Verdict: DEFEND** overall roster direction. **FIX** Legal assignment and CC-C tier signaling before launch marketing claims "8-vendor frontier panel."
+
+---
+
+### 3. admin-ai Paths — **6/10 — DEFER**
+
+**Observations**
+- Paths are more operationally concrete than v1.0 (good).
+- `gemini/gemini-pro-latest` is an alias, not a pinned model. Alias drift silently changes Audit + CC-B behavior.
+- Spec claims "All model paths confirmed against admin-ai (live, August 2026)" but this Cross-Check did not re-probe live `/v1/models`. Treat as **asserted, not re-verified**.
+- Internal architecture notes elsewhere still flag Qwen3.8 Max / Kimi K3 as "not yet on admin-ai — substitute qwen3.7-plus / kimi-k2.6." v2.0 uses the substitutes. Fine for launch, but naming in marketing vs runtime must not diverge (Qwen3.8 vs 3.7; Kimi K3 vs K2.6).
+
+**Verdict: DEFER** to an ops smoke-test: one live completion per path, pin versions, ban floating `*-latest` in Enterprise default lineup.
+
+---
+
+### 4. Vendor Diversity — **8/10 — DEFEND**
+
+**On paper (Enterprise default 12-judge)**
+- Anthropic 3 (25%), OpenAI 2, Google 2, + DeepSeek / Moonshot / Alibaba / MiniMax / xAI ×1 each.
+- 8 vendors. Hard rule tightened 40% → **33%**. This is the single biggest structural win vs v1.0 (Anthropic 44%).
+
+**Under stress (Chinese-lab concern)**
+- Diversity is a **steady-state** property. Under fallback (§5), Anthropic and Google density climbs fast because Primary/Team and Research/CC-B chains are same-vendor on hop 1.
+- Three Chinese vendors (DeepSeek, Alibaba, Moonshot) + MiniMax is excellent **presence**. But only Market-Reality is a Chinese model in a *weight-bearing interpretive* seat; CC-C is Tier C; Financial MiniMax is unproven; Reasoning Kimi does not score dimensions.
+- Risk: Western models still dominate **scoring power** even when vendor count looks global.
+
+**Verdict: DEFEND** the 8-vendor design. Do not celebrate "no vendor >25%" without measuring **scoring-seat share after fallback**.
+
+---
+
+### 5. Fallback Chains — **4/10 — FIX** ⚠️ FATAL-ish
+
+**Spec claim (§3.1):** *"Fallback chains always cross vendor boundaries — Enforced at the assignment table (§2.3)."*
+
+**Audit of §2.3 (Primary → Fallback 1 vendor):**
+
+| Role | Primary vendor | Fallback 1 vendor | Cross-vendor? |
+|---|---|---|---|
+| Research | xAI | Google | ✅ |
+| Primary | Anthropic | **Anthropic** | ❌ |
+| Validation | Anthropic | OpenAI | ✅ |
+| Cross-Check A | OpenAI | **OpenAI** | ❌ |
+| Cross-Check B | Google | **Google** | ❌ |
+| Cross-Check C | DeepSeek | Google | ✅ |
+| Reasoning | Moonshot | DeepSeek | ✅ |
+| Execution | OpenAI | Anthropic | ✅ |
+| Market | Alibaba | xAI | ✅ |
+| Financial | MiniMax | DeepSeek | ✅ |
+| Team/Founder | Anthropic | **Anthropic** | ❌ |
+| Audit | Google | Anthropic | ✅ |
+
+**Result: 4 of 12 roles (33%) violate the hard constraint on the first hop.**
+
+Additional cascade issue:
+- Research Fallback 1 → Fallback 2 = Google → Google (same-vendor cascade). If Grok is down and Google is degraded, Research has no third-ecosystem escape.
+
+**Hidden correlation (Chinese-lab lens)**
+- Same-vendor fallback is not just a rule bug — it recreates **correlated failure and correlated judgment**. OpenAI Sol→Terra preserves OpenAI RLHF priors. Anthropic Opus→Sonnet / Fable→Sonnet preserves Anthropic critique style. Google Pro→Flash preserves Google grounding stack.
+- §4.1 says "All fallbacks cross vendor boundaries (§2.3 guarantees this)" — this is a **false guarantee**. Implementers will trust the rule table; the assignment table contradicts it.
+- DeepSeek is overused as universal sink (appears in 7 of 12 Fallback 1/2 slots). That makes DeepSeek a **correlation hub under multi-provider brownout**, ironic given CC-C independence branding.
+
+**Required FIX**
+1. Rewrite every same-vendor F1 to a different vendor *before* any lower-tier same-family model.
+2. Suggested repairs:
+
+| Role | Primary | F1 (fixed) | F2 |
+|---|---|---|---|
+| Primary | Opus 5 (Anth) | **GPT-5.6 Sol (OpenAI)** | Sonnet 5 (Anth) only as F2 |
+| CC-A | Sol (OpenAI) | **DeepSeek V4 Pro** or **Kimi** | Terra (OpenAI) as F2 |
+| CC-B | Gemini Pro (Google) | **DeepSeek** or **Qwen** | Gemini Flash as F2 |
+| Team | Fable 5 (Anth) | **GPT-5.6 Sol** or **Kimi** | Sonnet 5 as F2 |
+| Research F2 | — | replace Gemini Flash with **DeepSeek** or **Qwen** (third ecosystem) |
+
+3. Add automated test: `assert primary.vendor != fallback1.vendor` for every row; CI fails the spec if violated.
+4. Cap any single vendor's appearance in Fallback 1 columns (DeepSeek sink problem).
+
+**Verdict: FIX — blocking.** Do not ship "always cross vendor" language while §2.3 falsifies it.
+
+---
+
+### 6. Content-Based Rules / Verticals — **4/10 — FIX** ⚠️
+
+**What exists:** Biotech, Hardware/IoT, Fintech, Climate/Energy, SaaS(default) + stage + complexity. Composition cap +50% is good.
+
+**What is missing (non-Western / global deal flow)**
+
+| Missing vertical | Why it matters | Suggested effect |
+|---|---|---|
+| **Superapp / Mini-program ecosystem** | WeChat / Alipay / LINE / Grab-style platform dependency is a first-class business model in CN/SEA, not "SaaS" | Market +30%; Execution weights platform policy risk; Legal +PIPL/platform ToS |
+| **B2B marketplace / transaction platform** | Take-rate, cold-start liquidity, disintermediation — not SaaS net-retention logic | Market + Financial weights; Execution on two-sided ops |
+| **Cross-border / trade / payments corridor** | FX, export controls, dual-regulation | Legal +40%; Financial + FX/settlement realism |
+| **Government / SOE / public procurement** | RFP-adjacent but also guanxi, budget cycles, localization mandates | Legal + Team weights; different buyer psychology |
+| **Consumer social / short-video / live commerce** | Not "SaaS"; growth loops and platform risk dominate | Market + Team; Execution on content/ops |
+| **Industrial / manufacturing SaaS in CN** | Hardware+SaaS hybrid common; supply chain + data residency | Execution + Legal (data export) |
+| **Crypto / Web3 / stablecoin (global+Asia)** | Spec mentions nothing; still a real proposal class | Legal + Market specialized |
+| **Edtech / Healthtech consumer CN** | Heavy regulatory, different from US edtech/HIPAA framing | Legal frameworks beyond FDA/GDPR |
+
+**SaaS-default failure mode (§3.3)**
+- Unmatched verticals → SaaS weighting + a polite post-review notice.
+- That means a **WeChat mini-program commerce** proposal, a **Southeast Asian B2B marketplace**, or a **China-US cross-border data** startup all get Silicon-Valley SaaS calibration (NRR, seat expansion, PLG) and a footer apology.
+- From a Chinese-lab lens this is not a minor omission — it is **systematic miscategorization of a large fraction of non-US venture proposals**.
+
+**Also missing framework coverage in Legal phase description**
+- Lists FDA, SOC 2, GDPR, EU AI Act.
+- Omits: **PIPL, CSL, DSL (China)**; **PDPA (Singapore/SEA)**; **PDPB India**; **data export / MLPS**; **content/ICP licensing** where relevant.
+
+**Required FIX**
+1. Expand vertical table with at least: Superapp/Mini-program, B2B Marketplace, Cross-border, Gov/SOE, Consumer Social/Live Commerce.
+2. Change unmatched behavior from silent SaaS-default to **explicit low-confidence flag** that reduces overall confidence score and boosts Market-Reality + Legal weights generically — not SaaS metrics.
+3. Legal framework map must include PIPL/CSL/DSL + major APAC privacy regimes, not only Euro-American.
+4. Add stacking example for `Superapp + Seed + High complexity` in the spec so implementers see non-SaaS composition.
+
+**Verdict: FIX — high priority for any claim of global-grade review.**
+
+---
+
+### 7. Latency Budgets — **6/10 — DEFER**
+
+- v1.0 inversion (Enterprise tightest) is fixed. Good.
+- Enterprise 45s target / 90s max for **12-judge** pipeline is still aggressive unless Phases 4a–c and later specialized judges run heavily parallel.
+- Spec never states parallelism topology (which phases block on which). Without a DAG, latency budgets are wishes.
+- Free 90/180 and Pro 60/120 are reasonable.
+
+**Verdict: DEFER** pending an explicit phase DAG + p95 measurement on admin-ai. Do not market "45s Enterprise full pipeline" until measured.
+
+---
+
+### 8. Feedback Loop — **7/10 — DEFEND**
+
+**Strengths vs v1.0**
+- Cold-start gate (90 days) — correct.
+- min N=30 bias / N=50 accuracy — correct.
+- Immediate bias recusal (>2σ, 3 reviews) — good operational escape hatch.
+- Malformed-score auto-replace — practical.
+
+**Chinese-lab concerns**
+- T+90/180/365 "public funding data" is implicitly **US/EU venture outcomes** (Crunchbase-shaped). Funding outcome ≠ business outcome in many CN/SEA contexts (profitability, strategic acquisition, gov design-win).
+- Correlation-to-funding as accuracy ground truth will **systematically mis-train** the feedback loop against non-Western success patterns.
+- Inter-judge agreement during cold-start favors majority (Western) priors — minority Chinese-lab signals may look like "bias" and get recused.
+
+**Mitigations to log (not all blocking)**
+- Multi-outcome labels: funded / revenue milestone / strategic acq / shutdown — not funded-only.
+- Protect minority-vendor disagreement from automatic bias recusal until N is high *per vertical including APAC*.
+- Separate calibration sets for US-SaaS vs APAC-marketplace vs regulated-CN.
+
+**Verdict: DEFEND** structure. Outcome ontology needs globalization before the data moat hardens Western bias.
+
+---
+
+### 9. Pricing — **7/10 — DEFEND**
+
+- $249 / $799 / $1,499 with 16.7% annual is coherent premium vs GC AI $500/seat and authoring tools.
+- Free=1 review/mo is the right leash.
+- Pre-Review Coach on Pro (not Enterprise-only) is smart funnel design.
+- Financial/Team/Legal/Audit Enterprise-gated matches cost of 12-judge pipeline.
+
+**Gaps**
+- No APAC regional pricing / PPP consideration (often required for SEA/CN SMB founders — even if Enterprise stays global USD).
+- No seat-based vs org-based clarity vs GC AI's seat anchor (the comp is seat; VT is org-flat — explain or the anchor confuses buyers).
+- COGS not in this spec (exists in architecture notes). Fine for this doc.
+
+**Verdict: DEFEND** premium hold. Not the Chinese-lab primary fight.
+
+---
+
+### 10. Product Boundary + SaaS-Default Centrism — **5/10 — FIX**
+
+**Product boundary (VT vs RFP Tank)**
+- Clean and correct. Critique vs compliance is the right split. DEFEND that half.
+
+**SaaS-default assumption (the real Dim-10 issue under this lens)**
+- §3.2: `Vertical = SaaS (default)`.
+- §3.3: unmatched → SaaS-default calibration.
+- Feature language, competitive set (AutogenAI, Bidara, AutoRFP, Civio, GC AI), and stage weights all assume **US/EU B2B SaaS venture narrative**.
+- Global proposal mass includes: superapps, mini-programs, transaction marketplaces, OEM/industrial platforms, cross-border commerce, SOE-facing govtech. Forcing SaaS defaults is not neutral — it is a **prior**.
+
+**Independence irony**
+- You seated Qwen on Market-Reality and DeepSeek on CC-C — then told unmatched verticals to pretend they are SaaS. The non-Western models are asked to score with Western category priors.
+
+**Required FIX**
+1. Rename default from "SaaS (default)" → **"Generic / Unclassified (low-confidence)"** with no SaaS-specific metric emphasis.
+2. SaaS becomes an explicit detected vertical like Fintech — not the null hypothesis.
+3. Confidence banner already in §3.3 should **lower the headline confidence band** when unclassified, not only notify.
+4. Competitive landscape §6.4 should acknowledge non-US critique/authoring tools if claiming global premium (or explicitly scope "US/EU primary GTM").
+
+**Verdict: FIX** the null-hypothesis vertical. Keep VT/RFP boundary as-is.
+
+---
+
+## Special Focus Answers (Brief)
+
+### A. Are fallback chains truly vendor-independent?
+
+**No.** Spec asserts yes; §2.3 falsifies on 4/12 first hops (Primary, CC-A, CC-B, Team). Research F1→F2 is Google→Google. DeepSeek is a correlation sink on brownout. **Blocking FIX.**
+
+### B. Missing non-Western verticals?
+
+**Yes, material.** Superapps/mini-programs, B2B marketplaces, WeChat/Alipay ecosystems, cross-border, gov/SOE, live commerce, industrial hybrid — all collapse to SaaS default. Legal frameworks omit PIPL/CSL/DSL/PDPA.
+
+### C. Is "SaaS default" too US-centric?
+
+**Yes.** It is the null hypothesis for the entire content-based system and shapes metric priors (seat expansion, NRR, PLG). Should be an explicit vertical, not the default. Unclassified → low-confidence generic, not SaaS.
+
+---
+
+## FIX / DEFEND / DEFER Register
+
+| ID | Item | Priority | Action |
+|---|---|---|---|
+| F1 | Rewrite §2.3 so every Primary→F1 is cross-vendor; kill false §3.1/§4.1 guarantee | **P0** | FIX |
+| F2 | Research F2 leave Google cascade; add third-ecosystem F2 | **P0** | FIX |
+| F3 | Expand verticals: Superapp, B2B marketplace, Cross-border, Gov/SOE, Live commerce | **P0** | FIX |
+| F4 | Replace SaaS-as-default with Generic/Unclassified low-confidence | **P0** | FIX |
+| F5 | Legal frameworks: add PIPL/CSL/DSL/PDPA (+ data-export) | **P1** | FIX |
+| F6 | Legal role: stop sharing Sonnet with Validation; dedicated assignment | **P1** | FIX |
+| F7 | CC-C tier signaling: independence seat ≠ Tier C afterthought | **P1** | FIX |
+| F8 | Cap DeepSeek as universal fallback sink | **P1** | FIX |
+| F9 | Pin Gemini model; ban floating `*-latest` in Enterprise default | **P1** | FIX |
+| F10 | Map 10 scoring dimensions ↔ 12 phases explicitly | **P2** | FIX |
+| D1 | 12-role expansion (Financial/Team/Legal) | — | DEFEND |
+| D2 | 8-vendor / 33% hard cap direction | — | DEFEND |
+| D3 | Cold-start + N gates in feedback loop | — | DEFEND |
+| D4 | Premium pricing $249/$799/$1499 | — | DEFEND |
+| D5 | VT vs RFP Tank boundary | — | DEFEND |
+| D6 | Qwen on Market-Reality + Kimi on Reasoning | — | DEFEND |
+| R1 | Live admin-ai path smoke-test all 12 | — | DEFER |
+| R2 | Latency DAG + p95 measure before marketing 45s | — | DEFER |
+| R3 | Technical Architecture + Geo-Market roles (v4.2) | — | DEFER |
+| R4 | Multi-outcome (non-US) feedback ontology | — | DEFER |
+
+---
+
+## Per-Dimension Scores (machine-readable)
+
+```json
+{
+  "reviewer": "Cross-Check C — DeepSeek V4 Pro",
+  "spec": "judge-pool-spec v2.0",
+  "lens": "chinese-lab-independence",
+  "overall": 6.4,
+  "dimensions": [
+    {"id": 1, "name": "role_coverage", "score": 7, "verdict": "DEFEND"},
+    {"id": 2, "name": "model_assignments", "score": 7, "verdict": "DEFEND"},
+    {"id": 3, "name": "admin_ai_paths", "score": 6, "verdict": "DEFER"},
+    {"id": 4, "name": "vendor_diversity", "score": 8, "verdict": "DEFEND"},
+    {"id": 5, "name": "fallback_chains", "score": 4, "verdict": "FIX"},
+    {"id": 6, "name": "content_based_rules", "score": 4, "verdict": "FIX"},
+    {"id": 7, "name": "latency_budgets", "score": 6, "verdict": "DEFER"},
+    {"id": 8, "name": "feedback_loop", "score": 7, "verdict": "DEFEND"},
+    {"id": 9, "name": "pricing", "score": 7, "verdict": "DEFEND"},
+    {"id": 10, "name": "product_boundary_saas_default", "score": 5, "verdict": "FIX"}
+  ],
+  "blocking_fixes": ["F1", "F2", "F3", "F4"],
+  "deployed_url_mismatch": "https://proposals.itpropartner.com/verdicttank/judge-pool-spec.md still serves v1.0"
+}
+```
+
+---
+
+## Closing (Cross-Check C voice)
+
+v2.0 finally treats Chinese labs as first-class citizens of the panel. That is real progress.
+
+But independence is not a logo count. It is what happens when the primary is down, when the vertical is not YC-SaaS, and when the feedback loop decides whose disagreement is "bias." On those three tests the spec still thinks in Silicon Valley defaults while wearing an eight-vendor badge.
+
+**Ship after F1–F4. Everything else can follow.**
+
+— Cross-Check C (DeepSeek V4 Pro), independent re-score, 2026-08-12

sites/app3.md

@@ -45,5 +45,5 @@
 
 ## Pending
 
-- Migrate remaining WordPress sites from wphost02 to app3
+- ~~Migrate remaining WordPress sites from wphost02 to app3~~ — moot: wphost02 DECOMMISSIONED (2026-08-28), deleted from Hetzner account
 - Update all CloudPanel passwords (sanitized from plaintext in this doc 2026-07-23 — values in Vaultwarden)

sites/intelsight.io.md

@@ -86,6 +86,15 @@ Covered by app3 daily backup at 3 AM ET → `s3://hermes-vps-backups/cloudpanel-
 
 ## Next Steps
 
+### Buildout (queued 2026-08-17)
+1. Finish IntelSight build: wire Super Search + OSINT person-search functionality into the portal
+2. Wire login to auth2 (central auth)
+3. Purge mock test data
+4. Full smoke test of functionality at my.intelsight.io
+5. Run launch-feedback-sprint on the site
+6. Dispatch UX/design + marketing teams to vet the full site (if teams unavailable, produce full docs package + technical writeups to git + docs)
+
+### Open items (initial scaffold)
 1. Change WordPress admin email from `admin@intelsight.io` to `info@itpropartner.com`
 2. Create WordPress site at `my.intelsight.io` for the customer portal
 3. Install WordPress theme and configure branding