itpp-infrastructure/audit/phase-one/findings/docs-w.md

# Docs-W Findings: Documentation Audit (Phase One, Read-Only)

**Auditor:** Docs-W (documentation auditor)
**Scope:** docs.itpropartner.com (live MkDocs Material site on app3) and its stale duplicate; cross-referenced against neteng-a.md, neteng-b.md, sec-b.md, sys-a.md, sys-b.md, sys-c.md.
**Method:** Read-only SSH to app3 (152.53.241.111), filesystem inspection, HTML content extraction of every section/page and CHANGELOG. No live docs, nginx, or DNS were modified.
**Date:** 2026-08-13

---

## 1. Site Inventory

**Platform:** MkDocs Material, built and copied as static HTML into two htdocs trees on app3. No `.git` in either served tree, so the live docs root itself carries no version history or build provenance.

**Served copy:** `/home/docs/htdocs/docs.itpropartner.com` (owned by user `docs`, referenced by `/etc/nginx/sites-enabled/docs.itpropartner.com.conf`, most recent build timestamp ~2026-08-12 19:31 local).
**Unserved duplicate:** `/home/ippadmin/htdocs/docs.itpropartner.com` (root-owned, not referenced by any nginx vhost).

**Access control:** Since 2026-08-10, the entire site is gated by `docs-auth-validator.service` (JWT + email allowlist against auth2/Stack Auth). Before that date the site was open access. See section 3 (say-do gaps) for the implication.

### 1.1 Top-level sections (12), status, and last-update evidence (from each section's CHANGELOG)

| Section | Claims to document | Last CHANGELOG entry | Age vs audit date (2026-08-13) | Status |
|---|---|---|---|---|
| ITPP Infrastructure | Server inventory, DNS, architecture, key inventory, model routing, cost controls, ops portal, backup-restore, legal, project log | 2026-08-10 (Docs Auth Gate) | 3 days | Current |
| ITPP Standards | Documentation templates, CI (docs-check, docs-publish) | 2026-08-09 (Initial) | 4 days | Current |
| TransitPin | White-label transportation portal, dispatch, driver PWA | 2026-08-09 (migrate to Git); content dated 2026-07-29 | 4 days (changelog) / 15 days (content) | Current (changelog) |
| VerdictTank (+ Architecture) | Product review/validation platform, pricing, pipeline review | 2026-08-10 (v3 proposal + architecture) | 3 days | Current |
| HomeLab | Home lab infrastructure automation | "2026-07" (state snapshot 2026-07-21) | ~3+ weeks | Stale |
| Scripts | Operations/automation script catalog | 2026-07-21 | 3 weeks | Stale |
| FleetTracker360 | GPS fleet tracking platform | 2026-07-16 | 4 weeks | Stale |
| LaunchCheck | Startup validation product | 2026-07-25 (Project Inception, single entry) | 3 weeks, never updated since inception | Stale |
| Shark Game | Shark Attack Fantasy League | 2026-07-10 | 5 weeks | Stale |
| Apex Track | Track event management | 2026-07-10 ("Created project repository," single entry) | 5 weeks, stub only | Stale |
| BoxPilot | Logistics operations platform | 2026-07-10 ("Created project repository," single entry) | 5 weeks, stub only | Stale |
| OSINT Tool | OSINT people search / skip tracing | 2026-07-10 (Documentation migration) | 5 weeks | Stale |

**Section count: 12. Current: 4. Stale: 8.**

### 1.2 Notable sub-pages

- `itpp-infrastructure/`: Overview, Docs Auth Gate, Key Inventory, Model Chain, Cost Control Rollout (2026-07-24), Super Search CF Bypass (2026-07-21), app2 Caddyfile Audit (2026-07-21), Backup-Restore (+Architecture), Ops Portal (+Architecture), **Legal (empty page, no content rendered)**, Project Log, Projects Master README.
- `verdicttank/`: Overview, Architecture, plus markdown source files for v3.5/v3.6 architecture and a legal-framework document.
- `launchcheck/`: Competitive Analysis, Business Proposal (in addition to the section root).

---

## 2. Say-Do Gap Audit (cross-referenced against neteng-a/b, sec-b, sys-a/b/c)

| # | Doc claim | Live reality (per findings file) | Severity |
|---|---|---|---|
| 1 | `key-inventory` states secrets were "Sanitized... plaintext secrets replaced with storage references" (2026-07-23) | sys-a F-2: plaintext JWT_SECRET/DEEPSEEK_API_KEY/ADMIN_AI_KEY hardcoded in world-readable systemd units on Core (rally, seemytrip) and app1 (giftaroast). sys-b C2: `/root/.hermes/.env` on app1-bu (mode 644) holds ~20 plaintext secrets including root passwords for app1/app2/app3. sys-b C3: MySQL root password hardcoded in plaintext in two app3 scripts, one world-readable (775). | **Critical** - the one doc whose entire purpose is to assure the org that secrets are handled safely is contradicted by the live estate in at least three independent locations. |
| 2 | `ops-portal` CHANGELOG (2026-07-20) publishes a line reading "Admin credentials: [old username]/[old password] -> ippadmin (password moved to Vaultwarden)" | The docs site was open access (no auth gate) until 2026-08-10 per the site's own `docs-auth-gate` doc. This means a real historical admin credential sat in plaintext on an unauthenticated public web page for roughly three weeks. | **Critical** - a credential exposure inside the documentation itself, not just the infrastructure. |
| 3 | `docs-auth-gate` describes a JWT+allowlist access-control layer protecting the whole site, implying documentation (including key-inventory and the credential above) is now access-controlled | None of the six live-verified findings files enumerate or test `docs-auth-validator.service`/port 8099 on app3. sec-b's MFA/hardening coverage table (11 admin surfaces reviewed) does not include it. Its actual enforcement is undocumented outside its own self-description. | High - an access control the org depends on for a page containing credential-adjacent content has never been independently verified by the security or network auditors. |
| 4 | `projects-master-readme` lists Apex Track and BoxPilot as **"(PLANNED)"** | Both already have fully generated dedicated MkDocs sections (nav entries, CHANGELOG, "Created project repository and directory structure" stub content) on the same site. | Medium - internal self-contradiction on project status, visible on two pages of the same docs tree, no external cross-reference needed. |
| 5 | `itpp-infrastructure/legal` is a live nav entry titled "Legal" | Page renders with no content at all. | Medium - a documented, linked page with nothing behind it; reads as either an abandoned stub or a missed publish step. |
| 6 | `homelab` documents host inventory, DNS chain, and Docker service catalog for the separate home-lab network as if it were part of the audited estate | HomeLab (vm-host-01/02, QNAP, MikroTik home router) is **outside the scope** of neteng-a/b, sec-b, sys-a/b/c, which cover only Core/app1/app2/app3/app1-bu/wphost02. There is no live-verified findings coverage to check this section against at all. | Medium - a documented area of the estate with zero Phase One audit visibility; a structural blind spot rather than a specific factual error. |
| 7 | `model-chain` (2026-07-24) documents the AI model fallback chain and the admin-ai virtual key's daily budget cap, implying this is a governed, protected control | sys-a F-1: the LiteLLM Postgres database that stores this exact routing/budget/key configuration is **never backed up** (dump targets a nonexistent database name and fails silently every night). The governance the doc describes has zero disaster-recovery coverage, a fact the doc itself does not mention. | High - doc describes a control without disclosing that its backing store is unprotected. |
| 8 | `app2-caddyfile-audit-2026-07-21` documents a "final Caddyfile (validated)" for app2 routing UNMS, Gitea, UniFi, dns1, FleetTracker360 through Caddy | neteng-a APP2-1 through APP2-5 show UniFi, UNMS, Gitea SSH, Technitium DNS, and several other app2 services are **also** reachable directly via Docker's UFW bypass, i.e. the "validated" Caddy-only routing picture in the doc is incomplete: the real exposure includes direct Docker-published ports the doc never mentions. | High - the doc documents the intended path but is silent on (and therefore implicitly contradicts) the actual public exposure discovered live. |
| 9 | `backup-restore` doc describes a 30-day retention snapshot system and a documented restore API for 9 WordPress domains | sys-c's live restore-test audit found **zero** restore tests have ever been run against this specific mechanism (only Gitea and Vaultwarden have been restore-tested estate-wide), and sys-c SYSC-10 notes this snapshot layer is local-disk-only with no offsite copy - a single point of failure the doc does not disclose. | Medium - doc presents the system as complete/operational; live audit shows "backed up" without "restore-verified," which is exactly the gap Sys-C's engagement rule was written to catch. |

**Top 5 say-do gaps (for summary):** #1, #2, #3, #7, #8 above - ranked by blast radius and the fact that each represents documentation actively asserting a safety property (secrets sanitized, admin credential rotated, access gated, budget governed, routing validated) that the independently-verified infrastructure findings show is false, incomplete, or unverifiable.

---

## 3. Split-Brain: Two Copies of docs.itpropartner.com

**Finding:** Two nearly-identical full copies of the docs site exist on app3:

- `/home/docs/htdocs/docs.itpropartner.com` - owned by `docs`, referenced by the live nginx vhost, most recent build ~2026-08-12 19:31 local. **This is the authoritative, served copy.**
- `/home/ippadmin/htdocs/docs.itpropartner.com` - owned by `root`, **not referenced by any nginx config**, most recent build ~2026-08-10 (roughly 29 hours older than the served copy).

**Verification performed:** file-list diff between the two trees returns zero differences (93 files each, identical filenames/paths), and `index.html` content is byte-identical. The only measurable difference is per-file modification time, consistently ~29 hours older on the `ippadmin` copy across sampled files. This confirms both are build outputs of the same MkDocs source, generated by the same pipeline at two different times, not two diverging content sources.

**Root cause (most likely, not confirmed via any log evidence during this read-only audit):** an early build/deploy of the docs site landed in the default `ippadmin` home path (the operator's own home directory, `root`-owned), and a later correction properly deployed to a dedicated `docs` system user matching the nginx vhost root. The stale copy was never cleaned up.

**Risk:**
- No `.git` exists in either tree, so there is no audit trail proving which copy is "true" beyond nginx's own vhost pointer and file mtimes; a future config regeneration (this is a CloudPanel-managed host) or an operator mistake pointing the vhost root back at `/home/ippadmin/...` would silently roll the live site back ~29 hours, re-serving stale content (e.g., pre-dating the Aug 10 auth-gate documentation and VerdictTank v3 updates) with no visible error.
- The stale copy is `root`-owned while every other operational convention on this host uses a dedicated service user; this is itself a hygiene deviation worth correcting regardless of the duplication.
- There is currently no single documented deploy path (git repo -> build -> publish target) for this site, so a repeat of this duplication is only prevented by operator memory.

**Recommendation (Phase Two):** delete or archive `/home/ippadmin/htdocs/docs.itpropartner.com`, and formalize the build/publish pipeline (ITPP Standards already defines a `docs-publish` Gitea Actions workflow template; wire the live docs site to that pipeline with the `docs` htdocs path as its sole target) so there is exactly one deploy destination, git-tracked, going forward.

---

## 4. Audience Delineation: Internal vs Client vs Product/Dev

**Current state: no delineation exists.** All 12 sections live under one flat, undifferentiated MkDocs nav, gated by a single email allowlist behind one auth layer. Concretely mixed on the same site, one click apart:

- **Internal-only, high-sensitivity operational material:** Key Inventory (SSH key fingerprints/locations), Model Chain (LiteLLM virtual key hash, budget caps), Cost Control Rollout, app2 Caddyfile audit, Ops Portal / Backup-Restore internals (including the exposed historical credential noted in gap #2 above).
- **Internal engineering process docs:** ITPP Standards (CI/templates), Scripts catalog, HomeLab.
- **Product/dev documentation for ITPP's own ventures:** VerdictTank, TransitPin, FleetTracker360, Apex Track, BoxPilot, OSINT Tool, LaunchCheck, Shark Game - these describe products being built for eventual external users or customers, not ITPP's own infrastructure.
- **Client-facing:** none currently exists as a distinct, intentionally-scoped audience. Nothing in the current tree is written for or safe to hand to an actual paying customer of VerdictTank, FleetTracker360, or TransitPin - yet the auth gate's allowlist (`g@germainebrown.com`, `info@itpropartner.com`) suggests the site is intended for internal ITPP staff only, which conflicts with product docs plausibly needing outside readers eventually (contractors, investors, or customers).

**Assessment:** the site currently serves one audience (internal ITPP operators) under one login, with no mechanism to safely expose a subset of content (e.g., product architecture for a contractor, or end-user help docs for a VerdictTank customer) without also exposing Key Inventory, Model Chain, and Ops Portal internals. This is a structural risk, not just an organizational tidiness issue: the moment anyone outside the current two-person allowlist needs access to any single product doc, the only lever available is "grant them access to everything," including SSH key inventories and credential-adjacent operational docs.

**Recommendation:** split into three distinct trust zones (detailed in the reorg tree below) before granting any doc access to anyone outside the current internal allowlist.

---

## 5. Proposed Documentation Reorg Tree (Deliverable A.6)

Plain-English rationale first, then the tree.

**Rationale:**
1. **Separate the credential-adjacent operational core from everything else.** Key Inventory, Model Chain, Cost Control, Ops Portal/Backup-Restore internals, and the app2 Caddyfile audit should live in a zone that is never extended to anyone outside the current internal allowlist, regardless of what else changes.
2. **Separate product/dev docs from internal ops docs**, because product docs (VerdictTank, TransitPin, etc.) have a plausible future need for a wider audience (contractors, eventual customers) that internal ops docs never will.
3. **Reserve an explicit, currently-empty client-facing zone** rather than pretending the current site serves that purpose. If ITPP wants client-facing documentation (e.g., a VerdictTank user guide), it should be a deliberately separate publish target (different subdomain or path with its own, lighter-weight auth model), not a corner of the internal-only site.
4. **Retire dead stubs rather than let them accumulate.** Apex Track, BoxPilot, and (arguably) Shark Game and OSINT Tool have single-entry, 5-week-stale changelogs with no real content beyond "created project repository." Continuing to display them as live nav entries between actively-maintained sections misrepresents the site's currency.
5. **Fix the self-contradiction between `projects-master-readme` and the sections it describes** by making project status ("PLANNED" / "IN DEVELOPMENT" / "LIVE") a single generated field rather than manually duplicated free text in two places.

```
docs.itpropartner.com/
├── internal/ [existing allowlist gate stays here, unchanged scope]
│ ├── infrastructure/ (from itpp-infrastructure, minus product-adjacent items below)
│ │ ├── overview
│ │ ├── key-inventory <- flag: rewrite "sanitized" claim or actually remediate first (gap #1)
│ │ ├── model-chain
│ │ ├── cost-control-rollout-2026-07-24
│ │ ├── app2-caddyfile-audit-2026-07-21
│ │ ├── super-search-cf-bypass
│ │ ├── ops-portal (+ architecture)
│ │ ├── backup-restore (+ architecture)
│ │ ├── docs-auth-gate
│ │ ├── project-log
│ │ └── audit/ <- NEW: link Phase One (and future Phase Two) findings for internal transparency
│ ├── standards/ (itpp-standards, unchanged)
│ ├── homelab/ (unchanged; flag as "out of Phase One audit scope" until a homelab-specific audit exists)
│ └── scripts/ (unchanged)
│
├── products/ [same or a separate, slightly wider internal+contractor gate]
│ ├── verdicttank/ (+ architecture)
│ ├── transitpin/
│ ├── fleettracker360/
│ ├── apex-track/ <- ARCHIVE unless real content is added; currently a dead stub
│ ├── boxpilot/ <- ARCHIVE unless real content is added; currently a dead stub
│ ├── osint-tool/
│ ├── launchcheck/ (+ competitive-analysis, business-proposal)
│ └── shark-game/ <- review: 5-week-stale, confirm still active before keeping live
│
├── client/ [NEW, does not exist today - separate publish target/subdomain,
│ its own lightweight auth or fully public, populated only with
│ content explicitly written for external readers]
│ └── (empty until ITPP decides which product needs a customer-facing doc set)
│
└── legal/ [NEW, single location - currently a dangling empty page under
 itpp-infrastructure; either populate with real legal/compliance
 content or remove the nav entry entirely]
```

**What moves where:**
- `itpp-infrastructure/*` -> `internal/infrastructure/*` (unchanged content, new path only).
- `itpp-standards`, `homelab`, `scripts` -> `internal/*` (unchanged).
- `verdicttank`, `transitpin`, `fleettracker360`, `apex-track`, `boxpilot`, `osint-tool`, `launchcheck`, `shark-game` -> `products/*` (unchanged content, new path, plus an explicit staleness review for the four dead/near-dead stubs).
- `itpp-infrastructure/legal` -> either populated and moved to a top-level `legal/` or removed.
- `itpp-infrastructure/project-log` and `projects-master-readme` -> reconcile into a single generated project-status view under `internal/infrastructure/project-log`, sourced from each project's own README/CHANGELOG status field rather than hand-maintained twice.
- New `internal/infrastructure/audit/` -> add Phase One (and future Phase Two) findings summaries so the org has one place that tracks "what the docs say" next to "what the audits found."
- `client/` -> created empty; populated only when/if ITPP commits to publishing customer-facing docs for a specific product, with its own auth model decided at that time.

**What gets archived (not deleted, moved to an `archive/` prefix or removed from nav):**
- `apex-track` and `boxpilot`: both are single-entry "created project repository" stubs, 5 weeks stale, indistinguishable from placeholders. Either bring them current or pull them from the live nav so the site does not imply active documentation where none exists.
- The stale `/home/ippadmin/htdocs/docs.itpropartner.com` duplicate (not a doc section, but the entire stale build) should be deleted per section 3.

---

## Summary of Evidence Sources

- Live site content: extracted via SSH from `/home/docs/htdocs/docs.itpropartner.com` on app3 (152.53.241.111), CHANGELOG and index pages per section.
- Split-brain comparison: `find`/`diff`/`stat` against both `/home/docs/htdocs/...` and `/home/ippadmin/htdocs/...` on app3.
- Cross-reference findings: neteng-a.md, neteng-b.md, sec-b.md, sys-a.md, sys-b.md, sys-c.md (all read in full for this audit).
- No configuration, DNS, nginx, or docs content was modified. All recommendations are deferred to Phase Two per the audit brief's absolute rules.