itpp-infrastructure/audit/phase-one/findings/docs-w.md

Docs-W Findings: Documentation Audit (Phase One, Read-Only)

Auditor: Docs-W (documentation auditor) Scope: docs.itpropartner.com (live MkDocs Material site on app3) and its stale duplicate; cross-referenced against neteng-a.md, neteng-b.md, sec-b.md, sys-a.md, sys-b.md, sys-c.md. Method: Read-only SSH to app3 (152.53.241.111), filesystem inspection, HTML content extraction of every section/page and CHANGELOG. No live docs, nginx, or DNS were modified. Date: 2026-08-13

---

1. Site Inventory

Platform: MkDocs Material, built and copied as static HTML into two htdocs trees on app3. No .git in either served tree, so the live docs root itself carries no version history or build provenance.

Served copy: /home/docs/htdocs/docs.itpropartner.com (owned by user docs, referenced by /etc/nginx/sites-enabled/docs.itpropartner.com.conf, most recent build timestamp ~2026-08-12 19:31 local). Unserved duplicate: /home/ippadmin/htdocs/docs.itpropartner.com (root-owned, not referenced by any nginx vhost).

Access control: Since 2026-08-10, the entire site is gated by docs-auth-validator.service (JWT + email allowlist against auth2/Stack Auth). Before that date the site was open access. See section 3 (say-do gaps) for the implication.

1.1 Top-level sections (12), status, and last-update evidence (from each section's CHANGELOG)

Section	Claims to document	Last CHANGELOG entry	Age vs audit date (2026-08-13)	Status
ITPP Infrastructure	Server inventory, DNS, architecture, key inventory, model routing, cost controls, ops portal, backup-restore, legal, project log	2026-08-10 (Docs Auth Gate)	3 days	Current
ITPP Standards	Documentation templates, CI (docs-check, docs-publish)	2026-08-09 (Initial)	4 days	Current
TransitPin	White-label transportation portal, dispatch, driver PWA	2026-08-09 (migrate to Git); content dated 2026-07-29	4 days (changelog) / 15 days (content)	Current (changelog)
VerdictTank (+ Architecture)	Product review/validation platform, pricing, pipeline review	2026-08-10 (v3 proposal + architecture)	3 days	Current
HomeLab	Home lab infrastructure automation	"2026-07" (state snapshot 2026-07-21)	~3+ weeks	Stale
Scripts	Operations/automation script catalog	2026-07-21	3 weeks	Stale
FleetTracker360	GPS fleet tracking platform	2026-07-16	4 weeks	Stale
LaunchCheck	Startup validation product	2026-07-25 (Project Inception, single entry)	3 weeks, never updated since inception	Stale
Shark Game	Shark Attack Fantasy League	2026-07-10	5 weeks	Stale
Apex Track	Track event management	2026-07-10 ("Created project repository," single entry)	5 weeks, stub only	Stale
BoxPilot	Logistics operations platform	2026-07-10 ("Created project repository," single entry)	5 weeks, stub only	Stale
OSINT Tool	OSINT people search / skip tracing	2026-07-10 (Documentation migration)	5 weeks	Stale

Section count: 12. Current: 4. Stale: 8.

1.2 Notable sub-pages

• itpp-infrastructure/: Overview, Docs Auth Gate, Key Inventory, Model Chain, Cost Control Rollout (2026-07-24), Super Search CF Bypass (2026-07-21), app2 Caddyfile Audit (2026-07-21), Backup-Restore (+Architecture), Ops Portal (+Architecture), Legal (empty page, no content rendered), Project Log, Projects Master README.
• verdicttank/: Overview, Architecture, plus markdown source files for v3.5/v3.6 architecture and a legal-framework document.
• launchcheck/: Competitive Analysis, Business Proposal (in addition to the section root).

---

2. Say-Do Gap Audit (cross-referenced against neteng-a/b, sec-b, sys-a/b/c)

#	Doc claim	Live reality (per findings file)	Severity
1	key-inventory states secrets were "Sanitized... plaintext secrets replaced with storage references" (2026-07-23)	sys-a F-2: plaintext JWT_SECRET/DEEPSEEK_API_KEY/ADMIN_AI_KEY hardcoded in world-readable systemd units on Core (rally, seemytrip) and app1 (giftaroast). sys-b C2: /root/.hermes/.env on app1-bu (mode 644) holds ~20 plaintext secrets including root passwords for app1/app2/app3. sys-b C3: MySQL root password hardcoded in plaintext in two app3 scripts, one world-readable (775).	Critical - the one doc whose entire purpose is to assure the org that secrets are handled safely is contradicted by the live estate in at least three independent locations.
2	ops-portal CHANGELOG (2026-07-20) publishes a line reading "Admin credentials: [old username]/[old password] -> ippadmin (password moved to Vaultwarden)"	The docs site was open access (no auth gate) until 2026-08-10 per the site's own docs-auth-gate doc. This means a real historical admin credential sat in plaintext on an unauthenticated public web page for roughly three weeks.	Critical - a credential exposure inside the documentation itself, not just the infrastructure.
3	docs-auth-gate describes a JWT+allowlist access-control layer protecting the whole site, implying documentation (including key-inventory and the credential above) is now access-controlled	None of the six live-verified findings files enumerate or test docs-auth-validator.service/port 8099 on app3. sec-b's MFA/hardening coverage table (11 admin surfaces reviewed) does not include it. Its actual enforcement is undocumented outside its own self-description.	High - an access control the org depends on for a page containing credential-adjacent content has never been independently verified by the security or network auditors.
4	projects-master-readme lists Apex Track and BoxPilot as "(PLANNED)"	Both already have fully generated dedicated MkDocs sections (nav entries, CHANGELOG, "Created project repository and directory structure" stub content) on the same site.	Medium - internal self-contradiction on project status, visible on two pages of the same docs tree, no external cross-reference needed.
5	itpp-infrastructure/legal is a live nav entry titled "Legal"	Page renders with no content at all.	Medium - a documented, linked page with nothing behind it; reads as either an abandoned stub or a missed publish step.
6	homelab documents host inventory, DNS chain, and Docker service catalog for the separate home-lab network as if it were part of the audited estate	HomeLab (vm-host-01/02, QNAP, MikroTik home router) is outside the scope of neteng-a/b, sec-b, sys-a/b/c, which cover only Core/app1/app2/app3/app1-bu/wphost02. There is no live-verified findings coverage to check this section against at all.	Medium - a documented area of the estate with zero Phase One audit visibility; a structural blind spot rather than a specific factual error.
7	model-chain (2026-07-24) documents the AI model fallback chain and the admin-ai virtual key's daily budget cap, implying this is a governed, protected control	sys-a F-1: the LiteLLM Postgres database that stores this exact routing/budget/key configuration is never backed up (dump targets a nonexistent database name and fails silently every night). The governance the doc describes has zero disaster-recovery coverage, a fact the doc itself does not mention.	High - doc describes a control without disclosing that its backing store is unprotected.
8	app2-caddyfile-audit-2026-07-21 documents a "final Caddyfile (validated)" for app2 routing UNMS, Gitea, UniFi, dns1, FleetTracker360 through Caddy	neteng-a APP2-1 through APP2-5 show UniFi, UNMS, Gitea SSH, Technitium DNS, and several other app2 services are also reachable directly via Docker's UFW bypass, i.e. the "validated" Caddy-only routing picture in the doc is incomplete: the real exposure includes direct Docker-published ports the doc never mentions.	High - the doc documents the intended path but is silent on (and therefore implicitly contradicts) the actual public exposure discovered live.
9	backup-restore doc describes a 30-day retention snapshot system and a documented restore API for 9 WordPress domains	sys-c's live restore-test audit found zero restore tests have ever been run against this specific mechanism (only Gitea and Vaultwarden have been restore-tested estate-wide), and sys-c SYSC-10 notes this snapshot layer is local-disk-only with no offsite copy - a single point of failure the doc does not disclose.	Medium - doc presents the system as complete/operational; live audit shows "backed up" without "restore-verified," which is exactly the gap Sys-C's engagement rule was written to catch.

Top 5 say-do gaps (for summary): #1, #2, #3, #7, #8 above - ranked by blast radius and the fact that each represents documentation actively asserting a safety property (secrets sanitized, admin credential rotated, access gated, budget governed, routing validated) that the independently-verified infrastructure findings show is false, incomplete, or unverifiable.

---

3. Split-Brain: Two Copies of docs.itpropartner.com

Finding: Two nearly-identical full copies of the docs site exist on app3:

• /home/docs/htdocs/docs.itpropartner.com - owned by docs, referenced by the live nginx vhost, most recent build ~2026-08-12 19:31 local. This is the authoritative, served copy.
• /home/ippadmin/htdocs/docs.itpropartner.com - owned by root, not referenced by any nginx config, most recent build ~2026-08-10 (roughly 29 hours older than the served copy).

Verification performed: file-list diff between the two trees returns zero differences (93 files each, identical filenames/paths), and index.html content is byte-identical. The only measurable difference is per-file modification time, consistently ~29 hours older on the ippadmin copy across sampled files. This confirms both are build outputs of the same MkDocs source, generated by the same pipeline at two different times, not two diverging content sources.

Root cause (most likely, not confirmed via any log evidence during this read-only audit): an early build/deploy of the docs site landed in the default ippadmin home path (the operator's own home directory, root-owned), and a later correction properly deployed to a dedicated docs system user matching the nginx vhost root. The stale copy was never cleaned up.

Risk:

• No .git exists in either tree, so there is no audit trail proving which copy is "true" beyond nginx's own vhost pointer and file mtimes; a future config regeneration (this is a CloudPanel-managed host) or an operator mistake pointing the vhost root back at /home/ippadmin/... would silently roll the live site back ~29 hours, re-serving stale content (e.g., pre-dating the Aug 10 auth-gate documentation and VerdictTank v3 updates) with no visible error.
• The stale copy is root-owned while every other operational convention on this host uses a dedicated service user; this is itself a hygiene deviation worth correcting regardless of the duplication.
• There is currently no single documented deploy path (git repo -> build -> publish target) for this site, so a repeat of this duplication is only prevented by operator memory.

Recommendation (Phase Two): delete or archive /home/ippadmin/htdocs/docs.itpropartner.com, and formalize the build/publish pipeline (ITPP Standards already defines a docs-publish Gitea Actions workflow template; wire the live docs site to that pipeline with the docs htdocs path as its sole target) so there is exactly one deploy destination, git-tracked, going forward.

---

4. Audience Delineation: Internal vs Client vs Product/Dev

Current state: no delineation exists. All 12 sections live under one flat, undifferentiated MkDocs nav, gated by a single email allowlist behind one auth layer. Concretely mixed on the same site, one click apart:

• Internal-only, high-sensitivity operational material: Key Inventory (SSH key fingerprints/locations), Model Chain (LiteLLM virtual key hash, budget caps), Cost Control Rollout, app2 Caddyfile audit, Ops Portal / Backup-Restore internals (including the exposed historical credential noted in gap #2 above).
• Internal engineering process docs: ITPP Standards (CI/templates), Scripts catalog, HomeLab.
• Product/dev documentation for ITPP's own ventures: VerdictTank, TransitPin, FleetTracker360, Apex Track, BoxPilot, OSINT Tool, LaunchCheck, Shark Game - these describe products being built for eventual external users or customers, not ITPP's own infrastructure.
• Client-facing: none currently exists as a distinct, intentionally-scoped audience. Nothing in the current tree is written for or safe to hand to an actual paying customer of VerdictTank, FleetTracker360, or TransitPin - yet the auth gate's allowlist (g@germainebrown.com, info@itpropartner.com) suggests the site is intended for internal ITPP staff only, which conflicts with product docs plausibly needing outside readers eventually (contractors, investors, or customers).

Assessment: the site currently serves one audience (internal ITPP operators) under one login, with no mechanism to safely expose a subset of content (e.g., product architecture for a contractor, or end-user help docs for a VerdictTank customer) without also exposing Key Inventory, Model Chain, and Ops Portal internals. This is a structural risk, not just an organizational tidiness issue: the moment anyone outside the current two-person allowlist needs access to any single product doc, the only lever available is "grant them access to everything," including SSH key inventories and credential-adjacent operational docs.

Recommendation: split into three distinct trust zones (detailed in the reorg tree below) before granting any doc access to anyone outside the current internal allowlist.

---

5. Proposed Documentation Reorg Tree (Deliverable A.6)

Plain-English rationale first, then the tree.

Rationale:

1. Separate the credential-adjacent operational core from everything else. Key Inventory, Model Chain, Cost Control, Ops Portal/Backup-Restore internals, and the app2 Caddyfile audit should live in a zone that is never extended to anyone outside the current internal allowlist, regardless of what else changes.
2. Separate product/dev docs from internal ops docs, because product docs (VerdictTank, TransitPin, etc.) have a plausible future need for a wider audience (contractors, eventual customers) that internal ops docs never will.
3. Reserve an explicit, currently-empty client-facing zone rather than pretending the current site serves that purpose. If ITPP wants client-facing documentation (e.g., a VerdictTank user guide), it should be a deliberately separate publish target (different subdomain or path with its own, lighter-weight auth model), not a corner of the internal-only site.
4. Retire dead stubs rather than let them accumulate. Apex Track, BoxPilot, and (arguably) Shark Game and OSINT Tool have single-entry, 5-week-stale changelogs with no real content beyond "created project repository." Continuing to display them as live nav entries between actively-maintained sections misrepresents the site's currency.
5. Fix the self-contradiction between projects-master-readme and the sections it describes by making project status ("PLANNED" / "IN DEVELOPMENT" / "LIVE") a single generated field rather than manually duplicated free text in two places.

docs.itpropartner.com/
├── internal/ [existing allowlist gate stays here, unchanged scope]
│ ├── infrastructure/ (from itpp-infrastructure, minus product-adjacent items below)
│ │ ├── overview
│ │ ├── key-inventory <- flag: rewrite "sanitized" claim or actually remediate first (gap #1)
│ │ ├── model-chain
│ │ ├── cost-control-rollout-2026-07-24
│ │ ├── app2-caddyfile-audit-2026-07-21
│ │ ├── super-search-cf-bypass
│ │ ├── ops-portal (+ architecture)
│ │ ├── backup-restore (+ architecture)
│ │ ├── docs-auth-gate
│ │ ├── project-log
│ │ └── audit/ <- NEW: link Phase One (and future Phase Two) findings for internal transparency
│ ├── standards/ (itpp-standards, unchanged)
│ ├── homelab/ (unchanged; flag as "out of Phase One audit scope" until a homelab-specific audit exists)
│ └── scripts/ (unchanged)
│
├── products/ [same or a separate, slightly wider internal+contractor gate]
│ ├── verdicttank/ (+ architecture)
│ ├── transitpin/
│ ├── fleettracker360/
│ ├── apex-track/ <- ARCHIVE unless real content is added; currently a dead stub
│ ├── boxpilot/ <- ARCHIVE unless real content is added; currently a dead stub
│ ├── osint-tool/
│ ├── launchcheck/ (+ competitive-analysis, business-proposal)
│ └── shark-game/ <- review: 5-week-stale, confirm still active before keeping live
│
├── client/ [NEW, does not exist today - separate publish target/subdomain,
│ its own lightweight auth or fully public, populated only with
│ content explicitly written for external readers]
│ └── (empty until ITPP decides which product needs a customer-facing doc set)
│
└── legal/ [NEW, single location - currently a dangling empty page under
 itpp-infrastructure; either populate with real legal/compliance
 content or remove the nav entry entirely]

What moves where:

• itpp-infrastructure/* -> internal/infrastructure/* (unchanged content, new path only).
• itpp-standards, homelab, scripts -> internal/* (unchanged).
• verdicttank, transitpin, fleettracker360, apex-track, boxpilot, osint-tool, launchcheck, shark-game -> products/* (unchanged content, new path, plus an explicit staleness review for the four dead/near-dead stubs).
• itpp-infrastructure/legal -> either populated and moved to a top-level legal/ or removed.
• itpp-infrastructure/project-log and projects-master-readme -> reconcile into a single generated project-status view under internal/infrastructure/project-log, sourced from each project's own README/CHANGELOG status field rather than hand-maintained twice.
• New internal/infrastructure/audit/ -> add Phase One (and future Phase Two) findings summaries so the org has one place that tracks "what the docs say" next to "what the audits found."
• client/ -> created empty; populated only when/if ITPP commits to publishing customer-facing docs for a specific product, with its own auth model decided at that time.

What gets archived (not deleted, moved to an archive/ prefix or removed from nav):

• apex-track and boxpilot: both are single-entry "created project repository" stubs, 5 weeks stale, indistinguishable from placeholders. Either bring them current or pull them from the live nav so the site does not imply active documentation where none exists.
• The stale /home/ippadmin/htdocs/docs.itpropartner.com duplicate (not a doc section, but the entire stale build) should be deleted per section 3.

---

Summary of Evidence Sources

• Live site content: extracted via SSH from /home/docs/htdocs/docs.itpropartner.com on app3 (152.53.241.111), CHANGELOG and index pages per section.
• Split-brain comparison: find/diff/stat against both /home/docs/htdocs/... and /home/ippadmin/htdocs/... on app3.
• Cross-reference findings: neteng-a.md, neteng-b.md, sec-b.md, sys-a.md, sys-b.md, sys-c.md (all read in full for this audit).
• No configuration, DNS, nginx, or docs content was modified. All recommendations are deferred to Phase Two per the audit brief's absolute rules.