itpp-infrastructure/audit/phase-one/report.md

# ITPP Infrastructure Audit - Phase One Final Report

**Engagement:** Read-only discovery, audit, and documentation certification.
**Date:** 2026-08-13
**Conductor:** Sho'Nuff (deepseek-v4-pro) + claude-sonnet-5 (report/QA synthesis)
**Status:** COMPLETE (read-only). Zero live modifications performed on any target.

---

## Table of Contents

1. Executive Brief
2. Discovery Summary (verified inventory)
3. Findings (prioritized, severity tiers with rationale)
4. Recommendations (mapped to findings, effort estimate)
5. Documentation Status
6. Infrastructure Separation Assessment
7. Disagreements (documented, not resolved - Germaine resolves)
8. Independence-Check Appendix (Indep severity review - complete)

---

## 1. Executive Brief

Six servers were audited read-only on 2026-08-13: **Core, app1, app2, app3** (Netcup) and **app1-bu, wphost02** (Hetzner), plus the Gitea repository estate, DNS, Wasabi S3 backups, and the docs site.

The estate is functionally sound at the application layer but has **five structural weaknesses that compound each other**. Any one of them is a real finding; together they mean a single compromise today is an estate-wide incident, not a contained one.

**The five systemic themes:**

1. **No segmentation, and the firewall does not do what operators think it does.** Docker-published ports bypass UFW on 4 of 6 hosts, leaving roughly 20 management consoles (Wazuh, UniFi, UNMS/UISP, Grafana, CloudPanel, backup-restore UI, Gitea, MinIO, RAGFlow, Infinity DB, browserless, camofox) reachable from the public internet despite none of them appearing in any allow-list. There is no VLAN or subnet boundary between hosts, and the Tailscale mesh has no ACL tags - personal devices sit in the same allow-all group as production.

2. **Credentials are stored in plaintext in world-readable locations.** Two full unsanitized credential-inventory files remain on Core; app1-bu holds ~20 plaintext secrets (including root passwords for app1/app2/app3) in a world-readable `.env`; production API keys are hardcoded in world-readable systemd units; app3's MySQL root password sits in a world-readable backup script; and the public repo `itpp-infrastructure` re-leaks live admin passwords inside a prior audit's own report.

3. **Backups are write-only.** 30 of 34 live targets have a backup, but only **2 (Gitea, Vaultwarden) have ever been restore-tested**. LiteLLM's Postgres database - which holds every AI API key, routing table, and spend record - is never backed up at all (the dump targets a nonexistent database name and fails silently nightly). wphost02 has no scheduled backup for 6 of 8 WordPress databases.

4. **The SIEM monitors nothing but itself.** Wazuh is deployed and healthy, but `agent_control -l` shows zero enrolled agents across the estate. No centralized log forwarding exists anywhere. Grafana runs with default `admin/admin` credentials, publicly reachable, with no MFA - and no admin console in the estate has enforced MFA.

5. **The warm standby is not actually ready.** app1-bu syncs config files but deliberately skips the state databases; its `state.db` is ~28 days stale. On failover today, Hermes would come up without the last month of session, memory, and state.

**Bottom line:** the platform works day to day, but resilience, credential hygiene, and detection are all materially weaker than the documentation claims. This report maps every gap to a concrete Phase Two remediation with an effort estimate. Nothing was changed during Phase One.

---

## 2. Discovery Summary (verified inventory)

### 2.1 Servers

| Server | Provider / model | Public IP | OS / kernel | Role (verified) |
|---|---|---|---|---|
| Core | Netcup RS 2000 | 152.53.192.33 | Debian 13 / 6.12.94 | Hermes host + Grafana :3002, Prometheus, Super Search MCP :8899, backup orchestration |
| app1 | Netcup RS 4000 | 152.53.36.131 | Debian 13 / 6.12.95 | LiteLLM/admin-ai, Wazuh SIEM, Twenty CRM, Komodo, n8n, Vaultwarden, Caddy |
| app2 | Netcup RS 4000 | 152.53.39.202 | Debian 13 / 6.12.95 | Hudu, UNMS/UISP, UniFi, Traccar, Gitea, Dawarich, Technitium DNS (~40 containers) |
| app3 | Netcup RS 4000 | 152.53.241.111 | Debian 13 / 6.12.95 | CloudPanel shared web host (~24 sites) + shared Percona MySQL + Hexclave/Buzz Docker |
| app1-bu | Hetzner CPX21 | 5.161.225.131 | Ubuntu 24.04 | Warm standby for Core (config-only sync; state DB ~28d stale) |
| wphost02 | Hetzner | 5.161.62.38 | Ubuntu 24.04 | Legacy WordPress/RunCloud - **still live, not decommissioned** (split-brain with app3) |

All six reachable via the shared `itpp-infra` SSH key (single key, single blast radius).

### 2.2 Shadow IT / drift surfaced

- **wphost02 is still serving 8 WordPress sites** that also exist on app3 - an unresolved split-brain migration state with no cutover.
- **HotNow** was not found deployed on any audited host (needs confirmation of live/decommissioned status).
- **Three ad-hoc `python3 -m http.server`** processes on Core, one serving `/tmp`, all bound to 0.0.0.0 as root.
- **Three runaway `fix_dict.py`** processes pegging 3 cores on app1 for 14+ days.
- **Orphaned container** `happy_rosalind` (2nd BookStack, no compose project) on app2.
- **Stale duplicate docs build** on app3 (root-owned, ~29h older than the served copy).

### 2.3 Repository estate (Gitea)

56 total repos under org `ippadmin`. ~9 active, ~35 stale (mostly one Aug-8 scaffolding event), 1 orphaned (`itpp-infra`, remote deleted - local clone is now the only copy), 0 abandoned stubs. 11 repos exist on Gitea but were never cloned locally. Default-branch mismatch (`itpp-infrastructure` tracks `main` locally vs `master` on Gitea) risks fresh-clone confusion.

---

## 3. Findings (prioritized)

Severity legend: **Critical** = publicly reachable control plane, unrecoverable data-loss risk, or single-compromise = estate-wide blast. **High** = material security or resilience gap with significant impact. **Medium** = defense-in-depth gap or hygiene issue. **Low** = minor.

Consolidated severity counts across all nine auditor files (deduplicated where multiple auditors surfaced the same root cause): **Critical ~23, High ~34, Medium ~36, Low ~17.** Many overlap - the same Docker/UFW bypass is the root mechanism behind findings in NetEng-A, NetEng-B, and several Sys-B items. The Indep severity review (§8) net-adjusted three ratings: Sec-A-02 restored to Critical (C10), Git-A Finding 2 escalated High→Critical (folded into C4), and Sec-B-03 Technitium confirmed High - all documented in §7.

### 3.1 Critical findings (consolidated by root cause)

**C1. Docker published-port rules bypass UFW, exposing ~20 management consoles to the internet.**
Evidence (NetEng-A): on Core/app1/app2/app3, `docker run -p <port>` inserts DNAT rules into `nat/PREROUTING` and ACCEPT rules into `filter/FORWARD/DOCKER`, which are processed before UFW's `filter/INPUT`. UFW's allow-lists do not contain these ports, yet they are publicly reachable.
Why it matters: every operator believes UFW is the security boundary. It is not. Wazuh indexer/dashboard/API (app1), UniFi controller (app2), UNMS/UISP (app2), Gitea SSH (app2), MinIO console, RAGFlow, Infinity DB, browserless, camofox, Twenty CRM, Komodo, and the Buzz relay are all on the public internet with no firewall gate.

**C2. No network segmentation exists anywhere in the estate.**
Evidence (NetEng-B): all hosts are flat on public IP space with no VLAN/subnet; the Tailscale overlay has no ACL tags on any node (default allow-all); personal devices share the same group as production. Verified Core→app1 and app1→app2 reachable over public IPs, unfiltered.
Why it matters: compromise of any single host is a direct network path to every other host and to personal devices. No lateral-movement friction.

**C3. Wazuh SIEM is the highest-leverage target and it is both public and monitoring nothing.**
Evidence (NetEng-A + Sec-B): the Wazuh stack is publicly exposed via the Docker bypass, AND `agent_control -l` shows zero enrolled remote agents - the SIEM monitors only its own loopback.
Why it matters: the one system built to detect compromise is itself the most exposed control plane and is blind to the other five hosts.

**C4. Plaintext credentials in world-readable locations across the estate.**
Evidence (Sec-A + Sec-B + Sys-A + Sys-B + Git-A):
- Two unsanitized copies of the full `key-inventory.md` credential inventory remain on Core (one inside Hermes's own reference directory).
- Private `scripts` repo hardcodes the MSP-backdoor admin password that is reused across client onboardings - the single most consequential live credential in the estate (Git-A Finding 1).
- app1-bu `/root/.hermes/.env` (mode 644) holds ~20 plaintext secrets including root passwords for app1/app2/app3, Telegram bot token, Cloudflare/Netcup/SyncroMSP tokens, and eight AI-provider keys.
- `rally.service`, `seemytrip.service`, `giftaroast.service` hardcode `JWT_SECRET`, `DEEPSEEK_API_KEY`, `ADMIN_AI_KEY`, Twilio creds in world-readable unit files.
- app3 MySQL root password in `/opt/backup-restore/snapshot.sh` (mode 775) and `/root/backup.sh`.
- Public repo `itpp-infrastructure` re-leaks live admin passwords inside a prior audit report (see §7 - deferred per Germaine).
Why it matters: any local user or any single compromised service can read the entire secret estate.

**C5. LiteLLM Postgres database is never backed up.**
Evidence (Sys-A): `aws s3 ls s3://hermes-vps-backups/app1/litellm/` shows only config YAML objects. app1 `/root/backup.sh` runs `pg_dump` against database `litellm`, but the live `config.yaml` uses `litellm_db` - the dump targets a nonexistent database and fails silently every night.
Why it matters: LiteLLM's Postgres holds every API key, model routing table, spend/budget record, and the admin-ai provider config. A failure means total reconstruction by hand.

**C6. app3 runs ~24 sites (internal + client + product) on one shared MySQL instance with no tenant boundary.**
Evidence (NetEng-B + Sys-B): a single Percona `mysqld` backs internal ops sites (mainwp, support, panel), client sites (katiewatts, modelortho, vigilanttac, boxpilotlogistics, timapta), and products (transitpin, myverdicttank, buzz, hexclave).
Why it matters: a SQLi or credential leak on any one site is a plausible path to every other site's data on the same engine.

**C7. Grafana running default admin credentials, publicly reachable, no MFA.**
Evidence (Sec-B + NetEng-A): `docker inspect grafana` shows `GF_SECURITY_ADMIN_PASSWORD=admin`; port 3002 is explicitly allowed by UFW to Anywhere; no SSO/OAuth configured.
Why it matters: default `admin/admin` on an internet-reachable observability console is a near-zero-effort compromise path, and Grafana holds dashboards of the entire monitoring estate.

**C8. wphost02 has no effective backup for 6 of 8 WordPress databases, yet is still live.**
Evidence (Sys-B): scheduled `db-dump.sh` backs up only 2 of 8 DBs to local disk (7-day retention); the full offsite S3 script is not in any crontab. The host is still serving all 8 sites.
Why it matters: a live client-content host with ~75% of its databases unprotected.

**C9. Warm standby (app1-bu) is not data-ready.**
Evidence (Sys-B + Sys-C): sync covers only config/skills/plugins/cron/references; `state.db` (~2.1GB), `memory_store.db`, and `sessions/` are stale at Jul 15 (~4 weeks). Failover today would restore Hermes without the last month of state.
Why it matters: the DR plan's core assumption (warm standby can take over) is false for application state.

**C10. A single SSH key unlocks passwordless root on 5 of 6 hosts with no MFA and no segmentation to contain it.**
Evidence (Sec-A + NetEng-B): the shared `itpp-infra` key gives passwordless root sudo across the estate; it lives on Core alongside WireGuard keys to the home network + WISP towers and a live autossh tunnel into wphost02's MySQL. Per the severity legend, this is the definition of "single-compromise = estate-wide blast."
Why it matters: one key compromise, one leaked private key, or one compromised workstation with the key loaded is a full estate takeover. Restored to Critical per the Indep severity review - see §7 D4.

### 3.2 High findings (representative)

- **`ippadmin` has `NOPASSWD:ALL` sudo on 4 of 6 hosts**, reachable by the same shared key (Sec-A).
- **app3's `clpctlWrapper` sudoers rule grants ALL accounts** (including ~28 per-client site accounts) a passwordless root escalation path if the wrapper has any input-validation gap (Sec-A).
- **Gitea and CloudPanel each run on one shared admin account** with zero per-person accountability (Sec-A).
- **No admin console has enforced MFA** - Grafana, Wazuh, Gitea (open registration, no captcha), Hudu, UniFi, UNMS (SSO explicitly nulled), Technitium (literal `changeme` in env), CloudPanel, LiteLLM, Vaultwarden (Sec-B).
- **Technitium DNS runs with `DNS_SERVER_ADMIN_PASSWORD=changeme`** in the live container env - a default-credential flag on the estate's authoritative DNS (Sec-B).
- **app3 patch cadence ~4 weeks stale** with an unapplied security kernel + postfix update (Sec-B).
- **Core has no fail2ban, no unattended-upgrades, no auditd** - the weakest OS hardening of all 6 hosts, on the highest-value control-plane host (Sec-B).
- **Every custom service runs as root**; only hermes-voice and Caddy run non-root. One compromised service = full host takeover (Sys-A).
- **Very old images in production:** browserless/chrome (2yr), grafana 11.4.0 (20mo), wazuh-indexer 4.9.2 (21mo), rabbitmq 3.7.28 (5yr, EOL CVEs) (Sys-A, Sys-B).
- **UNMS/UISP is EOL software** (Ubiquiti discontinued 2021); the entire 13-container stack should be migrated (Sys-B).
- **app1 Caddyfile is not backed up** - the entire reverse-proxy routing config would need manual reconstruction (Sys-A).
- **Hermes gateway supervised only by the root user-manager**, with a socat unit referencing a nonexistent system unit (fragile SPOF) (Sys-A).
- **Port 8200 collision** between `hermes-control-deck` and `pipeline-api` - one is silently shadowed (Sys-A).
- **Three runaway `fix_dict.py` processes pegging 3 cores for 14+ days** on app1 (Sys-A).
- **app3 Docker services (Hexclave Stack Auth, Buzz relay) + TransitPin have no backup** (Sys-B).
- **app2 backup.sh silently skips Hudu/UNMS/UniFi** (local scripts missing) - see §7 reconciliation (Sys-B, conductor-verified).
- **Public DNS hygiene:** apex A record + ~10 legacy subdomains point to a decommissioned GCP host; SPF record malformed (concatenated strings); DMARC `p=none`; fleettracker360.com has no MX/SPF/DMARC (NetEng-A).
- **Private `hermes-recovery` repo commits a live MySQL password + Gitea API token** (Git-A).
- **WISP tower router (DR-017) has zero backup coverage** - `s3://mikrotik-ccr-backups/wisp-backups/configs/tower*` returns zero objects versus 30+ dailies for the home gateway at the same prefix pattern. An operational device with a total absence of config backup, not merely an untested one (Sys-C SYSC-04).

### 3.3 Medium / Low (summarized)

Medium: monitoring exporters bound to 0.0.0.0; ad-hoc http.servers; socat→Hermes on 0.0.0.0; MySQL X on `*`; weak L2TP/IKEv1 crypto on the tower VPN; plaintext VPN creds; no centralized log forwarding; Gitea open registration; secrets-sprawl git-grep hits needing per-file triage (Core 6,296 / app1bu 5,504 / app2 ragflow 1,275 - mostly false positives); orphan container; duplicate WordPress install; single-host SPOFs; mysql-tunnel disables host-key verification; dead cron (`docker-volume-sync.sh`); no swap on Core/app1; duplicate/conflicting auth-api-backup cron jobs (a working 03:15 job + a failing leftover 04:35 job - alert-fatigue risk, not a live data-loss condition today; Sys-C SYSC-02).

Low: avahi on public interface; app1-bu stale WireGuard rule + Tailscale name drift; leftover Docker volumes; disk 82% full on wphost02; `.aws` dir 775 on app1; 5 EOL PHP-FPM runtimes on app3.

---

## 4. Recommendations (mapped to findings)

Phase Two ordering. Effort: **S** = under 1 hour, **M** = half day, **L** = 1-2 days, **XL** = multi-day project.

| # | Recommendation | Maps to | Effort | Notes |
|---|---|---|---|---|
| 1 | Bind all Docker publishes to `127.0.0.1` and route through the reverse proxy, OR add a `DOCKER-USER` UFW chain. Do this before any other firewall work. | C1 | M-L | Highest leverage single change; closes ~20 public holes at once |
| 2 | Stand up Tailscale ACL tags: separate `servers`, `personal`, `mgmt` groups; drop the default allow-all. | C2 | M | Tailscale ACLs exist precisely for this |
| 3 | Enroll a Wazuh agent on all 5 non-manager hosts. | C3 | M | Turns the SIEM from self-monitoring to estate-wide |
| 4 | Rotate every credential found in plaintext (inventory the full list first), move to Vaultwarden, and chmod 600 all secret-bearing files. | C4 | L-XL | Highest-risk secrets first: app1-bu .env, key-inventory.md, systemd units, app3 snapshot.sh |
| 5 | Fix LiteLLM `pg_dump` to target `litellm_db`, verify a restore, then schedule it. | C5 | S-M | The dump command exists but has the wrong DB name |
| 6 | Segment app3 databases per tenant (separate MySQL users/schemas per site; move products to their own DB servers). | C6 | L-XL | Product isolation depends on this |
| 7 | Change Grafana admin password, enable SSO or TOTP, restrict :3002 to Tailscale. | C7 | S | Memory already has the rotated Grafana credential in Vaultwarden |
| 8 | Schedule wphost02's full S3 backup, then decommission wphost02 after confirming the app3 cutover. | C8 | M | Resolves both the backup gap and the split-brain |
| 9 | Extend standby sync to include state DBs (or accept a documented config-only standby with a revised RPO). | C9 | M | Explicitly contradicts DR-009 as-is |
| 10 | Split the single `itpp-infra` key into per-host or per-role keys; retire `NOPASSWD:ALL` for `ippadmin` in favor of named sudo. | High | M | Reduces single-key blast radius |
| 11 | Enforce MFA on Gitea, CloudPanel, Vaultwarden; close Gitea registration + enable captcha. | High | M | Native TOTP support already exists in all three |
| 12 | Enable fail2ban + unattended-upgrades + auditd on Core; remediate app3's stale patch cadence. | High | S-M | Bring Core to parity with the other 5 hosts |
| 13 | Back up app1 Caddyfile; back up app3 Docker services (Hexclave, Buzz) + TransitPin; restore-test Hudu/UNMS/UniFi. | High | M | Closes the write-only backup gap |
| 14 | Migrate UNMS→UISP; pin/upgrade old images (browserless, rabbitmq, grafana, wazuh-indexer). | High | L | UNMS is EOL; rabbitmq 3.7 has known CVEs |
| 15 | Fix port 8200 collision, kill runaway `fix_dict.py`, move Hermes gateway to a system unit. | High | S | Hygiene cleanup with real availability impact |

### 4.1 Effort rollup

- **Quick wins (S):** #7, #12 (partial), #15
- **Half-day (M):** #2, #3, #5, #8, #9, #10, #11, #13
- **Multi-day (L/XL):** #1, #4, #6, #14

Recommended sequencing: **#1 → #4 → #3** first (close the biggest exposure, rotate the secrets, turn on detection), then the remainder in listed order.

---

## 5. Documentation Status

The documentation is **materially out of sync with reality** across two dimensions:

**Say-do gaps (Docs-W):**
1. Key Inventory claims secrets are "sanitized" (2026-07-23), but Sys-A/Sys-B found plaintext secrets in world-readable locations (see C4).
2. Ops Portal changelog published a real historical admin credential in plaintext on an unauthenticated public site for ~3 weeks before the auth gate went live 2026-08-10.
3. The Docs Auth Gate doc describes JWT + allowlist protecting the whole site, but no test verifies the validator (port 8099) - unverified access control.
4. The Model Chain doc governs LiteLLM routing, but the LiteLLM Postgres DB is never backed up (see C5).
5. The app2 Caddyfile audit presents "validated" routing but omits the directly reachable Docker/UFW bypass services (see C1).

**Coverage drift (Docs-W):** 12 top-level sections, only 4 current (ITPP Infrastructure, ITPP Standards, TransitPin, VerdictTank - changelogs ≤4 days); 8 stale (HomeLab, Scripts, FleetTracker360, LaunchCheck, Shark Game, Apex Track, BoxPilot, OSINT Tool - 3-5+ weeks; two are dead stubs).

**Backup-plan undercount (Sys-C):** `backup-plan.md` documents 27 targets; live discovery found **34**. The documented plan undercounts reality by 7 - itself a documentation-drift finding.

**Split-brain docs build (Docs-W):** two nearly identical MkDocs builds on app3 - the served `/home/docs/htdocs/` (nginx target, newest) vs a stale root-owned `/home/ippadmin/htdocs/` copy (~29h older). Only the nginx vhost pointer proves authority; a config regen could silently repoint at the stale copy.

---

## 6. Infrastructure Separation Assessment

**Current state: there is effectively ONE trust zone.** Internal ITPP operations, client sites, and micro-SaaS products share:

- The same 6 hosts with no VLAN/subnet boundary between them.
- One flat Tailscale mesh with no ACL tags (personal devices included).
- One shared SSH key for root on all hosts.
- app1: Wazuh (SIEM) + Twenty CRM (client PII) + LiteLLM (AI control plane) + Komodo (deploy) alongside TransitPin and giftaroast.com (products) as sibling Docker containers behind one Caddy.
- app3: one shared MySQL/Percona instance backing internal, client, and product sites simultaneously.

**Positive controls already present (preserve in Phase Two):**
- app1-bu standby sync is **pull-only from S3**, not a live tunnel to Core - a correct blast-radius design worth keeping (don't add a live push tunnel later).
- Per-service DB passwords are scoped per container (Twenty `APP_SECRET`, Komodo `KOMODO_JWT_SECRET`, etc.) - correctly separated, just not vaulted.
- wphost02 is the only host outside the shared-key radius (uses a different key) and outside the Tailscale mesh.

**Recommended target (Phase Two):** three logical tiers - `internal` (ops tooling, tight allowlist + MFA), `client` (client sites, isolated DB per tenant), `product` (micro-SaaS, dedicated DB + credential vault per product) - enforced by Tailscale ACL tags, per-tenant MySQL users on app3, and binding Docker publishes to loopback.

---

## 7. Disagreements (documented, not resolved - Germaine resolves)

The following conflicts between auditor findings are logged here for Germaine's decision. Conductor did not silently resolve any of them; the resolutions below are read-only factual checks, with the open question flagged.

**D1. Standby watchdog target IP - Sys-B H1 vs NetEng-A/NetEng-B.**
Sys-B H1 claims the app1-bu watchdog pings the "wrong IP" (152.53.192.33), asserting Core is at 152.53.36.131.
**Conductor verification (read-only):** Core's public IP is **152.53.192.33** (confirmed via `ip addr` on this host). 152.53.36.131 is **app1**, whose password appears in `.env` as `SERVER_152_53_36_131_PASS`. The watchdog's `LIVE_HOST=152.53.192.33` is **correct**.
**Resolution:** Sys-B H1 is a false positive (IP conflation). The watchdog is targeting Core correctly. **No action required.**

**D2. Gitea/Hudu/UNMS/UniFi backup coverage - Sys-B C1 vs Sys-C.**
Sys-B C1 rates these four as Critical "no effective backup" because app2's `/root/backup.sh` references local scripts that do not exist on app2.
Sys-C found Gitea restore-tested PASS (2026-08-10) and 30/34 targets "OK".
**Conductor verification (read-only):** Both are partially correct. app2's own backup.sh silently skips Gitea/Hudu/UNMS/UniFi (local `gitea-backup.sh`, `hudu-backup.sh`, `unms-backup-sync.sh`, `unifi-backup-sync.sh` confirmed absent on app2). BUT Core owns the real backup scripts (present in `/root/.hermes/scripts/`, scheduled via Hermes cron jobs.json), and Gitea's backup was restore-tested PASS from `s3://hermes-vps-backups/gitea/daily/`. So Gitea has a working, tested backup via the Core-side path.
**Resolution:** Sys-B's mechanism observation is correct; its severity conclusion **overstates Gitea** (which has a tested backup). The durable truth: there is a redundant broken app2 job creating false failure-log confidence, and **Hudu/UNMS/UniFi remain untested** (no restore evidence) even though Core-side scripts are scheduled. Recommended severity: **High** (untested coverage + broken parallel job), not Critical for Gitea. **Open question for Germaine:** whether to treat Hudu/UNMS/UniFi's untested-but-scheduled backup as acceptable or as a Critical gap.

**Indep refinement (final):** Gitea should be **dropped from this finding entirely** (its restore test is a documented PASS, so "unrecoverable" is factually wrong for Gitea). The Hudu/UNMS/UniFi gap is a subset of Sys-C's already-Critical estate-wide "94% of backup targets never restore-tested" pattern, not an independent Critical. Final rating: **High** for Hudu/UNMS/UniFi, Gitea removed.

**D3. Public repo `itpp-infrastructure` credential exposure - Git-A (Critical/High).**
Git-A found live admin credentials re-leaked verbatim inside a prior audit report in the public repo.
**Germaine decision (2026-08-13):** "leave the repo alone for now."
**Resolution:** Deferred. The exposure remains, queued as a Phase Two finding. **No lockdown, rotation, or history scrub was performed.** Re-surface at Phase Two planning.

**Indep note on D3 severity:** git-a.md rated this (Git-A Finding 2) High, but the report treats it Critical-tier. Indep agrees with the escalation - a live reusable credential in a searchable *public* repo is a worse exposure than the same secret in a private repo, so Critical is the more defensible rating. This does not change Germaine's deferral, which was made with full knowledge of the finding.

**D4. Sec-A-02 "single SSH key" severity - Critical in sec-a.md, silently listed as High in §3.2.**
The source auditor (Sec-A) rated the single-key blast-radius finding Critical; the consolidated report placed it under "High findings (representative)" with no Section 7 entry explaining the change. Indep flagged this as both a severity error and a process gap: any time the conductor changes a source auditor's severity, it must appear here.
**Resolution:** Restored to **Critical** (now C10). The downgrade itself was the process gap - corrected.

**D5. Sec-B-03 Technitium `changeme` default credential - Critical in sec-b.md, listed as High in §3.2.**
Sec-B rated it Critical; the report listed it High with no Section 7 entry. Indep agrees **High** is the correct rating (the audit could not confirm the live in-app credential, and Technitium may not re-apply the env var after first bootstrap), but the silent downgrade should have been logged.
**Resolution:** Confirmed **High**. Documented here for the record. The finding stands as a legitimate hardening signal regardless of whether the string is literally the current password.

---

## 8. Independence-Check Appendix

**Reviewer:** Indep (claude-sonnet-5), independent QA pass. **Method:** re-read all 9 findings files and this report, then independently judged every Critical/High rating against its own stated evidence without deferring to the conductor's synthesis. Full review on disk at `findings/indep-review.md`.

### 8.1 Re-score verdicts (Critical/High)

| Finding | Conductor | Indep | Verdict |
|---|---|---|---|
| D1 / Sys-B H1 (watchdog "wrong IP") | False positive | False positive | FALSE-POSITIVE (agree) |
| D2 / Sys-B C1 (Gitea/Hudu/UNMS/UniFi backup) | High | High (Gitea dropped) | AGREE + drop Gitea |
| C1-C9 (Docker bypass, no segmentation, Wazuh, plaintext creds, LiteLLM, app3 MySQL, Grafana, wphost02, standby) | Critical | Critical | AGREE (all 9) |
| Sec-A-02 (single SSH key) | High (silent) | Critical | UPGRADE → C10 |
| NetEng-B NETB-6 (same single-key fact) | High | Critical | UPGRADE (duplicate of C10) |
| Sec-B-03 (Technitium `changeme`) | High | High | AGREE (downgrade was silent - logged D5) |
| Sys-C SYSC-02 (duplicate auth-api cron) | omitted | Medium | DOWNGRADE + add to §3.3 |
| Sys-C SYSC-04 (WISP tower router, no backup) | omitted | High | MISSED + add to §3.2 |
| Git-A Finding 1 (scripts repo backdoor password) | under-cited | Critical | AGREE + named in C4 |
| Git-A Finding 2 (public repo re-leak) | Critical-tier | Critical | AGREE (escalated from High) |
| Git-A Finding 3 (hermes-recovery) | High | High | AGREE |

### 8.2 False positives

- **Sys-B H1** - confirmed false positive (D1). Core's IP is 152.53.192.33, not 152.53.36.131 (that is app1).
- **Sys-B C1 as applied to Gitea** - "unrecoverable" is factually wrong; Gitea has a passing restore test (2026-08-10). Dropped from the finding.

No other Critical/High in the nine files was found factually wrong on re-read.

### 8.3 Under-weighted or missed

- **Sec-A-02** (single key = estate-wide blast) restored to Critical - see D4.
- **Sys-C SYSC-04** (WISP tower router, zero backup coverage) was missing from the consolidated report - now in §3.2.
- **Git-A Finding 1** (backdoor password reused across client onboards) was not named in C4 - now named.
- **Sys-C SYSC-02** (duplicate cron) was missing - now in §3.3, downgraded to Medium.
- **Sec-B-03** (Technitium) - Critical overstated given the audit could not confirm the live credential; confirmed High.

### 8.4 Verdicts on D1/D2/D3

- **D1:** agree with conductor - clean false positive.
- **D2:** agree with direction, go further - Gitea dropped entirely; Hudu/UNMS/UniFi is a subset of Sys-C's estate-wide "94% untested" Critical, not an independent Critical.
- **D3:** no grounds to disagree; Germaine's deferral is his call. Severity on the public-repo leak is better as Critical than git-a.md's High.

### 8.5 Overall confidence

High on D1 (unambiguous) and D2 (well-supported by Sys-C's independent S3 check); reasonably high on the SSH-key and Technitium re-scores (they turn on the report's own severity legend and an evidence gap the auditors themselves flagged); lower on SYSC-02's exact Medium-vs-High and on whether Technitium's live credential is literally still the default (out of scope for a read-only audit). No evidence of systematic severity inflation or deflation across the nine files - the surfaced issues are individual scoring errors plus one process gap (silent downgrades, now logged as D4/D5), not a pattern that casts doubt on the other 50+ findings.

---

## Appendix A: Cost - Estimate vs Actual

- **Estimate (approved):** subtotal ~$5.40, realistic $8-10, ceiling ~$13.
- **Actual (verified):** the audit triggered the LiteLLM per-key budget cap (`hermes-agent-v5` hit its $300 max_budget during the run; Germaine raised it to $400). This was driven by claude-sonnet-5 subagent usage plus re-runs from the 429 failures (Git-A and Sec-A each needed one resume).
- **Attribution caveat:** precise audit-attributable spend requires a LiteLLM SpendLogs query scoped to the 2026-08-13 subagent window. The 7-day estate-wide DeepSeek total was $61.81 (all usage, not audit-only). The audit **exceeded the $13 ceiling** - exact overage will be itemized in the SpendLogs reconciliation before the report is closed. No opus-tier models were used, consistent with the brief.

---

## Appendix B: Artifacts

Findings files (all read-only, no secrets as values):

| File | Auditor | Lines |
|---|---|---|
| `findings/neteng-a.md` | NetEng-A (network exposure) | 268 |
| `findings/neteng-b.md` | NetEng-B (segmentation/blast radius) | 197 |
| `findings/sec-a.md` | Sec-A (IAM/secrets) | 245 |
| `findings/sec-b.md` | Sec-B (hardening/patch/MFA/logging) | 160 |
| `findings/sys-a.md` | Sys-A (Core + app1) | 388 |
| `findings/sys-b.md` | Sys-B (app2/app3/app1-bu/wphost02) | 330 |
| `findings/sys-c.md` | Sys-C (backup/DR) | 215 |
| `findings/git-a.md` | Git-A (repos/secrets-in-history) | 238 |
| `findings/docs-w.md` | Docs-W (docs say-do gap) | 169 |

Raw capture files (for Phase Two re-verification): `/root/audit_*.txt` on Core.

---

*End of Phase One Final Report. No live modification was performed on any target system. All remediation is deferred to Phase Two.*