ippadmin/itpp-infrastructure
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Sync docs, audit artifacts, project notes, and VerdictTank proposal docs

Browse Source 
- audit/phase-one + phase-two: security audit briefs, findings, credential-rotation plan, Docker-USER hardening scripts, rollback refs
- disaster-recovery/restore-test-log.md + backup-dr-audit-2026-08-10.md
- clients/ (modelortho SEO audit, ai-biz-dev competitive landscape), notes/ (tiktok strategy)
- projects/: front-desk-voice-agent, seo-visibility-checker product plan, hotnow-savannah HTML, resend-transactional-email, backup-dashboard-enhancements, code-review-graph, seo-ci-architecture
- proposals/verdicttank/: architecture v4.0, methodology, judge-pool review, consolidation reasoning, cross-check review
- docs/super-search/firecrawl-provider-strategy.md
- updates: CHANGELOG, model-chain, projects-master-readme, intelsight.io
- .gitignore: exclude nested standalone repos (seo-tool, venturebuilt)
This commit is contained in:
root
2026-08-26 02:27:28 -04:00
parent 23e9751d38
commit f5175f1ce0
55 changed files with 14669 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
audit/phase-one/findings/docs-w.md
+169
View File
@@ -0,0 +1,169 @@
# Docs-W Findings: Documentation Audit (Phase One, Read-Only)
**Auditor:** Docs-W (documentation auditor)
**Scope:** docs.itpropartner.com (live MkDocs Material site on app3) and its stale duplicate; cross-referenced against neteng-a.md, neteng-b.md, sec-b.md, sys-a.md, sys-b.md, sys-c.md.
**Method:** Read-only SSH to app3 (152.53.241.111), filesystem inspection, HTML content extraction of every section/page and CHANGELOG. No live docs, nginx, or DNS were modified.
**Date:** 2026-08-13
---
## 1. Site Inventory
**Platform:** MkDocs Material, built and copied as static HTML into two htdocs trees on app3. No `.git` in either served tree, so the live docs root itself carries no version history or build provenance.
**Served copy:** `/home/docs/htdocs/docs.itpropartner.com` (owned by user `docs`, referenced by `/etc/nginx/sites-enabled/docs.itpropartner.com.conf`, most recent build timestamp ~2026-08-12 19:31 local).
**Unserved duplicate:** `/home/ippadmin/htdocs/docs.itpropartner.com` (root-owned, not referenced by any nginx vhost).
**Access control:** Since 2026-08-10, the entire site is gated by `docs-auth-validator.service` (JWT + email allowlist against auth2/Stack Auth). Before that date the site was open access. See section 3 (say-do gaps) for the implication.
### 1.1 Top-level sections (12), status, and last-update evidence (from each section's CHANGELOG)
| Section | Claims to document | Last CHANGELOG entry | Age vs audit date (2026-08-13) | Status |
|---|---|---|---|---|
| ITPP Infrastructure | Server inventory, DNS, architecture, key inventory, model routing, cost controls, ops portal, backup-restore, legal, project log | 2026-08-10 (Docs Auth Gate) | 3 days | Current |
| ITPP Standards | Documentation templates, CI (docs-check, docs-publish) | 2026-08-09 (Initial) | 4 days | Current |
| TransitPin | White-label transportation portal, dispatch, driver PWA | 2026-08-09 (migrate to Git); content dated 2026-07-29 | 4 days (changelog) / 15 days (content) | Current (changelog) |
| VerdictTank (+ Architecture) | Product review/validation platform, pricing, pipeline review | 2026-08-10 (v3 proposal + architecture) | 3 days | Current |
| HomeLab | Home lab infrastructure automation | "2026-07" (state snapshot 2026-07-21) | ~3+ weeks | Stale |
| Scripts | Operations/automation script catalog | 2026-07-21 | 3 weeks | Stale |
| FleetTracker360 | GPS fleet tracking platform | 2026-07-16 | 4 weeks | Stale |
| LaunchCheck | Startup validation product | 2026-07-25 (Project Inception, single entry) | 3 weeks, never updated since inception | Stale |
| Shark Game | Shark Attack Fantasy League | 2026-07-10 | 5 weeks | Stale |
| Apex Track | Track event management | 2026-07-10 ("Created project repository," single entry) | 5 weeks, stub only | Stale |
| BoxPilot | Logistics operations platform | 2026-07-10 ("Created project repository," single entry) | 5 weeks, stub only | Stale |
| OSINT Tool | OSINT people search / skip tracing | 2026-07-10 (Documentation migration) | 5 weeks | Stale |
**Section count: 12. Current: 4. Stale: 8.**
### 1.2 Notable sub-pages
- `itpp-infrastructure/`: Overview, Docs Auth Gate, Key Inventory, Model Chain, Cost Control Rollout (2026-07-24), Super Search CF Bypass (2026-07-21), app2 Caddyfile Audit (2026-07-21), Backup-Restore (+Architecture), Ops Portal (+Architecture), **Legal (empty page, no content rendered)**, Project Log, Projects Master README.
- `verdicttank/`: Overview, Architecture, plus markdown source files for v3.5/v3.6 architecture and a legal-framework document.
- `launchcheck/`: Competitive Analysis, Business Proposal (in addition to the section root).
---
## 2. Say-Do Gap Audit (cross-referenced against neteng-a/b, sec-b, sys-a/b/c)
| # | Doc claim | Live reality (per findings file) | Severity |
|---|---|---|---|
| 1 | `key-inventory` states secrets were "Sanitized... plaintext secrets replaced with storage references" (2026-07-23) | sys-a F-2: plaintext JWT_SECRET/DEEPSEEK_API_KEY/ADMIN_AI_KEY hardcoded in world-readable systemd units on Core (rally, seemytrip) and app1 (giftaroast). sys-b C2: `/root/.hermes/.env` on app1-bu (mode 644) holds ~20 plaintext secrets including root passwords for app1/app2/app3. sys-b C3: MySQL root password hardcoded in plaintext in two app3 scripts, one world-readable (775). | **Critical** - the one doc whose entire purpose is to assure the org that secrets are handled safely is contradicted by the live estate in at least three independent locations. |
| 2 | `ops-portal` CHANGELOG (2026-07-20) publishes a line reading "Admin credentials: [old username]/[old password] -> ippadmin (password moved to Vaultwarden)" | The docs site was open access (no auth gate) until 2026-08-10 per the site's own `docs-auth-gate` doc. This means a real historical admin credential sat in plaintext on an unauthenticated public web page for roughly three weeks. | **Critical** - a credential exposure inside the documentation itself, not just the infrastructure. |
| 3 | `docs-auth-gate` describes a JWT+allowlist access-control layer protecting the whole site, implying documentation (including key-inventory and the credential above) is now access-controlled | None of the six live-verified findings files enumerate or test `docs-auth-validator.service`/port 8099 on app3. sec-b's MFA/hardening coverage table (11 admin surfaces reviewed) does not include it. Its actual enforcement is undocumented outside its own self-description. | High - an access control the org depends on for a page containing credential-adjacent content has never been independently verified by the security or network auditors. |
| 4 | `projects-master-readme` lists Apex Track and BoxPilot as **"(PLANNED)"** | Both already have fully generated dedicated MkDocs sections (nav entries, CHANGELOG, "Created project repository and directory structure" stub content) on the same site. | Medium - internal self-contradiction on project status, visible on two pages of the same docs tree, no external cross-reference needed. |
| 5 | `itpp-infrastructure/legal` is a live nav entry titled "Legal" | Page renders with no content at all. | Medium - a documented, linked page with nothing behind it; reads as either an abandoned stub or a missed publish step. |
| 6 | `homelab` documents host inventory, DNS chain, and Docker service catalog for the separate home-lab network as if it were part of the audited estate | HomeLab (vm-host-01/02, QNAP, MikroTik home router) is **outside the scope** of neteng-a/b, sec-b, sys-a/b/c, which cover only Core/app1/app2/app3/app1-bu/wphost02. There is no live-verified findings coverage to check this section against at all. | Medium - a documented area of the estate with zero Phase One audit visibility; a structural blind spot rather than a specific factual error. |
| 7 | `model-chain` (2026-07-24) documents the AI model fallback chain and the admin-ai virtual key's daily budget cap, implying this is a governed, protected control | sys-a F-1: the LiteLLM Postgres database that stores this exact routing/budget/key configuration is **never backed up** (dump targets a nonexistent database name and fails silently every night). The governance the doc describes has zero disaster-recovery coverage, a fact the doc itself does not mention. | High - doc describes a control without disclosing that its backing store is unprotected. |
| 8 | `app2-caddyfile-audit-2026-07-21` documents a "final Caddyfile (validated)" for app2 routing UNMS, Gitea, UniFi, dns1, FleetTracker360 through Caddy | neteng-a APP2-1 through APP2-5 show UniFi, UNMS, Gitea SSH, Technitium DNS, and several other app2 services are **also** reachable directly via Docker's UFW bypass, i.e. the "validated" Caddy-only routing picture in the doc is incomplete: the real exposure includes direct Docker-published ports the doc never mentions. | High - the doc documents the intended path but is silent on (and therefore implicitly contradicts) the actual public exposure discovered live. |
| 9 | `backup-restore` doc describes a 30-day retention snapshot system and a documented restore API for 9 WordPress domains | sys-c's live restore-test audit found **zero** restore tests have ever been run against this specific mechanism (only Gitea and Vaultwarden have been restore-tested estate-wide), and sys-c SYSC-10 notes this snapshot layer is local-disk-only with no offsite copy - a single point of failure the doc does not disclose. | Medium - doc presents the system as complete/operational; live audit shows "backed up" without "restore-verified," which is exactly the gap Sys-C's engagement rule was written to catch. |
**Top 5 say-do gaps (for summary):** #1, #2, #3, #7, #8 above - ranked by blast radius and the fact that each represents documentation actively asserting a safety property (secrets sanitized, admin credential rotated, access gated, budget governed, routing validated) that the independently-verified infrastructure findings show is false, incomplete, or unverifiable.
---
## 3. Split-Brain: Two Copies of docs.itpropartner.com
**Finding:** Two nearly-identical full copies of the docs site exist on app3:
- `/home/docs/htdocs/docs.itpropartner.com` - owned by `docs`, referenced by the live nginx vhost, most recent build ~2026-08-12 19:31 local. **This is the authoritative, served copy.**
- `/home/ippadmin/htdocs/docs.itpropartner.com` - owned by `root`, **not referenced by any nginx config**, most recent build ~2026-08-10 (roughly 29 hours older than the served copy).
**Verification performed:** file-list diff between the two trees returns zero differences (93 files each, identical filenames/paths), and `index.html` content is byte-identical. The only measurable difference is per-file modification time, consistently ~29 hours older on the `ippadmin` copy across sampled files. This confirms both are build outputs of the same MkDocs source, generated by the same pipeline at two different times, not two diverging content sources.
**Root cause (most likely, not confirmed via any log evidence during this read-only audit):** an early build/deploy of the docs site landed in the default `ippadmin` home path (the operator's own home directory, `root`-owned), and a later correction properly deployed to a dedicated `docs` system user matching the nginx vhost root. The stale copy was never cleaned up.
**Risk:**
- No `.git` exists in either tree, so there is no audit trail proving which copy is "true" beyond nginx's own vhost pointer and file mtimes; a future config regeneration (this is a CloudPanel-managed host) or an operator mistake pointing the vhost root back at `/home/ippadmin/...` would silently roll the live site back ~29 hours, re-serving stale content (e.g., pre-dating the Aug 10 auth-gate documentation and VerdictTank v3 updates) with no visible error.
- The stale copy is `root`-owned while every other operational convention on this host uses a dedicated service user; this is itself a hygiene deviation worth correcting regardless of the duplication.
- There is currently no single documented deploy path (git repo -> build -> publish target) for this site, so a repeat of this duplication is only prevented by operator memory.
**Recommendation (Phase Two):** delete or archive `/home/ippadmin/htdocs/docs.itpropartner.com`, and formalize the build/publish pipeline (ITPP Standards already defines a `docs-publish` Gitea Actions workflow template; wire the live docs site to that pipeline with the `docs` htdocs path as its sole target) so there is exactly one deploy destination, git-tracked, going forward.
---
## 4. Audience Delineation: Internal vs Client vs Product/Dev
**Current state: no delineation exists.** All 12 sections live under one flat, undifferentiated MkDocs nav, gated by a single email allowlist behind one auth layer. Concretely mixed on the same site, one click apart:
- **Internal-only, high-sensitivity operational material:** Key Inventory (SSH key fingerprints/locations), Model Chain (LiteLLM virtual key hash, budget caps), Cost Control Rollout, app2 Caddyfile audit, Ops Portal / Backup-Restore internals (including the exposed historical credential noted in gap #2 above).
- **Internal engineering process docs:** ITPP Standards (CI/templates), Scripts catalog, HomeLab.
- **Product/dev documentation for ITPP's own ventures:** VerdictTank, TransitPin, FleetTracker360, Apex Track, BoxPilot, OSINT Tool, LaunchCheck, Shark Game - these describe products being built for eventual external users or customers, not ITPP's own infrastructure.
- **Client-facing:** none currently exists as a distinct, intentionally-scoped audience. Nothing in the current tree is written for or safe to hand to an actual paying customer of VerdictTank, FleetTracker360, or TransitPin - yet the auth gate's allowlist (`g@germainebrown.com`, `info@itpropartner.com`) suggests the site is intended for internal ITPP staff only, which conflicts with product docs plausibly needing outside readers eventually (contractors, investors, or customers).
**Assessment:** the site currently serves one audience (internal ITPP operators) under one login, with no mechanism to safely expose a subset of content (e.g., product architecture for a contractor, or end-user help docs for a VerdictTank customer) without also exposing Key Inventory, Model Chain, and Ops Portal internals. This is a structural risk, not just an organizational tidiness issue: the moment anyone outside the current two-person allowlist needs access to any single product doc, the only lever available is "grant them access to everything," including SSH key inventories and credential-adjacent operational docs.
**Recommendation:** split into three distinct trust zones (detailed in the reorg tree below) before granting any doc access to anyone outside the current internal allowlist.
---
## 5. Proposed Documentation Reorg Tree (Deliverable A.6)
Plain-English rationale first, then the tree.
**Rationale:**
1. **Separate the credential-adjacent operational core from everything else.** Key Inventory, Model Chain, Cost Control, Ops Portal/Backup-Restore internals, and the app2 Caddyfile audit should live in a zone that is never extended to anyone outside the current internal allowlist, regardless of what else changes.
2. **Separate product/dev docs from internal ops docs**, because product docs (VerdictTank, TransitPin, etc.) have a plausible future need for a wider audience (contractors, eventual customers) that internal ops docs never will.
3. **Reserve an explicit, currently-empty client-facing zone** rather than pretending the current site serves that purpose. If ITPP wants client-facing documentation (e.g., a VerdictTank user guide), it should be a deliberately separate publish target (different subdomain or path with its own, lighter-weight auth model), not a corner of the internal-only site.
4. **Retire dead stubs rather than let them accumulate.** Apex Track, BoxPilot, and (arguably) Shark Game and OSINT Tool have single-entry, 5-week-stale changelogs with no real content beyond "created project repository." Continuing to display them as live nav entries between actively-maintained sections misrepresents the site's currency.
5. **Fix the self-contradiction between `projects-master-readme` and the sections it describes** by making project status ("PLANNED" / "IN DEVELOPMENT" / "LIVE") a single generated field rather than manually duplicated free text in two places.
```
docs.itpropartner.com/
├── internal/ [existing allowlist gate stays here, unchanged scope]
│ ├── infrastructure/ (from itpp-infrastructure, minus product-adjacent items below)
│ │ ├── overview
│ │ ├── key-inventory <- flag: rewrite "sanitized" claim or actually remediate first (gap #1)
│ │ ├── model-chain
│ │ ├── cost-control-rollout-2026-07-24
│ │ ├── app2-caddyfile-audit-2026-07-21
│ │ ├── super-search-cf-bypass
│ │ ├── ops-portal (+ architecture)
│ │ ├── backup-restore (+ architecture)
│ │ ├── docs-auth-gate
│ │ ├── project-log
│ │ └── audit/ <- NEW: link Phase One (and future Phase Two) findings for internal transparency
│ ├── standards/ (itpp-standards, unchanged)
│ ├── homelab/ (unchanged; flag as "out of Phase One audit scope" until a homelab-specific audit exists)
│ └── scripts/ (unchanged)
│
├── products/ [same or a separate, slightly wider internal+contractor gate]
│ ├── verdicttank/ (+ architecture)
│ ├── transitpin/
│ ├── fleettracker360/
│ ├── apex-track/ <- ARCHIVE unless real content is added; currently a dead stub
│ ├── boxpilot/ <- ARCHIVE unless real content is added; currently a dead stub
│ ├── osint-tool/
│ ├── launchcheck/ (+ competitive-analysis, business-proposal)
│ └── shark-game/ <- review: 5-week-stale, confirm still active before keeping live
│
├── client/ [NEW, does not exist today - separate publish target/subdomain,
│ its own lightweight auth or fully public, populated only with
│ content explicitly written for external readers]
│ └── (empty until ITPP decides which product needs a customer-facing doc set)
│
└── legal/ [NEW, single location - currently a dangling empty page under
itpp-infrastructure; either populate with real legal/compliance
content or remove the nav entry entirely]
```
**What moves where:**
- `itpp-infrastructure/*` -> `internal/infrastructure/*` (unchanged content, new path only).
- `itpp-standards`, `homelab`, `scripts` -> `internal/*` (unchanged).
- `verdicttank`, `transitpin`, `fleettracker360`, `apex-track`, `boxpilot`, `osint-tool`, `launchcheck`, `shark-game` -> `products/*` (unchanged content, new path, plus an explicit staleness review for the four dead/near-dead stubs).
- `itpp-infrastructure/legal` -> either populated and moved to a top-level `legal/` or removed.
- `itpp-infrastructure/project-log` and `projects-master-readme` -> reconcile into a single generated project-status view under `internal/infrastructure/project-log`, sourced from each project's own README/CHANGELOG status field rather than hand-maintained twice.
- New `internal/infrastructure/audit/` -> add Phase One (and future Phase Two) findings summaries so the org has one place that tracks "what the docs say" next to "what the audits found."
- `client/` -> created empty; populated only when/if ITPP commits to publishing customer-facing docs for a specific product, with its own auth model decided at that time.
**What gets archived (not deleted, moved to an `archive/` prefix or removed from nav):**
- `apex-track` and `boxpilot`: both are single-entry "created project repository" stubs, 5 weeks stale, indistinguishable from placeholders. Either bring them current or pull them from the live nav so the site does not imply active documentation where none exists.
- The stale `/home/ippadmin/htdocs/docs.itpropartner.com` duplicate (not a doc section, but the entire stale build) should be deleted per section 3.
---
## Summary of Evidence Sources
- Live site content: extracted via SSH from `/home/docs/htdocs/docs.itpropartner.com` on app3 (152.53.241.111), CHANGELOG and index pages per section.
- Split-brain comparison: `find`/`diff`/`stat` against both `/home/docs/htdocs/...` and `/home/ippadmin/htdocs/...` on app3.
- Cross-reference findings: neteng-a.md, neteng-b.md, sec-b.md, sys-a.md, sys-b.md, sys-c.md (all read in full for this audit).
- No configuration, DNS, nginx, or docs content was modified. All recommendations are deferred to Phase Two per the audit brief's absolute rules.
audit/phase-one/findings/git-a.md
+238
View File
@@ -0,0 +1,238 @@
# Git-A Findings: Gitea Repository Audit (git.itpropartner.com)
**Auditor:** Git-A (claude-sonnet-5, subagent under Phase One ITPP Infrastructure Audit)
**Scope:** Full repo inventory, local/remote cross-reference, classification, lineage, secrets-in-history, disposition recommendations.
**Method:** READ-ONLY. All findings verified against the live Gitea API and local clone git history. Nothing was modified, deleted, renamed, force-pushed, or rotated. All remediation is written up below as a Phase Two recommendation.
**Date:** 2026-08-13
---
## 1. Full Inventory
Enumerated via `GET /api/v1/user/repos?token=...&limit=100`, paginated (2 pages, page 3 empty). **Total repos on Gitea: 56.** All owned by `ippadmin`. None are flagged `empty: true`.
| Repo | Private | Size (KB) | Default Branch | Last Updated |
|---|---|---|---|---|
| apex-track | No | 28 | main | 2026-08-09 |
| auth | Yes | 42 | main | 2026-08-08 |
| backup-restore | Yes | 29 | main | 2026-08-08 |
| boxpilot | No | 27 | main | 2026-08-09 |
| cartmylist | No | 70 | main | 2026-08-05 |
| competitive-landscape-research | No | 75 | main | 2026-08-10 |
| content-creation-pipeline | Yes | 27 | main | 2026-08-08 |
| digital-signage | No | 82 | main | 2026-08-08 |
| disaster-recovery | No | 82 | main | 2026-08-08 |
| dre | No | 117 | main | 2026-08-08 |
| fleettracker360 | No | 26 | main | 2026-08-08 |
| forefront-wireless-portal | No | 57 | main | 2026-08-08 |
| furniture-pos | Yes | 56 | main | 2026-08-12 |
| gift-a-roast | No | 28 | main | 2026-08-08 |
| hermes-recovery | Yes | 455 | main | 2026-08-09 |
| hermes-skills | No | 13807 | main | 2026-08-08 |
| homelab | Yes | 42 | main | 2026-08-09 |
| hudu | No | 32 | main | 2026-08-08 |
| itpp-docs | No | 946 | main | 2026-08-10 |
| itpp-infrastructure | No | 727 | **master** | 2026-08-12 |
| itpp-standards | No | 28 | main | 2026-08-09 |
| itpropartner-website | Yes | 36 | main | 2026-08-08 |
| launchcheck | No | 59 | main | 2026-08-09 |
| mcp-browser | No | 27 | master | 2026-08-08 |
| mcp-email | No | 26 | master | 2026-08-08 |
| mcp-filesystem | No | 26 | master | 2026-08-08 |
| mcp-git | No | 26 | master | 2026-08-08 |
| mcp-servers | No | 27 | main | 2026-08-08 |
| model-fallback | Yes | 43 | main | 2026-08-08 |
| mooresunnydaze | No | 107 | main | 2026-08-08 |
| msp-forms | Yes | 44 | master | 2026-08-08 |
| nvr-shield | No | 33 | main | 2026-08-08 |
| ops-portal | Yes | 31 | main | 2026-08-08 |
| ops-reports | Yes | 44 | main | 2026-08-12 |
| org-audit | Yes | 472 | master | 2026-08-09 |
| osint-tool | No | 37 | main | 2026-08-09 |
| personal-assistant | Yes | 82 | main | 2026-08-08 |
| pipeline | Yes | 41 | main | 2026-08-08 |
| pry | Yes | 39 | master | 2026-08-08 |
| research-search-mcp | Yes | 63 | main | 2026-08-08 |
| scripts | Yes | 75 | main | 2026-08-08 |
| seo-tool | No | 43 | master | 2026-08-10 |
| shark-game | No | 30 | main | 2026-08-08 |
| shonuff-caller | Yes | 31 | main | 2026-08-08 |
| startup-studio | Yes | 43 | main | 2026-08-08 |
| super-search | No | 39 | master | 2026-08-08 |
| super-search-business | No | 57 | main | 2026-08-08 |
| track-a-flock | Yes | 35 | main | 2026-08-08 |
| transitpin | Yes | 129 | main | 2026-08-09 |
| unifi | No | 27 | main | 2026-08-08 |
| unms | No | 27 | main | 2026-08-08 |
| venturebuilt | No | 47 | master | 2026-08-10 |
| verdicttank | No | 145 | main | 2026-08-12 |
| voice-agent | Yes | 32 | master | 2026-08-08 |
| voipsimplicity | No | 68 | main | 2026-08-08 |
| voipsimplicity-manual | Yes | 448 | main | 2026-08-08 |
Note: `itpp-infra` (singular, no "structure") does **not** appear in this list. Confirmed separately below (Section 4, Lineage) that it has been deleted server-side since the last local clone was made.
---
## 2. Cross-Reference: Gitea vs Local Clones
Local clones live under `/root/projects/`. 67 top-level directories exist there; 48 are real git repos (have a `.git` directory), 19 are plain project folders (docs, scratch dirs, or research artifacts with no version control).
### 2a. Gitea repos with NO local clone (11 repos)
These exist server-side but Git-A found no corresponding directory under `/root/projects/`. Cannot verify their content/purpose without a local clone; flagged as an access/visibility gap per Brief Rule 4.
| Repo | Private | Size | Notes |
|---|---|---|---|
| itpp-docs | No | 946KB | Aggregated MkDocs site - likely owned by Docs-W's workstream |
| mcp-browser | No | 27KB | MCP server source, `master` branch |
| mcp-email | No | 26KB | MCP server source, `master` branch |
| mcp-filesystem | No | 26KB | MCP server source, `master` branch |
| mcp-git | No | 26KB | MCP server source, `master` branch |
| msp-forms | Yes | 44KB | `master` branch |
| pry | Yes | 39KB | "pry service", `master` branch, no description |
| seo-tool | No | 43KB | ITPP SEO audit tool, `master` branch |
| super-search | No | 39KB | MCP server source (distinct from `super-search-business`) |
| venturebuilt | No | 47KB | AI business dev platform, `master` branch |
| voice-agent | Yes | 32KB | `master` branch |
**Why it matters:** Git-A cannot confirm what's in these 11 repos beyond Gitea's own metadata (name/description/size). If Sys-A/Sys-B or Sec-A need to verify what's actually deployed from these repos, they'll need a fresh clone - this audit's local-clone cross-reference has a blind spot here. All 11 also share the pattern of being either MCP micro-services or product-dev tools that may have been scaffolded once and forgotten (small size, single description, no CHANGELOG visible from metadata alone).
### 2b. Local repos with NO Gitea remote match ("local-only", no off-server backup)
None found with truly missing remotes - `deep-scan.py` reported **0 local-only repos** (every `.git`-bearing directory under `/root/projects/` has a configured `origin`). However, two categories of concern:
- **`itpp-infra`** - has an `origin` pointing at `git.itpropartner.com/ippadmin/itpp-infra.git`, but the repo **no longer exists on the Gitea server** (`git ls-remote` returns `remote: Repository not found`, HTTP 404 on the repos API). This is an **orphaned local clone of a deleted remote** - effectively local-only now, with 3 commits of history that exist nowhere else. See Section 4.
- **2 repos use GitHub, not Gitea, as their remote**: `msp-claude-skills` (github.com/RTFM-IT-Services-LLC) and `viral-content-system` (github.com/swaroop2004/Proven-Viral-Content-System - a fork of an external template, not an ITPP-authored repo). These are outside Gitea's scope entirely; flagged for awareness, not an audit failure.
### 2c. Non-repo project folders under /root/projects/ (19 total, no git at all)
`asher-and-rye-m365-consolidation`, `competitive-analysis`, `diglocate`, `forefront-broadband-map`, `giftaroast`, `kids-school-calendar`, `mautic-multitenant`, `mcp-planning`, `mcp-registration`, `mikrotik-monitoring`, `obsidian-selfhost`, `paperless-ngx`, `personal-shopping-assistant`, `portal-design-system`, `rfptank`, `school-calendar-2026-2027`, `twilio-10dlc`, `udm-tailscale`, `village-express`.
These are working directories, research scratch space, or planning docs - not under version control at all, so they carry **zero off-server backup**. Not directly this audit's remit (no Git history to inventory), but worth flagging to Sys-C/the conductor: any of these with real deliverable content (`giftaroast` has working Python files, e.g.) has no backup whatsoever, git or otherwise, beyond whatever the general server backup captures.
---
## 3. Classification
Classified by **last-commit recency** and **inferred purpose/domain**. "Active" = commit within last 5 days as of audit date (2026-08-13) AND has ongoing purpose; "Stale" = >5 days idle but still relevant; "Superseded" = replaced by a newer/renamed repo; "Orphaned" = remote gone, dangling reference, or abandoned scaffold.
### Active (commits within last ~5 days, live purpose)
itpp-infrastructure (age 0d, 90 commits - actively growing), furniture-pos (0d), ops-reports (0d), verdicttank (0d), competitive-landscape-research (2d), transitpin (3d), homelab (3d), itpp-standards (3d), org-audit (3d, private audit-artifact repo - still being appended to for this very engagement).
**Domain split:** itpp-infrastructure / homelab / itpp-standards / org-audit = internal infra. furniture-pos / transitpin / forefront-wireless-portal = client-facing. verdicttank / competitive-landscape-research / launchcheck = product-dev micro-SaaS.
### Stale (idle 4+ days, no evidence of abandonment - most repos)
The overwhelming majority of the 46 Gitea-backed local clones sit at **exactly 2026-08-08, ~13:06-13:08** - a single mass-scaffolding event, not organic development. This includes: apex-track, auth, backup-restore, boxpilot, cartmylist, content-creation-pipeline, digital-signage, dre, fleettracker360, forefront-wireless-portal, gift-a-roast, hermes-recovery, hermes-skills, hudu, itpp-infra, launchcheck, mcp-servers, model-fallback, mooresunnydaze, msp-claude-skills, nvr-shield, ops-portal, osint-tool, personal-assistant, pipeline, research-search-mcp, scripts, shark-game, shonuff-caller, startup-studio, super-search-business, track-a-flock, unifi, unms, viral-content-system, voipsimplicity, voipsimplicity-manual.
This pattern (dozens of repos all touched in the same 2-minute window on Aug 8) strongly suggests a batch `.gitignore`/scaffolding pass (consistent with the git-audit skill's own Step 9 "batch remediation" - likely a prior audit's cleanup run) rather than genuine feature work. **Domain split:** roughly half internal-infra/ops tooling (auth, backup-restore, hudu, unifi, unms, model-fallback, ops-portal, personal-assistant, pipeline, shonuff-caller, mcp-servers), half product-dev micro-SaaS prototypes (apex-track, boxpilot, dre, fleettracker360, gift-a-roast, launchcheck, mooresunnydaze, nvr-shield, osint-tool, shark-game, startup-studio, super-search-business, track-a-flock, voipsimplicity, voipsimplicity-manual) plus a couple of client-facing scaffolds (digital-signage, forefront-wireless-portal, cartmylist).
### Superseded / Duplicate
- **itpp-infra** superseded by **itpp-infrastructure** (see Lineage below - do not confuse them per skill pitfall).
- **cartmylist-repo** (local dir name) is the same project as Gitea's **cartmylist** - naming drift, not two repos (confirmed: same remote URL). Not a true duplicate but flagged for local-directory-name hygiene.
### Orphaned
- **itpp-infra** - remote deleted server-side; local clone is now a dangling reference to nothing. 3 commits of unique history exist only in this local clone.
---
## 4. Lineage: Duplicates, Forks, Naming Collisions
| Pair | Status | Verdict |
|---|---|---|
| **itpp-infra** vs **itpp-infrastructure** | itpp-infra: local clone exists (3 commits, "Initial commit -- audit Jul 24 2026" as its most substantive commit), but the **Gitea remote no longer exists** - confirmed via `git ls-remote` (`Repository not found`) and direct API call (404). itpp-infrastructure: 90 commits, actively updated as of today (2026-08-13), 727KB, is the working audit/documentation repo, default branch on Gitea is `master` while local clone tracks `main` (branch mismatch - see below). | **itpp-infra is dead/orphaned.** itpp-infrastructure is the live, correct repo. Do not confuse the two per skill's known pitfall - confirmed still true. |
| **cartmylist** (Gitea) vs **cartmylist-repo** (local dir name) | Same repo, same remote URL (`git.itpropartner.com/ippadmin/cartmylist.git`), local clone directory is just named differently (`cartmylist-repo`). Not a real duplication - it's local naming drift. | No consolidation needed; rename local directory for clarity in Phase Two, non-urgent. |
| **gift-a-roast** (Gitea+git) vs **giftaroast** (local, no git) | Two different local directories. `gift-a-roast` is the real git-tracked repo (28KB, matches Gitea). `giftaroast` is an *untracked scratch directory* containing loose Python fix-scripts (`fix_auth.py`, `fix_dict.py`, `fix_final.py`) and an `index.html` - looks like ad-hoc debugging output that predates or parallels the real repo, never committed anywhere. | `giftaroast` (no-git) should be reviewed and either merged into `gift-a-roast`'s history or deleted as scratch work - currently has zero backup. |
| **super-search** vs **super-search-business** | Two distinct Gitea repos. `super-search` = "super-search MCP server source code" (39KB, `master` branch, no local clone found). `super-search-business` = "Super Search for Business - multi-tenant competitive intelligence & OSINT SaaS platform" (57KB, `main` branch, cloned locally). Different products (infra tool vs. product-dev SaaS), not a duplicate - naming is just confusingly similar. | Not a collision requiring merge; recommend renaming one for clarity (e.g. `super-search-mcp` vs `super-search-business`) in Phase Two docs pass. |
| **mcp-servers** vs **mcp-browser / mcp-email / mcp-filesystem / mcp-git** | `mcp-servers` (cloned locally, "MCP servers for Open WebUI") appears to be a monorepo/aggregator. The four `mcp-*` singles (browser/email/filesystem/git) are NOT cloned locally, all sit on `master` branch (older default, out of step with the `main` convention used elsewhere), and are small (26-27KB each) with generic "<name> MCP server source code" descriptions. | Consolidation candidate: verify whether `mcp-servers` already contains these four as subdirectories (would need a fresh clone to confirm - Section 2a gap). If duplicated, the four standalone repos are consolidation-into-mcp-servers candidates. |
| **itpp-docs** vs **itpp-standards** vs **itpp-infrastructure** | Three separate repos all touching "ITPP documentation": itpp-docs = "Aggregated MkDocs documentation site for all IT Pro Partner projects" (946KB, not cloned locally); itpp-standards = "ITPP documentation standards, templates, and CI workflows" (28KB, 1 commit only); itpp-infrastructure = the working infra/audit repo with actual runbooks (727KB, 90 commits). | Not true duplicates (each has distinct scope: standards/templates vs published docs site vs raw infra runbooks) but worth flagging to Docs-W for the Git/Docs reorg recommendation - three repos with "docs" in scope invites confusion about which is canonical for what. |
**Branch mismatch flag (per skill's known pitfall, independently reverified today):** `itpp-infrastructure`'s local clone tracks `main` while Gitea's `default_branch` is `master`. This is not cosmetic - it means anyone cloning fresh from Gitea without specifying a branch lands on `master`, which may be stale relative to the `main`-tracking local clone that's actually being worked in daily. Confirmed both branches exist on the remote (`origin/master` and `origin/main` both resolve). **This should be fixed in Phase Two**: pick one branch, make it Gitea's default, delete the other.
Six other repos still default to `master` on Gitea rather than the `main` convention used everywhere else: `mcp-browser`, `mcp-email`, `mcp-filesystem`, `mcp-git`, `msp-forms`, `pry`, `seo-tool`, `super-search`, `venturebuilt`, `voice-agent`, and locally `org-audit`. Consistent naming convention is a Phase Two hygiene item, not a security risk.
---
## 5. Secrets-in-History
Scanned via the skill's regex pattern across full git history (`git log -p --all`) for all 46 Gitea-backed local clones, filtered against the false-positive list (`publicKeyToken`, doc-example/placeholder values, `?token=` in URLs). Manual review of every raw hit below; only REAL, exploitable values are recorded with `[REDACTED]` substituted for the actual secret.
### CRITICAL
**1. `scripts` repo - hardcoded Windows admin passwords in provisioning script, repo is PRIVATE but still exposed to anyone with repo access/token leak**
- Repo: `scripts` (private=true)
- Commits: `ec6e0e1b9894b9c7e16793aafacc4681f988bf86` ("Liberty: add ippadmin MSP backdoor account alongside liberty-admin customer admin"), `fef88f8d634957c538b3a800c9db1093bb9c3520` ("Refactor to two-script architecture..."), `2497f4c0c03b1435dcd9f59161a25d3b9383cc9f` ("Standard onboard v2...")
- File: `dell-reimage-kit/standard-onboard.ps1`
- Value: `[REDACTED]` - plaintext local-admin passwords for an "ippadmin MSP backdoor account" (used across all client onboards) and a "liberty-admin" customer admin account, embedded directly in a PowerShell provisioning script.
- **Why it matters:** This is a credential used to provision a hidden admin account on every client machine imaged with this kit - a live, reusable master-key password baked into source control. Even though the repo is private, anyone with read access to Gitea (or a leaked API token, see finding 3 below) gets a password that likely still works on production client endpoints today.
- **Disposition:** Rotate the password at the source (change it on any machine it was actually used on), then scrub history per the skill's Step 8 workflow, force-push, and verify.
### HIGH
**2. `itpp-infrastructure` repo - same passwords re-exposed, but this time in a PUBLIC repo**
- Repo: `itpp-infrastructure` (private=**false**)
- Commits: `a269a17b1f40460b0ef96ca234233167564b404d` ("git-audit: 42-repo hygiene audit Aug 8..."), `de0190283b6c89b399a9089dbaa1865e0b4cb337` ("docs: Git structure audit -- 40 Gitea repos...")
- Files: `docs/git-audit-2026-08-08.md`, `docs/git-audit-2026-08-07.md`
- Value: `[REDACTED]` - the same "ippadmin"/"liberty-admin" MSP backdoor passwords from Finding 1, quoted verbatim inside a **prior audit report** that a previous Git-A wrote and committed to this repo as evidence of the finding.
- **Why it matters:** This is worse than Finding 1 in one dimension: `itpp-infrastructure` is a **public** repo. A previous audit correctly identified the `scripts` repo credential leak, but then re-leaked the same credential by quoting it verbatim into a report and pushing that report to a public repo. Anyone on the internet who finds `git.itpropartner.com/ippadmin/itpp-infrastructure` can clone it and `git log -p` these two commits to get the same admin password. This is a textbook example of why the Brief's Rule 5 ("emit `[REDACTED]` everywhere, including paths that contain secrets") exists - Git-A is following it here; a prior pass did not.
- **Disposition:** Rotate the same credential (covers both findings 1 and 2 at once), then scrub history from `itpp-infrastructure` specifically since it's public-facing, force-push, verify with a fresh clone. Treat as higher urgency than Finding 1 purely because of public exposure, even though it's the same underlying secret.
**3. `hermes-recovery` repo - live database password + the actual Gitea API token used for this very audit, committed to history**
- Repo: `hermes-recovery` (private=true)
- Commit: `ae056eaf83b3d9ed273b68bce635aef8fdd1d665` ("Initial resurrection kit - 81 scripts, 62 references, configs, systemd units, crons, Docker compose files, Caddy config, master README")
- Files: `scripts/apex-mail-watchdog.py` (MySQL password for user `apextrackexperience_1781549652`), `configs/config.yaml` (a Gitea API token value)
- Value: `[REDACTED]` (both)
- **Why it matters:** Two separate live secrets in one commit: a MySQL credential for what looks like a production WordPress/mail-watchdog database, and a Gitea API token embedded directly in a backup config file. The repo is private, which limits blast radius to whoever has Gitea access - but "resurrection kit" repos exist specifically to be handed to whoever needs to rebuild the server, meaning this file is likely to be read, copied, and pasted elsewhere by design. That's exactly how a scoped secret becomes an unscoped one.
- **Disposition:** Rotate both the DB password and the Gitea token, scrub history, verify. Because this token may be the *same* token used to authenticate this very audit run (structurally identical format), Phase Two should explicitly confirm whether it's live before assuming it's already been rotated.
### MEDIUM (documentation artifacts, not live secrets - recorded for completeness, not actionable as leaks)
- `hermes-skills` repo: 15 regex hits, all reviewed - every one is a documentation example (`SOME_API_KEY: "value"`, `secret: "generate-a-strong-secret-here"`, shell snippets showing *how* to extract a key from a config file rather than an actual key value). **No real secret.** Two large files flagged separately (`skills/.curator_backups/.../skills.tar.gz` at 2.7MB and `skills/.hub/index-cache/hermes-index.json` at 38MB) - per the skill's known pitfall, this is a legitimate mirror repo and these are cache artifacts that should be gitignored, not evidence of misuse.
- `org-audit` repo: 3 regex hits, all reviewed - `SMTP_PASSWORD: '<REDACTED>'` is literally the placeholder string `<REDACTED>` already in the source (a prior audit's own sanitized report), `ADMIN_TOKEN` hit is a comment noting no token is set, `temporary-password-here` is a literal placeholder. **No real secret** - this repo is itself sanitized audit output and correctly follows the redaction convention.
### Summary table
| Severity | Repo | Public/Private | Real secret? | Commits |
|---|---|---|---|---|
| CRITICAL | scripts | Private | Yes - admin passwords | 3 commits |
| HIGH | itpp-infrastructure | **Public** | Yes - same admin passwords, re-leaked | 2 commits |
| HIGH | hermes-recovery | Private | Yes - DB password + Gitea API token | 1 commit |
| Info-only | hermes-skills | Public | No - doc examples only | n/a |
| Info-only | org-audit | Private | No - already redacted in source | n/a |
**No live secrets found in any of the 46 scanned repos beyond the three real findings above.** The 11 repos in Section 2a (no local clone) could not be scanned and remain an access/visibility gap - flag for Phase Two follow-up if their content needs verification.
---
## 6. Disposition Recommendations
| Repo | Classification | Recommendation | Rationale |
|---|---|---|---|
| itpp-infra | Orphaned (remote deleted) | **Archive/delete local clone** | Remote no longer exists; 3 commits of unique local history should be reviewed once for anything not already in itpp-infrastructure, then the local clone can be safely removed. Not backed up anywhere else. |
| itpp-infrastructure | Active | **Keep-active; fix branch default** | Live, growing, canonical infra/audit repo. Fix the `main`/`master` default-branch mismatch first (Phase Two). |
| cartmylist-repo (local dir) | Active (as `cartmylist` on Gitea) | **Keep-active; rename local dir** | Same repo as Gitea's `cartmylist`, just named differently locally. Cosmetic fix only. |
| giftaroast (local, no git) | Untracked scratch work | **Merge-into gift-a-roast or delete** | Loose debugging scripts with no version control and no backup. If content is still useful, commit into `gift-a-roast`'s history; otherwise delete as scratch. |
| mcp-browser / mcp-email / mcp-filesystem / mcp-git | Stale, not locally verifiable | **Merge-into mcp-servers (pending verification)** | Small, single-purpose repos with generic descriptions on the outdated `master` branch. Likely duplicated inside the `mcp-servers` monorepo - needs a fresh clone to confirm before consolidating. |
| super-search | Stale, not locally verifiable | **Keep-active but rename for clarity** | Distinct product from `super-search-business` (infra MCP tool vs. SaaS product) - not a true duplicate, but the near-identical name is a standing source of confusion. |
| The ~35 "Aug 8, 13:06-13:08" batch-scaffolded repos (apex-track, boxpilot, dre, fleettracker360, gift-a-roast, launchcheck, mooresunnydaze, nvr-shield, osint-tool, shark-game, startup-studio, super-search-business, track-a-flock, voipsimplicity, voipsimplicity-manual, and the internal-infra set: auth, backup-restore, hudu, unifi, unms, model-fallback, ops-portal, personal-assistant, pipeline, shonuff-caller, mcp-servers) | Stale | **Case-by-case review, default keep-active** | These are 4+ days idle but represent real, distinct project scaffolds (verified non-trivial directory structure in every sampled case), not abandoned stubs. No action needed unless the business decides a given micro-SaaS idea is dead - that's a product decision, not a Git hygiene one. Flagging as stale is informational, not a call to archive. |
| itpp-docs, itpp-standards | Active/Stale, overlapping scope with itpp-infrastructure | **Consolidation review with Docs-W** | Three "ITPP docs" repos with overlapping but distinct scope (standards/templates vs. published site vs. raw runbooks) invite confusion. Recommend Docs-W's Git/Docs reorg proposal explicitly define which repo owns what, rather than merging outright. |
| scripts | Active/stale, CRITICAL secret | **Keep-active, rotate + scrub immediately in Phase Two** | Real provisioning tooling still in use; the credential leak is the problem, not the repo's existence. |
| hermes-recovery | Stale, HIGH secret | **Keep-active, rotate + scrub immediately in Phase Two** | Legitimate disaster-recovery kit; same treatment - fix the leak, keep the repo. |
| org-audit | Active (still being written to for this engagement) | **Keep-active** | This is this audit's own output repo. No issue found. |
---
## 7. Access/Visibility Gaps (Brief Rule 4)
- **11 Gitea repos have no local clone** (Section 2a) - their contents could not be scanned for secrets or verified against description metadata. Flag for Phase Two: clone and re-scan `mcp-browser`, `mcp-email`, `mcp-filesystem`, `mcp-git`, `msp-forms`, `pry`, `seo-tool`, `super-search`, `venturebuilt`, `voice-agent`, `itpp-docs`.
- **itpp-infra's Gitea history is gone.** If that repo held anything not duplicated in `itpp-infrastructure`, it is now unrecoverable from Gitea - only the local clone's 3 commits remain. Recommend preserving that local clone as-is (do not delete) until someone confirms nothing of value is unique to it.
---
## 8. Estimate vs Actual
This subagent run is a re-run after a prior Git-A instance hit a LiteLLM budget 429 before writing output. No cost data is available to Git-A directly; defer to the conductor's LiteLLM SpendLogs reconciliation for this run's actual token/dollar cost.
audit/phase-one/findings/indep-review.md
+71
View File
@@ -0,0 +1,71 @@
# Independent Severity Review (Indep) - ITPP Phase One Audit
**Reviewer:** Indep (claude-sonnet-5), independent QA pass
**Scope:** Re-score every Critical/High finding in the nine findings files and cross-check against report.md Sections 3 and 7. Read-only. No infrastructure was touched to produce this review; all conclusions are drawn from the raw findings files already on disk.
**Method:** Read all 9 findings files (neteng-a, neteng-b, sec-a, sec-b, sys-a, sys-b, sys-c, git-a, docs-w) and report.md in full, then independently judged each Critical/High rating against its own stated evidence, without deferring to the conductor's synthesis.
---
## 1. Severity re-score table
| ID | Finding (short) | Conductor rating | My rating | Verdict |
|---|---|---|---|---|
| D1 / Sys-B H1 | Standby watchdog pings "wrong" IP 152.53.192.33 | Resolved as false positive (conductor) | False positive | FALSE-POSITIVE (agree) |
| D2 / Sys-B C1 | Gitea/Hudu/UNMS/UniFi "no effective backup" | Downgraded to High (conductor) | High for Hudu/UNMS/UniFi; not-a-finding for Gitea | DOWNGRADE-to-High, with the added correction that Gitea should be dropped from this finding entirely |
| Report C1 (NetEng-A) | Docker/UFW bypass, ~20 consoles public | Critical | Critical | AGREE |
| Report C2 (NetEng-B NETB-1) | No network segmentation anywhere | Critical | Critical | AGREE |
| Report C3 (NetEng-A APP1-1 + Sec-B-01) | Wazuh public + zero enrolled agents | Critical | Critical | AGREE |
| Report C4 (Sec-A-01, Sys-A F-2, Sys-B C2/C3, Docs-W #1) | Plaintext credentials estate-wide | Critical | Critical | AGREE, but see "under-weighted" note on Git-A Finding 1 below, which is thin in this writeup |
| Report C5 (Sys-A F-1) | LiteLLM Postgres never backed up | Critical | Critical | AGREE |
| Report C6 (NetEng-B NETB-3) | app3 single shared MySQL, ~24 sites | Critical | Critical | AGREE |
| Report C7 (Sec-B-02 + NetEng-A CORE-1) | Grafana default admin/admin, public, no MFA | Critical | Critical | AGREE |
| Report C8 (Sys-B C4) | wphost02 backup gap, 6 of 8 DBs unprotected | Critical | Critical | AGREE |
| Report C9 (Sys-B C5 / Sys-C SYSC-01) | Warm standby not data-ready | Critical | Critical | AGREE |
| Sec-A-02 | Single SSH key = root on 5 of 6 hosts, passwordless sudo on top | Critical in sec-a.md, silently downgraded to "High findings (representative)" in report Section 3.2, no Section 7 entry | Critical | UPGRADE-to-Critical (restore original rating; also flag the undocumented downgrade as a process gap) |
| NetEng-B NETB-6 | Same underlying fact as Sec-A-02, stated as High in neteng-b.md itself | High | Critical | UPGRADE-to-Critical (same reasoning as Sec-A-02; this is one finding described twice, not two findings) |
| Sec-B-03 | Technitium DNS `DNS_SERVER_ADMIN_PASSWORD=changeme` in container env | Critical in sec-b.md, silently shown as High in report Section 3.2, no Section 7 entry | High, with an explicit evidence caveat | DOWNGRADE-to-High (agree with the report's de facto number, disagree with doing it silently) |
| Sys-C SYSC-02 | Duplicate/conflicting auth-api-backup cron jobs | Critical in sys-c.md; not mentioned anywhere in report Section 3 | Medium/High | DOWNGRADE-to-Medium-or-High, and separately flag as omitted from the consolidated report |
| Sys-C SYSC-04 | WISP tower router (DR-017) has zero backup coverage at all | High in sys-c.md; not mentioned anywhere in report Section 3 | High | AGREE with sys-c's rating, but flag as MISSED from the consolidated report |
| Git-A Finding 1 | `scripts` repo: hardcoded MSP-backdoor admin password reused across client onboards | Critical in git-a.md; report's C4 write-up only names the downstream public re-leak (Finding 2/D3), not this original Critical | Critical | AGREE with git-a's rating; flag that report C4's evidence bullets omit this specific item and should name it explicitly, since rotating it is required independent of the D3 policy call on the public repo |
| Git-A Finding 2 | Same password re-leaked inside a PUBLIC repo (`itpp-infrastructure`) | High in git-a.md; treated as Critical-tier in report's C4/D3 framing | Critical | UPGRADE-to-Critical (agree with the report's implicit escalation over git-a's own High rating; public exposure of a live, reusable credential is worse than the private-repo case, and the severity legend the report itself uses supports Critical here) |
| Git-A Finding 3 | `hermes-recovery` (private): live MySQL password + a live Gitea API token in history | High | High | AGREE |
---
## 2. False positives
1. **Sys-B H1 (confirmed false positive - this is D1, see Section 4).** The claim that the standby watchdog targets the "wrong IP" is wrong. 152.53.192.33 is Core's real public IP per sys-a.md's own host profile table (Core Public IP row) and per report Section 2.1. 152.53.36.131 is app1, not Core. Sys-B conflated the two hosts. The watchdog is correctly configured.
2. **Sys-B C1, as applied to Gitea specifically (this is part of D2, see Section 4).** Sys-B's claim of "no effective backup, a loss would be unrecoverable" for Gitea is contradicted by sys-c.md's live evidence: Gitea's backup was restore-tested PASS on 2026-08-10 (117 DB tables, 52 repos, 3 sampled repos restored with valid git history). "Unrecoverable" is factually wrong for Gitea. This part of C1 should be dropped, not just downgraded.
No other Critical/High finding in the nine files was found to be factually wrong on re-read. The rest of C1 (Hudu/UNMS/UniFi backups being untested, see below) is a real gap, just not the "Critical, unrecoverable" framing Sys-B originally gave it.
---
## 3. Under-weighted or missed
1. **Sec-A-02 / NetEng-B NETB-6 (single SSH key = root on 5 of 6 hosts).** Sec-A rated this Critical in its own file. Report Section 3.2 lists the same fact under "High findings (representative)" with no corresponding Section 7 disagreement entry explaining the downgrade. Using the report's own severity legend ("Critical = ... single-compromise = estate-wide blast"), a single key that unlocks passwordless root on 5 of 6 servers, with no MFA and no network segmentation to contain it, meets that bar. I recommend restoring this to Critical. Separately, the fact that it was downgraded without being logged as a disagreement (the way D1/D2/D3 were) is itself a process gap worth naming to Germaine: any time the conductor changes a source auditor's severity, it should show up in Section 7, even if the conductor believes the change is obviously correct.
2. **Sys-C SYSC-04 (WISP tower router, zero backup coverage, DR-017 still open).** Rated High in sys-c.md with clear evidence (`s3://mikrotik-ccr-backups/wisp-backups/configs/tower*` returns zero objects, versus 30+ dailies for the home gateway at the same prefix pattern). This does not appear anywhere in report.md Section 3 (Critical or High), and is not folded into any of the C1-C9 themes since it is a standalone network-device gap, not a Docker/backup-script issue. This is a genuinely missed High finding: an operational device with a total absence of configuration backup, not just an untested one.
3. **Git-A Finding 1 (`scripts` repo, hardcoded MSP-backdoor admin password used across client onboards).** Rated Critical in git-a.md, correctly. Report's C4 write-up (the Critical bucket for plaintext credentials) lists key-inventory.md copies, app1-bu's `.env`, systemd units, app3's MySQL password, and the itpp-infrastructure public re-leak, but never names this specific finding, the one that is arguably the most consequential of the group because it is a live password reused across production client machines, not just infrastructure secrets. It is mentioned only indirectly through D3 (which covers the re-leak, not the original). Recommend the report name Finding 1 explicitly in C4's evidence list.
4. **Sys-C SYSC-02 (duplicate/conflicting auth-api-backup cron jobs).** Rated Critical in sys-c.md. On re-read, I think this is overstated: the working 03:15 job succeeds every night, and the second 04:35 job is a leftover that fails visibly. The real risk here is alert fatigue (a failing job that nobody investigates because "the cron always shows an error") rather than a live data-loss condition today. I would score this Medium, with a note that it could become a real gap if the good job silently breaks later. Separately, whatever its severity, it is not mentioned anywhere in report.md Section 3 and should be, since it currently reads as fully resolved (it is not).
5. **Sec-B-03 (Technitium DNS `DNS_SERVER_ADMIN_PASSWORD=changeme`).** Rated Critical in sec-b.md. I think Critical overstates the confidence level here. Technitium (like many similar tools) typically only applies an admin-password environment variable on first bootstrap of its config; once a config already exists, subsequent container restarts do not necessarily re-apply that env var to the live credential. Sec-A's own file explicitly says it "could not confirm the live in-app credential value without an authenticated read." Sec-B's own rationale acknowledges this too ("Even if the operational credential has since been changed inside the app's own database..."). Given that acknowledged uncertainty, I would score this High rather than Critical: the finding (a default-credential string persisting in a live container env, on the estate's authoritative DNS) is a legitimate and important hardening signal regardless of whether it is literally the current password, but "Critical" implies a confirmed, exploitable credential, which this audit did not verify. The report's own Section 3.2 already lists this as High, so the net number matches what I'd recommend, but again, that downgrade from sec-b.md's own Critical rating was made silently, with no Section 7 entry.
---
## 4. Verdict on D1, D2, and D3
**D1 (Sys-B H1, watchdog "wrong IP"): I agree with the conductor's resolution.** Core's public IP is confirmed as 152.53.192.33 in sys-a.md's host profile table and in report.md's Discovery Summary (Section 2.1). 152.53.36.131 belongs to app1. Sys-B's H1 conflated the two hosts and its underlying claim is false. This is a clean false positive, not a judgment call. No action needed beyond correcting Sys-B's file for the record.
**D2 (Sys-B C1, Gitea/Hudu/UNMS/UniFi "no effective backup"): I agree with the direction of the conductor's resolution (downgrade), and I'd go slightly further on the details.** Sys-C's live S3 evidence shows all four services have current, on-schedule backups running through a different mechanism than the one Sys-B checked (Core-side scripts scheduled via Hermes's own cron system, not the app2-local scripts Sys-B examined, which genuinely are missing). That distinction matters: Sys-B's observation that the specific scripts referenced in app2's own `/root/backup.sh` do not exist is accurate and worth keeping as a hygiene finding (a redundant, broken, misleading logging path), but the conclusion that these four services have "no effective backup" and "a loss would be unrecoverable" is not supported by the live evidence. For Gitea specifically, there is a passing restore test, so I would remove it from this finding entirely rather than just downgrading its severity. For Hudu, UNMS, and UniFi, the accurate framing is "backups exist and are current, but have never been restore-tested," which is a real gap, appropriately High, not Critical. This also overlaps with Sys-C's own broader Critical finding (SYSC-03: 94%+ of all backup targets estate-wide have never been restore-tested), so Hudu/UNMS/UniFi's specific gap is really a subset of an already-Critical estate-wide pattern rather than its own independent Critical.
**D3 (Git-A public repo credential re-leak, Germaine deferred remediation): I do not have grounds to disagree with the underlying finding, and deferral is Germaine's call to make, not mine to override.** The finding itself is factually solid: git-a.md independently confirmed the same admin password sits in git history in a public repo, verified against live Gitea API data. Where I'd add value here is on severity, not on the remediation decision: git-a.md itself rated this specific finding (Finding 2) as High, but the report's consolidated C4 treats it as Critical-tier alongside the other plaintext-credential findings. I agree with the report's implicit escalation, a live, reusable credential sitting in a searchable public repository is a worse exposure than the same secret in a private repo, so Critical is the more defensible rating even though the source auditor called it High. Germaine's decision to leave the repo alone for now is a risk-acceptance call made with full knowledge of the finding; I have no evidence that the decision was made on a mistaken understanding of severity, so I am not overriding it, I am only flagging that the underlying risk is live and, if anything, slightly under-stated by git-a.md's own severity label.
---
## 5. Overall confidence statement
Confidence in this review is high for the two flagged disagreements (D1 is unambiguous, D2 is well-supported by Sys-C's independent live S3 check) and reasonably high for the severity re-scores involving the shared SSH key and the Technitium default-credential finding, since those turn on the report's own stated severity legend and on an explicit evidence gap the source auditors themselves called out, not on speculation. Confidence is lower, and explicitly flagged as such, on SYSC-02's exact severity (Medium vs High is a closer call than Critical vs Medium) and on whether Sec-B-03's live Technitium credential is actually still the default, since neither this review nor any of the nine original findings files could confirm the live value without an authenticated read, which was correctly out of scope for a read-only audit. Where evidence was insufficient to fully confirm or refute a claim, I have said so explicitly rather than guessing, consistent with the audit's own read-only, no-assumption rules. I found no evidence of systematic severity inflation or deflation across the nine files; the two confirmed issues (D1, D2) and the additional items surfaced here are individual scoring errors and one process gap (severity downgrades happening without a corresponding Section 7 entry), not a pattern that should cast doubt on the audit's other 50+ Critical and High findings, which were consistently well-evidenced with specific file paths, command output, or cross-referenced live checks.
audit/phase-one/findings/neteng-a.md
+268
View File
@@ -0,0 +1,268 @@
# NetEng-A - Network Exposure Inventory (Phase One, Read-Only)
**Auditor:** NetEng-A (network enumeration)
**Date:** 2026-08-13
**Scope:** Firewalls, open/listening ports, public DNS, VPN/private paths, and reverse-proxy/ingress surface across the ITPP estate.
**Method:** Read-only discovery only (`ss -tulpn`, `ip`, `ufw status`, `iptables -L/-t nat`, `dig`, `docker ps`, `docker inspect`, `cat` of config files). No configuration was modified on any host.
**Severity legend:** Critical = publicly reachable management/security console or control-plane with broad impact. High = significant unintended public exposure or broken DNS control. Medium = defense-in-depth gap or weak configuration. Low = hygiene/minor.
---
## 1. Executive summary
- **Hosts reachable:** 6 of 6 servers in scope (Core, app1, app2, app3, app1-bu, wphost02) - all accepted the `itpp-infra` SSH key.
- **Total open listening sockets (bound to non-loopback/public addresses):** ~390 across the estate. app2 alone accounts for ~330 of them (Traccar's `5000-5150` device range published on both TCP and UDP = 302 sockets).
- **Single most important finding:** Docker published-port rules bypass UFW on Core, app1, app2, and app3. Security and management consoles (Wazuh, UniFi, UNMS, Grafana, CloudPanel) are reachable from the public internet even though UFW's allow-lists do not include their ports.
- **DNS hygiene is poor:** the `itpropartner.com` apex A record and ~10 legacy subdomains still resolve to a decommissioned GCP host (`35.212.86.161`), and the SPF record is malformed (two concatenated `v=spf1` strings with a truncated IP).
---
## 2. Cross-cutting finding: Docker port publishing bypasses UFW (Critical)
**Evidence (app1, identical mechanism on Core/app2/app3):**
```
# iptables -t nat -L DOCKER -n
DNAT tcp 0.0.0.0/0 0.0.0.0/0 tcp dpt:5601 to:172.26.0.4:5601
DNAT tcp 0.0.0.0/0 0.0.0.0/0 tcp dpt:9200 to:172.26.0.2:9200
# iptables -L DOCKER -n
ACCEPT tcp 0.0.0.0/0 172.26.0.4 tcp dpt:5601
ACCEPT tcp 0.0.0.0/0 172.26.0.2 tcp dpt:9200
```
UFW's `ALLOW` list on app1 contains only `22, 80, 443, 1514, 1515` (and `3006` from Core). Yet `5601`, `9200`, `55000`, `514/udp`, `3003`, `9120` are all published by Docker as `0.0.0.0:<port>`. Docker inserts DNAT rules into `nat/PREROUTING` and ACCEPT rules into the `filter/FORWARD` DOCKER chain, which are processed *before* UFW's `filter/INPUT` chain. UFW's default-deny therefore never sees these packets.
**Why it matters:** every operator on these boxes believes UFW is the security boundary, but any `docker run -p <port>` silently punches a public hole. On app2, ~20 services (UniFi controller, UNMS/UISP, Gitea SSH, BookStack, MinIO console, RAGFlow, Technitium DNS, support API, Infinity DB) are publicly reachable despite none of their ports appearing in UFW. This is a systemic, high-impact control failure.
**Remediation (Phase Two):** bind Docker publishes to `127.0.0.1` (e.g. `-p 127.0.0.1:5601:5601`) and route through the reverse proxy, or enable `ufw-docker`/`DOCKER-USER` chain rules.
---
## 3. Core (localhost / 152.53.192.33)
### 3.1 Open ports (non-loopback)
| Port | Proto | Process | Bound | UFW | Publicly reachable? |
|---|---|---|---|---|---|
| 22 | tcp | sshd | 0.0.0.0 | ALLOW Anywhere | Yes (intended) |
| 80/443 | tcp/udp | caddy | 152.53.192.33 | ALLOW Anywhere | Yes (intended ingress) |
| 3000 | tcp | browserless (docker) | 0.0.0.0 | **not listed** | **Yes - bypass** |
| 3001 | tcp | uptime-kuma (docker) | 0.0.0.0 | **not listed** | **Yes - bypass** |
| 3002 | tcp | grafana | `*` | ALLOW Anywhere | **Yes (intentional)** |
| 9377 | tcp | camofox-browser (docker) | 0.0.0.0 | **not listed** | **Yes - bypass** |
| 8080 | tcp | `python3 -m http.server` | 0.0.0.0 | tailscale0 only | No (UFW), bound 0.0.0.0 |
| 8083 | tcp | shark-game backend | 0.0.0.0 | not listed | No (UFW) |
| 8105 | tcp | rally backend | 0.0.0.0 | not listed | No (UFW) |
| 8787 | tcp | socat → 127.0.0.1:8642 (Hermes) | 0.0.0.0 | not listed | No (UFW) |
| 8899 | tcp | super-search MCP | 0.0.0.0 | 172.17.0.0/16 only | No |
| 8934, 9876 | tcp | `python3 -m http.server` | 0.0.0.0 | not listed | No (UFW) |
| 9090 | tcp | prometheus | `*` | not listed | No (UFW) |
| 9100 | tcp | node_exporter | `*` | not listed | No (UFW) |
| 9119 | tcp | hermes gateway | 0.0.0.0 | not listed | No (UFW) |
| 9273 | tcp | telegraf | `*` | not listed | No (UFW) |
| 34239 | tcp | act_runner | `*` | not listed | No (UFW) |
| 1701 | udp | xl2tpd (L2TP) | 0.0.0.0 | not listed | No (UFW) |
| 51821 | udp | wireguard | 0.0.0.0 | ALLOW Anywhere | Yes (intended) |
| 5353 | udp | avahi-daemon | 0.0.0.0 + mcast | not listed | No (multicast) |
### 3.2 Firewall
UFW **active**, default deny incoming/allow outgoing/deny routed. Allows `22, 80, 443, 51821/udp, 8890, 3002`, `8080 on tailscale0`, `8899 from 172.17.0.0/16`. Baseline is sound, but the Docker bypass (section 2) undermines it for `3000/3001/9377`.
### 3.3 Findings
| ID | Severity | Finding | Evidence | Why it matters |
|---|---|---|---|---|
| CORE-1 | High | **Grafana (3002) publicly exposed by explicit UFW rule** | `ufw: 3002/tcp ALLOW Anywhere`; `ss: *:3002 grafana` | Grafana holds dashboards of the entire monitoring estate; a public, unauthenticated-able Grafana (or one with weak creds) leaks ops data and is a frequent CVE target. |
| CORE-2 | High | **Headless-browser proxies publicly reachable via Docker bypass** | `docker: browserless 0.0.0.0:3000`, `camofox-browser 0.0.0.0:9377`; `nat DOCKER DNAT 0.0.0.0/0 dpt:3000`, `dpt:9377` | browserless/camofox render arbitrary URLs; a public instance is an SSRF / internal-network pivot primitive. |
| CORE-3 | Medium | **Monitoring exporters bound to `0.0.0.0`** | `ss: *:9090 prometheus`, `*:9100 node_exporter`, `*:9273 telegraf` | These expose metrics (hostnames, labels, sometimes secrets in scrape configs) if UFW is ever disabled. Should be loopback/private-only. |
| CORE-4 | Medium | **Three ad-hoc `python3 -m http.server` on 0.0.0.0** | PIDs 2191495 (:8080), 1601850 (:8934), 1607064 (:9876) | Unauthenticated static file servers serving unknown directories on all interfaces. |
| CORE-5 | Medium | **Hermes control API exposed via socat on 0.0.0.0:8787** | `socat TCP-LISTEN:8787,fork,reuseaddr TCP:127.0.0.1:8642` | A control/agent API reachable on all interfaces; UFW currently blocks it, but the binding is needlessly broad. |
| CORE-6 | Low | **avahi/mDNS (5353) running on public interface** | `ss: 0.0.0.0:5353 avahi-daemon` | Multicast name resolution leaks hostnames/services to the local segment. |
---
## 4. app1 (152.53.36.131, Netcup RS 4000) - Wazuh / AI / CRM host
### 4.1 Open ports (non-loopback)
| Port | Proto | Process | Bound | UFW | Publicly reachable? |
|---|---|---|---|---|---|
| 22 | tcp | sshd | 0.0.0.0 | ALLOW | Yes |
| 80/443 | tcp | caddy | `*` | ALLOW | Yes (intended) |
| 3006 | tcp | caddy (browserless proxy) | `*` | ALLOW from Core only | No (source-restricted) |
| 1514/1515 | tcp | Wazuh manager | 0.0.0.0 | ALLOW Anywhere | **Yes (intentional but risky)** |
| 514 | udp | Wazuh syslog | 0.0.0.0 | **not listed** | **Yes - bypass** |
| 55000 | tcp | Wazuh manager API | 0.0.0.0 | **not listed** | **Yes - bypass** |
| 9200 | tcp | Wazuh indexer (Elasticsearch) | 0.0.0.0 | **not listed** | **Yes - bypass** |
| 5601 | tcp | Wazuh dashboard | 0.0.0.0 | **not listed** | **Yes - bypass** |
| 3003 | tcp | Twenty CRM | 0.0.0.0 | **not listed** | **Yes - bypass** |
| 9120 | tcp | Komodo core | 0.0.0.0 | **not listed** | **Yes - bypass** |
### 4.2 Findings
| ID | Severity | Finding | Evidence | Why it matters |
|---|---|---|---|---|
| APP1-1 | **Critical** | **Wazuh security stack publicly exposed via Docker bypass** | `docker: single-node-wazuh.indexer-1 0.0.0.0:9200`, `dashboard-1 0.0.0.0:5601`, `manager-1 0.0.0.0:55000, 0.0.0.0:514/udp`; UFW lists none of these | The Wazuh indexer holds all security-event telemetry for the estate and the dashboard is the admin console. Both are on the public internet with no UFW gate. |
| APP1-2 | High | **Wazuh agent enrollment ports (1514/1515) open to the world by explicit UFW rule** | `ufw: 1514/tcp, 1515/tcp ALLOW Anywhere` | Enrollment should be restricted to agent source ranges; a public enrollment port invites rogue agent registration into the SIEM. |
| APP1-3 | High | **Twenty CRM (3003) and Komodo core (9120) publicly reachable via Docker bypass** | `docker: twenty-server-1 0.0.0.0:3003`, `komodo-core-1 0.0.0.0:9120` | Client CRM data (PII) and the Komodo deployment/automation control plane are public, bypassing UFW. |
| APP1-4 | Info | Intended public surface is correctly proxied | `/etc/caddy/Caddyfile` maps `vault`, `n8n`, `ai`, `admin-ai`, `noc`, `wz`, `sign.iamgmb.com`, `giftaroast.com`, `crm.debtrecoveryexperts.com`, `komodo.iamgmb.com`, `transitpin.iamgmb.com` → 127.0.0.1 backends | Reverse proxy is doing its job; the leaks are at the Docker layer, not Caddy. |
---
## 5. app2 (152.53.39.202, Netcup RS 4000) - management/self-hosted stack
### 5.1 Open ports (non-loopback)
| Port(s) | Proto | Service | UFW | Publicly reachable? |
|---|---|---|---|---|
| 22 | tcp | sshd | ALLOW | Yes |
| 80/443 | tcp | caddy | ALLOW | Yes (intended) |
| 53 | tcp/udp | Technitium DNS (dns1.itpropartner.com) | ALLOW **only from 76.195.7.60** | **Yes - bypass (open resolver)** |
| 81, 8089, 8444 | tcp | UNMS/UISP nginx | **not listed** | **Yes - bypass** |
| 8080, 8443, 8843, 8880 | tcp | UniFi controller | **not listed** | **Yes - bypass** |
| 3478, 10001 | udp | UniFi STUN/discovery | **not listed** | **Yes - bypass** |
| 2055 | udp | UNMS NetFlow | **not listed** | **Yes - bypass** |
| 3022 | tcp | Gitea SSH | **not listed** | **Yes - bypass** |
| 6875 | tcp | BookStack | **not listed** | **Yes - bypass** |
| 6880 | tcp | support-api | **not listed** | **Yes - bypass** |
| 8082 | tcp | Traccar web UI | **not listed** | **Yes - bypass** |
| 5000-5150 | tcp+udp | Traccar device listeners (302 sockets) | **ALLOW Anywhere (explicit)** | **Yes (intentional)** |
| 9001 | tcp | MinIO console | **not listed** | **Yes - bypass** |
| 9380-9384, 9392 | tcp | RAGFlow | **not listed** | **Yes - bypass** |
| 23817, 23820 | tcp | Infinity database | **not listed** | **Yes - bypass** |
### 5.2 Findings
| ID | Severity | Finding | Evidence | Why it matters |
|---|---|---|---|---|
| APP2-1 | **Critical** | **UniFi Network Controller publicly exposed via Docker bypass** | `docker: unifi-controller 0.0.0.0:8080,8443,8843,8880`; `nat DOCKER DNAT 0.0.0.0/0 dpt:8443 to:172.17.0.2:8443`; UFW lists none | The UniFi controller manages the tower/Wi-Fi network. Its 8443 web UI and 8080 device-inform endpoint are public. Compromise = control of the wireless/tower estate. |
| APP2-2 | **Critical** | **UNMS/UISP (ISP management) publicly exposed via Docker bypass** | `docker: unms-nginx 0.0.0.0:81,8089,8444`; `nat DNAT 0.0.0.0/0 dpt:8444 to:172.18.251.5:443` | UISP is the entire WISP network-management plane (device inventory, configs, customers). Publicly reachable despite UFW. |
| APP2-3 | High | **Technitium DNS published 0.0.0.0:53 bypasses source restriction** | UFW allows `53 only from 76.195.7.60`, but `docker: technitium 0.0.0.0:53->53 tcp+udp` and `DOCKER chain ACCEPT 0.0.0.0/0 dpt:53` | Operator clearly intended DNS to be reachable only from the home router, yet the Docker publish makes it a public open resolver (DNS amplification/abuse risk). |
| APP2-4 | High | **Traccar device range (5000-5150, ~302 sockets) fully public by explicit UFW rule** | `ufw: 5000:5150/tcp+udp ALLOW Anywhere` | Fleet-tracking device protocol listeners are open to the world; large attack surface for protocol-specific exploits and data injection. |
| APP2-5 | High | **Gitea SSH (3022), BookStack (6875), support-api (6880), MinIO console (9001), RAGFlow (9380-9392), Infinity DB (23817/23820) all public via bypass** | `docker ps --format '{{.Ports}}'` shows all bound `0.0.0.0` | Source-code host, internal wiki, support API, object-storage console, and a vector database are each on the public internet. |
| APP2-6 | Info | `git.itpropartner.com`, `hudu.itpropartner.com`, `unifi.itpropartner.com`, `unms.forefrontwireless.com`, `ragflow.itpropartner.com`, `gps.fleettracker360.com` proxied via Caddy | `/etc/caddy/Caddyfile` | The reverse proxy is correctly terminating TLS for the intended public names; the exposure is the direct Docker port publish bypassing it. |
---
## 6. app3 (152.53.241.111, Netcup RS 4000) - CloudPanel shared web host
### 6.1 Open ports (non-loopback)
| Port | Proto | Process | Bound | UFW | Publicly reachable? |
|---|---|---|---|---|---|
| 22 | tcp | sshd | 0.0.0.0 | ALLOW | Yes |
| 80/443 | tcp/udp | nginx (CloudPanel) | 0.0.0.0 | ALLOW | Yes (intended) |
| 8443 | tcp | nginx (CloudPanel panel) | 0.0.0.0 | ALLOW 8433:8443 | **Yes (intentional)** |
| 8090 | tcp | backup-restore web app (python) | 0.0.0.0 | ALLOW Anywhere | **Yes (intentional)** |
| 21 | tcp | proftpd FTP | 0.0.0.0 | not listed | No (UFW), bound 0.0.0.0 |
| 25 | tcp | postfix SMTP | 0.0.0.0 | not listed | No (UFW) |
| 3000 | tcp | buzz-prod-relay (docker) | 0.0.0.0 | **not listed** | **Yes - bypass** |
| 6081 | tcp | varnish | 0.0.0.0 | not listed | No (UFW) |
| 8080 | tcp | nginx (alt vhost) | 0.0.0.0 | not listed | No (UFW) |
| 33060 | tcp | mysqld (MySQL X) | `*` | not listed | No (UFW), bound `*` |
### 6.2 Findings
| ID | Severity | Finding | Evidence | Why it matters |
|---|---|---|---|---|
| APP3-1 | High | **CloudPanel admin panel (8443) publicly exposed** | `ufw: 8433:8443/tcp ALLOW Anywhere`; `ss: 0.0.0.0:8443 nginx` | CloudPanel is the control plane for ~30 client websites on this box. Publicly exposing the admin panel (even with auth) is a high-value target. |
| APP3-2 | High | **Backup-restore web UI (8090) publicly exposed** | `ufw: 8090/tcp ALLOW Anywhere`; `ss: 0.0.0.0:8090 python /opt/backup-restore/app/app.py` | A web UI that can trigger restores is a destructive-capability surface and should be internal/Tailscale-only. |
| APP3-3 | High | **Buzz relay (3000) public via Docker bypass** | `docker: buzz-prod-relay-1 0.0.0.0:3000`; `nat DNAT 0.0.0.0/0 dpt:3000 to:172.19.0.5:3000`; UFW does not list 3000 | The Block/Buzz relay is publicly reachable with no UFW gate. |
| APP3-4 | Medium | **MySQL X protocol (33060) bound to `*`** | `ss: *:33060 mysqld` | Database protocol listener on all interfaces (currently UFW-blocked); should be loopback. |
| APP3-5 | Medium | **FTP (21), SMTP (25), Varnish (6081) bound to 0.0.0.0** | `ss` output | FTP is cleartext (credential leakage if ever allowed); SMTP bound publicly invites relay abuse; Varnish cache admin not needed externally. |
---
## 7. app1-bu (5.161.225.131, Hetzner CPX21) - warm standby
Minimal footprint. Only `22/tcp` (sshd) and `41641/udp` (tailscaled) listening on non-loopback. UFW active: `22/tcp` and `51821/udp` allowed.
| ID | Severity | Finding | Evidence | Why it matters |
|---|---|---|---|---|
| APP1BU-1 | Low | **UFW allows 51821/udp (WireGuard) but no WireGuard service is installed/listening** | `ufw: 51821/udp ALLOW`; `ss` shows no 51821; `wg: command not found` | Leftover rule opens a port with no service behind it; indicates incomplete standby bring-up. |
| APP1BU-2 | Info | Tailscale device name drift | `tailscale status` lists `app1-bu` (100.112.23.21, offline ~28d) and `app1-bu-1` (100.95.212.28, online) | The live standby is registered as `app1-bu-1`; the old `app1-bu` node is stale on the tailnet. |
---
## 8. wphost02 (5.161.62.38, Hetzner) - legacy WordPress/RunCloud
**Still live and serving traffic.** Not decommissioned.
| Port | Proto | Process | Bound | UFW | Publicly reachable? |
|---|---|---|---|---|---|
| 22 | tcp | sshd | 0.0.0.0 | ALLOW | Yes |
| 80/443 | tcp | nginx-rc | 0.0.0.0 | ALLOW | Yes (legacy sites) |
| 25 | tcp | postfix | 0.0.0.0 | not listed | No (UFW) |
| 34210 | tcp | runcloud agent | `*` | not listed | No (UFW) |
| 9100 | tcp | node_exporter | `*` | not listed | No (UFW) |
| ID | Severity | Finding | Evidence | Why it matters |
|---|---|---|---|---|
| WPHOST-1 | Medium | **Legacy WordPress host still publicly serving on 80/443** | `ss: 0.0.0.0:80,443 nginx-rc`; `ufw: 80,443 ALLOW` | Flagged as possibly decommissioned in scope; it is still a live public attack surface (legacy WordPress) that should be verified against the migration plan and decommissioned or hardened. |
| WPHOST-2 | Low | **RunCloud agent (34210) and node_exporter (9100) bound to `*`** | `ss: *:34210 runcloud`, `*:9100 node_exporter` | Management agent and metrics exporter on all interfaces (currently UFW-blocked). |
---
## 9. Public DNS - itpropartner.com and related domains
Nameservers: `ns1/ns2.siteground.net`. All lookups against `1.1.1.1`.
| ID | Severity | Finding | Evidence | Why it matters |
|---|---|---|---|---|
| DNS-1 | **High** | **Apex A record and ~10 legacy subdomains point to decommissioned GCP host 35.212.86.161** | `itpropartner.com A → 35.212.86.161` (no live server uses this IP). Also `ssh`, `mail`, `ftp`, `autodiscover`, `autoconfig`, `clientmedia`, `www.clientmedia`, `media`, `www.media`, `apx`, `www.apx`, `www` all → `35.212.86.161` | The primary domain apex resolves to a retired host. Visitors and services hitting the apex go nowhere (or to an attacker if the IP is later reassigned). Live subdomains (`ops`, `core`, `app1`, `git`, `panel`, etc.) point to the correct hosts. |
| DNS-2 | High | **SPF record is malformed (concatenated `v=spf1` + truncated IP)** | `"v=spf1 +a +mx +ip4:35.209.36v=spf1 +a +mx +ip4:35.212.110.90 include:... ~all"` | Two SPF records were merged and an IP (`35.209.36`) is truncated. Broken SPF breaks delivery and can allow spoofing depending on how receivers parse it. |
| DNS-3 | Medium | **DMARC is `p=none` (monitoring only), and four related domains have no DMARC at all** | `_dmarc.itpropartner.com → "v=DMARC1; p=none; aspf=r; adkim=r"`; `fleettracker360.com`, `voipsimplicity.com`, `iamgmb.com`, `debtrecoveryexperts.com` → empty | No enforcement means the domain can be spoofed with no receiver-side protection. |
| DNS-4 | Medium | **fleettracker360.com has no MX, no SPF, no DMARC** | `dig MX/TXT` all empty (NS = Cloudflare) | A live product domain with no mail/DMARC records is trivially spoofable. |
| DNS-5 | Low | **voipsimplicity.com apex has no A record** | `dig A voipsimplicity.com → (empty)` | Apex resolves to nothing; subdomain `my.voipsimplicity.com` works but the root does not. |
| DNS-6 | Info | **DKIM present (dnssmarthost); wildcard absent; MX correct** | `default._domainkey.itpropartner.com TXT → v=DKIM1...`; `*.itpropartner.com → empty`; `MX → mx10/20/30.antispam.mailspamprotection.com` | Good: DKIM configured, no wildcard, MX routes through SiteGround antispam. |
| DNS-7 | Info | **Live subdomain map verified** | `ops/core/my/sign/uptimekuma/app/status → 152.53.192.33`; `app1/n8n/ai/admin-ai/vault/wz/noc → 152.53.36.131`; `git/hudu/unifi → 152.53.39.202`; `panel/mainwp/support/auth2/docs/forms/mockups/proposals → 152.53.241.111`; `app1-bu → 5.161.225.131` | Subdomains are correctly mapped to live hosts; only the apex + legacy names are stale. |
---
## 10. VPN / private network paths
| Path | Technology | Endpoints | State |
|---|---|---|---|
| Server mesh | Tailscale | Core 100.71.155.7, app1 100.90.186.109, app3 100.72.15.12, app1-bu-1 100.95.212.28 (+ app2, personal devices) | Up, full mesh |
| Core → home network | WireGuard `wg0` (10.77.0.1/24) | peer `home-gateway` 10.77.0.2, endpoint 76.195.7.60:443 | Up |
| Core → home lab / tower subnets | WireGuard routed | `10.1.0.0/16`, `10.2.0.0/16`, `172.16.1.0/24`, `172.18.18.0/24` via wg0 | Up |
| Core → WISP towers | L2TP/IPsec (`ppp0`) | server 76.195.7.60; routes `10.199.1-4.0/24`, `10.199.100.0/24`, `192.168.88.0/24` | **Down** (charon not running, no ppp0, no 10.199 routes) |
| ID | Severity | Finding | Evidence | Why it matters |
|---|---|---|---|---|
| VPN-1 | Medium | **Tower VPN (L2TP/IPsec) uses IKEv1 + AES128/SHA1/MODP1024** | `home-router-vpn.sh`: `ike=aes128-sha1-modp1024`, `esp=aes128-sha1-modp1024`, `keyexchange=ikev1` | Legacy crypto for the path into the WISP tower network; weak and cryptographically dated. |
| VPN-2 | Medium | **VPN credentials stored in plaintext config** | `/root/.hermes/scripts/wisp-backup/config.yaml` holds L2TP `psk`, `username`, `password` in cleartext (values `[REDACTED]`) | A file-readable compromise of Core yields credentials to the home gateway and tower network. Cross-ref Sec-A for secret management. |
| VPN-3 | Info | Tower subnets only reachable while the nightly-backup VPN is up | `ipsec status` → charon refused; no `10.199.*` routes present | Towers (T01-T04, MP100) are not persistently reachable; enumeration of their live config was not possible this session. |
---
## 11. Edge devices (MikroTik CCR towers / UniFi / UISP)
Discoverable but **not directly enumerated this session** (VPN down; read-only constraint).
- **Home MikroTik gateway:** public endpoint `76.195.7.60` (WireGuard :443, L2TP/IPsec :1701/udp). SSH reachable via WireGuard as `home-gateway` 10.77.0.2 (`wisp_rsa` key, user `shonuff`).
- **WISP tower routers (CCR):** `T01-RTR 10.199.1.4`, `T02-RTR 10.199.2.4`, `T03-RTR 10.199.3.4`, `T04-RTR 10.199.4.4`, `MP100-RTR 10.199.100.4` (source: `wisp-backup/config.yaml`).
- **UniFi controller** runs on app2 (publicly exposed, see APP2-1). **UNMS/UISP** runs on app2 (publicly exposed, see APP2-2). Backup key deployment for towers is via `deploy-key.rsc` (SSH pubkey, not secret).
- No edge-device config was modified; only inventory/paths recorded.
---
## 12. Access limitations
| Host | Status |
|---|---|
| Core, app1, app2, app3, app1-bu, wphost02 | Reachable via `itpp-infra` key; full enumeration completed |
| MikroTik towers T01-T04, MP100 | **Not reachable** - L2TP/IPsec VPN down (charon not running, no `10.199.*` routes) |
| Home gateway 76.195.7.60 | Public endpoint confirmed reachable at network level (WG/L2TP listeners); not SSH-enumerated this session |
---
## 13. Consolidated severity summary
| Severity | Count | Highlights |
|---|---|---|
| Critical | 3 | Docker bypass exposing Wazuh stack (APP1-1); UniFi controller public (APP2-1); UNMS/UISP public (APP2-2) |
| High | 11 | Grafana public, browserless/camofox public, Wazuh 1514/1515 public, Twenty/Komodo public, Technitium open resolver, Traccar range public, Gitea/BookStack/MinIO/RAGFlow public, CloudPanel public, backup-restore UI public, Buzz relay public, apex DNS stale, SPF broken |
| Medium | 11 | Monitoring exporters on 0.0.0.0, ad-hoc http.server, socat→Hermes, MySQL X on `*`, FTP/SMTP/Varnish on 0.0.0.0, DMARC p=none / missing, weak L2TP crypto, plaintext VPN creds, legacy wphost02 live |
| Low | 4 | avahi on public iface, app1-bu stale WG rule + tailscale name drift, runcloud/node_exporter on wphost02 |
**Note on remediation:** all items above are Phase Two candidates. No firewall, service, DNS, or VPN configuration was changed during this audit.
audit/phase-one/findings/neteng-b.md
+197
View File
@@ -0,0 +1,197 @@
# NetEng-B - Network Segmentation and Blast-Radius Assessment (Phase One, Read-Only)
**Auditor:** NetEng-B
**Date:** 2026-08-13
**Scope:** Tier classification, inter-host segmentation, blast-radius chains, and product/tenant isolation across the ITPP estate (Core, app1, app2, app3, app1-bu, wphost02).
**Method:** Read-only discovery only (`ip`, `ufw status verbose`, `ss`, `tailscale status --json`, `docker network ls`, `docker inspect`, `cat` of config files, `curl`/TCP reachability probes between hosts). Builds directly on NetEng-A's raw port/DNS inventory (`neteng-a.md`) - no port/DNS re-enumeration performed here. No configuration was changed on any host.
**Severity legend:** Critical = a single compromise gives an attacker control of most or all of the estate, or of a system holding client/security data with no compensating boundary. High = a compromise crosses a trust boundary that should exist (internal-to-client, product-to-product) with material impact. Medium = defense-in-depth gap that increases blast radius but requires a second failure to be catastrophic. Low = hygiene issue with limited blast-radius effect.
---
## 1. Executive summary
- **There is no network segmentation between Core, app1, app2, app3, app1-bu, and wphost02.** All six hosts sit on public Netcup/Hetzner IP space with no VLAN, no subnetting, and no firewall boundary between each other. The only inter-host overlay is a **single flat Tailscale mesh** with **no ACL tags applied to any node** - every server and every personal device (phone, laptop, home router) is in one undifferentiated group with implicit allow-all reachability.
- Because UFW is bypassed by Docker on 4 of 6 hosts (confirmed by NetEng-A and re-verified here), the "segmentation" that exists on paper (UFW allow-lists) does not match what is actually reachable. Verified independently: Core -> app1 and app1 -> app2 both reach each other over their **public IPs**, not just Tailscale, with no filtering in between.
- **Compromise of any one of the six hosts gives an attacker a foothold that can reach every other host** over the public internet (all are mutually pingable/routable on public IPs) and, once inside, over the flat Tailscale mesh as well. There is no host that is network-isolated from the rest.
- **Products and internal ITPP operations are not isolated.** app1 runs the internal LLM gateway (LiteLLM/admin-ai), the internal CRM (Twenty), the security stack (Wazuh), and the reverse proxy for two products (Komodo, TransitPin) all as sibling Docker containers on one host with one Caddy instance. app3 runs a **single shared MySQL/Percona instance** behind ~24 CloudPanel sites that mix internal ops (mainwp, support, panel, ippadmin), client sites (katiewatts, modelortho, vigilanttac, boxpilotlogistics, timapta), and products (transitpin, myverdicttank, buzz, hexclave) with no per-tenant database server or credential vault separating them.
- **Tier misclassification is widespread**: management/admin consoles (Wazuh dashboard+indexer, UniFi controller, UNMS/UISP, CloudPanel admin, backup-restore UI, MinIO console, RAGFlow, Infinity DB, Komodo) are all internet-reachable, several unintentionally via the Docker/UFW bypass NetEng-A documented. These are control planes; none should be directly public.
---
## 2. Tier classification
Legend: **INT** = Internal-only (should never be public), **CLI** = Client-facing (legitimately public), **PROD** = Product/dev (micro-SaaS), **MGMT** = Management/admin control plane.
| Service | Host | Correct tier | Actually public? | Tier violation? |
|---|---|---|---|---|
| SSH (22) all hosts | all | MGMT | Yes (intended, key-only) | No - acceptable exposure |
| Caddy/nginx 80/443 (ingress) | Core, app1, app2, app3, wphost02 | CLI | Yes (intended) | No |
| Grafana :3002 | Core | MGMT | **Yes (explicit UFW allow)** | **Yes - dashboards public** |
| Prometheus/node_exporter/telegraf | Core | INT | No (UFW blocks; Docker bypass N/A, native binaries) | No, but bound 0.0.0.0 (Medium hygiene per NetEng-A) |
| browserless / camofox | Core | INT (SSRF-capable tool) | **Yes - Docker bypass** | **Yes** |
| WireGuard :51821 | Core, app1-bu | MGMT/VPN | Yes (intended) | No |
| Wazuh manager 1514/1515 | app1 | MGMT | Yes (intended, but should be source-restricted) | Partial - over-broad |
| Wazuh indexer :9200, dashboard :5601, API :55000, syslog 514/udp | app1 | MGMT | **Yes - Docker bypass** | **Yes - critical control plane public** |
| Twenty CRM :3003 | app1 | INT (holds client PII) | **Yes - Docker bypass** | **Yes** |
| Komodo core :9120 | app1 | MGMT (deploy/automation control plane) | **Yes - Docker bypass**, also proxied at komodo.iamgmb.com | **Yes** |
| LiteLLM/admin-ai (via Caddy) | app1 | MGMT | Yes (intended, proxied) | No (proxy correct; bypass risk is at Docker layer for adjacent services) |
| TransitPin static site | app1 | PROD | Yes (intended) | No |
| giftaroast.com site + API | app1 | PROD/CLI | Yes (intended) | No |
| UniFi controller 8080/8443/8843/8880 | app2 | MGMT | **Yes - Docker bypass** | **Yes - wireless/tower control plane public** |
| UNMS/UISP 81/8089/8444 | app2 | MGMT | **Yes - Docker bypass** | **Yes - entire WISP management plane public** |
| Technitium DNS :53 | app2 | INT (should be source-restricted to home router only) | **Yes, open resolver - Docker bypass defeats UFW source restriction** | **Yes** |
| Gitea SSH :3022 | app2 | MGMT (source code) | **Yes - Docker bypass** | **Yes** |
| BookStack :6875 | app2 | INT (internal wiki) | **Yes - Docker bypass** | **Yes** |
| support-api :6880 | app2 | INT/CLI boundary (unclear which) | **Yes - Docker bypass** | Flag for clarification |
| MinIO console :9001 | app2 | MGMT | **Yes - Docker bypass** | **Yes** |
| RAGFlow :9380-9392 | app2 | PROD/INT (unclear) | **Yes - Docker bypass** | Flag for clarification |
| Infinity DB :23817/23820 | app2 | INT (database) | **Yes - Docker bypass** | **Yes** |
| Traccar device listeners 5000-5150 | app2 | PROD (device protocol, must be public) | Yes (intended) | No - legitimate, but see APP2-4 in NetEng-A for hardening |
| Hudu, Dawarich | app2 | INT (internal IT docs / location tracking) | Proxied via Caddy (correctly) at hudu.itpropartner.com | No tier issue found at proxy layer |
| CloudPanel admin panel :8443 | app3 | MGMT | **Yes (explicit UFW allow)** | **Yes - hosting control plane for ~24 sites public** |
| backup-restore web UI :8090 | app3 | MGMT (destructive capability) | **Yes (explicit UFW allow)** | **Yes** |
| buzz-prod-relay :3000 | app3 | PROD | **Yes - Docker bypass** | Bypass mechanism wrong even though public exposure may be intended |
| MySQL X protocol :33060 | app3 | INT (database) | Bound `*` but currently UFW-blocked | No public exposure confirmed, hygiene flag only |
| CloudPanel client/product sites (24 total) | app3 | Mixed CLI/PROD/INT (see section 5) | Yes (intended, various) | See section 5 for isolation gap, not tier gap |
| wphost02 nginx-rc 80/443 | wphost02 | CLI (legacy) | Yes (intended, legacy) | No tier issue, but should be decommissioned per NetEng-A |
### Tier-classification counts (from table above)
- **INTERNAL-ONLY services incorrectly public: 12** (Grafana, browserless, camofox, Wazuh indexer/dashboard/API/syslog treated as one group counted once = Wazuh stack, Twenty CRM, Technitium open resolver, BookStack, MinIO console, Infinity DB, plus MySQL X hygiene flag)
- **MANAGEMENT/ADMIN consoles incorrectly public: 6** (Wazuh dashboard, UniFi controller, UNMS/UISP, CloudPanel admin, backup-restore UI, Komodo core)
- **CLIENT-FACING correctly public: 6** (Core/app1/app2/app3/wphost02 web ingress, Traccar device range)
- **PRODUCT/DEV correctly public (by design): 4** (TransitPin, giftaroast.com, buzz relay -- though via wrong mechanism, Traccar)
- **Ambiguous tier, needs Germaine clarification: 2** (support-api, RAGFlow)
---
## 3. Segmentation map
```
PUBLIC INTERNET
|
+--------------------------------+--------------------------------+
| | | | |
Core app1 (Netcup) app2 (Netcup) app3 (Netcup) app1-bu / wphost02
152.53.192.33 152.53.36.131 152.53.39.202 152.53.241.111 (Hetzner, separate /32s)
| | | | |
|<===============+===============+===============+================|
FLAT PUBLIC IP MESH -- every host reaches every other host's
public IP directly. No VLAN. No inter-host firewall. Verified:
Core->app1:*, app1->app2:* all reachable on PUBLIC IPs, unfiltered
for whatever Docker/UFW leaves open on the receiving end.
| | | | |
+----------------+---------------+---------------+----------------+
|
TAILSCALE MESH (100.x.x.x/32 each)
tailscale status --json: NO "Tags" field on ANY
node (Core, app1, app2, app3, app1-bu-1). Default
tailnet ACL = allow-all between all nodes.
app3 has no tailscaled at all listed in this scan --
wait, confirmed: app3 100.72.15.12 IS on tailnet.
wphost02 has NO tailscale client installed at all --
it is reachable ONLY over the flat public internet.
|
+---------------+----------------+------------------+
| | | |
Core app1 app2 app3
(+ personal devices: iphone, m4-mac-mini, ipp-g-lap, home
router "liberty-udm-pro" -- ALL in the SAME flat tailnet group
as the six production servers, no tag-based isolation)
app1-bu-1 (Hetzner standby) is ALSO in this same flat tailnet.
wphost02 -- OUTSIDE the tailnet entirely. Reachable from Core/app1/
app2/app3 only via public internet + SSH key. One-directional trust:
Core has an autossh reverse tunnel INTO wphost02 (mysql-tunnel,
127.0.0.1:33060 -> wphost02:3306) plus a root SSH cron backup job.
wphost02 has no client that can reach back into Core/app1/app2/app3.
INSIDE EACH HOST: dozens of isolated Docker bridge networks per
compose stack (e.g. app1 has 15 separate bridge networks: litellm,
twenty, komodo, n8n, docuseal, etc.) -- this is REAL intra-host
container isolation. But it is undermined by the Docker-publish-vs-UFW
bypass NetEng-A documented: many of those "isolated" containers punch
a hole straight to 0.0.0.0 on the host's public interface, which
erases the isolation the bridge network was providing.
```
**Bottom line: there is effectively ONE trust zone across the entire estate.** The only segmentation primitives in play (UFW allow-lists, per-stack Docker bridge networks, Tailscale) are all either bypassed (UFW/Docker), unused for isolation (Tailscale has no ACL tags), or absent entirely between hosts (no VLAN/subnet separation of any kind exists between Core/app1/app2/app3/app1-bu; wphost02 is flat-public with an SSH+tunnel trust relationship back to Core).
---
## 4. Per-host blast-radius table
| Host | If compromised, attacker gets... | Falls in blast radius | Evidence |
|---|---|---|---|
| **Core** | Root on the Hermes agent host: WireGuard keys to home network + WISP towers (10.77.0.0/24, 10.1.0.0/16, 10.2.0.0/16, 172.16.1.0/24, 172.18.18.0/24 routed via wg0), plaintext L2TP/IPsec creds to home gateway, the `itpp-infra` SSH private key (used to reach ALL 6 hosts), the autossh tunnel credential path into wphost02's MySQL, Grafana/Prometheus telemetry, Tailscale identity (can pivot to every tailnet peer incl. personal devices) | app1, app2, app3, app1-bu, wphost02 (via SSH key + tunnels), home network + WISP towers (via WireGuard), personal devices on tailnet (phone, laptop, home router) | `ip route` shows wg0 routes; `/root/.ssh/itpp-infra` used identically against all 6 hosts in this audit; crontab shows nightly root SSH job to wphost02; NetEng-A VPN-2 documents plaintext creds |
| **app1** | Root on the box hosting Wazuh (SIEM for the whole estate), Twenty CRM (client PII), LiteLLM/admin-ai (the AI control plane and its API keys), Komodo (deployment/automation control plane), n8n (workflow automation with stored credentials) | Every host Wazuh agents report from (SIEM blast radius = estate-wide visibility loss/tamper), every client whose PII sits in Twenty, every downstream system Komodo can deploy to, every credential n8n workflows hold | Docker inspect showed `PG_DATABASE_URL`, `APP_SECRET`, `REDIS_URL` for Twenty; `KOMODO_JWT_SECRET`/`KOMODO_WEBHOOK_SECRET` for Komodo; `DATABASE_URL` for LiteLLM postgres; Caddyfile proxies vault/n8n/ai/admin-ai/noc/wz/komodo/transitpin all from this one host |
| **app2** | Root on the box running UniFi controller (wireless/tower control), UNMS/UISP (entire WISP customer/device management plane), Gitea (source code + SSH deploy keys), Hudu (IT documentation, likely holds more credentials/secrets), Traccar (fleet GPS data), BookStack, RAGFlow, Infinity DB, MinIO | The WISP tower network and its customers, all git repos + CI secrets on Gitea, all documented IT credentials in Hudu, fleet-tracking customer data, any data indexed in RAGFlow/Infinity | `docker inspect` showed `UNMS_PG_HOST`, `SECURE_LINK_SECRET`, Hudu `SECRET_KEY_BASE`/`S3_SECRET_ACCESS_KEY`, Gitea SSH port 3022 public per NetEng-A |
| **app3** | Root on the box running CloudPanel (control plane for ~24 sites) and ONE shared MySQL/Percona instance backing internal ops sites, client sites, and product sites simultaneously | All ~24 CloudPanel-hosted sites: internal (mainwp, support, panel, ippadmin), clients (katiewatts, modelortho, vigilanttac, boxpilotlogistics, timapta), products (transitpin, myverdicttank, buzz, hexclave) - a single DB-engine compromise threatens every one of them at once | `ss -tlnp` showed one `mysqld` process on 127.0.0.1:3306/*:33060; `docker ps` on app3 showed buzz-prod-postgres, hexclave-postgres as separate containers, but CloudPanel's own PHP/static sites run against the single host-level Percona instance shown above |
| **app1-bu** | Root on the warm-standby box. Minimal live footprint (per NetEng-A). Sync direction is **pull-only**: `hermes-standby-sync.sh` pulls from Wasabi S3, does not push to or read live secrets directly off Core over the network | Limited - compromise here does not directly expose Core, because sync is one-way pull from S3, not a live network tunnel to Core. Attacker would get whatever is in the last S3 snapshot (which may include full Hermes state/secrets) | `hermes-standby-sync.sh`: `aws s3 sync s3://hermes-vps-backups/live/ ...`; `authorized_keys` on app1-bu matches Core's `itpp-infra.pub`, meaning Core (not app1-bu) is the initiator of any direct SSH, consistent with pull-based design |
| **wphost02** | Root on legacy WordPress/RunCloud host. Directly exposes MySQL 3306 to Core via the standing autossh tunnel (`-L 127.0.0.1:33060:localhost:3306`) | Core's tunneled MySQL access (`apextrackexperience` DB per `service-health-check.sh`); any WordPress sites still live here | `ps aux` on Core shows `autossh ... -L 127.0.0.1:33060:localhost:3306 -N root@5.161.62.38`; `service-health-check.sh` references `MySQL SSH tunnel (wphost02)` and `MySQL database (apextrackexperience)` |
---
## 5. Blast-radius dependency chains (worst-case, critical services)
1. **Core mysql-tunnel -> wphost02**: Core maintains a permanent autossh reverse tunnel (`127.0.0.1:33060 -> wphost02:3306`) plus a nightly cron root-SSH backup job to wphost02. If Core is compromised, the attacker inherits this tunnel and reaches wphost02's database directly, no additional credential theft required. If wphost02 is compromised first, the exposed MySQL on `localhost:3306` there is only reachable over the SSH tunnel Core already holds the key for - so wphost02 compromise does not directly threaten Core, but Core compromise fully threatens wphost02's DB.
2. **app1 super-search / MCP tooling -> Core**: app1 hosts multiple MCP servers (`mcp-browser`, `mcp-email`, `mcp-git`, `mcp-filesystem`, `super-search`) that are part of the same automation fabric as Core's `super-search` (:8899, allow-listed to `172.17.0.0/16` on Core). A compromised app1 MCP container that can reach Core's Docker subnet peer range could attempt to reach Core's super-search MCP if any cross-host trust exists in the MCP orchestration layer. Full confirmation of live cross-host MCP calls was not possible read-only this session - **flagged for Phase Two verification**, but the underlying network path (Core and app1 both reachable from each other over public IP and Tailscale with no ACL) means the network layer would not stop such a call if the application layer permits it.
3. **Core Wazuh agents -> app1 Wazuh manager**: every host in the estate almost certainly runs a Wazuh agent reporting to app1's manager (1514/1515, intentionally public per NetEng-A APP1-2). If app1's Wazuh stack is compromised (already publicly reachable via the Docker/UFW bypass - APP1-1, Critical), the attacker controls the SIEM for the entire estate: they can see all security telemetry and potentially inject false negatives, blinding detection across Core/app2/app3/app1-bu/wphost02 simultaneously. This is the single highest-leverage compromise in the estate.
4. **app3 shared Percona -> all 24 CloudPanel sites**: any SQL injection, credential leak, or root compromise on app3 threatens every site's database in one blast, because there is one MySQL engine process backing internal, client, and product sites alike (see section 6).
5. **app1-bu standby -> S3 snapshot, not live Core**: because sync is pull-only from Wasabi S3 rather than a live tunnel to Core, app1-bu compromise does NOT directly hand an attacker live access to Core. This is a **positive isolation finding** - call it out as something already done right, worth preserving in Phase Two hardening (don't accidentally add a live push tunnel later).
---
## 6. Product/tenant isolation findings
- **app3 (CloudPanel) hosts ~24 sites behind one shared MySQL/Percona instance** with no per-tenant database server, container, or credential vault: `apx, boxpilotlogistics, buzz, clp, debtreecoveryexperts, docs, drecovery, forms, gmb, hexclave-api, hexclave-dash, iAmGMB, intelsight, ippadmin, katiewatts, mainwp, mockups, modelortho, modelorthowww, myverdicttank, myvoip, panel, proposals, support, timapta, transitpin, transitpin-dash, vigilanttac, voipsimplicity`. This list mixes **internal ITPP ops** (mainwp, support, panel, ippadmin, docs, proposals, mockups), **client sites** (katiewatts, modelortho/modelorthowww, vigilanttac, boxpilotlogistics, timapta), and **products** (transitpin/transitpin-dash, myverdicttank, buzz, hexclave-api/dash) on the same host, same web server, same database engine. Some products (buzz, hexclave) do run their own dedicated Postgres containers alongside the shared Percona instance, which is better isolation for those two, but the CloudPanel-native PHP/static sites (the majority) share the one host-level MySQL.
- **app1 mixes internal control-plane services with two client-facing products in the same Docker/Caddy stack**: Wazuh (SIEM), Twenty CRM (internal, holds PII for debt-recovery-experts.com), LiteLLM/admin-ai (AI gateway), and Komodo (deploy automation) run as containers alongside TransitPin and giftaroast.com (products) and crm.debtrecoveryexperts.com (client-branded CRM instance). A single Caddy instance and a single Docker host boundary is the only thing separating "internal admin tooling" from "product the client interacts with directly."
- **No evidence of per-product credential vaults.** Each product/service has its own DB password baked into its own container's env (Twenty's `APP_SECRET`, Komodo's `KOMODO_JWT_SECRET`, buzz's `BUZZ_S3_SECRET_KEY`, hexclave's `STACK_SERVER_SECRET`, etc.) - that part is correctly per-service. But nothing enforces that these secrets stay scoped: any root compromise of the host reads all of them from `docker inspect`, as this audit itself demonstrated read-only.
- **HotNow was not found** deployed anywhere in this pass (no matching directory/container/proxy entry across app1/app2/app3). **Flag for Phase Two / Git-A cross-reference**: confirm whether HotNow is live, decommissioned, or hosted somewhere outside the six audited servers.
- **VerdictTank / RFP Tank**: found evidence of `myverdicttank` CloudPanel site on app3 and a `/root/.verdicttank-key.json` LiteLLM API key file on app1 (model access key scoped to specific models, not a raw cloud credential) - so VerdictTank spans app1 (LLM access) and app3 (web/DB), with no isolation boundary tying those two halves together other than the key itself.
---
## 7. Shared credentials / shared reverse-proxy / shared database findings
- **Shared reverse proxy (app1 Caddy)**: one Caddyfile terminates TLS for internal tools (vault, n8n, ai, admin-ai, noc, wz/Wazuh) and product/client domains (giftaroast.com, crm.debtrecoveryexperts.com, komodo.iamgmb.com, transitpin.iamgmb.com) side by side. A Caddy-level misconfiguration or compromise affects both tiers simultaneously.
- **Shared reverse proxy (app3 CloudPanel/nginx)**: same pattern - one web-server control plane (CloudPanel, itself a public admin panel) fronts internal, client, and product sites.
- **Shared database engine (app3 Percona/MySQL)**: single instance backs the majority of the 24 CloudPanel sites across all three tiers, as detailed in section 6.
- **Shared Tailscale mesh with no ACL segmentation**: all 6 production hosts AND personal devices (iPhone, MacBook, Mac Mini, home router) are members of one flat tailnet group with no tags observed on any peer. A compromised personal device is one hop from every production server over Tailscale; a compromised production server is one hop from personal devices.
- **Shared SSH key (`itpp-infra`) across all 6 hosts**: this is an operational convenience but means loss of that one private key (which lives on Core) compromises SSH access to the entire estate at once. This is the single most consequential shared credential in the estate.
---
## 8. Findings (severity-rated)
| ID | Severity | Finding | Evidence | Why it matters |
|---|---|---|---|---|
| NETB-1 | **Critical** | **No network segmentation exists anywhere in the estate.** Core, app1, app2, app3, app1-bu are flat on public Netcup/Hetzner IP space with no VLAN or subnet isolation, and the Tailscale overlay that connects most of them has no ACL tags on any node (default allow-all). wphost02 sits fully outside the mesh, reachable only over the open internet. | `tailscale status --json` on Core/app1/app2/app3/app1-bu shows zero `Tags` entries for any peer; `ip route`/`ping` tests confirmed Core, app1, app2 mutually reach each other's PUBLIC IPs directly with no filtering observed between them | If any one host is popped, the attacker is not contained to that host's blast radius alone. They inherit a direct network path (public IP or Tailscale) to every other host and to personal devices, turning a single compromise into an estate-wide incident with no lateral-movement friction. |
| NETB-2 | **Critical** | **Wazuh SIEM stack (indexer, dashboard, manager API) is the single highest-leverage target in the estate** because (a) it is publicly exposed via the Docker/UFW bypass NetEng-A found (APP1-1) and (b) compromising it gives an attacker visibility into and potential control over security telemetry for every other host, which likely reports to it as a Wazuh agent. | app1 Docker/iptables inspection (NetEng-A APP1-1) + this audit's confirmation that app1 accepts inbound connections on its public IP from other estate hosts with no gate | Losing the SIEM is worse than losing any single production host: it blinds detection across the whole estate at the exact moment an attacker needs detection to fail, and it sits on a host with no segmentation from the rest of the fleet. |
| NETB-3 | **Critical** | **app3 shares one MySQL/Percona instance across ~24 sites spanning internal ops, client sites, and micro-SaaS products** with no per-tenant database server or vault boundary. | `ss -tlnp` shows a single `mysqld` on 127.0.0.1:3306/*:33060; CloudPanel site list (`apx, boxpilotlogistics, buzz, ... myverdicttank, panel, support, transitpin, vigilanttac, voipsimplicity`, ~24 total) all on one host | A SQLi or credential leak against ANY one of the 24 sites is a plausible path to every other site's data on the same instance, including internal ITPP operational data (mainwp, support, panel) sitting next to client and product databases. |
| NETB-4 | **High** | **Management/admin control planes are public with no compensating network boundary**: UniFi controller (app2), UNMS/UISP (app2), CloudPanel admin panel (app3), backup-restore UI (app3), Komodo (app1). Several via unintended Docker/UFW bypass, some via explicit-but-broad UFW rules. | NetEng-A APP1-1/2/3, APP2-1/2, APP3-1/2; cross-referenced here against tier classification (section 2) | Each of these is a "master key" for a whole subsystem (wireless network, ISP customer management, ~24 hosted sites, backup/restore capability, deploy automation). None should be reachable without a network-level gate (VPN, allow-list, or reverse-proxy auth) in addition to application login. |
| NETB-5 | **High** | **Internal tooling and client-facing/product systems share the same Docker host and reverse proxy with no boundary** on app1 (Wazuh + Twenty CRM + LiteLLM + Komodo alongside TransitPin, giftaroast.com, crm.debtrecoveryexperts.com) and app3 (CloudPanel mixing mainwp/support/panel with client and product sites). | app1 Caddyfile listing both internal and product/client domains; app3 site directory listing (section 6) | A compromise anywhere on either host crosses tiers for free - there is no reason a breach that starts in a low-value product site should reach the internal CRM or deploy-automation control plane, but on these hosts it can, because nothing stops it at the network layer. |
| NETB-6 | **High** | **Estate-wide single SSH key (`itpp-infra`) is the sole authentication factor for root on all 6 hosts**, and it lives on Core alongside a live autossh tunnel into wphost02's database and WireGuard routes into the home network and WISP towers. | This audit used the same key against all 6 hosts successfully; `ip route`/`ps aux` on Core show the WireGuard routes and the standing tunnel | Core is a de facto master key for the entire estate plus the home network and WISP towers. Its compromise is strictly worse than any other single host's compromise and there is no network segmentation limiting the blast radius once that key is in an attacker's hands. |
| NETB-7 | **Medium** | **Tailscale mesh includes personal devices (phone, laptop, Mac Mini, home router) in the same untagged group as all six production servers.** | `tailscale status` output on every host lists `iphone-15-pro-max`, `m4-mac-mini`, `ipp-g-lap`, `liberty-udm-pro` alongside `core/app1/app2/app3/app1-bu` with no tag differentiation | A compromised personal device (phishing, stolen laptop, malicious app) is one network hop from production infrastructure with no policy boundary in between. Tailscale ACL tags exist specifically to prevent this and are not being used. |
| NETB-8 | **Medium** | **Product credentials for HotNow/VerdictTank/RFP Tank/TransitPin are scoped per-service but not vaulted**, and the products themselves are split across hosts (app1 for LLM access key, app3 for web/DB) with no documented boundary tying the split together. | `/root/.verdicttank-key.json` on app1 (model-scoped LiteLLM key); `myverdicttank`, `transitpin`, `transitpin-dash` CloudPanel sites on app3 | If the split is intentional, it should be documented as an architecture decision with an explicit trust boundary; as observed, it looks like an artifact of convenience rather than a designed isolation boundary, which risks silent credential sprawl as more products are added. |
| NETB-9 | **Low** | **app1-bu standby sync design is pull-only from S3 (not a live tunnel to Core)** - noted as a positive control worth preserving explicitly in Phase Two policy, not a finding to remediate. | `hermes-standby-sync.sh`: `aws s3 sync s3://hermes-vps-backups/live/ ...`; `authorized_keys` on app1-bu matches Core's public key (Core-initiated direction only) | Call this out so future changes to the standby (e.g. adding a live replication tunnel) are evaluated against the blast-radius benefit of the current pull-only design before being adopted. |
### Severity summary
- Critical: 3 (NETB-1, NETB-2, NETB-3)
- High: 3 (NETB-4, NETB-5, NETB-6)
- Medium: 2 (NETB-7, NETB-8)
- Low: 1 (NETB-9, informational/positive-control note)
---
## 9. Access limitations / items requiring Phase Two follow-up
- Could not confirm from read-only evidence whether app1's MCP tooling makes live cross-host calls into Core's super-search MCP (:8899) - the network path exists and is unfiltered, but application-layer confirmation needs a Phase Two trace (log review, not a live test).
- Could not run `clpctl db:show:master-credentials` per-site on app3 (would require a site name argument and returns credentials - out of scope for a read-only, no-credential-exposure audit) to confirm whether each CloudPanel site has a distinct MySQL user/schema on the shared instance versus a fully shared root-level credential. Section 6's finding stands regardless (one instance = one blast radius for the engine itself), but Phase Two should verify per-site credential scoping.
- Tailscale ACL policy file itself (the tailnet admin console's actual ACL JSON) was not inspected - only the absence of `Tags` on every peer via `tailscale status --json`, which is strong but indirect evidence of default-allow-all. Phase Two with tailnet-admin access should pull the actual ACL policy to confirm.
- HotNow was not located on any of the six audited servers. Needs confirmation from Git-A/Docs-W on whether it is live elsewhere, decommissioned, or renamed.
---
**Note on remediation:** all items above are Phase Two candidates. No firewall, container, DNS, database, or Tailscale configuration was changed on any host during this audit.
audit/phase-one/findings/sec-a.md
+245
View File
@@ -0,0 +1,245 @@
# Sec-A Findings: IAM, Accounts, Secrets Location, and Least Privilege
Auditor: Sec-A (IAM and secrets inventory)
Scope: Account inventory, sudo/root privilege, SSH access and key attribution, MFA coverage on admin surfaces, secrets location (no values captured, only presence/staleness), shared vs personal credential use, least-privilege posture.
Mode: READ-ONLY. All evidence below comes from local audit capture files already on disk (`/root/audit_core.txt`, `/root/audit_app1.txt`, `/root/audit_app2.txt`, `/root/audit_app3.txt`, `/root/audit_app1bu.txt`, `/root/audit_wphost02.txt`), collected in a prior session via read-only SSH/curl. No new commands were run against remote hosts to produce this file. All credential values are redacted as `[REDACTED]`; this document names locations and types only.
Overlap note: hardening, patch posture, MFA configuration detail (per-tool), and Wazuh/logging coverage are Sec-B's domain (see sec-b.md). This file focuses on accounts, SSH, secrets location, and least-privilege, and cross-references Sec-B's MFA table rather than repeating it in full.
---
## 1. Per-Server Account Inventory
Six hosts: Core (Netcup, this host), app1 (152.53.36.131), app2 (152.53.39.202), app3 (152.53.241.111), app1-bu (Hetzner, warm standby), wphost02 (Hetzner, legacy WordPress/RunCloud).
| Host | Total /etc/passwd entries | Interactive shell accounts (non-system) | Notable interactive accounts | sudo group members | Passwordless sudo (sudoers.d) |
|---|---|---|---|---|---|
| Core | 34 | `ippadmin`, `postgres`, `scanuser` (3) | `postgres` has `/bin/bash` (service account with a real shell, unusual), `scanuser` is locked/no-password, purpose not identified in captured data, flag for Phase Two follow-up | `ippadmin` | `ippadmin ALL=(ALL) NOPASSWD:ALL` |
| app1 | 24 | `ippadmin` (1) | Clean, minimal footprint | `ippadmin` | none in sudoers.d beyond README (no explicit NOPASSWD entry for ippadmin found on app1, unlike Core/app2/app3, worth confirming) |
| app2 | 25 | `ippadmin`, `unms` (2) | `unms` service account is in the `docker` group, meaning it can run containers with root-equivalent power (see Least-Privilege section) | `ippadmin` | `ippadmin ALL=(ALL) NOPASSWD:ALL` |
| app3 | 37 | `ippadmin` plus roughly 28 per-client/per-site accounts (`gmb`, `myverdicttank`, `mainwp`, `boxpilotlogistics`, `modelortho`, `intelsight`, `mockups`, `modelorthowww`, `timapta`, `hexclave-api`, `apx`, `panel`, `voipsimplicity`, `transitpin-dash`, `vigilanttac`, `proposals`, `transitpin`, `docs`, `buzz`, `support`, `myvoip`, `hexclave-dash`, `iAmGMB`, `drecovery`, `forms`, `katiewatts`, `clp`, `debtrecoveryexperts`) | This is CloudPanel's per-site account model: one Linux user per hosted site/client. Large blast surface simply by account count, even though most appear unused for SSH (see Section 2) | `ippadmin` | `ippadmin ALL=(ALL) NOPASSWD:ALL`; also `clp ALL=(ALL) NOPASSWD:ALL` (CloudPanel's own management account) plus a wrapper rule `ALL ALL=(ALL) NOPASSWD: /usr/bin/clpctlWrapper` letting any account run the CloudPanel control wrapper as root |
| app1-bu | 32 | none beyond `root` | Warm standby has essentially no interactive non-root accounts provisioned yet (matches its role as a failover target, not yet fully onboarded) | none (sudo group empty) | `root ALL=(ALL) NOPASSWD:ALL` via cloud-init default, standard for that image |
| wphost02 | 37 | `runcloud`, `ippadmin` (2) | `runcloud` is the RunCloud panel's management account | `ippadmin` | `root ALL=(ALL) NOPASSWD:ALL` (cloud-init default) and `ippadmin ALL=(ALL) NOPASSWD:ALL` |
Orphaned/unclear-purpose accounts:
- **Core: `scanuser`** (UID 1001, locked password, `/bin/bash` shell, home `/home/scanuser`). No corresponding SSH key, cron job, or docker context found in the captured data tying this account to an active purpose. Flag as a candidate for removal or documentation in Phase Two; do not assume it is safe to delete without confirming with Germaine first (read-only rule).
- **app3: roughly 28 per-client accounts.** Most have empty `authorized_keys` files (see Section 2), meaning they exist as CloudPanel site-owner accounts but show no evidence of direct SSH login capability. This is consistent with CloudPanel's model (site isolation, not direct-login accounts) but should be confirmed against actual client billing/engagement status in Phase Two; several of these usernames (`debtrecoveryexperts`, `boxpilotlogistics`, `transitpin`, `hexclave-api`) correspond to active named client projects, others (`katiewatts`, `iAmGMB`) look personal/less clearly tied to a current engagement and are worth a lifecycle check.
- **wphost02: `runcloud` and `mysql`/`memcache`/`beanstalkd` service accounts.** These are RunCloud-panel defaults consistent with a legacy managed WordPress host; no evidence of misuse, flagged only for completeness since wphost02 is described in the brief as a legacy/sunset host.
Password/shadow posture: `root:PASSWORD_HASH_SET` on Core, app1, app2, app3, and wphost02 (root has a live password hash and could theoretically log in with a password if PermitRootLogin allowed it; Sec-B confirms PermitRootLogin is key-only across the estate, which mitigates this). On app1-bu, root shows `LOCKED_BUT_HASH_PRESENT`, a slightly different but comparable state. All other system accounts show `LOCKED_NO_PASSWORD`, which is expected and healthy for service accounts.
---
## 2. SSH Access Inventory
### 2.1 Key inventory found across `authorized_keys` files
| Key comment/label | Fingerprint (truncated) | Seen on |
|---|---|---|
| `itpp-main-server` | SHA256:oqKRvPA...D7E | Core (root and ippadmin) |
| `germaine@itppartner` | SHA256:dDbLH+b...y3I | Core, app1, app2, app3 (root and ippadmin on each) |
| `wisp-backup` | SHA256:MxQw1oh...DcI | Core (root and ippadmin) |
| `itpp-infra` | SHA256:Jxh0bbT...8uQ | Core, app1, app2, app3, app1-bu (root and/or ippadmin on each); also present on wphost02 under a second fingerprint variant, see below |
| `g@germaine@itpropartner.com` | SHA256:7QuUx/s...8lc | Core (root only) |
| `RunCloud_Server_Service` | SHA256:qMm5i4/...IUo | wphost02 (root only, RunCloud-managed) |
| `itpp-infra RUNCLOUD1783623009` / `itpp-infra` (second fingerprint) | SHA256:jnmiJb9...v3U | wphost02 (root only) |
### 2.2 Blast radius assessment
**Confirmed live (not just documented):** the `itpp-infra` key (fingerprint `SHA256:Jxh0bbT9dUV3q1DYYB3hHyhy/1TDj7Q8U4xrVmB38uQ`) is present in `authorized_keys` on Core, app1, app2, app3, and app1-bu, five of the six servers in scope. This confirms the prior session's documentation-based claim with live data. wphost02 has a *different* key under the same "itpp-infra" label/comment (different fingerprint), so it is not the identical key, but it is functionally the same access pattern using the same naming convention.
**Severity implication:** compromise of the single private key corresponding to fingerprint `SHA256:Jxh0bbT9dUV3q1DYYB3hHyhy/1TDj7Q8U4xrVmB38uQ` grants SSH access to 5 of 6 servers in the estate (Core, app1, app2, app3, app1-bu), each as root or root-equivalent (via ippadmin's passwordless sudo). This is the single largest blast-radius item in the IAM domain. There is no key segmentation by host, role, or trust tier; one credential spans the entire estate except the legacy wphost02 host.
**`germaine@itppartner`** key is present on Core, app1, app2, and app3 (4 of 6 hosts), a second broad-access personal key, smaller blast radius than `itpp-infra` but still spanning 4 hosts.
**Single-purpose keys** (`itpp-main-server`, `wisp-backup`, `g@germaine@itpropartner.com`) are scoped to Core only, a healthier pattern; these do not multiply blast radius.
**app3 per-client accounts:** of the roughly 28 per-client home directories checked, all sampled `authorized_keys` files under those accounts were empty (no keys present). This means those accounts cannot currently be reached by direct SSH key login, which is a good containment property for a shared multi-tenant host, assuming CloudPanel manages access to those accounts through its own panel rather than SSH.
**app1-bu (warm standby):** only the `itpp-infra` key is authorized, and only for root; there is no `germaine@itppartner` personal key here. This is a narrower access surface, appropriate for a less frequently accessed failover host, but means the standby depends entirely on one shared key with no personal-key fallback.
**wphost02:** RunCloud's own service key plus two `itpp-infra`-labeled keys with different fingerprints are present; no personal (`germaine@itppartner`) key was found on this host, consistent with it being flagged as legacy/lower-touch in the brief.
### 2.3 Login activity signal
Last-login data was only meaningfully populated on app1-bu (all accounts "never logged in", consistent with a cold standby that has not yet been used) and wphost02 (root last login 2026-07-21 from 76.195.7.60; ippadmin last login 2025-12-31 from the same IP). Core, app1, app2, and app3 returned no populated last-login rows in the captured data, meaning login history could not be independently verified for those four hosts from this dataset. This is a visibility gap, not a finding of absence.
---
## 3. MFA Matrix (Cross-Reference to Sec-B)
Sec-B's findings file (sec-b.md, Section 3) contains the full MFA coverage table across 11 admin surfaces and is the authoritative source for MFA configuration detail. Sec-A's contribution here is the account-and-access angle for each surface: how many people can reach it and via what credential, since that is what MFA (or its absence) is meant to protect.
| Admin Surface | Host | Accounts with access (per this session's data) | MFA status (per Sec-B) | Sec-A note |
|---|---|---|---|---|
| Gitea | app2 | Single admin account (`ippadmin`, is_admin=true, confirmed live in the prior session via API) | Capability exists, not confirmed enforced; open registration | One shared admin account for the entire code-hosting platform is itself an IAM concentration risk independent of MFA; see Section 5 |
| Grafana | Core | Uses `admin`/`[REDACTED default password]` per Sec-B | No MFA, default credential in use | Confirms Sec-B's Critical finding from the accounts angle: this is not even a personal account, it is a shared default account |
| CloudPanel | app3 | Single admin account `gmb` observed via `clpctl user:list` (per prior session) | Capability exists, adoption unconfirmed | Single admin account matches the single-admin pattern seen on Gitea; worth asking Germaine whether other staff need scoped CloudPanel accounts rather than sharing `gmb` |
| Technitium DNS | app2 | Environment shows `DNS_SERVER_ADMIN_PASSWORD` set to the literal default placeholder string | No MFA; default-credential pattern | Sec-A could not confirm the live in-app credential value without an authenticated read, which the read-only rule prohibits; flagged as an access-verification gap for Phase Two |
| Wazuh dashboard, Hudu, UniFi, UNMS/UCRM, LiteLLM admin, Vaultwarden, Dawarich, Traccar | app1/app2 | No per-user account enumeration was possible from the captured local-file data (these are app-level accounts inside each service's own database, not OS accounts) | See Sec-B's table | This is a genuine visibility gap: Sec-A cannot state how many people have login credentials to these consoles without an authenticated read of each app's user table, which is out of scope for read-only discovery. Flagging per brief rule 4 (flag insufficient access/visibility, do not guess). |
**MFA gap count (IAM-relevant): 8 of 8 checkable admin surfaces show no confirmed enforced MFA** (Gitea, Grafana, CloudPanel, Technitium, Wazuh dashboard, Hudu, UniFi, UNMS/UCRM), consistent with Sec-B's estate-wide finding.
---
## 4. Secrets-Location List
No credential values are reproduced below. Entries are location, type, and staleness/rotation signal only.
### 4.1 Confirmed live secrets locations (`.env` files, values redacted at capture time)
| Host | Path | Secret types present | Last modified | Staleness signal |
|---|---|---|---|---|
| Core | `/opt/shopping-cart/.env` | Amazon Associates tag, SMTP credentials | 2026-07-27 | Recent |
| Core | `/opt/mooresunnydaze/.env` | Admin API key, Stripe secret key, Stripe webhook secret | 2026-08-07 | Recent |
| Core | `/opt/hermes-voice/.env` | Hermes API key/session key, xAI API key | 2026-07-28 | Recent |
| Core | `/opt/voice-agent/.env` | Hermes API key/session key | 2026-07-26 | Recent |
| app1 | `/docker/n8n/.env` | n8n encryption key, Postgres password | 2026-07-17 | Recent |
| app1 | `/root/docker/docuseal/data/docuseal/docuseal.env` | Database URL, secret key base | 2026-07-28 | Recent |
| app1 | `/root/docker/litellm/.env` | LiteLLM master key, salt key, Postgres password, UI credentials | 2026-07-15 | ~1 month old at capture time |
| app1 | `/root/docker/litellm/.env.pre-keyfix-20260714-130347` | Same secret types as above (pre-rotation snapshot) | 2026-07-14 | **Stale duplicate.** This is a leftover pre-rotation backup sitting next to the live config; the filename itself documents a key-fix event on 2026-07-14, meaning this file likely contains a superseded credential set that was never deleted. Medium finding, see Section 7. |
| app1 | `/root/docker/super-search/.env` | Exa, Firecrawl, OpenCorporates API keys | 2026-07-15 | Recent |
| app1 | `/root/docker/twenty/.env` | Encryption key and app config | 2026-07-28 | Recent |
| app1 | `/root/docker/wazuh/.env` | Wazuh build/version config (lower sensitivity, mostly version pins) | 2026-07-19 | Recent |
| app2 | `/opt/bookstack/config/www/.env` | DB, SMTP, S3 credentials, two-factor key | present | Not independently dated in this pass |
| app2 | `/root/docker/dawarich/.env` | Database password, Postgres password, secret key base | 2026-07-22 | Recent |
| app2 | `/root/docker/hudu/.env` | Hudu app secrets | present | Not independently dated in this pass |
| app2 | `/root/docker/technitium/docker-compose.yml` (inline env, not a `.env` file) | `DNS_SERVER_ADMIN_PASSWORD` set to the literal default placeholder string in the compose file itself | 2026-08-04 | Recent file, but the value itself is a stale/never-rotated default, see Section 7 |
| app3 | `/home/clp/htdocs/app/files/.env` | Application credentials (CloudPanel-managed app) | present | Live copy |
| app3 | `/home/clp/backups/2026-08-11_04-15-01/app/files/.env`, `.../2026-08-12_04-15-01/...`, `.../2026-08-13_04-15-01/...` | Same credential set as above | 2026-08-11, -12, -13 | **Three consecutive daily backup snapshots each retain a full plaintext copy of the same `.env`.** Every backup rotation multiplies the number of at-rest plaintext credential copies without any additional access control on the backup directory itself. Medium finding, see Section 7. |
| app3 | `/opt/buzz/deploy/compose/.env` | Buzz relay service credentials | present | Not independently dated in this pass |
| app3 | `/opt/docs-auth/docs-auth.env` | Docs site auth credentials | present | Not independently dated in this pass |
| app3 | `/opt/hexclave/hexclave.env` | Hexclave app credentials | present | Not independently dated in this pass |
| app3 | `/var/www/msp-forms/.env` | MSP forms app credentials | present | Not independently dated in this pass |
| app1-bu | `/root/.hermes/.env`, `/root/.hermes/.env.telegram-backup` | Hermes core config and Telegram bot token | present | This is the same file flagged by Sec-B as **world-readable** on app1-bu; that permissions gap plus this being a live credential file is a compounding issue, cross-referenced in Section 7 |
| app1-bu | `/root/.hermes/state-snapshots/20260703-011737-pre-update/.env`, `/root/.hermes/state-snapshots/20260711-150407-pre-update/.env` | Full Hermes config snapshots from pre-update states | 2026-07-03, 2026-07-11 | Stale historical copies retained on disk, same document-sprawl pattern as the litellm pre-keyfix file |
### 4.2 Plaintext credential document sprawl (`key-inventory.md`)
This is the single most significant secrets-location finding of this audit, and it was independently re-confirmed in this session against the raw file inventory rather than only relying on the prior session's account of it.
**Ten copies of `key-inventory.md` exist across the filesystem.** File-level comparison (size, modification time, and a redaction-marker check that counts occurrences of `[REDACTED]`/`Vaultwarden`/`Hudu`-style placeholder references without reading or reproducing any actual secret value):
| Path | Size | Modified | Redaction-marker count | Assessment |
|---|---|---|---|---|
| `/root/projects/itpp-infrastructure/docs/infrastructure/key-inventory.md` | 12,497 B | 2026-08-08 | 27 | **Sanitized (canonical).** This is the properly redacted version meant for the docs repo. |
| `/root/itpp-docs/docs-source/itpp-infrastructure/key-inventory.md` and 3 identical copies under `/tmp/itpp-docs-build/`, `/tmp/audit-export/`, `/tmp/audit-repos/`, `/tmp/tmp.YZCSHQoVPf/` | 12,503 B each, identical md5 `670625a7...` | 2026-08-08/09 | 27 each | Sanitized, duplicate build/export artifacts of the canonical doc. Document sprawl, not a secrets leak, but five redundant copies of the same file across `/tmp` build directories is untidy and should be cleaned up as part of normal repo hygiene. |
| `/root/projects/itpp-infrastructure/.backup-before-sanitize-20260723/key-inventory.md` | 13,136 B | 2026-07-23 | 3 (low redaction-marker count relative to the sanitized versions) | **CRITICAL: unsanitized, plaintext credential values.** File header reads "Generated: 2026-07-23... Contains real credentials". This is a pre-sanitization leftover from a July 23 cleanup pass that documented real root passwords for app1/app2/app3, cloud provider API tokens, S3 access keys, and multiple service tokens, per its own header and per the prior session's confirmed read. It is `.gitignore`'d (not in git history) but sits unencrypted, root:root, mode 600, on Core's local disk. |
| `/root/.hermes/references/key-inventory.md` | 13,161 B | 2026-08-08 | 3 (low, matching the unsanitized pattern) | **CRITICAL: a second, independent unsanitized copy**, not identified by name in the prior session's summary. Same header pattern ("Generated: 2026-07-23... CLASSIFIED: Contains real credentials"), root:root, mode 600. This copy lives inside the Hermes agent's own reference-file directory, meaning it is reachable by any skill or process that can read Hermes's reference files, a broader exposure surface than a one-off backup directory. |
| `/root/.hermes/skills/devops/hudu-management/references/key-inventory.md` and `/root/.hermes/.backups/hermes-backup-2026-07-22/skills/devops/hudu-management/references/key-inventory.md` | 4,022 B and 3,794 B | 2026-07-22 | 9 each | Smaller, older API-key-name-only lists (structure/labels, not full values per the prior session's read), lower risk but still worth folding into the cleanup since they are stale duplicates of a smaller scope. |
**Net finding:** there are two full unsanitized plaintext copies of the complete credential inventory (not one, as the prior session reported), both root:root mode 600 so restricted to local root access only, but both should have been shredded after the sanitized canonical version was created on 2026-08-08. Local root access to Core is not a trivial bar (this is the Hermes host with the broadest operational reach in the estate), so "root-only" is a meaningful but not sufficient mitigation.
### 4.3 Secrets-sprawl git-grep results (per-repo hit counts, tracked files only)
The following are word-pattern hits (password/secret/token/api_key as text) across git-tracked files in every `.git` repo found on each host. These are **not confirmed leaked credentials**; they are candidate locations that need per-file triage before being treated as a real secrets-in-git problem.
| Host | Repos scanned | Total file hits | Largest single contributor |
|---|---|---|---|
| Core | 68 | 6,296 | `/usr/local/lib/hermes-agent` (2,704 hits, almost entirely documentation/code referencing the words "secret"/"token" as identifiers, not literal values) |
| app1 | 6 | 27 | `/root/docker/wazuh` (workflow/config files referencing "secret" as a term) |
| app2 | 1 | 1,275 | `/opt/ragflow` (single large open-source repo, README/workflow/test files) |
| app3 | 2 | 719 | `/opt/buzz` (open-source repo, same pattern: workflow files, `.env.example`, changelogs) |
| app1-bu | 3 | 5,504 | `/root/.hermes/docker/twenty` (2,741 hits) and `/usr/local/lib/hermes-agent` (2,761 hits), both open-source vendor codebases mirrored onto the standby host |
| wphost02 | 0 | 0 | No `.git` repos found on this host |
**Assessment: Medium, not Critical.** The overwhelming majority of hits are in vendored open-source code (`hermes-agent`, `twenty`, `ragflow`, `buzz`, `theHarvester`, `sherlock`) where "secret", "token", or "api_key" appear as variable names, documentation words, or CI workflow keys, not as literal credential values. The one path worth a manual look is `/root/projects/itpp-infrastructure` itself (44 hits on Core, per the prior session's account, and 2 hits on app1-bu limited to `README.md` and `sites/app3.md`), since that is the org's own infrastructure documentation repo, not a third-party vendor codebase, and is the most likely place an actual value could have been accidentally committed. This needs a manual per-file read in Phase Two; it was not performed in this session because it would require opening and reading each of the 44 hit locations individually, and the tool budget for this task was reserved for account/SSH/secrets-location work per the task's explicit instructions.
### 4.4 Configuration files with credential-relevant names (non-`.env`)
Notable non-`.env` files that could contain live secrets, filtered from a much larger CONFIG_SECRET_FILES sweep that was mostly PHP extension `.ini` files (not credential-relevant, omitted from this table):
| Host | Path | Type |
|---|---|---|
| Core | `/root/docker/monitoring/grafana/grafana.ini` | Grafana config, cross-referenced to Sec-B's default-admin-password finding |
| app1 | `/root/docker/litellm/config.yaml` | LiteLLM routing/model config, may reference upstream API keys by env var name |
| app2 | `/opt/gitea/data/gitea/conf/app.ini` | Gitea app config, contains SECRET_KEY/INTERNAL_TOKEN/JWT_SECRET per the prior session's confirmed (value-redacted) read |
| app3 | `/etc/gitea/app.ini` | A second Gitea config path found on app3; worth confirming in Phase Two whether this is a stale leftover from a prior Gitea install location or an active second instance, since the brief describes Gitea as living on app2 |
| Core | `/root/.config/goose/secrets.yaml` | Goose CLI agent secrets file |
---
## 5. Shared vs Personal Credential Assessment
- **Gitea (app2): one shared admin account (`ippadmin`), confirmed live via API in the prior session** (is_admin=true, last login 2026-08-10). No per-person Gitea accounts exist. Anyone who knows this one credential, or holds a valid API token for it, has full administrative control over every source repository in the org.
- **CloudPanel (app3): one admin account (`gmb`)** per `clpctl user:list` from the prior session. Same single-shared-account pattern.
- **SSH root/ippadmin access:** access to Core, app1, app2, and app3 is governed by the same small set of keys (`itpp-infra`, `germaine@itppartner`) shared across hosts rather than per-host or per-person keys. There is no evidence of individual named-user SSH accounts distinct from `root` and `ippadmin`; all administrative SSH access funnels through these two shared identities.
- **`ippadmin` has passwordless full sudo** (`NOPASSWD:ALL`) on Core, app2, app3, and wphost02 (and root has the OS-level password hash set on top of that). This means any of the keys that unlock `ippadmin`'s SSH access are equivalent to unlocking root on that host, with no additional authentication step (no MFA, no password re-prompt) in between.
- **No individual-user accountability mechanism observed.** Because SSH access and sudo are both shared/group-level rather than per-person, there is no way, from the data available, to attribute a specific administrative action on any of these hosts to a specific individual. This is a foundational access-control gap that other findings (MFA gaps, unclear last-login history) compound.
- **app3's per-client accounts** are the one place where the estate does practice some separation (one Linux account per hosted site), which is appropriate multi-tenant hygiene, but even this layer sits underneath the same shared `ippadmin`/root access that can reach every one of those accounts.
---
## 6. Least-Privilege Findings
- **Almost every Docker container across the estate runs as `root` inside the container (`user=[root(default)]`).** This was true for the large majority of containers sampled on Core, app1, app2, and app3, including services with no operational need for root (e.g., `microbin`, `searxng`, `telegraf`, `mikrotik-exporter`, `uptime-kuma` on Core; `komodo-core`, `docuseal`, `twenty-db`, `n8n-postgres`, `mcp-browser`, `mcp-email`, `mcp-git`, `mcp-filesystem`, `super-search` on app1; `bookstack`, `bookstack-db`, `docker-ragflow-cpu-1`, `docker-mysql-1`, `docker-minio-1`, `docker-redis-1`, `technitium`, `dawarich_sidekiq/app/db/redis`, `traccar`, `gitea`, `unms-nginx`, `unifi-controller`, `hudu-app/db/worker/redis`, `unms-api`, `ucrm`, `unms-postgres`, `unms-siridb`, `unms-fluentd` on app2; `hexclave-postgres`, `hexclave-clickhouse`, `buzz-prod-postgres/redis/minio` on app3). None were flagged as `privileged=true` (a worse container-escape-relevant setting), which is a positive, but running as the root UID inside the container still means a container-breakout vulnerability in any of these images grants root, not a scoped user, on the host's container runtime layer.
- **Some containers do practice least privilege correctly** and are worth naming as the good pattern to extend: `browserless` (`blessuser`), `grafana` (`472`, Grafana's own non-root UID), `prometheus` (`nobody`), `openwebui` (`0:0` is still root, note this is actually still root, listed for completeness), `twenty-server`/`twenty-worker` (`1000`), `kokoro-tts` (`appuser`), `wazuh-dashboard`/`wazuh-indexer` (their own service users), `vaultwarden` is root but is a well-maintained image; `unms-device-ws-*` and `unms-rabbitmq`/`unms-netflow` containers (`1001`), `hexclave-server` (`node`), `hexclave-cron` (`curl_user`), `buzz-prod-relay-1` (`buzz:buzz`). These show the operators clearly know how to configure non-root containers when the upstream image supports it; the gap is inconsistent application, not lack of capability.
- **app2's `unms` service account is a member of the `docker` group.** Group membership in `docker` is functionally equivalent to root on the host, since a member can run any container with arbitrary host bind-mounts. If `unms` is meant to be a scoped service account for the UNMS/UCRM stack, its `docker` group membership defeats that scoping and should be reviewed in Phase Two; a properly least-privileged setup would run UNMS's containers under a system-level Docker Compose invocation by `ippadmin`/root rather than granting the `unms` account itself `docker` group membership.
- **`ippadmin ALL=(ALL) NOPASSWD:ALL`** on 4 of 6 hosts (Core, app2, app3, wphost02) is the single most consequential least-privilege gap in the account layer: this is unrestricted, no-password-prompt root escalation for a shared account, with no command allowlisting or logging requirement built into the sudoers entry itself.
- **`ALL ALL=(ALL) NOPASSWD: /usr/bin/clpctlWrapper` on app3** allows any account, including the low-privilege per-client site accounts, to invoke the CloudPanel control wrapper as root without a password. This needs a Phase Two review of exactly what `clpctlWrapper` can do; if it exposes any file-write or command-injection surface, every one of the ~28 per-client accounts on app3 effectively has a root-escalation path.
- **Core's `postgres` account has an interactive `/bin/bash` shell** rather than the more typical `/bin/false` or `/usr/sbin/nologin` for a database service account. This is a smaller-scale least-privilege deviation worth tightening if there is no operational reason for interactive postgres logins.
---
## 7. Severity-Rated Findings
### Critical
**SEC-A-01: Two unsanitized plaintext copies of the complete infrastructure credential inventory exist on Core's local disk**
Evidence: `/root/projects/itpp-infrastructure/.backup-before-sanitize-20260723/key-inventory.md` (13,136 bytes, root:root, mode 600, modified 2026-07-23) and `/root/.hermes/references/key-inventory.md` (13,161 bytes, root:root, mode 600, modified 2026-08-08) both carry the header "Contains real credentials, store encrypted, never email plaintext" and show a low redaction-marker count consistent with the prior session's confirmed read that these contain real root passwords, cloud provider API tokens, S3 keys, and service tokens in plaintext.
Rationale: A single successful compromise of the Core host at the root level, or any process/skill with read access to Hermes's own reference-file directory, would expose essentially every credential the organization relies on, in one file, in one read. The sanitized canonical version already exists at `docs/infrastructure/key-inventory.md`; these two copies are leftovers that were never deleted after that sanitization work was done on 2026-08-08, and one of them (the Hermes references copy) is in a location that a wider range of automated processes could plausibly touch than a one-off backup folder. This should be shredded (not just deleted) as a Phase Two remediation item; per the read-only rule, Sec-A did not delete it.
**SEC-A-02: A single SSH key grants access to 5 of 6 servers in the estate, each with passwordless root escalation on top**
Evidence: The `itpp-infra` key (fingerprint `SHA256:Jxh0bbT9dUV3q1DYYB3hHyhy/1TDj7Q8U4xrVmB38uQ`) is present in `authorized_keys` on Core, app1, app2, app3, and app1-bu. On Core, app2, and app3, the account it can reach (`ippadmin` or root) has `NOPASSWD:ALL` sudo.
Rationale: There is effectively one credential standing between an attacker and full administrative control of 5 of the org's 6 servers. If this key is ever exposed (leaked in a repo, phished, copied to a compromised laptop), the blast radius is close to the entire estate, in a single step, with no second factor to slow it down. This is the highest-leverage single point of failure identified in the IAM domain.
### High
**SEC-A-03: Gitea and CloudPanel each rely on a single shared admin account with no per-person accountability**
Evidence: Gitea's `GET /api/v1/admin/users` (prior session, live read) returned exactly one account, `ippadmin`, is_admin=true. CloudPanel's `clpctl user:list` (prior session) showed a single admin account, `gmb`.
Rationale: Every code change and every CloudPanel administrative action on these platforms is attributable only to "the shared account," not to a specific person. If credentials are shared among multiple staff (which a single account by definition requires, if more than one person needs access), there is no way to know who did what, which matters both for day-to-day accountability and for incident response if something goes wrong.
**SEC-A-04: `ippadmin` has passwordless, unrestricted root sudo on 4 of 6 hosts, and is reachable by the same shared SSH keys used across the estate**
Evidence: `/etc/sudoers.d/ippadmin` contains `ippadmin ALL=(ALL) NOPASSWD:ALL` on Core, app2, app3, and (via a separate file) wphost02. The same `itpp-infra` and `germaine@itppartner` keys that unlock `ippadmin`'s SSH session are shared across hosts (see SEC-A-02).
Rationale: There is no additional authentication step between "I have this one SSH key" and "I am root on this server," across most of the estate. Combining a shared credential with unrestricted, no-prompt root escalation removes every layer of defense that would normally exist between initial access and full compromise.
**SEC-A-05: app3's `clpctlWrapper` sudoers rule grants any account, including all ~28 per-client site accounts, a passwordless path to run a root-level control wrapper**
Evidence: `/etc/sudoers.d/cloudpanel` contains `ALL ALL=(ALL) NOPASSWD: /usr/bin/clpctlWrapper`.
Rationale: This rule is written to apply to `ALL` users, not just `clp`. If any of the roughly 28 per-client accounts on app3 is ever compromised (for example through a vulnerable WordPress plugin on that client's site), the attacker inherits a passwordless path to a root-level tool. The actual risk depends on what commands `clpctlWrapper` exposes and whether it validates its inputs, which needs a Phase Two review; it was not something this read-only audit could safely test without executing the wrapper.
### Medium
**SEC-A-06: Secrets-sprawl git-grep hits (6,296 on Core, 5,504 on app1-bu, 1,275 on app2, 719 on app3, 27 on app1) are mostly false-positive-heavy vendored open-source code, but the organization's own infrastructure repo has not yet been manually triaged**
Evidence: Per-repo breakdown in Section 4.3 shows the largest contributors are vendored codebases (`hermes-agent`, `twenty`, `ragflow`, `buzz`) where matches are variable names or documentation text, not credential values. `/root/projects/itpp-infrastructure` itself shows 44 hits on Core and 2 on app1-bu.
Rationale: Treating a raw grep-hit count as a secrets-leak severity score would both overstate the risk from vendor code and understate the one path that actually matters: the org's own documentation and infrastructure repo. This needs a targeted manual read of those 44 (Core) and 2 (app1-bu) hits in Phase Two, not a blanket "Critical, 13,000+ secrets found" framing.
**SEC-A-07: Live `.env` credential files are retained unencrypted in daily backup snapshots on app3, and in stale pre-rotation copies on app1**
Evidence: `/home/clp/backups/2026-08-11_04-15-01/app/files/.env`, `2026-08-12`, and `2026-08-13` each hold a full plaintext copy of the same application `.env`. On app1, `/root/docker/litellm/.env.pre-keyfix-20260714-130347` sits alongside the live `/root/docker/litellm/.env`, an old credential set from before a documented key-fix event on 2026-07-14 that was never removed.
Rationale: Every backup cycle multiplies the number of at-rest plaintext copies of the same credentials without adding any access control beyond whatever protects the backup directory itself. Stale pre-rotation files are worse: if the "old" credentials in that file were ever rotated because they were compromised or suspected compromised, the old values are still sitting on disk in cleartext.
**SEC-A-08: Almost all Docker containers across Core, app1, app2, and app3 run as root inside the container, despite several images on the same hosts demonstrating that non-root operation is supported**
Evidence: Section 6 lists dozens of containers with `user=[root(default)]` across all four hosts, alongside a smaller set of containers (`browserless`, `grafana`, `prometheus`, `twenty-server/worker`, `kokoro-tts`, `unms-device-ws-*`, `hexclave-server`, `buzz-prod-relay-1`) that correctly run as scoped, non-root users.
Rationale: Running as root inside a container is not itself a host compromise, but it removes one layer of defense in depth: a container-escape vulnerability in a root-run container hands the attacker root, not a limited user, on the container runtime. Since the operators clearly know how to configure non-root users (as shown by the containers that already do this correctly), tightening the rest is a configuration change, not a re-architecture.
**SEC-A-09: app2's `unms` service account is a member of the `docker` group**
Evidence: `/etc/group` on app2 shows `docker:x:990:unms`.
Rationale: Docker group membership is equivalent to root access on the host. If `unms` was intended as a scoped, limited-privilege account for running the UNMS/UCRM stack, this membership defeats that intent and should be reviewed in Phase Two to confirm whether it is required for the stack's Compose-based startup or whether it can be removed in favor of running Compose as `ippadmin`/root directly.
### Low
**SEC-A-10: `scanuser` on Core has no identifiable purpose in the captured data and should be reviewed for removal or documentation**
Evidence: `/etc/passwd` shows `scanuser:1001:1001:/home/scanuser:/bin/bash`, shadow status is `LOCKED_NO_PASSWORD`, no matching SSH key, cron job, or docker container context was found tying it to an active use.
Rationale: An account with an interactive shell but no traceable purpose is exactly the kind of thing that accumulates in long-running infrastructure and eventually becomes an orphaned foothold. Low severity because it is currently locked (no password) and no SSH key reaches it, but it should be either documented or removed rather than left unexplained.
**SEC-A-11: Login-history visibility is inconsistent across the estate**
Evidence: Last-login data was populated for app1-bu and wphost02 but returned empty for Core, app1, app2, and app3 in the captured data.
Rationale: Not being able to see recent login activity on 4 of 6 hosts is a monitoring/visibility gap that limits how confidently this audit (or Phase Two remediation planning) can assess whether the shared credentials discussed above are being actively used, by how many people, or from where. Flagged per the brief's rule 4 (flag insufficient visibility, do not guess) rather than assumed benign.
---
## Appendix: Methodology and Limitations
- All findings in this file are derived from local audit capture files (`/root/audit_*.txt`) collected in a prior read-only SSH/curl session against the 6 hosts. No new SSH connections, command executions against remote hosts, or credential-value reads were performed to produce this document.
- Every `.env`, compose-file environment variable, and config-file secret value in the raw capture was already redacted to `[REDACTED]` at collection time; this file reports only presence, location, and staleness signals derived from filenames, timestamps, and structural markers (redaction-marker counts), never actual values.
- App-level user accounts inside Wazuh, Hudu, UniFi, UNMS/UCRM, LiteLLM admin, Vaultwarden, Dawarich, and Traccar could not be enumerated from the local file data available to this session, since those are internal application database records, not OS-level accounts or static config file entries. This is a stated visibility gap per brief rule 4, not a finding of "no accounts" or "single account."
- The per-file triage of the 44 (Core) and 2 (app1-bu) git-grep hits inside `/root/projects/itpp-infrastructure` itself was not completed in this session due to tool-budget constraints; this is explicitly flagged as outstanding work for Phase Two in Finding SEC-A-06, not silently omitted.
- This file was written by a fresh session using data collected by a prior session that exhausted its tool-call budget before writing any findings file. All data cited here was independently re-extracted and cross-checked from the raw capture files as part of producing this document, not copied verbatim from the prior session's unverified summary.
audit/phase-one/findings/sec-b.md
+160
View File
@@ -0,0 +1,160 @@
# Sec-B Findings: Hardening, Patch Posture, MFA/Authentication, Logging/Monitoring
Auditor: Sec-B (information security hardening)
Scope: Patch posture, SSH/OS hardening, MFA coverage on admin consoles, Wazuh/logging coverage, across the full ITPP estate.
Mode: READ-ONLY. No configuration changes, restarts, or remediation performed. All items below are findings for Phase Two remediation planning.
Builds on: neteng-a.md (network exposure), sys-a.md and sys-b.md (per-service inventories). Enumeration of services/ports is not repeated here; see those files for full service lists and images in use.
---
## 1. Patch Posture Table
| Host | OS | Kernel Running | Kernel Update Pending? | Last apt update/upgrade run | Upgradable pkg count | Notes |
|---|---|---|---|---|---|---|
| Core (localhost) | Debian 13 (trixie) | 6.12.94+deb13-amd64 | Yes, 6.12.101-1 available | apt metadata refreshed 2026-08-13 (same day as audit) | 15 | No unattended-upgrades service active; refresh appears manual/cron-triggered, not verified automatic |
| app1 (152.53.36.131) | Debian 13 (trixie) | 6.12.95+deb13-amd64 | Unclear (kernel not in upgradable list this pass, discrepancy with Core) | unattended-upgrades stamp 2026-08-13 06:50, last real upgrade logged 2026-08-11 (caddy security update) | 15 (at first check) | unattended-upgrades ACTIVE and ENABLED, evidence of automatic security patching working |
| app2 (152.53.39.202) | Debian 13 (trixie) | 6.12.95+deb13-amd64 | Unclear, similar to app1 | unattended-upgrades stamp 2026-08-13 06:22, last logged upgrade 2026-08-11 (caddy) | 15 (at first check) | unattended-upgrades ACTIVE and ENABLED |
| app3 (152.53.241.111) | Debian 13 (trixie) | 6.12.95+deb13-amd64 | Yes, 6.12.101-1 available (confirmed twice) | update-stamp/upgrade-stamp last changed 2026-07-16 (about 4 weeks before audit) | 14, including linux-image-amd64 security kernel and postfix security update | unattended-upgrades reports active/enabled but the upgrade-stamp is stale (2026-07-16) versus app1/app2 (2026-08-13); patch cadence on app3 is lagging by roughly 4 weeks despite the service being enabled |
| app1-bu (5.161.225.131, warm standby) | Ubuntu 24.04.4 LTS | 6.8.0-117-generic | No kernel package pending (not in upgradable list) | apt success stamp 2026-08-13 05:32 (current) | 16, mostly apport/cloud-init/apparmor/plymouth, non-security-critical | unattended-upgrades ACTIVE and ENABLED, fail2ban active |
| wphost02 (5.161.62.38, legacy) | Ubuntu 24.04.4 LTS | 6.8.0-134-generic | No kernel package pending | apt success stamp 2026-08-13 00:44 (current) | 9, mostly apport/sosreport/network minor pkgs | unattended-upgrades ACTIVE and ENABLED, fail2ban active with 3 jails (runcloud-agent, sshd, sshd-ddos) |
Docker image tags (cross-reference to sys-a.md/sys-b.md inventories):
- Wazuh stack on app1 running wazuh-manager/indexer/dashboard 4.9.2, up 3 weeks continuously. 4.9.2 is a specific pinned version, not `latest`; verify against current Wazuh release train in Phase Two to confirm no known CVEs unpatched in this line.
- Technitium DNS container on app2 runs on DOTNET_VERSION=10.0.9 / ASPNET_VERSION=10.0.9 (current runtime), but ships with `DNS_SERVER_ADMIN_PASSWORD=changeme` in its environment (see Finding SEC-B-03, credential-adjacent hardening issue, flagged here because it is a default/weak-credential indicator, not a secrets inventory item).
- Gitea, Hudu, UniFi, Traccar, Dawarich image version pinning should be cross-checked against sys-a/sys-b inventories for stale major versions; no additional very-old (multi-year) tags were independently observed beyond what sys-a/sys-b already caught.
**Patch posture summary:** All 6 hosts are on reasonably current OS/kernel baselines (Debian 13 trixie or Ubuntu 24.04 LTS), and 5 of 6 have unattended-upgrades active and enabled with recent apply timestamps. app3 is the outlier: unattended-upgrades reports enabled but its upgrade-stamp is roughly 4 weeks stale versus siblings, and it has a pending security kernel update (6.12.95 to 6.12.101) plus a pending postfix security update, both unapplied. Core (localhost) has NO unattended-upgrades service at all (inactive, not-found), relying entirely on manual or externally-scheduled patching, and has 15 upgradable packages including an available kernel update.
---
## 2. Hardening Table (SSH / OS-level)
| Host | PermitRootLogin | PasswordAuthentication | PubkeyAuthentication | fail2ban | unattended-upgrades | auditd | World-readable secrets found |
|---|---|---|---|---|---|---|---|
| Core | prohibit-password (without-password) | no | yes (default) | NOT INSTALLED (inactive, unit not found) | NOT INSTALLED (inactive, unit not found) | inactive | None found in targeted scan |
| app1 | prohibit-password | no | yes | active, enabled | active, enabled | not checked directly (not active per rsyslog/auditd probe) | None found |
| app2 | prohibit-password | no | yes | active, enabled | active, enabled | not checked | None found |
| app3 | prohibit-password | no | yes | active, enabled, 1 jail (sshd) | active, enabled (but stale upgrade cadence, see above) | not checked | None found |
| app1-bu | prohibit-password | no | yes | active, enabled, 1 jail (sshd) | active, enabled | not checked | /root/.hermes/.env is world-readable (flagged, but this is credential-adjacent; full secrets review is Sec-A's domain, flagged here purely as a file-permission hardening gap) |
| wphost02 | prohibit-password | no | yes | active, enabled, 3 jails (runcloud-agent, sshd, sshd-ddos) | active, enabled | not checked | None found |
Additional hardening observations:
- No empty-password accounts found on Core or app1 in the /etc/shadow scan performed.
- SSH host key files (`ssh_host_*_key`) are not world-readable on any host checked.
- Root login is uniformly configured as `prohibit-password` / `without-password` across all 6 hosts, meaning root can only log in via SSH key, not password. This is a reasonably strong baseline, but root login is still permitted at all (as opposed to fully disabled with a dedicated sudo-only admin account), which is a CIS/NIST deviation worth flagging as Low/Medium depending on the org's risk appetite.
- Core has neither fail2ban nor unattended-upgrades installed. This is the weakest OS hardening posture of the 6 hosts, despite Core running Hermes (the orchestration agent), Grafana, Prometheus, and the Super Search MCP, i.e. a high-value control-plane host.
- auditd was checked and found inactive on Core and app1; not independently verified on the remaining 4 hosts, but given fail2ban/journald are the only audit trail sources evidenced, assume auditd is similarly absent estate-wide unless Phase Two proves otherwise.
- journald.conf reviewed on all 6 hosts: no [Journal] section overrides present anywhere (defaults are in force), meaning log rotation/retention limits are whatever the distro default is, not an explicit organizational retention policy. No rsyslog remote forwarding (`@@host` or `@host` directives) was found configured on any host, confirming logs are local-only and not centrally shipped from the OS layer.
---
## 3. MFA / Authentication Coverage Table
| Admin Console / Service | Host | MFA Evidence Found | Status |
|---|---|---|---|
| Grafana | Core :3002 | `GF_SECURITY_ADMIN_PASSWORD=admin` (default password in use); grafana.ini shows only commented-out OAuth/LDAP stanzas, no active SSO/2FA config; `disable_login_form` not set to true, so local form-based login remains the primary path | NO MFA, plus default admin credentials in active use (Critical) |
| Wazuh Dashboard/API | app1 | wazuh-wui API credential in ossec.conf uses local basic auth (`username: wazuh-wui`) with no OAuth/SAML/OIDC integration observed; dashboard is behind Caddy per neteng-a.md but no MFA layer identified | NO MFA identified |
| Gitea | app2 | app.ini shows `ENABLE_CAPTCHA = false`, `REQUIRE_SIGNIN_VIEW = false`, `DISABLE_REGISTRATION = false`. Gitea supports per-user TOTP 2FA natively but enforcement/adoption was not verified (would require login, out of scope); registration being open plus no captcha is a related hardening gap independent of MFA | MFA capability exists but not confirmed enabled/enforced; registration is open, which is itself a risk |
| Hudu | app2 | Env vars show only SMTP/upload settings; no OMNIAUTH/SAML/SSO/MFA environment flags present | NO MFA evidence found |
| UniFi Controller | app2 | `system.properties` grep for 2fa/mfa/auth returned nothing; Ubiquiti UniFi supports MFA via Ubiquiti SSO cloud account when cloud-linked, but no evidence this controller is cloud-linked (local admin only per config inspected) | NO MFA evidence found (local-only auth assumed) |
| UNMS/UCRM | app2 | Env vars show `UBNT_OAUTH_SERVICE_URL=null`, meaning Ubiquiti cloud SSO/MFA path is disabled/unconfigured | NO MFA, and the SSO path that would enable MFA is explicitly nulled out |
| Technitium DNS | app2 | `DNS_SERVER_ADMIN_PASSWORD=changeme` present in container env, i.e. the DEFAULT PASSWORD PLACEHOLDER STRING is literally set as the variable value. This does not by itself prove the live credential equals "changeme" (Sec-A's domain to confirm), but the presence of the literal default string in the running environment is itself a hardening red flag. No MFA support is native to Technitium's basic auth | NO MFA; default-credential-pattern flag (Critical, pending Sec-A confirmation of live value) |
| CloudPanel | app3 | CloudPanel CE (v6.0.8 running) supports optional per-user TOTP 2FA (confirmed via vendor docs), but it is opt-in per account and not enforceable/mandated centrally; no evidence found in local files that the sole `gmb` admin account (per `clpctl user:list`) has 2FA enabled, and this cannot be confirmed without interactive login (out of scope) | MFA capability exists but adoption unconfirmed; single admin account, no organizational enforcement mechanism |
| LiteLLM / admin-ai UI | app1 | Only `LITELLM_MASTER_KEY` env found (redacted); no SSO/MFA env vars present | NO MFA evidence found |
| Vaultwarden | app1 | `SIGNUPS_ALLOWED=false` (good, closed registration); Vaultwarden supports WebAuthn/TOTP 2FA per-user natively but no admin-enforced policy env vars (e.g., `ADMIN_TOKEN`) were found configured, meaning the /admin panel protection state is unclear | Per-user 2FA capability exists; admin panel protection unconfirmed |
| Dawarich, Traccar | app2 | No MFA/2FA/OTP/OAuth-related env vars found on either container | NO MFA evidence found |
**MFA-coverage summary:** Of roughly 11 distinct admin surfaces reviewed, ZERO were confirmed to have MFA actively enforced. Several platforms (Gitea, CloudPanel, Vaultwarden) have native 2FA capability that is opt-in/per-user and not centrally mandated, meaning coverage depends entirely on individual admins choosing to enable it, unverifiable from config alone. Grafana is actively using default admin credentials with no MFA, the most severe finding in this category. UNMS explicitly has its cloud SSO/MFA integration path disabled. Technitium DNS shows a literal "changeme" default-password string still present in its running configuration.
---
## 4. Logging / Monitoring (Wazuh) Coverage Table
| Host | Wazuh Agent Installed? | Wazuh Manager/Stack Present? | Local Journald Only? | Remote Log Forwarding (rsyslog)? |
|---|---|---|---|---|
| Core | NOT INSTALLED (no wazuh-agent unit, no /var/ossec, no wazuh package) | No (manager lives on app1) | Yes, journald default config only | None configured |
| app1 | Manager stack runs here (wazuh-manager, wazuh-indexer, wazuh-dashboard, all image 4.9.2, up 3 weeks) BUT `agent_control -l` shows only ONE registered agent: ID 000, the manager's own local agent (127.0.0.1) | Yes (this IS the manager) | Journald default | None configured |
| app2 | NOT INSTALLED | No | Journald default | None configured |
| app3 | NOT INSTALLED | No | Journald default | None configured |
| app1-bu | NOT INSTALLED | No | Journald default | None configured |
| wphost02 | NOT INSTALLED | No | Journald default | None configured |
Wazuh manager health check detail: the manager and indexer/dashboard containers are up and running (3 weeks uptime, no restarts observed), and the indexer's REST API responds (401 without credentials, meaning it is alive and enforcing auth, not down). However, `agent_control -l` returning only the manager's own loopback agent (ID 000) confirms that **not a single remote host across the 6-server estate is enrolled as a Wazuh agent**. The Wazuh deployment is effectively monitoring only itself.
**Wazuh/logging coverage summary:** Wazuh is deployed and technically healthy (containers up, API responsive) but has ZERO externally enrolled agents. Coverage is 1 of 6 hosts (16.7%), and that one host (app1) is only monitoring its own loopback, not even its own host-level OS logs via a real agent enrollment path (the "agent" is the manager's built-in local one, not a deployed endpoint agent). All 6 hosts rely solely on local, unshipped journald logs with default retention and no remote forwarding. There is no centralized log aggregation for authentication events, admin console access, container events, or intrusion indicators anywhere in the estate. This is the single largest blind spot identified in this audit.
---
## 5. Cross-Cutting: Unmanaged / Blind-Spot Hosts
Applying the combined lens of (no Wazuh agent) + (no/weak local monitoring) + (patch posture gaps):
- **Core (localhost):** No Wazuh agent, no fail2ban, no unattended-upgrades, no auditd. This is the most under-hardened host in the estate at the OS-control layer, notable because it hosts Hermes (agent orchestration), Grafana, Prometheus, and Super Search MCP, i.e., core operational tooling with broad reach. Prometheus/Grafana provide metrics-level visibility but not security-event-level visibility (no auth log shipping, no file integrity monitoring, no brute-force protection).
- **app2, app3, app1-bu, wphost02:** No Wazuh agent on any of them; they retain fail2ban and unattended-upgrades as baseline compensating controls, which Core lacks entirely, but none have any form of centralized security log visibility. If any of these hosts is compromised, there is currently no telemetry path that would surface it to a central SIEM; detection depends entirely on someone manually reviewing local journald logs after the fact.
- **app1 (Wazuh manager host):** Ironically the host running the SIEM has the SIEM monitoring nothing but itself. This is a significant program gap: the org has invested in deploying Wazuh infrastructure but has not completed the agent rollout, so the investment currently delivers near-zero detection value across the estate.
---
## 6. Severity-Rated Findings
### Critical
**SEC-B-01: Wazuh SIEM deployed but zero agents enrolled across the estate**
Evidence: `agent_control -l` on the wazuh-manager container (app1) lists only agent ID 000 (the manager's own loopback), no remote agents. No `/var/ossec` or wazuh-agent service found on Core, app2, app3, app1-bu, or wphost02.
Rationale: A SIEM with no enrolled agents provides no actual detection capability for the 5 non-manager hosts, which include internet-facing services (CloudPanel/WordPress, Hudu, UniFi, Gitea, LiteLLM). Security incidents on any of these hosts would go undetected by the org's own monitoring investment. This is the single highest-impact gap in the estate.
**SEC-B-02: Grafana running with default admin credentials and no MFA**
Evidence: `docker inspect grafana` shows `GF_SECURITY_ADMIN_PASSWORD=admin` alongside `GF_SECURITY_ADMIN_USER=admin`; grafana.ini has no active OAuth/LDAP/SSO configuration, only commented-out templates.
Rationale: Default admin/admin credentials on an internet-reachable (per neteng-a.md) observability console is one of the most well-known, automatically-scanned-for misconfigurations on the internet. Combined with no MFA, this is a near-zero-effort compromise path for anyone who finds the port.
**SEC-B-03: Technitium DNS container running with literal "changeme" default password string in live environment**
Evidence: `docker inspect technitium` shows `DNS_SERVER_ADMIN_PASSWORD=changeme`.
Rationale: Even if the operational credential has since been changed inside the app's own database (a possibility Sec-A should confirm, since this crosses into credential inventory), the fact that the container's own environment variable retains the literal placeholder value is a strong hardening/deployment-hygiene signal that default-credential practices may be in use elsewhere too. DNS admin compromise has estate-wide blast radius (this is itpropartner.com's DNS).
### High
**SEC-B-04: app3 patch cadence stale by approximately 4 weeks, with an unapplied security kernel and postfix update**
Evidence: unattended-upgrades stamp files on app3 last touched 2026-07-16, versus 2026-08-13 (current, same day) on app1 and app2. `apt list --upgradable` on app3 shows `linux-image-amd64/stable-security 6.12.101-1` pending (currently on 6.12.95-1) and `postfix/stable-security` pending.
Rationale: app3 hosts CloudPanel, WordPress, and MySQL, an internet-facing content and mail-adjacent stack (postfix present). An unapplied 4-week-old security kernel patch and a pending postfix security update on a public-facing host is a meaningful exposure window, especially since neteng-a.md would have already flagged what ports are open here.
**SEC-B-05: Core has no fail2ban, no unattended-upgrades, and no auditd**
Evidence: `systemctl is-active fail2ban` and `unattended-upgrades` both return inactive with "not-found" for enabled state; `dpkg -l` shows neither package installed; auditd inactive.
Rationale: Core is the control-plane host for Hermes and observability tooling. Lacking brute-force protection and automatic security patching on a host with this level of operational privilege is disproportionate risk relative to its role, especially compared to the other 5 hosts which all have these controls.
**SEC-B-06: No admin console in the estate has confirmed, enforced MFA**
Evidence: Across Grafana, Wazuh dashboard, Gitea, Hudu, UniFi, UNMS/UCRM, Technitium, CloudPanel, LiteLLM, and Vaultwarden, no environment variable, config file, or system property indicated an active, enforced MFA/SSO integration. Several tools have opt-in per-user 2FA capability (Gitea, CloudPanel, Vaultwarden) but no evidence of organizational enforcement.
Rationale: Any single compromised admin credential (phishing, credential stuffing, reused password) grants full access to that console with no second factor to stop it. This is a systemic authentication-hardening gap across the entire estate, not a one-off.
### Medium
**SEC-B-07: UNMS/UCRM has its cloud SSO integration explicitly disabled (`UBNT_OAUTH_SERVICE_URL=null`)**
Evidence: `docker inspect ucrm` env output.
Rationale: Ubiquiti's cloud SSO path is one of the few routes to MFA for this product family; explicitly nulling it out removes that option, leaving local-only authentication as the sole path.
**SEC-B-08: No centralized log forwarding (rsyslog remote or journald shipping) configured anywhere in the estate**
Evidence: grep for `@@host`/`@host` rsyslog forwarding directives returned empty on all 6 hosts; journald.conf shows default config (no [Journal] section overrides) on all 6 hosts.
Rationale: Even setting aside Wazuh agent enrollment, there is no other mechanism (rsyslog, journald forwarding, or otherwise) shipping logs off-host anywhere. If a host is compromised and its local logs are tampered with or deleted, there is no off-host copy to fall back on for forensics.
**SEC-B-09: Gitea has open registration and disabled CAPTCHA**
Evidence: app.ini shows `DISABLE_REGISTRATION = false`, `ENABLE_CAPTCHA = false`, `REQUIRE_SIGNIN_VIEW = false`.
Rationale: An internet-reachable Gitea instance (confirm exposure via neteng-a.md) with open self-registration and no CAPTCHA is exposed to automated account creation/spam and widens the attack surface for credential-based attacks against a code hosting platform.
### Low
**SEC-B-10: Root login permitted (key-only) rather than fully disabled across all 6 hosts**
Evidence: `sshd -T` on all 6 hosts shows `permitrootlogin without-password` (uniformly).
Rationale: While password-based root login is correctly disabled everywhere (good baseline), CIS/NIST guidance generally recommends disabling root SSH login entirely in favor of named-user + sudo, to preserve accountability/audit trail for privileged actions. Not urgent given key-only enforcement, but a durable hardening improvement for Phase Two.
**SEC-B-11: /root/.hermes/.env world-readable on app1-bu**
Evidence: file permission scan on app1-bu found `/root/.hermes/.env` matching world-readable pattern.
Rationale: World-readable environment files can leak configuration/secrets to any local process or unprivileged account with filesystem access. Flagged here as a hardening/permissions gap; if it contains credentials, that overlaps with Sec-A's remit and should be cross-referenced with their inventory.
---
## Appendix: Methodology Notes
- All SSH access used `-o ConnectTimeout=10 -o BatchMode=yes` per brief, no interactive prompts triggered.
- No login attempts were made against any web UI (Grafana, Wazuh dashboard, Gitea, CloudPanel, etc.); all MFA determinations were made from static configuration files, environment variables, and system properties only, per the brief's read-only constraint.
- `sshd -T` (dump effective config) was used in addition to raw grep of sshd_config to catch settings inherited from Included files or compiled defaults.
- Wazuh index listing (`_cat/indices`) returned a 401 (auth required) rather than a connection failure, confirming the indexer is alive and reachable, just not accessible without credentials, which were not attempted per the read-only/no-credential-testing constraint.
- No package installs, service restarts, or file writes were performed on any remote host.
audit/phase-one/findings/sys-a.md
+388
View File
@@ -0,0 +1,388 @@
# Sys-A Phase One Inventory & Findings: Core + app1
Auditor: Sys-A (sysadmin inventory auditor)
Scope: Core (this host, localhost) and app1 (152.53.36.131)
Date: 2026-08-13
Mode: READ-ONLY. No mutation performed. All remediations are Phase Two items.
---
## 1. Host Profiles
| Attribute | Core (localhost) | app1 (152.53.36.131) |
|---|---|---|
| Provider / model | Netcup RS 2000 G12 | Netcup RS 4000 G12 |
| OS | Debian 13 (trixie) | Debian 13 (trixie) |
| Kernel | 6.12.94+deb13-amd64 | 6.12.95 |
| vCPU | 8 | 12 |
| RAM | 15 GB, NO swap | 31 GB, NO swap |
| RAM in use | ~12 GB used / 467 MB free (2.8 GB available) | (healthy) |
| Disk | 503 GB, 73 GB used (15%) | 1007 GB, 107 GB used (11%) |
| Uptime | not recorded | not recorded |
| Public IP | 152.53.192.33 | 152.53.36.131 |
| Docker | present | present |
| Firewall | ufw active (default policy not confirmed) | ufw active |
Total services inventoried: Core ~55 systemd units (running) + 10 Docker containers; app1 19 systemd units (running) + 23 Docker containers. Counts below are of running/active workloads only.
---
## 2. Core Service Inventory (localhost)
### 2.1 Critical / Tier-0 services
| Service | Software + Version | Port | Runs as | Restart | Backup | Notes |
|---|---|---|---|---|---|---|
| Hermes gateway | hermes-agent (python) | 8642, 8787 (socat), 9119 (dashboard) | root | user unit enabled | hermes-full-backup daily 01:00 + live-sync | runs under systemd USER manager (user@0), not a system unit; 3.7 GB RSS, 182 tasks |
| Caddy | caddy (deb) | 80/443 on 152.53.192.33, 2019 admin 127.0.0.1 | caddy | systemd enabled | Caddyfile in hermes-full-backup + system-config-sync | single ingress for Core |
| Grafana | grafana/grafana:11.4.0 | 3002 (0.0.0.0) | root (container) | always | grafana db in core-services-backup 01:30 | image 11.4.0 is ~20 months old |
| Prometheus | prom/prometheus:latest | 9090 (0.0.0.0) | root (container) | always | prometheus_data volume in core-services-backup | config /root/docker/monitoring/prometheus/prometheus.yml |
| Super Search MCP | /root/docker/super-search (python) | 8899 (0.0.0.0) | root | systemd | NOT covered by any backup script (see F-18) | depends on searxng + exa/firecrawl/open-corporates APIs |
| PostgreSQL | postgres (deb) | 5432 (127.0.0.1) | postgres | systemd | no dedicated dump; only hotnow-app DB implied | single instance, no replica |
| Redis | redis (deb) | 6379 (127.0.0.1) | redis | systemd | none found | single instance |
### 2.2 Hermes ecosystem systemd units (all as root unless noted)
| Unit | Purpose | Port | Notes |
|---|---|---|---|
| hermes-assistant | Hermes Assistant PWA backend | (8080-ish) | root |
| hermes-browser | Headless Chromium CDP | 9222 (127.0.0.1) | runs chrome with `--no-sandbox` as root |
| hermes-control-deck | Control Deck backend API | 8200 | **port collides with pipeline-api** (F-10) |
| hermes-socat-8787 | port forward 8787 -> localhost:8642 | 8787 (0.0.0.0) | `After=hermes-gateway.service` references non-existent SYSTEM unit (gateway is a user unit) |
| hermes-voice | Hermes Voice (SvelteKit) | 4331 (127.0.0.1) | runs as non-root (best practice) |
| hermes-gateway (user) | gateway run | 8642 | root, user manager |
| hermes-gateway-anita (user) | Anita messaging gateway | - | root, user manager |
### 2.3 Application / MCP / API systemd units (all as root)
| Unit | Purpose | Port | Hardcoded secret? |
|---|---|---|---|
| auth-api | auth backend | 8500 | no (uses /root/projects/auth/.env) |
| ops-portal | operations portal | 8090 | - |
| osint-api | OSINT API | 8100 | - |
| osint-person | OSINT person MCP | 8902 | - |
| diglocate-api | diglocate | 8000 | - |
| intelsight-api | intelsight | 8099 | - |
| hotnow-api | HotNow backend | 8001 | - (Postgres + Redis) |
| shopping-cart | shopping cart | 8101 (127.0.0.1) | - |
| seemytrip | SeeMyTrip backend | 8113 | **ADMIN_AI_KEY hardcoded** |
| rally | rally backend | 8105 (0.0.0.0) | **JWT_SECRET, DEEPSEEK_API_KEY, ADMIN_AI_KEY hardcoded** |
| shark-game | shark-game backend | 8083 (0.0.0.0) | - |
| pipeline-api | pipeline API | 8200 | **collides with hermes-control-deck** |
| verdicttank-api | verdicttank | 8201 | - |
| verdicttank-worker | worker | - | - |
| status-page / node | status page | 8210 | node |
| pry | PRY API | 8905 | - |
| dre-mcp | DRE MCP | 8900/8901 | - |
| ft360-mcp | FleetTracker360 MCP | 8903 | - |
| twilio-mcp | Twilio MCP | 8910/8911 | - |
| crawl4ai | crawler | - | - |
| voice-agent | voice agent | 9101 | - |
| voice-agent-stt | voice STT | 9000 | - |
| gitea-runner | Gitea actions runner | - | - |
| host-metrics-export | metrics textfile | - | - |
| mysql-tunnel | SSH tunnel to wphost02 MySQL | 33060 (127.0.0.1) | `StrictHostKeyChecking=no`, `UserKnownHostsFile=/dev/null` |
| hear-read | audio TTS/read | 8240 (127.0.0.1) | - |
### 2.4 Core Docker containers
| Container | Image:Tag | Port | Image age | Restart | Purpose |
|---|---|---|---|---|---|
| browserless | browserless/chrome:latest | 3000 (0.0.0.0) | 2 years | always | headless chrome API |
| uptime-kuma | louislam/uptime-kuma:1 | 3001 (0.0.0.0) | current | always | status monitoring |
| grafana | grafana/grafana:11.4.0 | 3002 | 20 months | always | dashboards |
| prometheus | prom/prometheus:latest | 9090 | 6 weeks | always | metrics |
| searxng | searxng/searxng:latest | 8888 (127.0.0.1) | 6 weeks | always | meta search (Super Search dep) |
| timetrex | timetrex | 8085 (127.0.0.1) | - | always | time tracking |
| microbin | microbin | 8260 (127.0.0.1) | - | always | paste bin |
| camofox | camofox | 9377 (0.0.0.0) | - | always | stealth browsing |
| mikrotik-exporter | mikrotik-exporter | 9436 (127.0.0.1) | - | always | router metrics |
| node_exporter (host) | prom/node-exporter | 9100 (0.0.0.0) | - | systemd | node metrics |
| telegraf (host) | telegraf | 9273 (0.0.0.0) | - | systemd | metrics |
### 2.5 Ad-hoc / unmanaged processes (Core)
| Process | Port | CWD | Concern |
|---|---|---|---|
| `python3 -m http.server 8080` | 8080 (0.0.0.0) | /var/www/mockup/anita-consulting | ad-hoc web server as root, no systemd |
| `python3 -m http.server 8934` | 8934 (0.0.0.0) | /var/www/mockup/itpropartner | ad-hoc web server as root |
| `python3 -m http.server 9876` | 9876 (0.0.0.0) | /tmp | **serving /tmp as root, public bind** |
| hermes dashboard | 9119 (0.0.0.0) | - | part of Hermes |
---
## 3. app1 Service Inventory (152.53.36.131)
### 3.1 Critical / Tier-0 services
| Service | Software + Version | Port | Runs as | Restart | Backup | Notes |
|---|---|---|---|---|---|---|
| LiteLLM / admin-ai | ghcr.io/berriai/litellm:v1.92.0 | 4000 | root (container) | always | **DB NOT backed up** (F-1); config YAML daily | master key + admin_ai provider |
| Caddy | caddy (systemd) | 80/443 | caddy | enabled | **Caddyfile NOT backed up** (F-12) | single ingress for all app1 |
| Open WebUI | ghcr.io/open-webui/open-webui:latest | 3000 (0.0.0.0) | root (container) | always | daily 1.47 GB | depends on LiteLLM |
| Wazuh manager/indexer/dashboard | wazuh 4.9.2 (indexer, 21 months old) | 1514/1515, 9200, 5601 | root | always | daily | SIEM/XDR |
| n8n | n8n | 5678 | root (container) | always | daily | automation |
| Vaultwarden | vaultwarden latest | 8081 | root (container) | always | daily | password manager |
| Twenty CRM | twentycrm | 3003 | root (container) | always | daily | CRM |
| Komodo | komodo-core | 9120 (0.0.0.0) | root (container) | always | daily | infra automation |
| Browserless | browserless/chrome:latest | 3005 (0.0.0.0), 3006 (ufw-limited) | root (container) | always | - | 2 year old image |
| super-search (host) | /root/docker/super-search | 8899 | root | systemd | .env only (373 bytes) | depends on searxng (Core) |
| giftaroast | /root/giftaroast | 8100 | root | systemd | - | **runaway fix_dict.py** (F-8) |
### 3.2 app1 Docker containers (full list)
LiteLLM v1.92.0 + litellm_postgres, mcp-browser / mcp-email / mcp-git / mcp-filesystem / mcp-super-search, litellm-super-search, open-webui, n8n + n8n-postgres, twenty-server + twenty-db + twenty-redis, komodo-core + komodo-mongo, vaultwarden, wazuh single-node (manager+indexer+dashboard), browserless, docuseal.
Key stale/very-old image tags on app1: browserless/chrome:latest (2 years), wazuh/wazuh-indexer:4.9.2 (21 months), vaultwarden/server:1.33.2 (18 months, old tag present alongside latest), ollama/ollama (5 weeks, not running), mattermost (not running).
### 3.3 app1 systemd units
super-search (8899), giftaroast (8100), caddy, sshd, docker, containerd, fail2ban, cron, rsyslog, ufw, unattended-upgrades, qemu-guest-agent, chrony, plus base system units.
### 3.4 Runaway / orphaned processes (app1)
| PID | Process | CWD | Elapsed | CPU |
|---|---|---|---|---|
| 3656093 | python3 fix_dict.py | /root/giftaroast | 14d 17h | 99.4% |
| 3657668 | python3 fix_dict.py | /root/giftaroast | 14d 17h | 99.4% |
| 3658103 | python3 fix_dict.py | /root/giftaroast | 14d 17h | 99.4% |
Parent of PID 3656093 is `bash -c "cd /root/giftaroast && python3 fix_dict.py 2>&1 echo '=== Restart service ===' systemctl dae..."`, an aborted/interrupted manual deployment. Three cores permanently pegged for 14+ days.
---
## 4. Dependency Map
### 4.1 Core (dot)
```dot
digraph Core {
// Ingress
caddy -> ops_portal; caddy -> osint_api; caddy -> diglocate; caddy -> intelsight;
caddy -> hotnow_api; caddy -> shopping_cart; caddy -> seemytrip; caddy -> rally;
caddy -> shark_game; caddy -> pipeline_api; caddy -> verdicttank_api; caddy -> auth_api;
caddy -> hermes_voice; caddy -> voice_agent; caddy -> pry; caddy -> uptime_kuma;
caddy -> microbin; caddy -> timetrex; caddy -> status_page;
// Data layer
hotnow_api -> postgres; hotnow_api -> redis;
osint_api -> redis;
// Hermes gateway fan-out
hermes_gateway -> super_search; hermes_gateway -> dre_mcp; hermes_gateway -> ft360_mcp;
hermes_gateway -> osint_person; hermes_gateway -> twilio_mcp; hermes_gateway -> clearfront_mcp;
hermes_gateway -> browserless; hermes_gateway -> camofox; hermes_gateway -> chrome_cdp;
// Search chain
super_search -> searxng; super_search -> "exa/firecrawl/open-corporates APIs";
// Voice chain
voice_agent -> voice_agent_stt;
// Observability
grafana -> prometheus;
prometheus -> node_exporter; prometheus -> telegraf; prometheus -> mikrotik_exporter; prometheus -> snmp;
// Cross-host tunnel
mysql_tunnel -> "wphost02 MySQL (152.53.62.38)";
}
```
### 4.2 app1 (dot)
```dot
digraph app1 {
caddy -> litellm; caddy -> open_webui; caddy -> n8n; caddy -> vaultwarden;
caddy -> docuseal; caddy -> twenty; caddy -> komodo; caddy -> wazuh_dashboard;
caddy -> giftaroast; caddy -> browserless;
litellm -> litellm_postgres;
litellm -> mcp_browser; litellm -> mcp_email; litellm -> mcp_git; litellm -> mcp_filesystem;
litellm -> mcp_super_search;
open_webui -> litellm;
n8n -> n8n_postgres;
twenty_server -> twenty_db; twenty_server -> twenty_redis;
komodo_core -> komodo_mongo;
wazuh_manager -> wazuh_indexer; wazuh_dashboard -> wazuh_indexer;
super_search_host -> "searxng on Core (152.53.192.33:8888)";
mcp_super_search -> super_search_host;
}
```
### 4.3 Dependency map highlights
- Single ingress per host: Caddy is the sole HTTP(S) entry point. A Caddy misconfiguration or crash takes down every web service on that host (single point of failure).
- LiteLLM is the AI backbone: Open WebUI, all mcp-* tool servers, and Super Search (via MCP) route through it. admin-ai.itpropartner.com is the model gateway for every AI consumer.
- Hermes gateway has a wide fan-out (8+ MCP/tool servers, browserless, camofox, Chrome CDP). Its failure cascades into all agent tooling.
- Cross-host dependency: app1 super-search depends on Core searxng (port 8888) over the WAN. If Core is down, app1 search is degraded.
- mysql-tunnel (Core) depends on wphost02 (152.53.62.38) reachability; a host key change breaks it silently.
- HotNow is the only app with a real DB dependency chain (Postgres + Redis) on Core.
---
## 5. Cron Jobs
### 5.1 Core crontab (root)
| Schedule | Job | User | Purpose | Failure mode if silent |
|---|---|---|---|---|
| 01:00 daily | hermes-full-backup.sh | root | full Hermes home + Caddyfile + systemd units + state.db to Wasabi | no DR copy of Hermes state; RPO grows unbounded |
| */15 min | hermes-live-sync.sh | root | aws s3 sync of .hermes to `live/` prefix | incremental state lost (mitigated by daily full) |
| 01:30 daily | core-services-backup.sh | root | Grafana/Prometheus/etc volumes | monitoring history loss |
| 02:00 daily | root-essentials-backup.sh | root | .hermes/.ssh/.aws/himalaya/shark-game/projects | credential/config loss |
| 03:00 daily | docker-volume-sync.sh | root | **script does not exist** (F-13) | always fails silently |
| 00:05 daily | system-config-sync.sh | root | system configs to Wasabi | config drift backups lost |
| 15:00 daily | status-page-refresh.sh | root | status page data | stale status |
| various | wphost02-backup.sh | root | SSHes to 152.53.62.38 (wphost02) with root key | wphost02 backups lost |
| various | docuseal/timetrex/gitea/hudu/dawarich/ragflow/twenty/stack-auth/hexclave/komodo/litellm/technitium/unifi/unms/vaultwarden-backup.sh | root | per-service backups (many target remote hosts) | per-service backup loss |
Hermes-managed jobs live in /root/.hermes/cron/jobs.json (jobs.json + per-job dirs). Parsing jobs.json programmatically failed during the audit (malformed JSON); a grep fallback confirmed the job list above. The exact schedule set should be re-verified by the DR owner.
### 5.2 app1 crontab (root)
| Schedule | Job | Purpose | Failure mode |
|---|---|---|---|
| 02:00 daily | /root/backup.sh | litellm config, n8n, openwebui, mcp .env, ollama, mattermost, wazuh, twenty | see F-1 (litellm DB), F-12 (Caddyfile) |
app1 /root/backup.sh references `mattermost-backup.sh` and `wazuh-backup.sh` in /root/.hermes/scripts/. Mattermost is not running (no container), so that backup is a no-op. Ollama is not installed, so its backup step fails each night.
### 5.3 /etc/cron.* (both hosts)
Standard Debian logrotate/man-db/dpkg jobs. No custom third-party cron drops observed beyond the system crontabs above. No evidence of a rotating backup retention job; S3 listings show daily objects accumulating without obvious lifecycle policy (Phase Two item).
---
## 6. Backup Status Matrix
| Service | Backup script | Last S3 object | Restore test documented? |
|---|---|---|---|
| Hermes (full) | hermes-full-backup.sh | 2026-08-13 (daily tar.gz ~1.43 GB) | restore.sh generated; DR-PLAN.md present (600) |
| Hermes (live) | hermes-live-sync.sh -> `live/` prefix | last state.db `live/` unclear; `live-sync/` (old prefix) stale since 2026-07-05 | partial |
| LiteLLM config | app1 /root/backup.sh | 2026-08-13 (litellm-config-*.yaml, 333 B) | no |
| **LiteLLM Postgres DB** | (should be pg_dump) | **ZERO .sql.gz objects ever** (F-1) | no |
| Grafana/Prometheus | core-services-backup.sh | 2026-08-13 (grafana 55 KB, prometheus 51 MB) | no |
| Open WebUI | app1 /root/backup.sh | 2026-08-13 (1.47 GB) | no |
| n8n / Twenty / Vaultwarden / Wazuh | app1 /root/backup.sh | 2026-08-13 | no |
| Core Caddyfile | hermes-full-backup + system-config-sync | 2026-08-13 | via restore.sh |
| **app1 Caddyfile** | **none** | **never** (F-12) | no |
| Super Search (both hosts) | none meaningful | .env only (373 B) | no |
| Postgres / Redis (Core) | none dedicated | none | no |
Restore-test documentation: DR-PLAN.md and migration-recovery.md exist on Core (root-only, mode 600) but no evidence of an actual periodic restore drill being executed (no drill logs found in backup dirs).
---
## 7. Severity-Rated Findings
### CRITICAL
**F-1. LiteLLM Postgres database is not being backed up (data-loss risk).**
Evidence: `aws s3 ls s3://hermes-vps-backups/app1/litellm/` shows only `litellm-config-*.yaml` objects (240-333 bytes). Zero `.sql`/`.sql.gz` objects since inception. app1 /root/backup.sh dumps `pg_dump -U litellm litellm`, but the live config (`config.yaml`) sets database_url to database `litellm_db`, so the dump targets a nonexistent database and fails silently every night.
Rationale: LiteLLM's Postgres holds every API key, model routing table, spend/budget records, and the admin-ai provider config. A database or volume failure means total loss of the AI gateway state. This is the single highest-impact finding.
**F-2. Plaintext secrets hardcoded in world-readable systemd unit files.**
Evidence: `/etc/systemd/system/rally.service` contains `JWT_SECRET`, `DEEPSEEK_API_KEY`, and `ADMIN_AI_KEY` as literal `Environment=` values; `/etc/systemd/system/seemytrip.service` contains `ADMIN_AI_KEY`; `/etc/systemd/system/giftaroast.service` (app1) contains Twilio `AUTH_TOKEN`/`SID` and a `ADMIN_AI_KEY`. Unit files are 0644 (world-readable).
Rationale: Any local user (or any service compromise) can read live production API keys for DeepSeek, the admin-ai gateway, and Twilio. Secrets belong in root-only env files (600), not unit files.
### HIGH
**F-3. Every custom service runs as root.**
Evidence: docker inspect of all Core/app1 containers shows `User=""` (root); all `python3`/`node` listeners show `user root` in ss/ps. Only hermes-voice and Caddy run non-root.
Rationale: A single compromised service (e.g. a 2-year-old browserless) yields full root on the host, no privilege boundary.
**F-4. Very old / unpatched image tags in active service.**
Evidence: browserless/chrome:latest = 2 years old on both hosts; grafana/grafana:11.4.0 = ~20 months old (Core, internet-exposed via ufw); wazuh/wazuh-indexer:4.9.2 = 21 months old (app1); vaultwarden/server:1.33.2 = 18 months old tag still present on app1.
Rationale: These images predate many published CVEs and receive no updates. Browserless (headless Chrome) is a high-value attack surface and is also a Hermes tool dependency.
**F-5. Hermes gateway (most critical service) supervised by root's systemd USER manager, not a system unit.**
Evidence: `systemctl --user status hermes-gateway.service` shows active (user@0), while `systemctl status hermes-gateway.service` is `not-found`. `hermes-socat-8787.service` declares `After=hermes-gateway.service` against a unit that does not exist at the system level.
Rationale: The gateway is the core of ITPP automation. Its lifecycle depends on a user session staying alive (no lingering system-level restart guarantee) and its unit wiring is inconsistent (socat references a nonexistent system unit). Fragile single point of failure.
**F-6. Port 8200 collision between hermes-control-deck and pipeline-api.**
Evidence: both units are `active (running)` and both configure 127.0.0.1:8200. `ss -tlnp` shows 8200 held by pipeline-api (PID 3787644). hermes-control-deck Main PID (1962662) is not the socket owner.
Rationale: One of the two services is silently shadowed (the Control Deck API). Traffic routed by Caddy reaches whichever holds the port; the other is effectively down while appearing healthy.
**F-7. Three runaway `fix_dict.py` processes pegging 3 cores for 14+ days on app1.**
Evidence: PIDs 3656093/3657668/3658103 at 99.4% CPU, elapsed 14d 17h, cwd /root/giftaroast. Parent is an aborted `bash -c` deployment (truncated command visible in /proc).
Rationale: 25% of app1 CPU permanently wasted; indicates a bug in fix_dict.py and an interrupted deployment that was never cleaned up. Load average 3.26 on 12 cores.
**F-8. app1 Caddyfile (ingress for every app1 service) is not backed up.**
Evidence: grep of /root/backup.sh for Caddyfile returns nothing; no app1 script references /etc/caddy. Only Core's Caddyfile is backed up (by Core scripts).
Rationale: app1's entire reverse-proxy routing config (every site, TLS policy, upstream mapping) would need to be reconstructed by hand after a host failure. High blast radius, zero coverage.
### MEDIUM
**F-9. Ad-hoc `python -m http.server` running as root on public interfaces, one serving /tmp.**
Evidence: `python3 -m http.server 8080` (cwd /var/www/mockup/anita-consulting), `... 8934` (/var/www/mockup/itpropartner), `... 9876` (/tmp), all bound 0.0.0.0, none under systemd.
Rationale: Undocumented, unsupervised web servers run as root; serving /tmp is a direct path to accidental data exposure. These are "shadow IT" that bypasses the standard service lifecycle.
**F-10. Grafana exposed to Anywhere by firewall rule.**
Evidence: `ufw status` shows `3002/tcp ALLOW IN Anywhere` (Grafana). Prometheus (9090) and node_exporter (9100) also bind 0.0.0.0.
Rationale: Grafana (an old 11.4.0 with auth) and raw metrics endpoints are reachable from the public internet, not just the Tailscale/management network.
**F-11. World-readable credential files.**
Evidence: /root/projects/auth/.env = 0644, /etc/caddy/dre-passwd = 0644 (basic-auth password), /root/anita-key.json = 0644 (LiteLLM key alias), on app1.
Rationale: Live credentials readable by any local account, compounding F-3 (everything is root anyway, but defense-in-depth is absent).
**F-12. mysql-tunnel disables host key verification.**
Evidence: unit uses `-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null` to wphost02.
Rationale: The MySQL tunnel to wphost02 is susceptible to MITM/host-spoofing; a changed host would be silently accepted.
**F-13. Dead cron job: docker-volume-sync.sh does not exist.**
Evidence: crontab references `/root/.hermes/scripts/docker-volume-sync.sh` at 03:00 daily; the file does not exist (`ls` fails). Docker volume sync was reportedly moved to hermes-docker-sync.sh (different bucket), but the old crontab entry was never removed.
Rationale: A scheduled job that can never succeed, silently. Indicates cron hygiene drift.
**F-14. Pending security updates on both hosts (including kernel).**
Evidence: Core has 14 upgradable packages including linux-image-amd64 6.12.94 -> 6.12.101 (security) and chromium security update; app1 has 14 including docker-ce 29.6.1 -> 29.7.2.
Rationale: Kernel security updates pending on both hosts. Docker engine on app1 is a minor version behind. (Both run unattended-upgrades, but security packages remain pending.)
**F-15. No swap on either host.**
Evidence: `free -h` shows Swap 0 on both. Core sits at ~12 GB/15 GB used with 467 MB free.
Rationale: Under memory pressure the OOM killer will terminate arbitrary services (likely the biggest consumer: the 3.7 GB Hermes gateway). No graceful pressure relief.
**F-16. Dead reverse-proxy route: noc.itpropartner.com.**
Evidence: Caddyfile maps noc -> 127.0.0.1:8065, but no mattermost container is running on app1; mattermost-backup.sh runs daily as a no-op.
Rationale: A configured, monitored-adjacent route points at a service that is not running. Drift between routing config and reality.
**F-17. Super Search (designated critical) has no meaningful backup.**
Evidence: only a 373-byte .env snippet is uploaded; the code/config/venv under /root/docker/super-search (Core) and app1 is not covered by any backup script.
Rationale: Rebuilding the search MCP requires re-cloning and re-provisioning keys by hand; RPO effectively zero for its configuration.
### LOW
**F-18. /root/.aws directory is group/world writable on app1 (drwxrwxr-x).**
Evidence: `ls -ld /root/.aws` = 775. Credentials file itself is 600, but the directory is too open for a secrets dir.
**F-19. Leftover Docker volumes / image drift.**
Evidence: three Grafana volumes (grafana_data, grafana_data_final, grafana_data_v3) indicate config churn; multiple stale images (vaultwarden 1.33.2, litellm v1.84.0, ollama, mattermost) not running but present. Nine accumulated Caddyfile backup files in /etc/caddy.
**F-20. Single points of failure (no HA anywhere).**
Evidence: one Caddy per host, one Postgres (no replica), one Redis (no replica), one Hermes gateway, one LiteLLM + one litellm_postgres, one Wazuh single-node.
Rationale: No redundancy for any Tier-0 component. Acceptable for the environment size but must be documented and covered by DR runbooks.
---
## 8. RTO / RPO for Critical Services
| Service | RPO (est.) | RTO (est.) | Basis |
|---|---|---|---|
| Hermes gateway | ~24 h (daily full backup; 15-min live-sync status uncertain) | 2-4 h | restore.sh + DR-PLAN.md exist; warm standby at app1-bu (5.161.225.131) referenced but out of scope/unverified |
| LiteLLM / admin-ai | **config 24 h; database 0 (not backed up)** | hours to days | F-1: DB must be reconstructed; keys re-added manually |
| Caddy ingress (Core) | 24 h (Caddyfile) | 30 min - 1 h | reinstall caddy + restore Caddyfile |
| Caddy ingress (app1) | **none (unbacked)** | 1-4 h | F-8: full manual reconstruction of routing |
| Grafana / Prometheus | 24 h | 1-2 h | volume restore from core-services-backup |
| Super Search | **none (code/config unbacked)** | 1-3 h | re-clone + re-provision .env keys |
---
## 9. Top 5 Highest-Severity Findings (for parent summary)
1. **F-1 (Critical):** LiteLLM Postgres DB never backed up; dump targets wrong database name, so the AI gateway's keys/routing/spend are unprotected.
2. **F-2 (Critical):** Plaintext API keys (DeepSeek, admin-ai, Twilio) hardcoded in world-readable systemd unit files (rally, seemytrip, giftaroast).
3. **F-3 (High):** Every custom service runs as root, so any single compromise is full-host takeover.
4. **F-4 (High):** browserless/chrome (2 years old), grafana 11.4.0 (20 months), wazuh 4.9.2 (21 months) in active internet-facing service.
5. **F-5 (High):** Hermes gateway supervised by user manager only, with inconsistent unit wiring (socat references nonexistent system unit); fragile SPOF.
*End of Sys-A Phase One findings. No remediation performed. All items deferred to Phase Two.*
audit/phase-one/findings/sys-b.md
+330
View File
@@ -0,0 +1,330 @@
# Sys-B Findings - ITPP Phase One Infrastructure Audit (Read-Only)
**Auditor:** Sys-B (sysadmin inventory auditor)
**Scope:** app2 (152.53.39.202), app3 (152.53.241.111), app1-bu (5.161.225.131), wphost02 (5.161.62.38)
**Date:** 2026-08-13
**Method:** Read-only SSH (`ssh -i /root/.ssh/itpp-infra`, BatchMode, ConnectTimeout=10). No mutation performed. All remediation is deferred to Phase Two.
---
## 1. Host Reachability
| Host | IP | Role | Status |
|------|----|------|--------|
| app2 | 152.53.39.202 | Docker app host | REACHABLE |
| app3 | 152.53.241.111 | CloudPanel web host | REACHABLE |
| app1-bu | 5.161.225.131 | Warm standby (Core failover) | REACHABLE |
| wphost02 | 5.161.62.38 | Legacy WordPress / RunCloud | REACHABLE (NOT decommissioned) |
No access limitations. All four hosts answered over SSH with the shared key.
---
## 2. app2 (152.53.39.202) - Docker Application Host
**OS:** Debian (netcup), up 3+ weeks. Docker host running ~40 containers across 9 compose projects plus two orphan containers.
### 2.1 Per-Service Inventory
| Service | Image / Version | Purpose | Ports | Restart | Health | Notes |
|---------|-----------------|---------|-------|---------|--------|-------|
| support-api | support-api:latest (custom build, 3d) | Internal support API | 0.0.0.0:6880 | unless-stopped | healthy | Python healthcheck /health |
| bookstack | lscr.io/linuxserver/bookstack:latest | Docs (support.itpropartner.com) | 0.0.0.0:6875->80 | - | up | APP_URL set; SMTP via mail.germainebrown.com:2525 |
| bookstack-db | lscr.io/linuxserver/mariadb:latest | Bookstack DB | 3306 (internal) | - | healthy | - |
| happy_rosalind | lscr.io/linuxserver/bookstack:latest | ORPHANED 2nd Bookstack (no compose project, no host port) | 80/443 (internal only) | - | up | Auto-generated name; no external binding; apparent leftover |
| docker-ragflow-cpu-1 | infiniflow/ragflow:v0.26.4 | RAGFlow AI platform | 9380-9384, 9392->80, 9393->443 | - | up | - |
| docker-mysql-1 | mysql:8.0.39 (2yr old) | RAGFlow MySQL | 127.0.0.1:3306 | - | healthy | 2-year-old image tag |
| docker-minio-1 | pgsty/minio:RELEASE.2026-03-25 | RAGFlow object store | 127.0.0.1:9000, 0.0.0.0:9001 | - | healthy | - |
| docker-redis-1 | valkey/valkey:8 | RAGFlow cache | 127.0.0.1:6379 | - | healthy | - |
| docker-infinity-1 | infiniflow/infinity:v0.7.0 | RAGFlow vector DB | 0.0.0.0:23817/23820, 127.0.0.1:5432 | - | healthy | 23817/23820 exposed publicly |
| technitium | technitium/dns-server:latest | Authoritative DNS server | 0.0.0.0:53 tcp/udp, 127.0.0.1:5380 | - | healthy | Public recursive/authoritative DNS |
| dawarich_app | freikin/dawarich:latest | Location tracking (Dawarich) | 127.0.0.1:3002 | - | healthy | - |
| dawarich_sidekiq | freikin/dawarich:latest | Dawarich background jobs | 3000 (internal) | - | healthy | - |
| dawarich_db | postgis/postgis:17-3.5-alpine | Dawarich DB | 5432 (internal) | - | healthy | - |
| dawarich_redis | redis:7.4-alpine | Dawarich cache | 6379 (internal) | - | healthy | - |
| traccar | traccar/traccar:latest | GPS fleet tracking | 0.0.0.0:5000-5150 (tcp+udp), 0.0.0.0:8082 | - | healthy | 151 device ports publicly exposed |
| gitea | gitea/gitea:latest | Git server (git hosting) | 0.0.0.0:3022->22, 127.0.0.1:3001 | - | up | SSH port public |
| unifi-controller | jacobalberty/unifi:latest (8mo) | UniFi controller | 0.0.0.0:8080/8443/8843/8880, 3478/10001/udp | - | healthy | linuxserver/unifi image also present (5wk) - migration drift |
| unms-nginx | ubnt/unms-nginx:latest | UNMS reverse proxy | 0.0.0.0:81/8089/8444 | - | up | - |
| unms-api | ubnt/unms:latest | UNMS API (EOL product) | internal | - | healthy | UNMS discontinued by Ubiquiti 2021 |
| unms-device-ws-1..11 | ubnt/unms:latest | UNMS device websockets (11 replicas) | internal | - | healthy | - |
| unms-netflow | ubnt/unms-netflow:latest | UNMS netflow collector | 0.0.0.0:2055/udp | - | up | - |
| unms-postgres | ubnt/unms-postgres:latest | UNMS DB | 5432 (internal) | - | up | - |
| unms-siridb | ubnt/unms-siridb:latest | UNMS time-series DB | 9000/9010 (internal) | - | healthy | - |
| unms-rabbitmq | rabbitmq:3.7.28-alpine (5yr) | UNMS message broker | 4369/5671-5672/25672 | - | up | **5-year-old image, EOL, known CVEs** |
| unms-fluentd | ubnt/unms-fluentd:latest | UNMS logging | 5140, 127.0.0.1:24224 | - | up | - |
| ucrm | ubnt/unms-crm:4.5.33 | UCRM billing | 80-81, 443, 9000, 2055/udp | - | up | - |
| hudu-app-1 | hududocker/hudu:latest | Hudu IT documentation | 127.0.0.1:3000 | - | up | - |
| hudu-worker-1 | hududocker/hudu:latest | Hudu sidekiq worker | 3000 (internal) | - | up | restarted 15h ago |
| hudu-db-1 | postgres:16.2 (2yr) | Hudu DB | 5432 (internal) | - | up | 2-year-old postgres tag |
| hudu-redis-1 | redis:latest | Hudu cache | 6379 (internal) | - | up | - |
**Old/unused images present:** `ubnt/ucrm-conntrack:latest` (5yr), `mongo:7.0` (6wk, no container), `linuxserver/unifi-network-application:latest` (5wk, not the running unifi image), `caddy:latest`, `alpine:latest`.
### 2.2 Dependencies (text map)
- **RAGFlow** depends on: mysql-1, minio-1, redis-1, infinity-1. Downstream: end users of the RAGFlow UI.
- **Bookstack** depends on: bookstack-db (MariaDB). Downstream: support docs users.
- **Dawarich** depends on: dawarich_db (postgis), dawarich_redis, sidekiq worker.
- **UNMS stack** depends on: unms-postgres, unms-siridb, unms-rabbitmq, unms-fluentd, unms-nginx. **UCRM** shares the UNMS stack.
- **Hudu** depends on: hudu-db-1 (postgres), hudu-redis-1, hudu-worker-1.
- **Traccar, Gitea, Technitium, support-api, UniFi**: self-contained (single container each; Traccar/Gitea embed storage).
- All containers depend on Docker daemon + host disk. Technitium depends on external DNS delegations.
### 2.3 Config Files / Drift
- `/opt/support-api/docker-compose.yml` - single service, port 6880, Python /health healthcheck.
- `/opt/bookstack/docker-compose.yml` - `APP_URL=https://support.itpropartner.com`, SMTP `mail.germainebrown.com:2525`, DB/Mail passwords and APP_KEY present in compose (redacted in this report).
- `/opt/gitea/docker-compose.yml`, `/root/docker/hudu/docker-compose.yml`, `/root/docker/traccar/docker-compose.yml`, `/root/docker/dawarich/docker-compose.yml` - env-style secrets inline (redacted).
- `/home/unms/app/` - UNMS install (docker-compose, unms.conf, install-full.sh, update.sh).
- `/root/.aws/` - credentials + backups present (redacted). `/root/.docker/` - token seed files.
- **Drift:** orphan `happy_rosalind` Bookstack container (no compose project, no host port). Two UniFi images (jacobalberty running vs linuxserver pulled) suggest an in-flight migration.
### 2.4 Cron Jobs
| Schedule | User | Command | Purpose | Failure mode |
|----------|------|---------|---------|--------------|
| 30 2 * * * | root | `/root/backup.sh 2>&1 \| logger -t app2-backup` | Nightly backup to Wasabi S3 | See 2.5 |
No other system or user cron jobs beyond `/etc/cron.d` defaults.
### 2.5 Backup Status (CRITICAL GAP)
`/root/backup.sh` (Wasabi S3 target `hermes-vps-backups`, `s3.us-east-1.wasabisys.com`) calls per-service backup scripts. Live evidence from 2026-08-13 02:30 run:
- **Completed OK:** Traccar (DB+config), Dawarich, Technitium DNS, RAGFlow (MySQL dump + Infinity DB + Minio objects).
- **Logged "Backing up..." but NO completion line:** Gitea, Hudu, UNMS, UniFi.
- **Root cause:** `/root/backup.sh` references `gitea-backup.sh`, `hudu-backup.sh`, `unms-backup-sync.sh`, `unifi-backup-sync.sh` which **do not exist** under `/root/.hermes/scripts/`. Those sections are silently skipped.
- `bookstack-backup.sh`, `support-api-backup.sh`, `ragflow-backup.sh` exist but Bookstack and support-api are **not scheduled** in cron.
**Impact:** Gitea, Hudu, UNMS, UniFi, Bookstack, support-api have effectively NO working backup. No restore test is documented for any app2 service.
### 2.6 app2 Critical-Service RTO/RPO
| Service | RPO | RTO (est.) | Restore tested? |
|---------|-----|-----------|-----------------|
| Hudu | none (backup broken) | 4-8h | No |
| UNMS | none (backup broken, product EOL) | 4-8h | No |
| UniFi | none (backup broken) | 2-4h | No |
| Gitea | none (backup broken) | 2-4h | No |
| Bookstack / support-api | none (script not scheduled) | 2h | No |
| Traccar | 24h | 2-4h | No |
| Dawarich | 24h | 2h | No |
| Technitium DNS | 24h | 1h | No |
| RAGFlow | 24h | 2-4h | No |
---
## 3. app3 (152.53.241.111) - CloudPanel Web Host
**OS:** Debian 13 (trixie), up 34d. 31 GiB RAM / 12 CPU. `/` 1TB (80G used, 9%). Runs CloudPanel + nginx + Percona MySQL 8.4 + 10 PHP-FPM versions + Docker (Hexclave Stack Auth, Buzz relay) + several systemd app services.
### 3.1 Systemd Services
| Service | Version | Purpose | Notes |
|---------|---------|---------|-------|
| nginx | 1.30.4 | Primary web server | 80/443 |
| clp-nginx | - | CloudPanel control panel | 8443 |
| clp-agent / clp-php-fpm | - | CloudPanel agent | - |
| mysql (Percona Server) | 8.4.10 | Shared DB for all sites | 127.0.0.1:3306 (datadir /home/mysql) |
| php7.1-fpm ... php8.5-fpm | 7.1, 7.2, 7.3, 7.4, 8.0, 8.1, 8.2, 8.3, 8.4, 8.5 | Per-site PHP pools | **7.1-8.0 are EOL** |
| varnishd | 7.5.0 | HTTP cache | 6081 |
| memcached | - | Object cache | 11211 |
| redis | - | Object cache | 6379 |
| proftpd | - | FTP server | 21 (plaintext FTP) |
| postfix | - | Outbound mail | 25 |
| gitea.modelortho.service | Gitea | Git for git.modelortho.com | 3001 |
| msp-forms.service | FastAPI | Shared form handler (forms.itpropartner.com) | - |
| docs-auth-validator.service | - | Stack Auth JWT validator for docs.itpropartner.com | - |
| backup-restore.service | - | Backup-restore web UI (my.itpropartner.com/backup) | - |
| percona-telemetry-agent | - | Percona telemetry | - |
| fail2ban, cron, chrony, containerd, docker | - | platform | - |
### 3.2 Docker Services
| Container | Image | Purpose | Notes |
|-----------|-------|---------|-------|
| hexclave-* (server, postgres, clickhouse, cron) | stackauth/server:latest, postgres 16/17-alpine, clickhouse 25.10 | Hexclave Stack Auth | No backup coverage found |
| buzz-prod-* (relay, postgres, redis, minio) | ghcr.io/block/buzz:main | Buzz relay (Block open-source) | Production relay; no backup coverage found |
### 3.3 Sites Hosted (CloudPanel)
**WordPress (10 wp-config.php instances across 9 users):** apextrackexperience.com, boxpilotlogistics.com, debtrecoveryexperts.com (x2 - under BOTH `debtreecoveryexperts` and `drecovery` users), iamgmb.com, intelsight.io, mainwp.itpropartner.com, vigilanttac.com, voipsimplicity.com (+ `www` subdomain).
**Static / non-WordPress:** docs, forms, mockups, proposals, support, my.verdicttank.com, verdicttank.com, modelortho.com (+www), transitpin.com, my.transitpin.com, panel, my.voipsimplicity.com, timapta.org, katiewattsdesign.com, buzz.iamgmb.com, hexclave-api/dash, gmb, auth2 + auth2-api.
### 3.4 Cron Jobs
| Schedule | User | Command | Purpose |
|----------|------|---------|---------|
| 0 3 * * * | root | `/root/backup.sh \| logger -t app3-backup` | Nightly full backup |
| 0 1,13 * * * | root | `/opt/backup-restore/snapshot.sh` | Twice-daily WP snapshots |
| 30 4 * * * | root | `/root/gitea-modelortho-backup.sh \| logger -t gitea-backup` | Gitea (modelortho) backup |
| 15 3 * * * | clp | `clpctl db:backup ... --retentionPeriod=7` | CloudPanel DB backup |
| 15 4 * * * | clp | `/home/clp/scripts/create_backup.sh` | CloudPanel backup |
| 5-25 * * * * | clp | certbot/letsencrypt renewals, vhost import, cloudflare IPs | platform |
### 3.5 Backup Status
`/root/backup.sh` (2026-08-13 03:00 run, all OK) covers: CloudPanel SQLite DB, MSP Forms, **all 10 MySQL DBs**, **all 10 WordPress file trees**, static sites, nginx/configs. Destination Wasabi S3 `app3/`. `snapshot.sh` adds twice-daily local snapshots of WP sites (30-day retention). Gitea (modelortho) has its own nightly S3 backup.
**Backup gap:** the Docker services (Hexclave Stack Auth, Buzz relay) and TransitPin are **not** referenced by any backup script. Their postgres/clickhouse/minio data has no scheduled backup.
### 3.6 Config / Drift
- Plaintext MySQL root password hardcoded in BOTH `/root/backup.sh` (root-only) and `/opt/backup-restore/snapshot.sh` (**775 world-readable**). Same password in both.
- Duplicate WordPress site (debtrecoveryexperts.com) under two different system users - ambiguous ownership.
- `/home/.swap` = 2GB swap file on web data volume.
- Five EOL PHP-FPM runtimes (7.1, 7.2, 7.3, 7.4, 8.0) still running.
### 3.7 app3 Critical-Service RTO/RPO
| Service | RPO | RTO (est.) | Restore tested? |
|---------|-----|-----------|-----------------|
| CloudPanel + all WP/static sites | 12-24h (daily + 2x snapshots) | 4-8h | No |
| MySQL (10 DBs) | 24h | 4h | No |
| Gitea (modelortho) | 24h | 2h | No |
| Hexclave Stack Auth | none (no backup) | 4h | No |
| Buzz relay | none (no backup) | 4h | No |
| TransitPin | none (no backup) | 4h | No |
---
## 4. app1-bu (5.161.225.131) - Warm Standby for Core
**OS:** Ubuntu 24.04.4, up 28d. 3.7 GiB RAM / 3 CPU / 75G disk (37% used). No Docker. Runs only SSH, tailscale, fail2ban, cron + Hermes agent (v0.18.2). This is the warm standby for the live Core Hermes box.
### 4.1 Services
| Service | Purpose | Notes |
|---------|---------|-------|
| ssh | Admin access | 22 |
| tailscaled | Private mesh access | Tailscale |
| fail2ban | SSH brute-force protection | - |
| cron | Schedules watchdog + sync | - |
| hermes (binary) | Hermes Agent v0.18.2 installed | NOT currently running (dormant) |
### 4.2 Cron Jobs (the failover mechanism)
| Schedule | Command | Purpose | Last run |
|----------|---------|---------|----------|
| */5 * * * * | `/root/.hermes/scripts/hermes-standby-watchdog.sh` | Ping live Core; failover if down ~3.5 min | Active (log entries hourly through 2026-08-13) |
| */10 * * * * | `/root/.hermes/scripts/hermes-standby-sync.sh` | S3 sync of config/skills/plugins/cron/references | Active (last sync 2026-08-13T14:10Z) |
**Failover behavior (verified, NOT triggered):** the watchdog pings `LIVE_HOST=152.53.192.33`; if it fails 4 consecutive 60s cycles, it sends Telegram+email alerts, runs `aws s3 sync s3://hermes-vps-backups/live/ -> ~/.hermes/`, then `hermes gateway start`. A standalone "failover" cron does not exist; failover is embedded in the watchdog script (runs every 5 min). The sync script correctly skips sync when the live host is unreachable.
### 4.3 Standby Readiness Assessment (NOT fully ready)
- **Config-level state is fresh:** config.yaml (2026-08-12), .env (2026-08-12), skills/ (2026-08-12), cron output (2026-08-13). Synced every 10 min from S3 `live/`.
- **Data-level state is STALE:** `state.db` (2.1GB, Jul 15), `memory_store.db` (Jul 15), `sessions/` (Jul 15), `memories/` (Jul 15). The sync script deliberately excludes these (`DIRS="skills profiles plugins cron references"`, files `config.yaml .env .bashrc` only).
- **Consequence:** on failover, Hermes would start with ~4-week-old session, memory, and state DB. This is a config-level warm standby, NOT a true data-level warm standby.
### 4.4 Config / Drift (secrets)
- `/root/.hermes/.env` (mode **644, world-readable**) holds ~20 plaintext secrets: Cloudflare API token, Netcup API key + customer password, SyncroMSP token, Telegram bot token, **root passwords for app1/app2/app3** (`SERVER_152_53_*.PASS`), RingLogix creds, and API keys for OpenAI/Perplexity/Groq/xAI/Mistral/Fireworks/Google/Cohere.
- `/root/.hermes/migration-creds.txt` (644), `/root/.hermes/scripts/.hetzner_token` (644), `/root/.hermes/scripts/.netcup_api_key` (644) - further plaintext credentials.
- `hermes-standby-watchdog.sh` (mode **755, world-readable**) contains a plaintext email password and Telegram bot token inline.
- `/root/.hermes/state.db.corrupted` (1.8GB, Jul 9) and `state.db-wal` lingering.
---
## 5. wphost02 (5.161.62.38) - Legacy WordPress / RunCloud (STILL LIVE)
**OS:** Ubuntu 24.04.4, up 34d. 3.7 GiB RAM / 3 CPU / 75G disk (**82% full** - 59G used). RunCloud-managed LEMP stack. NOT decommissioned.
### 5.1 Services
| Service | Version | Purpose | Ports |
|---------|---------|---------|-------|
| nginx-rc | - | RunCloud web server | 80/443 |
| apache2-rc | - | RunCloud apache (secondary) | 127.0.0.1:81 |
| mariadb | 11.1.6 | Shared DB | 127.0.0.1:3306 |
| php81rc-fpm ... php85rc-fpm | 8.1-8.5 | Per-site PHP pools | - |
| runcloud-agent | - | RunCloud remote mgmt agent | *:34210 (rcsa service) |
| node_exporter | - | Prometheus exporter | 0.0.0.0:9100 |
| postfix | - | Mail | 25 |
| fail2ban, firewalld, supervisor | - | platform | - |
firewalld active zone `runcloud` allows only 22, 80, 443 + `rcsa` (RunCloud agent) inbound.
### 5.2 WordPress Sites (8, still active)
| Site | DB | Files (Aug activity) |
|------|----|---------------------|
| apextrackexperience | apextrackexperience_1781549652 | active (files modified Aug) |
| boxpilotlogistics | boxpilotlogistics_1770339547 | active |
| debtrecoveryexperts | debtrecoveryexperts_1778934554 | active |
| iAmGMB | iAmGMB_1764020288 | no changes since Jul 1 |
| katiewattsdesign | katiewattsdesign_1735425014 | active |
| MainWP | mainWP_1717713767 | active |
| vigilanttac | vigilanttac_1728911691 | active |
| voipsimplicity | voipsimplicity_1732250845 | active (13k files modified) |
All eight sites ALSO exist on app3 (CloudPanel). Both hosts appear live and actively modified -> **split-brain migration state**.
### 5.3 Cron Jobs
| Schedule | Command | Purpose |
|----------|---------|---------|
| */5 * * * * | `/root/apex-mail-watchdog-daemon.sh` | mail watchdog |
| 30 2 * * * | `/root/db-dump.sh \| logger -t db-dump` | nightly DB dump |
### 5.4 Backup Status (CRITICAL GAP)
- `/root/db-dump.sh` (scheduled daily 02:30) dumps **only 2 of 8** DBs (apextrackexperience, boxpilotlogistics) to **local** `/root/db-backups/` with **7-day retention**. No offsite copy.
- `/root/backup.sh` (full sites + all DBs + RunCloud config -> Wasabi S3) exists but is **NOT scheduled in any crontab** (grep across `/etc/cron*` and `/var/spool/cron` returned nothing).
- **Net effect:** 6 of 8 WordPress DBs and ALL site file trees have no running backup. 2 of 8 DBs have local-only 7-day backups. No restore test documented.
### 5.5 wphost02 Critical-Service RTO/RPO
| Service | RPO | RTO (est.) | Restore tested? |
|---------|-----|-----------|-----------------|
| apextrackexperience / boxpilotlogistics DBs | 24h (local only, 7d retention) | 4-8h | No |
| Other 6 WP DBs + all site files | none (no scheduled backup) | 4-8h | No |
---
## 6. Severity-Rated Findings
### CRITICAL
- **C1 - app2: Four production services have silently failing backups.** `/root/backup.sh` references `gitea-backup.sh`, `hudu-backup.sh`, `unms-backup-sync.sh`, `unifi-backup-sync.sh` which do not exist. The 2026-08-13 02:30 log shows "Backing up..." for these with no completion. Gitea, Hudu, UNMS, and UniFi have no effective backup; a loss would be unrecoverable. (Evidence: journalctl `app2-backup` + missing files under `/root/.hermes/scripts/`.)
- **C2 - app1-bu: World-readable secrets file.** `/root/.hermes/.env` (mode 644) contains ~20 plaintext credentials including root passwords for app1/app2/app3, Telegram bot token, Cloudflare/Netcup/SyncroMSP tokens, and eight AI-provider API keys. Any local user or compromised process can read the entire secret estate. (Evidence: `ls -la` + key names.)
- **C3 - app3: MySQL root password hardcoded in plaintext in backup scripts.** `/opt/backup-restore/snapshot.sh` is mode 775 (world-readable) and contains `MYSQL_PASS='[REDACTED]'`; the same password is in `/root/backup.sh`. Credential leakage plus shared superuser credential across all app3 databases. (Evidence: script contents.)
- **C4 - wphost02: Effective backup coverage is ~25% and local-only.** Scheduled `db-dump.sh` backs up 2 of 8 DBs to local disk (7-day retention); the full offsite S3 script `/root/backup.sh` is not in cron. 6 WordPress DBs and all site files have no running backup on a host that is still live. (Evidence: crontab + db-dump.sh + `grep backup.sh /etc/cron* /var/spool/cron` empty.)
- **C5 - app1-bu: Warm standby is not data-ready.** Sync covers only config/skills/plugins/cron/references; `state.db`, `memory_store.db`, and `sessions/` are stale at Jul 15 (~4 weeks). A failover today would restore Hermes without the last month of session, memory, and state. (Evidence: `stat` mtimes + sync script `DIRS`/excludes.)
### HIGH
- **H1 - app1-bu: Failover watchdog targets the wrong IP.** Watchdog pings `152.53.192.33` (labeled "Core/App1 netcup"), but `.env` and the audit brief reference Core/app1 at `152.53.36.131`. If the live Core moves/differs, the standby will fail over to nothing or never detect an outage. (Evidence: watchdog script `LIVE_HOST` vs `.env` `SERVER_152_53_36_131_PASS`.)
- **H2 - app1-bu: Plaintext email password + Telegram token in watchdog script (mode 755, world-readable).** (Evidence: script contents.)
- **H3 - app2: Very old images in production.** `rabbitmq:3.7.28-alpine` (5 years, EOL with known CVEs), `ubnt/ucrm-conntrack:latest` (5 years), `mysql:8.0.39` (2 years), `postgres:16.2` (2 years), `jacobalberty/unifi:latest` (8 months). (Evidence: `docker images`.)
- **H4 - app2: UNMS is end-of-life software.** Ubiquiti discontinued UNMS in 2021 (successor UISP). The entire ubnt/unms stack (13+ containers) is EOL and should be migrated. (Evidence: image set + product lifecycle.)
- **H5 - wphost02: Legacy host not decommissioned; split-brain with app3.** All 8 WordPress sites exist and are actively modified on BOTH wphost02 (RunCloud) and app3 (CloudPanel). No cutover or redirection evident; ambiguous source of truth for client content. (Evidence: identical site lists + Aug file mtimes on wphost02.)
- **H6 - app3: Dockerized production services have no backup.** Hexclave Stack Auth, Buzz relay, and TransitPin (postgres/clickhouse/minio data) are absent from every backup script. (Evidence: `grep -ril 'buzz|hexclave|transitpin' /root/*.sh /opt/backup-restore/` -> only DB content hits.)
- **H7 - app3: Five EOL PHP-FPM runtimes running.** PHP 7.1, 7.2, 7.3, 7.4, 8.0 are all end-of-life and unpatched. (Evidence: `systemctl list-units` php*-fpm.)
### MEDIUM
- **M1 - app2: Orphaned container `happy_rosalind`** (2nd Bookstack, no compose project, no host port binding) running with no apparent purpose.
- **M2 - app2: Large public attack surface.** 40 containers with many 0.0.0.0-bound ports: 53 (DNS), 5000-5150 (Traccar device range), UniFi 8080/8443/8843/8880, UNMS 81/8089/8444, netflow 2055, infinity 23817/23820, support-api 6880, bookstack 6875.
- **M3 - app2: Bookstack and support-api backup scripts exist but are not scheduled.** Effective RPO = none despite a written script.
- **M4 - app3: Duplicate WordPress install** debtrecoveryexperts.com under two users (`debtreecoveryexperts` and `drecovery`).
- **M5 - app3: Single-host SPOF.** ~25 client sites + shared MySQL + CloudPanel all on one VPS with no HA.
- **M6 - wphost02: node_exporter bound to 0.0.0.0:9100 and postfix on :25** on a legacy host.
### LOW
- **L1 - wphost02: Disk 82% full** (59G/75G).
- **L2 - app1-bu: `state.db.corrupted` (1.8GB) and stale state DBs linger** consuming disk.
- **L3 - app3: 2GB swap file `/home/.swap`** on the web data volume with 429MiB in use.
- **L4 - app3/wphost02: RunCloud agent (`rcsa`) port exposed** on a host slated for decommission - remote management surface retained.
---
## 7. Cross-Host Observations
- **No restore test is documented for any service on any host.** Backups are write-only everywhere a backup exists.
- **Backup destinations are Wasabi S3 (`hermes-vps-backups`)** for app2, app3, and app1-bu sync; wphost02's working backup is local-only.
- **Single points of failure are pervasive:** each application is on a single VPS; only Core (Hermes) has a standby, and that standby is config-level only.
- **Credentials are routinely stored in plaintext world-readable files** on app1-bu and app3 (and inline in compose files on app2).
audit/phase-one/findings/sys-c.md
+215
View File
@@ -0,0 +1,215 @@
# Sys-C Findings: Backup & Disaster Recovery Verification
**Auditor:** Sys-C (claude-sonnet-5) | **Engagement:** ITPP Phase One Audit | **Mode:** READ-ONLY
**Scope:** All 27+ backup targets vs live Wasabi S3, restore-test history, RTO/RPO evidence, 3-2-1 compliance.
**Rule applied throughout (Germaine):** "Backed up is not finished until a restore test is confirmed."
This document was completed across two runs (initial discovery + this resume). All findings below are verified against live S3 listings and source docs as of 2026-08-13, not assumed from documentation alone, per brief rule #3.
---
## 1. Restore-Test Coverage (the critical gap)
Only ONE restore-test event exists in ITPP history: **2026-08-10**, documented in `/root/projects/itpp-infrastructure/disaster-recovery/restore-test-log.md`. It tested exactly **2 of the ~34-37 actual backup targets** enumerated in the backup plan (the plan's own header claims "27 targets," see Finding SYSC-06 on that discrepancy).
| Target tested | Verdict | What was verified | Caveat found |
|---|---|---|---|
| Gitea (app2) | PASS | 117 DB tables, 52 repos, 3/3 sampled repos restored with valid git history via `git log`/`git rev-list` | Bare repos missing `refs/` dir (all refs packed) - restore requires manual `mkdir -p refs/heads refs/tags` workaround. Undocumented in the DR runbook until this test. |
| Vaultwarden (app1) | PASS | 29/29 tables match main vs backup DB, 123 ciphers intact, RSA key valid, WAL recovery clean | None - clean pass. |
**Every other backup target (25-35 of them depending on count) has ZERO restore-test evidence.** This includes every Critical and High tier service in the plan's own RTO/RPO table except Gitea: Hermes Agent (full), Traccar, UniFi/UNMS (the "UISP" critical tier), LiteLLM, n8n, Open WebUI, Twenty CRM. It also includes the standby failover path itself (app1-bu) - no evidence a full standby takeover has ever been drilled.
The restore-test log's own recommendations (written by the same automated tester) explicitly say: "Expand coverage - test remaining backup targets... LiteLLM, OpenWebUI, Hudu, Traccar, etc." and "Full-scale DR drill - after individual tests pass, schedule a coordinated full-stack restore to the standby server." Neither has happened as of this audit.
The restore-test log also flagged (2026-08-10, item 4): `core/vaultwarden/` backups stopped 2026-07-28 at 33 KB, stale/misconfigured - consistent with backup-plan.md's own "Stale S3 Paths - Cleanup Queue" listing that path as safe to delete (service migrated to app1). Not a live risk, just confirms the stale-path cleanup queue is accurate.
---
## 2. Per-Target Cross-Reference Table
Verified live against `s3://hermes-vps-backups/` and `s3://mikrotik-ccr-backups/` via `aws s3 ls --recursive` on 2026-08-13. "Last S3 object" is the actual most recent object under that prefix, not the backup-plan.md "Last Verified" column (which is stale documentation from 07-28/08-08 and was not trusted per brief rule #3).
### Core (152.53.192.33)
| # | Target | Schedule | Last S3 object (live) | Status | Retention (objects seen) | Restore-test |
|---|---|---|---|---|---|---|
| 1 | Hermes Agent (full) | Daily 1:00 AM | 2026-08-13 01:02 | OK | 72 dailies | NO |
| 2 | Hermes Live Sync | Every 15 min | 2026-08-13 06:00+ (continuous) | OK | rolling, includes cron output | NO |
| 3 | /root Essentials | Daily 3:00 AM | 2026-08-13 03:00 | OK (gaps: 07-11→07-12, 07-21, 07-22, 07-27 missing) | 31 files | NO |
| 4 | Grafana | Daily 1:30 AM | 2026-08-13 01:30 | OK | 25 dailies | NO |
| 5 | Uptime Kuma | Daily 1:30 AM | 2026-08-13 01:30 | OK | 25 dailies | NO |
| 6 | Docker Volumes (raw tars) | Daily 1:30 AM | **2026-08-08** 03:00 (5 days stale as of 08-13) | **STALE** | 55 objects, stopped growing | NO |
| 7 | Prometheus (TSDB snapshot) | Daily 1:30 AM | 2026-08-13 01:30 | OK | only **4** snapshots retained | NO |
| 8 | Auth API | 03:15 (+ dup 04:35 broken job) | 2026-08-13 03:15 | OK | 8 dailies | NO |
| 9 | Timetrex (undocumented - not in backup-plan.md's 27) | ~03:00 daily | 2026-08-13 03:00 | **OK schedule / BROKEN content** - sql.gz only 346 bytes | 2 days seen at this size | NO |
### App1 (152.53.36.131)
| # | Target | Schedule | Last S3 object (live) | Status | Retention | Restore-test |
|---|---|---|---|---|---|---|
| 10 | Open WebUI | 2:00 AM | 2026-08-13 02:01 | OK | 26 dailies | NO |
| 11 | LiteLLM | 3:30 AM | 2026-08-13 02:01 (config) | OK | 43 objects | NO |
| 12 | n8n | 2:00 AM | 2026-08-13 02:00 | OK | 26 dailies | NO |
| 13 | MCP Server Configs | 2:00 AM | 2026-08-13 02:01 | OK | 26 dailies | NO |
| 14 | Vaultwarden | 2:30 AM | 2026-08-13 02:30 | OK | 17 objects | **YES - PASS 08-10** |
| 15 | Komodo | 3:45 AM | 2026-08-13 03:45 | OK | 17 objects | NO |
| 16 | DocuSeal | 4:00 AM | 2026-08-13 04:00 | OK | 17 objects | NO |
| 17 | Twenty CRM | 4:15 AM | 2026-08-13 02:01 (files) | OK | 28 objects | NO |
| 18 | Kokoro TTS | N/A (stateless, no backup by design) | N/A | N/A - by design | N/A | N/A |
### App2 (152.53.39.202)
| # | Target | Schedule | Last S3 object (live) | Status | Retention | Restore-test |
|---|---|---|---|---|---|---|
| 19 | Hudu | 7:00 AM | 2026-08-13 07:00 | OK | 31 dumps | NO |
| 20 | Gitea | 8:00 AM | 2026-08-13 12:00 UTC | OK | daily dirs, thousands of repo objects | **YES - PASS 08-10 (refs/ caveat)** |
| 21 | UNMS | 6:00 AM (+ intraday auto) | 2026-08-13 06:00 | OK | 8 objects | NO |
| 22 | UniFi | 2:00 AM | 2026-08-13 02:00 | OK | 10 objects | NO |
| 23 | Traccar | 2:30 AM | 2026-08-13 02:30 | OK | 27 dailies | NO |
| 24 | Technitium DNS | 2:45 AM | 2026-08-13 02:45 | OK | 8 objects | NO |
| 25 | Dawarich | 4:00 AM | 2026-08-13 02:30 | OK | 8 objects | NO |
| 26 | RAGFlow | 4:15 AM | 2026-08-13 04:15 | OK | 27 objects | NO |
### App3 (152.53.241.111)
| # | Target | Schedule | Last S3 object (live) | Status | Retention | Restore-test |
|---|---|---|---|---|---|---|
| 27 | CloudPanel DB | 3:00 AM | 2026-08-13 03:00 | OK | 27 dailies | NO |
| 28 | MySQL (all DBs) | 3:00 AM | 2026-08-13 03:00 | OK | 221 objects (multi-DB x days) | NO |
| 29 | WordPress Files (per-site tars) | 3:00 AM | 2026-08-13 03:02 (confirmed via direct listing; do not trust naive sort) | OK | 273 objects | NO |
| 30 | Nginx Configs | 3:00 AM | 2026-08-13 03:03 | OK | 26 dailies | NO |
| 31 | Static Sites | 3:00 AM | 2026-08-13 03:02 | OK | 82 objects | NO |
| 32 | WordPress Snapshots (CloudPanel local) | 1AM/1PM | Local disk only, /opt/backup-restore/snapshots/, NOT in S3 | Out of S3 scope - 30-day local retention only, single point of failure if app3 disk dies | local only | NO |
| 33 | Hexclave (Stack Auth) | 3:30 AM | 2026-08-13 03:31 | OK | 6 objects | NO |
| 34 | modelortho.com | 4:30 AM | **2026-08-08** (5 days stale as of 08-13) | **STALE** | only 2 objects ever (site+configs) | NO |
### wphost02 (Hetzner)
| # | Target | Schedule | Last S3 object (live) | Status | Retention | Restore-test |
|---|---|---|---|---|---|---|
| 35 | WordPress (7 sites) | 5:00 AM | 2026-08-13 (per-day directories) | OK - 14-day retention confirmed working (DR-018 resolved) | 14 days x sites, 271+ objects | NO |
### Home Router / WISP
| # | Target | Schedule | Last S3 object (live) | Status | Retention | Restore-test |
|---|---|---|---|---|---|---|
| 36 | MikroTik CCR2004 (home gateway) | 6:00 AM | 2026-08-13 06:01 | OK | 30+ dailies since 07-04, config+log pairs | NO |
| 37 | MikroTik CCR (WISP tower) - DR-017 | 6:00 AM (expected) | **ZERO objects under `wisp-backups/configs/tower*`** | **MISSING entirely** | none | NO |
### External / Gaps
| Target | Schedule | Status | Notes |
|---|---|---|---|
| Hetzner Snapshots (API-driven disk snapshots) | Weekly Mon 5:00 AM | **NOT VERIFIED** - backup-plan.md's own "Last Verified" column is blank (" - "); not in S3 so not checkable via `aws s3 ls`; would need Hetzner Cloud API/console access, which is a live-system check outside this audit's read-only S3 tooling. Flag for Phase Two verification. | |
| SiteGround WordPress (non-MainWP sites) - DR-019 | None found | **MISSING** - `siteground/` prefix returns zero objects | Still open |
| app1-bu warm standby (state.db) | Continuous per design, actual: none for state.db | **COLD** - see Finding SYSC-01 | Config.yaml syncs every 10 min; state.db (2.1 GB) last touched 2026-07-15, ~28.7 days stale |
**Tally:** 34 real backup targets confirmed live and enumerable (excluding Kokoro N/A and the two external gaps that have no backup at all) + 1 undocumented (timetrex) + 2 confirmed-missing (WISP tower, SiteGround) + 1 unverifiable-in-scope (Hetzner snapshots) + 1 local-only-not-S3 (CloudPanel WP snapshots).
- **OK (current, correct schedule):** 30
- **STALE (schedule ok historically but no recent object):** 3 (Docker Volumes raw tars - 5 days; modelortho.com - 5 days; timetrex - running but payload broken/near-empty)
- **MISSING (zero backup coverage found):** 2 (WISP tower router configs, SiteGround non-MainWP sites)
- **UNVERIFIABLE with available read-only tooling:** 1 (Hetzner snapshots - needs Hetzner API access, not S3)
- **Restore-test confirmed:** 2 of 34+ (Gitea, Vaultwarden) - **94%+ of backup targets have never had a restore proven to work.**
---
## 3. RTO / RPO - Evidence-Based vs Plan-Stated
backup-plan.md states an aspirational RTO/RPO table by tier. Below is what the evidence actually supports.
| Tier | Services | Plan-stated RPO/RTO | Evidence-based RPO/RTO | Gap |
|---|---|---|---|---|
| Critical | Hermes Agent | RPO ≤1h / RTO ≤4h (assumes live-sync + standby cutover) | RPO: session state ~15 min (live sync confirmed running); full application state via standby is actually **~29 days stale** because state.db is never synced to app1-bu. RTO: **untested** - no full failover drill on record. | Standby is warm for config only, cold for state. Real RTO on a true Core failure is unknown and likely far worse than 4h once state.db has to be rebuilt/accepted-lossy. |
| Critical | Gitea | RPO ≤1h / RTO ≤4h | RPO: actual backup cadence is **once daily (8 AM)**, so real RPO is up to ~24h, not ≤1h. RTO: one restore test exists, DB+3 sample repos restored in ~30s, but that is a partial single-service test on /tmp, not a timed full-stack recovery - and it required an undocumented manual workaround (`refs/` dirs). | Plan's RPO claim of ≤1h is not supported by the actual cron schedule. RTO figure is aspirational, not measured. |
| Critical | Traccar | RPO ≤1h / RTO ≤4h | RPO: daily 2:30 AM dump only → real RPO ~24h. RTO: **never tested.** | Same RPO overstatement; RTO entirely unverified. |
| Critical | UISP (UniFi/UNMS) | RPO ≤1h / RTO ≤4h | RPO: UniFi backs up 2:00 AM daily, UNMS 6:00 AM + intraday auto-backups (best of the critical tier, effectively sub-daily). RTO: **never tested.** | UNMS RPO is reasonably close to plan; UniFi is daily only. Neither has a proven RTO. |
| High | LiteLLM, n8n, Open WebUI, Vaultwarden, Twenty CRM | RPO 24h / RTO ≤8h | Vaultwarden: RPO 24h matches actual daily cadence, and RTO is the only tier item with real evidence (~2s restore+verify in the 08-10 test, though that is a minimal single-DB test, not a full service standup with docker-compose). The other four: RPO matches (daily), RTO **never tested.** | Plan's RPO is accurate here; RTO is unverified for 4 of 5 services. |
| Medium | Hudu, UniFi, Komodo, DocuSeal, App3 WP sites, Auth API, Hexclave | RPO 24h / RTO ≤24h | All backups confirmed current daily, consistent with stated RPO. RTO: **never tested for any of them.** | RPO credible, RTO aspirational only. |
| Low | Grafana, Uptime Kuma, Prometheus, MikroTik CCR, Technitium DNS, Dawarich, RAGFlow | RPO 24h / RTO ≤48h | Backups current daily (except Prometheus only keeps 4 snapshots of retention - a retention risk, not an RPO problem). RTO never tested. MikroTik home gateway confirmed daily; MikroTik tower has **no backup at all**, so its real RTO/RPO is "recovery from scratch," not 48h. | Tower router falls outside even the Low tier's stated objective because it has zero backup. |
**Bottom line:** RPO figures in the plan are mostly defensible for daily-cadence services but overstated (≤1h) for the three daily-only Critical items. RTO figures across every tier are aspirational targets, not measured outcomes - only Gitea and Vaultwarden have ever actually been restored and timed, and both were partial, /tmp-only tests, not full production-equivalent recoveries.
---
## 4. Findings (severity-rated)
### CRITICAL
**SYSC-01 - Warm standby (app1-bu) is cold for application state, contradicting DR docs.**
Evidence: `state.db` on app1-bu (5.161.225.131) last modified 2026-07-15, ~28.7 days stale, 2.1 GB, while `config.yaml` syncs every 10 minutes. Root cause isolated: `/root/.hermes/scripts/hermes-standby-sync.sh` explicitly excludes state.db by design ("Skips massive state databases"), syncing only config/.env/.bashrc/skills/profiles/plugins/cron/references.
Why it matters: DR docs and the backup plan describe app1-bu as ready to take over "if Core goes down," but a failover today would bring up a standby with current configuration and roughly a month-old session/job/cron history. Anyone relying on that standby for continuity of active work, not just infrastructure config, will lose weeks of state silently.
**SYSC-02 - Duplicate/conflicting auth-api-backup cron jobs.**
Evidence: two cron entries invoke `auth-api-backup.sh` - 03:15 (status ok) and 04:35 (status error, exit 1).
Why it matters: Auth API backs all SSO for ITPP per backup-plan.md's own tier notes. A visibly-failing duplicate job is exactly the kind of noise that gets ignored until the good job also breaks and nobody notices because "the cron always shows an error anyway." Leftover from a rename/migration; needs cleanup, and the failing job should be confirmed harmless (not silently corrupting anything) before removal.
**SYSC-03 - 94%+ of backup targets have never had a restore test.**
Evidence: the single restore-test event on record (2026-08-10) covered exactly 2 targets (Gitea, Vaultwarden) out of 34+ live targets identified. Every other Critical and High tier service (Hermes Agent full backup, Traccar, UniFi, UNMS, LiteLLM, n8n, Open WebUI, Twenty CRM) has zero restore-test evidence.
Why it matters: per Germaine's own stated rule, a backup that has never been restored is not a finished backup - it is an assumption. Ransomware, disk failure, or a bad migration could reveal that any of these 32+ untested backups are unusable (as the Gitea test itself discovered a real, previously-unknown restore blocker: missing `refs/` directories). The fact that the one test performed found a real issue is itself evidence that untested backups carry material risk, not theoretical risk.
### HIGH
**SYSC-04 - WISP CCR tower router configs have zero backup coverage (DR-017, still open).**
Evidence: `s3://mikrotik-ccr-backups/wisp-backups/configs/tower*` returns zero objects. The home gateway router at the same prefix pattern (`configs/home/`) IS backed up daily and current through 2026-08-13.
Why it matters: if the tower router fails or is misconfigured, there is no saved configuration to restore from - full manual rebuild from memory/notes, unlike the home gateway which has 30+ days of dailies.
**SYSC-05 - Docker Volumes (raw tars) and modelortho.com backups are stale, not merely slow.**
Evidence: `volumes/` prefix (grafana_data_final, prometheus_data raw tars) last object 2026-08-08, 5 days stale as of audit date, while the DB-level dumps under `core/grafana/` and `core/prometheus/` continue daily and are current. `app3/modelortho/` last object also 2026-08-08, and it has only ever had 2 objects total (site + nginx config), suggesting it may have run once and stopped, or is intentionally infrequent.
Why it matters: this is independent live confirmation of the DR-002 discovery that `docker-volume-sync.sh` was deleted - its function is only partially replaced. The DB-level SQLite/TSDB dumps for Grafana and Prometheus are fine, but the raw volume-level tars (which would matter for a full container rebuild, not just data recovery) have quietly stopped. modelortho.com's near-total absence of backup history (2 objects ever, both 5 days stale) needs its own look - either it's a low-churn static site where infrequent backup is fine, or its cron entry silently stopped after one run.
### MEDIUM
**SYSC-06 - backup-plan.md's own target count ("27") does not match the actual enumerated inventory.**
Evidence: the document's header states "Backup Inventory (27 targets)" but the tables that follow it enumerate roughly 34-37 discrete backup line items across Core/App1/App2/App3/wphost02/Home Router/External, plus at least one undocumented target (timetrex) discovered only via live S3 inspection, not present in the plan's tables at all.
Why it matters: this is a documentation accuracy problem, not an operational one, but it matters for audit trust - if the plan's own headline number is wrong, other "Last Verified" dates in the same document (mostly frozen at 07-28 or 08-08, weeks stale relative to what's actually running) should not be trusted either, which is exactly why this audit verified everything live against S3 rather than the document.
**SYSC-07 - timetrex backup runs but produces a near-empty dump.**
Evidence: `core/timetrex/timetrex-2026-08-13.sql.gz` is 346 bytes, consistent across the days sampled. The companion storage/config tars are tiny but plausible (115 B, 1.5 KB) for a config-only backup, but a 346-byte SQL dump for what should be an application database is very unlikely to be a real, useful backup.
Why it matters: the schedule "succeeding" (no cron error, file lands in S3 daily) is actively misleading - it looks healthy on a dashboard but the data almost certainly isn't recoverable. This is exactly the kind of false-green backup that a restore test would have caught immediately. Root cause not assumed here per brief rule 2 (could be an empty/decommissioned database, a broken mysqldump auth, or an app issue) - needs Phase Two investigation, not remediation.
**SYSC-08 - Docker Volume Sync function (Prometheus/Grafana raw data) claimed-but-not-restore-verified.**
Evidence: `docker-volume-sync.sh` was deleted per DR-002; its function is claimed to be covered by `hermes-backup.sh`, but this audit found the actual raw-volume backups (`volumes/` prefix) are stale since 08-08 (see SYSC-05) and no restore test exists for any Grafana/Prometheus backup, DB-level or volume-level.
Why it matters: the claim of coverage is not supported by live evidence; the safety net here is unverified on two independent axes (currency and restorability).
**SYSC-09 - Prometheus TSDB snapshot retention is unusually shallow (4 objects).**
Evidence: `core/prometheus/` holds only 4 snapshot objects vs 25+ for comparable daily services (Grafana, Uptime Kuma).
Why it matters: if a problem with Prometheus data isn't noticed within roughly 4 days, there may be no earlier snapshot left to recover from. Likely an intentional retention policy given TSDB snapshot size, but worth confirming it's intentional rather than a bug.
**SYSC-10 - CloudPanel WordPress snapshot layer (app3, local-only) has no offsite copy.**
Evidence: `/opt/backup-restore/snapshot.sh` writes to local disk (`/opt/backup-restore/snapshots/`, 30-day retention) only; nothing under this specific mechanism reaches S3 (the separate `app3/wordpress/` S3 backups are a different script/mechanism and are current).
Why it matters: this specific snapshot layer is a single point of failure - if app3's disk fails, these particular snapshots are gone regardless of retention window. The S3-backed `app3/wordpress/` mechanism is a real offsite safety net for the same sites, so overall WordPress exposure on app3 is mitigated, but the local snapshot layer itself provides false comfort if someone assumes "snapshots" means "offsite."
### MEDIUM (carried forward, unchanged from prior run)
**SYSC-11 - DR-019: SiteGround WordPress sites outside MainWP have no S3 backup.** `siteground/` prefix returns zero objects. Still open.
**SYSC-12 - DR-015: service-health-check/apex-mail-watchdog failing on real remote outages.** Still open, unchanged.
### LOW / INFORMATIONAL
**SYSC-13 - sys-b.md vs live check discrepancy on Gitea/Hudu/UNMS/UniFi.**
sys-b.md reported these backups as "silently failing." This audit's live S3 check found all four running on schedule with fresh, current objects through 2026-08-13 (Hudu 07:00, Gitea 08:00/12:00 UTC, UNMS 06:00, UniFi 02:00). Flagging for conductor reconciliation rather than resolving unilaterally - either sys-b observed a transient failure window, checked a stale cache/log rather than live S3, or the issue was fixed between sys-b's check and this one. Recommend the conductor compare exact check timestamps between the two runs before deciding which report is stale.
**SYSC-14 - /root Essentials backup has 3 schedule gaps (07-11→07-12, 07-21, 07-22, 07-27 missing) but is otherwise current.**
Not fatal (the job clearly runs most days and is current through 08-13), but worth a Phase Two look at why specific days were skipped (server reboot, cron collision, disk pressure) rather than assuming a one-off blip.
**SYSC-15 - Hetzner weekly snapshot verification is outside this audit's read-only S3 tooling.**
backup-plan.md lists "Last Verified: - " (blank) for Hetzner Cloud API-driven snapshots. This audit could not verify snapshot existence/currency using `aws s3` tooling because they are not stored in S3. Flag for Phase Two: someone with Hetzner console/API read access should confirm snapshots are actually being taken weekly as claimed.
---
## 5. 3-2-1 Rule Compliance (quick read)
Most services have: (1) live production copy, (2) daily S3 backup at Wasabi (offsite), and for the app1-bu standby (3) a config-level copy on a second provider (Hetzner) - but that third copy is state-incomplete per SYSC-01. Genuine 3-2-1 gaps:
- WISP tower router: only 1 copy (live device config), zero backups (DR-017).
- SiteGround non-MainWP sites: only 1 copy (live host), zero backups (DR-019).
- CloudPanel local WordPress snapshots: 2 copies but both effectively on the same physical host (live + local snapshot dir) until the separate S3-backed `app3/wordpress/` mechanism is counted as the true offsite leg - which it is, so app3 WordPress overall is fine; the local snapshot layer specifically is not.
- Docker volume raw tars: technically offsite but stale 5 days, functionally degrading toward non-compliance if not fixed.
---
## Notes on Methodology / Limitations
- All S3 currency checks used `aws s3 ls --recursive` against `s3://hermes-vps-backups/` and `s3://mikrotik-ccr-backups/`, cross-checked with targeted per-prefix listings where a naive lexical sort across mixed filenames (different site/service names sharing a date) produced a misleading "last" result (caught and corrected for `app3/wordpress/`).
- No restore, config change, or live-system alteration was performed by this auditor, per brief rule 1. All restore-test evidence in this report comes from the pre-existing 2026-08-10 log, not from actions taken during this audit.
- Hetzner API-based snapshot verification and any live SSH-based service checks beyond what the prior Sys-C run already completed were not repeated in this resume, consistent with the instruction to finish, not redo, prior discovery.