itpp-infrastructure/audit/phase-one/findings/sys-b.md

# Sys-B Findings - ITPP Phase One Infrastructure Audit (Read-Only)

**Auditor:** Sys-B (sysadmin inventory auditor)
**Scope:** app2 (152.53.39.202), app3 (152.53.241.111), app1-bu (5.161.225.131), wphost02 (5.161.62.38, decommissioned 2026-08-28)
**Date:** 2026-08-13
**Method:** Read-only SSH (`ssh -i /root/.ssh/itpp-infra`, BatchMode, ConnectTimeout=10). No mutation performed. All remediation is deferred to Phase Two.

---

## 1. Host Reachability

| Host | IP | Role | Status |
|------|----|------|--------|
| app2 | 152.53.39.202 | Docker app host | REACHABLE |
| app3 | 152.53.241.111 | CloudPanel web host | REACHABLE |
| app1-bu | 5.161.225.131 | Warm standby (Core failover) | REACHABLE |
| wphost02 | 5.161.62.38 | Legacy WordPress / RunCloud | DECOMMISSIONED 2026-08-28 |

No access limitations. All four hosts answered over SSH with the shared key.

---

## 2. app2 (152.53.39.202) - Docker Application Host

**OS:** Debian (netcup), up 3+ weeks. Docker host running ~40 containers across 9 compose projects plus two orphan containers.

### 2.1 Per-Service Inventory

| Service | Image / Version | Purpose | Ports | Restart | Health | Notes |
|---------|-----------------|---------|-------|---------|--------|-------|
| support-api | support-api:latest (custom build, 3d) | Internal support API | 0.0.0.0:6880 | unless-stopped | healthy | Python healthcheck /health |
| bookstack | lscr.io/linuxserver/bookstack:latest | Docs (support.itpropartner.com) | 0.0.0.0:6875->80 | - | up | APP_URL set; SMTP via mail.germainebrown.com:2525 |
| bookstack-db | lscr.io/linuxserver/mariadb:latest | Bookstack DB | 3306 (internal) | - | healthy | - |
| happy_rosalind | lscr.io/linuxserver/bookstack:latest | ORPHANED 2nd Bookstack (no compose project, no host port) | 80/443 (internal only) | - | up | Auto-generated name; no external binding; apparent leftover |
| docker-ragflow-cpu-1 | infiniflow/ragflow:v0.26.4 | RAGFlow AI platform | 9380-9384, 9392->80, 9393->443 | - | up | - |
| docker-mysql-1 | mysql:8.0.39 (2yr old) | RAGFlow MySQL | 127.0.0.1:3306 | - | healthy | 2-year-old image tag |
| docker-minio-1 | pgsty/minio:RELEASE.2026-03-25 | RAGFlow object store | 127.0.0.1:9000, 0.0.0.0:9001 | - | healthy | - |
| docker-redis-1 | valkey/valkey:8 | RAGFlow cache | 127.0.0.1:6379 | - | healthy | - |
| docker-infinity-1 | infiniflow/infinity:v0.7.0 | RAGFlow vector DB | 0.0.0.0:23817/23820, 127.0.0.1:5432 | - | healthy | 23817/23820 exposed publicly |
| technitium | technitium/dns-server:latest | Authoritative DNS server | 0.0.0.0:53 tcp/udp, 127.0.0.1:5380 | - | healthy | Public recursive/authoritative DNS |
| dawarich_app | freikin/dawarich:latest | Location tracking (Dawarich) | 127.0.0.1:3002 | - | healthy | - |
| dawarich_sidekiq | freikin/dawarich:latest | Dawarich background jobs | 3000 (internal) | - | healthy | - |
| dawarich_db | postgis/postgis:17-3.5-alpine | Dawarich DB | 5432 (internal) | - | healthy | - |
| dawarich_redis | redis:7.4-alpine | Dawarich cache | 6379 (internal) | - | healthy | - |
| traccar | traccar/traccar:latest | GPS fleet tracking | 0.0.0.0:5000-5150 (tcp+udp), 0.0.0.0:8082 | - | healthy | 151 device ports publicly exposed |
| gitea | gitea/gitea:latest | Git server (git hosting) | 0.0.0.0:3022->22, 127.0.0.1:3001 | - | up | SSH port public |
| unifi-controller | jacobalberty/unifi:latest (8mo) | UniFi controller | 0.0.0.0:8080/8443/8843/8880, 3478/10001/udp | - | healthy | linuxserver/unifi image also present (5wk) - migration drift |
| unms-nginx | ubnt/unms-nginx:latest | UNMS reverse proxy | 0.0.0.0:81/8089/8444 | - | up | - |
| unms-api | ubnt/unms:latest | UNMS API (EOL product) | internal | - | healthy | UNMS discontinued by Ubiquiti 2021 |
| unms-device-ws-1..11 | ubnt/unms:latest | UNMS device websockets (11 replicas) | internal | - | healthy | - |
| unms-netflow | ubnt/unms-netflow:latest | UNMS netflow collector | 0.0.0.0:2055/udp | - | up | - |
| unms-postgres | ubnt/unms-postgres:latest | UNMS DB | 5432 (internal) | - | up | - |
| unms-siridb | ubnt/unms-siridb:latest | UNMS time-series DB | 9000/9010 (internal) | - | healthy | - |
| unms-rabbitmq | rabbitmq:3.7.28-alpine (5yr) | UNMS message broker | 4369/5671-5672/25672 | - | up | **5-year-old image, EOL, known CVEs** |
| unms-fluentd | ubnt/unms-fluentd:latest | UNMS logging | 5140, 127.0.0.1:24224 | - | up | - |
| ucrm | ubnt/unms-crm:4.5.33 | UCRM billing | 80-81, 443, 9000, 2055/udp | - | up | - |
| hudu-app-1 | hududocker/hudu:latest | Hudu IT documentation | 127.0.0.1:3000 | - | up | - |
| hudu-worker-1 | hududocker/hudu:latest | Hudu sidekiq worker | 3000 (internal) | - | up | restarted 15h ago |
| hudu-db-1 | postgres:16.2 (2yr) | Hudu DB | 5432 (internal) | - | up | 2-year-old postgres tag |
| hudu-redis-1 | redis:latest | Hudu cache | 6379 (internal) | - | up | - |

**Old/unused images present:** `ubnt/ucrm-conntrack:latest` (5yr), `mongo:7.0` (6wk, no container), `linuxserver/unifi-network-application:latest` (5wk, not the running unifi image), `caddy:latest`, `alpine:latest`.

### 2.2 Dependencies (text map)

- **RAGFlow** depends on: mysql-1, minio-1, redis-1, infinity-1. Downstream: end users of the RAGFlow UI.
- **Bookstack** depends on: bookstack-db (MariaDB). Downstream: support docs users.
- **Dawarich** depends on: dawarich_db (postgis), dawarich_redis, sidekiq worker.
- **UNMS stack** depends on: unms-postgres, unms-siridb, unms-rabbitmq, unms-fluentd, unms-nginx. **UCRM** shares the UNMS stack.
- **Hudu** depends on: hudu-db-1 (postgres), hudu-redis-1, hudu-worker-1.
- **Traccar, Gitea, Technitium, support-api, UniFi**: self-contained (single container each; Traccar/Gitea embed storage).
- All containers depend on Docker daemon + host disk. Technitium depends on external DNS delegations.

### 2.3 Config Files / Drift

- `/opt/support-api/docker-compose.yml` - single service, port 6880, Python /health healthcheck.
- `/opt/bookstack/docker-compose.yml` - `APP_URL=https://support.itpropartner.com`, SMTP `mail.germainebrown.com:2525`, DB/Mail passwords and APP_KEY present in compose (redacted in this report).
- `/opt/gitea/docker-compose.yml`, `/root/docker/hudu/docker-compose.yml`, `/root/docker/traccar/docker-compose.yml`, `/root/docker/dawarich/docker-compose.yml` - env-style secrets inline (redacted).
- `/home/unms/app/` - UNMS install (docker-compose, unms.conf, install-full.sh, update.sh).
- `/root/.aws/` - credentials + backups present (redacted). `/root/.docker/` - token seed files.
- **Drift:** orphan `happy_rosalind` Bookstack container (no compose project, no host port). Two UniFi images (jacobalberty running vs linuxserver pulled) suggest an in-flight migration.

### 2.4 Cron Jobs

| Schedule | User | Command | Purpose | Failure mode |
|----------|------|---------|---------|--------------|
| 30 2 * * * | root | `/root/backup.sh 2>&1 \| logger -t app2-backup` | Nightly backup to Wasabi S3 | See 2.5 |

No other system or user cron jobs beyond `/etc/cron.d` defaults.

### 2.5 Backup Status (CRITICAL GAP)

`/root/backup.sh` (Wasabi S3 target `hermes-vps-backups`, `s3.us-east-1.wasabisys.com`) calls per-service backup scripts. Live evidence from 2026-08-13 02:30 run:

- **Completed OK:** Traccar (DB+config), Dawarich, Technitium DNS, RAGFlow (MySQL dump + Infinity DB + Minio objects).
- **Logged "Backing up..." but NO completion line:** Gitea, Hudu, UNMS, UniFi.
- **Root cause:** `/root/backup.sh` references `gitea-backup.sh`, `hudu-backup.sh`, `unms-backup-sync.sh`, `unifi-backup-sync.sh` which **do not exist** under `/root/.hermes/scripts/`. Those sections are silently skipped.
- `bookstack-backup.sh`, `support-api-backup.sh`, `ragflow-backup.sh` exist but Bookstack and support-api are **not scheduled** in cron.

**Impact:** Gitea, Hudu, UNMS, UniFi, Bookstack, support-api have effectively NO working backup. No restore test is documented for any app2 service.

### 2.6 app2 Critical-Service RTO/RPO

| Service | RPO | RTO (est.) | Restore tested? |
|---------|-----|-----------|-----------------|
| Hudu | none (backup broken) | 4-8h | No |
| UNMS | none (backup broken, product EOL) | 4-8h | No |
| UniFi | none (backup broken) | 2-4h | No |
| Gitea | none (backup broken) | 2-4h | No |
| Bookstack / support-api | none (script not scheduled) | 2h | No |
| Traccar | 24h | 2-4h | No |
| Dawarich | 24h | 2h | No |
| Technitium DNS | 24h | 1h | No |
| RAGFlow | 24h | 2-4h | No |

---

## 3. app3 (152.53.241.111) - CloudPanel Web Host

**OS:** Debian 13 (trixie), up 34d. 31 GiB RAM / 12 CPU. `/` 1TB (80G used, 9%). Runs CloudPanel + nginx + Percona MySQL 8.4 + 10 PHP-FPM versions + Docker (Hexclave Stack Auth, Buzz relay) + several systemd app services.

### 3.1 Systemd Services

| Service | Version | Purpose | Notes |
|---------|---------|---------|-------|
| nginx | 1.30.4 | Primary web server | 80/443 |
| clp-nginx | - | CloudPanel control panel | 8443 |
| clp-agent / clp-php-fpm | - | CloudPanel agent | - |
| mysql (Percona Server) | 8.4.10 | Shared DB for all sites | 127.0.0.1:3306 (datadir /home/mysql) |
| php7.1-fpm ... php8.5-fpm | 7.1, 7.2, 7.3, 7.4, 8.0, 8.1, 8.2, 8.3, 8.4, 8.5 | Per-site PHP pools | **7.1-8.0 are EOL** |
| varnishd | 7.5.0 | HTTP cache | 6081 |
| memcached | - | Object cache | 11211 |
| redis | - | Object cache | 6379 |
| proftpd | - | FTP server | 21 (plaintext FTP) |
| postfix | - | Outbound mail | 25 |
| gitea.modelortho.service | Gitea | Git for git.modelortho.com | 3001 |
| msp-forms.service | FastAPI | Shared form handler (forms.itpropartner.com) | - |
| docs-auth-validator.service | - | Stack Auth JWT validator for docs.itpropartner.com | - |
| backup-restore.service | - | Backup-restore web UI (my.itpropartner.com/backup) | - |
| percona-telemetry-agent | - | Percona telemetry | - |
| fail2ban, cron, chrony, containerd, docker | - | platform | - |

### 3.2 Docker Services

| Container | Image | Purpose | Notes |
|-----------|-------|---------|-------|
| hexclave-* (server, postgres, clickhouse, cron) | stackauth/server:latest, postgres 16/17-alpine, clickhouse 25.10 | Hexclave Stack Auth | No backup coverage found |
| buzz-prod-* (relay, postgres, redis, minio) | ghcr.io/block/buzz:main | Buzz relay (Block open-source) | Production relay; no backup coverage found |

### 3.3 Sites Hosted (CloudPanel)

**WordPress (10 wp-config.php instances across 9 users):** apextrackexperience.com, boxpilotlogistics.com, debtrecoveryexperts.com (x2 - under BOTH `debtreecoveryexperts` and `drecovery` users), iamgmb.com, intelsight.io, mainwp.itpropartner.com, vigilanttac.com, voipsimplicity.com (+ `www` subdomain).

**Static / non-WordPress:** docs, forms, mockups, proposals, support, my.verdicttank.com, verdicttank.com, modelortho.com (+www), transitpin.com, my.transitpin.com, panel, my.voipsimplicity.com, timapta.org, katiewattsdesign.com, buzz.iamgmb.com, hexclave-api/dash, gmb, auth2 + auth2-api.

### 3.4 Cron Jobs

| Schedule | User | Command | Purpose |
|----------|------|---------|---------|
| 0 3 * * * | root | `/root/backup.sh \| logger -t app3-backup` | Nightly full backup |
| 0 1,13 * * * | root | `/opt/backup-restore/snapshot.sh` | Twice-daily WP snapshots |
| 30 4 * * * | root | `/root/gitea-modelortho-backup.sh \| logger -t gitea-backup` | Gitea (modelortho) backup |
| 15 3 * * * | clp | `clpctl db:backup ... --retentionPeriod=7` | CloudPanel DB backup |
| 15 4 * * * | clp | `/home/clp/scripts/create_backup.sh` | CloudPanel backup |
| 5-25 * * * * | clp | certbot/letsencrypt renewals, vhost import, cloudflare IPs | platform |

### 3.5 Backup Status

`/root/backup.sh` (2026-08-13 03:00 run, all OK) covers: CloudPanel SQLite DB, MSP Forms, **all 10 MySQL DBs**, **all 10 WordPress file trees**, static sites, nginx/configs. Destination Wasabi S3 `app3/`. `snapshot.sh` adds twice-daily local snapshots of WP sites (30-day retention). Gitea (modelortho) has its own nightly S3 backup.

**Backup gap:** the Docker services (Hexclave Stack Auth, Buzz relay) and TransitPin are **not** referenced by any backup script. Their postgres/clickhouse/minio data has no scheduled backup.

### 3.6 Config / Drift

- Plaintext MySQL root password hardcoded in BOTH `/root/backup.sh` (root-only) and `/opt/backup-restore/snapshot.sh` (**775 world-readable**). Same password in both.
- Duplicate WordPress site (debtrecoveryexperts.com) under two different system users - ambiguous ownership.
- `/home/.swap` = 2GB swap file on web data volume.
- Five EOL PHP-FPM runtimes (7.1, 7.2, 7.3, 7.4, 8.0) still running.

### 3.7 app3 Critical-Service RTO/RPO

| Service | RPO | RTO (est.) | Restore tested? |
|---------|-----|-----------|-----------------|
| CloudPanel + all WP/static sites | 12-24h (daily + 2x snapshots) | 4-8h | No |
| MySQL (10 DBs) | 24h | 4h | No |
| Gitea (modelortho) | 24h | 2h | No |
| Hexclave Stack Auth | none (no backup) | 4h | No |
| Buzz relay | none (no backup) | 4h | No |
| TransitPin | none (no backup) | 4h | No |

---

## 4. app1-bu (5.161.225.131) - Warm Standby for Core

**OS:** Ubuntu 24.04.4, up 28d. 3.7 GiB RAM / 3 CPU / 75G disk (37% used). No Docker. Runs only SSH, tailscale, fail2ban, cron + Hermes agent (v0.18.2). This is the warm standby for the live Core Hermes box.

### 4.1 Services

| Service | Purpose | Notes |
|---------|---------|-------|
| ssh | Admin access | 22 |
| tailscaled | Private mesh access | Tailscale |
| fail2ban | SSH brute-force protection | - |
| cron | Schedules watchdog + sync | - |
| hermes (binary) | Hermes Agent v0.18.2 installed | NOT currently running (dormant) |

### 4.2 Cron Jobs (the failover mechanism)

| Schedule | Command | Purpose | Last run |
|----------|---------|---------|----------|
| */5 * * * * | `/root/.hermes/scripts/hermes-standby-watchdog.sh` | Ping live Core; failover if down ~3.5 min | Active (log entries hourly through 2026-08-13) |
| */10 * * * * | `/root/.hermes/scripts/hermes-standby-sync.sh` | S3 sync of config/skills/plugins/cron/references | Active (last sync 2026-08-13T14:10Z) |

**Failover behavior (verified, NOT triggered):** the watchdog pings `LIVE_HOST=152.53.192.33`; if it fails 4 consecutive 60s cycles, it sends Telegram+email alerts, runs `aws s3 sync s3://hermes-vps-backups/live/ -> ~/.hermes/`, then `hermes gateway start`. A standalone "failover" cron does not exist; failover is embedded in the watchdog script (runs every 5 min). The sync script correctly skips sync when the live host is unreachable.

### 4.3 Standby Readiness Assessment (NOT fully ready)

- **Config-level state is fresh:** config.yaml (2026-08-12), .env (2026-08-12), skills/ (2026-08-12), cron output (2026-08-13). Synced every 10 min from S3 `live/`.
- **Data-level state is STALE:** `state.db` (2.1GB, Jul 15), `memory_store.db` (Jul 15), `sessions/` (Jul 15), `memories/` (Jul 15). The sync script deliberately excludes these (`DIRS="skills profiles plugins cron references"`, files `config.yaml .env .bashrc` only).
- **Consequence:** on failover, Hermes would start with ~4-week-old session, memory, and state DB. This is a config-level warm standby, NOT a true data-level warm standby.

### 4.4 Config / Drift (secrets)

- `/root/.hermes/.env` (mode **644, world-readable**) holds ~20 plaintext secrets: Cloudflare API token, Netcup API key + customer password, SyncroMSP token, Telegram bot token, **root passwords for app1/app2/app3** (`SERVER_152_53_*.PASS`), RingLogix creds, and API keys for OpenAI/Perplexity/Groq/xAI/Mistral/Fireworks/Google/Cohere.
- `/root/.hermes/migration-creds.txt` (644), `/root/.hermes/scripts/.hetzner_token` (644), `/root/.hermes/scripts/.netcup_api_key` (644) - further plaintext credentials.
- `hermes-standby-watchdog.sh` (mode **755, world-readable**) contains a plaintext email password and Telegram bot token inline.
- `/root/.hermes/state.db.corrupted` (1.8GB, Jul 9) and `state.db-wal` lingering.

---

## 5. wphost02 (5.161.62.38) - Legacy WordPress / RunCloud (DECOMMISSIONED 2026-08-28)

**OS:** Ubuntu 24.04.4, up 34d. 3.7 GiB RAM / 3 CPU / 75G disk (**82% full** - 59G used). RunCloud-managed LEMP stack. Decommissioned 2026-08-28 (all 8 WordPress sites migrated to app3).

### 5.1 Services

| Service | Version | Purpose | Ports |
|---------|---------|---------|-------|
| nginx-rc | - | RunCloud web server | 80/443 |
| apache2-rc | - | RunCloud apache (secondary) | 127.0.0.1:81 |
| mariadb | 11.1.6 | Shared DB | 127.0.0.1:3306 |
| php81rc-fpm ... php85rc-fpm | 8.1-8.5 | Per-site PHP pools | - |
| runcloud-agent | - | RunCloud remote mgmt agent | *:34210 (rcsa service) |
| node_exporter | - | Prometheus exporter | 0.0.0.0:9100 |
| postfix | - | Mail | 25 |
| fail2ban, firewalld, supervisor | - | platform | - |

firewalld active zone `runcloud` allows only 22, 80, 443 + `rcsa` (RunCloud agent) inbound.

### 5.2 WordPress Sites (8, still active)

| Site | DB | Files (Aug activity) |
|------|----|---------------------|
| apextrackexperience | apextrackexperience_1781549652 | active (files modified Aug) |
| boxpilotlogistics | boxpilotlogistics_1770339547 | active |
| debtrecoveryexperts | debtrecoveryexperts_1778934554 | active |
| iAmGMB | iAmGMB_1764020288 | no changes since Jul 1 |
| katiewattsdesign | katiewattsdesign_1735425014 | active |
| MainWP | mainWP_1717713767 | active |
| vigilanttac | vigilanttac_1728911691 | active |
| voipsimplicity | voipsimplicity_1732250845 | active (13k files modified) |

All eight sites ALSO exist on app3 (CloudPanel). Both hosts appear live and actively modified -> **split-brain migration state**.

### 5.3 Cron Jobs

| Schedule | Command | Purpose |
|----------|---------|---------|
| */5 * * * * | `/root/apex-mail-watchdog-daemon.sh` | mail watchdog |
| 30 2 * * * | `/root/db-dump.sh \| logger -t db-dump` | nightly DB dump |

### 5.4 Backup Status (CRITICAL GAP)

- `/root/db-dump.sh` (scheduled daily 02:30) dumps **only 2 of 8** DBs (apextrackexperience, boxpilotlogistics) to **local** `/root/db-backups/` with **7-day retention**. No offsite copy.
- `/root/backup.sh` (full sites + all DBs + RunCloud config -> Wasabi S3) exists but is **NOT scheduled in any crontab** (grep across `/etc/cron*` and `/var/spool/cron` returned nothing).
- **Net effect:** 6 of 8 WordPress DBs and ALL site file trees have no running backup. 2 of 8 DBs have local-only 7-day backups. No restore test documented.

### 5.5 wphost02 Critical-Service RTO/RPO

| Service | RPO | RTO (est.) | Restore tested? |
|---------|-----|-----------|-----------------|
| apextrackexperience / boxpilotlogistics DBs | 24h (local only, 7d retention) | 4-8h | No |
| Other 6 WP DBs + all site files | none (no scheduled backup) | 4-8h | No |

---

## 6. Severity-Rated Findings

### CRITICAL

- **C1 - app2: Four production services have silently failing backups.** `/root/backup.sh` references `gitea-backup.sh`, `hudu-backup.sh`, `unms-backup-sync.sh`, `unifi-backup-sync.sh` which do not exist. The 2026-08-13 02:30 log shows "Backing up..." for these with no completion. Gitea, Hudu, UNMS, and UniFi have no effective backup; a loss would be unrecoverable. (Evidence: journalctl `app2-backup` + missing files under `/root/.hermes/scripts/`.)
- **C2 - app1-bu: World-readable secrets file.** `/root/.hermes/.env` (mode 644) contains ~20 plaintext credentials including root passwords for app1/app2/app3, Telegram bot token, Cloudflare/Netcup/SyncroMSP tokens, and eight AI-provider API keys. Any local user or compromised process can read the entire secret estate. (Evidence: `ls -la` + key names.)
- **C3 - app3: MySQL root password hardcoded in plaintext in backup scripts.** `/opt/backup-restore/snapshot.sh` is mode 775 (world-readable) and contains `MYSQL_PASS='[REDACTED]'`; the same password is in `/root/backup.sh`. Credential leakage plus shared superuser credential across all app3 databases. (Evidence: script contents.)
- **C4 - wphost02: Effective backup coverage is ~25% and local-only.** Scheduled `db-dump.sh` backs up 2 of 8 DBs to local disk (7-day retention); the full offsite S3 script `/root/backup.sh` is not in cron. 6 WordPress DBs and all site files have no running backup on a host that is still live. (Evidence: crontab + db-dump.sh + `grep backup.sh /etc/cron* /var/spool/cron` empty.)
- **C5 - app1-bu: Warm standby is not data-ready.** Sync covers only config/skills/plugins/cron/references; `state.db`, `memory_store.db`, and `sessions/` are stale at Jul 15 (~4 weeks). A failover today would restore Hermes without the last month of session, memory, and state. (Evidence: `stat` mtimes + sync script `DIRS`/excludes.)

### HIGH

- **H1 - app1-bu: Failover watchdog targets the wrong IP.** Watchdog pings `152.53.192.33` (labeled "Core/App1 netcup"), but `.env` and the audit brief reference Core/app1 at `152.53.36.131`. If the live Core moves/differs, the standby will fail over to nothing or never detect an outage. (Evidence: watchdog script `LIVE_HOST` vs `.env` `SERVER_152_53_36_131_PASS`.)
- **H2 - app1-bu: Plaintext email password + Telegram token in watchdog script (mode 755, world-readable).** (Evidence: script contents.)
- **H3 - app2: Very old images in production.** `rabbitmq:3.7.28-alpine` (5 years, EOL with known CVEs), `ubnt/ucrm-conntrack:latest` (5 years), `mysql:8.0.39` (2 years), `postgres:16.2` (2 years), `jacobalberty/unifi:latest` (8 months). (Evidence: `docker images`.)
- **H4 - app2: UNMS is end-of-life software.** Ubiquiti discontinued UNMS in 2021 (successor UISP). The entire ubnt/unms stack (13+ containers) is EOL and should be migrated. (Evidence: image set + product lifecycle.)
- **H5 - wphost02: Legacy host not decommissioned; split-brain with app3.** All 8 WordPress sites exist and are actively modified on BOTH wphost02 (RunCloud) and app3 (CloudPanel). No cutover or redirection evident; ambiguous source of truth for client content. (Evidence: identical site lists + Aug file mtimes on wphost02.)
- **H6 - app3: Dockerized production services have no backup.** Hexclave Stack Auth, Buzz relay, and TransitPin (postgres/clickhouse/minio data) are absent from every backup script. (Evidence: `grep -ril 'buzz|hexclave|transitpin' /root/*.sh /opt/backup-restore/` -> only DB content hits.)
- **H7 - app3: Five EOL PHP-FPM runtimes running.** PHP 7.1, 7.2, 7.3, 7.4, 8.0 are all end-of-life and unpatched. (Evidence: `systemctl list-units` php*-fpm.)

### MEDIUM

- **M1 - app2: Orphaned container `happy_rosalind`** (2nd Bookstack, no compose project, no host port binding) running with no apparent purpose.
- **M2 - app2: Large public attack surface.** 40 containers with many 0.0.0.0-bound ports: 53 (DNS), 5000-5150 (Traccar device range), UniFi 8080/8443/8843/8880, UNMS 81/8089/8444, netflow 2055, infinity 23817/23820, support-api 6880, bookstack 6875.
- **M3 - app2: Bookstack and support-api backup scripts exist but are not scheduled.** Effective RPO = none despite a written script.
- **M4 - app3: Duplicate WordPress install** debtrecoveryexperts.com under two users (`debtreecoveryexperts` and `drecovery`).
- **M5 - app3: Single-host SPOF.** ~25 client sites + shared MySQL + CloudPanel all on one VPS with no HA.
- **M6 - wphost02: node_exporter bound to 0.0.0.0:9100 and postfix on :25** on a legacy host.

### LOW

- **L1 - wphost02: Disk 82% full** (59G/75G).
- **L2 - app1-bu: `state.db.corrupted` (1.8GB) and stale state DBs linger** consuming disk.
- **L3 - app3: 2GB swap file `/home/.swap`** on the web data volume with 429MiB in use.
- **L4 - app3/wphost02: RunCloud agent (`rcsa`) port exposed** on a host slated for decommission - remote management surface retained.

---

## 7. Cross-Host Observations

- **No restore test is documented for any service on any host.** Backups are write-only everywhere a backup exists.
- **Backup destinations are Wasabi S3 (`hermes-vps-backups`)** for app2, app3, and app1-bu sync; wphost02's working backup is local-only.
- **Single points of failure are pervasive:** each application is on a single VPS; only Core (Hermes) has a standby, and that standby is config-level only.
- **Credentials are routinely stored in plaintext world-readable files** on app1-bu and app3 (and inline in compose files on app2).