itpp-infrastructure/audit/phase-one/findings/sys-b.md

Sys-B Findings - ITPP Phase One Infrastructure Audit (Read-Only)

Auditor: Sys-B (sysadmin inventory auditor) Scope: app2 (152.53.39.202), app3 (152.53.241.111), app1-bu (5.161.225.131), wphost02 (5.161.62.38) Date: 2026-08-13 Method: Read-only SSH (ssh -i /root/.ssh/itpp-infra, BatchMode, ConnectTimeout=10). No mutation performed. All remediation is deferred to Phase Two.

---

1. Host Reachability

Host	IP	Role	Status
app2	152.53.39.202	Docker app host	REACHABLE
app3	152.53.241.111	CloudPanel web host	REACHABLE
app1-bu	5.161.225.131	Warm standby (Core failover)	REACHABLE
wphost02	5.161.62.38	Legacy WordPress / RunCloud	REACHABLE (NOT decommissioned)

No access limitations. All four hosts answered over SSH with the shared key.

---

2. app2 (152.53.39.202) - Docker Application Host

OS: Debian (netcup), up 3+ weeks. Docker host running ~40 containers across 9 compose projects plus two orphan containers.

2.1 Per-Service Inventory

Service	Image / Version	Purpose	Ports	Restart	Health	Notes
support-api	support-api:latest (custom build, 3d)	Internal support API	0.0.0.0:6880	unless-stopped	healthy	Python healthcheck /health
bookstack	lscr.io/linuxserver/bookstack:latest	Docs (support.itpropartner.com)	0.0.0.0:6875->80	-	up	APP_URL set; SMTP via mail.germainebrown.com:2525
bookstack-db	lscr.io/linuxserver/mariadb:latest	Bookstack DB	3306 (internal)	-	healthy	-
happy_rosalind	lscr.io/linuxserver/bookstack:latest	ORPHANED 2nd Bookstack (no compose project, no host port)	80/443 (internal only)	-	up	Auto-generated name; no external binding; apparent leftover
docker-ragflow-cpu-1	infiniflow/ragflow:v0.26.4	RAGFlow AI platform	9380-9384, 9392->80, 9393->443	-	up	-
docker-mysql-1	mysql:8.0.39 (2yr old)	RAGFlow MySQL	127.0.0.1:3306	-	healthy	2-year-old image tag
docker-minio-1	pgsty/minio:RELEASE.2026-03-25	RAGFlow object store	127.0.0.1:9000, 0.0.0.0:9001	-	healthy	-
docker-redis-1	valkey/valkey:8	RAGFlow cache	127.0.0.1:6379	-	healthy	-
docker-infinity-1	infiniflow/infinity:v0.7.0	RAGFlow vector DB	0.0.0.0:23817/23820, 127.0.0.1:5432	-	healthy	23817/23820 exposed publicly
technitium	technitium/dns-server:latest	Authoritative DNS server	0.0.0.0:53 tcp/udp, 127.0.0.1:5380	-	healthy	Public recursive/authoritative DNS
dawarich_app	freikin/dawarich:latest	Location tracking (Dawarich)	127.0.0.1:3002	-	healthy	-
dawarich_sidekiq	freikin/dawarich:latest	Dawarich background jobs	3000 (internal)	-	healthy	-
dawarich_db	postgis/postgis:17-3.5-alpine	Dawarich DB	5432 (internal)	-	healthy	-
dawarich_redis	redis:7.4-alpine	Dawarich cache	6379 (internal)	-	healthy	-
traccar	traccar/traccar:latest	GPS fleet tracking	0.0.0.0:5000-5150 (tcp+udp), 0.0.0.0:8082	-	healthy	151 device ports publicly exposed
gitea	gitea/gitea:latest	Git server (git hosting)	0.0.0.0:3022->22, 127.0.0.1:3001	-	up	SSH port public
unifi-controller	jacobalberty/unifi:latest (8mo)	UniFi controller	0.0.0.0:8080/8443/8843/8880, 3478/10001/udp	-	healthy	linuxserver/unifi image also present (5wk) - migration drift
unms-nginx	ubnt/unms-nginx:latest	UNMS reverse proxy	0.0.0.0:81/8089/8444	-	up	-
unms-api	ubnt/unms:latest	UNMS API (EOL product)	internal	-	healthy	UNMS discontinued by Ubiquiti 2021
unms-device-ws-1..11	ubnt/unms:latest	UNMS device websockets (11 replicas)	internal	-	healthy	-
unms-netflow	ubnt/unms-netflow:latest	UNMS netflow collector	0.0.0.0:2055/udp	-	up	-
unms-postgres	ubnt/unms-postgres:latest	UNMS DB	5432 (internal)	-	up	-
unms-siridb	ubnt/unms-siridb:latest	UNMS time-series DB	9000/9010 (internal)	-	healthy	-
unms-rabbitmq	rabbitmq:3.7.28-alpine (5yr)	UNMS message broker	4369/5671-5672/25672	-	up	5-year-old image, EOL, known CVEs
unms-fluentd	ubnt/unms-fluentd:latest	UNMS logging	5140, 127.0.0.1:24224	-	up	-
ucrm	ubnt/unms-crm:4.5.33	UCRM billing	80-81, 443, 9000, 2055/udp	-	up	-
hudu-app-1	hududocker/hudu:latest	Hudu IT documentation	127.0.0.1:3000	-	up	-
hudu-worker-1	hududocker/hudu:latest	Hudu sidekiq worker	3000 (internal)	-	up	restarted 15h ago
hudu-db-1	postgres:16.2 (2yr)	Hudu DB	5432 (internal)	-	up	2-year-old postgres tag
hudu-redis-1	redis:latest	Hudu cache	6379 (internal)	-	up	-

Old/unused images present: ubnt/ucrm-conntrack:latest (5yr), mongo:7.0 (6wk, no container), linuxserver/unifi-network-application:latest (5wk, not the running unifi image), caddy:latest, alpine:latest.

2.2 Dependencies (text map)

• RAGFlow depends on: mysql-1, minio-1, redis-1, infinity-1. Downstream: end users of the RAGFlow UI.
• Bookstack depends on: bookstack-db (MariaDB). Downstream: support docs users.
• Dawarich depends on: dawarich_db (postgis), dawarich_redis, sidekiq worker.
• UNMS stack depends on: unms-postgres, unms-siridb, unms-rabbitmq, unms-fluentd, unms-nginx. UCRM shares the UNMS stack.
• Hudu depends on: hudu-db-1 (postgres), hudu-redis-1, hudu-worker-1.
• Traccar, Gitea, Technitium, support-api, UniFi: self-contained (single container each; Traccar/Gitea embed storage).
• All containers depend on Docker daemon + host disk. Technitium depends on external DNS delegations.

2.3 Config Files / Drift

• /opt/support-api/docker-compose.yml - single service, port 6880, Python /health healthcheck.
• /opt/bookstack/docker-compose.yml - APP_URL=https://support.itpropartner.com, SMTP mail.germainebrown.com:2525, DB/Mail passwords and APP_KEY present in compose (redacted in this report).
• /opt/gitea/docker-compose.yml, /root/docker/hudu/docker-compose.yml, /root/docker/traccar/docker-compose.yml, /root/docker/dawarich/docker-compose.yml - env-style secrets inline (redacted).
• /home/unms/app/ - UNMS install (docker-compose, unms.conf, install-full.sh, update.sh).
• /root/.aws/ - credentials + backups present (redacted). /root/.docker/ - token seed files.
• Drift: orphan happy_rosalind Bookstack container (no compose project, no host port). Two UniFi images (jacobalberty running vs linuxserver pulled) suggest an in-flight migration.

2.4 Cron Jobs

Schedule	User	Command	Purpose	Failure mode
30 2 * * *	root	/root/backup.sh 2>&1 | logger -t app2-backup	Nightly backup to Wasabi S3	See 2.5

No other system or user cron jobs beyond /etc/cron.d defaults.

2.5 Backup Status (CRITICAL GAP)

/root/backup.sh (Wasabi S3 target hermes-vps-backups, s3.us-east-1.wasabisys.com) calls per-service backup scripts. Live evidence from 2026-08-13 02:30 run:

• Completed OK: Traccar (DB+config), Dawarich, Technitium DNS, RAGFlow (MySQL dump + Infinity DB + Minio objects).
• Logged "Backing up..." but NO completion line: Gitea, Hudu, UNMS, UniFi.
• Root cause: /root/backup.sh references gitea-backup.sh, hudu-backup.sh, unms-backup-sync.sh, unifi-backup-sync.sh which do not exist under /root/.hermes/scripts/. Those sections are silently skipped.
• bookstack-backup.sh, support-api-backup.sh, ragflow-backup.sh exist but Bookstack and support-api are not scheduled in cron.

Impact: Gitea, Hudu, UNMS, UniFi, Bookstack, support-api have effectively NO working backup. No restore test is documented for any app2 service.

2.6 app2 Critical-Service RTO/RPO

Service	RPO	RTO (est.)	Restore tested?
Hudu	none (backup broken)	4-8h	No
UNMS	none (backup broken, product EOL)	4-8h	No
UniFi	none (backup broken)	2-4h	No
Gitea	none (backup broken)	2-4h	No
Bookstack / support-api	none (script not scheduled)	2h	No
Traccar	24h	2-4h	No
Dawarich	24h	2h	No
Technitium DNS	24h	1h	No
RAGFlow	24h	2-4h	No

---

3. app3 (152.53.241.111) - CloudPanel Web Host

OS: Debian 13 (trixie), up 34d. 31 GiB RAM / 12 CPU. / 1TB (80G used, 9%). Runs CloudPanel + nginx + Percona MySQL 8.4 + 10 PHP-FPM versions + Docker (Hexclave Stack Auth, Buzz relay) + several systemd app services.

3.1 Systemd Services

Service	Version	Purpose	Notes
nginx	1.30.4	Primary web server	80/443
clp-nginx	-	CloudPanel control panel	8443
clp-agent / clp-php-fpm	-	CloudPanel agent	-
mysql (Percona Server)	8.4.10	Shared DB for all sites	127.0.0.1:3306 (datadir /home/mysql)
php7.1-fpm ... php8.5-fpm	7.1, 7.2, 7.3, 7.4, 8.0, 8.1, 8.2, 8.3, 8.4, 8.5	Per-site PHP pools	7.1-8.0 are EOL
varnishd	7.5.0	HTTP cache	6081
memcached	-	Object cache	11211
redis	-	Object cache	6379
proftpd	-	FTP server	21 (plaintext FTP)
postfix	-	Outbound mail	25
gitea.modelortho.service	Gitea	Git for git.modelortho.com	3001
msp-forms.service	FastAPI	Shared form handler (forms.itpropartner.com)	-
docs-auth-validator.service	-	Stack Auth JWT validator for docs.itpropartner.com	-
backup-restore.service	-	Backup-restore web UI (my.itpropartner.com/backup)	-
percona-telemetry-agent	-	Percona telemetry	-
fail2ban, cron, chrony, containerd, docker	-	platform	-

3.2 Docker Services

Container	Image	Purpose	Notes
hexclave-* (server, postgres, clickhouse, cron)	stackauth/server:latest, postgres 16/17-alpine, clickhouse 25.10	Hexclave Stack Auth	No backup coverage found
buzz-prod-* (relay, postgres, redis, minio)	ghcr.io/block/buzz:main	Buzz relay (Block open-source)	Production relay; no backup coverage found

3.3 Sites Hosted (CloudPanel)

WordPress (10 wp-config.php instances across 9 users): apextrackexperience.com, boxpilotlogistics.com, debtrecoveryexperts.com (x2 - under BOTH debtreecoveryexperts and drecovery users), iamgmb.com, intelsight.io, mainwp.itpropartner.com, vigilanttac.com, voipsimplicity.com (+ www subdomain).

Static / non-WordPress: docs, forms, mockups, proposals, support, my.verdicttank.com, verdicttank.com, modelortho.com (+www), transitpin.com, my.transitpin.com, panel, my.voipsimplicity.com, timapta.org, katiewattsdesign.com, buzz.iamgmb.com, hexclave-api/dash, gmb, auth2 + auth2-api.

3.4 Cron Jobs

Schedule	User	Command	Purpose
0 3 * * *	root	/root/backup.sh | logger -t app3-backup	Nightly full backup
0 1,13 * * *	root	/opt/backup-restore/snapshot.sh	Twice-daily WP snapshots
30 4 * * *	root	/root/gitea-modelortho-backup.sh | logger -t gitea-backup	Gitea (modelortho) backup
15 3 * * *	clp	clpctl db:backup ... --retentionPeriod=7	CloudPanel DB backup
15 4 * * *	clp	/home/clp/scripts/create_backup.sh	CloudPanel backup
5-25 * * * *	clp	certbot/letsencrypt renewals, vhost import, cloudflare IPs	platform

3.5 Backup Status

/root/backup.sh (2026-08-13 03:00 run, all OK) covers: CloudPanel SQLite DB, MSP Forms, all 10 MySQL DBs, all 10 WordPress file trees, static sites, nginx/configs. Destination Wasabi S3 app3/. snapshot.sh adds twice-daily local snapshots of WP sites (30-day retention). Gitea (modelortho) has its own nightly S3 backup.

Backup gap: the Docker services (Hexclave Stack Auth, Buzz relay) and TransitPin are not referenced by any backup script. Their postgres/clickhouse/minio data has no scheduled backup.

3.6 Config / Drift

• Plaintext MySQL root password hardcoded in BOTH /root/backup.sh (root-only) and /opt/backup-restore/snapshot.sh (775 world-readable). Same password in both.
• Duplicate WordPress site (debtrecoveryexperts.com) under two different system users - ambiguous ownership.
• /home/.swap = 2GB swap file on web data volume.
• Five EOL PHP-FPM runtimes (7.1, 7.2, 7.3, 7.4, 8.0) still running.

3.7 app3 Critical-Service RTO/RPO

Service	RPO	RTO (est.)	Restore tested?
CloudPanel + all WP/static sites	12-24h (daily + 2x snapshots)	4-8h	No
MySQL (10 DBs)	24h	4h	No
Gitea (modelortho)	24h	2h	No
Hexclave Stack Auth	none (no backup)	4h	No
Buzz relay	none (no backup)	4h	No
TransitPin	none (no backup)	4h	No

---

4. app1-bu (5.161.225.131) - Warm Standby for Core

OS: Ubuntu 24.04.4, up 28d. 3.7 GiB RAM / 3 CPU / 75G disk (37% used). No Docker. Runs only SSH, tailscale, fail2ban, cron + Hermes agent (v0.18.2). This is the warm standby for the live Core Hermes box.

4.1 Services

Service	Purpose	Notes
ssh	Admin access	22
tailscaled	Private mesh access	Tailscale
fail2ban	SSH brute-force protection	-
cron	Schedules watchdog + sync	-
hermes (binary)	Hermes Agent v0.18.2 installed	NOT currently running (dormant)

4.2 Cron Jobs (the failover mechanism)

Schedule	Command	Purpose	Last run
*/5 * * * *	/root/.hermes/scripts/hermes-standby-watchdog.sh	Ping live Core; failover if down ~3.5 min	Active (log entries hourly through 2026-08-13)
*/10 * * * *	/root/.hermes/scripts/hermes-standby-sync.sh	S3 sync of config/skills/plugins/cron/references	Active (last sync 2026-08-13T14:10Z)

Failover behavior (verified, NOT triggered): the watchdog pings LIVE_HOST=152.53.192.33; if it fails 4 consecutive 60s cycles, it sends Telegram+email alerts, runs aws s3 sync s3://hermes-vps-backups/live/ -> ~/.hermes/, then hermes gateway start. A standalone "failover" cron does not exist; failover is embedded in the watchdog script (runs every 5 min). The sync script correctly skips sync when the live host is unreachable.

4.3 Standby Readiness Assessment (NOT fully ready)

• Config-level state is fresh: config.yaml (2026-08-12), .env (2026-08-12), skills/ (2026-08-12), cron output (2026-08-13). Synced every 10 min from S3 live/.
• Data-level state is STALE: state.db (2.1GB, Jul 15), memory_store.db (Jul 15), sessions/ (Jul 15), memories/ (Jul 15). The sync script deliberately excludes these (DIRS="skills profiles plugins cron references", files config.yaml .env .bashrc only).
• Consequence: on failover, Hermes would start with ~4-week-old session, memory, and state DB. This is a config-level warm standby, NOT a true data-level warm standby.

4.4 Config / Drift (secrets)

• /root/.hermes/.env (mode 644, world-readable) holds ~20 plaintext secrets: Cloudflare API token, Netcup API key + customer password, SyncroMSP token, Telegram bot token, root passwords for app1/app2/app3 (SERVER_152_53_*.PASS), RingLogix creds, and API keys for OpenAI/Perplexity/Groq/xAI/Mistral/Fireworks/Google/Cohere.
• /root/.hermes/migration-creds.txt (644), /root/.hermes/scripts/.hetzner_token (644), /root/.hermes/scripts/.netcup_api_key (644) - further plaintext credentials.
• hermes-standby-watchdog.sh (mode 755, world-readable) contains a plaintext email password and Telegram bot token inline.
• /root/.hermes/state.db.corrupted (1.8GB, Jul 9) and state.db-wal lingering.

---

5. wphost02 (5.161.62.38) - Legacy WordPress / RunCloud (STILL LIVE)

OS: Ubuntu 24.04.4, up 34d. 3.7 GiB RAM / 3 CPU / 75G disk (82% full - 59G used). RunCloud-managed LEMP stack. NOT decommissioned.

5.1 Services

Service	Version	Purpose	Ports
nginx-rc	-	RunCloud web server	80/443
apache2-rc	-	RunCloud apache (secondary)	127.0.0.1:81
mariadb	11.1.6	Shared DB	127.0.0.1:3306
php81rc-fpm ... php85rc-fpm	8.1-8.5	Per-site PHP pools	-
runcloud-agent	-	RunCloud remote mgmt agent	*:34210 (rcsa service)
node_exporter	-	Prometheus exporter	0.0.0.0:9100
postfix	-	Mail	25
fail2ban, firewalld, supervisor	-	platform	-

firewalld active zone runcloud allows only 22, 80, 443 + rcsa (RunCloud agent) inbound.

5.2 WordPress Sites (8, still active)

Site	DB	Files (Aug activity)
apextrackexperience	apextrackexperience_1781549652	active (files modified Aug)
boxpilotlogistics	boxpilotlogistics_1770339547	active
debtrecoveryexperts	debtrecoveryexperts_1778934554	active
iAmGMB	iAmGMB_1764020288	no changes since Jul 1
katiewattsdesign	katiewattsdesign_1735425014	active
MainWP	mainWP_1717713767	active
vigilanttac	vigilanttac_1728911691	active
voipsimplicity	voipsimplicity_1732250845	active (13k files modified)

All eight sites ALSO exist on app3 (CloudPanel). Both hosts appear live and actively modified -> split-brain migration state.

5.3 Cron Jobs

Schedule	Command	Purpose
*/5 * * * *	/root/apex-mail-watchdog-daemon.sh	mail watchdog
30 2 * * *	/root/db-dump.sh | logger -t db-dump	nightly DB dump

5.4 Backup Status (CRITICAL GAP)

• /root/db-dump.sh (scheduled daily 02:30) dumps only 2 of 8 DBs (apextrackexperience, boxpilotlogistics) to local /root/db-backups/ with 7-day retention. No offsite copy.
• /root/backup.sh (full sites + all DBs + RunCloud config -> Wasabi S3) exists but is NOT scheduled in any crontab (grep across /etc/cron* and /var/spool/cron returned nothing).
• Net effect: 6 of 8 WordPress DBs and ALL site file trees have no running backup. 2 of 8 DBs have local-only 7-day backups. No restore test documented.

5.5 wphost02 Critical-Service RTO/RPO

Service	RPO	RTO (est.)	Restore tested?
apextrackexperience / boxpilotlogistics DBs	24h (local only, 7d retention)	4-8h	No
Other 6 WP DBs + all site files	none (no scheduled backup)	4-8h	No

---

6. Severity-Rated Findings

CRITICAL

• C1 - app2: Four production services have silently failing backups. /root/backup.sh references gitea-backup.sh, hudu-backup.sh, unms-backup-sync.sh, unifi-backup-sync.sh which do not exist. The 2026-08-13 02:30 log shows "Backing up..." for these with no completion. Gitea, Hudu, UNMS, and UniFi have no effective backup; a loss would be unrecoverable. (Evidence: journalctl app2-backup + missing files under /root/.hermes/scripts/.)
• C2 - app1-bu: World-readable secrets file. /root/.hermes/.env (mode 644) contains ~20 plaintext credentials including root passwords for app1/app2/app3, Telegram bot token, Cloudflare/Netcup/SyncroMSP tokens, and eight AI-provider API keys. Any local user or compromised process can read the entire secret estate. (Evidence: ls -la + key names.)
• C3 - app3: MySQL root password hardcoded in plaintext in backup scripts. /opt/backup-restore/snapshot.sh is mode 775 (world-readable) and contains MYSQL_PASS='[REDACTED]'; the same password is in /root/backup.sh. Credential leakage plus shared superuser credential across all app3 databases. (Evidence: script contents.)
• C4 - wphost02: Effective backup coverage is ~25% and local-only. Scheduled db-dump.sh backs up 2 of 8 DBs to local disk (7-day retention); the full offsite S3 script /root/backup.sh is not in cron. 6 WordPress DBs and all site files have no running backup on a host that is still live. (Evidence: crontab + db-dump.sh + grep backup.sh /etc/cron* /var/spool/cron empty.)
• C5 - app1-bu: Warm standby is not data-ready. Sync covers only config/skills/plugins/cron/references; state.db, memory_store.db, and sessions/ are stale at Jul 15 (~4 weeks). A failover today would restore Hermes without the last month of session, memory, and state. (Evidence: stat mtimes + sync script DIRS/excludes.)

HIGH

• H1 - app1-bu: Failover watchdog targets the wrong IP. Watchdog pings 152.53.192.33 (labeled "Core/App1 netcup"), but .env and the audit brief reference Core/app1 at 152.53.36.131. If the live Core moves/differs, the standby will fail over to nothing or never detect an outage. (Evidence: watchdog script LIVE_HOST vs .env SERVER_152_53_36_131_PASS.)
• H2 - app1-bu: Plaintext email password + Telegram token in watchdog script (mode 755, world-readable). (Evidence: script contents.)
• H3 - app2: Very old images in production. rabbitmq:3.7.28-alpine (5 years, EOL with known CVEs), ubnt/ucrm-conntrack:latest (5 years), mysql:8.0.39 (2 years), postgres:16.2 (2 years), jacobalberty/unifi:latest (8 months). (Evidence: docker images.)
• H4 - app2: UNMS is end-of-life software. Ubiquiti discontinued UNMS in 2021 (successor UISP). The entire ubnt/unms stack (13+ containers) is EOL and should be migrated. (Evidence: image set + product lifecycle.)
• H5 - wphost02: Legacy host not decommissioned; split-brain with app3. All 8 WordPress sites exist and are actively modified on BOTH wphost02 (RunCloud) and app3 (CloudPanel). No cutover or redirection evident; ambiguous source of truth for client content. (Evidence: identical site lists + Aug file mtimes on wphost02.)
• H6 - app3: Dockerized production services have no backup. Hexclave Stack Auth, Buzz relay, and TransitPin (postgres/clickhouse/minio data) are absent from every backup script. (Evidence: grep -ril 'buzz|hexclave|transitpin' /root/*.sh /opt/backup-restore/ -> only DB content hits.)
• H7 - app3: Five EOL PHP-FPM runtimes running. PHP 7.1, 7.2, 7.3, 7.4, 8.0 are all end-of-life and unpatched. (Evidence: systemctl list-units php*-fpm.)

MEDIUM

• M1 - app2: Orphaned container happy_rosalind (2nd Bookstack, no compose project, no host port binding) running with no apparent purpose.
• M2 - app2: Large public attack surface. 40 containers with many 0.0.0.0-bound ports: 53 (DNS), 5000-5150 (Traccar device range), UniFi 8080/8443/8843/8880, UNMS 81/8089/8444, netflow 2055, infinity 23817/23820, support-api 6880, bookstack 6875.
• M3 - app2: Bookstack and support-api backup scripts exist but are not scheduled. Effective RPO = none despite a written script.
• M4 - app3: Duplicate WordPress install debtrecoveryexperts.com under two users (debtreecoveryexperts and drecovery).
• M5 - app3: Single-host SPOF. ~25 client sites + shared MySQL + CloudPanel all on one VPS with no HA.
• M6 - wphost02: node_exporter bound to 0.0.0.0:9100 and postfix on :25 on a legacy host.

LOW

• L1 - wphost02: Disk 82% full (59G/75G).
• L2 - app1-bu: state.db.corrupted (1.8GB) and stale state DBs linger consuming disk.
• L3 - app3: 2GB swap file /home/.swap on the web data volume with 429MiB in use.
• L4 - app3/wphost02: RunCloud agent (rcsa) port exposed on a host slated for decommission - remote management surface retained.

---

7. Cross-Host Observations

• No restore test is documented for any service on any host. Backups are write-only everywhere a backup exists.
• Backup destinations are Wasabi S3 (hermes-vps-backups) for app2, app3, and app1-bu sync; wphost02's working backup is local-only.
• Single points of failure are pervasive: each application is on a single VPS; only Core (Hermes) has a standby, and that standby is config-level only.
• Credentials are routinely stored in plaintext world-readable files on app1-bu and app3 (and inline in compose files on app2).