itpp-infrastructure/audit/phase-one/findings/sys-a.md

Sys-A Phase One Inventory & Findings: Core + app1

Auditor: Sys-A (sysadmin inventory auditor) Scope: Core (this host, localhost) and app1 (152.53.36.131) Date: 2026-08-13 Mode: READ-ONLY. No mutation performed. All remediations are Phase Two items.

---

1. Host Profiles

Attribute	Core (localhost)	app1 (152.53.36.131)
Provider / model	Netcup RS 2000 G12	Netcup RS 4000 G12
OS	Debian 13 (trixie)	Debian 13 (trixie)
Kernel	6.12.94+deb13-amd64	6.12.95
vCPU	8	12
RAM	15 GB, NO swap	31 GB, NO swap
RAM in use	~12 GB used / 467 MB free (2.8 GB available)	(healthy)
Disk	503 GB, 73 GB used (15%)	1007 GB, 107 GB used (11%)
Uptime	not recorded	not recorded
Public IP	152.53.192.33	152.53.36.131
Docker	present	present
Firewall	ufw active (default policy not confirmed)	ufw active

Total services inventoried: Core ~55 systemd units (running) + 10 Docker containers; app1 19 systemd units (running) + 23 Docker containers. Counts below are of running/active workloads only.

---

2. Core Service Inventory (localhost)

2.1 Critical / Tier-0 services

Service	Software + Version	Port	Runs as	Restart	Backup	Notes
Hermes gateway	hermes-agent (python)	8642, 8787 (socat), 9119 (dashboard)	root	user unit enabled	hermes-full-backup daily 01:00 + live-sync	runs under systemd USER manager (user@0), not a system unit; 3.7 GB RSS, 182 tasks
Caddy	caddy (deb)	80/443 on 152.53.192.33, 2019 admin 127.0.0.1	caddy	systemd enabled	Caddyfile in hermes-full-backup + system-config-sync	single ingress for Core
Grafana	grafana/grafana:11.4.0	3002 (0.0.0.0)	root (container)	always	grafana db in core-services-backup 01:30	image 11.4.0 is ~20 months old
Prometheus	prom/prometheus:latest	9090 (0.0.0.0)	root (container)	always	prometheus_data volume in core-services-backup	config /root/docker/monitoring/prometheus/prometheus.yml
Super Search MCP	/root/docker/super-search (python)	8899 (0.0.0.0)	root	systemd	NOT covered by any backup script (see F-18)	depends on searxng + exa/firecrawl/open-corporates APIs
PostgreSQL	postgres (deb)	5432 (127.0.0.1)	postgres	systemd	no dedicated dump; only hotnow-app DB implied	single instance, no replica
Redis	redis (deb)	6379 (127.0.0.1)	redis	systemd	none found	single instance

2.2 Hermes ecosystem systemd units (all as root unless noted)

Unit	Purpose	Port	Notes
hermes-assistant	Hermes Assistant PWA backend	(8080-ish)	root
hermes-browser	Headless Chromium CDP	9222 (127.0.0.1)	runs chrome with --no-sandbox as root
hermes-control-deck	Control Deck backend API	8200	port collides with pipeline-api (F-10)
hermes-socat-8787	port forward 8787 -> localhost:8642	8787 (0.0.0.0)	After=hermes-gateway.service references non-existent SYSTEM unit (gateway is a user unit)
hermes-voice	Hermes Voice (SvelteKit)	4331 (127.0.0.1)	runs as non-root (best practice)
hermes-gateway (user)	gateway run	8642	root, user manager
hermes-gateway-anita (user)	Anita messaging gateway	-	root, user manager

2.3 Application / MCP / API systemd units (all as root)

Unit	Purpose	Port	Hardcoded secret?
auth-api	auth backend	8500	no (uses /root/projects/auth/.env)
ops-portal	operations portal	8090	-
osint-api	OSINT API	8100	-
osint-person	OSINT person MCP	8902	-
diglocate-api	diglocate	8000	-
intelsight-api	intelsight	8099	-
hotnow-api	HotNow backend	8001	- (Postgres + Redis)
shopping-cart	shopping cart	8101 (127.0.0.1)	-
seemytrip	SeeMyTrip backend	8113	ADMIN_AI_KEY hardcoded
rally	rally backend	8105 (0.0.0.0)	JWT_SECRET, DEEPSEEK_API_KEY, ADMIN_AI_KEY hardcoded
shark-game	shark-game backend	8083 (0.0.0.0)	-
pipeline-api	pipeline API	8200	collides with hermes-control-deck
verdicttank-api	verdicttank	8201	-
verdicttank-worker	worker	-	-
status-page / node	status page	8210	node
pry	PRY API	8905	-
dre-mcp	DRE MCP	8900/8901	-
ft360-mcp	FleetTracker360 MCP	8903	-
twilio-mcp	Twilio MCP	8910/8911	-
crawl4ai	crawler	-	-
voice-agent	voice agent	9101	-
voice-agent-stt	voice STT	9000	-
gitea-runner	Gitea actions runner	-	-
host-metrics-export	metrics textfile	-	-
mysql-tunnel	SSH tunnel to wphost02 MySQL	33060 (127.0.0.1)	StrictHostKeyChecking=no, UserKnownHostsFile=/dev/null
hear-read	audio TTS/read	8240 (127.0.0.1)	-

2.4 Core Docker containers

Container	Image:Tag	Port	Image age	Restart	Purpose
browserless	browserless/chrome:latest	3000 (0.0.0.0)	2 years	always	headless chrome API
uptime-kuma	louislam/uptime-kuma:1	3001 (0.0.0.0)	current	always	status monitoring
grafana	grafana/grafana:11.4.0	3002	20 months	always	dashboards
prometheus	prom/prometheus:latest	9090	6 weeks	always	metrics
searxng	searxng/searxng:latest	8888 (127.0.0.1)	6 weeks	always	meta search (Super Search dep)
timetrex	timetrex	8085 (127.0.0.1)	-	always	time tracking
microbin	microbin	8260 (127.0.0.1)	-	always	paste bin
camofox	camofox	9377 (0.0.0.0)	-	always	stealth browsing
mikrotik-exporter	mikrotik-exporter	9436 (127.0.0.1)	-	always	router metrics
node_exporter (host)	prom/node-exporter	9100 (0.0.0.0)	-	systemd	node metrics
telegraf (host)	telegraf	9273 (0.0.0.0)	-	systemd	metrics

2.5 Ad-hoc / unmanaged processes (Core)

Process	Port	CWD	Concern
python3 -m http.server 8080	8080 (0.0.0.0)	/var/www/mockup/anita-consulting	ad-hoc web server as root, no systemd
python3 -m http.server 8934	8934 (0.0.0.0)	/var/www/mockup/itpropartner	ad-hoc web server as root
python3 -m http.server 9876	9876 (0.0.0.0)	/tmp	serving /tmp as root, public bind
hermes dashboard	9119 (0.0.0.0)	-	part of Hermes

---

3. app1 Service Inventory (152.53.36.131)

3.1 Critical / Tier-0 services

Service	Software + Version	Port	Runs as	Restart	Backup	Notes
LiteLLM / admin-ai	ghcr.io/berriai/litellm:v1.92.0	4000	root (container)	always	DB NOT backed up (F-1); config YAML daily	master key + admin_ai provider
Caddy	caddy (systemd)	80/443	caddy	enabled	Caddyfile NOT backed up (F-12)	single ingress for all app1
Open WebUI	ghcr.io/open-webui/open-webui:latest	3000 (0.0.0.0)	root (container)	always	daily 1.47 GB	depends on LiteLLM
Wazuh manager/indexer/dashboard	wazuh 4.9.2 (indexer, 21 months old)	1514/1515, 9200, 5601	root	always	daily	SIEM/XDR
n8n	n8n	5678	root (container)	always	daily	automation
Vaultwarden	vaultwarden latest	8081	root (container)	always	daily	password manager
Twenty CRM	twentycrm	3003	root (container)	always	daily	CRM
Komodo	komodo-core	9120 (0.0.0.0)	root (container)	always	daily	infra automation
Browserless	browserless/chrome:latest	3005 (0.0.0.0), 3006 (ufw-limited)	root (container)	always	-	2 year old image
super-search (host)	/root/docker/super-search	8899	root	systemd	.env only (373 bytes)	depends on searxng (Core)
giftaroast	/root/giftaroast	8100	root	systemd	-	runaway fix_dict.py (F-8)

3.2 app1 Docker containers (full list)

LiteLLM v1.92.0 + litellm_postgres, mcp-browser / mcp-email / mcp-git / mcp-filesystem / mcp-super-search, litellm-super-search, open-webui, n8n + n8n-postgres, twenty-server + twenty-db + twenty-redis, komodo-core + komodo-mongo, vaultwarden, wazuh single-node (manager+indexer+dashboard), browserless, docuseal.

Key stale/very-old image tags on app1: browserless/chrome:latest (2 years), wazuh/wazuh-indexer:4.9.2 (21 months), vaultwarden/server:1.33.2 (18 months, old tag present alongside latest), ollama/ollama (5 weeks, not running), mattermost (not running).

3.3 app1 systemd units

super-search (8899), giftaroast (8100), caddy, sshd, docker, containerd, fail2ban, cron, rsyslog, ufw, unattended-upgrades, qemu-guest-agent, chrony, plus base system units.

3.4 Runaway / orphaned processes (app1)

PID	Process	CWD	Elapsed	CPU
3656093	python3 fix_dict.py	/root/giftaroast	14d 17h	99.4%
3657668	python3 fix_dict.py	/root/giftaroast	14d 17h	99.4%
3658103	python3 fix_dict.py	/root/giftaroast	14d 17h	99.4%

Parent of PID 3656093 is bash -c "cd /root/giftaroast && python3 fix_dict.py 2>&1 echo '=== Restart service ===' systemctl dae...", an aborted/interrupted manual deployment. Three cores permanently pegged for 14+ days.

---

4. Dependency Map

4.1 Core (dot)

digraph Core {
 // Ingress
 caddy -> ops_portal; caddy -> osint_api; caddy -> diglocate; caddy -> intelsight;
 caddy -> hotnow_api; caddy -> shopping_cart; caddy -> seemytrip; caddy -> rally;
 caddy -> shark_game; caddy -> pipeline_api; caddy -> verdicttank_api; caddy -> auth_api;
 caddy -> hermes_voice; caddy -> voice_agent; caddy -> pry; caddy -> uptime_kuma;
 caddy -> microbin; caddy -> timetrex; caddy -> status_page;

 // Data layer
 hotnow_api -> postgres; hotnow_api -> redis;
 osint_api -> redis;

 // Hermes gateway fan-out
 hermes_gateway -> super_search; hermes_gateway -> dre_mcp; hermes_gateway -> ft360_mcp;
 hermes_gateway -> osint_person; hermes_gateway -> twilio_mcp; hermes_gateway -> clearfront_mcp;
 hermes_gateway -> browserless; hermes_gateway -> camofox; hermes_gateway -> chrome_cdp;

 // Search chain
 super_search -> searxng; super_search -> "exa/firecrawl/open-corporates APIs";

 // Voice chain
 voice_agent -> voice_agent_stt;

 // Observability
 grafana -> prometheus;
 prometheus -> node_exporter; prometheus -> telegraf; prometheus -> mikrotik_exporter; prometheus -> snmp;

 // Cross-host tunnel
 mysql_tunnel -> "wphost02 MySQL (5.161.62.38, decommissioned 2026-08-28)";
}

4.2 app1 (dot)

digraph app1 {
 caddy -> litellm; caddy -> open_webui; caddy -> n8n; caddy -> vaultwarden;
 caddy -> docuseal; caddy -> twenty; caddy -> komodo; caddy -> wazuh_dashboard;
 caddy -> giftaroast; caddy -> browserless;

 litellm -> litellm_postgres;
 litellm -> mcp_browser; litellm -> mcp_email; litellm -> mcp_git; litellm -> mcp_filesystem;
 litellm -> mcp_super_search;
 open_webui -> litellm;

 n8n -> n8n_postgres;
 twenty_server -> twenty_db; twenty_server -> twenty_redis;
 komodo_core -> komodo_mongo;
 wazuh_manager -> wazuh_indexer; wazuh_dashboard -> wazuh_indexer;

 super_search_host -> "searxng on Core (152.53.192.33:8888)";
 mcp_super_search -> super_search_host;
}

4.3 Dependency map highlights

• Single ingress per host: Caddy is the sole HTTP(S) entry point. A Caddy misconfiguration or crash takes down every web service on that host (single point of failure).
• LiteLLM is the AI backbone: Open WebUI, all mcp-* tool servers, and Super Search (via MCP) route through it. admin-ai.itpropartner.com is the model gateway for every AI consumer.
• Hermes gateway has a wide fan-out (8+ MCP/tool servers, browserless, camofox, Chrome CDP). Its failure cascades into all agent tooling.
• Cross-host dependency: app1 super-search depends on Core searxng (port 8888) over the WAN. If Core is down, app1 search is degraded.
• mysql-tunnel (Core) depends on wphost02 (5.161.62.38, decommissioned 2026-08-28) reachability; a host key change breaks it silently.
• HotNow is the only app with a real DB dependency chain (Postgres + Redis) on Core.

---

5. Cron Jobs

5.1 Core crontab (root)

Schedule	Job	User	Purpose	Failure mode if silent
01:00 daily	hermes-full-backup.sh	root	full Hermes home + Caddyfile + systemd units + state.db to Wasabi	no DR copy of Hermes state; RPO grows unbounded
*/15 min	hermes-live-sync.sh	root	aws s3 sync of .hermes to live/ prefix	incremental state lost (mitigated by daily full)
01:30 daily	core-services-backup.sh	root	Grafana/Prometheus/etc volumes	monitoring history loss
02:00 daily	root-essentials-backup.sh	root	.hermes/.ssh/.aws/himalaya/shark-game/projects	credential/config loss
03:00 daily	docker-volume-sync.sh	root	script does not exist (F-13)	always fails silently
00:05 daily	system-config-sync.sh	root	system configs to Wasabi	config drift backups lost
15:00 daily	status-page-refresh.sh	root	status page data	stale status
various	wphost02-backup.sh	root	SSHes to 5.161.62.38 (wphost02, decommissioned 2026-08-28) with root key	wphost02 backups lost
various	docuseal/timetrex/gitea/hudu/dawarich/ragflow/twenty/stack-auth/hexclave/komodo/litellm/technitium/unifi/unms/vaultwarden-backup.sh	root	per-service backups (many target remote hosts)	per-service backup loss

Hermes-managed jobs live in /root/.hermes/cron/jobs.json (jobs.json + per-job dirs). Parsing jobs.json programmatically failed during the audit (malformed JSON); a grep fallback confirmed the job list above. The exact schedule set should be re-verified by the DR owner.

5.2 app1 crontab (root)

Schedule	Job	Purpose	Failure mode
02:00 daily	/root/backup.sh	litellm config, n8n, openwebui, mcp .env, ollama, mattermost, wazuh, twenty	see F-1 (litellm DB), F-12 (Caddyfile)

app1 /root/backup.sh references mattermost-backup.sh and wazuh-backup.sh in /root/.hermes/scripts/. Mattermost is not running (no container), so that backup is a no-op. Ollama is not installed, so its backup step fails each night.

5.3 /etc/cron.* (both hosts)

Standard Debian logrotate/man-db/dpkg jobs. No custom third-party cron drops observed beyond the system crontabs above. No evidence of a rotating backup retention job; S3 listings show daily objects accumulating without obvious lifecycle policy (Phase Two item).

---

6. Backup Status Matrix

Service	Backup script	Last S3 object	Restore test documented?
Hermes (full)	hermes-full-backup.sh	2026-08-13 (daily tar.gz ~1.43 GB)	restore.sh generated; DR-PLAN.md present (600)
Hermes (live)	hermes-live-sync.sh -> live/ prefix	last state.db live/ unclear; live-sync/ (old prefix) stale since 2026-07-05	partial
LiteLLM config	app1 /root/backup.sh	2026-08-13 (litellm-config-*.yaml, 333 B)	no
LiteLLM Postgres DB	(should be pg_dump)	ZERO .sql.gz objects ever (F-1)	no
Grafana/Prometheus	core-services-backup.sh	2026-08-13 (grafana 55 KB, prometheus 51 MB)	no
Open WebUI	app1 /root/backup.sh	2026-08-13 (1.47 GB)	no
n8n / Twenty / Vaultwarden / Wazuh	app1 /root/backup.sh	2026-08-13	no
Core Caddyfile	hermes-full-backup + system-config-sync	2026-08-13	via restore.sh
app1 Caddyfile	none	never (F-12)	no
Super Search (both hosts)	none meaningful	.env only (373 B)	no
Postgres / Redis (Core)	none dedicated	none	no

Restore-test documentation: DR-PLAN.md and migration-recovery.md exist on Core (root-only, mode 600) but no evidence of an actual periodic restore drill being executed (no drill logs found in backup dirs).

---

7. Severity-Rated Findings

CRITICAL

F-1. LiteLLM Postgres database is not being backed up (data-loss risk). Evidence: aws s3 ls s3://hermes-vps-backups/app1/litellm/ shows only litellm-config-*.yaml objects (240-333 bytes). Zero .sql/.sql.gz objects since inception. app1 /root/backup.sh dumps pg_dump -U litellm litellm, but the live config (config.yaml) sets database_url to database litellm_db, so the dump targets a nonexistent database and fails silently every night. Rationale: LiteLLM's Postgres holds every API key, model routing table, spend/budget records, and the admin-ai provider config. A database or volume failure means total loss of the AI gateway state. This is the single highest-impact finding.

F-2. Plaintext secrets hardcoded in world-readable systemd unit files. Evidence: /etc/systemd/system/rally.service contains JWT_SECRET, DEEPSEEK_API_KEY, and ADMIN_AI_KEY as literal Environment= values; /etc/systemd/system/seemytrip.service contains ADMIN_AI_KEY; /etc/systemd/system/giftaroast.service (app1) contains Twilio AUTH_TOKEN/SID and a ADMIN_AI_KEY. Unit files are 0644 (world-readable). Rationale: Any local user (or any service compromise) can read live production API keys for DeepSeek, the admin-ai gateway, and Twilio. Secrets belong in root-only env files (600), not unit files.

HIGH

F-3. Every custom service runs as root. Evidence: docker inspect of all Core/app1 containers shows User="" (root); all python3/node listeners show user root in ss/ps. Only hermes-voice and Caddy run non-root. Rationale: A single compromised service (e.g. a 2-year-old browserless) yields full root on the host, no privilege boundary.

F-4. Very old / unpatched image tags in active service. Evidence: browserless/chrome:latest = 2 years old on both hosts; grafana/grafana:11.4.0 = ~20 months old (Core, internet-exposed via ufw); wazuh/wazuh-indexer:4.9.2 = 21 months old (app1); vaultwarden/server:1.33.2 = 18 months old tag still present on app1. Rationale: These images predate many published CVEs and receive no updates. Browserless (headless Chrome) is a high-value attack surface and is also a Hermes tool dependency.

F-5. Hermes gateway (most critical service) supervised by root's systemd USER manager, not a system unit. Evidence: systemctl --user status hermes-gateway.service shows active (user@0), while systemctl status hermes-gateway.service is not-found. hermes-socat-8787.service declares After=hermes-gateway.service against a unit that does not exist at the system level. Rationale: The gateway is the core of ITPP automation. Its lifecycle depends on a user session staying alive (no lingering system-level restart guarantee) and its unit wiring is inconsistent (socat references a nonexistent system unit). Fragile single point of failure.

F-6. Port 8200 collision between hermes-control-deck and pipeline-api. Evidence: both units are active (running) and both configure 127.0.0.1:8200. ss -tlnp shows 8200 held by pipeline-api (PID 3787644). hermes-control-deck Main PID (1962662) is not the socket owner. Rationale: One of the two services is silently shadowed (the Control Deck API). Traffic routed by Caddy reaches whichever holds the port; the other is effectively down while appearing healthy.

F-7. Three runaway fix_dict.py processes pegging 3 cores for 14+ days on app1. Evidence: PIDs 3656093/3657668/3658103 at 99.4% CPU, elapsed 14d 17h, cwd /root/giftaroast. Parent is an aborted bash -c deployment (truncated command visible in /proc). Rationale: 25% of app1 CPU permanently wasted; indicates a bug in fix_dict.py and an interrupted deployment that was never cleaned up. Load average 3.26 on 12 cores.

F-8. app1 Caddyfile (ingress for every app1 service) is not backed up. Evidence: grep of /root/backup.sh for Caddyfile returns nothing; no app1 script references /etc/caddy. Only Core's Caddyfile is backed up (by Core scripts). Rationale: app1's entire reverse-proxy routing config (every site, TLS policy, upstream mapping) would need to be reconstructed by hand after a host failure. High blast radius, zero coverage.

MEDIUM

F-9. Ad-hoc python -m http.server running as root on public interfaces, one serving /tmp. Evidence: python3 -m http.server 8080 (cwd /var/www/mockup/anita-consulting), ... 8934 (/var/www/mockup/itpropartner), ... 9876 (/tmp), all bound 0.0.0.0, none under systemd. Rationale: Undocumented, unsupervised web servers run as root; serving /tmp is a direct path to accidental data exposure. These are "shadow IT" that bypasses the standard service lifecycle.

F-10. Grafana exposed to Anywhere by firewall rule. Evidence: ufw status shows 3002/tcp ALLOW IN Anywhere (Grafana). Prometheus (9090) and node_exporter (9100) also bind 0.0.0.0. Rationale: Grafana (an old 11.4.0 with auth) and raw metrics endpoints are reachable from the public internet, not just the Tailscale/management network.

F-11. World-readable credential files. Evidence: /root/projects/auth/.env = 0644, /etc/caddy/dre-passwd = 0644 (basic-auth password), /root/anita-key.json = 0644 (LiteLLM key alias), on app1. Rationale: Live credentials readable by any local account, compounding F-3 (everything is root anyway, but defense-in-depth is absent).

F-12. mysql-tunnel disables host key verification. Evidence: unit uses -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null to wphost02. Rationale: The MySQL tunnel to wphost02 is susceptible to MITM/host-spoofing; a changed host would be silently accepted.

F-13. Dead cron job: docker-volume-sync.sh does not exist. Evidence: crontab references /root/.hermes/scripts/docker-volume-sync.sh at 03:00 daily; the file does not exist (ls fails). Docker volume sync was reportedly moved to hermes-docker-sync.sh (different bucket), but the old crontab entry was never removed. Rationale: A scheduled job that can never succeed, silently. Indicates cron hygiene drift.

F-14. Pending security updates on both hosts (including kernel). Evidence: Core has 14 upgradable packages including linux-image-amd64 6.12.94 -> 6.12.101 (security) and chromium security update; app1 has 14 including docker-ce 29.6.1 -> 29.7.2. Rationale: Kernel security updates pending on both hosts. Docker engine on app1 is a minor version behind. (Both run unattended-upgrades, but security packages remain pending.)

F-15. No swap on either host. Evidence: free -h shows Swap 0 on both. Core sits at ~12 GB/15 GB used with 467 MB free. Rationale: Under memory pressure the OOM killer will terminate arbitrary services (likely the biggest consumer: the 3.7 GB Hermes gateway). No graceful pressure relief.

F-16. Dead reverse-proxy route: noc.itpropartner.com. Evidence: Caddyfile maps noc -> 127.0.0.1:8065, but no mattermost container is running on app1; mattermost-backup.sh runs daily as a no-op. Rationale: A configured, monitored-adjacent route points at a service that is not running. Drift between routing config and reality.

F-17. Super Search (designated critical) has no meaningful backup. Evidence: only a 373-byte .env snippet is uploaded; the code/config/venv under /root/docker/super-search (Core) and app1 is not covered by any backup script. Rationale: Rebuilding the search MCP requires re-cloning and re-provisioning keys by hand; RPO effectively zero for its configuration.

LOW

F-18. /root/.aws directory is group/world writable on app1 (drwxrwxr-x). Evidence: ls -ld /root/.aws = 775. Credentials file itself is 600, but the directory is too open for a secrets dir.

F-19. Leftover Docker volumes / image drift. Evidence: three Grafana volumes (grafana_data, grafana_data_final, grafana_data_v3) indicate config churn; multiple stale images (vaultwarden 1.33.2, litellm v1.84.0, ollama, mattermost) not running but present. Nine accumulated Caddyfile backup files in /etc/caddy.

F-20. Single points of failure (no HA anywhere). Evidence: one Caddy per host, one Postgres (no replica), one Redis (no replica), one Hermes gateway, one LiteLLM + one litellm_postgres, one Wazuh single-node. Rationale: No redundancy for any Tier-0 component. Acceptable for the environment size but must be documented and covered by DR runbooks.

---

8. RTO / RPO for Critical Services

Service	RPO (est.)	RTO (est.)	Basis
Hermes gateway	~24 h (daily full backup; 15-min live-sync status uncertain)	2-4 h	restore.sh + DR-PLAN.md exist; warm standby at app1-bu (5.161.225.131) referenced but out of scope/unverified
LiteLLM / admin-ai	config 24 h; database 0 (not backed up)	hours to days	F-1: DB must be reconstructed; keys re-added manually
Caddy ingress (Core)	24 h (Caddyfile)	30 min - 1 h	reinstall caddy + restore Caddyfile
Caddy ingress (app1)	none (unbacked)	1-4 h	F-8: full manual reconstruction of routing
Grafana / Prometheus	24 h	1-2 h	volume restore from core-services-backup
Super Search	none (code/config unbacked)	1-3 h	re-clone + re-provision .env keys

---

9. Top 5 Highest-Severity Findings (for parent summary)

1. F-1 (Critical): LiteLLM Postgres DB never backed up; dump targets wrong database name, so the AI gateway's keys/routing/spend are unprotected.
2. F-2 (Critical): Plaintext API keys (DeepSeek, admin-ai, Twilio) hardcoded in world-readable systemd unit files (rally, seemytrip, giftaroast).
3. F-3 (High): Every custom service runs as root, so any single compromise is full-host takeover.
4. F-4 (High): browserless/chrome (2 years old), grafana 11.4.0 (20 months), wazuh 4.9.2 (21 months) in active internet-facing service.
5. F-5 (High): Hermes gateway supervised by user manager only, with inconsistent unit wiring (socat references nonexistent system unit); fragile SPOF.

End of Sys-A Phase One findings. No remediation performed. All items deferred to Phase Two.