itpp-infrastructure/audit/phase-one/findings/sys-a.md

# Sys-A Phase One Inventory & Findings: Core + app1

Auditor: Sys-A (sysadmin inventory auditor)
Scope: Core (this host, localhost) and app1 (152.53.36.131)
Date: 2026-08-13
Mode: READ-ONLY. No mutation performed. All remediations are Phase Two items.

---

## 1. Host Profiles

| Attribute | Core (localhost) | app1 (152.53.36.131) |
|---|---|---|
| Provider / model | Netcup RS 2000 G12 | Netcup RS 4000 G12 |
| OS | Debian 13 (trixie) | Debian 13 (trixie) |
| Kernel | 6.12.94+deb13-amd64 | 6.12.95 |
| vCPU | 8 | 12 |
| RAM | 15 GB, NO swap | 31 GB, NO swap |
| RAM in use | ~12 GB used / 467 MB free (2.8 GB available) | (healthy) |
| Disk | 503 GB, 73 GB used (15%) | 1007 GB, 107 GB used (11%) |
| Uptime | not recorded | not recorded |
| Public IP | 152.53.192.33 | 152.53.36.131 |
| Docker | present | present |
| Firewall | ufw active (default policy not confirmed) | ufw active |

Total services inventoried: Core ~55 systemd units (running) + 10 Docker containers; app1 19 systemd units (running) + 23 Docker containers. Counts below are of running/active workloads only.

---

## 2. Core Service Inventory (localhost)

### 2.1 Critical / Tier-0 services

| Service | Software + Version | Port | Runs as | Restart | Backup | Notes |
|---|---|---|---|---|---|---|
| Hermes gateway | hermes-agent (python) | 8642, 8787 (socat), 9119 (dashboard) | root | user unit enabled | hermes-full-backup daily 01:00 + live-sync | runs under systemd USER manager (user@0), not a system unit; 3.7 GB RSS, 182 tasks |
| Caddy | caddy (deb) | 80/443 on 152.53.192.33, 2019 admin 127.0.0.1 | caddy | systemd enabled | Caddyfile in hermes-full-backup + system-config-sync | single ingress for Core |
| Grafana | grafana/grafana:11.4.0 | 3002 (0.0.0.0) | root (container) | always | grafana db in core-services-backup 01:30 | image 11.4.0 is ~20 months old |
| Prometheus | prom/prometheus:latest | 9090 (0.0.0.0) | root (container) | always | prometheus_data volume in core-services-backup | config /root/docker/monitoring/prometheus/prometheus.yml |
| Super Search MCP | /root/docker/super-search (python) | 8899 (0.0.0.0) | root | systemd | NOT covered by any backup script (see F-18) | depends on searxng + exa/firecrawl/open-corporates APIs |
| PostgreSQL | postgres (deb) | 5432 (127.0.0.1) | postgres | systemd | no dedicated dump; only hotnow-app DB implied | single instance, no replica |
| Redis | redis (deb) | 6379 (127.0.0.1) | redis | systemd | none found | single instance |

### 2.2 Hermes ecosystem systemd units (all as root unless noted)

| Unit | Purpose | Port | Notes |
|---|---|---|---|
| hermes-assistant | Hermes Assistant PWA backend | (8080-ish) | root |
| hermes-browser | Headless Chromium CDP | 9222 (127.0.0.1) | runs chrome with `--no-sandbox` as root |
| hermes-control-deck | Control Deck backend API | 8200 | **port collides with pipeline-api** (F-10) |
| hermes-socat-8787 | port forward 8787 -> localhost:8642 | 8787 (0.0.0.0) | `After=hermes-gateway.service` references non-existent SYSTEM unit (gateway is a user unit) |
| hermes-voice | Hermes Voice (SvelteKit) | 4331 (127.0.0.1) | runs as non-root (best practice) |
| hermes-gateway (user) | gateway run | 8642 | root, user manager |
| hermes-gateway-anita (user) | Anita messaging gateway | - | root, user manager |

### 2.3 Application / MCP / API systemd units (all as root)

| Unit | Purpose | Port | Hardcoded secret? |
|---|---|---|---|
| auth-api | auth backend | 8500 | no (uses /root/projects/auth/.env) |
| ops-portal | operations portal | 8090 | - |
| osint-api | OSINT API | 8100 | - |
| osint-person | OSINT person MCP | 8902 | - |
| diglocate-api | diglocate | 8000 | - |
| intelsight-api | intelsight | 8099 | - |
| hotnow-api | HotNow backend | 8001 | - (Postgres + Redis) |
| shopping-cart | shopping cart | 8101 (127.0.0.1) | - |
| seemytrip | SeeMyTrip backend | 8113 | **ADMIN_AI_KEY hardcoded** |
| rally | rally backend | 8105 (0.0.0.0) | **JWT_SECRET, DEEPSEEK_API_KEY, ADMIN_AI_KEY hardcoded** |
| shark-game | shark-game backend | 8083 (0.0.0.0) | - |
| pipeline-api | pipeline API | 8200 | **collides with hermes-control-deck** |
| verdicttank-api | verdicttank | 8201 | - |
| verdicttank-worker | worker | - | - |
| status-page / node | status page | 8210 | node |
| pry | PRY API | 8905 | - |
| dre-mcp | DRE MCP | 8900/8901 | - |
| ft360-mcp | FleetTracker360 MCP | 8903 | - |
| twilio-mcp | Twilio MCP | 8910/8911 | - |
| crawl4ai | crawler | - | - |
| voice-agent | voice agent | 9101 | - |
| voice-agent-stt | voice STT | 9000 | - |
| gitea-runner | Gitea actions runner | - | - |
| host-metrics-export | metrics textfile | - | - |
| mysql-tunnel | SSH tunnel to wphost02 MySQL | 33060 (127.0.0.1) | `StrictHostKeyChecking=no`, `UserKnownHostsFile=/dev/null` |
| hear-read | audio TTS/read | 8240 (127.0.0.1) | - |

### 2.4 Core Docker containers

| Container | Image:Tag | Port | Image age | Restart | Purpose |
|---|---|---|---|---|---|
| browserless | browserless/chrome:latest | 3000 (0.0.0.0) | 2 years | always | headless chrome API |
| uptime-kuma | louislam/uptime-kuma:1 | 3001 (0.0.0.0) | current | always | status monitoring |
| grafana | grafana/grafana:11.4.0 | 3002 | 20 months | always | dashboards |
| prometheus | prom/prometheus:latest | 9090 | 6 weeks | always | metrics |
| searxng | searxng/searxng:latest | 8888 (127.0.0.1) | 6 weeks | always | meta search (Super Search dep) |
| timetrex | timetrex | 8085 (127.0.0.1) | - | always | time tracking |
| microbin | microbin | 8260 (127.0.0.1) | - | always | paste bin |
| camofox | camofox | 9377 (0.0.0.0) | - | always | stealth browsing |
| mikrotik-exporter | mikrotik-exporter | 9436 (127.0.0.1) | - | always | router metrics |
| node_exporter (host) | prom/node-exporter | 9100 (0.0.0.0) | - | systemd | node metrics |
| telegraf (host) | telegraf | 9273 (0.0.0.0) | - | systemd | metrics |

### 2.5 Ad-hoc / unmanaged processes (Core)

| Process | Port | CWD | Concern |
|---|---|---|---|
| `python3 -m http.server 8080` | 8080 (0.0.0.0) | /var/www/mockup/anita-consulting | ad-hoc web server as root, no systemd |
| `python3 -m http.server 8934` | 8934 (0.0.0.0) | /var/www/mockup/itpropartner | ad-hoc web server as root |
| `python3 -m http.server 9876` | 9876 (0.0.0.0) | /tmp | **serving /tmp as root, public bind** |
| hermes dashboard | 9119 (0.0.0.0) | - | part of Hermes |

---

## 3. app1 Service Inventory (152.53.36.131)

### 3.1 Critical / Tier-0 services

| Service | Software + Version | Port | Runs as | Restart | Backup | Notes |
|---|---|---|---|---|---|---|
| LiteLLM / admin-ai | ghcr.io/berriai/litellm:v1.92.0 | 4000 | root (container) | always | **DB NOT backed up** (F-1); config YAML daily | master key + admin_ai provider |
| Caddy | caddy (systemd) | 80/443 | caddy | enabled | **Caddyfile NOT backed up** (F-12) | single ingress for all app1 |
| Open WebUI | ghcr.io/open-webui/open-webui:latest | 3000 (0.0.0.0) | root (container) | always | daily 1.47 GB | depends on LiteLLM |
| Wazuh manager/indexer/dashboard | wazuh 4.9.2 (indexer, 21 months old) | 1514/1515, 9200, 5601 | root | always | daily | SIEM/XDR |
| n8n | n8n | 5678 | root (container) | always | daily | automation |
| Vaultwarden | vaultwarden latest | 8081 | root (container) | always | daily | password manager |
| Twenty CRM | twentycrm | 3003 | root (container) | always | daily | CRM |
| Komodo | komodo-core | 9120 (0.0.0.0) | root (container) | always | daily | infra automation |
| Browserless | browserless/chrome:latest | 3005 (0.0.0.0), 3006 (ufw-limited) | root (container) | always | - | 2 year old image |
| super-search (host) | /root/docker/super-search | 8899 | root | systemd | .env only (373 bytes) | depends on searxng (Core) |
| giftaroast | /root/giftaroast | 8100 | root | systemd | - | **runaway fix_dict.py** (F-8) |

### 3.2 app1 Docker containers (full list)

LiteLLM v1.92.0 + litellm_postgres, mcp-browser / mcp-email / mcp-git / mcp-filesystem / mcp-super-search, litellm-super-search, open-webui, n8n + n8n-postgres, twenty-server + twenty-db + twenty-redis, komodo-core + komodo-mongo, vaultwarden, wazuh single-node (manager+indexer+dashboard), browserless, docuseal.

Key stale/very-old image tags on app1: browserless/chrome:latest (2 years), wazuh/wazuh-indexer:4.9.2 (21 months), vaultwarden/server:1.33.2 (18 months, old tag present alongside latest), ollama/ollama (5 weeks, not running), mattermost (not running).

### 3.3 app1 systemd units

super-search (8899), giftaroast (8100), caddy, sshd, docker, containerd, fail2ban, cron, rsyslog, ufw, unattended-upgrades, qemu-guest-agent, chrony, plus base system units.

### 3.4 Runaway / orphaned processes (app1)

| PID | Process | CWD | Elapsed | CPU |
|---|---|---|---|---|
| 3656093 | python3 fix_dict.py | /root/giftaroast | 14d 17h | 99.4% |
| 3657668 | python3 fix_dict.py | /root/giftaroast | 14d 17h | 99.4% |
| 3658103 | python3 fix_dict.py | /root/giftaroast | 14d 17h | 99.4% |

Parent of PID 3656093 is `bash -c "cd /root/giftaroast && python3 fix_dict.py 2>&1 echo '=== Restart service ===' systemctl dae..."`, an aborted/interrupted manual deployment. Three cores permanently pegged for 14+ days.

---

## 4. Dependency Map

### 4.1 Core (dot)

```dot
digraph Core {
 // Ingress
 caddy -> ops_portal; caddy -> osint_api; caddy -> diglocate; caddy -> intelsight;
 caddy -> hotnow_api; caddy -> shopping_cart; caddy -> seemytrip; caddy -> rally;
 caddy -> shark_game; caddy -> pipeline_api; caddy -> verdicttank_api; caddy -> auth_api;
 caddy -> hermes_voice; caddy -> voice_agent; caddy -> pry; caddy -> uptime_kuma;
 caddy -> microbin; caddy -> timetrex; caddy -> status_page;

 // Data layer
 hotnow_api -> postgres; hotnow_api -> redis;
 osint_api -> redis;

 // Hermes gateway fan-out
 hermes_gateway -> super_search; hermes_gateway -> dre_mcp; hermes_gateway -> ft360_mcp;
 hermes_gateway -> osint_person; hermes_gateway -> twilio_mcp; hermes_gateway -> clearfront_mcp;
 hermes_gateway -> browserless; hermes_gateway -> camofox; hermes_gateway -> chrome_cdp;

 // Search chain
 super_search -> searxng; super_search -> "exa/firecrawl/open-corporates APIs";

 // Voice chain
 voice_agent -> voice_agent_stt;

 // Observability
 grafana -> prometheus;
 prometheus -> node_exporter; prometheus -> telegraf; prometheus -> mikrotik_exporter; prometheus -> snmp;

 // Cross-host tunnel
 mysql_tunnel -> "wphost02 MySQL (152.53.62.38)";
}
```

### 4.2 app1 (dot)

```dot
digraph app1 {
 caddy -> litellm; caddy -> open_webui; caddy -> n8n; caddy -> vaultwarden;
 caddy -> docuseal; caddy -> twenty; caddy -> komodo; caddy -> wazuh_dashboard;
 caddy -> giftaroast; caddy -> browserless;

 litellm -> litellm_postgres;
 litellm -> mcp_browser; litellm -> mcp_email; litellm -> mcp_git; litellm -> mcp_filesystem;
 litellm -> mcp_super_search;
 open_webui -> litellm;

 n8n -> n8n_postgres;
 twenty_server -> twenty_db; twenty_server -> twenty_redis;
 komodo_core -> komodo_mongo;
 wazuh_manager -> wazuh_indexer; wazuh_dashboard -> wazuh_indexer;

 super_search_host -> "searxng on Core (152.53.192.33:8888)";
 mcp_super_search -> super_search_host;
}
```

### 4.3 Dependency map highlights

- Single ingress per host: Caddy is the sole HTTP(S) entry point. A Caddy misconfiguration or crash takes down every web service on that host (single point of failure).
- LiteLLM is the AI backbone: Open WebUI, all mcp-* tool servers, and Super Search (via MCP) route through it. admin-ai.itpropartner.com is the model gateway for every AI consumer.
- Hermes gateway has a wide fan-out (8+ MCP/tool servers, browserless, camofox, Chrome CDP). Its failure cascades into all agent tooling.
- Cross-host dependency: app1 super-search depends on Core searxng (port 8888) over the WAN. If Core is down, app1 search is degraded.
- mysql-tunnel (Core) depends on wphost02 (152.53.62.38) reachability; a host key change breaks it silently.
- HotNow is the only app with a real DB dependency chain (Postgres + Redis) on Core.

---

## 5. Cron Jobs

### 5.1 Core crontab (root)

| Schedule | Job | User | Purpose | Failure mode if silent |
|---|---|---|---|---|
| 01:00 daily | hermes-full-backup.sh | root | full Hermes home + Caddyfile + systemd units + state.db to Wasabi | no DR copy of Hermes state; RPO grows unbounded |
| */15 min | hermes-live-sync.sh | root | aws s3 sync of .hermes to `live/` prefix | incremental state lost (mitigated by daily full) |
| 01:30 daily | core-services-backup.sh | root | Grafana/Prometheus/etc volumes | monitoring history loss |
| 02:00 daily | root-essentials-backup.sh | root | .hermes/.ssh/.aws/himalaya/shark-game/projects | credential/config loss |
| 03:00 daily | docker-volume-sync.sh | root | **script does not exist** (F-13) | always fails silently |
| 00:05 daily | system-config-sync.sh | root | system configs to Wasabi | config drift backups lost |
| 15:00 daily | status-page-refresh.sh | root | status page data | stale status |
| various | wphost02-backup.sh | root | SSHes to 152.53.62.38 (wphost02) with root key | wphost02 backups lost |
| various | docuseal/timetrex/gitea/hudu/dawarich/ragflow/twenty/stack-auth/hexclave/komodo/litellm/technitium/unifi/unms/vaultwarden-backup.sh | root | per-service backups (many target remote hosts) | per-service backup loss |

Hermes-managed jobs live in /root/.hermes/cron/jobs.json (jobs.json + per-job dirs). Parsing jobs.json programmatically failed during the audit (malformed JSON); a grep fallback confirmed the job list above. The exact schedule set should be re-verified by the DR owner.

### 5.2 app1 crontab (root)

| Schedule | Job | Purpose | Failure mode |
|---|---|---|---|
| 02:00 daily | /root/backup.sh | litellm config, n8n, openwebui, mcp .env, ollama, mattermost, wazuh, twenty | see F-1 (litellm DB), F-12 (Caddyfile) |

app1 /root/backup.sh references `mattermost-backup.sh` and `wazuh-backup.sh` in /root/.hermes/scripts/. Mattermost is not running (no container), so that backup is a no-op. Ollama is not installed, so its backup step fails each night.

### 5.3 /etc/cron.* (both hosts)

Standard Debian logrotate/man-db/dpkg jobs. No custom third-party cron drops observed beyond the system crontabs above. No evidence of a rotating backup retention job; S3 listings show daily objects accumulating without obvious lifecycle policy (Phase Two item).

---

## 6. Backup Status Matrix

| Service | Backup script | Last S3 object | Restore test documented? |
|---|---|---|---|
| Hermes (full) | hermes-full-backup.sh | 2026-08-13 (daily tar.gz ~1.43 GB) | restore.sh generated; DR-PLAN.md present (600) |
| Hermes (live) | hermes-live-sync.sh -> `live/` prefix | last state.db `live/` unclear; `live-sync/` (old prefix) stale since 2026-07-05 | partial |
| LiteLLM config | app1 /root/backup.sh | 2026-08-13 (litellm-config-*.yaml, 333 B) | no |
| **LiteLLM Postgres DB** | (should be pg_dump) | **ZERO .sql.gz objects ever** (F-1) | no |
| Grafana/Prometheus | core-services-backup.sh | 2026-08-13 (grafana 55 KB, prometheus 51 MB) | no |
| Open WebUI | app1 /root/backup.sh | 2026-08-13 (1.47 GB) | no |
| n8n / Twenty / Vaultwarden / Wazuh | app1 /root/backup.sh | 2026-08-13 | no |
| Core Caddyfile | hermes-full-backup + system-config-sync | 2026-08-13 | via restore.sh |
| **app1 Caddyfile** | **none** | **never** (F-12) | no |
| Super Search (both hosts) | none meaningful | .env only (373 B) | no |
| Postgres / Redis (Core) | none dedicated | none | no |

Restore-test documentation: DR-PLAN.md and migration-recovery.md exist on Core (root-only, mode 600) but no evidence of an actual periodic restore drill being executed (no drill logs found in backup dirs).

---

## 7. Severity-Rated Findings

### CRITICAL

**F-1. LiteLLM Postgres database is not being backed up (data-loss risk).**
Evidence: `aws s3 ls s3://hermes-vps-backups/app1/litellm/` shows only `litellm-config-*.yaml` objects (240-333 bytes). Zero `.sql`/`.sql.gz` objects since inception. app1 /root/backup.sh dumps `pg_dump -U litellm litellm`, but the live config (`config.yaml`) sets database_url to database `litellm_db`, so the dump targets a nonexistent database and fails silently every night.
Rationale: LiteLLM's Postgres holds every API key, model routing table, spend/budget records, and the admin-ai provider config. A database or volume failure means total loss of the AI gateway state. This is the single highest-impact finding.

**F-2. Plaintext secrets hardcoded in world-readable systemd unit files.**
Evidence: `/etc/systemd/system/rally.service` contains `JWT_SECRET`, `DEEPSEEK_API_KEY`, and `ADMIN_AI_KEY` as literal `Environment=` values; `/etc/systemd/system/seemytrip.service` contains `ADMIN_AI_KEY`; `/etc/systemd/system/giftaroast.service` (app1) contains Twilio `AUTH_TOKEN`/`SID` and a `ADMIN_AI_KEY`. Unit files are 0644 (world-readable).
Rationale: Any local user (or any service compromise) can read live production API keys for DeepSeek, the admin-ai gateway, and Twilio. Secrets belong in root-only env files (600), not unit files.

### HIGH

**F-3. Every custom service runs as root.**
Evidence: docker inspect of all Core/app1 containers shows `User=""` (root); all `python3`/`node` listeners show `user root` in ss/ps. Only hermes-voice and Caddy run non-root.
Rationale: A single compromised service (e.g. a 2-year-old browserless) yields full root on the host, no privilege boundary.

**F-4. Very old / unpatched image tags in active service.**
Evidence: browserless/chrome:latest = 2 years old on both hosts; grafana/grafana:11.4.0 = ~20 months old (Core, internet-exposed via ufw); wazuh/wazuh-indexer:4.9.2 = 21 months old (app1); vaultwarden/server:1.33.2 = 18 months old tag still present on app1.
Rationale: These images predate many published CVEs and receive no updates. Browserless (headless Chrome) is a high-value attack surface and is also a Hermes tool dependency.

**F-5. Hermes gateway (most critical service) supervised by root's systemd USER manager, not a system unit.**
Evidence: `systemctl --user status hermes-gateway.service` shows active (user@0), while `systemctl status hermes-gateway.service` is `not-found`. `hermes-socat-8787.service` declares `After=hermes-gateway.service` against a unit that does not exist at the system level.
Rationale: The gateway is the core of ITPP automation. Its lifecycle depends on a user session staying alive (no lingering system-level restart guarantee) and its unit wiring is inconsistent (socat references a nonexistent system unit). Fragile single point of failure.

**F-6. Port 8200 collision between hermes-control-deck and pipeline-api.**
Evidence: both units are `active (running)` and both configure 127.0.0.1:8200. `ss -tlnp` shows 8200 held by pipeline-api (PID 3787644). hermes-control-deck Main PID (1962662) is not the socket owner.
Rationale: One of the two services is silently shadowed (the Control Deck API). Traffic routed by Caddy reaches whichever holds the port; the other is effectively down while appearing healthy.

**F-7. Three runaway `fix_dict.py` processes pegging 3 cores for 14+ days on app1.**
Evidence: PIDs 3656093/3657668/3658103 at 99.4% CPU, elapsed 14d 17h, cwd /root/giftaroast. Parent is an aborted `bash -c` deployment (truncated command visible in /proc).
Rationale: 25% of app1 CPU permanently wasted; indicates a bug in fix_dict.py and an interrupted deployment that was never cleaned up. Load average 3.26 on 12 cores.

**F-8. app1 Caddyfile (ingress for every app1 service) is not backed up.**
Evidence: grep of /root/backup.sh for Caddyfile returns nothing; no app1 script references /etc/caddy. Only Core's Caddyfile is backed up (by Core scripts).
Rationale: app1's entire reverse-proxy routing config (every site, TLS policy, upstream mapping) would need to be reconstructed by hand after a host failure. High blast radius, zero coverage.

### MEDIUM

**F-9. Ad-hoc `python -m http.server` running as root on public interfaces, one serving /tmp.**
Evidence: `python3 -m http.server 8080` (cwd /var/www/mockup/anita-consulting), `... 8934` (/var/www/mockup/itpropartner), `... 9876` (/tmp), all bound 0.0.0.0, none under systemd.
Rationale: Undocumented, unsupervised web servers run as root; serving /tmp is a direct path to accidental data exposure. These are "shadow IT" that bypasses the standard service lifecycle.

**F-10. Grafana exposed to Anywhere by firewall rule.**
Evidence: `ufw status` shows `3002/tcp ALLOW IN Anywhere` (Grafana). Prometheus (9090) and node_exporter (9100) also bind 0.0.0.0.
Rationale: Grafana (an old 11.4.0 with auth) and raw metrics endpoints are reachable from the public internet, not just the Tailscale/management network.

**F-11. World-readable credential files.**
Evidence: /root/projects/auth/.env = 0644, /etc/caddy/dre-passwd = 0644 (basic-auth password), /root/anita-key.json = 0644 (LiteLLM key alias), on app1.
Rationale: Live credentials readable by any local account, compounding F-3 (everything is root anyway, but defense-in-depth is absent).

**F-12. mysql-tunnel disables host key verification.**
Evidence: unit uses `-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null` to wphost02.
Rationale: The MySQL tunnel to wphost02 is susceptible to MITM/host-spoofing; a changed host would be silently accepted.

**F-13. Dead cron job: docker-volume-sync.sh does not exist.**
Evidence: crontab references `/root/.hermes/scripts/docker-volume-sync.sh` at 03:00 daily; the file does not exist (`ls` fails). Docker volume sync was reportedly moved to hermes-docker-sync.sh (different bucket), but the old crontab entry was never removed.
Rationale: A scheduled job that can never succeed, silently. Indicates cron hygiene drift.

**F-14. Pending security updates on both hosts (including kernel).**
Evidence: Core has 14 upgradable packages including linux-image-amd64 6.12.94 -> 6.12.101 (security) and chromium security update; app1 has 14 including docker-ce 29.6.1 -> 29.7.2.
Rationale: Kernel security updates pending on both hosts. Docker engine on app1 is a minor version behind. (Both run unattended-upgrades, but security packages remain pending.)

**F-15. No swap on either host.**
Evidence: `free -h` shows Swap 0 on both. Core sits at ~12 GB/15 GB used with 467 MB free.
Rationale: Under memory pressure the OOM killer will terminate arbitrary services (likely the biggest consumer: the 3.7 GB Hermes gateway). No graceful pressure relief.

**F-16. Dead reverse-proxy route: noc.itpropartner.com.**
Evidence: Caddyfile maps noc -> 127.0.0.1:8065, but no mattermost container is running on app1; mattermost-backup.sh runs daily as a no-op.
Rationale: A configured, monitored-adjacent route points at a service that is not running. Drift between routing config and reality.

**F-17. Super Search (designated critical) has no meaningful backup.**
Evidence: only a 373-byte .env snippet is uploaded; the code/config/venv under /root/docker/super-search (Core) and app1 is not covered by any backup script.
Rationale: Rebuilding the search MCP requires re-cloning and re-provisioning keys by hand; RPO effectively zero for its configuration.

### LOW

**F-18. /root/.aws directory is group/world writable on app1 (drwxrwxr-x).**
Evidence: `ls -ld /root/.aws` = 775. Credentials file itself is 600, but the directory is too open for a secrets dir.

**F-19. Leftover Docker volumes / image drift.**
Evidence: three Grafana volumes (grafana_data, grafana_data_final, grafana_data_v3) indicate config churn; multiple stale images (vaultwarden 1.33.2, litellm v1.84.0, ollama, mattermost) not running but present. Nine accumulated Caddyfile backup files in /etc/caddy.

**F-20. Single points of failure (no HA anywhere).**
Evidence: one Caddy per host, one Postgres (no replica), one Redis (no replica), one Hermes gateway, one LiteLLM + one litellm_postgres, one Wazuh single-node.
Rationale: No redundancy for any Tier-0 component. Acceptable for the environment size but must be documented and covered by DR runbooks.

---

## 8. RTO / RPO for Critical Services

| Service | RPO (est.) | RTO (est.) | Basis |
|---|---|---|---|
| Hermes gateway | ~24 h (daily full backup; 15-min live-sync status uncertain) | 2-4 h | restore.sh + DR-PLAN.md exist; warm standby at app1-bu (5.161.225.131) referenced but out of scope/unverified |
| LiteLLM / admin-ai | **config 24 h; database 0 (not backed up)** | hours to days | F-1: DB must be reconstructed; keys re-added manually |
| Caddy ingress (Core) | 24 h (Caddyfile) | 30 min - 1 h | reinstall caddy + restore Caddyfile |
| Caddy ingress (app1) | **none (unbacked)** | 1-4 h | F-8: full manual reconstruction of routing |
| Grafana / Prometheus | 24 h | 1-2 h | volume restore from core-services-backup |
| Super Search | **none (code/config unbacked)** | 1-3 h | re-clone + re-provision .env keys |

---

## 9. Top 5 Highest-Severity Findings (for parent summary)

1. **F-1 (Critical):** LiteLLM Postgres DB never backed up; dump targets wrong database name, so the AI gateway's keys/routing/spend are unprotected.
2. **F-2 (Critical):** Plaintext API keys (DeepSeek, admin-ai, Twilio) hardcoded in world-readable systemd unit files (rally, seemytrip, giftaroast).
3. **F-3 (High):** Every custom service runs as root, so any single compromise is full-host takeover.
4. **F-4 (High):** browserless/chrome (2 years old), grafana 11.4.0 (20 months), wazuh 4.9.2 (21 months) in active internet-facing service.
5. **F-5 (High):** Hermes gateway supervised by user manager only, with inconsistent unit wiring (socat references nonexistent system unit); fragile SPOF.

*End of Sys-A Phase One findings. No remediation performed. All items deferred to Phase Two.*