Files

T

root a522d11c35 docs: nest 19 files into audit/ clients/ infrastructure/ monitoring/ projects/ super-search/

2026-08-08 13:04:55 -04:00

16 KiB

Raw Blame History

ITPP Infrastructure Documentation Gap Assessment

Date: 2026-08-04 Auditor: Hermes Agent (subagent) Scope: All ITPP infrastructure — Core, app1, app2, app3, app1-bu, wphost02

Executive Summary

Total services discovered running: 65+ (across 5 hosts) Services with NO backup: 12 (CRITICAL: 3 with production data at risk) Services missing from API master list: 14 Documentation staleness issues: 7 Services with NO project documentation: 22+ Duplicate services (unintended): 2 (Twenty CRM, SearXNG running on both Core AND App1) Sites missing local snapshots (App3): 4

(A) CRITICAL GAPS — Services with NO Backup

🔴 Priority 1 — Production data at immediate risk

#	Service	Host	Risk	Data at stake
1	Ragflow (full stack)	App2	CRITICAL	MySQL DB, Minio objects, Infinity vector DB, Valkey cache — entire RAG/knowledge base platform. 6 Docker containers including mysql:8.0.39, minio, infinity vector DB
2	Mattermost	App1	CRITICAL	Team chat messages, channels, files, user accounts. postgres:16-alpine backend
3	Wazuh (SIEM)	App1	HIGH	Security events, alerts, agent data, compliance logs. 3 containers (dashboard, manager, indexer). Only the server itself is backed up via app1 general backup — Wazuh data is NOT

🟡 Priority 2 — Important services without backups

#	Service	Host	Risk	Data at stake
4	Dawarich	App2	MEDIUM	Location history data (PostGIS), Redis cache. NOT in app2-backup.sh
5	Technitium DNS	App2	MEDIUM	DNS zone configs, DHCP leases, blocklists. NOT in app2-backup.sh
6	SearXNG (App1)	App1	MEDIUM	Search engine config. Backup plan says "removed" but it's running on :8080
7	MCP containers (App1)	App1	LOW	mcp-browser, mcp-email, mcp-git, mcp-filesystem, super-search — config/state not backed up individually
8	browserless (App1)	App1	LOW	Stateless Chrome, but no restart config backup
9	Timetrex	Core	LOW	Time tracking data — Docker container, no compose file found
10	Microbin	Core	LOW	Paste bin data — Docker container, compose exists at /opt/microbin/
11	browserless (Core)	Core	LOW	Stateless Chrome, :3000 (conflicts with Grafana's documented port)
12	crawl4ai	Core	LOW	Python service :8910 — web crawling config. No compose, running from /root/docker/crawl4ai/

🟢 Services running natively (Core systemd/Python — minimal backup need)

These are stateless or backed up via hermes-backup.sh (skills/profiles/sessions) and root-essentials-backup.sh (scripts):

hotnow-api (:8001), auth-server (:8500), pipeline-server (:8200), hermes-voice (:4331), host-metrics-exporter (:9275), transitpin-mockup (:8912), http-server (:9876)

(B) DOCUMENTATION GAPS — Services not in API Master List

Missing from `/root/projects/itpp-infrastructure/api-master-list.md`

#	Service	Host	Port	Notes
1	Mattermost	App1	:8065	Team chat — not listed anywhere
2	n8n	App1	:5678	Workflow automation — not listed
3	Ragflow	App2	:9380-9384	RAG platform — not listed
4	SearXNG (App1)	App1	:8080	Listed only on Core :8888
5	Timetrex	Core	:8085	Time tracking — not listed
6	Microbin	Core	:8260	Paste bin — not listed
7	browserless (Core)	Core	:3000	Chrome automation — not listed
8	browserless (App1)	App1	:3005	Chrome automation — not listed
9	crawl4ai	Core	:8910	Web crawler — not listed
10	hermes-control-deck	Core	systemd	Control deck API — not listed
11	MCP containers (App1)	App1	:8900-8903	Litellm MCP gateway services — not listed individually
12	Super Search (App1)	App1	container	Duplicate of Core's — not listed
13	hotnow-api	Core	:8001	HotNow API (different from :8000 Diglocate)
14	auth-server	Core	:8500	Centralized auth project

(C) STALE DOCUMENTATION — Wrong/Outdated Information

🔴 API Master List errors

#	Issue	Doc says	Actual	Severity
1	Grafana port	Core :3000	Core :3002 (browserless/chrome occupies :3000)	MED — monitoring dashboards accessed at wrong port
2	DocuSeal port	App1 :3000	App1 :3002 (Open WebUI occupies :3000 on App1)	MED
3	Twenty CRM location	Core :3003 (listed as Core)	Running on BOTH Core :3003 AND App1 :3003	HIGH — duplicate service, unclear which is authoritative
4	SearXNG status	Core :8888, backup plan says "removed"	Actually running on BOTH Core :8888 AND App1 :8080	HIGH — backup plan says replaced by Super Search but still running on two hosts
5	Vaultwarden location	Core :8080 (in old API list), App1 :8081	Only on App1 :8081 (correct) but stale S3 paths remain	LOW
6	Komodo location	Migrated to App1 :9120	Correctly on App1 :9120	OK
7	DocuSeal location	Migrated to App1 :3002	Correctly on App1 :3002	OK
8	Twenty CRM migration	Backup plan says migrated to App1	Still running on Core too! The migration was partial or the Core instance was never shut down	HIGH

🔴 Backup Plan errors

#	Issue	Details
9	S3 stale paths	`core/vaultwarden/`, `core/twenty/`, `core/searxng/`, `core/komodo/` still in S3 hierarchy — services migrated but old paths not cleaned
10	Backup plan lists Core services that migrated	Table shows Vaultwarden, SearXNG, Twenty CRM, Komodo, DocuSeal, Kokoro TTS under Core section — all migrated to App1
11	app1-backup.sh (on App1) backs up MORE than documented	Script backs up n8n (not in plan), Ollama (not in plan) but does NOT back up Mattermost, Wazuh, SearXNG (App1), browserless
12	app2-backup.sh (on App2) coverage gaps	Script covers Traccar, Gitea, Hudu, UNMS, UniFi but misses Dawarich, Technitium DNS, Ragflow

🔴 App3 Snapshot Coverage

#	Site	Nginx config?	In backup-restore snapshots?	In S3 app3-backup.sh?
1	intelsight.io	✅	❌	✅ (via WordPress files backup)
2	my.voipsimplicity.com	✅	❌	✅
3	panel.itpropartner.com	✅	❌	N/A (CloudPanel itself)
4	transitpin.com	✅	❌	✅

(D) CREDENTIAL GAPS

Credential Management Assessment

Vaultwarden: Running on App1 :8081. Accessible via web UI. Not directly queryable via API without auth token.
Standard credentials (ippadmin/LoveMyBoys.1520!): Referenced in task context. Need to verify which services use these vs. unique credentials.
Key credential files: /root/.hermes/.env, ~/.aws/credentials, Vaultwarden vault
DR Issue Log references migration-creds.txt and dre-temp-passwords.txt — both now chmod 600 (DR-004, DR-005 fixed)

Recommendations:

Audit all services to confirm which use standard creds vs unique creds
Each service should have a Hudu asset documenting its credentials
Service-specific API keys (n8n, Mattermost, Ragflow internal admin) need to be inventoried

(E) SERVICES WITH NO PROJECT DOCUMENTATION

The /root/projects/itpp-infrastructure/ repo has documentation for only ~8 projects out of 30+ running services:

Have docs: ops-portal, backup-restore, hotnow, intelsight, schoolcart, tripflow, beachdirect, forefront-broadband-map, missed-call-lead-recovery

NO docs (22+ services): Mattermost, n8n, Ragflow, Wazuh, Dawarich, Technitium DNS, Timetrex, Microbin, browserless (both), crawl4ai, Vaultwarden, Komodo, DocuSeal, LiteLLM, Open WebUI, Twenty CRM, Kokoro TTS, PRY, TransitPin, Village Express, Shopping Cart, Voice Agent stack, Diglocate, Rally, Shark Game, hermes-assistant, hermes-control-deck, Camofox, Super Search, Gitea, Hudu, UNMS, UniFi

(F) DUPLICATE SERVICES

Two services are running redundantly on both Core and App1:

Service	Core	App1	Notes
Twenty CRM	:3003 (Docker, 5 containers)	:3003 (Docker, 4 containers)	Migration doc says moved to App1. Core instance was never shut down. Which is authoritative?
SearXNG	:8888 (Docker)	:8080 (Docker)	Backup plan says "removed, replaced by Super Search." Both still running.

(G) RECOMMENDED FIXES — Priority Order

🔴 Immediate (this week)

Backup Ragflow — Create ragflow-backup.sh on App2. Dump MySQL (mysql:8.0.39), backup Minio objects, backup Infinity DB. Add to cron. Risk: complete RAG platform data loss.
Backup Mattermost — Add to app1-backup.sh or create mattermost-backup.sh. Dump postgres:16-alpine DB, backup file uploads. Add to cron.
Backup Wazuh — Create wazuh-backup.sh on App1. Backup Elasticsearch indices and Wazuh manager config. Add to cron.
Shut down duplicate Twenty CRM on Core — The migration doc says it moved to App1. The Core instance (5 containers) is likely stale and consuming resources. Verify App1 instance is authoritative, then stop Core instance.
Shut down duplicate SearXNG on both hosts OR document the dual deployment — Backup plan says "removed, replaced by Super Search." If Super Search is sufficient, remove both SearXNG instances. If still needed, document why and add to backup plan.

🟡 This sprint (next 2 weeks)

Update API Master List — Add all 14 missing services. Fix stale port references (Grafana :3000→:3002, DocuSeal :3000→:3002).
Update Backup Plan — Remove stale Core entries (Vaultwarden, SearXNG, Twenty, Komodo, DocuSeal, Kokoro). Add Mattermost, n8n, Wazuh, Ragflow. Note that n8n IS backed up by app1-backup.sh but not documented.
Backup Dawarich — Add PostGIS dump to app2-backup.sh.
Backup Technitium DNS — Add DNS zone/config backup to app2-backup.sh.
Add App3 sites to local snapshots — Add intelsight.io, my.voipsimplicity.com, transitpin.com to /opt/backup-restore/snapshot.sh coverage.

🟢 Backlog (next month)

Create project docs for at minimum: Mattermost, n8n, Ragflow, Wazuh (the 4 services with no docs AND production data).
Create disaster-recovery-plan.md in itpp-infrastructure repo — it doesn't exist at the expected path. The DR issue log references it but the file is missing.
Audit S3 bucket — Clean stale paths (core/vaultwarden/, core/twenty/, core/searxng/, core/komodo/). Verify recent backups for all 26 documented targets.
Credential audit — Log into Vaultwarden, enumerate all entries, cross-reference with running services, identify gaps.
Timetrex and Microbin — Document purpose, add compose files to repo, add lightweight backup if they hold data.
Create per-service README template — Standardized format: purpose, host, ports, dependencies, backup method, restore procedure, credentials location.

Methodology

Baseline docs read: /root/projects/itpp-infrastructure/backup-plan.md, api-master-list.md, /root/.hermes/references/dr-issue-log.md
Hosts audited via SSH: Core (localhost), app1 (152.53.36.131), app2 (152.53.39.202), app3 (152.53.241.111), app1-bu (5.161.225.131), wphost02 (5.161.62.38)
Enumeration: docker ps, ss -tlnp, systemctl list-units, crontab -l, ls /etc/nginx/sites-enabled/
Cross-reference: Each running service checked against API master list, backup plan, and itpp-infrastructure project docs
Key: /root/.ssh/itpp-infra used for all remote SSH

Appendix: Complete Service Inventory

Core (152.53.192.33) — netcup RS 2000

Service	Type	Port	In API List?	Backed Up?	Has Docs?
Twenty CRM	Docker (5ctr)	:3003	✅ (stale: says Core)	⚠️ (S3: app1/twenty/, Core backup)	❌
SearXNG	Docker	:8888	✅ (stale: says removed)	⚠️ (stale S3 path)	❌
Prometheus	Docker	:9090	✅	✅ (core-services-backup.sh)	❌
Grafana	Docker	:3002	✅ (wrong port :3000)	✅	❌
Telegraf	Docker	:9273	✅	✅ (system)	❌
Uptime Kuma	Docker	:3001	✅	✅	❌
MikroTik Exporter	Docker	:9436	✅	✅ (system)	❌
Camofox Browser	Docker	:9377	✅	⚠️ (hermes-backup.sh)	❌
browserless	Docker	:3000	❌	❌	❌
Timetrex	Docker	:8085	❌	❌	❌
Microbin	Docker	:8260	❌	❌	❌
Super Search MCP	systemd	:8899	✅	✅ (hermes-backup.sh)	⚠️ (partial)
DRE MCP	systemd	:8900	✅	✅	❌
Twilio MCP	systemd	:8901	✅	✅	❌
OSINT Person MCP	systemd	:8902	✅	✅	❌
FT360 MCP	systemd	:8903	✅	✅	❌
PRY	systemd	:8905	✅	✅	❌
Ops Portal	systemd	:8090	✅	✅	✅
IntelSight API	systemd	:8099	✅	✅	✅
Diglocate API	systemd	:8000	✅	✅	❌
hotnow-api	systemd	:8001	❌	❌	⚠️ (project doc exists)
Rally	systemd	:8105	✅	✅	❌
Village Express	systemd	:8210	✅	✅	❌
Voice Agent STT	systemd	:9000	✅	✅	❌
Voice Agent	systemd	:9101	✅	✅	❌
Shopping Cart	systemd	:8101	✅	✅	❌
OSINT API	systemd	:8100	✅	✅	❌
Shark Game	systemd	:8083	✅	✅	❌
hermes-assistant	systemd	:8082	✅	✅	❌
hermes-control-deck	systemd	n/a	❌	✅	❌
hermes-voice	systemd	:4331	❌	✅	❌
auth-server	systemd	:8500	❌	❌	❌
pipeline-server	systemd	:8200	❌	❌	❌
crawl4ai	systemd	:8910	❌	❌	❌
host-metrics-exporter	systemd	:9275	❌	❌	❌
transitpin mockup	systemd	:8912	❌	❌	❌

App1 (152.53.36.131) — netcup RS 4000

Service	Type	Port	In API List?	Backed Up?	Has Docs?
Open WebUI	Docker	:3000	✅	✅ (app1-backup.sh)	❌
LiteLLM	Docker	:4000	✅	✅	❌
Komodo	Docker	:9120	✅	✅ (komodo-backup.sh)	❌
DocuSeal	Docker	:3002	✅ (port :3000)	✅ (docuseal-backup.sh)	❌
Twenty CRM	Docker (4ctr)	:3003	✅ (says Core)	✅ (twenty-backup.sh)	❌
Kokoro TTS	Docker	:8880	✅	N/A (stateless)	❌
SearXNG (App1)	Docker	:8080	❌ (only listed on Core)	❌	❌
Wazuh	Docker (3ctr)	:5601/:9200	✅	❌	❌
Vaultwarden	Docker	:8081	✅	✅ (vaultwarden-backup.sh)	❌
n8n	Docker	:5678	❌	✅ (in app1-backup.sh but not plan)	❌
Mattermost	Docker	:8065	❌	❌	❌
MCP Gateway services	Docker (5ctr)	:8900-8903	❌	❌	❌
browserless (App1)	Docker	:3005	❌	❌	❌
Super Search (App1)	Docker	n/a	❌	❌	❌

App2 (152.53.39.202) — netcup RS 4000

Service	Type	Port	In API List?	Backed Up?	Has Docs?
Hudu	Docker (4ctr)	:3000 (int)	✅	✅ (hudu-backup.sh)	❌
Gitea	Docker	:3001 (int)	✅	✅ (gitea-backup.sh)	❌
UNMS/UISP	Docker (full)	:8089	✅	✅ (unms-backup-sync.sh)	❌
UniFi	Docker	:8443	✅	✅ (unifi-backup-sync.sh)	❌
Traccar	Docker	:8082	✅	✅ (app2-backup.sh)	❌
Dawarich	Docker (4ctr)	:3002	✅	❌	❌
Technitium DNS	Docker	:5380	✅	❌	❌
Ragflow	Docker (6ctr)	:9380-9384	❌	❌	❌

App3 (152.53.241.111) — netcup RS 4000 (CloudPanel)

13 WordPress sites hosted. All backed up to S3 via app3-backup.sh (daily 3 AM). 4 of 13 sites NOT in local snapshot rotation (intelsight.io, my.voipsimplicity.com, panel.itpropartner.com, transitpin.com).

App1-BU (5.161.225.131) — Hetzner CPX21

Warm standby Hermes. No Docker. Cron: standby watchdog (every 5 min) + sync (every 10 min). Correctly configured per DR plan.

wphost02 (5.161.62.38) — Hetzner CPX21

RunCloud WordPress hosting. 2 users (ippadmin, runcloud). Backed up via SSH from Core at 5 AM daily. Verified functional (DR-018, Jul 19).

Files Created

/root/projects/itpp-infrastructure/docs/infrastructure-gap-assessment-2026-08-04.md — this report

16 KiB Raw Blame History