hermes-recovery/references/architecture-audit-2026-07-13.md

Server Architecture Plan — Independent Audit

Audited: July 13, 2026 00:21 EDT
Auditor: Sho'Nuff (Hermes Agent)
Purpose: Review approved architecture, identify gaps, present to unbiased 3rd‑party reviewer
Scope: Full architecture ...

Legend for all following sections:

Symbol	Meaning
✅	Sound / no issue found
⚠️	Concern — needs discussion or mitigation
❌	Flaw — contradicts stated goal or creates risk
—	Not applicable / excluded by directive

---

1. Guiding Principles (from User Memory / Context)

Principle	Source
Fault isolation matters — no all-in-one servers	Memory
Reliability / performance > savings	Memory
No single points of failure	Memory
Offline recovery capability required	Memory
Provider diversity for DR	Memory
Uptime / restorability > cost / convenience	Memory
Model / API spend now a real constraint	Memory
Clean direct solutions over proxy hops	Memory
Do only what is asked; no scope creep	Memory
Everything not needed for Hermes runtime should move off Core	This request
Ollama excluded from plan entirely	This request
Plan must survive independent 3rd‑party review	This request

---

2. Current Live State (observed Jul 13 00:21 EDT)

2.1 Core (152.53.192.33 — RS 2000 G12)

Resource	Value
vCPU	8 (non-dedicated)
RAM	15 GB
Disk	503 GB (16% used, ~406 GB free)
Docker	11 containers running
systemd services	33 running

Docker containers on Core:

Container	Purpose	Needed for Hermes to function?
vaultwarden	Password vault	No
komodo-core-1 + komodo-mongo-1	Docker management	No
prometheus	Metrics	No
grafana	Dashboards	No
twenty-server-1 + twenty-worker-1 + twenty-db-1 + twenty-redis-1	CRM (Twenty)	No
docuseal	Document signing	No
searxng	Meta-search engine	No (Super Search uses it but runs as separate service)

systemd services on Core relevant to architecture:

Service	Purpose	Needed for Hermes to function?
hermes.service	Hermes Agent gateway	Yes
hermes-browser.service	Headless Chromium for browser tool	Yes
hermes-assistant.service	PWA backend	Yes
ollama.service	Local LLM server	No (to be removed per this directive)
caddy.service	Reverse proxy	Partially — only for core.itpropartner.com paths; full Caddy can move
ops-portal.service	Ops Portal FastAPI backend	No
osint-api.service	OSINT API	No
osint-person.service	OSINT Person MCP	Yes (runs locally, needed for Hermes)
super-search.service	Super Search MCP	Yes (runs locally, needed for Hermes)
dre-mcp.service	DRE MCP	Yes (runs locally, needed for Hermes)
twilio-mcp.service	Twilio MCP	Yes (runs locally, needed for Hermes)
mysql-tunnel.service	MySQL SSH tunnel to Apex DB	Yes (needed for mysql MCP tool)
shark-game.service	Shark Game backend	No
node_exporter.service	Prometheus exporter	No
strongswan-starter.service	VPN	No (for remote access; not Hermes-critical)
xl2tpd.service	L2TP VPN	No

Current Hermes provider/failover chain:

Primary:   gpt-5.5             via admin-ai (https://admin-ai.itpropartner.com/v1)
Fallback 1: deepseek-v4-pro    via admin-ai
Fallback 2: deepseek/deepseek-chat  via OpenRouter (https://openrouter.ai/api/v1)
Fallback 3: llama3.2:3b        via ollama-local (http://localhost:11434/v1)

Delegation (subagent) primary: deepseek-v4-pro via admin-ai.

admin-ai.itpropartner.com currently points to: 178.156.167.181 (Hetzner old-ai box).

2.2 app1 (152.53.36.131 — RS 4000 G12)

Resource	Value
vCPU	12 dedicated EPYC 9645
RAM	32 GB
Disk	1 TB (3% used, ~943 GB free)

Running: Ollama, Open WebUI, n8n + n8n-postgres, Caddy.

2.3 app2 (152.53.39.202 — RS 4000 G12)

Running: UNMS/UISP stack, UniFi Controller (deployed Jul 12).

2.4 app3 (152.53.241.111 — RS 4000 G12)

Target for WordPress/Apex from SiteGround/wphost02.

2.5 core-bu (5.161.114.8 — Hetzner CPX11, was app1-bu)

Warm standby. Hermes gateway dormant. Watchdog checks Core every 5 min. Needs CPX11→CPX31 upgrade for DR capacity.

2.6 old-ai (178.156.167.181 — Hetzner CPX41)

Current admin-ai LiteLLM host. 93% disk. Target for decommission.

2.7 Remaining Hetzner hosts

Host	IP	Status
docker	178.156.168.35	To decommission
old app1	87.99.144.163	Deprecated
unms	5.161.225.131	DNS to app2 pending
unifi	143.198.185.17	DNS updated Jul 12, decom pending
hudu	178.156.130.130	Ready to migrate
fleettracker	178.156.149.32	Traccar ready to move
wphost02	5.161.62.38	WP to app3
tony-vps	87.99.159.142	Keep — Tony's Hermes

---

3. Ollama Audit — Why It Should Be Excluded

Per directive: "Exclude/strike Ollama from the plan altogether."

3.1 What Ollama provides today

Function	Model	Reality
Local LLM fallback	llama3.2:3b (2 GB, loaded on demand)	Extremely limited; useful only for "I am alive" pings. Not competent for any real work.
Attempted fallback model	llama3.3:70b (42 GB disk)	Cannot load. Requires ~35-40 GB RAM. Core has 15 GB. This model is wasted disk space and should be deleted immediately regardless of architecture decision.

3.2 Problems with Ollama on Core

❌	Issue
❌	42 GB of wasted disk (llama3.3:70b) that can never be used
❌	Creates false sense of security — "we have local fallback" — but the usable model is too dumb for any actual work
❌	llama3.2:3b cannot handle structured JSON, tool calling, code fixes, or even basic email summarization reliably
❌	RAM pressure: Ollama service consumes RAM even when idle, competing with Hermes gateway + browser + Postgres (future LiteLLM DB)
❌	Maintenance overhead: model updates, service health checks, logging
❌	Slows Core boot time

3.3 Verdict

Verdict
❌ Ollama should be completely removed from Core. Delete llama3.3:70b immediately. Remove ollama.service. Remove from failover chain.
❌ llama3.2:3b as last-resort fallback is an architectural lie — it creates the impression of resilience while providing no practical value. If all remote providers fail, Hermes should report "all providers exhausted" rather than silently degrading to a model that cannot perform any task.

---

4. Replacement Models for Low-Lift Tasks

Per directive: "Present other model options for low-lift tasks that Ollama could handle."

4.1 What "low-lift" means in practice

Ollama was supposed to handle:

• Survival pings (acknowledge a message exists)
• Very basic classification (is this email spam?)
• Simple summarization of short text
• "I'm alive" health checks

None of these require a local model. They can be handled by dirt-cheap remote APIs with far better results.

4.2 Cheapest available APIs (Jul 2026)

From live pricing research:

Model	Provider	Input/1M	Output/1M	Context	Notes
GPT-4.1 Nano	OpenAI/OpenRouter	$0.10	$0.40	1M	Cheapest competent model; good enough for classification/light summarization
Gemini 2.0 Flash / Flash-Lite	Google/OpenRouter	$0.10	$0.40	1M	Free tier available; multimodal
DeepSeek V4 Flash	DeepSeek direct	$0.14	$0.28	1M	Already in admin-ai; cheaper than Ollama electricity
Mistral Small	Mistral/OpenRouter	$0.10	$0.30	128K	Good for classification
GLM-4.7-Flash	Z.ai	Free	Free	128K	Actually free tier; limited but usable
GLM-4.6V-Flash	Z.ai	Free	Free	—	Free vision model
Qwen Qwen3-Flash	Alibaba/OpenRouter	$0.07	$0.14	32K	Ultra cheap
Llama 3.1 8B	Various/OpenRouter	$0.05–$0.10	$0.05–$0.20	128K	Open model, many providers; dirt cheap

4.3 Recommended replacement tier

Tier	Model	Route	Cost estimate	Use case
Survival ping	deepseek-v4-flash	admin-ai	~$0.14/M input	Already has access; extremely cheap; 1M context; way better than llama3.2:3b
Cheapest possible	openrouter/qwen/qwen3-flash	OpenRouter	~$0.07/$0.14	Classification, trivial summarization
Free option	openrouter/z-ai/glm-4.7-flash	OpenRouter	Free	Low-risk cron jobs, health checks, zero-cost classification
When admin-ai is down	openrouter/deepseek/deepseek-chat	OpenRouter	~$0.14/$0.28	Already in fallback chain; keep
When OpenRouter is also down	Report "all providers exhausted"	—	—	Honest failure > silent useless degradation

4.4 Cost comparison: Ollama vs cheapest API

Scenario	Ollama cost	API cost
Per month electricity/RAM	~$5–$10 (portion of VPS cost for RAM + CPU cycles)	~$0.01–$0.50 for all survival-ping volume
Quality	Unusable for real work	Usable for light tasks
Maintenance	Updates, service management, disk cleanup	Zero
Disk	2–42 GB wasted	Zero

Verdict: The cheapest API is economically better AND qualitatively better than Ollama for every single use case that Ollama was supposed to cover.

4.5 The honest fallback design

Remove the survival-model pretense:

Fallback 1: deepseek-v4-pro          via admin-ai          (different model, same provider)
Fallback 2: deepseek/deepseek-chat    via OpenRouter        (different provider)
Fallback 3: deepseek-v4-flash         via admin-ai          (cheapest competent fallback)
Fallback 4: openrouter/qwen/qwen3-flash via OpenRouter       (dirt cheap, still better than Ollama)
Fallback 5: openrouter/z-ai/glm-4.7-flash via OpenRouter     (free tier — zero cost last resort)

Or, simpler:

Fallback 1: deepseek-v4-pro          via admin-ai
Fallback 2: deepseek-v4-flash        via admin-ai
Fallback 3: deepseek/deepseek-chat   via OpenRouter
Fallback 4: openrouter/qwen/qwen3-flash via OpenRouter

When all are exhausted: Hermes reports "All providers exhausted. Check admin-ai and OpenRouter status." No silent degradation.

---

5. Architecture Audit — Original Plan vs Reality

5.1 The approved rebalance plan

From server-architecture-plan (rebalanced Jul 12):

Core:   Hermes + LiteLLM + Ollama fallback
app1:   Caddy (all domains), Ops Portal, OSINT API, Super Search MCP,
        SearXNG, Grafana, cron, n8n + Postgres, Docker services
app2:   UNMS/UISP, UniFi Controller, Hudu (target), Traccar (target)
app3:   WordPress / web apps
core-bu: Warm standby

5.2 Problems found

5.2.1 ❌ admin-ai.itpropartner.com DNS cutover risk

Plan says: LiteLLM → Core. DNS stays (implied localhost or unchanged).

Reality: Today admin-ai.itpropartner.com resolves to 178.156.167.181 (Hetzner old-ai). The core-rebalance-plan.md says "admin-ai.itpropartner.com → Core IP (stays)" — but this is contradictory. The domain currently points to old-ai, not Core. Moving LiteLLM to Core means DNS must change, OR Hermes must switch from admin-ai.itpropartner.com to localhost. The plan does not address this.

Impact: If LiteLLM moves to Core and Hermes uses https://admin-ai.itpropartner.com/v1, then during the DNS propagation window:

• Hermes on Core tries to reach admin-ai.itpropartner.com
• DNS may still point to old-ai
• Old-ai LiteLLM may already be down
• Result: Hermes falls back to OpenRouter unnecessarily

Fix: Either:

• (a) Change Hermes providers.admin-ai.base_url to http://localhost:4000/v1 after LiteLLM is running locally (no DNS dependency), OR
• (b) Switch admin-ai.itpropartner.com DNS to Core IP as a deliberate step in the migration, accepting the propagation window.

Option (a) is cleaner and eliminates DNS as a failure mode for Hermes→admin-ai entirely. The domain admin-ai.itpropartner.com would still serve external clients.

5.2.2 ⚠️ core-bu fails over to OpenRouter-only

Plan says: core-bu takes over when Core is down.

Problem: If Core hosts LiteLLM, then a Core outage kills both Hermes AND admin-ai simultaneously. core-bu's Hermes would have zero access to admin-ai. It would fall to OpenRouter.

Is this acceptable? Yes — IF we accept the tradeoff. But the plan does not acknowledge this explicitly.

Mitigation: core-bu should be pre-configured to use OpenRouter as primary during failover. The current standby config already uses admin-ai, which would be dead. This needs an explicit config change for core-bu's Hermes.

5.2.3 ⚠️ RAM on Core with LiteLLM + Postgres

Component	Estimated RAM
Hermes gateway + browser + assistant	~2-3 GB
LiteLLM + Postgres (Docker)	~1-2 GB
OSINT Person + Super Search + DRE + Twilio MCPs	~1 GB
Caddy (if kept)	~100 MB
MySQL tunnel	~200 MB
System + buffers	~2 GB
Docker daemon	~500 MB
Core Docker services to be evacuated	~3-4 GB (freed)
Total after evacuation	~7-9 GB of 15 GB

This is acceptable but tight. LiteLLM under load can spike Postgres memory. No headroom for Ollama (which we're removing anyway). No headroom for spikes.

Recommendation: Document the RAM budget explicitly. Monitor Postgres memory after migration. Be prepared to add LiteLLM resource limits.

5.2.4 ❌ core-rebalance-plan.md is too sparse

The full rebalance plan is only 25 lines. It lacks:

• Rollback steps per phase
• Verification gates between phases
• DNS propagation wait times
• Service dependency ordering
• What to do if a phase fails
• Cron job re-pinning steps
• Anita profile verification step
• Vision model verification step
• Gateway model override check
• S3 sync impact during migration

This is not a reviewable plan. It is an outline.

5.2.5 ⚠️ Caddy on Core is both needed and not needed

Core needs Caddy for:

• core.itpropartner.com paths (Vaultwarden, Twenty, DocuSeal, etc.) — but these are moving to app1
• Hermes PWA / browser tool — internal-only? or public?

If all web services move to app1, Core's Caddy becomes only:

• Hermes PWA backend proxy
• Possible browser tool endpoint

Question: After rebalance, does Core need a public Caddy at all? If Hermes is internally accessed and all web services are on app1, Core could run with no public ports except SSH and VPN.

5.2.6 ⚠️ MCP services running on Core but serving app1-hosted tools

Super Search depends on SearXNG (Docker container). After rebalance, SearXNG moves to app1. Super Search's server.py references http://127.0.0.1:8888/search for SearXNG.

Impact: Super Search on Core would be calling app1's SearXNG across the network. This introduces latency and a cross-server dependency where none existed before.

Fix: Either:

• (a) Point Super Search's SearXNG URL at app1's internal/private IP, OR
• (b) Keep a minimal SearXNG on Core for Super Search, OR
• (c) Move Super Search MCP to app1 alongside SearXNG

Option (c) is cleanest but then Super Search is not local to Hermes — it would be accessed via MCP over HTTP between servers.

5.2.7 ❌ Ollama is in the plan

Both server-architecture-plan and core-rebalance-plan reference Ollama as a fallback. Per the directive, this must be removed.

5.2.8 ❌ The plan fails the "fault isolation" principle

Core becomes:

Hermes + LiteLLM + all MCPs + Postgres + Caddy + Tailscale + WireGuard + SSH

This is a single box that:

• Runs the agent
• Runs the model proxy
• Runs all MCP tool servers
• Runs the reverse proxy
• Runs the VPN

A Core outage is now a total outage — no agent, no model access, no tools, no remote access. Even with core-bu failover, core-bu would have no admin-ai and limited MCP access.

Compare to the alternative:

Core:     Hermes + MCPs (tools only)
app1:     LiteLLM + Caddy + all web services + Super Search + SearXNG + Grafana + cron

Core outage → core-bu takes over Hermes, admin-ai still up on app1
app1 outage → Core Hermes loses admin-ai, falls to OpenRouter; web services down but agent still alive

This alternative provides partial degradation. Both boxes must fail simultaneously for a total outage.

---

6. Revised Architecture Recommendation

6.1 Target roles (Ollama removed)

Box	Model	Role	Critical for Hermes?
Core	RS 2000	Hermes Agent + essential MCPs only	Yes
app1	RS 4000	LiteLLM (admin-ai) + Caddy (all domains) + Ops Portal + Super Search + SearXNG + OSINT API + Grafana + cron + n8n + Postgres + all Docker services	Partially — admin-ai is critical; web services are not
app2	RS 4000	UNMS/UISP, UniFi, Hudu, Traccar	No
app3	RS 4000	WordPress, Apex, web apps	No
core-bu	CPX31	Warm standby for Core	Yes (DR)

6.2 Core — keep only what Hermes needs to function

Keep on Core	Reason
hermes.service	Core function
hermes-browser.service	Browser tool for Hermes
hermes-assistant.service	PWA backend
osint-person.service	OSINT Person MCP (local = fast, no cross-server latency)
super-search.service	Super Search MCP (local; SearXNG endpoint updated to app1)
dre-mcp.service	DRE MCP
twilio-mcp.service	Twilio MCP
mysql-tunnel.service	MySQL SSH tunnel to Apex DB
caddy.service	Only if needed for Hermes PWA; otherwise remove
tailscaled.service	Private infrastructure access
ssh.service	Administrative access
cron.service	System cron
docker.service	Only if MCPs containerized; otherwise remove

Move off Core → target	Service
→ app1	vaultwarden (Docker)
→ app1	komodo-core + komodo-mongo (Docker)
→ app1	prometheus + grafana (Docker)
→ app1	twenty-server + worker + db + redis (Docker)
→ app1	docuseal (Docker)
→ app1	searxng (Docker)
→ app1	ops-portal.service (systemd)
→ app1	osint-api.service (systemd)
→ app1	shark-game.service (systemd)
→ app1	node_exporter.service (systemd)
→ app1	caddy.service (systemd — if Core doesn't need it)
→ Delete	ollama.service + ollama models
→ Delete	strongswan-starter.service + xl2tpd.service (VPN — replace with Tailscale-only or keep if needed)

6.3 Revised fallback chain (no Ollama)

Primary:      gpt-5.5                via admin-ai (app1)
Fallback 1:   deepseek-v4-pro        via admin-ai (app1)
Fallback 2:   deepseek-v4-flash      via admin-ai (app1)
Fallback 3:   deepseek/deepseek-chat via OpenRouter
Fallback 4:   openrouter/qwen/qwen3-flash via OpenRouter

When all exhausted: Hermes reports "All providers exhausted. Check admin-ai and OpenRouter."

6.4 Revised DNS plan

Domain	Current →	Target
admin-ai.itpropartner.com	178.156.167.181 (old-ai)	152.53.36.131 (app1)
ops.itpropartner.com	Core	app1
core.itpropartner.com	Core	app1 (all web services move)
sign.itpropartner.com	Core	app1
n8n.itpropartner.com	Already app1	No change
ai.itpropartner.com	Already app1?	No change

Hermes on Core uses:

providers.admin-ai.base_url = https://admin-ai.itpropartner.com/v1

This means admin-ai DNS must point to app1 before Hermes can use it. The DNS cutover for admin-ai is the critical path. All other DNS changes can happen independently.

6.5 Revised migration phases

Phase 0 — Pre-flight (no downtime)

1. Inventory old-ai admin-ai completely
2. Pull LiteLLM image on app1
3. Create LiteLLM docker-compose on app1
4. Dump old-ai Postgres → import to app1
5. Start app1 LiteLLM, test locally
6. Verify all models work through app1 local LiteLLM
7. Verify encrypted provider keys survived (SALT_KEY match)
8. Move non-Hermes Docker services: Core → app1
9. Remove Ollama from Core (stop service, delete models, disable service)

Phase 1 — DNS cutover (planned window)

1. Switch Hermes to OpenRouter-only fallback temporarily
2. SiteGround DNS: admin-ai.itpropartner.com → 152.53.36.131
3. Wait for DNS propagation (check from Core: dig +short admin-ai.itpropartner.com)
4. Caddy on app1 issues cert for admin-ai.itpropartner.com
5. Verify: curl https://admin-ai.itpropartner.com/v1/models from Core
6. Restore Hermes admin-ai primary + full fallback chain
7. Test full model chain: gpt-5.5, deepseek-v4-pro, deepseek-v4-flash

Phase 2 — Remaining DNS and service moves

1. Caddy → app1
2. ops.itpropartner.com → app1
3. core.itpropartner.com → app1
4. sign.itpropartner.com → app1
5. Move systemd services (ops-portal, osint-api, shark-game) → app1
6. Move cron jobs → app1 (or re-create on app1)
7. Update Super Search SearXNG URL to app1's SearXNG

Phase 3 — Verification

1. Verify Hermes works with new admin-ai (app1)
2. Verify core-bu Hermes standby config
3. Verify all cron jobs
4. Verify Anita profile
5. Verify vision model
6. Verify gateway sessions have no stale model overrides

Phase 4 — Stabilization (7 days)

• Keep old-ai running for rollback
• Compare LiteLLM spend logs old vs new
• Monitor Core RAM/CPU after evacuation
• Confirm Caddy cert auto-renewal on app1
• Confirm all MCP tools function

Phase 5 — Decommission

• old-ai (178.156.167.181) → cancel
• docker (178.156.168.35) → cancel
• remaining Hetzner hosts → cancel as each service confirms migration

Rollback per phase

Phase	Rollback action
Phase 1	DNS back to old-ai IP; restore Hermes to admin-ai. Insert master key hash into old DB if needed.
Phase 2	DNS back to Core IPs; restart services on Core.
Phase 3	Fix individual failures without full rollback.
Phase 4	If app1 admin-ai unstable, Phase 1 rollback.

---

7. Open Questions for 3rd-Party Review

These are areas where the plan has ambiguity that a reviewer should flag:

1. Does Core need a public Caddy after all web services move? If Hermes PWA and browser tool are internal-only, Core could run with SSH + Tailscale only. This simplifies Core dramatically.

2. Should admin-ai be on Core or app1? This audit recommends app1 for fault isolation. The user's earlier intuition (Core colocation) favors simplicity. Both arguments are presented above. The reviewer should weigh them.

3. Is core-bu's CPX11 sufficient for DR if admin-ai moves to app1? CPX11 has 2 vCPU / 2 GB RAM. Hermes gateway alone may fit, but subagent delegation and browser tools may struggle. The plan notes CPX31 upgrade is needed.

4. What happens to Super Search if SearXNG moves to app1? Cross-server dependency is introduced. Should Super Search move to app1 too? Or should a minimal SearXNG stay on Core?

5. Are the free-tier models (GLM-4.7-Flash) reliable enough for the last fallback tier? Free tiers can disappear without notice. The plan should include a fallback that doesn't depend on a free tier.

6. Should VPN (WireGuard/StrongSwan) stay on Core or move? If Core goes down and VPN is on Core, remote access to the entire infrastructure is lost. VPN on a separate box (or redundant) would be safer.

---

8. Summary of Changes from Original Plan

Original Plan Element	Audit Finding	Revised
Ollama on Core	Removed entirely	Deleted. Replaced with cheap API fallbacks
admin-ai → Core	Risk of single-point failure	admin-ai → app1
Core keeps Caddy + all Docker services	Breaches "Core is lightweight" principle	Move all non-Hermes services to app1
3-step migration outline	Too sparse; no rollback	5-phase plan with per-phase rollback
DNS plan vague	admin-ai DNS path unclear	Explicit DNS cutover with propagation check
Fallback chain ends at Ollama	Useless fallback	Chain ends at cheap-but-competent API, not local model
No core-bu config plan	Standby would use dead admin-ai	core-bu must use OpenRouter during Core outage
Super Search ↔ SearXNG co-location broken	Cross-server dependency introduced	Explicitly flagged; recommendation to move Super Search to app1
3-day stabilization	Too short	7-day stabilization

---

End of audit. Ready for 3rd-party review.