Files
hermes-recovery/references/architecture-audit-2026-07-13.md
T

560 lines
24 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Server Architecture Plan — Independent Audit
**Audited:** July 13, 2026 00:21 EDT
**Auditor:** Sho'Nuff (Hermes Agent)
**Purpose:** Review approved architecture, identify gaps, present to unbiased 3rdparty reviewer
**Scope:** Full architecture ...
Legend for all following sections:
| Symbol | Meaning |
|---|---:|
| ✅ | Sound / no issue found |
| ⚠️ | Concern — needs discussion or mitigation |
| ❌ | Flaw — contradicts stated goal or creates risk |
| — | Not applicable / excluded by directive |
---
## 1. Guiding Principles (from User Memory / Context)
| Principle | Source |
|---|---|
| Fault isolation matters — no all-in-one servers | Memory |
| Reliability / performance > savings | Memory |
| No single points of failure | Memory |
| Offline recovery capability required | Memory |
| Provider diversity for DR | Memory |
| Uptime / restorability > cost / convenience | Memory |
| Model / API spend now a real constraint | Memory |
| Clean direct solutions over proxy hops | Memory |
| Do only what is asked; no scope creep | Memory |
| Everything not needed for Hermes runtime should move off Core | This request |
| Ollama excluded from plan entirely | This request |
| Plan must survive independent 3rdparty review | This request |
---
## 2. Current Live State (observed Jul 13 00:21 EDT)
### 2.1 Core (152.53.192.33 — RS 2000 G12)
| Resource | Value |
|---|---:|
| vCPU | 8 (non-dedicated) |
| RAM | 15 GB |
| Disk | 503 GB (16% used, ~406 GB free) |
| Docker | 11 containers running |
| systemd services | 33 running |
**Docker containers on Core:**
| Container | Purpose | Needed for Hermes to function? |
|---|---|---|
| vaultwarden | Password vault | No |
| komodo-core-1 + komodo-mongo-1 | Docker management | No |
| prometheus | Metrics | No |
| grafana | Dashboards | No |
| twenty-server-1 + twenty-worker-1 + twenty-db-1 + twenty-redis-1 | CRM (Twenty) | No |
| docuseal | Document signing | No |
| searxng | Meta-search engine | No (Super Search uses it but runs as separate service) |
**systemd services on Core relevant to architecture:**
| Service | Purpose | Needed for Hermes to function? |
|---|---|---|
| hermes.service | Hermes Agent gateway | **Yes** |
| hermes-browser.service | Headless Chromium for browser tool | **Yes** |
| hermes-assistant.service | PWA backend | **Yes** |
| ollama.service | Local LLM server | **No** (to be removed per this directive) |
| caddy.service | Reverse proxy | Partially — only for core.itpropartner.com paths; full Caddy can move |
| ops-portal.service | Ops Portal FastAPI backend | No |
| osint-api.service | OSINT API | No |
| osint-person.service | OSINT Person MCP | **Yes** (runs locally, needed for Hermes) |
| super-search.service | Super Search MCP | **Yes** (runs locally, needed for Hermes) |
| dre-mcp.service | DRE MCP | **Yes** (runs locally, needed for Hermes) |
| twilio-mcp.service | Twilio MCP | **Yes** (runs locally, needed for Hermes) |
| mysql-tunnel.service | MySQL SSH tunnel to Apex DB | **Yes** (needed for mysql MCP tool) |
| shark-game.service | Shark Game backend | No |
| node_exporter.service | Prometheus exporter | No |
| strongswan-starter.service | VPN | No (for remote access; not Hermes-critical) |
| xl2tpd.service | L2TP VPN | No |
**Current Hermes provider/failover chain:**
```text
Primary: gpt-5.5 via admin-ai (https://admin-ai.itpropartner.com/v1)
Fallback 1: deepseek-v4-pro via admin-ai
Fallback 2: deepseek/deepseek-chat via OpenRouter (https://openrouter.ai/api/v1)
Fallback 3: llama3.2:3b via ollama-local (http://localhost:11434/v1)
```
Delegation (subagent) primary: `deepseek-v4-pro` via admin-ai.
admin-ai.itpropartner.com currently points to: **178.156.167.181 (Hetzner old-ai box).**
### 2.2 app1 (152.53.36.131 — RS 4000 G12)
| Resource | Value |
|---|---:|
| vCPU | 12 dedicated EPYC 9645 |
| RAM | 32 GB |
| Disk | 1 TB (3% used, ~943 GB free) |
Running: Ollama, Open WebUI, n8n + n8n-postgres, Caddy.
### 2.3 app2 (152.53.39.202 — RS 4000 G12)
Running: UNMS/UISP stack, UniFi Controller (deployed Jul 12).
### 2.4 app3 (152.53.241.111 — RS 4000 G12)
Target for WordPress/Apex from SiteGround/wphost02.
### 2.5 core-bu (5.161.114.8 — Hetzner CPX11, was app1-bu)
Warm standby. Hermes gateway dormant. Watchdog checks Core every 5 min. Needs CPX11→CPX31 upgrade for DR capacity.
### 2.6 old-ai (178.156.167.181 — Hetzner CPX41)
Current admin-ai LiteLLM host. 93% disk. Target for decommission.
### 2.7 Remaining Hetzner hosts
| Host | IP | Status |
|---|---|---|
| docker | 178.156.168.35 | To decommission |
| old app1 | 87.99.144.163 | Deprecated |
| unms | 5.161.225.131 | DNS to app2 pending |
| unifi | 143.198.185.17 | DNS updated Jul 12, decom pending |
| hudu | 178.156.130.130 | Ready to migrate |
| fleettracker | 178.156.149.32 | Traccar ready to move |
| wphost02 | 5.161.62.38 | WP to app3 |
| tony-vps | 87.99.159.142 | Keep — Tony's Hermes |
---
## 3. Ollama Audit — Why It Should Be Excluded
Per directive: **"Exclude/strike Ollama from the plan altogether."**
### 3.1 What Ollama provides today
| Function | Model | Reality |
|---|---|---|
| Local LLM fallback | `llama3.2:3b` (2 GB, loaded on demand) | Extremely limited; useful only for "I am alive" pings. Not competent for any real work. |
| Attempted fallback model | `llama3.3:70b` (42 GB disk) | **Cannot load.** Requires ~35-40 GB RAM. Core has 15 GB. This model is wasted disk space and should be deleted immediately regardless of architecture decision. |
### 3.2 Problems with Ollama on Core
| ❌ | Issue |
|---|---|
| ❌ | 42 GB of wasted disk (`llama3.3:70b`) that can never be used |
| ❌ | Creates false sense of security — "we have local fallback" — but the usable model is too dumb for any actual work |
| ❌ | `llama3.2:3b` cannot handle structured JSON, tool calling, code fixes, or even basic email summarization reliably |
| ❌ | RAM pressure: Ollama service consumes RAM even when idle, competing with Hermes gateway + browser + Postgres (future LiteLLM DB) |
| ❌ | Maintenance overhead: model updates, service health checks, logging |
| ❌ | Slows Core boot time |
### 3.3 Verdict
| Verdict |
|---|
| ❌ Ollama should be completely removed from Core. Delete `llama3.3:70b` immediately. Remove `ollama.service`. Remove from failover chain. |
| ❌ `llama3.2:3b` as last-resort fallback is an architectural lie — it creates the impression of resilience while providing no practical value. If all remote providers fail, Hermes should report "all providers exhausted" rather than silently degrading to a model that cannot perform any task. |
---
## 4. Replacement Models for Low-Lift Tasks
Per directive: **"Present other model options for low-lift tasks that Ollama could handle."**
### 4.1 What "low-lift" means in practice
Ollama was supposed to handle:
- Survival pings (acknowledge a message exists)
- Very basic classification (is this email spam?)
- Simple summarization of short text
- "I'm alive" health checks
None of these require a local model. They can be handled by dirt-cheap remote APIs with far better results.
### 4.2 Cheapest available APIs (Jul 2026)
From live pricing research:
| Model | Provider | Input/1M | Output/1M | Context | Notes |
|---|---:|---:|---:|---:|---|
| **GPT-4.1 Nano** | OpenAI/OpenRouter | $0.10 | $0.40 | 1M | Cheapest competent model; good enough for classification/light summarization |
| **Gemini 2.0 Flash / Flash-Lite** | Google/OpenRouter | $0.10 | $0.40 | 1M | Free tier available; multimodal |
| **DeepSeek V4 Flash** | DeepSeek direct | $0.14 | $0.28 | 1M | Already in admin-ai; cheaper than Ollama electricity |
| **Mistral Small** | Mistral/OpenRouter | $0.10 | $0.30 | 128K | Good for classification |
| **GLM-4.7-Flash** | Z.ai | **Free** | **Free** | 128K | Actually free tier; limited but usable |
| **GLM-4.6V-Flash** | Z.ai | **Free** | **Free** | — | Free vision model |
| **Qwen Qwen3-Flash** | Alibaba/OpenRouter | $0.07 | $0.14 | 32K | Ultra cheap |
| **Llama 3.1 8B** | Various/OpenRouter | $0.05$0.10 | $0.05$0.20 | 128K | Open model, many providers; dirt cheap |
### 4.3 Recommended replacement tier
| Tier | Model | Route | Cost estimate | Use case |
|---|---:|---|---|---|
| **Survival ping** | `deepseek-v4-flash` | admin-ai | ~$0.14/M input | Already has access; extremely cheap; 1M context; way better than llama3.2:3b |
| **Cheapest possible** | `openrouter/qwen/qwen3-flash` | OpenRouter | ~$0.07/$0.14 | Classification, trivial summarization |
| **Free option** | `openrouter/z-ai/glm-4.7-flash` | OpenRouter | Free | Low-risk cron jobs, health checks, zero-cost classification |
| **When admin-ai is down** | `openrouter/deepseek/deepseek-chat` | OpenRouter | ~$0.14/$0.28 | Already in fallback chain; keep |
| **When OpenRouter is also down** | Report "all providers exhausted" | — | — | Honest failure > silent useless degradation |
### 4.4 Cost comparison: Ollama vs cheapest API
| Scenario | Ollama cost | API cost |
|---|---|---|
| Per month electricity/RAM | ~$5$10 (portion of VPS cost for RAM + CPU cycles) | ~$0.01$0.50 for all survival-ping volume |
| Quality | Unusable for real work | Usable for light tasks |
| Maintenance | Updates, service management, disk cleanup | Zero |
| Disk | 242 GB wasted | Zero |
**Verdict:** The cheapest API is economically better AND qualitatively better than Ollama for every single use case that Ollama was supposed to cover.
### 4.5 The honest fallback design
Remove the survival-model pretense:
```text
Fallback 1: deepseek-v4-pro via admin-ai (different model, same provider)
Fallback 2: deepseek/deepseek-chat via OpenRouter (different provider)
Fallback 3: deepseek-v4-flash via admin-ai (cheapest competent fallback)
Fallback 4: openrouter/qwen/qwen3-flash via OpenRouter (dirt cheap, still better than Ollama)
Fallback 5: openrouter/z-ai/glm-4.7-flash via OpenRouter (free tier — zero cost last resort)
```
Or, simpler:
```text
Fallback 1: deepseek-v4-pro via admin-ai
Fallback 2: deepseek-v4-flash via admin-ai
Fallback 3: deepseek/deepseek-chat via OpenRouter
Fallback 4: openrouter/qwen/qwen3-flash via OpenRouter
```
When all are exhausted: Hermes reports "All providers exhausted. Check admin-ai and OpenRouter status." No silent degradation.
---
## 5. Architecture Audit — Original Plan vs Reality
### 5.1 The approved rebalance plan
From `server-architecture-plan` (rebalanced Jul 12):
```text
Core: Hermes + LiteLLM + Ollama fallback
app1: Caddy (all domains), Ops Portal, OSINT API, Super Search MCP,
SearXNG, Grafana, cron, n8n + Postgres, Docker services
app2: UNMS/UISP, UniFi Controller, Hudu (target), Traccar (target)
app3: WordPress / web apps
core-bu: Warm standby
```
### 5.2 Problems found
#### 5.2.1 ❌ admin-ai.itpropartner.com DNS cutover risk
**Plan says:** LiteLLM → Core. DNS stays (implied localhost or unchanged).
**Reality:** Today admin-ai.itpropartner.com resolves to 178.156.167.181 (Hetzner old-ai). The `core-rebalance-plan.md` says "admin-ai.itpropartner.com → Core IP (stays)" — but this is contradictory. The domain currently points to old-ai, not Core. Moving LiteLLM to Core means DNS must change, OR Hermes must switch from `admin-ai.itpropartner.com` to `localhost`. The plan does not address this.
**Impact:** If LiteLLM moves to Core and Hermes uses `https://admin-ai.itpropartner.com/v1`, then during the DNS propagation window:
- Hermes on Core tries to reach admin-ai.itpropartner.com
- DNS may still point to old-ai
- Old-ai LiteLLM may already be down
- Result: Hermes falls back to OpenRouter unnecessarily
**Fix:** Either:
- (a) Change Hermes `providers.admin-ai.base_url` to `http://localhost:4000/v1` after LiteLLM is running locally (no DNS dependency), OR
- (b) Switch admin-ai.itpropartner.com DNS to Core IP as a deliberate step in the migration, accepting the propagation window.
Option (a) is cleaner and eliminates DNS as a failure mode for Hermes→admin-ai entirely. The domain `admin-ai.itpropartner.com` would still serve external clients.
#### 5.2.2 ⚠️ core-bu fails over to OpenRouter-only
**Plan says:** core-bu takes over when Core is down.
**Problem:** If Core hosts LiteLLM, then a Core outage kills both Hermes AND admin-ai simultaneously. core-bu's Hermes would have zero access to admin-ai. It would fall to OpenRouter.
**Is this acceptable?** Yes — IF we accept the tradeoff. But the plan does not acknowledge this explicitly.
**Mitigation:** core-bu should be pre-configured to use OpenRouter as primary during failover. The current standby config already uses admin-ai, which would be dead. This needs an explicit config change for core-bu's Hermes.
#### 5.2.3 ⚠️ RAM on Core with LiteLLM + Postgres
| Component | Estimated RAM |
|---|---:|
| Hermes gateway + browser + assistant | ~2-3 GB |
| LiteLLM + Postgres (Docker) | ~1-2 GB |
| OSINT Person + Super Search + DRE + Twilio MCPs | ~1 GB |
| Caddy (if kept) | ~100 MB |
| MySQL tunnel | ~200 MB |
| System + buffers | ~2 GB |
| Docker daemon | ~500 MB |
| Core Docker services to be evacuated | ~3-4 GB (freed) |
| **Total after evacuation** | **~7-9 GB of 15 GB** |
This is **acceptable but tight**. LiteLLM under load can spike Postgres memory. No headroom for Ollama (which we're removing anyway). No headroom for spikes.
**Recommendation:** Document the RAM budget explicitly. Monitor Postgres memory after migration. Be prepared to add LiteLLM resource limits.
#### 5.2.4 ❌ `core-rebalance-plan.md` is too sparse
The full rebalance plan is only 25 lines. It lacks:
- Rollback steps per phase
- Verification gates between phases
- DNS propagation wait times
- Service dependency ordering
- What to do if a phase fails
- Cron job re-pinning steps
- Anita profile verification step
- Vision model verification step
- Gateway model override check
- S3 sync impact during migration
This is not a reviewable plan. It is an outline.
#### 5.2.5 ⚠️ Caddy on Core is both needed and not needed
Core needs Caddy for:
- `core.itpropartner.com` paths (Vaultwarden, Twenty, DocuSeal, etc.) — but these are moving to app1
- Hermes PWA / browser tool — internal-only? or public?
If all web services move to app1, Core's Caddy becomes only:
- Hermes PWA backend proxy
- Possible browser tool endpoint
**Question:** After rebalance, does Core need a public Caddy at all? If Hermes is internally accessed and all web services are on app1, Core could run with no public ports except SSH and VPN.
#### 5.2.6 ⚠️ MCP services running on Core but serving app1-hosted tools
Super Search depends on SearXNG (Docker container). After rebalance, SearXNG moves to app1. Super Search's server.py references `http://127.0.0.1:8888/search` for SearXNG.
**Impact:** Super Search on Core would be calling app1's SearXNG across the network. This introduces latency and a cross-server dependency where none existed before.
**Fix:** Either:
- (a) Point Super Search's SearXNG URL at app1's internal/private IP, OR
- (b) Keep a minimal SearXNG on Core for Super Search, OR
- (c) Move Super Search MCP to app1 alongside SearXNG
Option (c) is cleanest but then Super Search is not local to Hermes — it would be accessed via MCP over HTTP between servers.
#### 5.2.7 ❌ Ollama is in the plan
Both `server-architecture-plan` and `core-rebalance-plan` reference Ollama as a fallback. Per the directive, this must be removed.
#### 5.2.8 ❌ The plan fails the "fault isolation" principle
Core becomes:
```text
Hermes + LiteLLM + all MCPs + Postgres + Caddy + Tailscale + WireGuard + SSH
```
This is a single box that:
- Runs the agent
- Runs the model proxy
- Runs all MCP tool servers
- Runs the reverse proxy
- Runs the VPN
A Core outage is now a **total outage** — no agent, no model access, no tools, no remote access. Even with core-bu failover, core-bu would have no admin-ai and limited MCP access.
Compare to the alternative:
```text
Core: Hermes + MCPs (tools only)
app1: LiteLLM + Caddy + all web services + Super Search + SearXNG + Grafana + cron
Core outage → core-bu takes over Hermes, admin-ai still up on app1
app1 outage → Core Hermes loses admin-ai, falls to OpenRouter; web services down but agent still alive
```
This alternative provides partial degradation. Both boxes must fail simultaneously for a total outage.
---
## 6. Revised Architecture Recommendation
### 6.1 Target roles (Ollama removed)
| Box | Model | Role | Critical for Hermes? |
|---:|---|---|---|
| **Core** | RS 2000 | **Hermes Agent + essential MCPs only** | Yes |
| **app1** | RS 4000 | **LiteLLM (admin-ai) + Caddy (all domains) + Ops Portal + Super Search + SearXNG + OSINT API + Grafana + cron + n8n + Postgres + all Docker services** | Partially — admin-ai is critical; web services are not |
| **app2** | RS 4000 | UNMS/UISP, UniFi, Hudu, Traccar | No |
| **app3** | RS 4000 | WordPress, Apex, web apps | No |
| **core-bu** | CPX31 | Warm standby for Core | Yes (DR) |
### 6.2 Core — keep only what Hermes needs to function
| Keep on Core | Reason |
|---|---|
| hermes.service | Core function |
| hermes-browser.service | Browser tool for Hermes |
| hermes-assistant.service | PWA backend |
| osint-person.service | OSINT Person MCP (local = fast, no cross-server latency) |
| super-search.service | Super Search MCP (local; SearXNG endpoint updated to app1) |
| dre-mcp.service | DRE MCP |
| twilio-mcp.service | Twilio MCP |
| mysql-tunnel.service | MySQL SSH tunnel to Apex DB |
| caddy.service | Only if needed for Hermes PWA; otherwise remove |
| tailscaled.service | Private infrastructure access |
| ssh.service | Administrative access |
| cron.service | System cron |
| docker.service | Only if MCPs containerized; otherwise remove |
| Move off Core → target | Service |
|---|---|
| → app1 | vaultwarden (Docker) |
| → app1 | komodo-core + komodo-mongo (Docker) |
| → app1 | prometheus + grafana (Docker) |
| → app1 | twenty-server + worker + db + redis (Docker) |
| → app1 | docuseal (Docker) |
| → app1 | searxng (Docker) |
| → app1 | ops-portal.service (systemd) |
| → app1 | osint-api.service (systemd) |
| → app1 | shark-game.service (systemd) |
| → app1 | node_exporter.service (systemd) |
| → app1 | caddy.service (systemd — if Core doesn't need it) |
| → Delete | ollama.service + ollama models |
| → Delete | strongswan-starter.service + xl2tpd.service (VPN — replace with Tailscale-only or keep if needed) |
### 6.3 Revised fallback chain (no Ollama)
```text
Primary: gpt-5.5 via admin-ai (app1)
Fallback 1: deepseek-v4-pro via admin-ai (app1)
Fallback 2: deepseek-v4-flash via admin-ai (app1)
Fallback 3: deepseek/deepseek-chat via OpenRouter
Fallback 4: openrouter/qwen/qwen3-flash via OpenRouter
```
When all exhausted: Hermes reports "All providers exhausted. Check admin-ai and OpenRouter."
### 6.4 Revised DNS plan
| Domain | Current → | Target |
|---|---|---|
| admin-ai.itpropartner.com | 178.156.167.181 (old-ai) | **152.53.36.131 (app1)** |
| ops.itpropartner.com | Core | **app1** |
| core.itpropartner.com | Core | **app1** (all web services move) |
| sign.itpropartner.com | Core | **app1** |
| n8n.itpropartner.com | Already app1 | No change |
| ai.itpropartner.com | Already app1? | No change |
Hermes on Core uses:
```text
providers.admin-ai.base_url = https://admin-ai.itpropartner.com/v1
```
This means admin-ai DNS must point to app1 before Hermes can use it. The DNS cutover for admin-ai is **the critical path**. All other DNS changes can happen independently.
### 6.5 Revised migration phases
#### Phase 0 — Pre-flight (no downtime)
1. Inventory old-ai admin-ai completely
2. Pull LiteLLM image on app1
3. Create LiteLLM docker-compose on app1
4. Dump old-ai Postgres → import to app1
5. Start app1 LiteLLM, test locally
6. Verify all models work through app1 local LiteLLM
7. Verify encrypted provider keys survived (SALT_KEY match)
8. Move non-Hermes Docker services: Core → app1
9. Remove Ollama from Core (stop service, delete models, disable service)
#### Phase 1 — DNS cutover (planned window)
1. Switch Hermes to OpenRouter-only fallback temporarily
2. SiteGround DNS: admin-ai.itpropartner.com → 152.53.36.131
3. Wait for DNS propagation (check from Core: `dig +short admin-ai.itpropartner.com`)
4. Caddy on app1 issues cert for admin-ai.itpropartner.com
5. Verify: `curl https://admin-ai.itpropartner.com/v1/models` from Core
6. Restore Hermes admin-ai primary + full fallback chain
7. Test full model chain: gpt-5.5, deepseek-v4-pro, deepseek-v4-flash
#### Phase 2 — Remaining DNS and service moves
1. Caddy → app1
2. ops.itpropartner.com → app1
3. core.itpropartner.com → app1
4. sign.itpropartner.com → app1
5. Move systemd services (ops-portal, osint-api, shark-game) → app1
6. Move cron jobs → app1 (or re-create on app1)
7. Update Super Search SearXNG URL to app1's SearXNG
#### Phase 3 — Verification
1. Verify Hermes works with new admin-ai (app1)
2. Verify core-bu Hermes standby config
3. Verify all cron jobs
4. Verify Anita profile
5. Verify vision model
6. Verify gateway sessions have no stale model overrides
#### Phase 4 — Stabilization (7 days)
- Keep old-ai running for rollback
- Compare LiteLLM spend logs old vs new
- Monitor Core RAM/CPU after evacuation
- Confirm Caddy cert auto-renewal on app1
- Confirm all MCP tools function
#### Phase 5 — Decommission
- old-ai (178.156.167.181) → cancel
- docker (178.156.168.35) → cancel
- remaining Hetzner hosts → cancel as each service confirms migration
#### Rollback per phase
| Phase | Rollback action |
|---|---|
| Phase 1 | DNS back to old-ai IP; restore Hermes to admin-ai. Insert master key hash into old DB if needed. |
| Phase 2 | DNS back to Core IPs; restart services on Core. |
| Phase 3 | Fix individual failures without full rollback. |
| Phase 4 | If app1 admin-ai unstable, Phase 1 rollback. |
---
## 7. Open Questions for 3rd-Party Review
These are areas where the plan has ambiguity that a reviewer should flag:
1. **Does Core need a public Caddy after all web services move?** If Hermes PWA and browser tool are internal-only, Core could run with SSH + Tailscale only. This simplifies Core dramatically.
2. **Should admin-ai be on Core or app1?** This audit recommends app1 for fault isolation. The user's earlier intuition (Core colocation) favors simplicity. Both arguments are presented above. The reviewer should weigh them.
3. **Is core-bu's CPX11 sufficient for DR if admin-ai moves to app1?** CPX11 has 2 vCPU / 2 GB RAM. Hermes gateway alone may fit, but subagent delegation and browser tools may struggle. The plan notes CPX31 upgrade is needed.
4. **What happens to Super Search if SearXNG moves to app1?** Cross-server dependency is introduced. Should Super Search move to app1 too? Or should a minimal SearXNG stay on Core?
5. **Are the free-tier models (GLM-4.7-Flash) reliable enough for the last fallback tier?** Free tiers can disappear without notice. The plan should include a fallback that doesn't depend on a free tier.
6. **Should VPN (WireGuard/StrongSwan) stay on Core or move?** If Core goes down and VPN is on Core, remote access to the entire infrastructure is lost. VPN on a separate box (or redundant) would be safer.
---
## 8. Summary of Changes from Original Plan
| Original Plan Element | Audit Finding | Revised |
|---|---|---|
| Ollama on Core | Removed entirely | Deleted. Replaced with cheap API fallbacks |
| admin-ai → Core | Risk of single-point failure | admin-ai → app1 |
| Core keeps Caddy + all Docker services | Breaches "Core is lightweight" principle | Move all non-Hermes services to app1 |
| 3-step migration outline | Too sparse; no rollback | 5-phase plan with per-phase rollback |
| DNS plan vague | admin-ai DNS path unclear | Explicit DNS cutover with propagation check |
| Fallback chain ends at Ollama | Useless fallback | Chain ends at cheap-but-competent API, not local model |
| No core-bu config plan | Standby would use dead admin-ai | core-bu must use OpenRouter during Core outage |
| Super Search ↔ SearXNG co-location broken | Cross-server dependency introduced | Explicitly flagged; recommendation to move Super Search to app1 |
| 3-day stabilization | Too short | 7-day stabilization |
---
**End of audit. Ready for 3rd-party review.**