171 lines
11 KiB
Markdown
171 lines
11 KiB
Markdown
---
|
||
name: server-architecture-plan
|
||
description: ITPP infrastructure architecture — server roles, service placement, and the migration plan from Hetzron to netcup. Approved July 2026.
|
||
tags: [infrastructure, architecture, netcup, migration, planning, RBAC]
|
||
---
|
||
|
||
# Server Architecture Plan
|
||
|
||
## Approved Architecture (July 2026) — Rebalanced Jul 12
|
||
|
||
### Base Server Standard
|
||
All new netcup servers are **RS 4000 G12** — 12 dedicated EPYC 9645 cores, 32 GB DDR5 ECC RAM, 1 TB NVMe, ~$44/mo. Exception: Core (RS 2000) keeps Hermes + LiteLLM + Ollama fallback. core-bu (Hetzner, was app1-bu) is warm standby, pending CPX11→CPX31 upgrade.
|
||
|
||
### Server Roles — Post-Rebalance
|
||
|
||
| Box | Model | Cost | IP | Role |
|
||
|-----|-------|------|-----|------|
|
||
| **Core** | RS 2000 | ~$24/mo | 152.53.192.33 | **Hermes + MCPs only**. LiteLLM, Caddy, Portal, Super Search, Ollama moved to app1/decommissioned. See audit Jul 13. |
|
||
| **app1** | RS 4000 | ~$44/mo | 152.53.36.131 | **Service hub** — Caddy (all domains), Ops Portal, OSINT API, Super Search MCP, SearXNG, Grafana, cron, n8n + Postgres, Docker services |
|
||
| **app2** | RS 4000 | ~$44/mo | 152.53.39.202 | **Infrastructure** — UNMS/UISP, UniFi Controller, Hudu (target), Traccar (target) |
|
||
| **app3** | RS 4000 | ~$44/mo | 152.53.241.111 | **Web apps** — WordPress/Apex from wphost02/SiteGround |
|
||
| **core-bu** (was app1-bu) | CPX11→CPX31 | ~$22/mo | 5.161.114.8 | Warm standby, needs upgrade for DR capacity |
|
||
| | | **~$174/mo** | | |
|
||
|
||
### Hetzner Cleanup Plan (Jul 15, 2026 — mostly complete)
|
||
|
||
**Deleted today (Jul 15):**
|
||
- unifi (178.156.131.57) — ✅ Old UniFi controller decommissioned, backups in S3, server deleted
|
||
- hudu (178.156.130.130) — ✅ Full data migration to app2 verified (135 assets, 125 passwords, 18 layouts restored), server deleted
|
||
- unms (5.161.225.131) — ✅ UNMS running on app2, full backup to S3 (994MB + 1,107 configs), server deleted
|
||
|
||
**Previously deleted:**
|
||
- fleettracker (178.156.149.32) — ✅ Deleted Jul 14
|
||
|
||
**Remaining Hetzner servers (4):**
|
||
- core-bu (5.161.114.8) — Warm standby, keep
|
||
- ai.itpropartner.com (178.156.167.181) — LiteLLM, pending migration to app1
|
||
- wphost02 (5.161.62.38) — WordPress, pending migration to app3
|
||
- app1.itpropartner.com (87.99.144.163) — Old app1, deprecated
|
||
|
||
### Why RS 4000 as standard
|
||
- 12 dedicated EPYC cores — no shared vCPU contention
|
||
- 32 GB ECC RAM — headroom for Java apps (UniFi, Traccar) + 20+ Docker containers
|
||
- 1 TB NVMe — Ollama models alone need 50+ GB, plus UNMS/UISP data
|
||
- Fault isolation — an outage on app2 doesn't touch Core or app1
|
||
|
||
## Service Placement
|
||
|
||
### Core (RS 2000) — Existing, paid
|
||
- Hermes Agent (systemd, port 8081)
|
||
- Caddy reverse proxy (systemd, ports 80/443)
|
||
- Ops Portal backend (FastAPI, systemd, port 8090)
|
||
- Prometheus + Grafana (Docker)
|
||
- **Uptime-Kuma** (Docker, `/root/docker/uptime-kuma/`, port 3001) — migrated from docker server Jul 13 with 9 monitors + Telegram/Discord/Slack notifications
|
||
- FT360 MCP server (FastMCP, systemd, port 8903, `/root/docker/ft360-mcp/`) — 3 tools: get_devices, get_positions, get_stats. Queries Traccar H2 on app2 via SSH+SCP pattern
|
||
- Vaultwarden, DocuSeal, SearXNG, Twenty CRM (Docker under /root/docker/*/)
|
||
- Tailscale (systemd)
|
||
- **Super Search** — FastMCP server at http://127.0.0.1:8899/mcp (systemd: super-search.service). Multi-provider web search: SearXNG → Exa → OpenCorporates → CourtListener → Firecrawl, plus Trafilatura extraction + premium Exa endpoint. All 3 exposed as Hermes MCP tools: `web_search`, `web_extract`, `web_search_premium`. v1.2.0: token-bucket rate limiting, TTL caching, exponential backoff per provider. Deployed at `/root/docker/super-search/server.py` via venv.
|
||
|
||
### app1 (RS 4000) — 152.53.36.131 — Currently Live
|
||
- Ollama (`ollama/ollama:latest`) — running
|
||
- Open WebUI (`ghcr.io/open-webui/open-webui:latest`) — running on :3000
|
||
- n8n (`n8nio/n8n:latest`) — running on :5678, proxied at n8n.itpropartner.com
|
||
- n8n-postgres (`postgres:16-alpine`) — running
|
||
- **LiteLLM** — NOT yet deployed (target for admin-ai migration)
|
||
- Caddy 2.6.2 — active on 80/443, proxying n8n and Open WebUI
|
||
- 1TB disk — 3% used (943 GB free), 32 GB RAM, 12C EPYC
|
||
- All deployments per docker-service-deployment standard at /root/docker/*/
|
||
|
||
See `references/admin-ai-migration-plan.md` for the detailed cutover plan (Phase 0–3).
|
||
|
||
See `references/core-rebalance-plan.md` (created Jul 12, 2026) for the full service rebalance — moving Caddy, Ops Portal, Super Search, SearXNG, Grafana, cron from Core → app1; LiteLLM from Hetzner old-ai → Core; Ollama fallback from app1 → Core. 6-phase executable plan with DNS cutover and rollback.
|
||
|
||
### app2 (RS 4000) — New
|
||
- UNMS/UISP full stack (10 Docker containers: nginx, API, device-ws, PostgreSQL, SiriDB, RabbitMQ, Fluentd, Netflow)
|
||
- UCRM (customer management CRM)
|
||
- UniFi Controller (native Java app + MongoDB)
|
||
- Hudu (**full migration complete Jul 15** — 21 companies, 135 assets, 125 passwords, 18 custom layouts all verified on app2. Old server 178.156.130.130 deleted. Daily backup cron active: 3 AM ET → S3, 30-day retention)
|
||
- Traccar (Docker at `/root/docker/traccar/` — port 8082 web dashboard + 5000-5150 TCP/UDP for GPS device protocols). Cloudflare DNS: fleettracker360.com, gps.fleettracker360.com, www.fleettracker360.com → 152.53.39.202
|
||
|
||
### app3 (RS 4000) — New
|
||
- WordPress/Apex Track Experience (from wphost02 via RunCloud or standalone Nginx)
|
||
- Other WordPress sites (from wphost02 / SiteGround)
|
||
- Web apps requiring PHP/Nginx stack
|
||
|
||
### app1-bu (Hetzner CPX11) — Stays
|
||
- Warm standby for Core
|
||
- Hermes gateway dormant
|
||
- StrongSwan + L2TP
|
||
- Watchdog checks Core every 5 min, 2 min confirmation window
|
||
|
||
## Migration Status
|
||
|
||
### Ready to migrate
|
||
- **ai.itpropartner.com** (LiteLLM, Open WebUI, Ollama, Qdrant) → **app1**
|
||
- LiteLLM uses PostgreSQL DB for model configs — dump/restore
|
||
- Open WebUI data in SQLite at /app/backend/data/webui.db (~121 MB, 1.9 GB total)
|
||
- Ollama models at /root/.ollama (need to copy or re-pull)
|
||
- CRITICAL: admin-ai.itpropartner.com must have zero downtime. Set up new proxy first, test, switch DNS, decommission old.
|
||
- **n8n** (current Hetzner app1) → **app1**
|
||
- Docker compose at /opt/n8n/ — migrate volumes
|
||
- Benefits from direct Ollama access on same box
|
||
|
||
### Approved for migration (pending execution)
|
||
- UNMS/UISP → app2 (Docker stack ready, DNS pending SiteGround update)
|
||
- **UniFi Controller** → app2 (deployed Jul 12 at 152.53.39.202:8443, backup .unf file import needed)
|
||
- **Traccar GPS** → app2 (ready for fresh Docker deploy, HERE Maps API for traffic-aware ETA)
|
||
- **Hudu → app2** (🔴 PARTIAL — DNS cut over, companies migrated, but assets/passwords tables empty. Old server still up with all data. Full DB re-dump/restore needed. See `docker-service-deployment` skill for verification queries.)
|
||
- WordPress → app3
|
||
|
||
### Stays at Hetzner
|
||
- app1-bu (warm standby)
|
||
- tony-vps (Anthony's Hermes)
|
||
- Container host "docker" (178.156.168.35) — NPM, RustDesk, StrongSwan, RunCloud stack remain. Uptime-Kuma migrated to Core Jul 13. Server decommissioning deferred until RunCloud web apps migrate to app3 and RustDesk/StrongSwan decisions are made.
|
||
|
||
### API Access Status (Jul 13, 2026)
|
||
- **Hetzner API token expired** — server deletion/provisioning requires manual Hetzner Cloud Console access until token is rotated
|
||
- **SiteGround DNS** — no API; all DNS changes for itpropartner.com and related domains must be manual through SiteGround web panel
|
||
|
||
## Estimated Monthly Cost Breakdown
|
||
|
||
| Server | Provider | Model | Cost |
|
||
|--------|----------|-------|------|
|
||
| Core | netcup | RS 2000 | ~$24 |
|
||
| app1 | netcup | RS 4000 | ~$44 |
|
||
| app2 | netcup | RS 4000 | ~$44 |
|
||
| app3 | netcup | RS 4000 | ~$44 |
|
||
| app1-bu | Hetzner | CPX11 | ~$8 |
|
||
| **Total** | | | **~$164/mo** |
|
||
| Current Hetzner total | Hetzner | 10 VPS | ~$150/mo (inflated) |
|
||
|
||
## RBAC Model (for ops portal)
|
||
|
||
| Role | Scope | Who |
|
||
|------|-------|-----|
|
||
| **Admin** | Full access — servers, customers, billing, provisioning, DNS, audit logs | Germaine |
|
||
| **Tech/Support** | Tickets, server status, network health. No billing, no provisioning. | Future hire |
|
||
| **Sales/Onboarding** | Customer activations, provisioning status, order forms | Future hire |
|
||
| **Viewer** | Read-only, limited scope | Anita, auditors |
|
||
|
||
Roles built into ops portal from day one. Not retrofitted.
|
||
|
||
## Docker Standard Reference
|
||
|
||
All services at /root/docker/<name>/ with:
|
||
- One `docker-compose.yml`, one `.env` (chmod 600)
|
||
- Healthchecks on every service
|
||
- Resource limits per container
|
||
- Named volumes (not bind mounts for app data)
|
||
- Internal-only networks for service-to-service traffic
|
||
- Logging with rotation (max-size 10m, max-file 3)
|
||
- Automated S3 backup of volumes
|
||
|
||
## Pitfalls
|
||
|
||
- **admin-ai goes down = Sho'Nuff loses model access.** Fallback chain: deepseek-v4-pro (admin-ai) → deepseek/deepseek-chat (OpenRouter) → cheap API fallbacks. No Ollama. admin-ai lives on app1 for fault isolation — Core and admin-ai should not share the same box.
|
||
- **Do NOT put all AI containers in one compose file.** Each service gets its own /root/docker/<name>/ — LiteLLM, Open WebUI are separate.
|
||
- **UniFi on app2 needs MongoDB.** Must migrate the native MongoDB database along with the controller. Cannot just copy files.
|
||
- **Traccar device ports: 5000-5150 TCP+UDP** — Ensure firewall allows this range. GPS devices connect directly (non-proxied DNS at `gps.fleettracker360.com`). Web dashboard on port 8082.
|
||
- **UNMS uses fixed version tags (3.0.140, 4.5.31).** Do not use `:latest` for Ubiquiti images — they break on minor releases.
|
||
- **admin-ai placement decided Jul 13: app1, not Core.** LiteLLM migration target is app1 (152.53.36.131). Core keeps Hermes + MCPs only. See architecture audit at `/root/.hermes/references/architecture-audit-2026-07-13.md`.
|
||
- **Ollama removed Jul 13.** Too weak for real work, wasted RAM on Core. Replacement: cheap API fallbacks (deepseek-v4-flash, qwen3-flash via OpenRouter). Remove ollama.service from Core, delete llama3.3:70b (42GB disk waste).
|
||
- **Caddy + existing containers port conflict (app2)** — When Caddy shares a box with services like UNMS nginx (port 443), Caddy's auto port 80 binding also conflicts. Fix: use `auto_https disable_redirects` + explicit `domain:443` blocks in Caddyfile. Change existing containers to only bind their required ports.
|
||
- **Hudu migration: verify ALL tables, not just companies (Jul 15, 2026).** Initial pg_dump/restore only copied the `companies` table — `assets` and `asset_passwords` were empty on the target. Post-migration verification must query every major table (`companies`, `assets`, `asset_passwords`) and compare row counts between old and new before declaring done. Use `curl --resolve` to hit both old and new IPs through the domain for side-by-side API comparison. See `docker-service-deployment` skill for the full pattern.
|
||
```
|
||
{
|
||
default_bind 152.53.192.33
|
||
}
|
||
```
|
||
This tells Caddy to bind only the public IP. Tailscale keeps its Tailnet IP. Both coexist without conflict. Verify with `caddy validate --config /etc/caddy/Caddyfile` then `systemctl start caddy`.
|