171 lines
7.6 KiB
Markdown
171 lines
7.6 KiB
Markdown
# Hetzner to netcup Migration Plan
|
|
|
|
Author: Sho'Nuff
|
|
Last updated: 2026-07-09
|
|
Status: PLAN (not yet executed)
|
|
Sources: app-inventory.csv (Germaine's Migrate/New Host columns), hetzner-server-audit.md,
|
|
server-provisioning-standard, ops-portal-and-collector skill.
|
|
|
|
---
|
|
|
|
## Objective
|
|
|
|
Consolidate the ITPP server fleet by moving apps/services off Hetzner (whose new-server
|
|
pricing is now premium) onto netcup root (dedicated) servers. Goal: fewer servers, lower
|
|
recurring cost, and NOT an all-in-one design -- workloads are spread so a single outage
|
|
does not take down everything and everyone.
|
|
|
|
Guiding constraints from Germaine (Jul 9, 2026):
|
|
- Cost savings is good but NOT the priority; plenty of overhead for growth on each box.
|
|
- Do not put everything on one server (blast-radius isolation).
|
|
- Keep the MikroTik router API functionality.
|
|
- Keep Ollama on core as a local fallback if admin-ai connectivity is lost.
|
|
- admin-ai (LiteLLM) must stay online at all times -- it is where Hermes gets its model.
|
|
- app1-bu stays as the warm standby at Hetzner (do NOT migrate).
|
|
|
|
---
|
|
|
|
## Target Architecture
|
|
|
|
Base hardware standard for all NEW netcup servers: RS 4000 G12
|
|
- 12 dedicated EPYC 9645 cores, 32 GB DDR5 ECC RAM, 1 TB NVMe
|
|
- ~$44/mo (monthly billing, Manassas VA / Virginia hosting)
|
|
- Monthly billing chosen (NOT annual).
|
|
|
|
Server roles after migration:
|
|
|
|
Core (existing, netcup RS 2000 G12, 152.53.192.33) -- KEEP AS-IS
|
|
- Role: Ops hub. Hermes, Caddy reverse proxy, ops portal, Docker services
|
|
(Twenty CRM, Vaultwarden, DocuSeal, SearXNG), Shark Game, Prometheus, Ollama (fallback), VPN.
|
|
- Also receives: Uptime-Kuma (migrated from Hetzner 'docker' box).
|
|
|
|
app1 (NEW, netcup RS 4000 G12) -- AI / ML + automation
|
|
- LiteLLM (admin-ai), Open WebUI, Ollama (primary), Qdrant, PostgreSQL (LiteLLM backend), Caddy
|
|
- n8n (workflow automation) + its PostgreSQL -- placed here so it can use local Ollama
|
|
- Follows the /docker/<service> install standard (current Hetzner install does NOT).
|
|
|
|
app2 (NEW, netcup RS 4000 G12) -- Infrastructure / WISP / network
|
|
- UNMS/UISP (10-container stack), UCRM, PostgreSQL, RabbitMQ, Netflow, SiriDB, Fluentd
|
|
- UniFi Controller + MongoDB + Postfix
|
|
- Hudu docs + SWAG + PostgreSQL + Redis
|
|
- Traccar (fleet GPS) + Apache frontend
|
|
|
|
app3 (NEW, netcup RS 4000 G12) -- Web hosting
|
|
- All WordPress sites currently on wphost02 (Apex Track Experience + others)
|
|
- MariaDB, Postfix, PHP-FPM
|
|
- New-site provisioning workflow (replacing RunCloud stand-up) lives here.
|
|
|
|
app1-bu (existing, Hetzner CPX11, 5.161.114.8) -- KEEP AS WARM STANDBY
|
|
- Stays at Hetzner. Warm standby clone of Hermes, always on, gateway dormant.
|
|
- Failover: checks Core every 5 min; if Core down, confirms over 2 min then activates gateway.
|
|
|
|
---
|
|
|
|
## Per-Service Migration Map
|
|
|
|
Source -> Destination (from app-inventory.csv "Migrate / New Host" columns):
|
|
|
|
From ai.itpropartner.com (Hetzner CPX41, 178.156.167.181) -> app1
|
|
- LiteLLM -> app1 (reinstall under /docker standard)
|
|
- Open WebUI -> app1 (conversations are in Postgres/DB -- migrate DB, not just app)
|
|
- Ollama (remote) -> app1
|
|
- Qdrant -> app1 (confirm whether actually wired to OWUI/LiteLLM before migrating)
|
|
- Caddy (ai) -> app1
|
|
- PostgreSQL (ai) -> app1 (LiteLLM uses DB method, not config file -- migrate the DB)
|
|
NOTE: This box is at 92% disk -- migrate before it fills. admin-ai must not have downtime;
|
|
stand up new LiteLLM on app1, cut DNS over, then decommission old.
|
|
|
|
From hudu.itpropartner.com (Hetzner CPX21, 178.156.130.130) -> app2
|
|
- Hudu Docs, SWAG, PostgreSQL, Redis, MariaDB, Postfix
|
|
|
|
From unms.forefrontwireless.com (Hetzner CPX21, 5.161.225.131) -> app2
|
|
- UNMS/UISP, UCRM, PostgreSQL, RabbitMQ, Netflow, SiriDB, Fluentd, MariaDB
|
|
- Full WISP management stack -- migrate existing app/service/DB to new host.
|
|
|
|
From unifi (Hetzner CPX21, 178.156.131.57) -> app2
|
|
- UniFi Controller + MongoDB + Postfix
|
|
- MariaDB (UniFi): NOT migrated (N/A).
|
|
|
|
From wphost02 (Hetzner CPX21, 5.161.62.38) -> app3
|
|
- WordPress (Apex) + all other WordPress sites hosted here, MariaDB, Postfix, PHP-FPM
|
|
- All WordPress sites must be transferred. SiteGround SFTP backups already captured (21 sites).
|
|
|
|
From app1.itpropartner.com (Hetzner CPX11, 87.99.144.163) -> app1
|
|
- n8n + PostgreSQL. Migrate automations AND saved credentials.
|
|
- Nginx reverse proxy for n8n. MariaDB (n8n): NOT migrated (N/A).
|
|
|
|
From docker (Hetzner CPX11, 178.156.168.35)
|
|
- Uptime-Kuma -> core (migrate app + data)
|
|
- Nginx Proxy Manager -> NOT migrated (N/A)
|
|
- RustDesk -> NOT migrated (N/A)
|
|
- StrongSwan (Docker) -> NOT migrated (N/A)
|
|
- RunCloud PHP -> NOT migrated (N/A)
|
|
NOTE: 73% disk. Once Uptime-Kuma is moved, this box can be freed.
|
|
|
|
From fleettracker360 (Hetzner CPX11, 178.156.149.32) -> app2
|
|
- Traccar (native Java) + Apache frontend. MariaDB: NOT migrated (N/A).
|
|
|
|
Stays in place (no migration):
|
|
- app1-bu (warm standby) -- remains at Hetzner.
|
|
- Home Router Keepalive / IPsec on app1-bu -- functionality already exists on core.
|
|
|
|
---
|
|
|
|
## Cost Summary (USD, monthly)
|
|
|
|
Current relevant Hetzner spend is being replaced by 3 netcup RS 4000 boxes.
|
|
- app1 (RS 4000): ~$44/mo
|
|
- app2 (RS 4000): ~$44/mo
|
|
- app3 (RS 4000): ~$44/mo
|
|
- Core (RS 2000): ~$24/mo (unchanged)
|
|
- app1-bu (CPX11): ~$8/mo (unchanged, stays at Hetzner)
|
|
|
|
Servers freed at Hetzner after migration: ai, hudu, unms, unifi, wphost02, app1(n8n),
|
|
docker, fleettracker360 (8 boxes) -- decommission after verification, avoiding Hetzner's
|
|
new premium pricing on any future provisioning.
|
|
|
|
---
|
|
|
|
## Migration Sequence (recommended, safety-first)
|
|
|
|
Precondition: full audit GREEN and all backups verified before starting (Germaine's rule).
|
|
Also fix DR-006 (schedule the full Hermes backup) BEFORE decommissioning anything.
|
|
|
|
Phase 0 -- Provision
|
|
1. Order app1, app2, app3 (RS 4000 G12, monthly, Virginia) via netcup SCP.
|
|
2. Base-harden each per server-provisioning-standard (SSH keys, firewall, Docker under /docker).
|
|
|
|
Phase 1 -- app1 (AI/ML) FIRST, because admin-ai is critical
|
|
3. Stand up LiteLLM + Postgres on app1 under /docker standard; import the LiteLLM DB.
|
|
4. Verify Hermes can reach the new admin-ai endpoint (test deepseek-chat) BEFORE cutover.
|
|
5. Move Open WebUI (migrate DB so conversations transfer), Ollama, Qdrant.
|
|
6. Move n8n + Postgres + credentials.
|
|
7. Cut DNS (admin-ai.itpropartner.com, ai.itpropartner.com, app1.itpropartner.com) to app1.
|
|
8. Verify, then decommission old ai + app1(n8n) Hetzner boxes.
|
|
|
|
Phase 2 -- app2 (Infrastructure/WISP)
|
|
9. Migrate UNMS/UISP + UCRM stack, UniFi + MongoDB, Hudu, Traccar (apps + services + DBs).
|
|
10. Cut DNS, verify each, decommission unms/unifi/hudu/fleettracker360.
|
|
|
|
Phase 3 -- app3 (Web)
|
|
11. Replicate WordPress hosting structure; transfer all sites + MariaDB from wphost02.
|
|
12. Build new-site provisioning workflow (RunCloud replacement) on app3.
|
|
13. Cut DNS, verify all sites, decommission wphost02.
|
|
|
|
Phase 4 -- Core cleanup
|
|
14. Migrate Uptime-Kuma to core; free the Hetzner 'docker' box.
|
|
|
|
Phase 5 -- Post-migration audit
|
|
15. Run a full DR audit again; confirm all backups, services, DNS, and failover healthy.
|
|
|
|
---
|
|
|
|
## Open Items / Risks
|
|
|
|
- IAM key lacks s3:CreateBucket -- new S3 buckets (system-configs, docker-volumes) must be
|
|
created in the Wasabi Console before their sync scripts can write (see dr-issue-log DR-011).
|
|
- Qdrant may not currently be wired to Open WebUI / LiteLLM -- confirm before migrating.
|
|
- admin-ai has zero-downtime requirement -- use stand-up-then-cutover, never in-place move.
|
|
- Docker installs on the Hetzner AI box do NOT follow the /docker/<service> standard;
|
|
the new app1 install must follow it.
|