Initial resurrection kit — 81 scripts, 62 references, configs, systemd units, crons, Docker compose files, Caddy config, master README
This commit is contained in:
root
@@ -0,0 +1,170 @@
|
||||
# Hetzner to netcup Migration Plan
|
||||
|
||||
Author: Sho'Nuff
|
||||
Last updated: 2026-07-09
|
||||
Status: PLAN (not yet executed)
|
||||
Sources: app-inventory.csv (Germaine's Migrate/New Host columns), hetzner-server-audit.md,
|
||||
server-provisioning-standard, ops-portal-and-collector skill.
|
||||
|
||||
---
|
||||
|
||||
## Objective
|
||||
|
||||
Consolidate the ITPP server fleet by moving apps/services off Hetzner (whose new-server
|
||||
pricing is now premium) onto netcup root (dedicated) servers. Goal: fewer servers, lower
|
||||
recurring cost, and NOT an all-in-one design -- workloads are spread so a single outage
|
||||
does not take down everything and everyone.
|
||||
|
||||
Guiding constraints from Germaine (Jul 9, 2026):
|
||||
- Cost savings is good but NOT the priority; plenty of overhead for growth on each box.
|
||||
- Do not put everything on one server (blast-radius isolation).
|
||||
- Keep the MikroTik router API functionality.
|
||||
- Keep Ollama on core as a local fallback if admin-ai connectivity is lost.
|
||||
- admin-ai (LiteLLM) must stay online at all times -- it is where Hermes gets its model.
|
||||
- app1-bu stays as the warm standby at Hetzner (do NOT migrate).
|
||||
|
||||
---
|
||||
|
||||
## Target Architecture
|
||||
|
||||
Base hardware standard for all NEW netcup servers: RS 4000 G12
|
||||
- 12 dedicated EPYC 9645 cores, 32 GB DDR5 ECC RAM, 1 TB NVMe
|
||||
- ~$44/mo (monthly billing, Manassas VA / Virginia hosting)
|
||||
- Monthly billing chosen (NOT annual).
|
||||
|
||||
Server roles after migration:
|
||||
|
||||
Core (existing, netcup RS 2000 G12, 152.53.192.33) -- KEEP AS-IS
|
||||
- Role: Ops hub. Hermes, Caddy reverse proxy, ops portal, Docker services
|
||||
(Twenty CRM, Vaultwarden, DocuSeal, SearXNG), Shark Game, Prometheus, Ollama (fallback), VPN.
|
||||
- Also receives: Uptime-Kuma (migrated from Hetzner 'docker' box).
|
||||
|
||||
app1 (NEW, netcup RS 4000 G12) -- AI / ML + automation
|
||||
- LiteLLM (admin-ai), Open WebUI, Ollama (primary), Qdrant, PostgreSQL (LiteLLM backend), Caddy
|
||||
- n8n (workflow automation) + its PostgreSQL -- placed here so it can use local Ollama
|
||||
- Follows the /docker/<service> install standard (current Hetzner install does NOT).
|
||||
|
||||
app2 (NEW, netcup RS 4000 G12) -- Infrastructure / WISP / network
|
||||
- UNMS/UISP (10-container stack), UCRM, PostgreSQL, RabbitMQ, Netflow, SiriDB, Fluentd
|
||||
- UniFi Controller + MongoDB + Postfix
|
||||
- Hudu docs + SWAG + PostgreSQL + Redis
|
||||
- Traccar (fleet GPS) + Apache frontend
|
||||
|
||||
app3 (NEW, netcup RS 4000 G12) -- Web hosting
|
||||
- All WordPress sites currently on wphost02 (Apex Track Experience + others)
|
||||
- MariaDB, Postfix, PHP-FPM
|
||||
- New-site provisioning workflow (replacing RunCloud stand-up) lives here.
|
||||
|
||||
app1-bu (existing, Hetzner CPX11, 5.161.114.8) -- KEEP AS WARM STANDBY
|
||||
- Stays at Hetzner. Warm standby clone of Hermes, always on, gateway dormant.
|
||||
- Failover: checks Core every 5 min; if Core down, confirms over 2 min then activates gateway.
|
||||
|
||||
---
|
||||
|
||||
## Per-Service Migration Map
|
||||
|
||||
Source -> Destination (from app-inventory.csv "Migrate / New Host" columns):
|
||||
|
||||
From ai.itpropartner.com (Hetzner CPX41, 178.156.167.181) -> app1
|
||||
- LiteLLM -> app1 (reinstall under /docker standard)
|
||||
- Open WebUI -> app1 (conversations are in Postgres/DB -- migrate DB, not just app)
|
||||
- Ollama (remote) -> app1
|
||||
- Qdrant -> app1 (confirm whether actually wired to OWUI/LiteLLM before migrating)
|
||||
- Caddy (ai) -> app1
|
||||
- PostgreSQL (ai) -> app1 (LiteLLM uses DB method, not config file -- migrate the DB)
|
||||
NOTE: This box is at 92% disk -- migrate before it fills. admin-ai must not have downtime;
|
||||
stand up new LiteLLM on app1, cut DNS over, then decommission old.
|
||||
|
||||
From hudu.itpropartner.com (Hetzner CPX21, 178.156.130.130) -> app2
|
||||
- Hudu Docs, SWAG, PostgreSQL, Redis, MariaDB, Postfix
|
||||
|
||||
From unms.forefrontwireless.com (Hetzner CPX21, 5.161.225.131) -> app2
|
||||
- UNMS/UISP, UCRM, PostgreSQL, RabbitMQ, Netflow, SiriDB, Fluentd, MariaDB
|
||||
- Full WISP management stack -- migrate existing app/service/DB to new host.
|
||||
|
||||
From unifi (Hetzner CPX21, 178.156.131.57) -> app2
|
||||
- UniFi Controller + MongoDB + Postfix
|
||||
- MariaDB (UniFi): NOT migrated (N/A).
|
||||
|
||||
From wphost02 (Hetzner CPX21, 5.161.62.38) -> app3
|
||||
- WordPress (Apex) + all other WordPress sites hosted here, MariaDB, Postfix, PHP-FPM
|
||||
- All WordPress sites must be transferred. SiteGround SFTP backups already captured (21 sites).
|
||||
|
||||
From app1.itpropartner.com (Hetzner CPX11, 87.99.144.163) -> app1
|
||||
- n8n + PostgreSQL. Migrate automations AND saved credentials.
|
||||
- Nginx reverse proxy for n8n. MariaDB (n8n): NOT migrated (N/A).
|
||||
|
||||
From docker (Hetzner CPX11, 178.156.168.35)
|
||||
- Uptime-Kuma -> core (migrate app + data)
|
||||
- Nginx Proxy Manager -> NOT migrated (N/A)
|
||||
- RustDesk -> NOT migrated (N/A)
|
||||
- StrongSwan (Docker) -> NOT migrated (N/A)
|
||||
- RunCloud PHP -> NOT migrated (N/A)
|
||||
NOTE: 73% disk. Once Uptime-Kuma is moved, this box can be freed.
|
||||
|
||||
From fleettracker360 (Hetzner CPX11, 178.156.149.32) -> app2
|
||||
- Traccar (native Java) + Apache frontend. MariaDB: NOT migrated (N/A).
|
||||
|
||||
Stays in place (no migration):
|
||||
- app1-bu (warm standby) -- remains at Hetzner.
|
||||
- Home Router Keepalive / IPsec on app1-bu -- functionality already exists on core.
|
||||
|
||||
---
|
||||
|
||||
## Cost Summary (USD, monthly)
|
||||
|
||||
Current relevant Hetzner spend is being replaced by 3 netcup RS 4000 boxes.
|
||||
- app1 (RS 4000): ~$44/mo
|
||||
- app2 (RS 4000): ~$44/mo
|
||||
- app3 (RS 4000): ~$44/mo
|
||||
- Core (RS 2000): ~$24/mo (unchanged)
|
||||
- app1-bu (CPX11): ~$8/mo (unchanged, stays at Hetzner)
|
||||
|
||||
Servers freed at Hetzner after migration: ai, hudu, unms, unifi, wphost02, app1(n8n),
|
||||
docker, fleettracker360 (8 boxes) -- decommission after verification, avoiding Hetzner's
|
||||
new premium pricing on any future provisioning.
|
||||
|
||||
---
|
||||
|
||||
## Migration Sequence (recommended, safety-first)
|
||||
|
||||
Precondition: full audit GREEN and all backups verified before starting (Germaine's rule).
|
||||
Also fix DR-006 (schedule the full Hermes backup) BEFORE decommissioning anything.
|
||||
|
||||
Phase 0 -- Provision
|
||||
1. Order app1, app2, app3 (RS 4000 G12, monthly, Virginia) via netcup SCP.
|
||||
2. Base-harden each per server-provisioning-standard (SSH keys, firewall, Docker under /docker).
|
||||
|
||||
Phase 1 -- app1 (AI/ML) FIRST, because admin-ai is critical
|
||||
3. Stand up LiteLLM + Postgres on app1 under /docker standard; import the LiteLLM DB.
|
||||
4. Verify Hermes can reach the new admin-ai endpoint (test deepseek-chat) BEFORE cutover.
|
||||
5. Move Open WebUI (migrate DB so conversations transfer), Ollama, Qdrant.
|
||||
6. Move n8n + Postgres + credentials.
|
||||
7. Cut DNS (admin-ai.itpropartner.com, ai.itpropartner.com, app1.itpropartner.com) to app1.
|
||||
8. Verify, then decommission old ai + app1(n8n) Hetzner boxes.
|
||||
|
||||
Phase 2 -- app2 (Infrastructure/WISP)
|
||||
9. Migrate UNMS/UISP + UCRM stack, UniFi + MongoDB, Hudu, Traccar (apps + services + DBs).
|
||||
10. Cut DNS, verify each, decommission unms/unifi/hudu/fleettracker360.
|
||||
|
||||
Phase 3 -- app3 (Web)
|
||||
11. Replicate WordPress hosting structure; transfer all sites + MariaDB from wphost02.
|
||||
12. Build new-site provisioning workflow (RunCloud replacement) on app3.
|
||||
13. Cut DNS, verify all sites, decommission wphost02.
|
||||
|
||||
Phase 4 -- Core cleanup
|
||||
14. Migrate Uptime-Kuma to core; free the Hetzner 'docker' box.
|
||||
|
||||
Phase 5 -- Post-migration audit
|
||||
15. Run a full DR audit again; confirm all backups, services, DNS, and failover healthy.
|
||||
|
||||
---
|
||||
|
||||
## Open Items / Risks
|
||||
|
||||
- IAM key lacks s3:CreateBucket -- new S3 buckets (system-configs, docker-volumes) must be
|
||||
created in the Wasabi Console before their sync scripts can write (see dr-issue-log DR-011).
|
||||
- Qdrant may not currently be wired to Open WebUI / LiteLLM -- confirm before migrating.
|
||||
- admin-ai has zero-downtime requirement -- use stand-up-then-cutover, never in-place move.
|
||||
- Docker installs on the Hetzner AI box do NOT follow the /docker/<service> standard;
|
||||
the new app1 install must follow it.
|
||||
Reference in New Issue
Block a user