hermes-skills/skills/devops/infrastructure-automation/references/netcup-rs-g12-consolidation.md

# Netcup RS G12 Server Consolidation Plan

## Overview

Replace 9 Hetzner Cloud VPSes (~$150/mo) with 2-4 Netcup RS G12 dedicated servers in Manassas, VA (US East Coast) for ~$55-86/mo with dedicated EPYC cores, DDR5 ECC RAM, NVMe storage, and 2.5Gbps networking.

## Current Estate (Hetzner, ~$150/mo)

| Server | Type | Role | IP |
|--------|------|------|-----|
| `agent` | CPX11 | Hermes, VPN, backups | 5.161.114.8 |
| `wphost02` | CPX21 | WordPress (RunCloud) | 5.161.62.38 |
| `unms.forefrontwireless.com` | CPX21 | UNMS (UISP) | 5.161.225.131 |
| `unifi` | CPX21 | UniFi Controller | 178.156.131.57 |
| `hudu.itpropartner.com` | CPX21 | Hudu docs | 178.156.130.130 |
| `fleettracker360` | CPX11 | Traccar GPS | 178.156.149.32 |
| `docker` | CPX11 | Docker host (N8N, Mautic) | (internal) |
| `ai.itpropartner.com` | CPX41 | Open WebUI, LiteLLM | 178.156.167.181 |
| `app1.itpropartner.com` | CPX11 | Apps/API | 87.99.144.163 |

## Proposed Architecture — 4-Box Layout (User-Approved)

The user prefers **multiple isolated boxes** (not a single consolidated server) for fault isolation. One box goes down, others keep running. Acceptable: app1+app4 can share if needed for cost optimization.

### Box Layout

| Box | DNS | Plan | Resources | Apps | Monthly (Manassas) |
|-----|-----|------|-----------|------|--------------------|
| **app1** | portal.itpropartner.com | RS 1000 G12 | 4 core, 8GB, 256GB | IT Pro Partner Portal | ~€14.55 (~$16) |
| **app2** | ai.itpropartner.com, admin-ai, ollama.itpropartner.com | RS 2000 G12 | 8 core, 16GB, 512GB | Open WebUI, LiteLLM, Ollama | ~€25.51 (~$27) |
| **app3** | auto.itpropartner.com, marketing, docs, gps, unifi, unms | RS 2000 G12 | 8 core, 16GB, 512GB | N8N, Mautic, Hudu, Traccar, UniFi, UISP | ~€25.51 (~$27) |
| **app4** | WordPress websites | RS 1000 G12 | 4 core, 8GB, 256GB | Nginx + PHP-FPM + MySQL | ~€14.55 (~$16) |
| **Total** | | | | | **~€80/mo ($86)** |

### Alternative: 2-Box Lean (app1+app4 together)

| Box | Plan | Cost |
|-----|------|------|
| **app1+app4** (shared RS 1000) | RS 1000 G12 | ~$16 |
| **app2+app3** (shared RS 2000) | RS 2000 G12 | ~$27 |
| **Total** | | **~$43/mo** |

### Alternative: Single RS 4000 (user rejected — want isolation)

An RS 4000 G12 (12 core, 32GB, 1TB NVMe, ~€40/mo) can run everything. User rejected this approach — prefers multiple boxes so a failure takes down only one service group.

## Netcup vs Hetzner Comparison (July 2026)

### Dedicated Servers — The Real Need

IT Pro Partner needs US servers (all customers US-based). Hetzner has NO US datacenters for cloud servers — only Germany/Finland. Netcup has Manassas, VA.

| Spec | Netcup RS 2000 G12 | Hetzner CCX23 |
|------|-------------------|---------------|
| **CPU** | **8 dedicated** EPYC 9645 (Zen 5) | 4 dedicated EPYC |
| **RAM** | 16 GB DDR5 ECC | 16 GB |
| **Storage** | **512 GB NVMe** | 160 GB NVMe |
| **Network** | **2.5 Gbps** (dedicated bandwidth) | 10 Gbps (shared) |
| **Traffic** | Flat (3TB/24h soft cap ≈ 90 TB/mo) | 20 TB |
| **US Location** | **✅ Manassas, VA** | ❌ No |
| **12-mo price** | €21.22/mo | €85.99/mo |
| **Monthly billing** | **~€25.51/mo (~$27)** | ~€85.99/mo |
| **Cores per euro** | **0.37 cores/€** | 0.05 cores/€ |
| **GB storage per €** | **23.8 GB/€** | 1.9 GB/€ |

Netcup is **3.4x cheaper** with double the cores and triple the storage — and actually in the US.

### VPS Lineup (for comparison)

| Spec | Netcup VPS 2000 G12 | Netcup VPS 4000 G12 | Hetzner CX43 |
|------|-------------------|-------------------|--------------|
| vCores | **8 vCore** (EPYC 9645) | **12 vCore** | 8 shared |
| RAM | 16 GB DDR5 ECC | 32 GB DDR5 ECC | 16 GB |
| Storage | **512 GB NVMe** | **1 TB NVMe** | 160 GB |
| Network | 2.5 Gbps | 2.5 Gbps | 10 Gbps (shared) |
| US Location | ✅ | ✅ | ❌ |
| Monthly (Manassas) | ~€19.08 (~$20) | ~€30.14 (~$32) | ~€15.99 |
| Notes | Shared vCPU, no dedicated guarantee | Shared vCPU, no dedicated guarantee | Grandfathered price, shared + throttled |

**Note:** VPS pricing at Netcup is competitive but uses **shared vCPU** — the RS (dedicated) line costs ~€6 more per tier for guaranteed cores. For production workloads, the RS line is a better choice.

### Verdict

| Factor | Winner |
|--------|--------|
| **Price per dedicated core** | Netcup (8x more cores/€) |
| **US datacenter** | Netcup (only option) |
| **Storage** | Netcup (12x more GB/€) |
| **Monthly billing flexibility** | Both offer month-to-month |
| **Cloud API polish** | Hetzner (snapshots-as-service) |
| **Hourly billing** | Hetzner (more flexible) |
| **ARM instances** | Hetzner (CAX line) |

For IT Pro Partner: **Netcup RS G12 in Manassas**. No contest.

## Migration Phases

### Phase 1: Provision Netcup (Day 1)

1. Order RS 2000 G12 (app2) + RS 2000 G12 (app3) in Manassas — 1-month billing
2. Add RS 1000 G12 (app1+app4) optionally later
3. Base OS: Ubuntu 24.04 LTS per box
4. Deploy wisp_rsa SSH key to each
5. Establish L2TP/IPsec VPN from app3 (management hub)
6. Point Cloudflare DNS at new IPs via Cloudflare API
7. Cancel RunCloud (~€15-20/mo savings) after WordPress migrated

### Phase 2: Move Hermes + VPN (Day 1-2)

1. Install Hermes on app3
2. Transfer profiles, skills, sessions from old `agent` box
3. Point Telegram bot webhook at new Hermes instance
4. Keep old agent running 48h as rollback

### Phase 3: Move Apps (Day 2-5)

1. **Docker containers** (N8N, Mautic) → app3 — docker save/load + env vars
2. **Traccar** → app3 — PostgreSQL dump + restore
3. **UniFi Controller** → app3 — backup export from old, import + readopt devices
4. **UNMS/UISP** → app3 — MongoDB dump + Ubiquiti migration tools
5. **Hudu** → app3 — Docker compose transfer + PostgreSQL restore
6. **Open WebUI + Ollama + LiteLLM** → app2 — model volumes, chroma DB, proxy config
7. **WordPress** → app4 — WP-CLI DB dump + rsync wp-content
8. **Portal** → app1 — fresh FastAPI project deployment

### Phase 4: Decommission (Day 7+)

After 7 days with zero issues:
1. Cancel Hetzner (save ~$150/mo)
2. Cancel RunCloud (save ~€15-20/mo)
3. Update all DNS records
4. Final S3 backup before teardown

## What We Need From User

| Item | When | Blocking |
|------|------|----------|
| Netcup account + order in Manassas | Phase 1 | Yes — can't start without servers |
| SSH key (wisp_rsa already exists) | Phase 1 | Deployable |
| Cloudflare API token (Zone:DNS:Edit) | Phase 1 | Replaces RunCloud DNS |
| Hudu API key + URL | Phase 3 | For docs sync |
| UniFi API key | Phase 3 | For monitoring |
| SyncroMSP API key | Phase 3 | For ticketing |
| RingLogix OAuth | Future | VoIP provisioning |
| Stripe/Xero API keys | Future | Billing integration |

## Per-Box Docker Compose Stack Notes

### app1 — Portal
- FastAPI + HTMX + Tailwind
- PostgreSQL (portal DB)
- Redis (sessions/cache)
- Nginx (reverse proxy + SSL)

### app2 — AI
- Open WebUI (Docker)
- Ollama (Docker + model volumes under `/opt/ollama`)
- LiteLLM (Docker config with `config.yaml`)
- Nginx (reverse proxy + SSL per subdomain)

### app3 — Apps (management hub)
- N8N (Docker + SQLite or PostgreSQL for workflow state)
- Mautic (Docker + MySQL/MariaDB)
- Hudu (Docker + PostgreSQL)
- Traccar (Docker + PostgreSQL)
- UniFi Controller (Docker, needs `--net=host` for device adoption)
- UISP (Docker + MongoDB)
- StrongSwan + xl2tpd (VPN to home CCR)
- Nginx (reverse proxy per subdomain)

### app4 — WordPress
- Nginx + PHP-FPM
- MariaDB
- certbot or Cloudflare origin certs
- Redis object cache (optional)
- WP-CLI for backup/sync

## RunCloud Replacement Details

RunCloud only backs up web apps — it has no reach into UNMS, UniFi, Traccar, Hudu, Docker containers, or anything that isn't a PHP web app. The S3 pipeline already built handles everything:

| RunCloud Feature | Replacement |
|---|---|
| WordPress Nginx/PHP config | Manual nginx config + certbot |
| Staging sites | `wp-cli` clone + DB dump |
| SSL certs | Certbot cron or Cloudflare edge certs |
| Cloudflare DNS | Cloudflare v4 API (portal will handle this) |
| Backups (web apps only) | S3 pipeline covers ALL apps, not just web |
| Monitoring | Beszel on the box |

Savings: ~€15-20/mo
Prerequisite: Cloudflare API token with Zone:DNS:Edit permission

## Portal Viewer/Observer Role

The portal will need RBAC for scenarios like:
- Customer employees who need to see phone logs but NOT modify anything
- Read-only access to call recordings, ticket history, billing

Role hierarchy:
1. **Admin** — full access, billing, user management, server controls
2. **Manager** — customer CRUD, team management, device operations
3. **Agent** — ticket handling, phone system ops, read customer data
4. **Viewer** — read-only: phone logs, call recordings, ticket history. No create/edit/delete. No server controls (reboot/restart). No billing/admin access.