# IT Pro Partner — Infrastructure Documentation

> **Last Updated:** July 16, 2026
> **Maintainer:** Sho'Nuff

---

## Server Inventory

### Core Server
- **Hostname:** Core
- **IP:** 152.53.192.33
- **Provider:** netcup RS 2000 G12
- **Specs:** 8 vCPU EPYC 9645, 15 GB DDR5 ECC, 512 GB NVMe
- **OS:** Debian 13
- Role: Hermes + Ollama fallback + Portals
- **Key Services:**
  - Hermes Agent (Telegram + cron, 22 cron jobs)
  - Caddy reverse proxy (11 domains, auto-TLS)
  - Ops Portal (FastAPI, port 8090)
  - Prometheus (native, port 9090) + Grafana (native, port 3002)
  - Uptime Kuma (Docker, port 3001) — 9+ monitors
  - Vaultwarden (Docker, port 8080) — vault.iamgmb.com
  - Super Search MCP v2.0.0 (FastMCP, port 8899)
  - Twenty CRM (Docker) — crm.debtrecoveryexperts.com
  - DocuSeal (Docker, port 3000) — sign.itpropartner.com
  - SearXNG (Docker, port 8888)
  - Komodo (Docker, port 9120)
  - Tailscale, StrongSwan, WireGuard (home CCR tunnel)
  - Redis cache
  - Ollama (systemd, port 11434) — lightweight fallback LLM

### App1 Server
- **Hostname:** app1
- **IP:** 152.53.36.131
- **Provider:** netcup RS 4000 G12
- **Specs:** 12 vCPU EPYC 9645, 32 GB DDR5 ECC, 1 TB NVMe
- **OS:** Debian 13
- **Role:** AI/Service Hub
- **Key Services:**
  - Ollama (Docker, port 11434)
  - Open WebUI (Docker, port 3000) — ai.itpropartner.com
  - n8n + Postgres (Docker, port 5678) — n8n.itpropartner.com
  - Caddy (systemd, 80/443)
  - LiteLLM (Docker) + Postgres — admin-ai.itpropartner.com
  - 5 MCP servers: Super Search (:8899), Browser (:8901), Filesystem (:8900), Email (:8902), Git (:8903)

### App2 Server
- **Hostname:** app2
- **IP:** 152.53.39.202
- **Provider:** netcup RS 4000 G12
- **Specs:** 12 vCPU EPYC 9645, 32 GB DDR5 ECC, 1 TB NVMe
- **OS:** Debian 13
- **Role:** Infrastructure Server
- **Key Services:**
  - Traccar GPS (Docker, port 8082 + 5000-5150) — fleettracker360.com
  - UniFi Controller (Docker, port 8443) — unifi.itpropartner.com
  - UNMS/UISP (10 Docker containers) — unms.forefrontwireless.com
  - Hudu (Docker) — hudu.itpropartner.com
  - Caddy (4 domains)

### App3 Server
- **Hostname:** app3
- **IP:** 152.53.241.111
- **Provider:** netcup RS 4000 G12
- **Specs:** 12 vCPU EPYC 9645, 32 GB DDR5 ECC, 1 TB NVMe
- **OS:** Debian 13
- **Role:** Web Hosting
- **Key Services:**
  - CloudPanel CE — panel.itpropartner.com
  - Nginx (80/443) + Percona MySQL 8.4 + PHP 8.3
  - voipsimplicity.com (WordPress, migrated Jul 16)
  - my.voipsimplicity.com (PHP customer portal)

### Core-BU (Warm Standby)
- **Hostname:** core-bu
- **IP:** 5.161.114.8
- **Provider:** Hetzner CPX11 (planned upgrade: CPX31)
- **Specs:** 2 vCPU, 2 GB RAM, 40 GB SSD
- **Role:** Warm standby — auto-failover if Core down
- **Watchdog:** 5-min check, 4-cycle confirmation, S3 sync every 10 min

### Other Active Servers
- **old-ai:** 178.156.167.181 (Hetzner CPX41) — LiteLLM + Open WebUI + Ollama — **deprecated, migrating to app1**
- **wphost02:** 5.161.62.38 (Hetzner CPX21) — WordPress host — **migrating to app3**
- **Tony Hermes:** 87.99.159.142 (Hetzner CPX21) — Anthony's personal Hermes

---

## Domain / DNS Map

### ⚠️ itpropartner.com — SiteGround Nameservers Only

`itpropartner.com` uses **SiteGround nameservers** exclusively. A Cloudflare zone exists (`0dc20632…`) but is NOT authoritative — records created there silently fail. All `*.itpropartner.com` changes must be manual through SiteGround panel.

| Domain | IP | Server | Service |
|---|---|---|---|
| core.itpropartner.com | 152.53.192.33 | Core | Static API docs |
| ops.itpropartner.com | 152.53.192.33 | Core | Ops dashboard |
| sign.itpropartner.com | 152.53.192.33 | Core | DocuSeal |
| uptimekuma.itpropartner.com | 152.53.192.33 | Core | Monitoring |
| hudu.itpropartner.com | 152.53.39.202 | app2 | IT documentation |
| unifi.itpropartner.com | 152.53.39.202 | app2 | UniFi controller |
| panel.itpropartner.com | 152.53.241.111 | app3 | CloudPanel CE |
| ai.itpropartner.com | 152.53.36.131 | app1 | Open WebUI |
| n8n.itpropartner.com | 152.53.36.131 | app1 | n8n automation |
| admin-ai.itpropartner.com | 152.53.36.131 | app1 | LiteLLM (currently → old-ai) |

### Cloudflare-Managed Domains

| Domain | IP | Server | Service |
|---|---|---|---|
| fleettracker360.com | 152.53.39.202 | app2 | Fleet tracking |
| gps.fleettracker360.com | 152.53.39.202 | app2 | GPS devices |
| voipsimplicity.com | 152.53.241.111 | app3 | VoIP site |
| support.voipsimplicity.com | 152.53.241.111 | app3 | Support portal |
| portal.debtrecoveryexperts.com | 152.53.192.33 | Core | DRE portal |
| crm.debtrecoveryexperts.com | Cloudflare Access | — | DRE CRM |

---

## Backup Pipeline

| Backup | Schedule | Target | Purpose |
|---|---|---|---|
| hermes-live-sync | Every 15 min | s3://hermes-vps-backups/live/ | Live state sync |
| hermes-full-backup | Daily 1 AM | s3://hermes-vps-backups/hermes-full-backup/ | Full Hermes backup |
| home-router-backup | Daily 6 AM | s3://mikrotik-ccr-backups/ | CCR config |
| root-essentials-backup | Daily 3 AM | S3 | /root essentials |
| docker-volume-sync | Daily 3 AM | S3 | Docker volumes |
| system-config-sync | Daily 4 AM | S3 | System configs |
| unms-backup-sync | Daily 6 AM | s3://hermes-vps-backups/unms-backups/ | UNMS data |
| unifi-backup-sync | Daily 2 AM | s3://hermes-vps-backups/unifi-backups/ | UniFi configs |
| hudu-backup | Daily 3 AM | s3://hermes-vps-backups/hudu-backups/ | Hudu (30-day ret) |
| wphost02-mariadb-dump | Daily 2:30 AM | S3 | WordPress DB |
| warm-standby-sync | Every 10 min | core-bu ← S3 | DR readiness |

---

## Caddy Configuration

### Core (152.53.192.33)
```
sign.itpropartner.com    → :3000 (DocuSeal)
core.itpropartner.com    → static files
crm.debtrecoveryexperts.com → :3001 (Twenty)
portal.debtrecoveryexperts.com → static
internal.debtrecoveryexperts.com → static
app.itpropartner.com     → static + :8082
shark.itpropartner.com   → static + :8083
shark.iamgmb.com         → static + :8083
ops.itpropartner.com     → :8090 + static
uptimekuma.itpropartner.com → :3001
vault.iamgmb.com         → :8080
```

### App1 (152.53.36.131)
```
ai.itpropartner.com      → :3000 (Open WebUI)
n8n.itpropartner.com     → :5678 (n8n)
```

### App2 (152.53.39.202)
```
hudu.itpropartner.com    → Hudu internal
gps.fleettracker360.com  → :8082 (Traccar)
unms.forefrontwireless.com → UNMS Nginx
unifi.itpropartner.com   → :8443 (UniFi)
```

---

## Service Access

| Service | URL | Location | Auth |
|---|---|---|---|
| Open WebUI | https://ai.itpropartner.com | app1 | Chat UI |
| Open WebUI Admin | https://admin-ai.itpropartner.com/ui | app1 | user: admin, pw: LITELLM_MASTER_KEY |
| Ops Portal | https://ops.itpropartner.com | Core | Internal dashboard |
| Uptime Kuma | https://uptimekuma.itpropartner.com | Core | Service monitoring |
| Vaultwarden | https://vault.iamgmb.com | Core | Password vault |
| CloudPanel | https://panel.itpropartner.com | app3 | SQLite DB at /home/clp/htdocs/app/data/db.sq3 |
| Traccar | https://gps.fleettracker360.com | app2 | GPS fleet tracking |
| UniFi | https://unifi.itpropartner.com | app2 | Network controller |
| UNMS | https://unms.forefrontwireless.com | app2 | WISP management |
| Hudu | https://hudu.itpropartner.com | app2 | IT documentation |
| n8n | https://n8n.itpropartner.com | app1 | Automation |
| CRM (DRE) | https://crm.debtrecoveryexperts.com | Cloudflare Access | TwentyCRM |

### MCP Access (from Open WebUI)

All 5 MCP servers run on **app1** and are wired into Open WebUI at https://ai.itpropartner.com:

| MCP | Port | Tools |
|---|---|---|
| Super Search | :8899 | web_search, web_extract, web_search_premium |
| Browser | :8901 | browser_navigate, browser_snapshot, browser_click, browser_type, browser_console |
| Filesystem | :8900 | read_file, write_file, search_files, list_dir, file_info |
| Email | :8902 | search_emails, send_email, get_email |
| Git/Gitea | :8903 | clone, commit, push, pull |

---

## SSH Access

- **Key:** `itpp-infra` (deployed to all servers)
- **User:** `ippadmin` (sudo privileges)
- **Root SSH:** Enabled on app1, app2, app3 (key-only exception per provisioning standard)
- **Core SSH:** `ssh -i /root/.ssh/itpp-infra root@152.53.192.33`

---

## Firewall

UFW is enabled on all servers. Standard rules:
- **Core:** 22, 80, 443, 3000, 3001, 8080, 8082, 8443, 9090, 3002
- **app1:** 22, 80, 443, 3000, 5678, 8899, 8900, 8901, 8902, 8903
- **app2:** 22, 80, 443, 3000, 8080, 8082, 8089, 8443, 8843, 3478, 10001, 5000:5150
- **app3:** 22, 80, 443, 8443

---

## SSL

All SSL certificates issued via Let's Encrypt through Caddy. All certs auto-renew. No manual management needed.

---

## Notes

- **DNS trap:** `itpropartner.com` uses SiteGround nameservers. Cloudflare zone is NOT authoritative. Always verify with `dig NS domain.com` before creating records.
- **SiteGround:** No API access. All DNS changes are manual through SiteGround panel.
- **Provider diversity:** core-bu stays at Hetzner specifically so a netcup outage can't kill both Core and standby simultaneously.
- **app3 MySQL:** Root password at `MOQMINFQIhklM0AF`, accessible via 127.0.0.1:3306.
- **CloudPanel:** SQLite DB at `/home/clp/htdocs/app/data/db.sq3` — users live here, not in MySQL.