# itpp-infrastructure


> **Last Updated:** July 17, 2026
> **Maintainer:** Sho'Nuff

---

## Server Inventory

### Core Server
- **Hostname:** Core
- **IP:** 152.53.192.33
- **Provider:** netcup RS 2000 G12
- **Specs:** 8 vCPU EPYC 9645, 15 GB DDR5 ECC, 512 GB NVMe
- **OS:** Debian 13
- **Role:** Hermes + Portals
- **Key Services:**
  - Hermes Agent (Telegram + cron, 22 cron jobs)
  - Caddy reverse proxy (12 domains, auto-TLS)
  - Ops Portal (FastAPI, port 8090)
  - Prometheus (native, port 9090) + Grafana (native, port 3002)
  - Uptime Kuma (Docker, port 3001) — 9+ monitors
  - Vaultwarden (Docker, port 8080) — vault.iamgmb.com
  - Twenty CRM (Docker) — crm.debtrecoveryexperts.com
  - DocuSeal (Docker, port 3000) — sign.core.itpropartner.com
  - SearXNG (Docker, port 8888)
  - Komodo (Docker, port 9120)
  - Tailscale, StrongSwan, WireGuard (home CCR tunnel 10.77.0.0/24)
  - Redis cache

### App1 Server
- **Hostname:** app1
- **IP:** 152.53.36.131
- **Provider:** netcup RS 4000 G12
- **Specs:** 12 vCPU EPYC 9645, 32 GB DDR5 ECC, 1 TB NVMe
- **OS:** Debian 13
- **Role:** AI/Service Hub
- **Key Services:**
  - Open WebUI (Docker, port 3000) — ai.itpropartner.com
  - n8n + Postgres (Docker, port 5678) — n8n.itpropartner.com
  - LiteLLM (Docker) + Postgres — admin-ai.itpropartner.com
  - Mattermost Team Edition (Docker, port 8065) — noc.itpropartner.com
  - Caddy (systemd, 80/443)
  - 4 MCP servers: Browser (:8901), Filesystem (:8900), Email (:8902), Git (:8903)
  - Super Search MCP (systemd, port 8899)

### App2 Server
- **Hostname:** app2
- **IP:** 152.53.39.202
- **Provider:** netcup RS 4000 G12
- **Specs:** 12 vCPU EPYC 9645, 32 GB DDR5 ECC, 1 TB NVMe
- **OS:** Debian 13
- **Role:** Infrastructure Server
- **Key Services:**
  - Traccar GPS (Docker, port 8082 + 5000-5150) — fleettracker360.com
  - UniFi Controller (Docker, port 8443) — unifi.itpropartner.com
  - UNMS/UISP (10 Docker containers) — unms.forefrontwireless.com
  - Hudu (Docker) — hudu.itpropartner.com
  - Caddy (4 domains)

### App3 Server
- **Hostname:** app3
- **IP:** 152.53.241.111
- **Provider:** netcup RS 4000 G12
- **Specs:** 12 vCPU EPYC 9645, 32 GB DDR5 ECC, 1 TB NVMe
- **OS:** Debian 13
- **Role:** Web Hosting + Backup Restore
- **Key Services:**
  - CloudPanel CE — panel.itpropartner.com
  - Nginx (80/443) + Percona MySQL 8.4 + PHP 8.3
  - Backup Restore System (Flask, port 8090) — my.itpropartner.com/backups
  - WordPress sites (7 migrated from wphost02, all live):
    - debtreecoveryexperts.com, boxpilotlogistics.com, iamgmb.com
    - katiewattsdesign.com, vigilanttac.com, apextrackexperience.com
    - mainwp.itpropartner.com, voipsimplicity.com, my.voipsimplicity.com
  - Daily snapshots: 1 AM + 1 PM, 60-day retention, /opt/backup-restore/snapshots

### Core-BU (Warm Standby)
- **Hostname:** core-bu
- **IP:** 5.161.225.131
- **Provider:** Hetzner CPX21
- **Specs:** 3 vCPU, 4 GB RAM, 80 GB SSD
- **Role:** Warm standby — auto-failover if Core down
- **Watchdog:** 5-min check, 4-cycle confirmation, S3 sync every 10 min

### Legacy / Decommissioned
- **old-ai:** 178.156.167.181 (Hetzner CPX41) — **decommissioned** (LiteLLM migrated to app1)
- **old app1:** 87.99.144.163 (Hetzner CPX11) — **deleted**
- **wphost02:** 5.161.62.38 (Hetzner CPX21) — **migrated to app3**
- **Ollama:** Removed from Core (systemd) and app1 (Docker) Jul 17

---

## Model Fallback Chain

All providers use direct API keys. GPT-5.5 quality survives through admin-ai → OpenRouter, then degrades through DeepSeek → Gemini → Grok.

| # | Model | Provider | Gateway |
|---|---|---|---|
| Primary | GPT-5.5 | admin-ai | Self-hosted LiteLLM (app1) |
| Fallback 1 | GPT-5.5 | OpenRouter | openrouter.ai |
| Fallback 2 | DeepSeek v4 Pro | DeepSeek | api.deepseek.com |
| Fallback 3 | Gemini 3.5 Flash | Google | generativelanguage.googleapis.com |
| Fallback 4 | Grok 4.5 | xAI | api.x.ai |

**Credits (Jul 17):** DeepSeek $58, OpenRouter ~$30 remaining, OpenAI/xAI/Google on pay-as-you-go
**Health check:** Daily 8 AM cron (`model-usage-check`)

---

## Domain / DNS Map

### ⚠️ itpropartner.com — SiteGround Nameservers Only

`itpropartner.com` uses **SiteGround nameservers** exclusively. A Cloudflare zone exists (`0dc20632…`) but is NOT authoritative — records created there silently fail. All `*.itpropartner.com` changes must be manual through SiteGround panel.

| Domain | IP | Server | Service |
|---|---|---|---|
| core.itpropartner.com | 152.53.192.33 | Core | Landing page + Grafana link |
| ops.itpropartner.com | 152.53.192.33 | Core | Ops dashboard |
| sign.core.itpropartner.com | 152.53.192.33 | Core | DocuSeal |
| uptimekuma.itpropartner.com | 152.53.192.33 | Core | Uptime monitoring |
| gps.fleettracker360.com | 152.53.192.33 | Core | Traccar HTTPS proxy → app2 |
| my.itpropartner.com | 152.53.192.33 | Core | Customer portal hub |
| hudu.itpropartner.com | 152.53.39.202 | app2 | IT documentation |
| unifi.itpropartner.com | 152.53.39.202 | app2 | UniFi controller |
| panel.itpropartner.com | 152.53.241.111 | app3 | CloudPanel CE |
| ai.itpropartner.com | 152.53.36.131 | app1 | Open WebUI |
| n8n.itpropartner.com | 152.53.36.131 | app1 | n8n automation |
| admin-ai.itpropartner.com | 152.53.36.131 | app1 | LiteLLM |

### Cloudflare-Managed Domains

| Domain | IP | Server | Service |
|---|---|---|---|
| fleettracker360.com | Cloudflare | app2 | Fleet tracking website |
| gps.fleettracker360.com | Cloudflare → Core | Core → app2 | Traccar devices |
| voipsimplicity.com | Cloudflare | app3 | VoIP marketing site |
| my.voipsimplicity.com | Cloudflare | app3 | VoIP customer portal |
| portal.debtrecoveryexperts.com | 152.53.192.33 | Core | DRE portal |
| crm.debtrecoveryexperts.com | Cloudflare Access | — | DRE CRM |
| vault.iamgmb.com | 152.53.192.33 | Core | Vaultwarden |
| sign.iamgmb.com | 152.53.192.33 | Core | Document signing |
| shark.iamgmb.com | 152.53.192.33 | Core | Shark game |

### DNS PENDING (create at SiteGround)

| Subdomain | → IP | Service |
|---|---|---|
| vault.itpropartner.com | 152.53.36.131 | Vaultwarden (after migration) |
| status.itpropartner.com | 152.53.192.33 | Public status page |

---

## Backup Pipeline

| Backup | Schedule | Target | Purpose |
|---|---|---|---|
| hermes-live-sync | Every 15 min | s3://hermes-vps-backups/live/ | Live state sync |
| hermes-full-backup | Daily 1 AM | s3://hermes-vps-backups/hermes-full-backup/ | Full Hermes backup |
| home-router-backup | Daily 6 AM | s3://mikrotik-ccr-backups/ | CCR config |
| root-essentials-backup | Daily 3 AM | S3 | /root essentials |
| docker-volume-sync | Daily 3 AM | S3 | Docker volumes |
| system-config-sync | Daily 4 AM | S3 | System configs |
| unms-backup-sync | Daily 6 AM (Core) | s3://hermes-vps-backups/unms-backups/ | UNMS data (pulled from app2) |
| unifi-backup-sync | Daily 2 AM (Core) | s3://hermes-vps-backups/unifi-backups/ | UniFi configs (pulled from app2) |
| hudu-backup | Daily 7 AM | s3://hermes-vps-backups/hudu/backups/ | Hudu volume dump |
| gitea-backup | Daily 8 AM | s3://hermes-vps-backups/gitea/daily/ | Gitea repos |
| app1-backup | Daily 2 AM | s3://hermes-vps-backups/app1/ | LiteLLM, n8n, OpenWebUI, MCP, Mattermost |
| app2-backup | Daily 2:30 AM | s3://hermes-vps-backups/app2/ | Traccar, Gitea, Hudu, UNMS, UniFi |
| app3-backup | Daily 3 AM | s3://hermes-vps-backups/app3/ | CloudPanel, MySQL, WordPress |
| wphost02-backup | Daily 5 AM | s3://hermes-vps-backups/wphost02-backup/ | Webapps + MySQL |
| warm-standby-sync | Every 10 min | core-bu ← S3 | DR readiness |

---

## Caddy Configuration

### Core (152.53.192.33)

```
core.itpropartner.com           → static files
sign.core.itpropartner.com      → localhost:3000 (DocuSeal)
ops.itpropartner.com            → 127.0.0.1:8090 + static
uptimekuma.itpropartner.com     → 127.0.0.1:3001 (Uptime Kuma)
gps.fleettracker360.com         → app2:8082 (Traccar)
my.itpropartner.com             → static files
portal.debtrecoveryexperts.com  → static files
vault.iamgmb.com                → localhost:8080 (Vaultwarden)
sign.iamgmb.com                 → 127.0.0.1:8090
shark.iamgmb.com                → static + :8083
```

### App1 (152.53.36.131)

```
ai.itpropartner.com             → :3000 (Open WebUI)
n8n.itpropartner.com            → :5678 (n8n)
admin-ai.itpropartner.com       → :4000 (LiteLLM)
app1.itpropartner.com           → static response
```

### App2 (152.53.39.202)

```
hudu.itpropartner.com           → Hudu internal
gps.fleettracker360.com         → :8082 (Traccar)
unms.forefrontwireless.com      → UNMS Nginx
unifi.itpropartner.com          → :8443 (UniFi)
```

---

## Service Access

| Service | URL | Location | Auth |
|---|---|---|---|
| Open WebUI | https://ai.itpropartner.com | app1 | Chat UI |
| Open WebUI Admin | https://admin-ai.itpropartner.com/ui | app1 | user: admin, pw: LITELLM_MASTER_KEY |
| Ops Portal | https://ops.itpropartner.com | Core | Internal dashboard |
| Grafana | http://core.itpropartner.com:3002 | Core | admin/admin |
| Uptime Kuma | https://uptimekuma.itpropartner.com | Core | Service monitoring |
| Vaultwarden | https://vault.iamgmb.com | Core | Password vault |
| CloudPanel | https://panel.itpropartner.com | app3 | user: gmb / SQLite auth |
| Traccar | https://gps.fleettracker360.com | app2 | GPS fleet tracking |
| UniFi | https://unifi.itpropartner.com | app2 | Network controller |
| UNMS | https://unms.forefrontwireless.com | app2 | WISP management |
| Hudu | https://hudu.itpropartner.com | app2 | IT documentation |
| n8n | https://n8n.itpropartner.com | app1 | Automation |
| CRM (DRE) | https://crm.debtrecoveryexperts.com | Cloudflare Access | TwentyCRM |

### MCP Access (from Open WebUI)

| MCP Server | Location | Port | Tools |
|---|---|---|---|
| Super Search | app1 | :8899 | 10 tools — web_search, web_extract, person_search, email_search, phone_search, etc. |
| Browser | app1 | :8901 | browser_navigate, browser_snapshot, browser_click, browser_type, browser_console |
| Filesystem | app1 | :8900 | read_file, write_file, search_files, list_dir, file_info |
| Email | app1 | :8902 | search_emails, send_email, get_email |
| Git/Gitea | app1 | :8903 | clone, commit, push, pull |

---

## SSH Access

- **Key:** `itpp-infra` (deployed to all servers)
- **User:** `ippadmin` (sudo privileges)
- **Root SSH:** Enabled on app1, app2, app3 (key-only exception per provisioning standard)
- **Core SSH:** `ssh -i /root/.ssh/itpp-infra root@152.53.192.33`

---

## Firewall

UFW is enabled on all servers. Standard rules:
- **Core:** 22, 80, 443, 3000, 3001, 3002, 8080, 8082, 8090, 8443, 9090
- **app1:** 22, 80, 443, 3000, 5678, 8899, 8900, 8901, 8902, 8903
- **app2:** 22, 80, 443, 3000, 8080, 8082, 8089, 8443, 8843, 3478, 10001, 5000:5150
- **app3:** 22, 80, 443, 8443

---

## SSL

All SSL certificates issued via Let's Encrypt through Caddy. All certs auto-renew. No manual management needed.

---

## Notes

- **DNS trap:** `itpropartner.com` uses SiteGround nameservers. Cloudflare zone is NOT authoritative. Always verify with `dig NS domain.com` before creating records.
- **SiteGround:** No API access. All DNS changes are manual through SiteGround panel.
- **Provider diversity:** core-bu stays at Hetzner specifically so a netcup outage can't kill both Core and standby simultaneously.
- **app3 MySQL:** Root password in Vaultwarden + `/root/.my.cnf` on app3, accessible via 127.0.0.1:3306.
- **CloudPanel:** SQLite DB at `/home/clp/htdocs/app/data/db.sq3` — users live here, not in MySQL.
- **AWS CLI PATH:** All backup scripts must use `/opt/awscli-venv/bin/aws` or `source /opt/awscli-venv/bin/activate` — `aws` bare fails in cron context (PATH doesn't include venv bin). Documented in server-provisioning-standard v1.3.0.
- **Backup verification:** Always run at least one manual backup after provisioning a server and verify it landed in S3 — never trust cron entries alone. Silent failures (`aws: command not found`, wrong file paths, S3 permission issues) won't surface otherwise.