235 lines
9.1 KiB
Markdown
235 lines
9.1 KiB
Markdown
# IT Pro Partner — Infrastructure Documentation
|
|
|
|
> **Last Updated:** July 16, 2026
|
|
> **Maintainer:** Sho'Nuff
|
|
|
|
---
|
|
|
|
## Server Inventory
|
|
|
|
### Core Server
|
|
- **Hostname:** Core
|
|
- **IP:** 152.53.192.33
|
|
- **Provider:** netcup RS 2000 G12
|
|
- **Specs:** 8 vCPU EPYC 9645, 15 GB DDR5 ECC, 512 GB NVMe
|
|
- **OS:** Debian 13
|
|
- **Role:** Hermes + LiteLLM + Ollama fallback + Portals
|
|
- **Key Services:**
|
|
- Hermes Agent (Telegram + cron, 22 cron jobs)
|
|
- Caddy reverse proxy (11 domains, auto-TLS)
|
|
- Ops Portal (FastAPI, port 8090)
|
|
- Prometheus (native, port 9090) + Grafana (native, port 3002)
|
|
- Uptime Kuma (Docker, port 3001) — 9+ monitors
|
|
- Vaultwarden (Docker, port 8080) — vault.iamgmb.com
|
|
- Super Search MCP v2.0.0 (FastMCP, port 8899)
|
|
- Twenty CRM (Docker) — crm.debtrecoveryexperts.com
|
|
- DocuSeal (Docker, port 3000) — sign.itpropartner.com
|
|
- SearXNG (Docker, port 8888)
|
|
- Komodo (Docker, port 9120)
|
|
- Tailscale, StrongSwan, WireGuard (home CCR tunnel)
|
|
- Redis cache
|
|
|
|
### App1 Server
|
|
- **Hostname:** app1
|
|
- **IP:** 152.53.36.131
|
|
- **Provider:** netcup RS 4000 G12
|
|
- **Specs:** 12 vCPU EPYC 9645, 32 GB DDR5 ECC, 1 TB NVMe
|
|
- **OS:** Debian 13
|
|
- **Role:** AI/Service Hub
|
|
- **Key Services:**
|
|
- Ollama (Docker, port 11434)
|
|
- Open WebUI (Docker, port 3000) — ai.itpropartner.com
|
|
- n8n + Postgres (Docker, port 5678) — n8n.itpropartner.com
|
|
- Caddy (systemd, 80/443)
|
|
- 5 MCP servers: Super Search (:8899), Browser (:8901), Filesystem (:8900), Email (:8902), Git (:8903)
|
|
- **Planned:** LiteLLM + Postgres (migrating from old-ai 178.156.167.181)
|
|
|
|
### App2 Server
|
|
- **Hostname:** app2
|
|
- **IP:** 152.53.39.202
|
|
- **Provider:** netcup RS 4000 G12
|
|
- **Specs:** 12 vCPU EPYC 9645, 32 GB DDR5 ECC, 1 TB NVMe
|
|
- **OS:** Debian 13
|
|
- **Role:** Infrastructure Server
|
|
- **Key Services:**
|
|
- Traccar GPS (Docker, port 8082 + 5000-5150) — fleettracker360.com
|
|
- UniFi Controller (Docker, port 8443) — unifi.itpropartner.com
|
|
- UNMS/UISP (10 Docker containers) — unms.forefrontwireless.com
|
|
- Hudu (Docker) — hudu.itpropartner.com
|
|
- Caddy (4 domains)
|
|
|
|
### App3 Server
|
|
- **Hostname:** app3
|
|
- **IP:** 152.53.241.111
|
|
- **Provider:** netcup RS 4000 G12
|
|
- **Specs:** 12 vCPU EPYC 9645, 32 GB DDR5 ECC, 1 TB NVMe
|
|
- **OS:** Debian 13
|
|
- **Role:** Web Hosting
|
|
- **Key Services:**
|
|
- CloudPanel CE — panel.itpropartner.com
|
|
- Nginx (80/443) + Percona MySQL 8.4 + PHP 8.3
|
|
- voipsimplicity.com (WordPress, migrated Jul 16)
|
|
- my.voipsimplicity.com (PHP customer portal)
|
|
|
|
### Core-BU (Warm Standby)
|
|
- **Hostname:** core-bu
|
|
- **IP:** 5.161.114.8
|
|
- **Provider:** Hetzner CPX11 (planned upgrade: CPX31)
|
|
- **Specs:** 2 vCPU, 2 GB RAM, 40 GB SSD
|
|
- **Role:** Warm standby — auto-failover if Core down
|
|
- **Watchdog:** 5-min check, 4-cycle confirmation, S3 sync every 10 min
|
|
|
|
### Other Active Servers
|
|
- **old-ai:** 178.156.167.181 (Hetzner CPX41) — LiteLLM + Open WebUI + Ollama — **deprecated, migrating to app1**
|
|
- **wphost02:** 5.161.62.38 (Hetzner CPX21) — WordPress host — **migrating to app3**
|
|
- **Tony Hermes:** 87.99.159.142 (Hetzner CPX21) — Anthony's personal Hermes
|
|
|
|
---
|
|
|
|
## Domain / DNS Map
|
|
|
|
### ⚠️ itpropartner.com — SiteGround Nameservers Only
|
|
|
|
`itpropartner.com` uses **SiteGround nameservers** exclusively. A Cloudflare zone exists (`0dc20632…`) but is NOT authoritative — records created there silently fail. All `*.itpropartner.com` changes must be manual through SiteGround panel.
|
|
|
|
| Domain | IP | Server | Service |
|
|
|---|---|---|---|
|
|
| core.itpropartner.com | 152.53.192.33 | Core | Static API docs |
|
|
| ops.itpropartner.com | 152.53.192.33 | Core | Ops dashboard |
|
|
| sign.itpropartner.com | 152.53.192.33 | Core | DocuSeal |
|
|
| uptimekuma.itpropartner.com | 152.53.192.33 | Core | Monitoring |
|
|
| hudu.itpropartner.com | 152.53.39.202 | app2 | IT documentation |
|
|
| unifi.itpropartner.com | 152.53.39.202 | app2 | UniFi controller |
|
|
| panel.itpropartner.com | 152.53.241.111 | app3 | CloudPanel CE |
|
|
| ai.itpropartner.com | 152.53.36.131 | app1 | Open WebUI |
|
|
| n8n.itpropartner.com | 152.53.36.131 | app1 | n8n automation |
|
|
| admin-ai.itpropartner.com | 152.53.36.131 | app1 | LiteLLM (currently → old-ai) |
|
|
|
|
### Cloudflare-Managed Domains
|
|
|
|
| Domain | IP | Server | Service |
|
|
|---|---|---|---|
|
|
| fleettracker360.com | 152.53.39.202 | app2 | Fleet tracking |
|
|
| gps.fleettracker360.com | 152.53.39.202 | app2 | GPS devices |
|
|
| voipsimplicity.com | 152.53.241.111 | app3 | VoIP site |
|
|
| support.voipsimplicity.com | 152.53.241.111 | app3 | Support portal |
|
|
| portal.debtrecoveryexperts.com | 152.53.192.33 | Core | DRE portal |
|
|
| crm.debtrecoveryexperts.com | Cloudflare Access | — | DRE CRM |
|
|
|
|
---
|
|
|
|
## Backup Pipeline
|
|
|
|
| Backup | Schedule | Target | Purpose |
|
|
|---|---|---|---|
|
|
| hermes-live-sync | Every 15 min | s3://hermes-vps-backups/live/ | Live state sync |
|
|
| hermes-full-backup | Daily 1 AM | s3://hermes-vps-backups/hermes-full-backup/ | Full Hermes backup |
|
|
| home-router-backup | Daily 6 AM | s3://mikrotik-ccr-backups/ | CCR config |
|
|
| root-essentials-backup | Daily 3 AM | S3 | /root essentials |
|
|
| docker-volume-sync | Daily 3 AM | S3 | Docker volumes |
|
|
| system-config-sync | Daily 4 AM | S3 | System configs |
|
|
| unms-backup-sync | Daily 6 AM | s3://hermes-vps-backups/unms-backups/ | UNMS data |
|
|
| unifi-backup-sync | Daily 2 AM | s3://hermes-vps-backups/unifi-backups/ | UniFi configs |
|
|
| hudu-backup | Daily 3 AM | s3://hermes-vps-backups/hudu-backups/ | Hudu (30-day ret) |
|
|
| wphost02-mariadb-dump | Daily 2:30 AM | S3 | WordPress DB |
|
|
| warm-standby-sync | Every 10 min | core-bu ← S3 | DR readiness |
|
|
|
|
---
|
|
|
|
## Caddy Configuration
|
|
|
|
### Core (152.53.192.33)
|
|
```
|
|
sign.itpropartner.com → :3000 (DocuSeal)
|
|
core.itpropartner.com → static files
|
|
crm.debtrecoveryexperts.com → :3001 (Twenty)
|
|
portal.debtrecoveryexperts.com → static
|
|
internal.debtrecoveryexperts.com → static
|
|
app.itpropartner.com → static + :8082
|
|
shark.itpropartner.com → static + :8083
|
|
shark.iamgmb.com → static + :8083
|
|
ops.itpropartner.com → :8090 + static
|
|
uptimekuma.itpropartner.com → :3001
|
|
vault.iamgmb.com → :8080
|
|
```
|
|
|
|
### App1 (152.53.36.131)
|
|
```
|
|
ai.itpropartner.com → :3000 (Open WebUI)
|
|
n8n.itpropartner.com → :5678 (n8n)
|
|
```
|
|
|
|
### App2 (152.53.39.202)
|
|
```
|
|
hudu.itpropartner.com → Hudu internal
|
|
gps.fleettracker360.com → :8082 (Traccar)
|
|
unms.forefrontwireless.com → UNMS Nginx
|
|
unifi.itpropartner.com → :8443 (UniFi)
|
|
```
|
|
|
|
---
|
|
|
|
## Service Access
|
|
|
|
| Service | URL | Location | Auth |
|
|
|---|---|---|---|
|
|
| Open WebUI | https://ai.itpropartner.com | app1 | Chat UI |
|
|
| Open WebUI Admin | https://admin-ai.itpropartner.com/ui | app1 | user: admin, pw: LITELLM_MASTER_KEY |
|
|
| Ops Portal | https://ops.itpropartner.com | Core | Internal dashboard |
|
|
| Uptime Kuma | https://uptimekuma.itpropartner.com | Core | Service monitoring |
|
|
| Vaultwarden | https://vault.iamgmb.com | Core | Password vault |
|
|
| CloudPanel | https://panel.itpropartner.com | app3 | SQLite DB at /home/clp/htdocs/app/data/db.sq3 |
|
|
| Traccar | https://gps.fleettracker360.com | app2 | GPS fleet tracking |
|
|
| UniFi | https://unifi.itpropartner.com | app2 | Network controller |
|
|
| UNMS | https://unms.forefrontwireless.com | app2 | WISP management |
|
|
| Hudu | https://hudu.itpropartner.com | app2 | IT documentation |
|
|
| n8n | https://n8n.itpropartner.com | app1 | Automation |
|
|
| CRM (DRE) | https://crm.debtrecoveryexperts.com | Cloudflare Access | TwentyCRM |
|
|
|
|
### MCP Access (from Open WebUI)
|
|
|
|
All 5 MCP servers run on **app1** and are wired into Open WebUI at https://ai.itpropartner.com:
|
|
|
|
| MCP | Port | Tools |
|
|
|---|---|---|
|
|
| Super Search | :8899 | web_search, web_extract, web_search_premium |
|
|
| Browser | :8901 | browser_navigate, browser_snapshot, browser_click, browser_type, browser_console |
|
|
| Filesystem | :8900 | read_file, write_file, search_files, list_dir, file_info |
|
|
| Email | :8902 | search_emails, send_email, get_email |
|
|
| Git/Gitea | :8903 | clone, commit, push, pull |
|
|
|
|
---
|
|
|
|
## SSH Access
|
|
|
|
- **Key:** `itpp-infra` (deployed to all servers)
|
|
- **User:** `ippadmin` (sudo privileges)
|
|
- **Root SSH:** Enabled on app1, app2, app3 (key-only exception per provisioning standard)
|
|
- **Core SSH:** `ssh -i /root/.ssh/itpp-infra root@152.53.192.33`
|
|
|
|
---
|
|
|
|
## Firewall
|
|
|
|
UFW is enabled on all servers. Standard rules:
|
|
- **Core:** 22, 80, 443, 3000, 3001, 8080, 8082, 8443, 9090, 3002
|
|
- **app1:** 22, 80, 443, 3000, 5678, 8899, 8900, 8901, 8902, 8903
|
|
- **app2:** 22, 80, 443, 3000, 8080, 8082, 8089, 8443, 8843, 3478, 10001, 5000:5150
|
|
- **app3:** 22, 80, 443, 8443
|
|
|
|
---
|
|
|
|
## SSL
|
|
|
|
All SSL certificates issued via Let's Encrypt through Caddy. All certs auto-renew. No manual management needed.
|
|
|
|
---
|
|
|
|
## Notes
|
|
|
|
- **DNS trap:** `itpropartner.com` uses SiteGround nameservers. Cloudflare zone is NOT authoritative. Always verify with `dig NS domain.com` before creating records.
|
|
- **SiteGround:** No API access. All DNS changes are manual through SiteGround panel.
|
|
- **Provider diversity:** core-bu stays at Hetzner specifically so a netcup outage can't kill both Core and standby simultaneously.
|
|
- **app3 MySQL:** Root password at `MOQMINFQIhklM0AF`, accessible via 127.0.0.1:3306.
|
|
- **CloudPanel:** SQLite DB at `/home/clp/htdocs/app/data/db.sq3` — users live here, not in MySQL.
|