itpp-infrastructure
Last Updated: July 17, 2026
Maintainer: Sho'Nuff
Server Inventory
Core Server
- Hostname: Core
- IP: 152.53.192.33
- Provider: netcup RS 2000 G12
- Specs: 8 vCPU EPYC 9645, 15 GB DDR5 ECC, 512 GB NVMe
- OS: Debian 13
- Role: Hermes + Portals
- Key Services:
- Hermes Agent (Telegram + cron, 22 cron jobs)
- Caddy reverse proxy (12 domains, auto-TLS)
- Ops Portal (FastAPI, port 8090)
- Prometheus (native, port 9090) + Grafana (native, port 3002)
- Uptime Kuma (Docker, port 3001) — 9+ monitors
- Vaultwarden (Docker, port 8080) — vault.iamgmb.com
- Twenty CRM (Docker) — crm.debtrecoveryexperts.com
- DocuSeal (Docker, port 3000) — sign.core.itpropartner.com
- SearXNG (Docker, port 8888)
- Komodo (Docker, port 9120)
- Tailscale, StrongSwan, WireGuard (home CCR tunnel 10.77.0.0/24)
- Redis cache
App1 Server
- Hostname: app1
- IP: 152.53.36.131
- Provider: netcup RS 4000 G12
- Specs: 12 vCPU EPYC 9645, 32 GB DDR5 ECC, 1 TB NVMe
- OS: Debian 13
- Role: AI/Service Hub
- Key Services:
- Open WebUI (Docker, port 3000) — ai.itpropartner.com
- n8n + Postgres (Docker, port 5678) — n8n.itpropartner.com
- LiteLLM (Docker) + Postgres — admin-ai.itpropartner.com
- Mattermost Team Edition (Docker, port 8065) — noc.itpropartner.com
- Caddy (systemd, 80/443)
- 4 MCP servers: Browser (:8901), Filesystem (:8900), Email (:8902), Git (:8903)
- Super Search MCP (systemd, port 8899)
App2 Server
- Hostname: app2
- IP: 152.53.39.202
- Provider: netcup RS 4000 G12
- Specs: 12 vCPU EPYC 9645, 32 GB DDR5 ECC, 1 TB NVMe
- OS: Debian 13
- Role: Infrastructure Server
- Key Services:
- Traccar GPS (Docker, port 8082 + 5000-5150) — fleettracker360.com
- UniFi Controller (Docker, port 8443) — unifi.itpropartner.com
- UNMS/UISP (10 Docker containers) — unms.forefrontwireless.com
- Hudu (Docker) — hudu.itpropartner.com
- Caddy (4 domains)
App3 Server
- Hostname: app3
- IP: 152.53.241.111
- Provider: netcup RS 4000 G12
- Specs: 12 vCPU EPYC 9645, 32 GB DDR5 ECC, 1 TB NVMe
- OS: Debian 13
- Role: Web Hosting + Backup Restore
- Key Services:
- CloudPanel CE — panel.itpropartner.com
- Nginx (80/443) + Percona MySQL 8.4 + PHP 8.3
- Backup Restore System (Flask, port 8090) — my.itpropartner.com/backups
- WordPress sites (7 migrated from wphost02, all live):
- debtreecoveryexperts.com, boxpilotlogistics.com, iamgmb.com
- katiewattsdesign.com, vigilanttac.com, apextrackexperience.com
- mainwp.itpropartner.com, voipsimplicity.com, my.voipsimplicity.com
- Daily snapshots: 1 AM + 1 PM, 60-day retention, /opt/backup-restore/snapshots
Core-BU (Warm Standby)
- Hostname: core-bu
- IP: 5.161.225.131
- Provider: Hetzner CPX21
- Specs: 3 vCPU, 4 GB RAM, 80 GB SSD
- Role: Warm standby — auto-failover if Core down
- Watchdog: 5-min check, 4-cycle confirmation, S3 sync every 10 min
Legacy / Decommissioned
- old-ai: 178.156.167.181 (Hetzner CPX41) — decommissioned (LiteLLM migrated to app1)
- old app1: 87.99.144.163 (Hetzner CPX11) — deleted
- wphost02: 5.161.62.38 (Hetzner CPX21) — migrated to app3
- Ollama: Removed from Core (systemd) and app1 (Docker) Jul 17
Model Fallback Chain
All providers use direct API keys. GPT-5.5 quality survives through admin-ai → OpenRouter, then degrades through DeepSeek → Gemini → Grok.
| # |
Model |
Provider |
Gateway |
| Primary |
GPT-5.5 |
admin-ai |
Self-hosted LiteLLM (app1) |
| Fallback 1 |
GPT-5.5 |
OpenRouter |
openrouter.ai |
| Fallback 2 |
DeepSeek v4 Pro |
DeepSeek |
api.deepseek.com |
| Fallback 3 |
Gemini 3.5 Flash |
Google |
generativelanguage.googleapis.com |
| Fallback 4 |
Grok 4.5 |
xAI |
api.x.ai |
Credits (Jul 17): DeepSeek $58, OpenRouter ~$30 remaining, OpenAI/xAI/Google on pay-as-you-go
Health check: Daily 8 AM cron (model-usage-check)
Domain / DNS Map
itpropartner.com uses SiteGround nameservers exclusively. A Cloudflare zone exists (0dc20632…) but is NOT authoritative — records created there silently fail. All *.itpropartner.com changes must be manual through SiteGround panel.
| Domain |
IP |
Server |
Service |
| core.itpropartner.com |
152.53.192.33 |
Core |
Landing page + Grafana link |
| ops.itpropartner.com |
152.53.192.33 |
Core |
Ops dashboard |
| sign.core.itpropartner.com |
152.53.192.33 |
Core |
DocuSeal |
| uptimekuma.itpropartner.com |
152.53.192.33 |
Core |
Uptime monitoring |
| gps.fleettracker360.com |
152.53.192.33 |
Core |
Traccar HTTPS proxy → app2 |
| my.itpropartner.com |
152.53.192.33 |
Core |
Customer portal hub |
| hudu.itpropartner.com |
152.53.39.202 |
app2 |
IT documentation |
| unifi.itpropartner.com |
152.53.39.202 |
app2 |
UniFi controller |
| panel.itpropartner.com |
152.53.241.111 |
app3 |
CloudPanel CE |
| ai.itpropartner.com |
152.53.36.131 |
app1 |
Open WebUI |
| n8n.itpropartner.com |
152.53.36.131 |
app1 |
n8n automation |
| admin-ai.itpropartner.com |
152.53.36.131 |
app1 |
LiteLLM |
Cloudflare-Managed Domains
| Domain |
IP |
Server |
Service |
| fleettracker360.com |
Cloudflare |
app2 |
Fleet tracking website |
| gps.fleettracker360.com |
Cloudflare → Core |
Core → app2 |
Traccar devices |
| voipsimplicity.com |
Cloudflare |
app3 |
VoIP marketing site |
| my.voipsimplicity.com |
Cloudflare |
app3 |
VoIP customer portal |
| portal.debtrecoveryexperts.com |
152.53.192.33 |
Core |
DRE portal |
| crm.debtrecoveryexperts.com |
Cloudflare Access |
— |
DRE CRM |
| vault.iamgmb.com |
152.53.192.33 |
Core |
Vaultwarden |
| sign.iamgmb.com |
152.53.192.33 |
Core |
Document signing |
| shark.iamgmb.com |
152.53.192.33 |
Core |
Shark game |
DNS PENDING (create at SiteGround)
| Subdomain |
→ IP |
Service |
| vault.itpropartner.com |
152.53.36.131 |
Vaultwarden (after migration) |
| status.itpropartner.com |
152.53.192.33 |
Public status page |
Backup Pipeline
| Backup |
Schedule |
Target |
Purpose |
| hermes-live-sync |
Every 15 min |
s3://hermes-vps-backups/live/ |
Live state sync |
| hermes-full-backup |
Daily 1 AM |
s3://hermes-vps-backups/hermes-full-backup/ |
Full Hermes backup |
| home-router-backup |
Daily 6 AM |
s3://mikrotik-ccr-backups/ |
CCR config |
| root-essentials-backup |
Daily 3 AM |
S3 |
/root essentials |
| docker-volume-sync |
Daily 3 AM |
S3 |
Docker volumes |
| system-config-sync |
Daily 4 AM |
S3 |
System configs |
| unms-backup-sync |
Daily 6 AM (Core) |
s3://hermes-vps-backups/unms-backups/ |
UNMS data (pulled from app2) |
| unifi-backup-sync |
Daily 2 AM (Core) |
s3://hermes-vps-backups/unifi-backups/ |
UniFi configs (pulled from app2) |
| hudu-backup |
Daily 7 AM |
s3://hermes-vps-backups/hudu/backups/ |
Hudu volume dump |
| gitea-backup |
Daily 8 AM |
s3://hermes-vps-backups/gitea/daily/ |
Gitea repos |
| app1-backup |
Daily 2 AM |
s3://hermes-vps-backups/app1/ |
LiteLLM, n8n, OpenWebUI, MCP, Mattermost |
| app2-backup |
Daily 2:30 AM |
s3://hermes-vps-backups/app2/ |
Traccar, Gitea, Hudu, UNMS, UniFi |
| app3-backup |
Daily 3 AM |
s3://hermes-vps-backups/app3/ |
CloudPanel, MySQL, WordPress |
| wphost02-backup |
Daily 5 AM |
s3://hermes-vps-backups/wphost02-backup/ |
Webapps + MySQL |
| warm-standby-sync |
Every 10 min |
core-bu ← S3 |
DR readiness |
Caddy Configuration
Core (152.53.192.33)
core.itpropartner.com → static files
sign.core.itpropartner.com → localhost:3000 (DocuSeal)
ops.itpropartner.com → 127.0.0.1:8090 + static
uptimekuma.itpropartner.com → 127.0.0.1:3001 (Uptime Kuma)
gps.fleettracker360.com → app2:8082 (Traccar)
my.itpropartner.com → static files
portal.debtrecoveryexperts.com → static files
vault.iamgmb.com → localhost:8080 (Vaultwarden)
sign.iamgmb.com → 127.0.0.1:8090
shark.iamgmb.com → static + :8083
App1 (152.53.36.131)
ai.itpropartner.com → :3000 (Open WebUI)
n8n.itpropartner.com → :5678 (n8n)
admin-ai.itpropartner.com → :4000 (LiteLLM)
app1.itpropartner.com → static response
App2 (152.53.39.202)
hudu.itpropartner.com → Hudu internal
gps.fleettracker360.com → :8082 (Traccar)
unms.forefrontwireless.com → UNMS Nginx
unifi.itpropartner.com → :8443 (UniFi)
Service Access
| Service |
URL |
Location |
Auth |
| Open WebUI |
https://ai.itpropartner.com |
app1 |
Chat UI |
| Open WebUI Admin |
https://admin-ai.itpropartner.com/ui |
app1 |
user: admin, pw: LITELLM_MASTER_KEY |
| Ops Portal |
https://ops.itpropartner.com |
Core |
Internal dashboard |
| Grafana |
http://core.itpropartner.com:3002 |
Core |
admin/admin |
| Uptime Kuma |
https://uptimekuma.itpropartner.com |
Core |
Service monitoring |
| Vaultwarden |
https://vault.iamgmb.com |
Core |
Password vault |
| CloudPanel |
https://panel.itpropartner.com |
app3 |
user: gmb / SQLite auth |
| Traccar |
https://gps.fleettracker360.com |
app2 |
GPS fleet tracking |
| UniFi |
https://unifi.itpropartner.com |
app2 |
Network controller |
| UNMS |
https://unms.forefrontwireless.com |
app2 |
WISP management |
| Hudu |
https://hudu.itpropartner.com |
app2 |
IT documentation |
| n8n |
https://n8n.itpropartner.com |
app1 |
Automation |
| CRM (DRE) |
https://crm.debtrecoveryexperts.com |
Cloudflare Access |
TwentyCRM |
MCP Access (from Open WebUI)
| MCP Server |
Location |
Port |
Tools |
| Super Search |
app1 |
:8899 |
10 tools — web_search, web_extract, person_search, email_search, phone_search, etc. |
| Browser |
app1 |
:8901 |
browser_navigate, browser_snapshot, browser_click, browser_type, browser_console |
| Filesystem |
app1 |
:8900 |
read_file, write_file, search_files, list_dir, file_info |
| Email |
app1 |
:8902 |
search_emails, send_email, get_email |
| Git/Gitea |
app1 |
:8903 |
clone, commit, push, pull |
SSH Access
- Key:
itpp-infra (deployed to all servers)
- User:
ippadmin (sudo privileges)
- Root SSH: Enabled on app1, app2, app3 (key-only exception per provisioning standard)
- Core SSH:
ssh -i /root/.ssh/itpp-infra root@152.53.192.33
Firewall
UFW is enabled on all servers. Standard rules:
- Core: 22, 80, 443, 3000, 3001, 3002, 8080, 8082, 8090, 8443, 9090
- app1: 22, 80, 443, 3000, 5678, 8899, 8900, 8901, 8902, 8903
- app2: 22, 80, 443, 3000, 8080, 8082, 8089, 8443, 8843, 3478, 10001, 5000:5150
- app3: 22, 80, 443, 8443
SSL
All SSL certificates issued via Let's Encrypt through Caddy. All certs auto-renew. No manual management needed.
Notes
- DNS trap:
itpropartner.com uses SiteGround nameservers. Cloudflare zone is NOT authoritative. Always verify with dig NS domain.com before creating records.
- SiteGround: No API access. All DNS changes are manual through SiteGround panel.
- Provider diversity: core-bu stays at Hetzner specifically so a netcup outage can't kill both Core and standby simultaneously.
- app3 MySQL: Root password in Vaultwarden +
/root/.my.cnf on app3, accessible via 127.0.0.1:3306.
- CloudPanel: SQLite DB at
/home/clp/htdocs/app/data/db.sq3 — users live here, not in MySQL.
- AWS CLI PATH: All backup scripts must use
/opt/awscli-venv/bin/aws or source /opt/awscli-venv/bin/activate — aws bare fails in cron context (PATH doesn't include venv bin). Documented in server-provisioning-standard v1.3.0.
- Backup verification: Always run at least one manual backup after provisioning a server and verify it landed in S3 — never trust cron entries alone. Silent failures (
aws: command not found, wrong file paths, S3 permission issues) won't surface otherwise.