Add DNS records doc + service routing plan

2026-07-16 20:44:53 -04:00
parent 231321fc95
commit d70665039f
2 changed files with 274 additions and 139 deletions
@@ -1,184 +1,234 @@
-# ITPP Infrastructure Documentation
+# IT Pro Partner — Infrastructure Documentation
+
+> **Last Updated:** July 16, 2026
+> **Maintainer:** Sho'Nuff
+
+---

 ## Server Inventory

 ### Core Server
- **Hostname**: core
- **IP**: 152.53.192.33
- **Provider**: netcup RS 2000 G12
- **Specs**: RS 2000 G12 — 8 vCPU EPYC 9645, 16 GB DDR5 ECC, 512 GB NVMe
- **OS**: Debian 13
- **Role**: Primary infrastructure server
- **Key Services**:
-  - Hermes Agent
-  - LiteLLM
-  - Vaultwarden
-  - Uptime Kuma
-  - ops-portal
-  - Super Search MCP
-  - Cron jobs
+- **Hostname:** Core
+- **IP:** 152.53.192.33
+- **Provider:** netcup RS 2000 G12
+- **Specs:** 8 vCPU EPYC 9645, 15 GB DDR5 ECC, 512 GB NVMe
+- **OS:** Debian 13
+- **Role:** Hermes + LiteLLM + Ollama fallback + Portals
+- **Key Services:**
+  - Hermes Agent (Telegram + cron, 22 cron jobs)
+  - Caddy reverse proxy (11 domains, auto-TLS)
+  - Ops Portal (FastAPI, port 8090)
+  - Prometheus (native, port 9090) + Grafana (native, port 3002)
+  - Uptime Kuma (Docker, port 3001) — 9+ monitors
+  - Vaultwarden (Docker, port 8080) — vault.iamgmb.com
+  - Super Search MCP v2.0.0 (FastMCP, port 8899)
+  - Twenty CRM (Docker) — crm.debtrecoveryexperts.com
+  - DocuSeal (Docker, port 3000) — sign.itpropartner.com
+  - SearXNG (Docker, port 8888)
+  - Komodo (Docker, port 9120)
+  - Tailscale, StrongSwan, WireGuard (home CCR tunnel)
+  - Redis cache

 ### App1 Server
- **Hostname**: app1
- **IP**: 152.53.36.131
- **Provider**: netcup RS 4000
- **Specs**: RS 4000 G12 — 12 vCPU EPYC 9645, 32 GB DDR5 ECC, 1 TB NVMe
- **OS**: Debian 13
- **Role**: Application server
- **Key Services**:
-  - Ollama
-  - Open WebUI
-  - n8n
-  - Postgres
-  - Caddy
-  - SearXNG
+- **Hostname:** app1
+- **IP:** 152.53.36.131
+- **Provider:** netcup RS 4000 G12
+- **Specs:** 12 vCPU EPYC 9645, 32 GB DDR5 ECC, 1 TB NVMe
+- **OS:** Debian 13
+- **Role:** AI/Service Hub
+- **Key Services:**
+  - Ollama (Docker, port 11434)
+  - Open WebUI (Docker, port 3000) — ai.itpropartner.com
+  - n8n + Postgres (Docker, port 5678) — n8n.itpropartner.com
+  - Caddy (systemd, 80/443)
+  - 5 MCP servers: Super Search (:8899), Browser (:8901), Filesystem (:8900), Email (:8902), Git (:8903)
+- **Planned:** LiteLLM + Postgres (migrating from old-ai 178.156.167.181)

 ### App2 Server
- **Hostname**: app2
- **IP**: 152.53.39.202
- **Provider**: netcup RS 4000
- **Specs**: RS 4000 G12 — 12 vCPU EPYC 9645, 32 GB DDR5 ECC, 1 TB NVMe
- **OS**: Debian 13
- **Role**: Application server
- **Key Services**:
-  - UNMS
-  - UniFi Controller (v10.0.162, Docker: jacobalberty/unifi)
-    - Source: unifi.itpropartner.com (Caddy → localhost:8443)
-    - Old controller: 178.156.131.57 (v9.5.21) — scheduled to decommission after migration
-    - Backup: /var/lib/unifi/backup/autobackup/ on old server → S3 daily 2 AM ET via unifi-backup-sync.sh
-  - Hudu
-  - Traccar
+- **Hostname:** app2
+- **IP:** 152.53.39.202
+- **Provider:** netcup RS 4000 G12
+- **Specs:** 12 vCPU EPYC 9645, 32 GB DDR5 ECC, 1 TB NVMe
+- **OS:** Debian 13
+- **Role:** Infrastructure Server
+- **Key Services:**
+  - Traccar GPS (Docker, port 8082 + 5000-5150) — fleettracker360.com
+  - UniFi Controller (Docker, port 8443) — unifi.itpropartner.com
+  - UNMS/UISP (10 Docker containers) — unms.forefrontwireless.com
+  - Hudu (Docker) — hudu.itpropartner.com
  - Caddy (4 domains)
-  - Docker

 ### App3 Server
- **Hostname**: app3
- **IP**: 152.53.241.111
- **Provider**: netcup RS 2000
- **Specs**: RS 2000 G12 — 8 vCPU EPYC 9645, 16 GB DDR5 ECC, 512 GB NVMe
- **OS**: Debian 13
- **Role**: Application server
- **Key Services**:
-  - WordPress
-  - Apex Track
+- **Hostname:** app3
+- **IP:** 152.53.241.111
+- **Provider:** netcup RS 4000 G12
+- **Specs:** 12 vCPU EPYC 9645, 32 GB DDR5 ECC, 1 TB NVMe
+- **OS:** Debian 13
+- **Role:** Web Hosting
+- **Key Services:**
+  - CloudPanel CE — panel.itpropartner.com
+  - Nginx (80/443) + Percona MySQL 8.4 + PHP 8.3
+  - voipsimplicity.com (WordPress, migrated Jul 16)
+  - my.voipsimplicity.com (PHP customer portal)

-### WPHost02 Server
- **Hostname**: wphost02
- **IP**: 5.161.62.38
- **Provider**: RunCloud
- **Role**: WordPress hosting
+### Core-BU (Warm Standby)
+- **Hostname:** core-bu
+- **IP:** 5.161.114.8
+- **Provider:** Hetzner CPX11 (planned upgrade: CPX31)
+- **Specs:** 2 vCPU, 2 GB RAM, 40 GB SSD
+- **Role:** Warm standby — auto-failover if Core down
+- **Watchdog:** 5-min check, 4-cycle confirmation, S3 sync every 10 min

-### App1 Backup Server
- **Hostname**: app1-bu
- **IP**: 5.161.114.8
- **Provider**: Hetzner CPX11
- **Role**: Warm standby for app1
+### Other Active Servers
+- **old-ai:** 178.156.167.181 (Hetzner CPX41) — LiteLLM + Open WebUI + Ollama — **deprecated, migrating to app1**
+- **wphost02:** 5.161.62.38 (Hetzner CPX21) — WordPress host — **migrating to app3**
+- **Tony Hermes:** 87.99.159.142 (Hetzner CPX21) — Anthony's personal Hermes

-### Old Hudu Server
- **Hostname**: hudu (old)
- **IP**: 178.156.130.130
- **Provider**: Hetzner
- **Role**: Fallback for Hudu (7-day retention)
+---

 ## Domain / DNS Map

-### itpropartner.com
- **DNS Provider**: SiteGround
- **Subdomains**:
-  - core.itpropartner.com → 152.53.192.33
-  - hudu.itpropartner.com → 152.53.39.202
-  - ops.itpropartner.com → 152.53.192.33
-  - portal.itpropartner.com → 152.53.192.33
-  - admin-ai.itpropartner.com → 152.53.36.131
-  - uptimekuma.itpropartner.com → 152.53.192.33
-  - unifi.itpropartner.com → 152.53.39.202
+### ⚠️ itpropartner.com — SiteGround Nameservers Only

-### fleettracker360.com
- **DNS Provider**: Cloudflare
- **Subdomains**:
-  - gps.fleettracker360.com → 152.53.39.202 (Caddy)
+`itpropartner.com` uses **SiteGround nameservers** exclusively. A Cloudflare zone exists (`0dc20632…`) but is NOT authoritative — records created there silently fail. All `*.itpropartner.com` changes must be manual through SiteGround panel.

-### forefrontwireless.com
- **DNS Provider**: Unknown
- **Subdomains**:
-  - unms.forefrontwireless.com → 152.53.39.202 (Caddy)
+| Domain | IP | Server | Service |
+|---|---|---|---|
+| core.itpropartner.com | 152.53.192.33 | Core | Static API docs |
+| ops.itpropartner.com | 152.53.192.33 | Core | Ops dashboard |
+| sign.itpropartner.com | 152.53.192.33 | Core | DocuSeal |
+| uptimekuma.itpropartner.com | 152.53.192.33 | Core | Monitoring |
+| hudu.itpropartner.com | 152.53.39.202 | app2 | IT documentation |
+| unifi.itpropartner.com | 152.53.39.202 | app2 | UniFi controller |
+| panel.itpropartner.com | 152.53.241.111 | app3 | CloudPanel CE |
+| ai.itpropartner.com | 152.53.36.131 | app1 | Open WebUI |
+| n8n.itpropartner.com | 152.53.36.131 | app1 | n8n automation |
+| admin-ai.itpropartner.com | 152.53.36.131 | app1 | LiteLLM (currently → old-ai) |

-### debtrecoveryexperts.com
- **DNS Provider**: Cloudflare
- **Subdomains**:
-  - portal.debtrecoveryexperts.com → 152.53.192.33
-  - crm.debtrecoveryexperts.com → Cloudflare Access
+### Cloudflare-Managed Domains

-### iamgmb.com
- **DNS Provider**: SiteGround
- **Subdomains**:
-  - vault.iamgmb.com
-  - shark.iamgmb.com
-  - sign.iamgmb.com
+| Domain | IP | Server | Service |
+|---|---|---|---|
+| fleettracker360.com | 152.53.39.202 | app2 | Fleet tracking |
+| gps.fleettracker360.com | 152.53.39.202 | app2 | GPS devices |
+| voipsimplicity.com | 152.53.241.111 | app3 | VoIP site |
+| support.voipsimplicity.com | 152.53.241.111 | app3 | Support portal |
+| portal.debtrecoveryexperts.com | 152.53.192.33 | Core | DRE portal |
+| crm.debtrecoveryexperts.com | Cloudflare Access | — | DRE CRM |
+
+---

 ## Backup Pipeline

- **hermes-live-sync**: Every 15 min → s3://hermes-vps-backups/live/
- **hermes-full-backup**: Daily 1 AM → s3://hermes-vps-backups/hermes-full-backup/
- **home-router-backup**: Daily 6 AM → s3://mikrotik-ccr-backups/ (currently empty)
- **unms-backup-sync**: Daily 6 AM
- **hetzner-weekly-snapshots**: Monday 5 AM
- **root-essentials-backup**: Daily 3 AM
- **backup-audit-check**: Daily 2 AM
- **Warm standby**: app1-bu checks Core every 10 min, auto-failover
+| Backup | Schedule | Target | Purpose |
+|---|---|---|---|
+| hermes-live-sync | Every 15 min | s3://hermes-vps-backups/live/ | Live state sync |
+| hermes-full-backup | Daily 1 AM | s3://hermes-vps-backups/hermes-full-backup/ | Full Hermes backup |
+| home-router-backup | Daily 6 AM | s3://mikrotik-ccr-backups/ | CCR config |
+| root-essentials-backup | Daily 3 AM | S3 | /root essentials |
+| docker-volume-sync | Daily 3 AM | S3 | Docker volumes |
+| system-config-sync | Daily 4 AM | S3 | System configs |
+| unms-backup-sync | Daily 6 AM | s3://hermes-vps-backups/unms-backups/ | UNMS data |
+| unifi-backup-sync | Daily 2 AM | s3://hermes-vps-backups/unifi-backups/ | UniFi configs |
+| hudu-backup | Daily 3 AM | s3://hermes-vps-backups/hudu-backups/ | Hudu (30-day ret) |
+| wphost02-mariadb-dump | Daily 2:30 AM | S3 | WordPress DB |
+| warm-standby-sync | Every 10 min | core-bu ← S3 | DR readiness |

-## Caddy Config
+---

-### Core Server
- Proxied paths:
-  - /vault/
-  - /n8n/
-  - /webui/
-  - /litellm/
+## Caddy Configuration

-### App1 Server
- Proxied domains:
-  - admin-ai.itpropartner.com
+### Core (152.53.192.33)
+```
+sign.itpropartner.com    → :3000 (DocuSeal)
+core.itpropartner.com    → static files
+crm.debtrecoveryexperts.com → :3001 (Twenty)
+portal.debtrecoveryexperts.com → static
+internal.debtrecoveryexperts.com → static
+app.itpropartner.com     → static + :8082
+shark.itpropartner.com   → static + :8083
+shark.iamgmb.com         → static + :8083
+ops.itpropartner.com     → :8090 + static
+uptimekuma.itpropartner.com → :3001
+vault.iamgmb.com         → :8080
+```

-### App2 Server
- Proxied domains:
-  - hudu.itpropartner.com
-  - gps.fleettracker360.com
-  - unms.forefrontwireless.com
-  - unifi.itpropartner.com
+### App1 (152.53.36.131)
+```
+ai.itpropartner.com      → :3000 (Open WebUI)
+n8n.itpropartner.com     → :5678 (n8n)
+```

-## SSL
+### App2 (152.53.39.202)
+```
+hudu.itpropartner.com    → Hudu internal
+gps.fleettracker360.com  → :8082 (Traccar)
+unms.forefrontwireless.com → UNMS Nginx
+unifi.itpropartner.com   → :8443 (UniFi)
+```

-All SSL certificates are issued via Let's Encrypt through Caddy. No manual certificate management is required. Certificates auto-renew.
+---

 ## Service Access

-| Service | URL | Location | Notes |
+| Service | URL | Location | Auth |
 |---|---|---|---|
-| Open WebUI | http://ai.itpropartner.com | app1 | User-facing chat UI |
-| Open WebUI Admin | https://admin-ai.itpropartner.com/ui | app1 | Admin panel (user: admin, password: LiteLLM master key from /root/docker/litellm/.env) |
-| LiteLLM Proxy | https://admin-ai.itpropartner.com | app1 | API gateway for all models |
-| Ops Portal | https://ops.itpropartner.com | Core | Internal operations dashboard |
+| Open WebUI | https://ai.itpropartner.com | app1 | Chat UI |
+| Open WebUI Admin | https://admin-ai.itpropartner.com/ui | app1 | user: admin, pw: LITELLM_MASTER_KEY |
+| Ops Portal | https://ops.itpropartner.com | Core | Internal dashboard |
 | Uptime Kuma | https://uptimekuma.itpropartner.com | Core | Service monitoring |
-| Vaultwarden | https://core.itpropartner.com/vault/ | Core | Password vault |
-| n8n | https://core.itpropartner.com/n8n/ | Core | Workflow automation |
-| CRM (DRE) | https://crm.debtrecoveryexperts.com | Cloudflare | TwentyCRM (Cloudflare Access) |
+| Vaultwarden | https://vault.iamgmb.com | Core | Password vault |
+| CloudPanel | https://panel.itpropartner.com | app3 | SQLite DB at /home/clp/htdocs/app/data/db.sq3 |
+| Traccar | https://gps.fleettracker360.com | app2 | GPS fleet tracking |
+| UniFi | https://unifi.itpropartner.com | app2 | Network controller |
+| UNMS | https://unms.forefrontwireless.com | app2 | WISP management |
+| Hudu | https://hudu.itpropartner.com | app2 | IT documentation |
+| n8n | https://n8n.itpropartner.com | app1 | Automation |
+| CRM (DRE) | https://crm.debtrecoveryexperts.com | Cloudflare Access | TwentyCRM |

-### MCP Access from Open WebUI
+### MCP Access (from Open WebUI)

-All app1 MCPs (Super Search, Browser, Filesystem, Email, Git) are wired into Open WebUI's tool system. See [mcp-servers/README.md](../mcp-servers/README.md#access) for details on adding and testing MCPs.
+All 5 MCP servers run on **app1** and are wired into Open WebUI at https://ai.itpropartner.com:
+
+| MCP | Port | Tools |
+|---|---|---|
+| Super Search | :8899 | web_search, web_extract, web_search_premium |
+| Browser | :8901 | browser_navigate, browser_snapshot, browser_click, browser_type, browser_console |
+| Filesystem | :8900 | read_file, write_file, search_files, list_dir, file_info |
+| Email | :8902 | search_emails, send_email, get_email |
+| Git/Gitea | :8903 | clone, commit, push, pull |
+
+---

 ## SSH Access

- **Key**: itpp-infra key on all servers
- **User**: ippadmin (sudo privileges)
- **Root SSH**: Enabled on app1, app2, app3 (key-only exception)
+- **Key:** `itpp-infra` (deployed to all servers)
+- **User:** `ippadmin` (sudo privileges)
+- **Root SSH:** Enabled on app1, app2, app3 (key-only exception per provisioning standard)
+- **Core SSH:** `ssh -i /root/.ssh/itpp-infra root@152.53.192.33`
+
+---

 ## Firewall

-UFW is enabled on all servers. Current rules for app2:
- Allowed ports: 22, 80, 443, 3000, 8080, 8082, 8089, 8443, 8843, 3478, 10001
+UFW is enabled on all servers. Standard rules:
+- **Core:** 22, 80, 443, 3000, 3001, 8080, 8082, 8443, 9090, 3002
+- **app1:** 22, 80, 443, 3000, 5678, 8899, 8900, 8901, 8902, 8903
+- **app2:** 22, 80, 443, 3000, 8080, 8082, 8089, 8443, 8843, 3478, 10001, 5000:5150
+- **app3:** 22, 80, 443, 8443
+
+---
+
+## SSL
+
+All SSL certificates issued via Let's Encrypt through Caddy. All certs auto-renew. No manual management needed.
+
+---

 ## Notes

-This document is a living record of the ITPP infrastructure. Update it as changes occur.
+- **DNS trap:** `itpropartner.com` uses SiteGround nameservers. Cloudflare zone is NOT authoritative. Always verify with `dig NS domain.com` before creating records.
+- **SiteGround:** No API access. All DNS changes are manual through SiteGround panel.
+- **Provider diversity:** core-bu stays at Hetzner specifically so a netcup outage can't kill both Core and standby simultaneously.
+- **app3 MySQL:** Root password at `MOQMINFQIhklM0AF`, accessible via 127.0.0.1:3306.
+- **CloudPanel:** SQLite DB at `/home/clp/htdocs/app/data/db.sq3` — users live here, not in MySQL.
@@ -0,0 +1,85 @@
+# DNS Records — itpropartner.com
+**Managed by:** SiteGround Site Tools (no API access)
+**Updated:** July 16, 2026
+**⚠️ Germaine must apply changes manually at SiteGround**
+
+## CREATE (pending)
+
+| Subdomain | Record | IP | Service |
+|---|---|---|---|
+| my.itpropartner.com | A | 152.53.192.33 | Customer portal hub |
+| vault.itpropartner.com | A | 152.53.36.131 | Vaultwarden (moving from Core) |
+| status.itpropartner.com | A | 152.53.192.33 | Public status page |
+
+## FIX
+
+| Subdomain | Current IP | Should Be | Reason |
+|---|---|---|---|
+| app1-bu.itpropartner.com | 5.161.114.8 | **5.161.225.131** | Old CPX11 → new CPX21 warm standby |
+
+## PRODUCTION (Netcup — no changes)
+
+| Subdomain | IP | Server | Service |
+|---|---|---|---|
+| ops.itpropartner.com | 152.53.192.33 | Core | Operations dashboard |
+| core.itpropartner.com | 152.53.192.33 | Core | Landing page + MikroTik |
+| sign.itpropartner.com | 152.53.192.33 | Core | Document signing |
+| uptimekuma.itpropartner.com | 152.53.192.33 | Core | Uptime monitoring |
+| app.itpropartner.com | 152.53.192.33 | Core | App entry |
+| admin-ai.itpropartner.com | 152.53.36.131 | app1 | LiteLLM |
+| ai.itpropartner.com | 152.53.36.131 | app1 | Open WebUI |
+| n8n.itpropartner.com | 152.53.36.131 | app1 | Workflow automation |
+| app1.itpropartner.com | 152.53.36.131 | app1 | App server 1 |
+| git.itpropartner.com | 152.53.39.202 | app2 | Gitea |
+| hudu.itpropartner.com | 152.53.39.202 | app2 | Documentation |
+| unifi.itpropartner.com | 152.53.39.202 | app2 | UniFi controller |
+| www.unifi.itpropartner.com | 152.53.39.202 | app2 | UniFi (www) |
+| panel.itpropartner.com | 152.53.241.111 | app3 | CloudPanel |
+
+## DELETE (all point to 35.212.86.161 — old host)
+
+| Subdomain | Notes |
+|---|---|
+| garyjavo.itpropartner.com | Old client site |
+| www.garyjavo.itpropartner.com | Old client site |
+| clientmedia.itpropartner.com | Old project |
+| www.clientmedia.itpropartner.com | Old project |
+| media.itpropartner.com | Old project |
+| www.media.itpropartner.com | Old project |
+| apx.itpropartner.com | Old project |
+| www.apx.itpropartner.com | Old project |
+| ssh.itpropartner.com | Old SSH endpoint |
+| support.itpropartner.com | Moving to my.itpropartner.com |
+| www.support.itpropartner.com | Moving to my.itpropartner.com |
+| mail.itpropartner.com | MXroute handles email |
+| ftp.itpropartner.com | Not used |
+| autodiscover.itpropartner.com | Exchange not used |
+| autoconfig.itpropartner.com | Exchange not used |
+| mainwp.itpropartner.com | Moving to ops |
+
+## Service Routing Plan
+
+### my.itpropartner.com (152.53.192.33) — Customer Hub
+DNS, Digital Signage, Networking, Status, Tech Support, Web Hosting, Marketplace, Phone (→ voipsimplicity.com), Fleet (→ fleettracker360.com)
+
+### ops.itpropartner.com (152.53.192.33) — Technical Hub
+- `/` — Operations Dashboard (port 8090)
+- `/grafana/` — Grafana (Core :3002)
+- `/uptime/` — Uptime Kuma (Core :3001)
+- `/gitea/` — Gitea (app2 :3000)
+- `/n8n/` — n8n (app1 :5678)
+- `/search/` — Super Search / OSINT (app1 :8100)
+- `/noc/` — Mattermost (future, app1)
+
+### core.itpropartner.com (152.53.192.33) — Minimal Landing
+Stripped down — just links to ops, my, and other services. Keep Core lean for Hermes.
+
+### Standalone (no changes)
+- vault.itpropartner.com → 152.53.36.131 (app1, after migration)
+- admin-ai.itpropartner.com → 152.53.36.131 (app1)
+- hudu.itpropartner.com → 152.53.39.202 (app2)
+- panel.itpropartner.com → 152.53.241.111 (app3)
+
+### Other Domains
+- fleettracker360.com → Cloudflare DNS
+- voipsimplicity.com → Cloudflare DNS