diff --git a/README.md b/README.md index 3eee3f8..5ab552f 100644 --- a/README.md +++ b/README.md @@ -1,184 +1,234 @@ -# ITPP Infrastructure Documentation +# IT Pro Partner — Infrastructure Documentation + +> **Last Updated:** July 16, 2026 +> **Maintainer:** Sho'Nuff + +--- ## Server Inventory ### Core Server -- **Hostname**: core -- **IP**: 152.53.192.33 -- **Provider**: netcup RS 2000 G12 -- **Specs**: RS 2000 G12 — 8 vCPU EPYC 9645, 16 GB DDR5 ECC, 512 GB NVMe -- **OS**: Debian 13 -- **Role**: Primary infrastructure server -- **Key Services**: - - Hermes Agent - - LiteLLM - - Vaultwarden - - Uptime Kuma - - ops-portal - - Super Search MCP - - Cron jobs +- **Hostname:** Core +- **IP:** 152.53.192.33 +- **Provider:** netcup RS 2000 G12 +- **Specs:** 8 vCPU EPYC 9645, 15 GB DDR5 ECC, 512 GB NVMe +- **OS:** Debian 13 +- **Role:** Hermes + LiteLLM + Ollama fallback + Portals +- **Key Services:** + - Hermes Agent (Telegram + cron, 22 cron jobs) + - Caddy reverse proxy (11 domains, auto-TLS) + - Ops Portal (FastAPI, port 8090) + - Prometheus (native, port 9090) + Grafana (native, port 3002) + - Uptime Kuma (Docker, port 3001) — 9+ monitors + - Vaultwarden (Docker, port 8080) — vault.iamgmb.com + - Super Search MCP v2.0.0 (FastMCP, port 8899) + - Twenty CRM (Docker) — crm.debtrecoveryexperts.com + - DocuSeal (Docker, port 3000) — sign.itpropartner.com + - SearXNG (Docker, port 8888) + - Komodo (Docker, port 9120) + - Tailscale, StrongSwan, WireGuard (home CCR tunnel) + - Redis cache ### App1 Server -- **Hostname**: app1 -- **IP**: 152.53.36.131 -- **Provider**: netcup RS 4000 -- **Specs**: RS 4000 G12 — 12 vCPU EPYC 9645, 32 GB DDR5 ECC, 1 TB NVMe -- **OS**: Debian 13 -- **Role**: Application server -- **Key Services**: - - Ollama - - Open WebUI - - n8n - - Postgres - - Caddy - - SearXNG +- **Hostname:** app1 +- **IP:** 152.53.36.131 +- **Provider:** netcup RS 4000 G12 +- **Specs:** 12 vCPU EPYC 9645, 32 GB DDR5 ECC, 1 TB NVMe +- **OS:** Debian 13 +- **Role:** AI/Service Hub +- **Key Services:** + - Ollama (Docker, port 11434) + - Open WebUI (Docker, port 3000) — ai.itpropartner.com + - n8n + Postgres (Docker, port 5678) — n8n.itpropartner.com + - Caddy (systemd, 80/443) + - 5 MCP servers: Super Search (:8899), Browser (:8901), Filesystem (:8900), Email (:8902), Git (:8903) +- **Planned:** LiteLLM + Postgres (migrating from old-ai 178.156.167.181) ### App2 Server -- **Hostname**: app2 -- **IP**: 152.53.39.202 -- **Provider**: netcup RS 4000 -- **Specs**: RS 4000 G12 — 12 vCPU EPYC 9645, 32 GB DDR5 ECC, 1 TB NVMe -- **OS**: Debian 13 -- **Role**: Application server -- **Key Services**: - - UNMS - - UniFi Controller (v10.0.162, Docker: jacobalberty/unifi) - - Source: unifi.itpropartner.com (Caddy → localhost:8443) - - Old controller: 178.156.131.57 (v9.5.21) — scheduled to decommission after migration - - Backup: /var/lib/unifi/backup/autobackup/ on old server → S3 daily 2 AM ET via unifi-backup-sync.sh - - Hudu - - Traccar +- **Hostname:** app2 +- **IP:** 152.53.39.202 +- **Provider:** netcup RS 4000 G12 +- **Specs:** 12 vCPU EPYC 9645, 32 GB DDR5 ECC, 1 TB NVMe +- **OS:** Debian 13 +- **Role:** Infrastructure Server +- **Key Services:** + - Traccar GPS (Docker, port 8082 + 5000-5150) — fleettracker360.com + - UniFi Controller (Docker, port 8443) — unifi.itpropartner.com + - UNMS/UISP (10 Docker containers) — unms.forefrontwireless.com + - Hudu (Docker) — hudu.itpropartner.com - Caddy (4 domains) - - Docker ### App3 Server -- **Hostname**: app3 -- **IP**: 152.53.241.111 -- **Provider**: netcup RS 2000 -- **Specs**: RS 2000 G12 — 8 vCPU EPYC 9645, 16 GB DDR5 ECC, 512 GB NVMe -- **OS**: Debian 13 -- **Role**: Application server -- **Key Services**: - - WordPress - - Apex Track +- **Hostname:** app3 +- **IP:** 152.53.241.111 +- **Provider:** netcup RS 4000 G12 +- **Specs:** 12 vCPU EPYC 9645, 32 GB DDR5 ECC, 1 TB NVMe +- **OS:** Debian 13 +- **Role:** Web Hosting +- **Key Services:** + - CloudPanel CE — panel.itpropartner.com + - Nginx (80/443) + Percona MySQL 8.4 + PHP 8.3 + - voipsimplicity.com (WordPress, migrated Jul 16) + - my.voipsimplicity.com (PHP customer portal) -### WPHost02 Server -- **Hostname**: wphost02 -- **IP**: 5.161.62.38 -- **Provider**: RunCloud -- **Role**: WordPress hosting +### Core-BU (Warm Standby) +- **Hostname:** core-bu +- **IP:** 5.161.114.8 +- **Provider:** Hetzner CPX11 (planned upgrade: CPX31) +- **Specs:** 2 vCPU, 2 GB RAM, 40 GB SSD +- **Role:** Warm standby — auto-failover if Core down +- **Watchdog:** 5-min check, 4-cycle confirmation, S3 sync every 10 min -### App1 Backup Server -- **Hostname**: app1-bu -- **IP**: 5.161.114.8 -- **Provider**: Hetzner CPX11 -- **Role**: Warm standby for app1 +### Other Active Servers +- **old-ai:** 178.156.167.181 (Hetzner CPX41) — LiteLLM + Open WebUI + Ollama — **deprecated, migrating to app1** +- **wphost02:** 5.161.62.38 (Hetzner CPX21) — WordPress host — **migrating to app3** +- **Tony Hermes:** 87.99.159.142 (Hetzner CPX21) — Anthony's personal Hermes -### Old Hudu Server -- **Hostname**: hudu (old) -- **IP**: 178.156.130.130 -- **Provider**: Hetzner -- **Role**: Fallback for Hudu (7-day retention) +--- -## Domain/DNS Map +## Domain / DNS Map -### itpropartner.com -- **DNS Provider**: SiteGround -- **Subdomains**: - - core.itpropartner.com → 152.53.192.33 - - hudu.itpropartner.com → 152.53.39.202 - - ops.itpropartner.com → 152.53.192.33 - - portal.itpropartner.com → 152.53.192.33 - - admin-ai.itpropartner.com → 152.53.36.131 - - uptimekuma.itpropartner.com → 152.53.192.33 - - unifi.itpropartner.com → 152.53.39.202 +### ⚠️ itpropartner.com — SiteGround Nameservers Only -### fleettracker360.com -- **DNS Provider**: Cloudflare -- **Subdomains**: - - gps.fleettracker360.com → 152.53.39.202 (Caddy) +`itpropartner.com` uses **SiteGround nameservers** exclusively. A Cloudflare zone exists (`0dc20632…`) but is NOT authoritative — records created there silently fail. All `*.itpropartner.com` changes must be manual through SiteGround panel. -### forefrontwireless.com -- **DNS Provider**: Unknown -- **Subdomains**: - - unms.forefrontwireless.com → 152.53.39.202 (Caddy) +| Domain | IP | Server | Service | +|---|---|---|---| +| core.itpropartner.com | 152.53.192.33 | Core | Static API docs | +| ops.itpropartner.com | 152.53.192.33 | Core | Ops dashboard | +| sign.itpropartner.com | 152.53.192.33 | Core | DocuSeal | +| uptimekuma.itpropartner.com | 152.53.192.33 | Core | Monitoring | +| hudu.itpropartner.com | 152.53.39.202 | app2 | IT documentation | +| unifi.itpropartner.com | 152.53.39.202 | app2 | UniFi controller | +| panel.itpropartner.com | 152.53.241.111 | app3 | CloudPanel CE | +| ai.itpropartner.com | 152.53.36.131 | app1 | Open WebUI | +| n8n.itpropartner.com | 152.53.36.131 | app1 | n8n automation | +| admin-ai.itpropartner.com | 152.53.36.131 | app1 | LiteLLM (currently → old-ai) | -### debtrecoveryexperts.com -- **DNS Provider**: Cloudflare -- **Subdomains**: - - portal.debtrecoveryexperts.com → 152.53.192.33 - - crm.debtrecoveryexperts.com → Cloudflare Access +### Cloudflare-Managed Domains -### iamgmb.com -- **DNS Provider**: SiteGround -- **Subdomains**: - - vault.iamgmb.com - - shark.iamgmb.com - - sign.iamgmb.com +| Domain | IP | Server | Service | +|---|---|---|---| +| fleettracker360.com | 152.53.39.202 | app2 | Fleet tracking | +| gps.fleettracker360.com | 152.53.39.202 | app2 | GPS devices | +| voipsimplicity.com | 152.53.241.111 | app3 | VoIP site | +| support.voipsimplicity.com | 152.53.241.111 | app3 | Support portal | +| portal.debtrecoveryexperts.com | 152.53.192.33 | Core | DRE portal | +| crm.debtrecoveryexperts.com | Cloudflare Access | — | DRE CRM | + +--- ## Backup Pipeline -- **hermes-live-sync**: Every 15 min → s3://hermes-vps-backups/live/ -- **hermes-full-backup**: Daily 1 AM → s3://hermes-vps-backups/hermes-full-backup/ -- **home-router-backup**: Daily 6 AM → s3://mikrotik-ccr-backups/ (currently empty) -- **unms-backup-sync**: Daily 6 AM -- **hetzner-weekly-snapshots**: Monday 5 AM -- **root-essentials-backup**: Daily 3 AM -- **backup-audit-check**: Daily 2 AM -- **Warm standby**: app1-bu checks Core every 10 min, auto-failover +| Backup | Schedule | Target | Purpose | +|---|---|---|---| +| hermes-live-sync | Every 15 min | s3://hermes-vps-backups/live/ | Live state sync | +| hermes-full-backup | Daily 1 AM | s3://hermes-vps-backups/hermes-full-backup/ | Full Hermes backup | +| home-router-backup | Daily 6 AM | s3://mikrotik-ccr-backups/ | CCR config | +| root-essentials-backup | Daily 3 AM | S3 | /root essentials | +| docker-volume-sync | Daily 3 AM | S3 | Docker volumes | +| system-config-sync | Daily 4 AM | S3 | System configs | +| unms-backup-sync | Daily 6 AM | s3://hermes-vps-backups/unms-backups/ | UNMS data | +| unifi-backup-sync | Daily 2 AM | s3://hermes-vps-backups/unifi-backups/ | UniFi configs | +| hudu-backup | Daily 3 AM | s3://hermes-vps-backups/hudu-backups/ | Hudu (30-day ret) | +| wphost02-mariadb-dump | Daily 2:30 AM | S3 | WordPress DB | +| warm-standby-sync | Every 10 min | core-bu ← S3 | DR readiness | -## Caddy Config +--- -### Core Server -- Proxied paths: - - /vault/ - - /n8n/ - - /webui/ - - /litellm/ +## Caddy Configuration -### App1 Server -- Proxied domains: - - admin-ai.itpropartner.com +### Core (152.53.192.33) +``` +sign.itpropartner.com → :3000 (DocuSeal) +core.itpropartner.com → static files +crm.debtrecoveryexperts.com → :3001 (Twenty) +portal.debtrecoveryexperts.com → static +internal.debtrecoveryexperts.com → static +app.itpropartner.com → static + :8082 +shark.itpropartner.com → static + :8083 +shark.iamgmb.com → static + :8083 +ops.itpropartner.com → :8090 + static +uptimekuma.itpropartner.com → :3001 +vault.iamgmb.com → :8080 +``` -### App2 Server -- Proxied domains: - - hudu.itpropartner.com - - gps.fleettracker360.com - - unms.forefrontwireless.com - - unifi.itpropartner.com +### App1 (152.53.36.131) +``` +ai.itpropartner.com → :3000 (Open WebUI) +n8n.itpropartner.com → :5678 (n8n) +``` -## SSL +### App2 (152.53.39.202) +``` +hudu.itpropartner.com → Hudu internal +gps.fleettracker360.com → :8082 (Traccar) +unms.forefrontwireless.com → UNMS Nginx +unifi.itpropartner.com → :8443 (UniFi) +``` -All SSL certificates are issued via Let's Encrypt through Caddy. No manual certificate management is required. Certificates auto-renew. +--- ## Service Access -| Service | URL | Location | Notes | +| Service | URL | Location | Auth | |---|---|---|---| -| Open WebUI | http://ai.itpropartner.com | app1 | User-facing chat UI | -| Open WebUI Admin | https://admin-ai.itpropartner.com/ui | app1 | Admin panel (user: admin, password: LiteLLM master key from /root/docker/litellm/.env) | -| LiteLLM Proxy | https://admin-ai.itpropartner.com | app1 | API gateway for all models | -| Ops Portal | https://ops.itpropartner.com | Core | Internal operations dashboard | +| Open WebUI | https://ai.itpropartner.com | app1 | Chat UI | +| Open WebUI Admin | https://admin-ai.itpropartner.com/ui | app1 | user: admin, pw: LITELLM_MASTER_KEY | +| Ops Portal | https://ops.itpropartner.com | Core | Internal dashboard | | Uptime Kuma | https://uptimekuma.itpropartner.com | Core | Service monitoring | -| Vaultwarden | https://core.itpropartner.com/vault/ | Core | Password vault | -| n8n | https://core.itpropartner.com/n8n/ | Core | Workflow automation | -| CRM (DRE) | https://crm.debtrecoveryexperts.com | Cloudflare | TwentyCRM (Cloudflare Access) | +| Vaultwarden | https://vault.iamgmb.com | Core | Password vault | +| CloudPanel | https://panel.itpropartner.com | app3 | SQLite DB at /home/clp/htdocs/app/data/db.sq3 | +| Traccar | https://gps.fleettracker360.com | app2 | GPS fleet tracking | +| UniFi | https://unifi.itpropartner.com | app2 | Network controller | +| UNMS | https://unms.forefrontwireless.com | app2 | WISP management | +| Hudu | https://hudu.itpropartner.com | app2 | IT documentation | +| n8n | https://n8n.itpropartner.com | app1 | Automation | +| CRM (DRE) | https://crm.debtrecoveryexperts.com | Cloudflare Access | TwentyCRM | -### MCP Access from Open WebUI +### MCP Access (from Open WebUI) -All app1 MCPs (Super Search, Browser, Filesystem, Email, Git) are wired into Open WebUI's tool system. See [mcp-servers/README.md](../mcp-servers/README.md#access) for details on adding and testing MCPs. +All 5 MCP servers run on **app1** and are wired into Open WebUI at https://ai.itpropartner.com: + +| MCP | Port | Tools | +|---|---|---| +| Super Search | :8899 | web_search, web_extract, web_search_premium | +| Browser | :8901 | browser_navigate, browser_snapshot, browser_click, browser_type, browser_console | +| Filesystem | :8900 | read_file, write_file, search_files, list_dir, file_info | +| Email | :8902 | search_emails, send_email, get_email | +| Git/Gitea | :8903 | clone, commit, push, pull | + +--- ## SSH Access -- **Key**: itpp-infra key on all servers -- **User**: ippadmin (sudo privileges) -- **Root SSH**: Enabled on app1, app2, app3 (key-only exception) +- **Key:** `itpp-infra` (deployed to all servers) +- **User:** `ippadmin` (sudo privileges) +- **Root SSH:** Enabled on app1, app2, app3 (key-only exception per provisioning standard) +- **Core SSH:** `ssh -i /root/.ssh/itpp-infra root@152.53.192.33` + +--- ## Firewall -UFW is enabled on all servers. Current rules for app2: -- Allowed ports: 22, 80, 443, 3000, 8080, 8082, 8089, 8443, 8843, 3478, 10001 +UFW is enabled on all servers. Standard rules: +- **Core:** 22, 80, 443, 3000, 3001, 8080, 8082, 8443, 9090, 3002 +- **app1:** 22, 80, 443, 3000, 5678, 8899, 8900, 8901, 8902, 8903 +- **app2:** 22, 80, 443, 3000, 8080, 8082, 8089, 8443, 8843, 3478, 10001, 5000:5150 +- **app3:** 22, 80, 443, 8443 + +--- + +## SSL + +All SSL certificates issued via Let's Encrypt through Caddy. All certs auto-renew. No manual management needed. + +--- ## Notes -This document is a living record of the ITPP infrastructure. Update it as changes occur. \ No newline at end of file +- **DNS trap:** `itpropartner.com` uses SiteGround nameservers. Cloudflare zone is NOT authoritative. Always verify with `dig NS domain.com` before creating records. +- **SiteGround:** No API access. All DNS changes are manual through SiteGround panel. +- **Provider diversity:** core-bu stays at Hetzner specifically so a netcup outage can't kill both Core and standby simultaneously. +- **app3 MySQL:** Root password at `MOQMINFQIhklM0AF`, accessible via 127.0.0.1:3306. +- **CloudPanel:** SQLite DB at `/home/clp/htdocs/app/data/db.sq3` — users live here, not in MySQL. diff --git a/dns-records.md b/dns-records.md new file mode 100644 index 0000000..d854079 --- /dev/null +++ b/dns-records.md @@ -0,0 +1,85 @@ +# DNS Records — itpropartner.com +**Managed by:** SiteGround Site Tools (no API access) +**Updated:** July 16, 2026 +**⚠️ Germaine must apply changes manually at SiteGround** + +## CREATE (pending) + +| Subdomain | Record | IP | Service | +|---|---|---|---| +| my.itpropartner.com | A | 152.53.192.33 | Customer portal hub | +| vault.itpropartner.com | A | 152.53.36.131 | Vaultwarden (moving from Core) | +| status.itpropartner.com | A | 152.53.192.33 | Public status page | + +## FIX + +| Subdomain | Current IP | Should Be | Reason | +|---|---|---|---| +| app1-bu.itpropartner.com | 5.161.114.8 | **5.161.225.131** | Old CPX11 → new CPX21 warm standby | + +## PRODUCTION (Netcup — no changes) + +| Subdomain | IP | Server | Service | +|---|---|---|---| +| ops.itpropartner.com | 152.53.192.33 | Core | Operations dashboard | +| core.itpropartner.com | 152.53.192.33 | Core | Landing page + MikroTik | +| sign.itpropartner.com | 152.53.192.33 | Core | Document signing | +| uptimekuma.itpropartner.com | 152.53.192.33 | Core | Uptime monitoring | +| app.itpropartner.com | 152.53.192.33 | Core | App entry | +| admin-ai.itpropartner.com | 152.53.36.131 | app1 | LiteLLM | +| ai.itpropartner.com | 152.53.36.131 | app1 | Open WebUI | +| n8n.itpropartner.com | 152.53.36.131 | app1 | Workflow automation | +| app1.itpropartner.com | 152.53.36.131 | app1 | App server 1 | +| git.itpropartner.com | 152.53.39.202 | app2 | Gitea | +| hudu.itpropartner.com | 152.53.39.202 | app2 | Documentation | +| unifi.itpropartner.com | 152.53.39.202 | app2 | UniFi controller | +| www.unifi.itpropartner.com | 152.53.39.202 | app2 | UniFi (www) | +| panel.itpropartner.com | 152.53.241.111 | app3 | CloudPanel | + +## DELETE (all point to 35.212.86.161 — old host) + +| Subdomain | Notes | +|---|---| +| garyjavo.itpropartner.com | Old client site | +| www.garyjavo.itpropartner.com | Old client site | +| clientmedia.itpropartner.com | Old project | +| www.clientmedia.itpropartner.com | Old project | +| media.itpropartner.com | Old project | +| www.media.itpropartner.com | Old project | +| apx.itpropartner.com | Old project | +| www.apx.itpropartner.com | Old project | +| ssh.itpropartner.com | Old SSH endpoint | +| support.itpropartner.com | Moving to my.itpropartner.com | +| www.support.itpropartner.com | Moving to my.itpropartner.com | +| mail.itpropartner.com | MXroute handles email | +| ftp.itpropartner.com | Not used | +| autodiscover.itpropartner.com | Exchange not used | +| autoconfig.itpropartner.com | Exchange not used | +| mainwp.itpropartner.com | Moving to ops | + +## Service Routing Plan + +### my.itpropartner.com (152.53.192.33) — Customer Hub +DNS, Digital Signage, Networking, Status, Tech Support, Web Hosting, Marketplace, Phone (→ voipsimplicity.com), Fleet (→ fleettracker360.com) + +### ops.itpropartner.com (152.53.192.33) — Technical Hub +- `/` — Operations Dashboard (port 8090) +- `/grafana/` — Grafana (Core :3002) +- `/uptime/` — Uptime Kuma (Core :3001) +- `/gitea/` — Gitea (app2 :3000) +- `/n8n/` — n8n (app1 :5678) +- `/search/` — Super Search / OSINT (app1 :8100) +- `/noc/` — Mattermost (future, app1) + +### core.itpropartner.com (152.53.192.33) — Minimal Landing +Stripped down — just links to ops, my, and other services. Keep Core lean for Hermes. + +### Standalone (no changes) +- vault.itpropartner.com → 152.53.36.131 (app1, after migration) +- admin-ai.itpropartner.com → 152.53.36.131 (app1) +- hudu.itpropartner.com → 152.53.39.202 (app2) +- panel.itpropartner.com → 152.53.241.111 (app3) + +### Other Domains +- fleettracker360.com → Cloudflare DNS +- voipsimplicity.com → Cloudflare DNS