feat: initial itpp-docs setup with MkDocs Material

- mkdocs.yml with dark slate theme, nav for 12 ITPP projects - build-docs.sh aggregates docs from all project repos - .gitea/workflows/docs-publish.yml for nightly rebuild+deploy - README and CHANGELOG for the itpp-docs repo itself - docs-source/ populated from all 12 repos - site/ ready for deployment to docs.itpropartner.com
2026-08-09 23:49:45 -04:00
parent 08ba2e62a8
commit 98073dc636
134 changed files with 55534 additions and 2 deletions
@@ -0,0 +1,278 @@
+# itpp-infrastructure
+
+
+> **Last Updated:** July 17, 2026
+> **Maintainer:** Sho'Nuff
+
+---
+
+## Server Inventory
+
+### Core Server
+- **Hostname:** Core
+- **IP:** 152.53.192.33
+- **Provider:** netcup RS 2000 G12
+- **Specs:** 8 vCPU EPYC 9645, 15 GB DDR5 ECC, 512 GB NVMe
+- **OS:** Debian 13
+- **Role:** Hermes + Portals
+- **Key Services:**
+  - Hermes Agent (Telegram + cron, 22 cron jobs)
+  - Caddy reverse proxy (12 domains, auto-TLS)
+  - Ops Portal (FastAPI, port 8090)
+  - Prometheus (native, port 9090) + Grafana (native, port 3002)
+  - Uptime Kuma (Docker, port 3001) — 9+ monitors
+  - Vaultwarden (Docker, port 8080) — vault.iamgmb.com
+  - Twenty CRM (Docker) — crm.debtrecoveryexperts.com
+  - DocuSeal (Docker, port 3000) — sign.core.itpropartner.com
+  - SearXNG (Docker, port 8888)
+  - Komodo (Docker, port 9120)
+  - Tailscale, StrongSwan, WireGuard (home CCR tunnel 10.77.0.0/24)
+  - Redis cache
+
+### App1 Server
+- **Hostname:** app1
+- **IP:** 152.53.36.131
+- **Provider:** netcup RS 4000 G12
+- **Specs:** 12 vCPU EPYC 9645, 32 GB DDR5 ECC, 1 TB NVMe
+- **OS:** Debian 13
+- **Role:** AI/Service Hub
+- **Key Services:**
+  - Open WebUI (Docker, port 3000) — ai.itpropartner.com
+  - n8n + Postgres (Docker, port 5678) — n8n.itpropartner.com
+  - LiteLLM (Docker) + Postgres — admin-ai.itpropartner.com
+  - Mattermost Team Edition (Docker, port 8065) — noc.itpropartner.com
+  - Caddy (systemd, 80/443)
+  - 4 MCP servers: Browser (:8901), Filesystem (:8900), Email (:8902), Git (:8903)
+  - Super Search MCP (systemd, port 8899)
+
+### App2 Server
+- **Hostname:** app2
+- **IP:** 152.53.39.202
+- **Provider:** netcup RS 4000 G12
+- **Specs:** 12 vCPU EPYC 9645, 32 GB DDR5 ECC, 1 TB NVMe
+- **OS:** Debian 13
+- **Role:** Infrastructure Server
+- **Key Services:**
+  - Traccar GPS (Docker, port 8082 + 5000-5150) — fleettracker360.com
+  - UniFi Controller (Docker, port 8443) — unifi.itpropartner.com
+  - UNMS/UISP (10 Docker containers) — unms.forefrontwireless.com
+  - Hudu (Docker) — hudu.itpropartner.com
+  - Caddy (4 domains)
+
+### App3 Server
+- **Hostname:** app3
+- **IP:** 152.53.241.111
+- **Provider:** netcup RS 4000 G12
+- **Specs:** 12 vCPU EPYC 9645, 32 GB DDR5 ECC, 1 TB NVMe
+- **OS:** Debian 13
+- **Role:** Web Hosting + Backup Restore
+- **Key Services:**
+  - CloudPanel CE — panel.itpropartner.com
+  - Nginx (80/443) + Percona MySQL 8.4 + PHP 8.3
+  - Backup Restore System (Flask, port 8090) — my.itpropartner.com/backups
+  - WordPress sites (7 migrated from wphost02, all live):
+    - debtreecoveryexperts.com, boxpilotlogistics.com, iamgmb.com
+    - katiewattsdesign.com, vigilanttac.com, apextrackexperience.com
+    - mainwp.itpropartner.com, voipsimplicity.com, my.voipsimplicity.com
+  - Daily snapshots: 1 AM + 1 PM, 60-day retention, /opt/backup-restore/snapshots
+
+### Core-BU (Warm Standby)
+- **Hostname:** core-bu
+- **IP:** 5.161.225.131
+- **Provider:** Hetzner CPX21
+- **Specs:** 3 vCPU, 4 GB RAM, 80 GB SSD
+- **Role:** Warm standby — auto-failover if Core down
+- **Watchdog:** 5-min check, 4-cycle confirmation, S3 sync every 10 min
+
+### Legacy / Decommissioned
+- **old-ai:** 178.156.167.181 (Hetzner CPX41) — **decommissioned** (LiteLLM migrated to app1)
+- **old app1:** 87.99.144.163 (Hetzner CPX11) — **deleted**
+- **wphost02:** 5.161.62.38 (Hetzner CPX21) — **migrated to app3**
+- **Ollama:** Removed from Core (systemd) and app1 (Docker) Jul 17
+
+---
+
+## Model Fallback Chain
+
+All providers use direct API keys. GPT-5.5 quality survives through admin-ai → OpenRouter, then degrades through DeepSeek → Gemini → Grok.
+
+| # | Model | Provider | Gateway |
+|---|---|---|---|
+| Primary | GPT-5.5 | admin-ai | Self-hosted LiteLLM (app1) |
+| Fallback 1 | GPT-5.5 | OpenRouter | openrouter.ai |
+| Fallback 2 | DeepSeek v4 Pro | DeepSeek | api.deepseek.com |
+| Fallback 3 | Gemini 3.5 Flash | Google | generativelanguage.googleapis.com |
+| Fallback 4 | Grok 4.5 | xAI | api.x.ai |
+
+**Credits (Jul 17):** DeepSeek $58, OpenRouter ~$30 remaining, OpenAI/xAI/Google on pay-as-you-go
+**Health check:** Daily 8 AM cron (`model-usage-check`)
+
+---
+
+## Domain / DNS Map
+
+### ⚠️ itpropartner.com — SiteGround Nameservers Only
+
+`itpropartner.com` uses **SiteGround nameservers** exclusively. A Cloudflare zone exists (`0dc20632…`) but is NOT authoritative — records created there silently fail. All `*.itpropartner.com` changes must be manual through SiteGround panel.
+
+| Domain | IP | Server | Service |
+|---|---|---|---|
+| core.itpropartner.com | 152.53.192.33 | Core | Landing page + Grafana link |
+| ops.itpropartner.com | 152.53.192.33 | Core | Ops dashboard |
+| sign.core.itpropartner.com | 152.53.192.33 | Core | DocuSeal |
+| uptimekuma.itpropartner.com | 152.53.192.33 | Core | Uptime monitoring |
+| gps.fleettracker360.com | 152.53.192.33 | Core | Traccar HTTPS proxy → app2 |
+| my.itpropartner.com | 152.53.192.33 | Core | Customer portal hub |
+| hudu.itpropartner.com | 152.53.39.202 | app2 | IT documentation |
+| unifi.itpropartner.com | 152.53.39.202 | app2 | UniFi controller |
+| panel.itpropartner.com | 152.53.241.111 | app3 | CloudPanel CE |
+| ai.itpropartner.com | 152.53.36.131 | app1 | Open WebUI |
+| n8n.itpropartner.com | 152.53.36.131 | app1 | n8n automation |
+| admin-ai.itpropartner.com | 152.53.36.131 | app1 | LiteLLM |
+
+### Cloudflare-Managed Domains
+
+| Domain | IP | Server | Service |
+|---|---|---|---|
+| fleettracker360.com | Cloudflare | app2 | Fleet tracking website |
+| gps.fleettracker360.com | Cloudflare → Core | Core → app2 | Traccar devices |
+| voipsimplicity.com | Cloudflare | app3 | VoIP marketing site |
+| my.voipsimplicity.com | Cloudflare | app3 | VoIP customer portal |
+| portal.debtrecoveryexperts.com | 152.53.192.33 | Core | DRE portal |
+| crm.debtrecoveryexperts.com | Cloudflare Access | — | DRE CRM |
+| vault.iamgmb.com | 152.53.192.33 | Core | Vaultwarden |
+| sign.iamgmb.com | 152.53.192.33 | Core | Document signing |
+| shark.iamgmb.com | 152.53.192.33 | Core | Shark game |
+
+### DNS PENDING (create at SiteGround)
+
+| Subdomain | → IP | Service |
+|---|---|---|
+| vault.itpropartner.com | 152.53.36.131 | Vaultwarden (after migration) |
+| status.itpropartner.com | 152.53.192.33 | Public status page |
+
+---
+
+## Backup Pipeline
+
+| Backup | Schedule | Target | Purpose |
+|---|---|---|---|
+| hermes-live-sync | Every 15 min | s3://hermes-vps-backups/live/ | Live state sync |
+| hermes-full-backup | Daily 1 AM | s3://hermes-vps-backups/hermes-full-backup/ | Full Hermes backup |
+| home-router-backup | Daily 6 AM | s3://mikrotik-ccr-backups/ | CCR config |
+| root-essentials-backup | Daily 3 AM | S3 | /root essentials |
+| docker-volume-sync | Daily 3 AM | S3 | Docker volumes |
+| system-config-sync | Daily 4 AM | S3 | System configs |
+| unms-backup-sync | Daily 6 AM (Core) | s3://hermes-vps-backups/unms-backups/ | UNMS data (pulled from app2) |
+| unifi-backup-sync | Daily 2 AM (Core) | s3://hermes-vps-backups/unifi-backups/ | UniFi configs (pulled from app2) |
+| hudu-backup | Daily 7 AM | s3://hermes-vps-backups/hudu/backups/ | Hudu volume dump |
+| gitea-backup | Daily 8 AM | s3://hermes-vps-backups/gitea/daily/ | Gitea repos |
+| app1-backup | Daily 2 AM | s3://hermes-vps-backups/app1/ | LiteLLM, n8n, OpenWebUI, MCP, Mattermost |
+| app2-backup | Daily 2:30 AM | s3://hermes-vps-backups/app2/ | Traccar, Gitea, Hudu, UNMS, UniFi |
+| app3-backup | Daily 3 AM | s3://hermes-vps-backups/app3/ | CloudPanel, MySQL, WordPress |
+| wphost02-backup | Daily 5 AM | s3://hermes-vps-backups/wphost02-backup/ | Webapps + MySQL |
+| warm-standby-sync | Every 10 min | core-bu ← S3 | DR readiness |
+
+---
+
+## Caddy Configuration
+
+### Core (152.53.192.33)
+
+```
+core.itpropartner.com           → static files
+sign.core.itpropartner.com      → localhost:3000 (DocuSeal)
+ops.itpropartner.com            → 127.0.0.1:8090 + static
+uptimekuma.itpropartner.com     → 127.0.0.1:3001 (Uptime Kuma)
+gps.fleettracker360.com         → app2:8082 (Traccar)
+my.itpropartner.com             → static files
+portal.debtrecoveryexperts.com  → static files
+vault.iamgmb.com                → localhost:8080 (Vaultwarden)
+sign.iamgmb.com                 → 127.0.0.1:8090
+shark.iamgmb.com                → static + :8083
+```
+
+### App1 (152.53.36.131)
+
+```
+ai.itpropartner.com             → :3000 (Open WebUI)
+n8n.itpropartner.com            → :5678 (n8n)
+admin-ai.itpropartner.com       → :4000 (LiteLLM)
+app1.itpropartner.com           → static response
+```
+
+### App2 (152.53.39.202)
+
+```
+hudu.itpropartner.com           → Hudu internal
+gps.fleettracker360.com         → :8082 (Traccar)
+unms.forefrontwireless.com      → UNMS Nginx
+unifi.itpropartner.com          → :8443 (UniFi)
+```
+
+---
+
+## Service Access
+
+| Service | URL | Location | Auth |
+|---|---|---|---|
+| Open WebUI | https://ai.itpropartner.com | app1 | Chat UI |
+| Open WebUI Admin | https://admin-ai.itpropartner.com/ui | app1 | user: admin, pw: LITELLM_MASTER_KEY |
+| Ops Portal | https://ops.itpropartner.com | Core | Internal dashboard |
+| Grafana | http://core.itpropartner.com:3002 | Core | admin/admin |
+| Uptime Kuma | https://uptimekuma.itpropartner.com | Core | Service monitoring |
+| Vaultwarden | https://vault.iamgmb.com | Core | Password vault |
+| CloudPanel | https://panel.itpropartner.com | app3 | user: gmb / SQLite auth |
+| Traccar | https://gps.fleettracker360.com | app2 | GPS fleet tracking |
+| UniFi | https://unifi.itpropartner.com | app2 | Network controller |
+| UNMS | https://unms.forefrontwireless.com | app2 | WISP management |
+| Hudu | https://hudu.itpropartner.com | app2 | IT documentation |
+| n8n | https://n8n.itpropartner.com | app1 | Automation |
+| CRM (DRE) | https://crm.debtrecoveryexperts.com | Cloudflare Access | TwentyCRM |
+
+### MCP Access (from Open WebUI)
+
+| MCP Server | Location | Port | Tools |
+|---|---|---|---|
+| Super Search | app1 | :8899 | 10 tools — web_search, web_extract, person_search, email_search, phone_search, etc. |
+| Browser | app1 | :8901 | browser_navigate, browser_snapshot, browser_click, browser_type, browser_console |
+| Filesystem | app1 | :8900 | read_file, write_file, search_files, list_dir, file_info |
+| Email | app1 | :8902 | search_emails, send_email, get_email |
+| Git/Gitea | app1 | :8903 | clone, commit, push, pull |
+
+---
+
+## SSH Access
+
+- **Key:** `itpp-infra` (deployed to all servers)
+- **User:** `ippadmin` (sudo privileges)
+- **Root SSH:** Enabled on app1, app2, app3 (key-only exception per provisioning standard)
+- **Core SSH:** `ssh -i /root/.ssh/itpp-infra root@152.53.192.33`
+
+---
+
+## Firewall
+
+UFW is enabled on all servers. Standard rules:
+- **Core:** 22, 80, 443, 3000, 3001, 3002, 8080, 8082, 8090, 8443, 9090
+- **app1:** 22, 80, 443, 3000, 5678, 8899, 8900, 8901, 8902, 8903
+- **app2:** 22, 80, 443, 3000, 8080, 8082, 8089, 8443, 8843, 3478, 10001, 5000:5150
+- **app3:** 22, 80, 443, 8443
+
+---
+
+## SSL
+
+All SSL certificates issued via Let's Encrypt through Caddy. All certs auto-renew. No manual management needed.
+
+---
+
+## Notes
+
+- **DNS trap:** `itpropartner.com` uses SiteGround nameservers. Cloudflare zone is NOT authoritative. Always verify with `dig NS domain.com` before creating records.
+- **SiteGround:** No API access. All DNS changes are manual through SiteGround panel.
+- **Provider diversity:** core-bu stays at Hetzner specifically so a netcup outage can't kill both Core and standby simultaneously.
+- **app3 MySQL:** Root password in Vaultwarden + `/root/.my.cnf` on app3, accessible via 127.0.0.1:3306.
+- **CloudPanel:** SQLite DB at `/home/clp/htdocs/app/data/db.sq3` — users live here, not in MySQL.
+- **AWS CLI PATH:** All backup scripts must use `/opt/awscli-venv/bin/aws` or `source /opt/awscli-venv/bin/activate` — `aws` bare fails in cron context (PATH doesn't include venv bin). Documented in server-provisioning-standard v1.3.0.
+- **Backup verification:** Always run at least one manual backup after provisioning a server and verify it landed in S3 — never trust cron entries alone. Silent failures (`aws: command not found`, wrong file paths, S3 permission issues) won't surface otherwise.