ippadmin/itpp-docs
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
98073dc636a56e8ef20c4e90d898e733b40f4ab6
itpp-docs/docs-source/itpp-infrastructure/index.md
T
Germaine Brown 98073dc636
Publish Docs Site / build (push) Failing after 5s
Details
feat: initial itpp-docs setup with MkDocs Material 
- mkdocs.yml with dark slate theme, nav for 12 ITPP projects
- build-docs.sh aggregates docs from all project repos
- .gitea/workflows/docs-publish.yml for nightly rebuild+deploy
- README and CHANGELOG for the itpp-docs repo itself
- docs-source/ populated from all 12 repos
- site/ ready for deployment to docs.itpropartner.com
2026-08-09 23:49:45 -04:00

279 lines
12 KiB
Markdown
Raw Blame History

# itpp-infrastructure
> **Last Updated:** July 17, 2026
> **Maintainer:** Sho'Nuff
---
## Server Inventory
### Core Server
- **Hostname:** Core
- **IP:** 152.53.192.33
- **Provider:** netcup RS 2000 G12
- **Specs:** 8 vCPU EPYC 9645, 15 GB DDR5 ECC, 512 GB NVMe
- **OS:** Debian 13
- **Role:** Hermes + Portals
- **Key Services:**
- Hermes Agent (Telegram + cron, 22 cron jobs)
- Caddy reverse proxy (12 domains, auto-TLS)
- Ops Portal (FastAPI, port 8090)
- Prometheus (native, port 9090) + Grafana (native, port 3002)
- Uptime Kuma (Docker, port 3001) — 9+ monitors
- Vaultwarden (Docker, port 8080) — vault.iamgmb.com
- Twenty CRM (Docker) — crm.debtrecoveryexperts.com
- DocuSeal (Docker, port 3000) — sign.core.itpropartner.com
- SearXNG (Docker, port 8888)
- Komodo (Docker, port 9120)
- Tailscale, StrongSwan, WireGuard (home CCR tunnel 10.77.0.0/24)
- Redis cache
### App1 Server
- **Hostname:** app1
- **IP:** 152.53.36.131
- **Provider:** netcup RS 4000 G12
- **Specs:** 12 vCPU EPYC 9645, 32 GB DDR5 ECC, 1 TB NVMe
- **OS:** Debian 13
- **Role:** AI/Service Hub
- **Key Services:**
- Open WebUI (Docker, port 3000) — ai.itpropartner.com
- n8n + Postgres (Docker, port 5678) — n8n.itpropartner.com
- LiteLLM (Docker) + Postgres — admin-ai.itpropartner.com
- Mattermost Team Edition (Docker, port 8065) — noc.itpropartner.com
- Caddy (systemd, 80/443)
- 4 MCP servers: Browser (:8901), Filesystem (:8900), Email (:8902), Git (:8903)
- Super Search MCP (systemd, port 8899)
### App2 Server
- **Hostname:** app2
- **IP:** 152.53.39.202
- **Provider:** netcup RS 4000 G12
- **Specs:** 12 vCPU EPYC 9645, 32 GB DDR5 ECC, 1 TB NVMe
- **OS:** Debian 13
- **Role:** Infrastructure Server
- **Key Services:**
- Traccar GPS (Docker, port 8082 + 5000-5150) — fleettracker360.com
- UniFi Controller (Docker, port 8443) — unifi.itpropartner.com
- UNMS/UISP (10 Docker containers) — unms.forefrontwireless.com
- Hudu (Docker) — hudu.itpropartner.com
- Caddy (4 domains)
### App3 Server
- **Hostname:** app3
- **IP:** 152.53.241.111
- **Provider:** netcup RS 4000 G12
- **Specs:** 12 vCPU EPYC 9645, 32 GB DDR5 ECC, 1 TB NVMe
- **OS:** Debian 13
- **Role:** Web Hosting + Backup Restore
- **Key Services:**
- CloudPanel CE — panel.itpropartner.com
- Nginx (80/443) + Percona MySQL 8.4 + PHP 8.3
- Backup Restore System (Flask, port 8090) — my.itpropartner.com/backups
- WordPress sites (7 migrated from wphost02, all live):
- debtreecoveryexperts.com, boxpilotlogistics.com, iamgmb.com
- katiewattsdesign.com, vigilanttac.com, apextrackexperience.com
- mainwp.itpropartner.com, voipsimplicity.com, my.voipsimplicity.com
- Daily snapshots: 1 AM + 1 PM, 60-day retention, /opt/backup-restore/snapshots
### Core-BU (Warm Standby)
- **Hostname:** core-bu
- **IP:** 5.161.225.131
- **Provider:** Hetzner CPX21
- **Specs:** 3 vCPU, 4 GB RAM, 80 GB SSD
- **Role:** Warm standby — auto-failover if Core down
- **Watchdog:** 5-min check, 4-cycle confirmation, S3 sync every 10 min
### Legacy / Decommissioned
- **old-ai:** 178.156.167.181 (Hetzner CPX41) — **decommissioned** (LiteLLM migrated to app1)
- **old app1:** 87.99.144.163 (Hetzner CPX11) — **deleted**
- **wphost02:** 5.161.62.38 (Hetzner CPX21) — **migrated to app3**
- **Ollama:** Removed from Core (systemd) and app1 (Docker) Jul 17
---
## Model Fallback Chain
All providers use direct API keys. GPT-5.5 quality survives through admin-ai → OpenRouter, then degrades through DeepSeek → Gemini → Grok.
| # | Model | Provider | Gateway |
|---|---|---|---|
| Primary | GPT-5.5 | admin-ai | Self-hosted LiteLLM (app1) |
| Fallback 1 | GPT-5.5 | OpenRouter | openrouter.ai |
| Fallback 2 | DeepSeek v4 Pro | DeepSeek | api.deepseek.com |
| Fallback 3 | Gemini 3.5 Flash | Google | generativelanguage.googleapis.com |
| Fallback 4 | Grok 4.5 | xAI | api.x.ai |
**Credits (Jul 17):** DeepSeek $58, OpenRouter ~$30 remaining, OpenAI/xAI/Google on pay-as-you-go
**Health check:** Daily 8 AM cron (`model-usage-check`)
---
## Domain / DNS Map
### ⚠️ itpropartner.com — SiteGround Nameservers Only
`itpropartner.com` uses **SiteGround nameservers** exclusively. A Cloudflare zone exists (`0dc20632…`) but is NOT authoritative — records created there silently fail. All `*.itpropartner.com` changes must be manual through SiteGround panel.
| Domain | IP | Server | Service |
|---|---|---|---|
| core.itpropartner.com | 152.53.192.33 | Core | Landing page + Grafana link |
| ops.itpropartner.com | 152.53.192.33 | Core | Ops dashboard |
| sign.core.itpropartner.com | 152.53.192.33 | Core | DocuSeal |
| uptimekuma.itpropartner.com | 152.53.192.33 | Core | Uptime monitoring |
| gps.fleettracker360.com | 152.53.192.33 | Core | Traccar HTTPS proxy → app2 |
| my.itpropartner.com | 152.53.192.33 | Core | Customer portal hub |
| hudu.itpropartner.com | 152.53.39.202 | app2 | IT documentation |
| unifi.itpropartner.com | 152.53.39.202 | app2 | UniFi controller |
| panel.itpropartner.com | 152.53.241.111 | app3 | CloudPanel CE |
| ai.itpropartner.com | 152.53.36.131 | app1 | Open WebUI |
| n8n.itpropartner.com | 152.53.36.131 | app1 | n8n automation |
| admin-ai.itpropartner.com | 152.53.36.131 | app1 | LiteLLM |
### Cloudflare-Managed Domains
| Domain | IP | Server | Service |
|---|---|---|---|
| fleettracker360.com | Cloudflare | app2 | Fleet tracking website |
| gps.fleettracker360.com | Cloudflare → Core | Core → app2 | Traccar devices |
| voipsimplicity.com | Cloudflare | app3 | VoIP marketing site |
| my.voipsimplicity.com | Cloudflare | app3 | VoIP customer portal |
| portal.debtrecoveryexperts.com | 152.53.192.33 | Core | DRE portal |
| crm.debtrecoveryexperts.com | Cloudflare Access | — | DRE CRM |
| vault.iamgmb.com | 152.53.192.33 | Core | Vaultwarden |
| sign.iamgmb.com | 152.53.192.33 | Core | Document signing |
| shark.iamgmb.com | 152.53.192.33 | Core | Shark game |
### DNS PENDING (create at SiteGround)
| Subdomain | → IP | Service |
|---|---|---|
| vault.itpropartner.com | 152.53.36.131 | Vaultwarden (after migration) |
| status.itpropartner.com | 152.53.192.33 | Public status page |
---
## Backup Pipeline
| Backup | Schedule | Target | Purpose |
|---|---|---|---|
| hermes-live-sync | Every 15 min | s3://hermes-vps-backups/live/ | Live state sync |
| hermes-full-backup | Daily 1 AM | s3://hermes-vps-backups/hermes-full-backup/ | Full Hermes backup |
| home-router-backup | Daily 6 AM | s3://mikrotik-ccr-backups/ | CCR config |
| root-essentials-backup | Daily 3 AM | S3 | /root essentials |
| docker-volume-sync | Daily 3 AM | S3 | Docker volumes |
| system-config-sync | Daily 4 AM | S3 | System configs |
| unms-backup-sync | Daily 6 AM (Core) | s3://hermes-vps-backups/unms-backups/ | UNMS data (pulled from app2) |
| unifi-backup-sync | Daily 2 AM (Core) | s3://hermes-vps-backups/unifi-backups/ | UniFi configs (pulled from app2) |
| hudu-backup | Daily 7 AM | s3://hermes-vps-backups/hudu/backups/ | Hudu volume dump |
| gitea-backup | Daily 8 AM | s3://hermes-vps-backups/gitea/daily/ | Gitea repos |
| app1-backup | Daily 2 AM | s3://hermes-vps-backups/app1/ | LiteLLM, n8n, OpenWebUI, MCP, Mattermost |
| app2-backup | Daily 2:30 AM | s3://hermes-vps-backups/app2/ | Traccar, Gitea, Hudu, UNMS, UniFi |
| app3-backup | Daily 3 AM | s3://hermes-vps-backups/app3/ | CloudPanel, MySQL, WordPress |
| wphost02-backup | Daily 5 AM | s3://hermes-vps-backups/wphost02-backup/ | Webapps + MySQL |
| warm-standby-sync | Every 10 min | core-bu ← S3 | DR readiness |
---
## Caddy Configuration
### Core (152.53.192.33)
```
core.itpropartner.com → static files
sign.core.itpropartner.com → localhost:3000 (DocuSeal)
ops.itpropartner.com → 127.0.0.1:8090 + static
uptimekuma.itpropartner.com → 127.0.0.1:3001 (Uptime Kuma)
gps.fleettracker360.com → app2:8082 (Traccar)
my.itpropartner.com → static files
portal.debtrecoveryexperts.com → static files
vault.iamgmb.com → localhost:8080 (Vaultwarden)
sign.iamgmb.com → 127.0.0.1:8090
shark.iamgmb.com → static + :8083
```
### App1 (152.53.36.131)
```
ai.itpropartner.com → :3000 (Open WebUI)
n8n.itpropartner.com → :5678 (n8n)
admin-ai.itpropartner.com → :4000 (LiteLLM)
app1.itpropartner.com → static response
```
### App2 (152.53.39.202)
```
hudu.itpropartner.com → Hudu internal
gps.fleettracker360.com → :8082 (Traccar)
unms.forefrontwireless.com → UNMS Nginx
unifi.itpropartner.com → :8443 (UniFi)
```
---
## Service Access
| Service | URL | Location | Auth |
|---|---|---|---|
| Open WebUI | https://ai.itpropartner.com | app1 | Chat UI |
| Open WebUI Admin | https://admin-ai.itpropartner.com/ui | app1 | user: admin, pw: LITELLM_MASTER_KEY |
| Ops Portal | https://ops.itpropartner.com | Core | Internal dashboard |
| Grafana | http://core.itpropartner.com:3002 | Core | admin/admin |
| Uptime Kuma | https://uptimekuma.itpropartner.com | Core | Service monitoring |
| Vaultwarden | https://vault.iamgmb.com | Core | Password vault |
| CloudPanel | https://panel.itpropartner.com | app3 | user: gmb / SQLite auth |
| Traccar | https://gps.fleettracker360.com | app2 | GPS fleet tracking |
| UniFi | https://unifi.itpropartner.com | app2 | Network controller |
| UNMS | https://unms.forefrontwireless.com | app2 | WISP management |
| Hudu | https://hudu.itpropartner.com | app2 | IT documentation |
| n8n | https://n8n.itpropartner.com | app1 | Automation |
| CRM (DRE) | https://crm.debtrecoveryexperts.com | Cloudflare Access | TwentyCRM |
### MCP Access (from Open WebUI)
| MCP Server | Location | Port | Tools |
|---|---|---|---|
| Super Search | app1 | :8899 | 10 tools — web_search, web_extract, person_search, email_search, phone_search, etc. |
| Browser | app1 | :8901 | browser_navigate, browser_snapshot, browser_click, browser_type, browser_console |
| Filesystem | app1 | :8900 | read_file, write_file, search_files, list_dir, file_info |
| Email | app1 | :8902 | search_emails, send_email, get_email |
| Git/Gitea | app1 | :8903 | clone, commit, push, pull |
---
## SSH Access
- **Key:** `itpp-infra` (deployed to all servers)
- **User:** `ippadmin` (sudo privileges)
- **Root SSH:** Enabled on app1, app2, app3 (key-only exception per provisioning standard)
- **Core SSH:** `ssh -i /root/.ssh/itpp-infra root@152.53.192.33`
---
## Firewall
UFW is enabled on all servers. Standard rules:
- **Core:** 22, 80, 443, 3000, 3001, 3002, 8080, 8082, 8090, 8443, 9090
- **app1:** 22, 80, 443, 3000, 5678, 8899, 8900, 8901, 8902, 8903
- **app2:** 22, 80, 443, 3000, 8080, 8082, 8089, 8443, 8843, 3478, 10001, 5000:5150
- **app3:** 22, 80, 443, 8443
---
## SSL
All SSL certificates issued via Let's Encrypt through Caddy. All certs auto-renew. No manual management needed.
---
## Notes
- **DNS trap:** `itpropartner.com` uses SiteGround nameservers. Cloudflare zone is NOT authoritative. Always verify with `dig NS domain.com` before creating records.
- **SiteGround:** No API access. All DNS changes are manual through SiteGround panel.
- **Provider diversity:** core-bu stays at Hetzner specifically so a netcup outage can't kill both Core and standby simultaneously.
- **app3 MySQL:** Root password in Vaultwarden + `/root/.my.cnf` on app3, accessible via 127.0.0.1:3306.
- **CloudPanel:** SQLite DB at `/home/clp/htdocs/app/data/db.sq3` — users live here, not in MySQL.
- **AWS CLI PATH:** All backup scripts must use `/opt/awscli-venv/bin/aws` or `source /opt/awscli-venv/bin/activate` — `aws` bare fails in cron context (PATH doesn't include venv bin). Documented in server-provisioning-standard v1.3.0.
- **Backup verification:** Always run at least one manual backup after provisioning a server and verify it landed in S3 — never trust cron entries alone. Silent failures (`aws: command not found`, wrong file paths, S3 permission issues) won't surface otherwise.
Reference in New Issue View Git Blame Copy Permalink