ippadmin/itpp-docs
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
98073dc636a56e8ef20c4e90d898e733b40f4ab6
itpp-docs/docs-source/itpp-infrastructure/index.md
T
Germaine Brown 98073dc636
Publish Docs Site / build (push) Failing after 5s
Details
feat: initial itpp-docs setup with MkDocs Material 
- mkdocs.yml with dark slate theme, nav for 12 ITPP projects
- build-docs.sh aggregates docs from all project repos
- .gitea/workflows/docs-publish.yml for nightly rebuild+deploy
- README and CHANGELOG for the itpp-docs repo itself
- docs-source/ populated from all 12 repos
- site/ ready for deployment to docs.itpropartner.com
2026-08-09 23:49:45 -04:00

12 KiB
Raw Blame History

itpp-infrastructure

Last Updated: July 17, 2026 Maintainer: Sho'Nuff

Server Inventory

Core Server

  • Hostname: Core
  • IP: 152.53.192.33
  • Provider: netcup RS 2000 G12
  • Specs: 8 vCPU EPYC 9645, 15 GB DDR5 ECC, 512 GB NVMe
  • OS: Debian 13
  • Role: Hermes + Portals
  • Key Services:
    • Hermes Agent (Telegram + cron, 22 cron jobs)
    • Caddy reverse proxy (12 domains, auto-TLS)
    • Ops Portal (FastAPI, port 8090)
    • Prometheus (native, port 9090) + Grafana (native, port 3002)
    • Uptime Kuma (Docker, port 3001) — 9+ monitors
    • Vaultwarden (Docker, port 8080) — vault.iamgmb.com
    • Twenty CRM (Docker) — crm.debtrecoveryexperts.com
    • DocuSeal (Docker, port 3000) — sign.core.itpropartner.com
    • SearXNG (Docker, port 8888)
    • Komodo (Docker, port 9120)
    • Tailscale, StrongSwan, WireGuard (home CCR tunnel 10.77.0.0/24)
    • Redis cache

App1 Server

  • Hostname: app1
  • IP: 152.53.36.131
  • Provider: netcup RS 4000 G12
  • Specs: 12 vCPU EPYC 9645, 32 GB DDR5 ECC, 1 TB NVMe
  • OS: Debian 13
  • Role: AI/Service Hub
  • Key Services:
    • Open WebUI (Docker, port 3000) — ai.itpropartner.com
    • n8n + Postgres (Docker, port 5678) — n8n.itpropartner.com
    • LiteLLM (Docker) + Postgres — admin-ai.itpropartner.com
    • Mattermost Team Edition (Docker, port 8065) — noc.itpropartner.com
    • Caddy (systemd, 80/443)
    • 4 MCP servers: Browser (:8901), Filesystem (:8900), Email (:8902), Git (:8903)
    • Super Search MCP (systemd, port 8899)

App2 Server

  • Hostname: app2
  • IP: 152.53.39.202
  • Provider: netcup RS 4000 G12
  • Specs: 12 vCPU EPYC 9645, 32 GB DDR5 ECC, 1 TB NVMe
  • OS: Debian 13
  • Role: Infrastructure Server
  • Key Services:
    • Traccar GPS (Docker, port 8082 + 5000-5150) — fleettracker360.com
    • UniFi Controller (Docker, port 8443) — unifi.itpropartner.com
    • UNMS/UISP (10 Docker containers) — unms.forefrontwireless.com
    • Hudu (Docker) — hudu.itpropartner.com
    • Caddy (4 domains)

App3 Server

  • Hostname: app3
  • IP: 152.53.241.111
  • Provider: netcup RS 4000 G12
  • Specs: 12 vCPU EPYC 9645, 32 GB DDR5 ECC, 1 TB NVMe
  • OS: Debian 13
  • Role: Web Hosting + Backup Restore
  • Key Services:
    • CloudPanel CE — panel.itpropartner.com
    • Nginx (80/443) + Percona MySQL 8.4 + PHP 8.3
    • Backup Restore System (Flask, port 8090) — my.itpropartner.com/backups
    • WordPress sites (7 migrated from wphost02, all live):
      • debtreecoveryexperts.com, boxpilotlogistics.com, iamgmb.com
      • katiewattsdesign.com, vigilanttac.com, apextrackexperience.com
      • mainwp.itpropartner.com, voipsimplicity.com, my.voipsimplicity.com
    • Daily snapshots: 1 AM + 1 PM, 60-day retention, /opt/backup-restore/snapshots

Core-BU (Warm Standby)

  • Hostname: core-bu
  • IP: 5.161.225.131
  • Provider: Hetzner CPX21
  • Specs: 3 vCPU, 4 GB RAM, 80 GB SSD
  • Role: Warm standby — auto-failover if Core down
  • Watchdog: 5-min check, 4-cycle confirmation, S3 sync every 10 min

Legacy / Decommissioned

  • old-ai: 178.156.167.181 (Hetzner CPX41) — decommissioned (LiteLLM migrated to app1)
  • old app1: 87.99.144.163 (Hetzner CPX11) — deleted
  • wphost02: 5.161.62.38 (Hetzner CPX21) — migrated to app3
  • Ollama: Removed from Core (systemd) and app1 (Docker) Jul 17

Model Fallback Chain

All providers use direct API keys. GPT-5.5 quality survives through admin-ai → OpenRouter, then degrades through DeepSeek → Gemini → Grok.

# Model Provider Gateway
Primary GPT-5.5 admin-ai Self-hosted LiteLLM (app1)
Fallback 1 GPT-5.5 OpenRouter openrouter.ai
Fallback 2 DeepSeek v4 Pro DeepSeek api.deepseek.com
Fallback 3 Gemini 3.5 Flash Google generativelanguage.googleapis.com
Fallback 4 Grok 4.5 xAI api.x.ai

Credits (Jul 17): DeepSeek $58, OpenRouter ~$30 remaining, OpenAI/xAI/Google on pay-as-you-go Health check: Daily 8 AM cron (model-usage-check)

Domain / DNS Map

⚠️ itpropartner.com — SiteGround Nameservers Only

itpropartner.com uses SiteGround nameservers exclusively. A Cloudflare zone exists (0dc20632…) but is NOT authoritative — records created there silently fail. All *.itpropartner.com changes must be manual through SiteGround panel.

Domain IP Server Service
core.itpropartner.com 152.53.192.33 Core Landing page + Grafana link
ops.itpropartner.com 152.53.192.33 Core Ops dashboard
sign.core.itpropartner.com 152.53.192.33 Core DocuSeal
uptimekuma.itpropartner.com 152.53.192.33 Core Uptime monitoring
gps.fleettracker360.com 152.53.192.33 Core Traccar HTTPS proxy → app2
my.itpropartner.com 152.53.192.33 Core Customer portal hub
hudu.itpropartner.com 152.53.39.202 app2 IT documentation
unifi.itpropartner.com 152.53.39.202 app2 UniFi controller
panel.itpropartner.com 152.53.241.111 app3 CloudPanel CE
ai.itpropartner.com 152.53.36.131 app1 Open WebUI
n8n.itpropartner.com 152.53.36.131 app1 n8n automation
admin-ai.itpropartner.com 152.53.36.131 app1 LiteLLM

Cloudflare-Managed Domains

Domain IP Server Service
fleettracker360.com Cloudflare app2 Fleet tracking website
gps.fleettracker360.com Cloudflare → Core Core → app2 Traccar devices
voipsimplicity.com Cloudflare app3 VoIP marketing site
my.voipsimplicity.com Cloudflare app3 VoIP customer portal
portal.debtrecoveryexperts.com 152.53.192.33 Core DRE portal
crm.debtrecoveryexperts.com Cloudflare Access — DRE CRM
vault.iamgmb.com 152.53.192.33 Core Vaultwarden
sign.iamgmb.com 152.53.192.33 Core Document signing
shark.iamgmb.com 152.53.192.33 Core Shark game

DNS PENDING (create at SiteGround)

Subdomain → IP Service
vault.itpropartner.com 152.53.36.131 Vaultwarden (after migration)
status.itpropartner.com 152.53.192.33 Public status page

Backup Pipeline

Backup Schedule Target Purpose
hermes-live-sync Every 15 min s3://hermes-vps-backups/live/ Live state sync
hermes-full-backup Daily 1 AM s3://hermes-vps-backups/hermes-full-backup/ Full Hermes backup
home-router-backup Daily 6 AM s3://mikrotik-ccr-backups/ CCR config
root-essentials-backup Daily 3 AM S3 /root essentials
docker-volume-sync Daily 3 AM S3 Docker volumes
system-config-sync Daily 4 AM S3 System configs
unms-backup-sync Daily 6 AM (Core) s3://hermes-vps-backups/unms-backups/ UNMS data (pulled from app2)
unifi-backup-sync Daily 2 AM (Core) s3://hermes-vps-backups/unifi-backups/ UniFi configs (pulled from app2)
hudu-backup Daily 7 AM s3://hermes-vps-backups/hudu/backups/ Hudu volume dump
gitea-backup Daily 8 AM s3://hermes-vps-backups/gitea/daily/ Gitea repos
app1-backup Daily 2 AM s3://hermes-vps-backups/app1/ LiteLLM, n8n, OpenWebUI, MCP, Mattermost
app2-backup Daily 2:30 AM s3://hermes-vps-backups/app2/ Traccar, Gitea, Hudu, UNMS, UniFi
app3-backup Daily 3 AM s3://hermes-vps-backups/app3/ CloudPanel, MySQL, WordPress
wphost02-backup Daily 5 AM s3://hermes-vps-backups/wphost02-backup/ Webapps + MySQL
warm-standby-sync Every 10 min core-bu ← S3 DR readiness

Caddy Configuration

Core (152.53.192.33)

core.itpropartner.com           → static files
sign.core.itpropartner.com      → localhost:3000 (DocuSeal)
ops.itpropartner.com            → 127.0.0.1:8090 + static
uptimekuma.itpropartner.com     → 127.0.0.1:3001 (Uptime Kuma)
gps.fleettracker360.com         → app2:8082 (Traccar)
my.itpropartner.com             → static files
portal.debtrecoveryexperts.com  → static files
vault.iamgmb.com                → localhost:8080 (Vaultwarden)
sign.iamgmb.com                 → 127.0.0.1:8090
shark.iamgmb.com                → static + :8083

App1 (152.53.36.131)

ai.itpropartner.com             → :3000 (Open WebUI)
n8n.itpropartner.com            → :5678 (n8n)
admin-ai.itpropartner.com       → :4000 (LiteLLM)
app1.itpropartner.com           → static response

App2 (152.53.39.202)

hudu.itpropartner.com           → Hudu internal
gps.fleettracker360.com         → :8082 (Traccar)
unms.forefrontwireless.com      → UNMS Nginx
unifi.itpropartner.com          → :8443 (UniFi)

Service Access

Service URL Location Auth
Open WebUI https://ai.itpropartner.com app1 Chat UI
Open WebUI Admin https://admin-ai.itpropartner.com/ui app1 user: admin, pw: LITELLM_MASTER_KEY
Ops Portal https://ops.itpropartner.com Core Internal dashboard
Grafana http://core.itpropartner.com:3002 Core admin/admin
Uptime Kuma https://uptimekuma.itpropartner.com Core Service monitoring
Vaultwarden https://vault.iamgmb.com Core Password vault
CloudPanel https://panel.itpropartner.com app3 user: gmb / SQLite auth
Traccar https://gps.fleettracker360.com app2 GPS fleet tracking
UniFi https://unifi.itpropartner.com app2 Network controller
UNMS https://unms.forefrontwireless.com app2 WISP management
Hudu https://hudu.itpropartner.com app2 IT documentation
n8n https://n8n.itpropartner.com app1 Automation
CRM (DRE) https://crm.debtrecoveryexperts.com Cloudflare Access TwentyCRM

MCP Access (from Open WebUI)

MCP Server Location Port Tools
Super Search app1 :8899 10 tools — web_search, web_extract, person_search, email_search, phone_search, etc.
Browser app1 :8901 browser_navigate, browser_snapshot, browser_click, browser_type, browser_console
Filesystem app1 :8900 read_file, write_file, search_files, list_dir, file_info
Email app1 :8902 search_emails, send_email, get_email
Git/Gitea app1 :8903 clone, commit, push, pull

SSH Access

  • Key: itpp-infra (deployed to all servers)
  • User: ippadmin (sudo privileges)
  • Root SSH: Enabled on app1, app2, app3 (key-only exception per provisioning standard)
  • Core SSH: ssh -i /root/.ssh/itpp-infra root@152.53.192.33

Firewall

UFW is enabled on all servers. Standard rules:

  • Core: 22, 80, 443, 3000, 3001, 3002, 8080, 8082, 8090, 8443, 9090
  • app1: 22, 80, 443, 3000, 5678, 8899, 8900, 8901, 8902, 8903
  • app2: 22, 80, 443, 3000, 8080, 8082, 8089, 8443, 8843, 3478, 10001, 5000:5150
  • app3: 22, 80, 443, 8443

SSL

All SSL certificates issued via Let's Encrypt through Caddy. All certs auto-renew. No manual management needed.

Notes

  • DNS trap: itpropartner.com uses SiteGround nameservers. Cloudflare zone is NOT authoritative. Always verify with dig NS domain.com before creating records.
  • SiteGround: No API access. All DNS changes are manual through SiteGround panel.
  • Provider diversity: core-bu stays at Hetzner specifically so a netcup outage can't kill both Core and standby simultaneously.
  • app3 MySQL: Root password in Vaultwarden + /root/.my.cnf on app3, accessible via 127.0.0.1:3306.
  • CloudPanel: SQLite DB at /home/clp/htdocs/app/data/db.sq3 — users live here, not in MySQL.
  • AWS CLI PATH: All backup scripts must use /opt/awscli-venv/bin/aws or source /opt/awscli-venv/bin/activate — aws bare fails in cron context (PATH doesn't include venv bin). Documented in server-provisioning-standard v1.3.0.
  • Backup verification: Always run at least one manual backup after provisioning a server and verify it landed in S3 — never trust cron entries alone. Silent failures (aws: command not found, wrong file paths, S3 permission issues) won't surface otherwise.
Reference in New Issue View Git Blame Copy Permalink