docs: restore sanitized operational documentation after credential purge

README.md

@@ -0,0 +1,277 @@
+# IT Pro Partner — Infrastructure Documentation
+
+> **Last Updated:** July 17, 2026
+> **Maintainer:** Sho'Nuff
+
+---
+
+## Server Inventory
+
+### Core Server
+- **Hostname:** Core
+- **IP:** 152.53.192.33
+- **Provider:** netcup RS 2000 G12
+- **Specs:** 8 vCPU EPYC 9645, 15 GB DDR5 ECC, 512 GB NVMe
+- **OS:** Debian 13
+- **Role:** Hermes + Portals
+- **Key Services:**
+  - Hermes Agent (Telegram + cron, 22 cron jobs)
+  - Caddy reverse proxy (12 domains, auto-TLS)
+  - Ops Portal (FastAPI, port 8090)
+  - Prometheus (native, port 9090) + Grafana (native, port 3002)
+  - Uptime Kuma (Docker, port 3001) — 9+ monitors
+  - Vaultwarden (Docker, port 8080) — vault.iamgmb.com
+  - Twenty CRM (Docker) — crm.debtrecoveryexperts.com
+  - DocuSeal (Docker, port 3000) — sign.core.itpropartner.com
+  - SearXNG (Docker, port 8888)
+  - Komodo (Docker, port 9120)
+  - Tailscale, StrongSwan, WireGuard (home CCR tunnel 10.77.0.0/24)
+  - Redis cache
+
+### App1 Server
+- **Hostname:** app1
+- **IP:** 152.53.36.131
+- **Provider:** netcup RS 4000 G12
+- **Specs:** 12 vCPU EPYC 9645, 32 GB DDR5 ECC, 1 TB NVMe
+- **OS:** Debian 13
+- **Role:** AI/Service Hub
+- **Key Services:**
+  - Open WebUI (Docker, port 3000) — ai.itpropartner.com
+  - n8n + Postgres (Docker, port 5678) — n8n.itpropartner.com
+  - LiteLLM (Docker) + Postgres — admin-ai.itpropartner.com
+  - Mattermost Team Edition (Docker, port 8065) — noc.itpropartner.com
+  - Caddy (systemd, 80/443)
+  - 4 MCP servers: Browser (:8901), Filesystem (:8900), Email (:8902), Git (:8903)
+  - Super Search MCP (systemd, port 8899)
+
+### App2 Server
+- **Hostname:** app2
+- **IP:** 152.53.39.202
+- **Provider:** netcup RS 4000 G12
+- **Specs:** 12 vCPU EPYC 9645, 32 GB DDR5 ECC, 1 TB NVMe
+- **OS:** Debian 13
+- **Role:** Infrastructure Server
+- **Key Services:**
+  - Traccar GPS (Docker, port 8082 + 5000-5150) — fleettracker360.com
+  - UniFi Controller (Docker, port 8443) — unifi.itpropartner.com
+  - UNMS/UISP (10 Docker containers) — unms.forefrontwireless.com
+  - Hudu (Docker) — hudu.itpropartner.com
+  - Caddy (4 domains)
+
+### App3 Server
+- **Hostname:** app3
+- **IP:** 152.53.241.111
+- **Provider:** netcup RS 4000 G12
+- **Specs:** 12 vCPU EPYC 9645, 32 GB DDR5 ECC, 1 TB NVMe
+- **OS:** Debian 13
+- **Role:** Web Hosting + Backup Restore
+- **Key Services:**
+  - CloudPanel CE — panel.itpropartner.com
+  - Nginx (80/443) + Percona MySQL 8.4 + PHP 8.3
+  - Backup Restore System (Flask, port 8090) — my.itpropartner.com/backups
+  - WordPress sites (7 migrated from wphost02, all live):
+    - debtreecoveryexperts.com, boxpilotlogistics.com, iamgmb.com
+    - katiewattsdesign.com, vigilanttac.com, apextrackexperience.com
+    - mainwp.itpropartner.com, voipsimplicity.com, my.voipsimplicity.com
+  - Daily snapshots: 1 AM + 1 PM, 60-day retention, /opt/backup-restore/snapshots
+
+### Core-BU (Warm Standby)
+- **Hostname:** core-bu
+- **IP:** 5.161.225.131
+- **Provider:** Hetzner CPX21
+- **Specs:** 3 vCPU, 4 GB RAM, 80 GB SSD
+- **Role:** Warm standby — auto-failover if Core down
+- **Watchdog:** 5-min check, 4-cycle confirmation, S3 sync every 10 min
+
+### Legacy / Decommissioned
+- **old-ai:** 178.156.167.181 (Hetzner CPX41) — **decommissioned** (LiteLLM migrated to app1)
+- **old app1:** 87.99.144.163 (Hetzner CPX11) — **deleted**
+- **wphost02:** 5.161.62.38 (Hetzner CPX21) — **migrated to app3**
+- **Ollama:** Removed from Core (systemd) and app1 (Docker) Jul 17
+
+---
+
+## Model Fallback Chain
+
+All providers use direct API keys. GPT-5.5 quality survives through admin-ai → OpenRouter, then degrades through DeepSeek → Gemini → Grok.
+
+| # | Model | Provider | Gateway |
+|---|---|---|---|
+| Primary | GPT-5.5 | admin-ai | Self-hosted LiteLLM (app1) |
+| Fallback 1 | GPT-5.5 | OpenRouter | openrouter.ai |
+| Fallback 2 | DeepSeek v4 Pro | DeepSeek | api.deepseek.com |
+| Fallback 3 | Gemini 3.5 Flash | Google | generativelanguage.googleapis.com |
+| Fallback 4 | Grok 4.5 | xAI | api.x.ai |
+
+**Credits (Jul 17):** DeepSeek $58, OpenRouter ~$30 remaining, OpenAI/xAI/Google on pay-as-you-go
+**Health check:** Daily 8 AM cron (`model-usage-check`)
+
+---
+
+## Domain / DNS Map
+
+### ⚠️ itpropartner.com — SiteGround Nameservers Only
+
+`itpropartner.com` uses **SiteGround nameservers** exclusively. A Cloudflare zone exists (`0dc20632…`) but is NOT authoritative — records created there silently fail. All `*.itpropartner.com` changes must be manual through SiteGround panel.
+
+| Domain | IP | Server | Service |
+|---|---|---|---|
+| core.itpropartner.com | 152.53.192.33 | Core | Landing page + Grafana link |
+| ops.itpropartner.com | 152.53.192.33 | Core | Ops dashboard |
+| sign.core.itpropartner.com | 152.53.192.33 | Core | DocuSeal |
+| uptimekuma.itpropartner.com | 152.53.192.33 | Core | Uptime monitoring |
+| gps.fleettracker360.com | 152.53.192.33 | Core | Traccar HTTPS proxy → app2 |
+| my.itpropartner.com | 152.53.192.33 | Core | Customer portal hub |
+| hudu.itpropartner.com | 152.53.39.202 | app2 | IT documentation |
+| unifi.itpropartner.com | 152.53.39.202 | app2 | UniFi controller |
+| panel.itpropartner.com | 152.53.241.111 | app3 | CloudPanel CE |
+| ai.itpropartner.com | 152.53.36.131 | app1 | Open WebUI |
+| n8n.itpropartner.com | 152.53.36.131 | app1 | n8n automation |
+| admin-ai.itpropartner.com | 152.53.36.131 | app1 | LiteLLM |
+
+### Cloudflare-Managed Domains
+
+| Domain | IP | Server | Service |
+|---|---|---|---|
+| fleettracker360.com | Cloudflare | app2 | Fleet tracking website |
+| gps.fleettracker360.com | Cloudflare → Core | Core → app2 | Traccar devices |
+| voipsimplicity.com | Cloudflare | app3 | VoIP marketing site |
+| my.voipsimplicity.com | Cloudflare | app3 | VoIP customer portal |
+| portal.debtrecoveryexperts.com | 152.53.192.33 | Core | DRE portal |
+| crm.debtrecoveryexperts.com | Cloudflare Access | — | DRE CRM |
+| vault.iamgmb.com | 152.53.192.33 | Core | Vaultwarden |
+| sign.iamgmb.com | 152.53.192.33 | Core | Document signing |
+| shark.iamgmb.com | 152.53.192.33 | Core | Shark game |
+
+### DNS PENDING (create at SiteGround)
+
+| Subdomain | → IP | Service |
+|---|---|---|
+| vault.itpropartner.com | 152.53.36.131 | Vaultwarden (after migration) |
+| status.itpropartner.com | 152.53.192.33 | Public status page |
+
+---
+
+## Backup Pipeline
+
+| Backup | Schedule | Target | Purpose |
+|---|---|---|---|
+| hermes-live-sync | Every 15 min | s3://hermes-vps-backups/live/ | Live state sync |
+| hermes-full-backup | Daily 1 AM | s3://hermes-vps-backups/hermes-full-backup/ | Full Hermes backup |
+| home-router-backup | Daily 6 AM | s3://mikrotik-ccr-backups/ | CCR config |
+| root-essentials-backup | Daily 3 AM | S3 | /root essentials |
+| docker-volume-sync | Daily 3 AM | S3 | Docker volumes |
+| system-config-sync | Daily 4 AM | S3 | System configs |
+| unms-backup-sync | Daily 6 AM (Core) | s3://hermes-vps-backups/unms-backups/ | UNMS data (pulled from app2) |
+| unifi-backup-sync | Daily 2 AM (Core) | s3://hermes-vps-backups/unifi-backups/ | UniFi configs (pulled from app2) |
+| hudu-backup | Daily 7 AM | s3://hermes-vps-backups/hudu/backups/ | Hudu volume dump |
+| gitea-backup | Daily 8 AM | s3://hermes-vps-backups/gitea/daily/ | Gitea repos |
+| app1-backup | Daily 2 AM | s3://hermes-vps-backups/app1/ | LiteLLM, n8n, OpenWebUI, MCP, Mattermost |
+| app2-backup | Daily 2:30 AM | s3://hermes-vps-backups/app2/ | Traccar, Gitea, Hudu, UNMS, UniFi |
+| app3-backup | Daily 3 AM | s3://hermes-vps-backups/app3/ | CloudPanel, MySQL, WordPress |
+| wphost02-backup | Daily 5 AM | s3://hermes-vps-backups/wphost02-backup/ | Webapps + MySQL |
+| warm-standby-sync | Every 10 min | core-bu ← S3 | DR readiness |
+
+---
+
+## Caddy Configuration
+
+### Core (152.53.192.33)
+
+```
+core.itpropartner.com           → static files
+sign.core.itpropartner.com      → localhost:3000 (DocuSeal)
+ops.itpropartner.com            → 127.0.0.1:8090 + static
+uptimekuma.itpropartner.com     → 127.0.0.1:3001 (Uptime Kuma)
+gps.fleettracker360.com         → app2:8082 (Traccar)
+my.itpropartner.com             → static files
+portal.debtrecoveryexperts.com  → static files
+vault.iamgmb.com                → localhost:8080 (Vaultwarden)
+sign.iamgmb.com                 → 127.0.0.1:8090
+shark.iamgmb.com                → static + :8083
+```
+
+### App1 (152.53.36.131)
+
+```
+ai.itpropartner.com             → :3000 (Open WebUI)
+n8n.itpropartner.com            → :5678 (n8n)
+admin-ai.itpropartner.com       → :4000 (LiteLLM)
+app1.itpropartner.com           → static response
+```
+
+### App2 (152.53.39.202)
+
+```
+hudu.itpropartner.com           → Hudu internal
+gps.fleettracker360.com         → :8082 (Traccar)
+unms.forefrontwireless.com      → UNMS Nginx
+unifi.itpropartner.com          → :8443 (UniFi)
+```
+
+---
+
+## Service Access
+
+| Service | URL | Location | Auth |
+|---|---|---|---|
+| Open WebUI | https://ai.itpropartner.com | app1 | Chat UI |
+| Open WebUI Admin | https://admin-ai.itpropartner.com/ui | app1 | user: admin, pw: LITELLM_MASTER_KEY |
+| Ops Portal | https://ops.itpropartner.com | Core | Internal dashboard |
+| Grafana | http://core.itpropartner.com:3002 | Core | admin/admin |
+| Uptime Kuma | https://uptimekuma.itpropartner.com | Core | Service monitoring |
+| Vaultwarden | https://vault.iamgmb.com | Core | Password vault |
+| CloudPanel | https://panel.itpropartner.com | app3 | user: gmb / SQLite auth |
+| Traccar | https://gps.fleettracker360.com | app2 | GPS fleet tracking |
+| UniFi | https://unifi.itpropartner.com | app2 | Network controller |
+| UNMS | https://unms.forefrontwireless.com | app2 | WISP management |
+| Hudu | https://hudu.itpropartner.com | app2 | IT documentation |
+| n8n | https://n8n.itpropartner.com | app1 | Automation |
+| CRM (DRE) | https://crm.debtrecoveryexperts.com | Cloudflare Access | TwentyCRM |
+
+### MCP Access (from Open WebUI)
+
+| MCP Server | Location | Port | Tools |
+|---|---|---|---|
+| Super Search | app1 | :8899 | 10 tools — web_search, web_extract, person_search, email_search, phone_search, etc. |
+| Browser | app1 | :8901 | browser_navigate, browser_snapshot, browser_click, browser_type, browser_console |
+| Filesystem | app1 | :8900 | read_file, write_file, search_files, list_dir, file_info |
+| Email | app1 | :8902 | search_emails, send_email, get_email |
+| Git/Gitea | app1 | :8903 | clone, commit, push, pull |
+
+---
+
+## SSH Access
+
+- **Key:** `itpp-infra` (deployed to all servers)
+- **User:** `ippadmin` (sudo privileges)
+- **Root SSH:** Enabled on app1, app2, app3 (key-only exception per provisioning standard)
+- **Core SSH:** `ssh -i /root/.ssh/itpp-infra root@152.53.192.33`
+
+---
+
+## Firewall
+
+UFW is enabled on all servers. Standard rules:
+- **Core:** 22, 80, 443, 3000, 3001, 3002, 8080, 8082, 8090, 8443, 9090
+- **app1:** 22, 80, 443, 3000, 5678, 8899, 8900, 8901, 8902, 8903
+- **app2:** 22, 80, 443, 3000, 8080, 8082, 8089, 8443, 8843, 3478, 10001, 5000:5150
+- **app3:** 22, 80, 443, 8443
+
+---
+
+## SSL
+
+All SSL certificates issued via Let's Encrypt through Caddy. All certs auto-renew. No manual management needed.
+
+---
+
+## Notes
+
+- **DNS trap:** `itpropartner.com` uses SiteGround nameservers. Cloudflare zone is NOT authoritative. Always verify with `dig NS domain.com` before creating records.
+- **SiteGround:** No API access. All DNS changes are manual through SiteGround panel.
+- **Provider diversity:** core-bu stays at Hetzner specifically so a netcup outage can't kill both Core and standby simultaneously.
+- **app3 MySQL:** Root password in Vaultwarden + `/root/.my.cnf` on app3, accessible via 127.0.0.1:3306.
+- **CloudPanel:** SQLite DB at `/home/clp/htdocs/app/data/db.sq3` — users live here, not in MySQL.
+- **AWS CLI PATH:** All backup scripts must use `/opt/awscli-venv/bin/aws` or `source /opt/awscli-venv/bin/activate` — `aws` bare fails in cron context (PATH doesn't include venv bin). Documented in server-provisioning-standard v1.3.0.
+- **Backup verification:** Always run at least one manual backup after provisioning a server and verify it landed in S3 — never trust cron entries alone. Silent failures (`aws: command not found`, wrong file paths, S3 permission issues) won't surface otherwise.

docs/key-inventory.md

@@ -0,0 +1,252 @@
+# IT Pro Partner — Complete Key Inventory
+
+**Generated:** 2026-07-23
+**Sanitized:** 2026-07-23 (plaintext secrets replaced with storage references)
+**Scope:** All SSH keys, API tokens, service credentials, device keys, and passwords across the infrastructure
+**⚠️ SENSITIVE:** All credential values live in the listed storage locations. See Hudu for API keys (layout 49).
+
+---
+
+## 1. SSH Keys
+
+| Key Name | File | Type | Fingerprint (SHA256) | Purpose | Deployed To |
+|----------|------|------|-----------------------|---------|-------------|
+| **itpp-infra** | `/root/.ssh/itpp-infra` | ED25519 | `Jxh0bbT9dUV3q1DYYB3hHyhy/1TDj7Q8U4xrVmB38uQ` | Universal server admin key | All servers (Core, app1, app2, app3, wphost02, app1-bu, home router) |
+| **wisp_rsa** | `/root/.ssh/wisp_rsa` | ED25519 | `MxQw1oh90NibSgN2mDbKP+07/jE4FEUEBbFAzuk5DcI` | WISP MikroTik CCR router SSH | Home CCR router (10.77.0.2 via WireGuard) |
+| **germaine-personal** | `/root/.ssh/germaine-personal` | ED25519 | `dDbLH+bdPFcGU0mm1DpGa43ec0nUZ88YnpCi4p63y3I` | Germaine's personal key (from his machines) | Germaine's devices → Core |
+| **homelab** | `/root/.ssh/homelab` | ED25519 | `c1nts4wR9EU06/O/k895Pb2tGZublgnGWG6NoQrK/qs` | Homelab Proxmox/QNAP access | vm-host-01, vm-host-02, QNAP NAS |
+| **siteground.key** | `/root/.ssh/siteground.key` | RSA (encrypted) | N/A (RSA, encrypted) | SiteGround SFTP backup (port 18765) | SiteGround shared hosting |
+| **authorized_keys** | `/root/.ssh/authorized_keys` | — | — | Who can SSH into Core | Core (this server) |
+
+### SSH Key Details
+
+```
+itpp-infra.pub:    ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAII4dxTH11aJkBqCY8lXl1kTfZ8yXWhTcthHnt1MtAuIE itpp-infra
+wisp_rsa.pub:      ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDnI4UwwPL8gJvtP/Jr7qiw0Qj/bQBwi2+f03p730xvn wisp-backup
+germaine-personal.pub: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAID2H/2VMn8i7YSUUpcag6yXiI6nB3T99h7JIOs5/+73r germaine@itppartner
+homelab.pub:       ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHT+727Cti4cZ2x6CiYDeDKZ9BhvCJCzTHlO9vMInHie homelab-itpp
+```
+
+---
+
+## 2. Server Root Passwords
+
+**Storage:** Hudu (Vaultwarden asset) + `/root/.hermes/.env` (netcup CCP section)
+
+| Server | IP | Provider | Access | Notes |
+|--------|-----|----------|--------|-------|
+| **Core** | 152.53.192.33 | netcup RS 2000 | SSH key only | `itpp-infra` key, password auth disabled |
+| **app1** | 152.53.36.131 | netcup RS 4000 | Via root or ippadmin+sudo | Root password in Vaultwarden |
+| **app2** | 152.53.39.202 | netcup RS 4000 | Via root or ippadmin+sudo | Root password in Vaultwarden |
+| **app3** | 152.53.241.111 | netcup RS 4000 | Via root or ippadmin+sudo | Root password in Vaultwarden |
+| **app1-bu** | 5.161.114.8 | Hetzner CPX11 | itpp-infra SSH key | Warm standby, offline by default |
+
+### Admin Account (all servers)
+
+- **Username:** `ippadmin`
+- **Password:** → Vaultwarden entry "ippadmin"
+- **Sudo:** Yes (full sudo access)
+- **SSH:** Key-based only (`itpp-infra`)
+
+---
+
+## 3. Cloud & Infrastructure API Keys
+
+**Storage:** All in `~/.hermes/.env` + Hudu API assets (layout 49)
+
+| Service | Hudu Asset | Storage Location | Status |
+|---------|-----------|------------------|--------|
+| **Hetzner Cloud** | [177] | `/root/.hermes/scripts/.hetzner_token` + `/root/.hermes/.env` | ✅ Verified Jul 22 |
+| **Cloudflare DNS** | [165] | `~/.hermes/.env` → `CLOUDFLARE_API_TOKEN` | ✅ Active (verified by health check) |
+| **Wasabi S3** | [176] | `/root/.aws/credentials` | ✅ Active |
+| **netcup API** | [166] | `~/.hermes/.env` → `NETCUP_API_KEY` | ✅ Active |
+| **netcup CCP** | [167] | `~/.hermes/.env` → `NETCUP_CUSTOMER_NUMBER` + `NETCUP_CCP_PASSWORD` | ✅ Active |
+| **Gitea (OLD/DEAD)** | — | ⚠️ **EXPIRED** — still in homelab + itpp-infrastructure remotes | ❌ INVALID (verified Jul 23) |
+| **Gitea (ACTIVE)** | — | All other repos + `gitea-backup.sh` — ippadmin | ✅ Active (verified Jul 23) |
+
+---
+
+## 4. AI Provider API Keys
+
+All stored in `/root/.hermes/.env` and Hudu API assets (layout 49).
+
+| Provider | Hudu Asset | Purpose | Status |
+|----------|-----------|---------|--------|
+| **admin-ai** (LiteLLM) | [126] Hermes Primary Key | Primary model gateway (all models) | ✅ Active |
+| **Anthropic** | [150] | Claude models | ✅ Active |
+| **OpenAI** | [149] | GPT models | ✅ Active |
+| **DeepSeek** | [152] | DeepSeek models | ✅ Active |
+| **Google Gemini** | [161] / [151] | Gemini models | ✅ Active |
+| **xAI (Grok)** | [154] | Grok models | ✅ Active |
+| **OpenRouter** | [153] | Multi-provider routing | ✅ Active |
+| **Mistral** | [155] | Mistral models | ✅ Active |
+| **Groq** | [157] | Fast inference | ✅ Active |
+| **Fireworks AI** | [156] | Serverless inference | ✅ Active |
+| **Perplexity** | [159] | Search-augmented LLM | ✅ Active |
+| **Cohere** | [158] | Cohere models | ✅ Active |
+| **AI21 Labs** | [160] | Jurassic models | ✅ Active |
+| **MiniMax** | [187] | MiniMax M3 | ✅ Active |
+| **Z.ai (GLM)** | [188] | GLM models | ✅ Active |
+| **Alibaba Qwen** | [189] Alibaba Qwen (DashScope) | Qwen models | ✅ Active |
+| **Deepgram** | [162] | STT (voice transcription) | ✅ Active |
+
+---
+
+## 5. Communication APIs
+
+**Storage:** All in `~/.hermes/.env` + Hudu API assets (layout 49)
+
+| Service | Hudu Asset | Storage | Status |
+|---------|-----------|---------|--------|
+| **Telegram Bot** | [170] | `~/.hermes/.env` → `TELEGRAM_BOT_TOKEN` | ✅ Active |
+| **Twilio (Live)** | [184] Twilio Live | `~/.hermes/.env` → `TWILIO_ACCOUNT_SID` + `TWILIO_AUTH_TOKEN` | ✅ Active |
+| **Twilio (Test)** | [185] Twilio Test | `~/.hermes/.env` → `TWILIO_TEST_ACCOUNT_SID` + `TWILIO_TEST_AUTH_TOKEN` | ✅ Active |
+| **Twilio API Key** | [186] Twilio API Key | `~/.hermes/.env` → `TWILIO_API_KEY_SID` + `TWILIO_API_KEY_SECRET` | ✅ Active |
+| **ElevenLabs** | [148] | `~/.hermes/config.yaml` (auxiliary vision / TTS) | ✅ Active |
+| **Email SMTP/IMAP** | — | `/root/.config/himalaya/shonuff.pass` | ✅ Active |
+| **Email account** | — | `shonuff@germainebrown.com` — MXroute via mail.germainebrown.com:2525 (SMTP) / :993 (IMAP) | ✅ Active |
+
+---
+
+## 6. VoIP / RingLogix
+
+**Storage:** `~/.hermes/.env` + Hudu API assets
+
+| Credential | Hudu Asset | Storage |
+|-----------|-----------|---------|
+| **RingLogix Client ID** | [174] | `~/.hermes/.env` → `RINGLOGIX_CLIENT_ID` |
+| **RingLogix Client Secret** | [175] | `~/.hermes/.env` → `RINGLOGIX_CLIENT_SECRET` |
+| **RingLogix Username** | — | `~/.hermes/.env` → `RINGLOGIX_USERNAME` |
+| **RingLogix Password** | — | `~/.hermes/.env` → `RINGLOGIX_PASSWORD` |
+| **RingLogix Domain** | — | `~/.hermes/.env` → `RINGLOGIX_DOMAIN` |
+
+---
+
+## 7. MSP / RMM / Security APIs
+
+**Storage:** All in `~/.hermes/.env` + Hudu API assets
+
+| Service | Hudu Asset | Storage | Status |
+|---------|-----------|---------|--------|
+| **SyncroMSP** | [168] Token + [169] API Key | `~/.hermes/.env` → `SYNCROMSP_API_TOKEN` + `SYNCROMSP_API_KEY` | ✅ Active |
+| **Bitdefender GZ** | [172] | `~/.hermes/.env` → `BITDEFENDER_API_KEY` | ✅ Active |
+| **VirusTotal** | [171] | `~/.hermes/.env` → `VIRUSTOTAL_API_KEY` | ✅ Active |
+| **UISP/UNMS** | [173] | `~/.hermes/.env` → `UISP_API_KEY` | ✅ Active |
+
+---
+
+## 8. Search & Data APIs
+
+| Service | Hudu Asset | Storage | Status |
+|---------|-----------|---------|--------|
+| **Firecrawl** | [164] | `~/.hermes/.env` → `FIRECRAWL_API_KEY` | ✅ Active |
+| **Exa AI Search** | [163] | `~/.hermes/.env` → `EXA_API_KEY` | ✅ Active |
+
+---
+
+## 9. Database Credentials
+
+| Database | Host | User | Password Location | Purpose |
+|----------|------|------|-------------------|---------|
+| **MySQL (apex track)** | 127.0.0.1:33060 (SSH tunnel from wphost02) | `apextrackexperience_1781549652` | `wp-config.php` on wphost02 | Apex Track Experience WordPress |
+| **MySQL (CloudPanel)** | app3:3306 | `root` | `/root/.my.cnf` on app3 (also in Vaultwarden) | CloudPanel WordPress hosting |
+| **LiteLLM Postgres** | app1 (Docker) | (in docker-compose) | `/root/docker/litellm/docker-compose.yml` on app1 | LiteLLM operational DB |
+
+---
+
+## 10. Docker Services
+
+| Service | URL | Credential Location | Storage |
+|---------|-----|--------------------|---------|
+| **Vaultwarden** | vault.itpropartner.com / vault.iamgmb.com | Admin Token → `/root/docker/vaultwarden/.env` on Core | Docker env file |
+| **DRE Portal** | portal.debtrecoveryexperts.com | Basic Auth (htpasswd) | `/etc/caddy/dre-passwd` |
+| **SearXNG** | (internal, no public endpoint) | (none) | — |
+| **DocuSeal** | sign.core.itpropartner.com / sign.iamgmb.com | (none / app-managed) | — |
+| **Uptime Kuma** | uptimekuma.itpropartner.com | (app-managed) | — |
+| **Open WebUI** | admin-ai.itpropartner.com | `admin@itpropartner.com` (password: ask Sho'Nuff) | Not in .env |
+| **Mealie** | recipe.iamgmb.com | `G@germainebrown.com` (password → Vaultwarden) | Vaultwarden |
+| **Ops Portal** | ops.itpropartner.com | `ippadmin` (password → `~/.hermes/.env`) | `~/.hermes/.env` |
+
+---
+
+## 11. VPN & Network Keys
+
+### WireGuard (Core)
+
+| Item | Value |
+|------|-------|
+| **Interface** | `wg0` |
+| **Core Private Key** | → `/etc/wireguard/wg0.conf` on Core |
+| **Core IP** | `10.77.0.1/24` |
+| **Listen Port** | `51821` |
+| **Home Peer Public Key** | `1fPwdGQ20CxlZCQZQV134olDcE91hfp78yNDeaKJZzg=` |
+| **Home Peer Endpoint** | `76.195.7.60:13231` |
+| **Routed Networks** | `10.1.0.0/16`, `10.2.0.0/16`, `172.16.1.0/24`, `172.18.18.0/24` |
+
+### Tailscale
+
+| Node | IP | Type | Status |
+|------|-----|------|--------|
+| core | 100.71.155.7 | Linux | ✅ Online |
+| app1 | 100.90.186.109 | Linux | ✅ Online |
+| app2 | 100.117.164.66 | Linux | ✅ Online |
+| app3 | 100.72.15.12 | Linux | ✅ Online |
+| app1-bu | 100.112.23.21 | Linux | ⚠️ Offline (7d) |
+| iphone-15-pro-max | 100.106.231.86 | iOS | ✅ Online |
+| ipp-g-lap | 100.120.64.120 | macOS | ✅ Online |
+| m4-mac-mini | 100.116.232.65 | macOS | ✅ Online |
+
+---
+
+## 12. UniFi / UDM Pro Device Keys
+
+| Site | Key Location | Type | Status |
+|------|-------------|------|--------|
+| **Grand Lake Club** | UniFi Network Controller → Settings → API | Local Network API Key | ✅ Stored, pending direct verification |
+| **Liberty Tire** | UniFi Network Controller → Settings → API | Local Network API Key | ✅ Stored, pending direct verification |
+
+---
+
+## 13. Unknown / Not Found
+
+The following credentials are known to exist but were not found in the standard locations:
+
+| Item | Notes |
+|------|-------|
+| **Open WebUI admin password** | Recovery manual says "in .env or ask Sho'Nuff" — NOT in current .env. Must ask Germaine. |
+| **Hudu API key** | In skill docs (`hudu-management`) — used programmatically, not in .env. |
+| **Traccar/FleetTracker360 admin** | Not in .env. May be Docker env or app-managed. |
+| **Twenty CRM credentials** | Docker on Core, env at `/root/docker/twenty/.env` (not read). |
+| **WordPress site DB passwords** | Various sites, typically in `wp-config.php` on wphost02 or app3. |
+| **app1-bu root password** | Hetzner CPX11 — accessed via itpp-infra SSH key only. |
+| **ComfyUI / Z4** | GPU server allocated for TripFlow — credentials not yet documented. |
+| **Home MikroTik admin** | SSH via `admin@10.77.0.2` with `wisp_rsa` key. RouterOS password in router config (not extracted). |
+
+---
+
+## 14. Key Rotation & Audit Notes
+
+- **Last full audit:** 2026-07-23
+- **Last sanitization:** 2026-07-23 — all plaintext secrets removed; use Hudu + file paths for values
+- **Hetzner token:** Rotated Jul 22 (old tokens in Hudu were invalid)
+- **Twilio:** Live + test credentials both present in Hudu [184]/[185]/[186]
+- **OpenRouter:** Fallback routing key — keep active even if not primary
+- **admin-ai:** Primary gateway — all model calls route through this
+- **Backups:** All .env + config files included in daily Hermes backup to S3
+
+### Recovery Priority
+
+If Core is lost, you need these to rebuild (in order):
+1. `/root/.ssh/itpp-infra` — SSH to all servers
+2. `/root/.hermes/.env` — All API keys and secrets
+3. `/root/.aws/credentials` — S3 access for backups
+4. `/root/.hermes/config.yaml` — Full Hermes config
+5. `/root/.config/himalaya/shonuff.pass` — Email access
+
+### Hudu API Assets (layout 49)
+
+All API keys are documented as Hudu assets. List them via:
+```
+GET https://hudu.itpropartner.com/api/v1/companies/1/assets?page=1&per_page=25
+```
+Filter by `asset_layout_id == 49` to see all API keys with their Hudu asset IDs and storage locations.

docs/model-chain.md

@@ -0,0 +1,26 @@
+# AI Model Chain — IT Pro Partner
+
+**Updated:** July 21, 2026
+**Sanitized:** July 23, 2026 (plaintext keys removed)
+**Gateways:** admin-ai (self-hosted LiteLLM), OpenRouter, DeepSeek, Google, xAI
+
+## Chain
+
+| # | Model | Provider | Key Storage |
+|---|---|---|---|
+| Primary | GPT-5.5 | admin-ai | Hudu [126] Hermes Primary Key |
+| F1 | GPT-5.5 | OpenRouter | Hudu [153] |
+| F2 | DeepSeek v4 Pro | DeepSeek | Hudu [152] |
+| F3 | Gemini 3.5 Flash | Google | Hudu [161] / [151] |
+| F4 | Grok 4.5 | xAI | Hudu [154] |
+
+## Admin-AI (LiteLLM)
+- URL: `admin-ai.itpropartner.com` (app1, 152.53.36.131)
+- Master key: `/root/docker/litellm/.env` on app1 (Hudu [178] LiteLLM Master Key)
+- Hermes virtual key: Hudu [126] Hermes Primary Key (GPT-5.5 + DeepSeek v4 Pro routing)
+
+## Credential Storage
+- Config: `/root/.hermes/config.yaml`
+- Hudu: All API assets under layout 49 at https://hudu.itpropartner.com — search "AI Provider" or by model name
+- Git: `itpp-infrastructure/docs/model-chain.md` (this file, sanitized)
+- Full inventory: `itpp-infrastructure/docs/key-inventory.md`

docs/ops-portal/CHANGELOG.md

@@ -0,0 +1,39 @@
+# Ops Portal Changelog
+
+## 2026-07-20 — Major audit and fix session
+
+### Fixed
+- **`/api/health` returning 404:** Caddy `handle_path` was stripping the path → changed to `handle`, port corrected to 8090
+- **`/api/servers` returning null:** Server list now returns all 5 servers with live ping health
+- **Server IPs stale:** app1-bu updated to 5.161.225.131, legacy entries removed
+- **Page titles inconsistent:** All 11 pages standardized to "X — IT Pro Partner Ops" format
+- **Missing nav icons:** All 11 nav items now have SVG icons
+- **FleetTracker360 missing from nav:** Added to navigation with car icon
+- **Backups page no data:** `s3_buckets` → `s3_backups` key fix
+- **FleetTracker360 page no nav:** Added ops.css, app.js, utils.js includes
+- **Network page dark sidebar:** Replaced with standard top nav bar
+- **Cache-busting broken:** All JS/CSS references now versioned with timestamps
+- **Mobile nav broken:** `.nav-links.open` CSS rule missing → hamburger menu now toggles properly on iOS/Android
+- **Auth guard race condition:** IIFE scripts replaced with DOMContentLoaded event listeners — pages now load data when user is authenticated
+- **Cost page broken:** Missing `loadData` function → defined and wired
+- **Dependency diagram 404:** File copied to static dir, link corrected
+- **Logs page mangled title:** Triple-nested `<title>` tags from sed accident → cleaned
+- **Config page scripts directory:** Now populates directory listing when clicked
+- **Services page:** Server column added showing "Core (152.53.192.33)"
+- **Dashboard auto-refresh on tab focus:** Visibility API handler added
+- **Critical service protection:** hermes, caddy, ops-portal restarts blocked via API
+
+### Removed
+- Duplicate server entries: "app1 (AI Stack)" and "Docker Box (legacy)"
+- Server count: 7 → 5 clean entries
+
+### Changed
+- Admin credentials: germaine/itpp2026! → ippadmin (password → Vaultwarden)
+- JWT_SECRET made persistent in /root/.hermes/.env to survive restarts
+- Collector timeout: 20s → 90s to handle 94K-file S3 bucket scanning
+
+## Jul 17, 2026 — Initial deployment
+- Ops portal deployed on Core as FastAPI app
+- Caddy reverse proxy configured
+- 10 HTML pages created
+- Ops collector built for S3, system health, server status

docs/ops-portal/README.md

@@ -0,0 +1,55 @@
+# Ops Portal — ops.itpropartner.com
+
+## Architecture
+- **Server:** Core (152.53.192.33, netcup RS 2000)
+- **Backend:** FastAPI at `/opt/ops-portal/server.py` (port 8090)
+- **Proxy:** Caddy → reverse_proxy to 127.0.0.1:8090
+- **Static files:** `/opt/ops-portal/static/` — 11 HTML pages, ops.css, app.js, utils.js
+- **Auth:** JWT via `POST /api/auth/login`, token in localStorage
+- **Data:** `/var/www/ops/data/ops-status.json` (5-min collector refresh)
+- **Collector:** `/root/.hermes/scripts/ops-data-collector.py` — Wazuh, Bitdefender, S3, UISP, system health
+- **Systemd:** `ops-portal.service`, env from `/root/.hermes/.env`
+- **Credentials:** ippadmin (password → Vaultwarden / `~/.hermes/.env`)
+
+## Pages (11 total)
+| Page | Path | Description |
+|------|------|-------------|
+| Dashboard | / | System health, widgets, audit log |
+| Services | /services.html | Systemd service control, audit log, server column |
+| Servers | /servers.html | 5 servers with ping health |
+| Network | /network.html | UISP data (44 sites, 90 devices), DNS zones |
+| Backups | /backups.html | S3 bucket status (6 buckets) |
+| FleetTracker | /fleettracker360.html | Traccar device tracking |
+| Cron Jobs | /cron.html | Hermes cron jobs with expandable scripts |
+| Config | /config.html | Active configs, /root/.hermes/scripts/ listing |
+| Logs | /logs.html | Aggregated log viewer |
+| Audit | /audit.html | Full audit trail |
+| Costs | /cost.html | API cost tracking by model |
+
+## API Endpoints
+| Method | Path | Auth | Description |
+|--------|------|------|-------------|
+| POST | /api/auth/login | No | JWT authentication |
+| GET | /api/health | No | Health check, DB status |
+| GET | /api/status | JWT | Full dashboard data (17 sections) |
+| GET | /api/servers | JWT | Server list with IPs |
+| GET | /api/servers/health | JWT | Ping health (all 7 LIVE) |
+| GET | /api/audit-log?limit=N | JWT | Audit trail entries |
+| GET | /api/ft360/status | JWT | FleetTracker device data |
+
+## Critical Services (API restart blocked)
+hermes, hermes-assistant, hermes-browser, caddy, ops-portal, mysql-tunnel
+
+## Dashboard Widgets
+- System Health — Core metrics (jobs, disk, memory, S3, APIs)
+- Wazuh Security — agent count, alerts
+- Bitdefender GravityZone — 9 managed endpoints
+- Alerts and Notifications — DR issues, backup failures, cron errors
+- Quick Actions — Restart Ops Portal
+
+## Recovery
+```
+systemctl restart ops-portal
+systemctl reload caddy
+python3 /root/.hermes/scripts/ops-data-collector.py
+```

docs/project-log.md

@@ -0,0 +1,68 @@
+# Project Log — All Completed Projects
+
+## 2026-07-20
+
+### Ops Portal Audit and Overhaul
+- Full audit of all 11 pages, 7 API endpoints, and 5 dashboard widgets
+- Fixed 15 bugs: auth guards, cache-busting, mobile nav, page titles, missing icons, data keys
+- Added 3 new widgets: Wazuh Security, Bitdefender GravityZone, Alerts and Notifications
+- Standardized credentials: ippadmin (password → Vaultwarden / `~/.hermes/.env`)
+- Added critical service protection (hermes/caddy/ops-portal restart blocked via API)
+- Server list cleaned up (7→5), dependency diagram fixed, config page scripts listing
+
+### Backup-Restore Enhancements
+- Added manual backup with domain dropdown and note field
+- Added restore history logging with formatted 4-column table
+- Fixed Caddy routing and timeouts (restore was returning 404 via proxy)
+- Fixed mobile toggle on domain expansion cards
+- 9 WordPress sites under daily backup (1 AM and 1 PM)
+
+### Docs Written
+- `/root/projects/ops-portal/README.md` + `CHANGELOG.md`
+- `/root/projects/backup-restore/README.md` + `CHANGELOG.md`
+
+---
+
+## 2026-07-17 — Backup-Restore Initial Deployment
+- Flask backup/restore app deployed on app3 (152.53.241.111)
+- Daily snapshots scheduled at 1 AM and 1 PM
+- Caddy reverse proxy from my.itpropartner.com
+- 9 WordPress sites configured
+
+## 2026-07-21 — Home Lab Consolidation
+
+### Proxmox Migration
+- vm-host-02 VMs migrated/destroyed: graylog, zabbix, fog, Ubuntu-Server
+- vm-host-01 now hosts: docker-host-01, adguard-home
+- vm-host-02 cleared for GPU installation (RTX 3090 pending verification)
+- QNAP NFS shared storage created (2TB pool, mounted on both Proxmox hosts)
+
+### DNS Infrastructure
+- Technitium DNS deployed on app2 (dns1.itpropartner.com)
+- DoH upstreams: Quad9, Cloudflare, Google
+- Home DNS chain: docker-host-01 AdGuard → dns1 Technitium → vm-host-01 AdGuard
+- Firewall locked: port 53 restricted to 76.195.7.60
+
+### Twilio
+- Toll-free number verification submitted for IT Pro Partner
+- Use case: customer notifications, appointment reminders, IVR
+
+### Mattermost
+- Branding configured: IT Pro Partner NOC
+- Channel structure designed (13 channels)
+- Mobile push investigation: HPNS required for background notifications
+
+### Gift-a-Roast
+- Domain giftaroast.com purchased, DNS live (Cloudflare → app1)
+- ElevenLabs TTS + Deepgram STT keys verified
+- Architecture: Twilio Voice → STT → AI → TTS → caller
+
+### Uptime Kuma
+- Backed up (361MB, 25 monitors), updated to latest
+
+### IRS
+- Name change letter drafted: CG Premier Transport LLC → IT Pro Partner LLC
+- Georgia Secretary of State filing confirmed
+
+### Skills Updated
+- 10 skills patched: docker-service-deployment, home-lab-*, server-architecture-plan, twilio-10dlc, vaultwarden-management, voip-portal, hudu, syncromsp, recurring-information-scout

hermes-skills/systems-networks-team.md

@@ -0,0 +1,66 @@
+---
+name: systems-networks-team
+description: Expert team for systems administration, network infrastructure, server deployment, backup/DR, and monitoring. Dispatch for any task involving servers, Docker, networking gear, cron jobs, security, or infrastructure automation.
+category: delegation
+---
+
+# Systems & Networks Team
+
+You are a senior infrastructure engineer. You build and maintain systems that never fail, networks that never drop, and automation that eliminates toil.
+
+## Team Members (Skills)
+
+Always load these before beginning a sys/net task:
+
+- **docker-service-deployment** -- Deploy and document Docker services
+- **server-provisioning-standard** -- ITPP base server deployment standards
+- **hermes-backup** -- Hermes backup and restore procedures
+- **disaster-recovery-audit** -- DR audit framework
+- **mikrotik-onboarding** -- MikroTik router onboarding
+- **reboot-with-health-check** -- Server reboots with verification
+- **script-audit** -- Systematic script review
+- **cloudflare-dns-and-domains** -- Cloudflare DNS management
+- **cloudpanel-deployment** -- CloudPanel server deployment
+- **status-page-deployment** -- Status page using Uptime Kuma + Caddy
+- **vaultwarden-management** -- Vaultwarden deploy, SMTP, migration
+- **tailscale-infrastructure-access** -- Tailscale private networking
+- **model-failover-and-credit-tracking** -- Model failover chain management
+
+## Standing Orders
+
+1. **Verify before reporting.** Never claim a service is up or a config is applied until you've actually tested it. SSH in, run the command, read the output.
+
+2. **Idempotency by design.** Every script, every config, every deployment must survive being run twice. Check state before changing it.
+
+3. **Backup before mutation.** Before touching a production config, ensure a backup exists and is verified.
+
+4. **Document as you build.** Every change goes into the infrastructure Git repo. No undocumented changes to production systems.
+
+5. **Safe to fail.** Every change must have a rollback path. If you can't undo it, don't do it without explicit approval.
+
+## Server Inventory
+
+| Server | IP | Role |
+|---|---|---|
+| Core | 152.53.192.33 | Hermes, portals, monitoring |
+| app1 | 152.53.36.131 | AI/services hub |
+| app2 | 152.53.39.202 | Infrastructure server |
+| app3 | 152.53.241.111 | Web hosting + backup |
+| core-bu | 5.161.225.131 | Warm standby |
+| wphost02 | 5.161.62.38 | Legacy RunCloud host (still live) |
+
+## Key Credentials
+
+- SSH key: `/root/.ssh/itpp-infra` (all servers, root access)
+- CloudPanel MySQL root: password in Vaultwarden + `/root/.my.cnf` on app3, accessible on 127.0.0.1:3306
+- CloudPanel admin: panel.itpropartner.com (gmb — password → Vaultwarden)
+- Wasabi S3: s3.us-east-1.wasabisys.com, bucket: hermes-vps-backups
+- Cloudflare API: token in /root/.hermes/.env (CLOUDFLARE_API_TOKEN)
+
+## Quality Gates
+
+- [ ] Change tested in staging/dry-run before production
+- [ ] Backup verified before mutation
+- [ ] Health check passes after deployment
+- [ ] Documentation committed to Git
+- [ ] Rollback procedure documented

sites/app3.md

@@ -0,0 +1,49 @@
+# App3 — Infrastructure Reference
+
+**Server:** netcup RS 4000 G12
+**IP:** 152.53.241.111
+**Provisioned:** July 16, 2026
+**Role:** Customer web hosting (WordPress sites)
+
+## Software Stack
+
+| Component | Version |
+|---|---|
+| Debian | 13 (Trixie) |
+| Nginx | 2.x |
+| PHP | 8.3 (FPM) |
+| MySQL | Percona 8.4 |
+| CloudPanel CE | Latest |
+| Tailscale | Enabled |
+
+## Hosted Sites
+
+| Domain | Type | Status |
+|---|---|---|
+| voipsimplicity.com | WordPress | ✅ Live |
+| support.voipsimplicity.com | DNS only | ✅ Created |
+
+## Issues Encountered & Fixed (Jul 16)
+
+1. **"Hello World" default page** — WordPress files were copied to `/home/voipsimplicity/htdocs/www/` but CloudPanel's Nginx root was `/home/voipsimplicity/htdocs/voipsimplicity.com/`. Fixed by rsyncing files to correct directory.
+
+2. **panel.itpropartner.com 404** — Manual Nginx config (`panel.conf`) was overriding CloudPanel's generated config. Removed manual config. CloudPanel's reverse proxy template had `try_files $uri =404` instead of `try_files $uri @reverse_proxy` — fixed.
+
+3. **CloudPanel MFA blocking login** — MFA was enabled on the `admin` user by default. Disabled via `clpctl user:disable:mfa`.
+
+4. **Let's Encrypt failure (unproxied DNS)** — DNS for `panel.itpropartner.com` was on SiteGround nameservers, not Cloudflare. Added manually via SiteGround panel. CloudPanel SSL via HTTP-01 challenge required `.well-known/acme-challenge` directory at reverse proxy site root.
+
+5. **"Error establishing a database connection"** — MySQL auth plugin mismatch. Site DB user was using `caching_sha2_password` but WordPress expects `mysql_native_password`. Fixed by converting auth plugin.
+
+6. **itpropartner.com not on Cloudflare** — Nameservers still at SiteGround. All DNS changes for `*.itpropartner.com` must be done at SiteGround manually.
+
+## Access
+
+- **CloudPanel:** https://panel.itpropartner.com — `info@itpropartner.com` (password → Vaultwarden)
+- **SSH:** root@152.53.241.111 (itpp-infra key)
+- **MySQL root:** `root` (password → Vaultwarden + `/root/.my.cnf` on app3) on `127.0.0.1:3306`
+
+## Pending
+
+- Migrate remaining WordPress sites from wphost02 to app3
+- Update all CloudPanel passwords (sanitized from plaintext in this doc 2026-07-23 — values in Vaultwarden)