feat: initial itpp-docs setup with MkDocs Material

- mkdocs.yml with dark slate theme, nav for 12 ITPP projects - build-docs.sh aggregates docs from all project repos - .gitea/workflows/docs-publish.yml for nightly rebuild+deploy - README and CHANGELOG for the itpp-docs repo itself - docs-source/ populated from all 12 repos - site/ ready for deployment to docs.itpropartner.com
2026-08-09 23:49:45 -04:00
parent 08ba2e62a8
commit 98073dc636
134 changed files with 55534 additions and 2 deletions
@@ -0,0 +1,6 @@
+# itpp-infrastructure — CHANGELOG
+
+## 2026-07-16 — Audit Remediation
+
+- Created CHANGELOG.md (missing per project documentation standard)
+- Project directory: `/root/projects/itpp-infrastructure`
@@ -0,0 +1,27 @@
+# app2 Caddyfile Audit — July 21, 2026
+
+## Root cause
+Technitium DNS was deployed on app2. During the Caddyfile rewrite to add `dns1.itpropartner.com`, two existing services were dropped:
+
+1. **UNMS** — `reverse_proxy localhost:80` failed because UNMS nginx exposes port 443 (via host 8444), not port 80. Fixed by proxying via HTTPS with `tls_insecure_skip_verify`.
+2. **Gitea** — entry was completely removed. Fixed by adding `reverse_proxy 127.0.0.1:3001`.
+
+## Prevention
+- Always audit `docker ps` output BEFORE rewriting Caddyfile
+- Verify every running container that exposes web ports has a Caddy entry
+- Test each domain with `curl -sk` after Caddy reload
+
+## Final Caddyfile (validated)
+```
+{
+    default_bind 152.53.39.202
+    auto_https disable_redirects
+}
+dns1.itpropartner.com:443 → 127.0.0.1:5380
+gps.fleettracker360.com:443 → localhost:8082
+fleettracker360.com:443 → localhost:8082
+unms.forefrontwireless.com:443 → https://localhost:8444 (tls_insecure_skip_verify)
+unifi.itpropartner.com:443 → https://localhost:8443 (tls_insecure_skip_verify)
+hudu.itpropartner.com:443 → localhost:3000
+git.itpropartner.com:443 → 127.0.0.1:3001
+```
@@ -0,0 +1,158 @@
+# Backup-Restore — Architecture
+
+## Topology
+
+```
+                         INTERNET
+                            |
+                    [Caddy on Core]
+                   my.itpropartner.com
+                            |
+            +---------------+---------------+
+            |               |               |
+     /backups/*      /api/restore    /api/backup
+     /api/log        /api/download   /api/delete
+            |               |               |
+            +-------+-------+-------+-------+
+                            |
+                    app3 (152.53.241.111)
+                     netcup RS 4000
+                            |
+                    [Flask :8090]
+                   /opt/backup-restore/
+                            |
+        +-------------------+-------------------+
+        |                   |                   |
+  snapshot.sh         app.py (UI+API)     snapshots/
+  (cron 1AM,1PM)      Jinja templates     /opt/backup-restore/
+        |                   |              snapshots/<domain>/
+        v                   v                   |
+  [tar files]          [render HTML]     +------+------+
+  [mysqldump]          [REST API]        |      |      |
+        |                   |          .tar.gz  .sql  note.txt
+        v                   v
+  /opt/backup-restore/  [Browser]
+  snapshots/<domain>/
+  <timestamp>/
+```
+
+## Data Flow — Manual Backup
+
+```
+Browser (user clicks "Backup Now")
+  |
+  |-- POST /api/backup {"domain":"x.com","note":"pre-deploy"}
+  |       |
+  |       v
+  |   Caddy → app3:8090
+  |       |
+  |       v
+  |   Flask api_backup()
+  |       |
+  |       |-- Parse nginx config → find htdocs path
+  |       |-- tar -czf files.tar.gz (timeout 300s)
+  |       |-- Parse wp-config.php → find DB_NAME
+  |       |-- mysqldump → database.sql (timeout 300s)
+  |       |-- Save note.txt, size.txt
+  |       |-- Return {"ok":true, "snapshot":"<timestamp>"}
+  |       |
+  |       v
+  |   snapshots/x.com/2026-07-20_163208/
+  |       files.tar.gz (16MB)
+  |       database.sql (74KB)
+  |       note.txt ("pre-deploy")
+  |       size.txt
+  |
+  v
+Browser reloads → new snapshot in list
+```
+
+## Data Flow — Restore
+
+```
+Browser (user clicks Restore on a snapshot)
+  |
+  |-- POST /api/restore {"domain":"x.com","snapshot":"2026-07-20_130001"}
+  |       |
+  |       v
+  |   Caddy → app3:8090 (flush_interval -1, 300s timeouts)
+  |       |
+  |       v
+  |   Flask api_restore()
+  |       |
+  |       |-- Find snapshot path
+  |       |-- tar -xzf files.tar.gz → htdocs (timeout 300s)
+  |       |-- mysql < database.sql → WordPress DB (timeout 300s)
+  |       |-- chown -R site-user:site-user
+  |       |-- Log to restore.log: "TS|x.com|snap_id|OK"
+  |       |-- Return {"ok":true, "msg":"x.com restored to <snap>"}
+  |       |
+  |       v
+  |   Site is restored
+  |
+  v
+Browser shows success toast → Restore History updates
+```
+
+## Components
+
+### 1. Flask App (`/opt/backup-restore/app/app.py`)
+- Single-file Flask application, port 8090
+- Jinja2 templating for backup dashboard (render_template_string)
+- 6 API endpoints (backup, restore, delete, download, log, index)
+- All HTML/CSS/JS inline in a single Python triple-quoted string
+- No auth — accessible via Caddy-only routing
+- Systemd: `backup-restore.service`
+
+### 2. Snapshot Engine (`/opt/backup-restore/snapshot.sh`)
+- Bash script, runs at 1 AM and 1 PM via cron
+- Iterates all WordPress sites in `/etc/nginx/sites-enabled/`
+- Creates: files.tar.gz (document root), database.sql (MySQL dump)
+- Auto-cleanup: deletes snapshots older than 30 days
+- Log: `/opt/backup-restore/logs/snapshots.log`
+
+### 3. Snapshot Storage (`/opt/backup-restore/snapshots/`)
+- Structure: `/<domain>/<YYYY-MM-DD_HHMMSS>/`
+- 9 WordPress domains, 10 snapshots each (10 days retention shown)
+- Average snapshot size: 16MB files + 74KB database
+- Total: ~1.4GB for full snapshot set
+
+### 4. Restore Log (`/opt/backup-restore/logs/restore.log`)
+- Pipe-delimited format: `timestamp|domain|snapshot_id|status`
+- Written by api_restore() on every restore attempt
+- Read by /api/log → displayed in Restore History table
+- Last 50 entries retained
+
+### 5. Caddy Proxy (on Core)
+- `handle /api/backup` → app3:8090
+- `handle /api/restore` → app3:8090 (flush_interval -1, 300s read/write timeouts)
+- `handle /api/download/*` → app3:8090
+- `handle /api/log` → app3:8090
+- `handle_path /backups/*` → app3:8090 (300s timeouts for long restores)
+- Domain: my.itpropartner.com
+
+## 9 Hosted WordPress Sites
+
+All served by CloudPanel on app3, backed up by this system:
+
+| Domain | htdocs Path | DB Pattern |
+|---|---|---|
+| apextrackexperience.com | /home/apx/htdocs/apextrackexperience.com | wp-config DB_NAME |
+| boxpilotlogistics.com | /home/boxpilotlogistics/htdocs/boxpilotlogistics.com | wp-config DB_NAME |
+| debtrecoveryexperts.com | /home/debtrecoveryexperts/... | wp-config DB_NAME |
+| iamgmb.com | /home/iamgmb/... | wp-config DB_NAME |
+| katiewattdesign.com | /home/katiewattdesign/htdocs/katiewattdesign.com | wp-config DB_NAME |
+| katiewattsdesign.com | /home/katiewattsdesign/... | wp-config DB_NAME |
+| mainwp.itpropartner.com | /home/mainwp/... | wp-config DB_NAME |
+| vigilanttac.com | /home/vigilanttac/... | wp-config DB_NAME |
+| voipsimplicity.com | /home/voipsimplicity/... | wp-config DB_NAME |
+
+## Key Design Decisions
+
+1. **Single-file Flask app:** No package structure needed — the app has 6 endpoints and one HTML template. Keeping it in one file makes deployment trivial (scp + systemctl restart).
+
+2. **Caddy on Core as single entry point:** app3 isn't exposed to the internet directly. All access goes through Core's Caddy with proper timeouts. The restore operation takes 30-45s and Caddy's default proxy timeout was killing connections mid-operation.
+
+3. **Tar + mysqldump over rsync:** Snapshots are point-in-time archives, not incremental backups. Each snapshot is self-contained (files.tar.gz + database.sql). Restore is a single operation with no dependency chain.
+
+4. **No auth on backup API:** The endpoints have no authentication. Access is controlled by Caddy routing — only requests through my.itpropartner.com reach the app. Internal network only.
@@ -0,0 +1,30 @@
+# Backup-Restore Changelog
+
+## 2026-07-20 — Restore logging and manual backup
+
+### Added
+- **Backup Now button:** Green "+ Backup Now" button on backup page
+- **Domain dropdown:** Select element with all 9 hosted domains
+- **Note field:** Optional "why" note saved as note.txt in snapshot
+- **Restore History section:** Auto-expanded table at bottom — Domain, Snapshot, Date/Time, Status
+- **Restore logging:** Every restore writes to `/opt/backup-restore/logs/restore.log`
+- **Status formatting:** Green OK / red FAILED with centered status column
+
+### Fixed
+- **Restore timing out:** Caddy flush_interval added + 300s transport timeouts
+- **Route ordering:** `/api/restore` decorator was stacked on backup function → reconnected to restore function
+- **API routing:** `/api/restore`, `/api/backup`, `/api/log` not proxied → added to Caddy config
+- **Mobile toggle:** Inline `display:none` on site tables removed → CSS class toggle now works
+- **Mobile touch:** role="button", tabindex="0", Enter key support added to card headers
+- **Auto-expand first domain:** Removed — all domains now start collapsed
+- **Restore History auto-expanded:** tbl-log has class="show", arrow is ▼
+
+### Changed
+- "Backup Log History" → "Restore History"
+- Config page scripts directory now shows content when clicked
+
+## 2026-07-17 — Initial deployment
+- Flask app deployed on app3 as systemd service
+- Snapshot script scheduled (1 AM, 1 PM)
+- 9 WordPress sites configured for backup
+- Caddy proxy from Core via my.itpropartner.com
@@ -0,0 +1,50 @@
+# Backup-Restore — my.itpropartner.com/backups/
+
+## Architecture
+- **Server:** app3 (152.53.241.111, netcup RS 4000)
+- **Backend:** Flask Python app at `/opt/backup-restore/app/app.py` (port 8090)
+- **Proxy:** Caddy on Core → reverse_proxy to 152.53.241.111:8090 with 300s timeouts
+- **Snapshots:** `/opt/backup-restore/snapshots/<domain>/<timestamp>/`
+- **Scheduled:** `0 1,13 * * * /opt/backup-restore/snapshot.sh` — 1 AM and 1 PM daily
+- **Systemd:** `backup-restore.service`
+- **Retention:** 30 days (auto-cleanup)
+
+## Sites Backed Up (9 domains)
+apextrackexperience.com, boxpilotlogistics.com, debtrecoveryexperts.com, iamgmb.com, katiewattdesign.com, katiewattsdesign.com, mainwp.itpropartner.com, vigilanttac.com, voipsimplicity.com
+
+## Snapshot Contents
+Each snapshot directory contains:
+- `files.tar.gz` — WordPress document root tarball
+- `database.sql` — MySQL dump
+- `size.txt` — Total backup size in bytes
+- `note.txt` — Optional manual backup note
+
+## API Endpoints
+| Method | Path | Auth | Description |
+|--------|------|------|-------------|
+| GET | /backups/ | No | Backup dashboard page |
+| POST | /api/backup | No | Trigger manual backup |
+| POST | /api/restore | No | Restore a snapshot |
+| POST | /api/delete | No | Delete a snapshot |
+| GET | /api/download/<domain>/<id> | No | Download snapshot archive |
+| GET | /api/log | No | Restore history |
+
+## Caddy Routes (on Core)
+```
+handle /api/backup → app3:8090
+handle /api/restore → app3:8090 (flush_interval -1, 300s timeouts)
+handle /api/download/* → app3:8090
+handle /api/log → app3:8090
+handle_path /backups/* → app3:8090 (300s timeouts)
+```
+
+## Recovery
+```
+systemctl restart backup-restore
+# Manual snapshot:
+/opt/backup-restore/snapshot.sh
+# Manual restore via curl:
+curl -X POST https://my.itpropartner.com/api/restore \
+  -H "Content-Type: application/json" \
+  -d '{"domain":"katiewattdesign.com","snapshot":"2026-07-20_130001"}'
+```
@@ -0,0 +1,37 @@
+# Cost Control Rollout — 2026-07-24
+
+**Trigger:** $46 additional unexpected spend on top of $155.76/3-day burn.
+**Root cause:** Unlimited LiteLLM key + GPT-5.6 Terra as default gateway model with no enforced budget, session-size, or model-allowlist guardrails.
+
+## Changes Deployed
+
+### 1. LiteLLM — New constrained team + key
+- **Team `hermes-normal-ops`**: $3.33 rolling daily cap + $100 rolling 30-day cap, 30 RPM, 250K TPM, max 3 parallel requests.
+- **Key `hermes-normal-ops-daily-capped`** (`...I3gQ`): Hard $3.33/day, model-restricted to approved list only.
+- **Allowed models**: `claude-sonnet-5`, `deepseek-v4-pro`, `deepseek-v4-flash`, `glm-5.2`, `MiniMax-M3`, `qwen3.7-plus`.
+- **Terra/GPT-5.6/Claude-Opus-4**: explicitly **excluded** from this key — LiteLLM returns HTTP 403.
+- **Legacy key `sk-...itzA`**: blocked (blocked=t in DB).
+
+### 2. Hermes config — Default routing
+- **Conductor**: `claude-sonnet-5` (admin-ai/LiteLLM proxy).
+- **Fallbacks**: `deepseek-v4-pro` → `deepseek-v4-flash` (admin-ai only).
+- **Delegation/workers**: `deepseek-v4-pro`, fallback `deepseek-v4-flash`.
+- **No automatic escalation to premium** — failure stops, not silently upgrades.
+
+### 3. Session controls
+- **Context length**: 128k tokens hard ceiling.
+- **Compression**: enabled at 50% fill, targets 20% ratio.
+- **Max turns**: 50 per session (prevents unbounded tool-call marathons).
+
+### 4. Rate limits (on LiteLLM key)
+- 30 RPM, 250K TPM, max 3 concurrent requests.
+- 429 throttle-backoff confirmed working in live logs.
+
+## Verification
+- Sonnet makes calls through new key: HTTP 200.
+- Terra through new key: HTTP 403 (blocked).
+- Old key blocked in DB: `blocked = t`.
+- Live gateway confirmed routing through `admin-ai` at `https://admin-ai.itpropartner.com/v1/`.
+
+## What's still behavioral (not enforced)
+- Model compliance is enforced at the proxy key layer. Cost caps are enforced at the key+team layer. Session size is a Hermes config setting — stickiness depends on the runtime respecting it.
@@ -0,0 +1,278 @@
+# itpp-infrastructure
+
+
+> **Last Updated:** July 17, 2026
+> **Maintainer:** Sho'Nuff
+
+---
+
+## Server Inventory
+
+### Core Server
+- **Hostname:** Core
+- **IP:** 152.53.192.33
+- **Provider:** netcup RS 2000 G12
+- **Specs:** 8 vCPU EPYC 9645, 15 GB DDR5 ECC, 512 GB NVMe
+- **OS:** Debian 13
+- **Role:** Hermes + Portals
+- **Key Services:**
+  - Hermes Agent (Telegram + cron, 22 cron jobs)
+  - Caddy reverse proxy (12 domains, auto-TLS)
+  - Ops Portal (FastAPI, port 8090)
+  - Prometheus (native, port 9090) + Grafana (native, port 3002)
+  - Uptime Kuma (Docker, port 3001) — 9+ monitors
+  - Vaultwarden (Docker, port 8080) — vault.iamgmb.com
+  - Twenty CRM (Docker) — crm.debtrecoveryexperts.com
+  - DocuSeal (Docker, port 3000) — sign.core.itpropartner.com
+  - SearXNG (Docker, port 8888)
+  - Komodo (Docker, port 9120)
+  - Tailscale, StrongSwan, WireGuard (home CCR tunnel 10.77.0.0/24)
+  - Redis cache
+
+### App1 Server
+- **Hostname:** app1
+- **IP:** 152.53.36.131
+- **Provider:** netcup RS 4000 G12
+- **Specs:** 12 vCPU EPYC 9645, 32 GB DDR5 ECC, 1 TB NVMe
+- **OS:** Debian 13
+- **Role:** AI/Service Hub
+- **Key Services:**
+  - Open WebUI (Docker, port 3000) — ai.itpropartner.com
+  - n8n + Postgres (Docker, port 5678) — n8n.itpropartner.com
+  - LiteLLM (Docker) + Postgres — admin-ai.itpropartner.com
+  - Mattermost Team Edition (Docker, port 8065) — noc.itpropartner.com
+  - Caddy (systemd, 80/443)
+  - 4 MCP servers: Browser (:8901), Filesystem (:8900), Email (:8902), Git (:8903)
+  - Super Search MCP (systemd, port 8899)
+
+### App2 Server
+- **Hostname:** app2
+- **IP:** 152.53.39.202
+- **Provider:** netcup RS 4000 G12
+- **Specs:** 12 vCPU EPYC 9645, 32 GB DDR5 ECC, 1 TB NVMe
+- **OS:** Debian 13
+- **Role:** Infrastructure Server
+- **Key Services:**
+  - Traccar GPS (Docker, port 8082 + 5000-5150) — fleettracker360.com
+  - UniFi Controller (Docker, port 8443) — unifi.itpropartner.com
+  - UNMS/UISP (10 Docker containers) — unms.forefrontwireless.com
+  - Hudu (Docker) — hudu.itpropartner.com
+  - Caddy (4 domains)
+
+### App3 Server
+- **Hostname:** app3
+- **IP:** 152.53.241.111
+- **Provider:** netcup RS 4000 G12
+- **Specs:** 12 vCPU EPYC 9645, 32 GB DDR5 ECC, 1 TB NVMe
+- **OS:** Debian 13
+- **Role:** Web Hosting + Backup Restore
+- **Key Services:**
+  - CloudPanel CE — panel.itpropartner.com
+  - Nginx (80/443) + Percona MySQL 8.4 + PHP 8.3
+  - Backup Restore System (Flask, port 8090) — my.itpropartner.com/backups
+  - WordPress sites (7 migrated from wphost02, all live):
+    - debtreecoveryexperts.com, boxpilotlogistics.com, iamgmb.com
+    - katiewattsdesign.com, vigilanttac.com, apextrackexperience.com
+    - mainwp.itpropartner.com, voipsimplicity.com, my.voipsimplicity.com
+  - Daily snapshots: 1 AM + 1 PM, 60-day retention, /opt/backup-restore/snapshots
+
+### Core-BU (Warm Standby)
+- **Hostname:** core-bu
+- **IP:** 5.161.225.131
+- **Provider:** Hetzner CPX21
+- **Specs:** 3 vCPU, 4 GB RAM, 80 GB SSD
+- **Role:** Warm standby — auto-failover if Core down
+- **Watchdog:** 5-min check, 4-cycle confirmation, S3 sync every 10 min
+
+### Legacy / Decommissioned
+- **old-ai:** 178.156.167.181 (Hetzner CPX41) — **decommissioned** (LiteLLM migrated to app1)
+- **old app1:** 87.99.144.163 (Hetzner CPX11) — **deleted**
+- **wphost02:** 5.161.62.38 (Hetzner CPX21) — **migrated to app3**
+- **Ollama:** Removed from Core (systemd) and app1 (Docker) Jul 17
+
+---
+
+## Model Fallback Chain
+
+All providers use direct API keys. GPT-5.5 quality survives through admin-ai → OpenRouter, then degrades through DeepSeek → Gemini → Grok.
+
+| # | Model | Provider | Gateway |
+|---|---|---|---|
+| Primary | GPT-5.5 | admin-ai | Self-hosted LiteLLM (app1) |
+| Fallback 1 | GPT-5.5 | OpenRouter | openrouter.ai |
+| Fallback 2 | DeepSeek v4 Pro | DeepSeek | api.deepseek.com |
+| Fallback 3 | Gemini 3.5 Flash | Google | generativelanguage.googleapis.com |
+| Fallback 4 | Grok 4.5 | xAI | api.x.ai |
+
+**Credits (Jul 17):** DeepSeek $58, OpenRouter ~$30 remaining, OpenAI/xAI/Google on pay-as-you-go
+**Health check:** Daily 8 AM cron (`model-usage-check`)
+
+---
+
+## Domain / DNS Map
+
+### ⚠️ itpropartner.com — SiteGround Nameservers Only
+
+`itpropartner.com` uses **SiteGround nameservers** exclusively. A Cloudflare zone exists (`0dc20632…`) but is NOT authoritative — records created there silently fail. All `*.itpropartner.com` changes must be manual through SiteGround panel.
+
+| Domain | IP | Server | Service |
+|---|---|---|---|
+| core.itpropartner.com | 152.53.192.33 | Core | Landing page + Grafana link |
+| ops.itpropartner.com | 152.53.192.33 | Core | Ops dashboard |
+| sign.core.itpropartner.com | 152.53.192.33 | Core | DocuSeal |
+| uptimekuma.itpropartner.com | 152.53.192.33 | Core | Uptime monitoring |
+| gps.fleettracker360.com | 152.53.192.33 | Core | Traccar HTTPS proxy → app2 |
+| my.itpropartner.com | 152.53.192.33 | Core | Customer portal hub |
+| hudu.itpropartner.com | 152.53.39.202 | app2 | IT documentation |
+| unifi.itpropartner.com | 152.53.39.202 | app2 | UniFi controller |
+| panel.itpropartner.com | 152.53.241.111 | app3 | CloudPanel CE |
+| ai.itpropartner.com | 152.53.36.131 | app1 | Open WebUI |
+| n8n.itpropartner.com | 152.53.36.131 | app1 | n8n automation |
+| admin-ai.itpropartner.com | 152.53.36.131 | app1 | LiteLLM |
+
+### Cloudflare-Managed Domains
+
+| Domain | IP | Server | Service |
+|---|---|---|---|
+| fleettracker360.com | Cloudflare | app2 | Fleet tracking website |
+| gps.fleettracker360.com | Cloudflare → Core | Core → app2 | Traccar devices |
+| voipsimplicity.com | Cloudflare | app3 | VoIP marketing site |
+| my.voipsimplicity.com | Cloudflare | app3 | VoIP customer portal |
+| portal.debtrecoveryexperts.com | 152.53.192.33 | Core | DRE portal |
+| crm.debtrecoveryexperts.com | Cloudflare Access | — | DRE CRM |
+| vault.iamgmb.com | 152.53.192.33 | Core | Vaultwarden |
+| sign.iamgmb.com | 152.53.192.33 | Core | Document signing |
+| shark.iamgmb.com | 152.53.192.33 | Core | Shark game |
+
+### DNS PENDING (create at SiteGround)
+
+| Subdomain | → IP | Service |
+|---|---|---|
+| vault.itpropartner.com | 152.53.36.131 | Vaultwarden (after migration) |
+| status.itpropartner.com | 152.53.192.33 | Public status page |
+
+---
+
+## Backup Pipeline
+
+| Backup | Schedule | Target | Purpose |
+|---|---|---|---|
+| hermes-live-sync | Every 15 min | s3://hermes-vps-backups/live/ | Live state sync |
+| hermes-full-backup | Daily 1 AM | s3://hermes-vps-backups/hermes-full-backup/ | Full Hermes backup |
+| home-router-backup | Daily 6 AM | s3://mikrotik-ccr-backups/ | CCR config |
+| root-essentials-backup | Daily 3 AM | S3 | /root essentials |
+| docker-volume-sync | Daily 3 AM | S3 | Docker volumes |
+| system-config-sync | Daily 4 AM | S3 | System configs |
+| unms-backup-sync | Daily 6 AM (Core) | s3://hermes-vps-backups/unms-backups/ | UNMS data (pulled from app2) |
+| unifi-backup-sync | Daily 2 AM (Core) | s3://hermes-vps-backups/unifi-backups/ | UniFi configs (pulled from app2) |
+| hudu-backup | Daily 7 AM | s3://hermes-vps-backups/hudu/backups/ | Hudu volume dump |
+| gitea-backup | Daily 8 AM | s3://hermes-vps-backups/gitea/daily/ | Gitea repos |
+| app1-backup | Daily 2 AM | s3://hermes-vps-backups/app1/ | LiteLLM, n8n, OpenWebUI, MCP, Mattermost |
+| app2-backup | Daily 2:30 AM | s3://hermes-vps-backups/app2/ | Traccar, Gitea, Hudu, UNMS, UniFi |
+| app3-backup | Daily 3 AM | s3://hermes-vps-backups/app3/ | CloudPanel, MySQL, WordPress |
+| wphost02-backup | Daily 5 AM | s3://hermes-vps-backups/wphost02-backup/ | Webapps + MySQL |
+| warm-standby-sync | Every 10 min | core-bu ← S3 | DR readiness |
+
+---
+
+## Caddy Configuration
+
+### Core (152.53.192.33)
+
+```
+core.itpropartner.com           → static files
+sign.core.itpropartner.com      → localhost:3000 (DocuSeal)
+ops.itpropartner.com            → 127.0.0.1:8090 + static
+uptimekuma.itpropartner.com     → 127.0.0.1:3001 (Uptime Kuma)
+gps.fleettracker360.com         → app2:8082 (Traccar)
+my.itpropartner.com             → static files
+portal.debtrecoveryexperts.com  → static files
+vault.iamgmb.com                → localhost:8080 (Vaultwarden)
+sign.iamgmb.com                 → 127.0.0.1:8090
+shark.iamgmb.com                → static + :8083
+```
+
+### App1 (152.53.36.131)
+
+```
+ai.itpropartner.com             → :3000 (Open WebUI)
+n8n.itpropartner.com            → :5678 (n8n)
+admin-ai.itpropartner.com       → :4000 (LiteLLM)
+app1.itpropartner.com           → static response
+```
+
+### App2 (152.53.39.202)
+
+```
+hudu.itpropartner.com           → Hudu internal
+gps.fleettracker360.com         → :8082 (Traccar)
+unms.forefrontwireless.com      → UNMS Nginx
+unifi.itpropartner.com          → :8443 (UniFi)
+```
+
+---
+
+## Service Access
+
+| Service | URL | Location | Auth |
+|---|---|---|---|
+| Open WebUI | https://ai.itpropartner.com | app1 | Chat UI |
+| Open WebUI Admin | https://admin-ai.itpropartner.com/ui | app1 | user: admin, pw: LITELLM_MASTER_KEY |
+| Ops Portal | https://ops.itpropartner.com | Core | Internal dashboard |
+| Grafana | http://core.itpropartner.com:3002 | Core | admin/admin |
+| Uptime Kuma | https://uptimekuma.itpropartner.com | Core | Service monitoring |
+| Vaultwarden | https://vault.iamgmb.com | Core | Password vault |
+| CloudPanel | https://panel.itpropartner.com | app3 | user: gmb / SQLite auth |
+| Traccar | https://gps.fleettracker360.com | app2 | GPS fleet tracking |
+| UniFi | https://unifi.itpropartner.com | app2 | Network controller |
+| UNMS | https://unms.forefrontwireless.com | app2 | WISP management |
+| Hudu | https://hudu.itpropartner.com | app2 | IT documentation |
+| n8n | https://n8n.itpropartner.com | app1 | Automation |
+| CRM (DRE) | https://crm.debtrecoveryexperts.com | Cloudflare Access | TwentyCRM |
+
+### MCP Access (from Open WebUI)
+
+| MCP Server | Location | Port | Tools |
+|---|---|---|---|
+| Super Search | app1 | :8899 | 10 tools — web_search, web_extract, person_search, email_search, phone_search, etc. |
+| Browser | app1 | :8901 | browser_navigate, browser_snapshot, browser_click, browser_type, browser_console |
+| Filesystem | app1 | :8900 | read_file, write_file, search_files, list_dir, file_info |
+| Email | app1 | :8902 | search_emails, send_email, get_email |
+| Git/Gitea | app1 | :8903 | clone, commit, push, pull |
+
+---
+
+## SSH Access
+
+- **Key:** `itpp-infra` (deployed to all servers)
+- **User:** `ippadmin` (sudo privileges)
+- **Root SSH:** Enabled on app1, app2, app3 (key-only exception per provisioning standard)
+- **Core SSH:** `ssh -i /root/.ssh/itpp-infra root@152.53.192.33`
+
+---
+
+## Firewall
+
+UFW is enabled on all servers. Standard rules:
+- **Core:** 22, 80, 443, 3000, 3001, 3002, 8080, 8082, 8090, 8443, 9090
+- **app1:** 22, 80, 443, 3000, 5678, 8899, 8900, 8901, 8902, 8903
+- **app2:** 22, 80, 443, 3000, 8080, 8082, 8089, 8443, 8843, 3478, 10001, 5000:5150
+- **app3:** 22, 80, 443, 8443
+
+---
+
+## SSL
+
+All SSL certificates issued via Let's Encrypt through Caddy. All certs auto-renew. No manual management needed.
+
+---
+
+## Notes
+
+- **DNS trap:** `itpropartner.com` uses SiteGround nameservers. Cloudflare zone is NOT authoritative. Always verify with `dig NS domain.com` before creating records.
+- **SiteGround:** No API access. All DNS changes are manual through SiteGround panel.
+- **Provider diversity:** core-bu stays at Hetzner specifically so a netcup outage can't kill both Core and standby simultaneously.
+- **app3 MySQL:** Root password in Vaultwarden + `/root/.my.cnf` on app3, accessible via 127.0.0.1:3306.
+- **CloudPanel:** SQLite DB at `/home/clp/htdocs/app/data/db.sq3` — users live here, not in MySQL.
+- **AWS CLI PATH:** All backup scripts must use `/opt/awscli-venv/bin/aws` or `source /opt/awscli-venv/bin/activate` — `aws` bare fails in cron context (PATH doesn't include venv bin). Documented in server-provisioning-standard v1.3.0.
+- **Backup verification:** Always run at least one manual backup after provisioning a server and verify it landed in S3 — never trust cron entries alone. Silent failures (`aws: command not found`, wrong file paths, S3 permission issues) won't surface otherwise.
@@ -0,0 +1,252 @@
+# IT Pro Partner — Complete Key Inventory
+
+**Generated:** 2026-07-23
+**Sanitized:** 2026-07-23 (plaintext secrets replaced with storage references)
+**Scope:** All SSH keys, API tokens, service credentials, device keys, and passwords across the infrastructure
+**⚠️ SENSITIVE:** All credential values live in the listed storage locations. See Hudu for API keys (layout 49).
+
+---
+
+## 1. SSH Keys
+
+| Key Name | File | Type | Fingerprint (SHA256) | Purpose | Deployed To |
+|----------|------|------|-----------------------|---------|-------------|
+| **itpp-infra** | `/root/.ssh/itpp-infra` | ED25519 | `Jxh0bbT9dUV3q1DYYB3hHyhy/1TDj7Q8U4xrVmB38uQ` | Universal server admin key | All servers (Core, app1, app2, app3, wphost02, app1-bu, home router) |
+| **wisp_rsa** | `/root/.ssh/wisp_rsa` | ED25519 | `MxQw1oh90NibSgN2mDbKP+07/jE4FEUEBbFAzuk5DcI` | WISP MikroTik CCR router SSH | Home CCR router (10.77.0.2 via WireGuard) |
+| **germaine-personal** | `/root/.ssh/germaine-personal` | ED25519 | `dDbLH+bdPFcGU0mm1DpGa43ec0nUZ88YnpCi4p63y3I` | Germaine's personal key (from his machines) | Germaine's devices → Core |
+| **homelab** | `/root/.ssh/homelab` | ED25519 | `c1nts4wR9EU06/O/k895Pb2tGZublgnGWG6NoQrK/qs` | Homelab Proxmox/QNAP access | vm-host-01, vm-host-02, QNAP NAS |
+| **siteground.key** | `/root/.ssh/siteground.key` | RSA (encrypted) | N/A (RSA, encrypted) | SiteGround SFTP backup (port 18765) | SiteGround shared hosting |
+| **authorized_keys** | `/root/.ssh/authorized_keys` | — | — | Who can SSH into Core | Core (this server) |
+
+### SSH Key Details
+
+```
+itpp-infra.pub:    ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAII4dxTH11aJkBqCY8lXl1kTfZ8yXWhTcthHnt1MtAuIE itpp-infra
+wisp_rsa.pub:      ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDnI4UwwPL8gJvtP/Jr7qiw0Qj/bQBwi2+f03p730xvn wisp-backup
+germaine-personal.pub: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAID2H/2VMn8i7YSUUpcag6yXiI6nB3T99h7JIOs5/+73r germaine@itppartner
+homelab.pub:       ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHT+727Cti4cZ2x6CiYDeDKZ9BhvCJCzTHlO9vMInHie homelab-itpp
+```
+
+---
+
+## 2. Server Root Passwords
+
+**Storage:** Hudu (Vaultwarden asset) + `/root/.hermes/.env` (netcup CCP section)
+
+| Server | IP | Provider | Access | Notes |
+|--------|-----|----------|--------|-------|
+| **Core** | 152.53.192.33 | netcup RS 2000 | SSH key only | `itpp-infra` key, password auth disabled |
+| **app1** | 152.53.36.131 | netcup RS 4000 | Via root or ippadmin+sudo | Root password in Vaultwarden |
+| **app2** | 152.53.39.202 | netcup RS 4000 | Via root or ippadmin+sudo | Root password in Vaultwarden |
+| **app3** | 152.53.241.111 | netcup RS 4000 | Via root or ippadmin+sudo | Root password in Vaultwarden |
+| **app1-bu** | 5.161.114.8 | Hetzner CPX11 | itpp-infra SSH key | Warm standby, offline by default |
+
+### Admin Account (all servers)
+
+- **Username:** `ippadmin`
+- **Password:** → Vaultwarden entry "ippadmin"
+- **Sudo:** Yes (full sudo access)
+- **SSH:** Key-based only (`itpp-infra`)
+
+---
+
+## 3. Cloud & Infrastructure API Keys
+
+**Storage:** All in `~/.hermes/.env` + Hudu API assets (layout 49)
+
+| Service | Hudu Asset | Storage Location | Status |
+|---------|-----------|------------------|--------|
+| **Hetzner Cloud** | [177] | `/root/.hermes/scripts/.hetzner_token` + `/root/.hermes/.env` | ✅ Verified Jul 22 |
+| **Cloudflare DNS** | [165] | `~/.hermes/.env` → `CLOUDFLARE_API_TOKEN` | ✅ Active (verified by health check) |
+| **Wasabi S3** | [176] | `/root/.aws/credentials` | ✅ Active |
+| **netcup API** | [166] | `~/.hermes/.env` → `NETCUP_API_KEY` | ✅ Active |
+| **netcup CCP** | [167] | `~/.hermes/.env` → `NETCUP_CUSTOMER_NUMBER` + `NETCUP_CCP_PASSWORD` | ✅ Active |
+| **Gitea (OLD/DEAD)** | — | ⚠️ **EXPIRED** — still in homelab + itpp-infrastructure remotes | ❌ INVALID (verified Jul 23) |
+| **Gitea (ACTIVE)** | — | All other repos + `gitea-backup.sh` — ippadmin | ✅ Active (verified Jul 23) |
+
+---
+
+## 4. AI Provider API Keys
+
+All stored in `/root/.hermes/.env` and Hudu API assets (layout 49).
+
+| Provider | Hudu Asset | Purpose | Status |
+|----------|-----------|---------|--------|
+| **admin-ai** (LiteLLM) | [126] Hermes Primary Key | Primary model gateway (all models) | ✅ Active |
+| **Anthropic** | [150] | Claude models | ✅ Active |
+| **OpenAI** | [149] | GPT models | ✅ Active |
+| **DeepSeek** | [152] | DeepSeek models | ✅ Active |
+| **Google Gemini** | [161] / [151] | Gemini models | ✅ Active |
+| **xAI (Grok)** | [154] | Grok models | ✅ Active |
+| **OpenRouter** | [153] | Multi-provider routing | ✅ Active |
+| **Mistral** | [155] | Mistral models | ✅ Active |
+| **Groq** | [157] | Fast inference | ✅ Active |
+| **Fireworks AI** | [156] | Serverless inference | ✅ Active |
+| **Perplexity** | [159] | Search-augmented LLM | ✅ Active |
+| **Cohere** | [158] | Cohere models | ✅ Active |
+| **AI21 Labs** | [160] | Jurassic models | ✅ Active |
+| **MiniMax** | [187] | MiniMax M3 | ✅ Active |
+| **Z.ai (GLM)** | [188] | GLM models | ✅ Active |
+| **Alibaba Qwen** | [189] Alibaba Qwen (DashScope) | Qwen models | ✅ Active |
+| **Deepgram** | [162] | STT (voice transcription) | ✅ Active |
+
+---
+
+## 5. Communication APIs
+
+**Storage:** All in `~/.hermes/.env` + Hudu API assets (layout 49)
+
+| Service | Hudu Asset | Storage | Status |
+|---------|-----------|---------|--------|
+| **Telegram Bot** | [170] | `~/.hermes/.env` → `TELEGRAM_BOT_TOKEN` | ✅ Active |
+| **Twilio (Live)** | [184] Twilio Live | `~/.hermes/.env` → `TWILIO_ACCOUNT_SID` + `TWILIO_AUTH_TOKEN` | ✅ Active |
+| **Twilio (Test)** | [185] Twilio Test | `~/.hermes/.env` → `TWILIO_TEST_ACCOUNT_SID` + `TWILIO_TEST_AUTH_TOKEN` | ✅ Active |
+| **Twilio API Key** | [186] Twilio API Key | `~/.hermes/.env` → `TWILIO_API_KEY_SID` + `TWILIO_API_KEY_SECRET` | ✅ Active |
+| **ElevenLabs** | [148] | `~/.hermes/config.yaml` (auxiliary vision / TTS) | ✅ Active |
+| **Email SMTP/IMAP** | — | `/root/.config/himalaya/shonuff.pass` | ✅ Active |
+| **Email account** | — | `shonuff@germainebrown.com` — MXroute via mail.germainebrown.com:2525 (SMTP) / :993 (IMAP) | ✅ Active |
+
+---
+
+## 6. VoIP / RingLogix
+
+**Storage:** `~/.hermes/.env` + Hudu API assets
+
+| Credential | Hudu Asset | Storage |
+|-----------|-----------|---------|
+| **RingLogix Client ID** | [174] | `~/.hermes/.env` → `RINGLOGIX_CLIENT_ID` |
+| **RingLogix Client Secret** | [175] | `~/.hermes/.env` → `RINGLOGIX_CLIENT_SECRET` |
+| **RingLogix Username** | — | `~/.hermes/.env` → `RINGLOGIX_USERNAME` |
+| **RingLogix Password** | — | `~/.hermes/.env` → `RINGLOGIX_PASSWORD` |
+| **RingLogix Domain** | — | `~/.hermes/.env` → `RINGLOGIX_DOMAIN` |
+
+---
+
+## 7. MSP / RMM / Security APIs
+
+**Storage:** All in `~/.hermes/.env` + Hudu API assets
+
+| Service | Hudu Asset | Storage | Status |
+|---------|-----------|---------|--------|
+| **SyncroMSP** | [168] Token + [169] API Key | `~/.hermes/.env` → `SYNCROMSP_API_TOKEN` + `SYNCROMSP_API_KEY` | ✅ Active |
+| **Bitdefender GZ** | [172] | `~/.hermes/.env` → `BITDEFENDER_API_KEY` | ✅ Active |
+| **VirusTotal** | [171] | `~/.hermes/.env` → `VIRUSTOTAL_API_KEY` | ✅ Active |
+| **UISP/UNMS** | [173] | `~/.hermes/.env` → `UISP_API_KEY` | ✅ Active |
+
+---
+
+## 8. Search & Data APIs
+
+| Service | Hudu Asset | Storage | Status |
+|---------|-----------|---------|--------|
+| **Firecrawl** | [164] | `~/.hermes/.env` → `FIRECRAWL_API_KEY` | ✅ Active |
+| **Exa AI Search** | [163] | `~/.hermes/.env` → `EXA_API_KEY` | ✅ Active |
+
+---
+
+## 9. Database Credentials
+
+| Database | Host | User | Password Location | Purpose |
+|----------|------|------|-------------------|---------|
+| **MySQL (apex track)** | 127.0.0.1:33060 (SSH tunnel from wphost02) | `apextrackexperience_1781549652` | `wp-config.php` on wphost02 | Apex Track Experience WordPress |
+| **MySQL (CloudPanel)** | app3:3306 | `root` | `/root/.my.cnf` on app3 (also in Vaultwarden) | CloudPanel WordPress hosting |
+| **LiteLLM Postgres** | app1 (Docker) | (in docker-compose) | `/root/docker/litellm/docker-compose.yml` on app1 | LiteLLM operational DB |
+
+---
+
+## 10. Docker Services
+
+| Service | URL | Credential Location | Storage |
+|---------|-----|--------------------|---------|
+| **Vaultwarden** | vault.itpropartner.com / vault.iamgmb.com | Admin Token → `/root/docker/vaultwarden/.env` on Core | Docker env file |
+| **DRE Portal** | portal.debtrecoveryexperts.com | Basic Auth (htpasswd) | `/etc/caddy/dre-passwd` |
+| **SearXNG** | (internal, no public endpoint) | (none) | — |
+| **DocuSeal** | sign.core.itpropartner.com / sign.iamgmb.com | (none / app-managed) | — |
+| **Uptime Kuma** | uptimekuma.itpropartner.com | (app-managed) | — |
+| **Open WebUI** | admin-ai.itpropartner.com | `admin@itpropartner.com` (password: ask Sho'Nuff) | Not in .env |
+| **Mealie** | recipe.iamgmb.com | `G@germainebrown.com` (password → Vaultwarden) | Vaultwarden |
+| **Ops Portal** | ops.itpropartner.com | `ippadmin` (password → `~/.hermes/.env`) | `~/.hermes/.env` |
+
+---
+
+## 11. VPN & Network Keys
+
+### WireGuard (Core)
+
+| Item | Value |
+|------|-------|
+| **Interface** | `wg0` |
+| **Core Private Key** | → `/etc/wireguard/wg0.conf` on Core |
+| **Core IP** | `10.77.0.1/24` |
+| **Listen Port** | `51821` |
+| **Home Peer Public Key** | `1fPwdGQ20CxlZCQZQV134olDcE91hfp78yNDeaKJZzg=` |
+| **Home Peer Endpoint** | `76.195.7.60:13231` |
+| **Routed Networks** | `10.1.0.0/16`, `10.2.0.0/16`, `172.16.1.0/24`, `172.18.18.0/24` |
+
+### Tailscale
+
+| Node | IP | Type | Status |
+|------|-----|------|--------|
+| core | 100.71.155.7 | Linux | ✅ Online |
+| app1 | 100.90.186.109 | Linux | ✅ Online |
+| app2 | 100.117.164.66 | Linux | ✅ Online |
+| app3 | 100.72.15.12 | Linux | ✅ Online |
+| app1-bu | 100.112.23.21 | Linux | ⚠️ Offline (7d) |
+| iphone-15-pro-max | 100.106.231.86 | iOS | ✅ Online |
+| ipp-g-lap | 100.120.64.120 | macOS | ✅ Online |
+| m4-mac-mini | 100.116.232.65 | macOS | ✅ Online |
+
+---
+
+## 12. UniFi / UDM Pro Device Keys
+
+| Site | Key Location | Type | Status |
+|------|-------------|------|--------|
+| **Grand Lake Club** | UniFi Network Controller → Settings → API | Local Network API Key | ✅ Stored, pending direct verification |
+| **Liberty Tire** | UniFi Network Controller → Settings → API | Local Network API Key | ✅ Stored, pending direct verification |
+
+---
+
+## 13. Unknown / Not Found
+
+The following credentials are known to exist but were not found in the standard locations:
+
+| Item | Notes |
+|------|-------|
+| **Open WebUI admin password** | Recovery manual says "in .env or ask Sho'Nuff" — NOT in current .env. Must ask Germaine. |
+| **Hudu API key** | In skill docs (`hudu-management`) — used programmatically, not in .env. |
+| **Traccar/FleetTracker360 admin** | Not in .env. May be Docker env or app-managed. |
+| **Twenty CRM credentials** | Docker on Core, env at `/root/docker/twenty/.env` (not read). |
+| **WordPress site DB passwords** | Various sites, typically in `wp-config.php` on wphost02 or app3. |
+| **app1-bu root password** | Hetzner CPX11 — accessed via itpp-infra SSH key only. |
+| **ComfyUI / Z4** | GPU server allocated for TripFlow — credentials not yet documented. |
+| **Home MikroTik admin** | SSH via `admin@10.77.0.2` with `wisp_rsa` key. RouterOS password in router config (not extracted). |
+
+---
+
+## 14. Key Rotation & Audit Notes
+
+- **Last full audit:** 2026-07-23
+- **Last sanitization:** 2026-07-23 — all plaintext secrets removed; use Hudu + file paths for values
+- **Hetzner token:** Rotated Jul 22 (old tokens in Hudu were invalid)
+- **Twilio:** Live + test credentials both present in Hudu [184]/[185]/[186]
+- **OpenRouter:** Fallback routing key — keep active even if not primary
+- **admin-ai:** Primary gateway — all model calls route through this
+- **Backups:** All .env + config files included in daily Hermes backup to S3
+
+### Recovery Priority
+
+If Core is lost, you need these to rebuild (in order):
+1. `/root/.ssh/itpp-infra` — SSH to all servers
+2. `/root/.hermes/.env` — All API keys and secrets
+3. `/root/.aws/credentials` — S3 access for backups
+4. `/root/.hermes/config.yaml` — Full Hermes config
+5. `/root/.config/himalaya/shonuff.pass` — Email access
+
+### Hudu API Assets (layout 49)
+
+All API keys are documented as Hudu assets. List them via:
+```
+GET https://hudu.itpropartner.com/api/v1/companies/1/assets?page=1&per_page=25
+```
+Filter by `asset_layout_id == 49` to see all API keys with their Hudu asset IDs and storage locations.
@@ -0,0 +1,29 @@
+July 21, 2026
+
+Department of the Treasury
+Internal Revenue Service
+Ogden, UT 84201-0027
+
+RE: Business Name Change Notification
+EIN: [INSERT EIN]
+Previous Legal Name: CG Premier Transport LLC
+New Legal Name: IT Pro Partner LLC
+
+To whom it may concern,
+
+This letter is to notify the Internal Revenue Service of a legal name change for the above-referenced entity. The name change was filed and approved by the Georgia Secretary of State.
+
+Enclosed:
+- Copy of filed Georgia Articles of Amendment confirming the name change from CG Premier Transport LLC to IT Pro Partner LLC
+- This notification letter
+
+Please update your records accordingly. The entity type (LLC) and EIN remain unchanged. All other information — business address, responsible party, and tax classification — remains the same as previously filed.
+
+If you require additional documentation, please contact me at the address or phone number below.
+
+Sincerely,
+
+_______________________________
+[Name of Authorized Member/Officer]
+[Title]
+[Phone Number]
@@ -0,0 +1,18 @@
+# AI Model Chain — IT Pro Partner
+
+**Updated:** July 24, 2026
+
+## Active Fallback Chain
+
+| Tier | Model | Provider | Notes |
+|---|---|---|---|
+| **Primary** | `claude-sonnet-5` | `admin-ai` | Conductor via LiteLLM ($3.33/day cap) |
+| **Fallback 1 (F1)** | `deepseek-v4-pro` | `deepseek` | Direct DeepSeek API |
+| **Fallback 2 (F2)** | `gpt-5.6-terra` | `admin-ai` | Demoted from Primary via LiteLLM ($3.33/day cap) |
+| **Fallback 3 (F3)** | `grok-4.5` (`grok-2-1212`) | `xai` | Direct xAI API |
+| **Fallback 4 (F4)** | `gemini-3.6-flash` | `google` | Direct Google API |
+
+## Admin-AI Virtual Key
+- **Key Hash**: `0237186aaff1ed90295c00d73103103900d4bd07252ec1806af8a4358e45d37a`
+- **Allowed Models**: `claude-sonnet-5`, `gpt-5.6-terra`, `deepseek-v4-pro`, `deepseek-v4-flash`, `glm-5.2`, `MiniMax-M3`, `qwen3.7-plus`
+- **Daily Budget Cap**: $3.33 / day
@@ -0,0 +1,112 @@
+# Ops Portal — Architecture
+
+## Topology
+
+```
+                         INTERNET
+                            |
+                      [Caddy :443]
+                            |
+                    Core (152.53.192.33)
+                            |
+            +---------------+---------------+
+            |               |               |
+     /api/* :8090    /data/* :files     /static/*
+            |               |               |
+     [FastAPI app]    ops-status.json   [HTML/CSS/JS]
+     server.py       /var/www/ops/     /opt/ops-portal/
+            |            /data/           static/
+            |
+    +-------+-------+-------+-------+
+    |       |       |       |       |
+  S3 API  UISP   Wazuh  Bitdef  systemd
+ (Wasabi) (FFW) (app1) (Cloud) (Core)
+```
+
+## Data Flow
+
+```
+[Collector]                     [Dashboard]
+     |                               |
+     |-- python3 ops-data-           |
+     |   collector.py                |
+     |                               |
+     v                               |
+  S3 buckets ----+                   |
+  UISP API ------+---> ops-status    |
+  Wazuh API -----+     .json ------> GET /api/status
+  Bitdefender ---+                   |
+  systemd -------+                   |
+  cron jobs -----+                   |
+                                     v
+                              [Browser renders
+                               health grid,
+                               widgets, alerts]
+```
+
+## Components
+
+### 1. Collector (`/root/.hermes/scripts/ops-data-collector.py`)
+- Runs every 5 min via cron
+- Gathers: S3 backup status (6 buckets), UISP devices (90), Wazuh agents/alerts, Bitdefender endpoints, systemd services, cron jobs, server health, disk/memory/CPU
+- Timeout: 90s (was 20s — too short for 94K-file S3 bucket)
+- Output: `/var/www/ops/data/ops-status.json`
+
+### 2. Backend (`/opt/ops-portal/server.py`)
+- FastAPI on port 8090
+- 7 API endpoints (health, status, servers, servers/health, audit-log, ft360/status)
+- JWT auth from `/root/.hermes/.env` (ADMIN_USERNAME, ADMIN_PASSWORD, JWT_SECRET)
+- Critical service restart protection (hermes, caddy, ops-portal blocked)
+- Systemd: `ops-portal.service`
+
+### 3. Frontend (`/opt/ops-portal/static/`)
+- 11 HTML pages with shared ops.css, app.js, utils.js
+- Auth: login overlay → localStorage JWT → all API calls Bearer
+- Auto-refresh: 60s interval + tab visibility API
+- Mobile: hamburger toggle with .nav-links.open CSS
+- Cache-busting: all assets versioned with timestamps
+
+### 4. Proxy (Caddy on Core)
+- `/` and `/*.html` → static file server from `/opt/ops-portal/static/`
+- `/api/*` → reverse_proxy to 127.0.0.1:8090
+- `/data/*` → file server from `/var/www/ops/data/`
+- Domain: ops.itpropartner.com
+
+## Cross-Service Dependencies
+
+| Dependency | Server | Purpose | Fallback |
+|---|---|---|---|
+| Wasabi S3 | External | Backup bucket status | Shows "Issues" |
+| UISP API | unms.forefrontwireless.com | Device/site count | Shows 0 devices |
+| Wazuh | app1 (152.53.36.131) | Agent count, alerts | Shows "Offline" |
+| Bitdefender | External API | Endpoint monitoring | Shows "Offline" |
+| Traccar | app2 (152.53.39.202) | FleetTracker data | Dedicated endpoint |
+| Core systemd | Local | Service health, disk, memory | N/A (local) |
+
+## Auth Flow
+
+```
+Browser                    Server
+  |                          |
+  |-- POST /api/auth/login ->|
+  |   {username, password}   |
+  |                          |-- Validate against ADMIN_USERNAME/ADMIN_PASSWORD
+  |                          |-- Generate JWT with JWT_SECRET
+  |<- {access_token} --------|
+  |                          |
+  |-- GET /api/status ------->|
+  |   Authorization: Bearer   |
+  |                          |-- Verify JWT
+  |                          |-- Read ops-status.json
+  |<- {full dashboard} ------|
+```
+
+## Key Design Decisions
+
+1. **Collector pattern over direct API calls:** Dashboard fetches one JSON blob rather than 6 separate APIs. Single point of failure but fast rendering and offline-capable (shows last-cached data).
+
+2. **Python/FastAPI over Node:** Already have Python toolchain on Core. FastAPI is lightweight, async-native, and the ops portal is read-heavy with minimal write paths.
+
+3. **Static HTML + vanilla JS over React/Vue:** 11-page dashboard with no SPA routing. Auth via localStorage JWT. Zero build step, zero dependencies beyond ops.css.
+
+4. **JWT over session cookies:** Cross-page auth without server-side session state. Token survives page navigations and ops-portal restarts (persistent JWT_SECRET in .env).
@@ -0,0 +1,39 @@
+# Ops Portal Changelog
+
+## 2026-07-20 — Major audit and fix session
+
+### Fixed
+- **`/api/health` returning 404:** Caddy `handle_path` was stripping the path → changed to `handle`, port corrected to 8090
+- **`/api/servers` returning null:** Server list now returns all 5 servers with live ping health
+- **Server IPs stale:** app1-bu updated to 5.161.225.131, legacy entries removed
+- **Page titles inconsistent:** All 11 pages standardized to "X — IT Pro Partner Ops" format
+- **Missing nav icons:** All 11 nav items now have SVG icons
+- **FleetTracker360 missing from nav:** Added to navigation with car icon
+- **Backups page no data:** `s3_buckets` → `s3_backups` key fix
+- **FleetTracker360 page no nav:** Added ops.css, app.js, utils.js includes
+- **Network page dark sidebar:** Replaced with standard top nav bar
+- **Cache-busting broken:** All JS/CSS references now versioned with timestamps
+- **Mobile nav broken:** `.nav-links.open` CSS rule missing → hamburger menu now toggles properly on iOS/Android
+- **Auth guard race condition:** IIFE scripts replaced with DOMContentLoaded event listeners — pages now load data when user is authenticated
+- **Cost page broken:** Missing `loadData` function → defined and wired
+- **Dependency diagram 404:** File copied to static dir, link corrected
+- **Logs page mangled title:** Triple-nested `<title>` tags from sed accident → cleaned
+- **Config page scripts directory:** Now populates directory listing when clicked
+- **Services page:** Server column added showing "Core (152.53.192.33)"
+- **Dashboard auto-refresh on tab focus:** Visibility API handler added
+- **Critical service protection:** hermes, caddy, ops-portal restarts blocked via API
+
+### Removed
+- Duplicate server entries: "app1 (AI Stack)" and "Docker Box (legacy)"
+- Server count: 7 → 5 clean entries
+
+### Changed
+- Admin credentials: germaine/itpp2026! → ippadmin (password → Vaultwarden)
+- JWT_SECRET made persistent in /root/.hermes/.env to survive restarts
+- Collector timeout: 20s → 90s to handle 94K-file S3 bucket scanning
+
+## Jul 17, 2026 — Initial deployment
+- Ops portal deployed on Core as FastAPI app
+- Caddy reverse proxy configured
+- 10 HTML pages created
+- Ops collector built for S3, system health, server status
@@ -0,0 +1,55 @@
+# Ops Portal — ops.itpropartner.com
+
+## Architecture
+- **Server:** Core (152.53.192.33, netcup RS 2000)
+- **Backend:** FastAPI at `/opt/ops-portal/server.py` (port 8090)
+- **Proxy:** Caddy → reverse_proxy to 127.0.0.1:8090
+- **Static files:** `/opt/ops-portal/static/` — 11 HTML pages, ops.css, app.js, utils.js
+- **Auth:** JWT via `POST /api/auth/login`, token in localStorage
+- **Data:** `/var/www/ops/data/ops-status.json` (5-min collector refresh)
+- **Collector:** `/root/.hermes/scripts/ops-data-collector.py` — Wazuh, Bitdefender, S3, UISP, system health
+- **Systemd:** `ops-portal.service`, env from `/root/.hermes/.env`
+- **Credentials:** ippadmin (password → Vaultwarden / `~/.hermes/.env`)
+
+## Pages (11 total)
+| Page | Path | Description |
+|------|------|-------------|
+| Dashboard | / | System health, widgets, audit log |
+| Services | /services.html | Systemd service control, audit log, server column |
+| Servers | /servers.html | 5 servers with ping health |
+| Network | /network.html | UISP data (44 sites, 90 devices), DNS zones |
+| Backups | /backups.html | S3 bucket status (6 buckets) |
+| FleetTracker | /fleettracker360.html | Traccar device tracking |
+| Cron Jobs | /cron.html | Hermes cron jobs with expandable scripts |
+| Config | /config.html | Active configs, /root/.hermes/scripts/ listing |
+| Logs | /logs.html | Aggregated log viewer |
+| Audit | /audit.html | Full audit trail |
+| Costs | /cost.html | API cost tracking by model |
+
+## API Endpoints
+| Method | Path | Auth | Description |
+|--------|------|------|-------------|
+| POST | /api/auth/login | No | JWT authentication |
+| GET | /api/health | No | Health check, DB status |
+| GET | /api/status | JWT | Full dashboard data (17 sections) |
+| GET | /api/servers | JWT | Server list with IPs |
+| GET | /api/servers/health | JWT | Ping health (all 7 LIVE) |
+| GET | /api/audit-log?limit=N | JWT | Audit trail entries |
+| GET | /api/ft360/status | JWT | FleetTracker device data |
+
+## Critical Services (API restart blocked)
+hermes, hermes-assistant, hermes-browser, caddy, ops-portal, mysql-tunnel
+
+## Dashboard Widgets
+- System Health — Core metrics (jobs, disk, memory, S3, APIs)
+- Wazuh Security — agent count, alerts
+- Bitdefender GravityZone — 9 managed endpoints
+- Alerts and Notifications — DR issues, backup failures, cron errors
+- Quick Actions — Restart Ops Portal
+
+## Recovery
+```
+systemctl restart ops-portal
+systemctl reload caddy
+python3 /root/.hermes/scripts/ops-data-collector.py
+```
@@ -0,0 +1,68 @@
+# Project Log — All Completed Projects
+
+## 2026-07-20
+
+### Ops Portal Audit and Overhaul
+- Full audit of all 11 pages, 7 API endpoints, and 5 dashboard widgets
+- Fixed 15 bugs: auth guards, cache-busting, mobile nav, page titles, missing icons, data keys
+- Added 3 new widgets: Wazuh Security, Bitdefender GravityZone, Alerts and Notifications
+- Standardized credentials: ippadmin (password → Vaultwarden / `~/.hermes/.env`)
+- Added critical service protection (hermes/caddy/ops-portal restart blocked via API)
+- Server list cleaned up (7→5), dependency diagram fixed, config page scripts listing
+
+### Backup-Restore Enhancements
+- Added manual backup with domain dropdown and note field
+- Added restore history logging with formatted 4-column table
+- Fixed Caddy routing and timeouts (restore was returning 404 via proxy)
+- Fixed mobile toggle on domain expansion cards
+- 9 WordPress sites under daily backup (1 AM and 1 PM)
+
+### Docs Written
+- `/root/projects/ops-portal/README.md` + `CHANGELOG.md`
+- `/root/projects/backup-restore/README.md` + `CHANGELOG.md`
+
+---
+
+## 2026-07-17 — Backup-Restore Initial Deployment
+- Flask backup/restore app deployed on app3 (152.53.241.111)
+- Daily snapshots scheduled at 1 AM and 1 PM
+- Caddy reverse proxy from my.itpropartner.com
+- 9 WordPress sites configured
+
+## 2026-07-21 — Home Lab Consolidation
+
+### Proxmox Migration
+- vm-host-02 VMs migrated/destroyed: graylog, zabbix, fog, Ubuntu-Server
+- vm-host-01 now hosts: docker-host-01, adguard-home
+- vm-host-02 cleared for GPU installation (RTX 3090 pending verification)
+- QNAP NFS shared storage created (2TB pool, mounted on both Proxmox hosts)
+
+### DNS Infrastructure
+- Technitium DNS deployed on app2 (dns1.itpropartner.com)
+- DoH upstreams: Quad9, Cloudflare, Google
+- Home DNS chain: docker-host-01 AdGuard → dns1 Technitium → vm-host-01 AdGuard
+- Firewall locked: port 53 restricted to 76.195.7.60
+
+### Twilio
+- Toll-free number verification submitted for IT Pro Partner
+- Use case: customer notifications, appointment reminders, IVR
+
+### Mattermost
+- Branding configured: IT Pro Partner NOC
+- Channel structure designed (13 channels)
+- Mobile push investigation: HPNS required for background notifications
+
+### Gift-a-Roast
+- Domain giftaroast.com purchased, DNS live (Cloudflare → app1)
+- ElevenLabs TTS + Deepgram STT keys verified
+- Architecture: Twilio Voice → STT → AI → TTS → caller
+
+### Uptime Kuma
+- Backed up (361MB, 25 monitors), updated to latest
+
+### IRS
+- Name change letter drafted: CG Premier Transport LLC → IT Pro Partner LLC
+- Georgia Secretary of State filing confirmed
+
+### Skills Updated
+- 10 skills patched: docker-service-deployment, home-lab-*, server-architecture-plan, twilio-10dlc, vaultwarden-management, voip-portal, hudu, syncromsp, recurring-information-scout
@@ -0,0 +1,10 @@
+# IT Pro Partner / GMB Projects
+
+Master index of all internal and client projects.
+
+- **[Debt Recovery Experts (DRE)](./dre/README.md)**: A specialized debt recovery platform focused on Texas mechanics liens and B2B collections. Handles client intake, compliance with Texas law, fee structures, and deliverables. (IN DEVELOPMENT)
+- **[Shark Attack Fantasy League](./shark-game/README.md)**: A fantasy league game where players draft coastal regions and earn points based on real-world shark sightings, bites, and fatalities. (IN DEVELOPMENT)
+- **[IT Pro Partner Infrastructure](./itpp-infra/README.md)**: Management of IT Pro Partner server infrastructure, backups, security baselines, and disaster recovery plans. (LIVE)
+- **[OSINT People Search](./osint-tool/README.md)**: An Open Source Intelligence tool for performing background checks, compiling data broker reports, and removing personal information. (IN DEVELOPMENT)
+- **[Apex Track Experience](./apex-track/README.md)**: Website and operations platform for track day experiences, vehicle registrations, and event logistics. (PLANNED)
+- **[BoxPilot Logistics](./boxpilot/README.md)**: Logistics and shipping management platform. (PLANNED)
@@ -0,0 +1,64 @@
+# Super Search — Cloudflare Bypass
+
+**Added:** July 21, 2026
+**Version:** 2.1.0
+
+## Extraction Chain
+
+| Tier | Provider | What it handles | Fallback Trigger |
+|---|---|---|---|
+| 1 | Trafilatura | Normal sites | Error OR CF challenge detected |
+| 2 | Browserless Chrome | CF-protected sites | Chrome render + Trafilatura |
+| 3 | Firecrawl | Everything else | API-based extraction |
+
+## CF Detection
+Nine detection markers for caught challenge pages (from Hound's code + additional):
+- cf-turnstile, challenges.cloudflare.com/turnstile
+- cf_chl_opt, __cf_chl
+- cf-browser-verification, challenge-platform, cf-mitigated
+- "Checking your browser", "Just a moment"
+
+## Infrastructure
+- Browserless Chrome on app1 (152.53.36.131), port 3005
+- Caddy proxy: app1:3006 → localhost:3005
+- Firewall: only Core (152.53.192.33) can reach port 3006
+- Super Search: `/root/docker/super-search/server.py`
+
+## Verify
+```bash
+# Test CF bypass
+cd /root/docker/super-search && source venv/bin/activate
+python3 -c "
+from server import _extract_one
+import asyncio
+r = asyncio.run(_extract_one('https://nowsecure.nl'))
+print(r['provider'])  # Should be 'trafilatura' or 'browserless'
+"
+```
+
+---
+
+## v2.1 Features (Added July 21, 2026)
+
+### 13 Tools
+| # | Tool | Description |
+|---|---|---|
+| 1-10 | Original 10 | Search, extract, lookup, suggest, images |
+| 11 | `web_search_fast` | Parallel racing: all providers fire simultaneously |
+| 12 | `screenshot` | Browserless Chrome → base64 PNG |
+| 13 | `circuit_status` | Provider health states (closed/open/half-open) |
+
+### Circuit Breaker
+- 8 provider circuits: 3 failures → open for 60s
+- Prevents hammering dead providers
+- Auto-recovers when provider comes back
+
+### Parallel Racing
+- `web_search_fast`: SearXNG ∥ Exa ∥ DuckDuckGo ∥ Wikipedia
+- First successful result wins — others cancelled
+- Typically 2-3x faster than sequential fallback
+
+### Screenshot
+- Browserless Chrome on port 3006 (Caddy HTTP proxy)
+- Full-page or viewport capture
+- Base64-encoded PNG in JSON response