sanitize: redact secrets from docs, add forefront-broadband-map, exclude backup dir

- Redact credentials/keys from README, key-inventory, model-chain, ops-portal docs, project-log, systems-networks-team, app3
- Add forefront-broadband-map.md with sanitized content
- Add .gitignore to permanently exclude .backup-before-sanitize-20260723/ and .env files
- No credentials rotated; no history rewritten

docs/key-inventory.md

@@ -1,8 +1,9 @@
 # IT Pro Partner — Complete Key Inventory
 
 **Generated:** 2026-07-23
+**Sanitized:** 2026-07-23 (plaintext secrets replaced with storage references)
 **Scope:** All SSH keys, API tokens, service credentials, device keys, and passwords across the infrastructure
-**⚠️ CLASSIFIED:** Contains real credentials — store encrypted, never email plaintext
+**⚠️ SENSITIVE:** All credential values live in the listed storage locations. See Hudu for API keys (layout 49).
 
 ---
 
@@ -30,18 +31,20 @@ homelab.pub:       ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHT+727Cti4cZ2x6CiYDeDKZ9
 
 ## 2. Server Root Passwords
 
-| Server | IP | Provider | Password | Notes |
-|--------|-----|----------|----------|-------|
-| **Core** | 152.53.192.33 | netcup RS 2000 | (SSH key only) | `itpp-infra` key, password auth disabled |
-| **app1** | 152.53.36.131 | netcup RS 4000 | `heUa1ucN6Xwta2O` | Via root or ippadmin+sudo |
-| **app2** | 152.53.39.202 | netcup RS 4000 | `NKCS0I9wMn3yy86` | Via root or ippadmin+sudo |
-| **app3** | 152.53.241.111 | netcup RS 4000 | `ab6eogprILQI6UX` | Via root or ippadmin+sudo |
-| **app1-bu** | 5.161.114.8 | Hetzner CPX11 | (itpp-infra SSH key) | Warm standby, offline by default |
+**Storage:** Hudu (Vaultwarden asset) + `/root/.hermes/.env` (netcup CCP section)
+
+| Server | IP | Provider | Access | Notes |
+|--------|-----|----------|--------|-------|
+| **Core** | 152.53.192.33 | netcup RS 2000 | SSH key only | `itpp-infra` key, password auth disabled |
+| **app1** | 152.53.36.131 | netcup RS 4000 | Via root or ippadmin+sudo | Root password in Vaultwarden |
+| **app2** | 152.53.39.202 | netcup RS 4000 | Via root or ippadmin+sudo | Root password in Vaultwarden |
+| **app3** | 152.53.241.111 | netcup RS 4000 | Via root or ippadmin+sudo | Root password in Vaultwarden |
+| **app1-bu** | 5.161.114.8 | Hetzner CPX11 | itpp-infra SSH key | Warm standby, offline by default |
 
 ### Admin Account (all servers)
 
 - **Username:** `ippadmin`
-- **Password:** `LoveMyBoys.1520!`
+- **Password:** → Vaultwarden entry "ippadmin"
 - **Sudo:** Yes (full sudo access)
 - **SSH:** Key-based only (`itpp-infra`)
 
@@ -49,116 +52,120 @@ homelab.pub:       ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHT+727Cti4cZ2x6CiYDeDKZ9
 
 ## 3. Cloud & Infrastructure API Keys
 
-| Service | Key/Tenant | Storage Location | Status |
-|---------|------------|------------------|--------|
-| **Hetzner Cloud** | `KI4DrpOPGLnQLtiEGPtu9Rap1Cw2Hg75O38hzdllqNKynkn8eFi6J2qxg4S17xxN` | `/root/.hermes/scripts/.hetzner_token` + `/root/.hermes/secrets/.hetzner_token` | ✅ Verified Jul 22 |
-| **Cloudflare DNS** | `cfut_MNpfpQeLK96lh6vnS9hK1WbrKjGWp1l1Y7jDDCBHd95bf25a` | `~/.hermes/.env` → `CLOUDFLARE_API_TOKEN` | ✅ Active (verified by health check) |
-| **Wasabi S3** | Access Key: `JGDE34XQVXTJKGAZIJYS` | `/root/.aws/credentials` | ✅ Active |
-| **Wasabi S3** | Secret Key: `tYBoEClD6jywZ3OlWHXf4nCbn1oyKZJJyGFle1Mh` | `/root/.aws/credentials` | ✅ Active |
-| **netcup CCP** | Customer: `389212`, Password: `LoveMyBoys1520!` | `~/.hermes/.env` | ✅ Active |
-| **netcup CCP** | API Key: `NHJSb0k3NG1HcjM4dEJ0NDNURDFiMThmUkg1NkEySFFTYTYxdj` | `~/.hermes/.env` → `NETCUP_API_KEY` | ✅ Active |
-| **Gitea (OLD/DEAD)** | `92ab79ea9b889fdbbce415f0ec5e54c2f7dcfd7c` | ⚠️ **EXPIRED** — still in `homelab` + `itpp-infrastructure` remotes | ❌ INVALID (verified Jul 23) |
-| **Gitea (ACTIVE)** | `1761daa2c537fb72b365e54619208329d8e3ad33` | All other repos + `gitea-backup.sh` — ippadmin | ✅ Active (verified Jul 23) |
+**Storage:** All in `~/.hermes/.env` + Hudu API assets (layout 49)
+
+| Service | Hudu Asset | Storage Location | Status |
+|---------|-----------|------------------|--------|
+| **Hetzner Cloud** | [177] | `/root/.hermes/scripts/.hetzner_token` + `/root/.hermes/.env` | ✅ Verified Jul 22 |
+| **Cloudflare DNS** | [165] | `~/.hermes/.env` → `CLOUDFLARE_API_TOKEN` | ✅ Active (verified by health check) |
+| **Wasabi S3** | [176] | `/root/.aws/credentials` | ✅ Active |
+| **netcup API** | [166] | `~/.hermes/.env` → `NETCUP_API_KEY` | ✅ Active |
+| **netcup CCP** | [167] | `~/.hermes/.env` → `NETCUP_CUSTOMER_NUMBER` + `NETCUP_CCP_PASSWORD` | ✅ Active |
+| **Gitea (OLD/DEAD)** | — | ⚠️ **EXPIRED** — still in homelab + itpp-infrastructure remotes | ❌ INVALID (verified Jul 23) |
+| **Gitea (ACTIVE)** | — | All other repos + `gitea-backup.sh` — ippadmin | ✅ Active (verified Jul 23) |
 
 ---
 
 ## 4. AI Provider API Keys
 
-All stored in `/root/.hermes/.env` unless noted.
+All stored in `/root/.hermes/.env` and Hudu API assets (layout 49).
 
-| Provider | Key Prefix | Purpose | Status |
+| Provider | Hudu Asset | Purpose | Status |
 |----------|-----------|---------|--------|
-| **admin-ai** (LiteLLM) | `sk-aEJ...itzA` | Primary model gateway (all models) | ✅ Active |
-| **Anthropic** | `sk-ant...rgAA` | Claude models | ✅ Active |
-| **OpenAI** | `sk-pro...t8MA` | GPT models | ✅ Active |
-| **DeepSeek** | `sk-038...4a1d` | DeepSeek models | ✅ Active |
-| **Google (Gemini)** | `AQ.Ab8RN6IZmtXPrQewzo3DGqbvMWfwUwSLKg16SVTx8nidn_L14A` | Gemini models | ✅ Active |
-| **xAI (Grok)** | `xai-Kl...yULd` | Grok models | ✅ Active |
-| **OpenRouter** | `sk-or-...1df6` | Multi-provider routing | ✅ Active |
-| **Mistral** | `hWEysFDGrU6jPaZIhZsXVY6QuuYON40t` | Mistral models | ✅ Active |
-| **Groq** | `gsk_mq...C4Ir` | Fast inference | ✅ Active |
-| **Fireworks** | `fw_EpWLvWX3i5XDAAfwPScbwB` | Serverless inference | ✅ Active |
-| **Perplexity** | `pplx-I...jkHR` | Search-augmented LLM | ✅ Active |
-| **Cohere** | `Sc6gSYGtm2Cszco0mIngGitFTKR7LFRhPWRS4Smj` | Cohere models | ✅ Active |
-| **AI21** | `3f510393-9523-45ef-9f31-2fc1b2d3eb5f` | Jurassic models | ✅ Active |
-| **MiniMax** | `sk-cp-...OANE` | MiniMax M3 | ✅ Active |
-| **Z.ai** | `1a1cce5be0c14d9289aa45f610679790.dik1gBAVo3ujjQFU` | GLM models | ✅ Active |
-| **Alibaba** | `sk-ws-H.XLPLPP.fHrU...` (ref file) | Qwen models | ✅ Active |
-| **Deepgram** | `50b872c92df0b9342c64d76888bdd77b9d0bd4fb` | STT (voice transcription) | ✅ Active |
+| **admin-ai** (LiteLLM) | [126] Hermes Primary Key | Primary model gateway (all models) | ✅ Active |
+| **Anthropic** | [150] | Claude models | ✅ Active |
+| **OpenAI** | [149] | GPT models | ✅ Active |
+| **DeepSeek** | [152] | DeepSeek models | ✅ Active |
+| **Google Gemini** | [161] / [151] | Gemini models | ✅ Active |
+| **xAI (Grok)** | [154] | Grok models | ✅ Active |
+| **OpenRouter** | [153] | Multi-provider routing | ✅ Active |
+| **Mistral** | [155] | Mistral models | ✅ Active |
+| **Groq** | [157] | Fast inference | ✅ Active |
+| **Fireworks AI** | [156] | Serverless inference | ✅ Active |
+| **Perplexity** | [159] | Search-augmented LLM | ✅ Active |
+| **Cohere** | [158] | Cohere models | ✅ Active |
+| **AI21 Labs** | [160] | Jurassic models | ✅ Active |
+| **MiniMax** | [187] | MiniMax M3 | ✅ Active |
+| **Z.ai (GLM)** | [188] | GLM models | ✅ Active |
+| **Alibaba Qwen** | [189] Alibaba Qwen (DashScope) | Qwen models | ✅ Active |
+| **Deepgram** | [162] | STT (voice transcription) | ✅ Active |
 
 ---
 
 ## 5. Communication APIs
 
-| Service | Credential | Storage | Status |
+**Storage:** All in `~/.hermes/.env` + Hudu API assets (layout 49)
+
+| Service | Hudu Asset | Storage | Status |
 |---------|-----------|---------|--------|
-| **Telegram Bot** | Token: `8359374835:***` (full in .env) | `~/.hermes/.env` → `TELEGRAM_BOT_TOKEN` | ✅ Active |
-| **Twilio (Live)** | SID: `AC0eb47065ac789aeda2113ec226afef08` | `~/.hermes/.env` → `TWILIO_ACCOUNT_SID` | ✅ Active |
-| **Twilio (Live)** | Auth Token: `314fd0c61f4117d54f2e8c47d821735e` | `~/.hermes/.env` → `TWILIO_AUTH_TOKEN` | ✅ Active |
-| **Twilio (Test)** | SID: `AC0e920a39a21165a06a8711ae4bcccaf2` | `~/.hermes/.env` → `TWILIO_TEST_ACCOUNT_SID` | ✅ Active |
-| **Twilio (Test)** | Auth Token: `ab6eebcedf479c7b7aa662ec394a6705` | `~/.hermes/.env` → `TWILIO_TEST_AUTH_TOKEN` | ✅ Active |
-| **Twilio API Key** | SID: `SKb8af6974d638bfc69f3e3d8fb99f49e7` | `~/.hermes/.env` → `TWILIO_API_KEY_SID` | ✅ Active |
-| **Twilio API Key** | Secret: `VXlP49wVKV0CtApPhK8q6E9TSvWKQ5oc` | `~/.hermes/.env` → `TWILIO_API_KEY_SECRET` | ✅ Active |
-| **Email SMTP/IMAP** | Password: `Catches.bullets1985` | `/root/.config/himalaya/shonuff.pass` | ✅ Active |
-| **Email account** | `shonuff@germainebrown.com` | MXroute via mail.germainebrown.com:2525 (SMTP) / :993 (IMAP) | ✅ Active |
+| **Telegram Bot** | [170] | `~/.hermes/.env` → `TELEGRAM_BOT_TOKEN` | ✅ Active |
+| **Twilio (Live)** | [184] Twilio Live | `~/.hermes/.env` → `TWILIO_ACCOUNT_SID` + `TWILIO_AUTH_TOKEN` | ✅ Active |
+| **Twilio (Test)** | [185] Twilio Test | `~/.hermes/.env` → `TWILIO_TEST_ACCOUNT_SID` + `TWILIO_TEST_AUTH_TOKEN` | ✅ Active |
+| **Twilio API Key** | [186] Twilio API Key | `~/.hermes/.env` → `TWILIO_API_KEY_SID` + `TWILIO_API_KEY_SECRET` | ✅ Active |
+| **ElevenLabs** | [148] | `~/.hermes/config.yaml` (auxiliary vision / TTS) | ✅ Active |
+| **Email SMTP/IMAP** | — | `/root/.config/himalaya/shonuff.pass` | ✅ Active |
+| **Email account** | — | `shonuff@germainebrown.com` — MXroute via mail.germainebrown.com:2525 (SMTP) / :993 (IMAP) | ✅ Active |
 
 ---
 
 ## 6. VoIP / RingLogix
 
-| Credential | Value | Storage | Status |
-|-----------|-------|---------|--------|
-| **RingLogix Client ID** | `0-t1706-c284088-r284061` | `~/.hermes/.env` → `RINGLOGIX_CLIENT_ID` | ✅ Active |
-| **RingLogix Client Secret** | `f3665f9738de9a7834b429a96b39f8cb` | `~/.hermes/.env` → `RINGLOGIX_CLIENT_SECRET` | ✅ Active |
-| **RingLogix Username** | `106@284088` | `~/.hermes/.env` → `RINGLOGIX_USERNAME` | ✅ Active |
-| **RingLogix Password** | `Bruce.leroy85` | `~/.hermes/.env` → `RINGLOGIX_PASSWORD` | ✅ Active |
-| **RingLogix Domain** | `284088` | `~/.hermes/.env` → `RINGLOGIX_DOMAIN` | ✅ Active |
+**Storage:** `~/.hermes/.env` + Hudu API assets
+
+| Credential | Hudu Asset | Storage |
+|-----------|-----------|---------|
+| **RingLogix Client ID** | [174] | `~/.hermes/.env` → `RINGLOGIX_CLIENT_ID` |
+| **RingLogix Client Secret** | [175] | `~/.hermes/.env` → `RINGLOGIX_CLIENT_SECRET` |
+| **RingLogix Username** | — | `~/.hermes/.env` → `RINGLOGIX_USERNAME` |
+| **RingLogix Password** | — | `~/.hermes/.env` → `RINGLOGIX_PASSWORD` |
+| **RingLogix Domain** | — | `~/.hermes/.env` → `RINGLOGIX_DOMAIN` |
 
 ---
 
 ## 7. MSP / RMM / Security APIs
 
-| Service | Credential | Storage | Status |
+**Storage:** All in `~/.hermes/.env` + Hudu API assets
+
+| Service | Hudu Asset | Storage | Status |
 |---------|-----------|---------|--------|
-| **SyncroMSP** | API Token: `Ta9f9d1b462271a2f4-8d63a3f025eb89451edb16f2308c2e40` | `~/.hermes/.env` → `SYNCROMSP_API_TOKEN` | ✅ Active |
-| **SyncroMSP** | API Key: `T861e9ea26ad45fc95-0eb0f2474e7a55a03a3212af80fb645e` | `~/.hermes/.env` → `SYNCROMSP_API_KEY` | ✅ Active |
-| **Bitdefender GZ** | `1af2732323e2b535b716f56bff9e7f46ddc39a01e8e2c044b4d91e120b0ab70f` | `~/.hermes/.env` → `BITDEFENDER_API_KEY` | ✅ Active |
-| **VirusTotal** | `846c46f921b26436962f569c2cbf7cb3d18a1222a112a21c93a499ece0eec21e` | `~/.hermes/.env` → `VIRUSTOTAL_API_KEY` | ✅ Active |
-| **UISP/UNMS** | `e46e0170-2ff9-4bd3-9b42-37a644969da9` | `~/.hermes/.env` → `UISP_API_KEY` | ✅ Active |
+| **SyncroMSP** | [168] Token + [169] API Key | `~/.hermes/.env` → `SYNCROMSP_API_TOKEN` + `SYNCROMSP_API_KEY` | ✅ Active |
+| **Bitdefender GZ** | [172] | `~/.hermes/.env` → `BITDEFENDER_API_KEY` | ✅ Active |
+| **VirusTotal** | [171] | `~/.hermes/.env` → `VIRUSTOTAL_API_KEY` | ✅ Active |
+| **UISP/UNMS** | [173] | `~/.hermes/.env` → `UISP_API_KEY` | ✅ Active |
 
 ---
 
 ## 8. Search & Data APIs
 
-| Service | Credential | Storage | Status |
+| Service | Hudu Asset | Storage | Status |
 |---------|-----------|---------|--------|
-| **Firecrawl** | `fc-4f9...63c1` (full in .env) | `~/.hermes/.env` → `FIRECRAWL_API_KEY` | ✅ Active |
-| **Exa AI Search** | `d1720047-10bb-47d5-95a3-74e6eabeea16` | `~/.hermes/.env` → `EXA_API_KEY` | ✅ Active |
+| **Firecrawl** | [164] | `~/.hermes/.env` → `FIRECRAWL_API_KEY` | ✅ Active |
+| **Exa AI Search** | [163] | `~/.hermes/.env` → `EXA_API_KEY` | ✅ Active |
 
 ---
 
 ## 9. Database Credentials
 
-| Database | Host | User | Password | Purpose |
-|----------|------|------|----------|---------|
-| **MySQL (apex track)** | 127.0.0.1:33060 (SSH tunnel from wphost02) | `apextrackexperience_1781549652` | `K3E1ZZWvHDu0q8ZmoBCAhzKUZawEapdGBlbaPME1sOTKgGk9FCuYS` | Apex Track Experience WordPress |
-| **MySQL (CloudPanel)** | app3:3306 | `root` | `MOQMINFQIhklM0AF` | CloudPanel WordPress hosting |
-| **LiteLLM Postgres** | app1 (Docker) | (in docker-compose) | (in docker-compose) | LiteLLM operational DB |
+| Database | Host | User | Password Location | Purpose |
+|----------|------|------|-------------------|---------|
+| **MySQL (apex track)** | 127.0.0.1:33060 (SSH tunnel from wphost02) | `apextrackexperience_1781549652` | `wp-config.php` on wphost02 | Apex Track Experience WordPress |
+| **MySQL (CloudPanel)** | app3:3306 | `root` | `/root/.my.cnf` on app3 (also in Vaultwarden) | CloudPanel WordPress hosting |
+| **LiteLLM Postgres** | app1 (Docker) | (in docker-compose) | `/root/docker/litellm/docker-compose.yml` on app1 | LiteLLM operational DB |
 
 ---
 
 ## 10. Docker Services
 
-| Service | URL | Credential | Storage |
-|---------|-----|-----------|---------|
-| **Vaultwarden** | vault.itpropartner.com / vault.iamgmb.com | Admin Token: `fJtoaIYPUb7K9pTE8d8NterWWZdgvDdwhyKGiD/+MVjxMddEF3NCNZIodC6rjoL6` | `/root/docker/vaultwarden/.env` |
-| **DRE Portal** | portal.debtrecoveryexperts.com | Basic Auth (htpasswd): Germaine/Tony/Anita | `/etc/caddy/dre-passwd` |
+| Service | URL | Credential Location | Storage |
+|---------|-----|--------------------|---------|
+| **Vaultwarden** | vault.itpropartner.com / vault.iamgmb.com | Admin Token → `/root/docker/vaultwarden/.env` on Core | Docker env file |
+| **DRE Portal** | portal.debtrecoveryexperts.com | Basic Auth (htpasswd) | `/etc/caddy/dre-passwd` |
 | **SearXNG** | (internal, no public endpoint) | (none) | — |
 | **DocuSeal** | sign.core.itpropartner.com / sign.iamgmb.com | (none / app-managed) | — |
 | **Uptime Kuma** | uptimekuma.itpropartner.com | (app-managed) | — |
 | **Open WebUI** | admin-ai.itpropartner.com | `admin@itpropartner.com` (password: ask Sho'Nuff) | Not in .env |
-| **Mealie** | recipe.iamgmb.com | `G@germainebrown.com` / `LoveMyBoys73!` | Memory (not in .env) |
-| **Ops Portal** | ops.itpropartner.com | `ippadmin` / `LoveMyBoys.1520!` | `~/.hermes/.env` |
+| **Mealie** | recipe.iamgmb.com | `G@germainebrown.com` (password → Vaultwarden) | Vaultwarden |
+| **Ops Portal** | ops.itpropartner.com | `ippadmin` (password → `~/.hermes/.env`) | `~/.hermes/.env` |
 
 ---
 
@@ -169,7 +176,7 @@ All stored in `/root/.hermes/.env` unless noted.
 | Item | Value |
 |------|-------|
 | **Interface** | `wg0` |
-| **Core Private Key** | `KkXVsdKyiYQVBbA5HbC9sF4dMX16WSmVOkVPYTq8mh8=` |
+| **Core Private Key** | → `/etc/wireguard/wg0.conf` on Core |
 | **Core IP** | `10.77.0.1/24` |
 | **Listen Port** | `51821` |
 | **Home Peer Public Key** | `1fPwdGQ20CxlZCQZQV134olDcE91hfp78yNDeaKJZzg=` |
@@ -193,10 +200,10 @@ All stored in `/root/.hermes/.env` unless noted.
 
 ## 12. UniFi / UDM Pro Device Keys
 
-| Site | Key | Type | Status |
-|------|-----|------|--------|
-| **Grand Lake Club** | `I_0UDCxrO_M5wg4hz0gd5ZfXdNlwCjkm` | Local Network API Key | ✅ Stored, pending direct verification |
-| **Liberty Tire** | `ovTGg-esdc3WbV1oMlHLxK0WSemE58kq` | Local Network API Key | ✅ Stored, pending direct verification |
+| Site | Key Location | Type | Status |
+|------|-------------|------|--------|
+| **Grand Lake Club** | UniFi Network Controller → Settings → API | Local Network API Key | ✅ Stored, pending direct verification |
+| **Liberty Tire** | UniFi Network Controller → Settings → API | Local Network API Key | ✅ Stored, pending direct verification |
 
 ---
 
@@ -207,7 +214,7 @@ The following credentials are known to exist but were not found in the standard
 | Item | Notes |
 |------|-------|
 | **Open WebUI admin password** | Recovery manual says "in .env or ask Sho'Nuff" — NOT in current .env. Must ask Germaine. |
-| **Hudu API key** | Referenced in skill docs but not found in .env or config.yaml. May be in Vaultwarden. |
+| **Hudu API key** | In skill docs (`hudu-management`) — used programmatically, not in .env. |
 | **Traccar/FleetTracker360 admin** | Not in .env. May be Docker env or app-managed. |
 | **Twenty CRM credentials** | Docker on Core, env at `/root/docker/twenty/.env` (not read). |
 | **WordPress site DB passwords** | Various sites, typically in `wp-config.php` on wphost02 or app3. |
@@ -220,8 +227,9 @@ The following credentials are known to exist but were not found in the standard
 ## 14. Key Rotation & Audit Notes
 
 - **Last full audit:** 2026-07-23
+- **Last sanitization:** 2026-07-23 — all plaintext secrets removed; use Hudu + file paths for values
 - **Hetzner token:** Rotated Jul 22 (old tokens in Hudu were invalid)
-- **Twilio:** Live + test credentials both present
+- **Twilio:** Live + test credentials both present in Hudu [184]/[185]/[186]
 - **OpenRouter:** Fallback routing key — keep active even if not primary
 - **admin-ai:** Primary gateway — all model calls route through this
 - **Backups:** All .env + config files included in daily Hermes backup to S3
@@ -234,3 +242,11 @@ If Core is lost, you need these to rebuild (in order):
 3. `/root/.aws/credentials` — S3 access for backups
 4. `/root/.hermes/config.yaml` — Full Hermes config
 5. `/root/.config/himalaya/shonuff.pass` — Email access
+
+### Hudu API Assets (layout 49)
+
+All API keys are documented as Hudu assets. List them via:
+```
+GET https://hudu.itpropartner.com/api/v1/companies/1/assets?page=1&per_page=25
+```
+Filter by `asset_layout_id == 49` to see all API keys with their Hudu asset IDs and storage locations.