From f5285ae7f1341fb14fb1dc34d3b51ce3d7bf1d3f Mon Sep 17 00:00:00 2001 From: root Date: Thu, 23 Jul 2026 12:05:47 -0400 Subject: [PATCH] docs: restore sanitized operational documentation after credential purge --- README.md | 277 +++++++++++++++++++++++++ docs/key-inventory.md | 252 ++++++++++++++++++++++ docs/model-chain.md | 26 +++ docs/ops-portal/CHANGELOG.md | 39 ++++ docs/ops-portal/README.md | 55 +++++ docs/project-log.md | 68 ++++++ hermes-skills/systems-networks-team.md | 66 ++++++ sites/app3.md | 49 +++++ 8 files changed, 832 insertions(+) create mode 100644 README.md create mode 100644 docs/key-inventory.md create mode 100644 docs/model-chain.md create mode 100644 docs/ops-portal/CHANGELOG.md create mode 100644 docs/ops-portal/README.md create mode 100644 docs/project-log.md create mode 100644 hermes-skills/systems-networks-team.md create mode 100644 sites/app3.md diff --git a/README.md b/README.md new file mode 100644 index 0000000..29a6cae --- /dev/null +++ b/README.md @@ -0,0 +1,277 @@ +# IT Pro Partner — Infrastructure Documentation + +> **Last Updated:** July 17, 2026 +> **Maintainer:** Sho'Nuff + +--- + +## Server Inventory + +### Core Server +- **Hostname:** Core +- **IP:** 152.53.192.33 +- **Provider:** netcup RS 2000 G12 +- **Specs:** 8 vCPU EPYC 9645, 15 GB DDR5 ECC, 512 GB NVMe +- **OS:** Debian 13 +- **Role:** Hermes + Portals +- **Key Services:** + - Hermes Agent (Telegram + cron, 22 cron jobs) + - Caddy reverse proxy (12 domains, auto-TLS) + - Ops Portal (FastAPI, port 8090) + - Prometheus (native, port 9090) + Grafana (native, port 3002) + - Uptime Kuma (Docker, port 3001) — 9+ monitors + - Vaultwarden (Docker, port 8080) — vault.iamgmb.com + - Twenty CRM (Docker) — crm.debtrecoveryexperts.com + - DocuSeal (Docker, port 3000) — sign.core.itpropartner.com + - SearXNG (Docker, port 8888) + - Komodo (Docker, port 9120) + - Tailscale, StrongSwan, WireGuard (home CCR tunnel 10.77.0.0/24) + - Redis cache + +### App1 Server +- **Hostname:** app1 +- **IP:** 152.53.36.131 +- **Provider:** netcup RS 4000 G12 +- **Specs:** 12 vCPU EPYC 9645, 32 GB DDR5 ECC, 1 TB NVMe +- **OS:** Debian 13 +- **Role:** AI/Service Hub +- **Key Services:** + - Open WebUI (Docker, port 3000) — ai.itpropartner.com + - n8n + Postgres (Docker, port 5678) — n8n.itpropartner.com + - LiteLLM (Docker) + Postgres — admin-ai.itpropartner.com + - Mattermost Team Edition (Docker, port 8065) — noc.itpropartner.com + - Caddy (systemd, 80/443) + - 4 MCP servers: Browser (:8901), Filesystem (:8900), Email (:8902), Git (:8903) + - Super Search MCP (systemd, port 8899) + +### App2 Server +- **Hostname:** app2 +- **IP:** 152.53.39.202 +- **Provider:** netcup RS 4000 G12 +- **Specs:** 12 vCPU EPYC 9645, 32 GB DDR5 ECC, 1 TB NVMe +- **OS:** Debian 13 +- **Role:** Infrastructure Server +- **Key Services:** + - Traccar GPS (Docker, port 8082 + 5000-5150) — fleettracker360.com + - UniFi Controller (Docker, port 8443) — unifi.itpropartner.com + - UNMS/UISP (10 Docker containers) — unms.forefrontwireless.com + - Hudu (Docker) — hudu.itpropartner.com + - Caddy (4 domains) + +### App3 Server +- **Hostname:** app3 +- **IP:** 152.53.241.111 +- **Provider:** netcup RS 4000 G12 +- **Specs:** 12 vCPU EPYC 9645, 32 GB DDR5 ECC, 1 TB NVMe +- **OS:** Debian 13 +- **Role:** Web Hosting + Backup Restore +- **Key Services:** + - CloudPanel CE — panel.itpropartner.com + - Nginx (80/443) + Percona MySQL 8.4 + PHP 8.3 + - Backup Restore System (Flask, port 8090) — my.itpropartner.com/backups + - WordPress sites (7 migrated from wphost02, all live): + - debtreecoveryexperts.com, boxpilotlogistics.com, iamgmb.com + - katiewattsdesign.com, vigilanttac.com, apextrackexperience.com + - mainwp.itpropartner.com, voipsimplicity.com, my.voipsimplicity.com + - Daily snapshots: 1 AM + 1 PM, 60-day retention, /opt/backup-restore/snapshots + +### Core-BU (Warm Standby) +- **Hostname:** core-bu +- **IP:** 5.161.225.131 +- **Provider:** Hetzner CPX21 +- **Specs:** 3 vCPU, 4 GB RAM, 80 GB SSD +- **Role:** Warm standby — auto-failover if Core down +- **Watchdog:** 5-min check, 4-cycle confirmation, S3 sync every 10 min + +### Legacy / Decommissioned +- **old-ai:** 178.156.167.181 (Hetzner CPX41) — **decommissioned** (LiteLLM migrated to app1) +- **old app1:** 87.99.144.163 (Hetzner CPX11) — **deleted** +- **wphost02:** 5.161.62.38 (Hetzner CPX21) — **migrated to app3** +- **Ollama:** Removed from Core (systemd) and app1 (Docker) Jul 17 + +--- + +## Model Fallback Chain + +All providers use direct API keys. GPT-5.5 quality survives through admin-ai → OpenRouter, then degrades through DeepSeek → Gemini → Grok. + +| # | Model | Provider | Gateway | +|---|---|---|---| +| Primary | GPT-5.5 | admin-ai | Self-hosted LiteLLM (app1) | +| Fallback 1 | GPT-5.5 | OpenRouter | openrouter.ai | +| Fallback 2 | DeepSeek v4 Pro | DeepSeek | api.deepseek.com | +| Fallback 3 | Gemini 3.5 Flash | Google | generativelanguage.googleapis.com | +| Fallback 4 | Grok 4.5 | xAI | api.x.ai | + +**Credits (Jul 17):** DeepSeek $58, OpenRouter ~$30 remaining, OpenAI/xAI/Google on pay-as-you-go +**Health check:** Daily 8 AM cron (`model-usage-check`) + +--- + +## Domain / DNS Map + +### ⚠️ itpropartner.com — SiteGround Nameservers Only + +`itpropartner.com` uses **SiteGround nameservers** exclusively. A Cloudflare zone exists (`0dc20632…`) but is NOT authoritative — records created there silently fail. All `*.itpropartner.com` changes must be manual through SiteGround panel. + +| Domain | IP | Server | Service | +|---|---|---|---| +| core.itpropartner.com | 152.53.192.33 | Core | Landing page + Grafana link | +| ops.itpropartner.com | 152.53.192.33 | Core | Ops dashboard | +| sign.core.itpropartner.com | 152.53.192.33 | Core | DocuSeal | +| uptimekuma.itpropartner.com | 152.53.192.33 | Core | Uptime monitoring | +| gps.fleettracker360.com | 152.53.192.33 | Core | Traccar HTTPS proxy → app2 | +| my.itpropartner.com | 152.53.192.33 | Core | Customer portal hub | +| hudu.itpropartner.com | 152.53.39.202 | app2 | IT documentation | +| unifi.itpropartner.com | 152.53.39.202 | app2 | UniFi controller | +| panel.itpropartner.com | 152.53.241.111 | app3 | CloudPanel CE | +| ai.itpropartner.com | 152.53.36.131 | app1 | Open WebUI | +| n8n.itpropartner.com | 152.53.36.131 | app1 | n8n automation | +| admin-ai.itpropartner.com | 152.53.36.131 | app1 | LiteLLM | + +### Cloudflare-Managed Domains + +| Domain | IP | Server | Service | +|---|---|---|---| +| fleettracker360.com | Cloudflare | app2 | Fleet tracking website | +| gps.fleettracker360.com | Cloudflare → Core | Core → app2 | Traccar devices | +| voipsimplicity.com | Cloudflare | app3 | VoIP marketing site | +| my.voipsimplicity.com | Cloudflare | app3 | VoIP customer portal | +| portal.debtrecoveryexperts.com | 152.53.192.33 | Core | DRE portal | +| crm.debtrecoveryexperts.com | Cloudflare Access | — | DRE CRM | +| vault.iamgmb.com | 152.53.192.33 | Core | Vaultwarden | +| sign.iamgmb.com | 152.53.192.33 | Core | Document signing | +| shark.iamgmb.com | 152.53.192.33 | Core | Shark game | + +### DNS PENDING (create at SiteGround) + +| Subdomain | → IP | Service | +|---|---|---| +| vault.itpropartner.com | 152.53.36.131 | Vaultwarden (after migration) | +| status.itpropartner.com | 152.53.192.33 | Public status page | + +--- + +## Backup Pipeline + +| Backup | Schedule | Target | Purpose | +|---|---|---|---| +| hermes-live-sync | Every 15 min | s3://hermes-vps-backups/live/ | Live state sync | +| hermes-full-backup | Daily 1 AM | s3://hermes-vps-backups/hermes-full-backup/ | Full Hermes backup | +| home-router-backup | Daily 6 AM | s3://mikrotik-ccr-backups/ | CCR config | +| root-essentials-backup | Daily 3 AM | S3 | /root essentials | +| docker-volume-sync | Daily 3 AM | S3 | Docker volumes | +| system-config-sync | Daily 4 AM | S3 | System configs | +| unms-backup-sync | Daily 6 AM (Core) | s3://hermes-vps-backups/unms-backups/ | UNMS data (pulled from app2) | +| unifi-backup-sync | Daily 2 AM (Core) | s3://hermes-vps-backups/unifi-backups/ | UniFi configs (pulled from app2) | +| hudu-backup | Daily 7 AM | s3://hermes-vps-backups/hudu/backups/ | Hudu volume dump | +| gitea-backup | Daily 8 AM | s3://hermes-vps-backups/gitea/daily/ | Gitea repos | +| app1-backup | Daily 2 AM | s3://hermes-vps-backups/app1/ | LiteLLM, n8n, OpenWebUI, MCP, Mattermost | +| app2-backup | Daily 2:30 AM | s3://hermes-vps-backups/app2/ | Traccar, Gitea, Hudu, UNMS, UniFi | +| app3-backup | Daily 3 AM | s3://hermes-vps-backups/app3/ | CloudPanel, MySQL, WordPress | +| wphost02-backup | Daily 5 AM | s3://hermes-vps-backups/wphost02-backup/ | Webapps + MySQL | +| warm-standby-sync | Every 10 min | core-bu ← S3 | DR readiness | + +--- + +## Caddy Configuration + +### Core (152.53.192.33) + +``` +core.itpropartner.com → static files +sign.core.itpropartner.com → localhost:3000 (DocuSeal) +ops.itpropartner.com → 127.0.0.1:8090 + static +uptimekuma.itpropartner.com → 127.0.0.1:3001 (Uptime Kuma) +gps.fleettracker360.com → app2:8082 (Traccar) +my.itpropartner.com → static files +portal.debtrecoveryexperts.com → static files +vault.iamgmb.com → localhost:8080 (Vaultwarden) +sign.iamgmb.com → 127.0.0.1:8090 +shark.iamgmb.com → static + :8083 +``` + +### App1 (152.53.36.131) + +``` +ai.itpropartner.com → :3000 (Open WebUI) +n8n.itpropartner.com → :5678 (n8n) +admin-ai.itpropartner.com → :4000 (LiteLLM) +app1.itpropartner.com → static response +``` + +### App2 (152.53.39.202) + +``` +hudu.itpropartner.com → Hudu internal +gps.fleettracker360.com → :8082 (Traccar) +unms.forefrontwireless.com → UNMS Nginx +unifi.itpropartner.com → :8443 (UniFi) +``` + +--- + +## Service Access + +| Service | URL | Location | Auth | +|---|---|---|---| +| Open WebUI | https://ai.itpropartner.com | app1 | Chat UI | +| Open WebUI Admin | https://admin-ai.itpropartner.com/ui | app1 | user: admin, pw: LITELLM_MASTER_KEY | +| Ops Portal | https://ops.itpropartner.com | Core | Internal dashboard | +| Grafana | http://core.itpropartner.com:3002 | Core | admin/admin | +| Uptime Kuma | https://uptimekuma.itpropartner.com | Core | Service monitoring | +| Vaultwarden | https://vault.iamgmb.com | Core | Password vault | +| CloudPanel | https://panel.itpropartner.com | app3 | user: gmb / SQLite auth | +| Traccar | https://gps.fleettracker360.com | app2 | GPS fleet tracking | +| UniFi | https://unifi.itpropartner.com | app2 | Network controller | +| UNMS | https://unms.forefrontwireless.com | app2 | WISP management | +| Hudu | https://hudu.itpropartner.com | app2 | IT documentation | +| n8n | https://n8n.itpropartner.com | app1 | Automation | +| CRM (DRE) | https://crm.debtrecoveryexperts.com | Cloudflare Access | TwentyCRM | + +### MCP Access (from Open WebUI) + +| MCP Server | Location | Port | Tools | +|---|---|---|---| +| Super Search | app1 | :8899 | 10 tools — web_search, web_extract, person_search, email_search, phone_search, etc. | +| Browser | app1 | :8901 | browser_navigate, browser_snapshot, browser_click, browser_type, browser_console | +| Filesystem | app1 | :8900 | read_file, write_file, search_files, list_dir, file_info | +| Email | app1 | :8902 | search_emails, send_email, get_email | +| Git/Gitea | app1 | :8903 | clone, commit, push, pull | + +--- + +## SSH Access + +- **Key:** `itpp-infra` (deployed to all servers) +- **User:** `ippadmin` (sudo privileges) +- **Root SSH:** Enabled on app1, app2, app3 (key-only exception per provisioning standard) +- **Core SSH:** `ssh -i /root/.ssh/itpp-infra root@152.53.192.33` + +--- + +## Firewall + +UFW is enabled on all servers. Standard rules: +- **Core:** 22, 80, 443, 3000, 3001, 3002, 8080, 8082, 8090, 8443, 9090 +- **app1:** 22, 80, 443, 3000, 5678, 8899, 8900, 8901, 8902, 8903 +- **app2:** 22, 80, 443, 3000, 8080, 8082, 8089, 8443, 8843, 3478, 10001, 5000:5150 +- **app3:** 22, 80, 443, 8443 + +--- + +## SSL + +All SSL certificates issued via Let's Encrypt through Caddy. All certs auto-renew. No manual management needed. + +--- + +## Notes + +- **DNS trap:** `itpropartner.com` uses SiteGround nameservers. Cloudflare zone is NOT authoritative. Always verify with `dig NS domain.com` before creating records. +- **SiteGround:** No API access. All DNS changes are manual through SiteGround panel. +- **Provider diversity:** core-bu stays at Hetzner specifically so a netcup outage can't kill both Core and standby simultaneously. +- **app3 MySQL:** Root password in Vaultwarden + `/root/.my.cnf` on app3, accessible via 127.0.0.1:3306. +- **CloudPanel:** SQLite DB at `/home/clp/htdocs/app/data/db.sq3` — users live here, not in MySQL. +- **AWS CLI PATH:** All backup scripts must use `/opt/awscli-venv/bin/aws` or `source /opt/awscli-venv/bin/activate` — `aws` bare fails in cron context (PATH doesn't include venv bin). Documented in server-provisioning-standard v1.3.0. +- **Backup verification:** Always run at least one manual backup after provisioning a server and verify it landed in S3 — never trust cron entries alone. Silent failures (`aws: command not found`, wrong file paths, S3 permission issues) won't surface otherwise. diff --git a/docs/key-inventory.md b/docs/key-inventory.md new file mode 100644 index 0000000..bb34bd8 --- /dev/null +++ b/docs/key-inventory.md @@ -0,0 +1,252 @@ +# IT Pro Partner — Complete Key Inventory + +**Generated:** 2026-07-23 +**Sanitized:** 2026-07-23 (plaintext secrets replaced with storage references) +**Scope:** All SSH keys, API tokens, service credentials, device keys, and passwords across the infrastructure +**⚠️ SENSITIVE:** All credential values live in the listed storage locations. See Hudu for API keys (layout 49). + +--- + +## 1. SSH Keys + +| Key Name | File | Type | Fingerprint (SHA256) | Purpose | Deployed To | +|----------|------|------|-----------------------|---------|-------------| +| **itpp-infra** | `/root/.ssh/itpp-infra` | ED25519 | `Jxh0bbT9dUV3q1DYYB3hHyhy/1TDj7Q8U4xrVmB38uQ` | Universal server admin key | All servers (Core, app1, app2, app3, wphost02, app1-bu, home router) | +| **wisp_rsa** | `/root/.ssh/wisp_rsa` | ED25519 | `MxQw1oh90NibSgN2mDbKP+07/jE4FEUEBbFAzuk5DcI` | WISP MikroTik CCR router SSH | Home CCR router (10.77.0.2 via WireGuard) | +| **germaine-personal** | `/root/.ssh/germaine-personal` | ED25519 | `dDbLH+bdPFcGU0mm1DpGa43ec0nUZ88YnpCi4p63y3I` | Germaine's personal key (from his machines) | Germaine's devices → Core | +| **homelab** | `/root/.ssh/homelab` | ED25519 | `c1nts4wR9EU06/O/k895Pb2tGZublgnGWG6NoQrK/qs` | Homelab Proxmox/QNAP access | vm-host-01, vm-host-02, QNAP NAS | +| **siteground.key** | `/root/.ssh/siteground.key` | RSA (encrypted) | N/A (RSA, encrypted) | SiteGround SFTP backup (port 18765) | SiteGround shared hosting | +| **authorized_keys** | `/root/.ssh/authorized_keys` | — | — | Who can SSH into Core | Core (this server) | + +### SSH Key Details + +``` +itpp-infra.pub: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAII4dxTH11aJkBqCY8lXl1kTfZ8yXWhTcthHnt1MtAuIE itpp-infra +wisp_rsa.pub: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDnI4UwwPL8gJvtP/Jr7qiw0Qj/bQBwi2+f03p730xvn wisp-backup +germaine-personal.pub: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAID2H/2VMn8i7YSUUpcag6yXiI6nB3T99h7JIOs5/+73r germaine@itppartner +homelab.pub: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHT+727Cti4cZ2x6CiYDeDKZ9BhvCJCzTHlO9vMInHie homelab-itpp +``` + +--- + +## 2. Server Root Passwords + +**Storage:** Hudu (Vaultwarden asset) + `/root/.hermes/.env` (netcup CCP section) + +| Server | IP | Provider | Access | Notes | +|--------|-----|----------|--------|-------| +| **Core** | 152.53.192.33 | netcup RS 2000 | SSH key only | `itpp-infra` key, password auth disabled | +| **app1** | 152.53.36.131 | netcup RS 4000 | Via root or ippadmin+sudo | Root password in Vaultwarden | +| **app2** | 152.53.39.202 | netcup RS 4000 | Via root or ippadmin+sudo | Root password in Vaultwarden | +| **app3** | 152.53.241.111 | netcup RS 4000 | Via root or ippadmin+sudo | Root password in Vaultwarden | +| **app1-bu** | 5.161.114.8 | Hetzner CPX11 | itpp-infra SSH key | Warm standby, offline by default | + +### Admin Account (all servers) + +- **Username:** `ippadmin` +- **Password:** → Vaultwarden entry "ippadmin" +- **Sudo:** Yes (full sudo access) +- **SSH:** Key-based only (`itpp-infra`) + +--- + +## 3. Cloud & Infrastructure API Keys + +**Storage:** All in `~/.hermes/.env` + Hudu API assets (layout 49) + +| Service | Hudu Asset | Storage Location | Status | +|---------|-----------|------------------|--------| +| **Hetzner Cloud** | [177] | `/root/.hermes/scripts/.hetzner_token` + `/root/.hermes/.env` | ✅ Verified Jul 22 | +| **Cloudflare DNS** | [165] | `~/.hermes/.env` → `CLOUDFLARE_API_TOKEN` | ✅ Active (verified by health check) | +| **Wasabi S3** | [176] | `/root/.aws/credentials` | ✅ Active | +| **netcup API** | [166] | `~/.hermes/.env` → `NETCUP_API_KEY` | ✅ Active | +| **netcup CCP** | [167] | `~/.hermes/.env` → `NETCUP_CUSTOMER_NUMBER` + `NETCUP_CCP_PASSWORD` | ✅ Active | +| **Gitea (OLD/DEAD)** | — | ⚠️ **EXPIRED** — still in homelab + itpp-infrastructure remotes | ❌ INVALID (verified Jul 23) | +| **Gitea (ACTIVE)** | — | All other repos + `gitea-backup.sh` — ippadmin | ✅ Active (verified Jul 23) | + +--- + +## 4. AI Provider API Keys + +All stored in `/root/.hermes/.env` and Hudu API assets (layout 49). + +| Provider | Hudu Asset | Purpose | Status | +|----------|-----------|---------|--------| +| **admin-ai** (LiteLLM) | [126] Hermes Primary Key | Primary model gateway (all models) | ✅ Active | +| **Anthropic** | [150] | Claude models | ✅ Active | +| **OpenAI** | [149] | GPT models | ✅ Active | +| **DeepSeek** | [152] | DeepSeek models | ✅ Active | +| **Google Gemini** | [161] / [151] | Gemini models | ✅ Active | +| **xAI (Grok)** | [154] | Grok models | ✅ Active | +| **OpenRouter** | [153] | Multi-provider routing | ✅ Active | +| **Mistral** | [155] | Mistral models | ✅ Active | +| **Groq** | [157] | Fast inference | ✅ Active | +| **Fireworks AI** | [156] | Serverless inference | ✅ Active | +| **Perplexity** | [159] | Search-augmented LLM | ✅ Active | +| **Cohere** | [158] | Cohere models | ✅ Active | +| **AI21 Labs** | [160] | Jurassic models | ✅ Active | +| **MiniMax** | [187] | MiniMax M3 | ✅ Active | +| **Z.ai (GLM)** | [188] | GLM models | ✅ Active | +| **Alibaba Qwen** | [189] Alibaba Qwen (DashScope) | Qwen models | ✅ Active | +| **Deepgram** | [162] | STT (voice transcription) | ✅ Active | + +--- + +## 5. Communication APIs + +**Storage:** All in `~/.hermes/.env` + Hudu API assets (layout 49) + +| Service | Hudu Asset | Storage | Status | +|---------|-----------|---------|--------| +| **Telegram Bot** | [170] | `~/.hermes/.env` → `TELEGRAM_BOT_TOKEN` | ✅ Active | +| **Twilio (Live)** | [184] Twilio Live | `~/.hermes/.env` → `TWILIO_ACCOUNT_SID` + `TWILIO_AUTH_TOKEN` | ✅ Active | +| **Twilio (Test)** | [185] Twilio Test | `~/.hermes/.env` → `TWILIO_TEST_ACCOUNT_SID` + `TWILIO_TEST_AUTH_TOKEN` | ✅ Active | +| **Twilio API Key** | [186] Twilio API Key | `~/.hermes/.env` → `TWILIO_API_KEY_SID` + `TWILIO_API_KEY_SECRET` | ✅ Active | +| **ElevenLabs** | [148] | `~/.hermes/config.yaml` (auxiliary vision / TTS) | ✅ Active | +| **Email SMTP/IMAP** | — | `/root/.config/himalaya/shonuff.pass` | ✅ Active | +| **Email account** | — | `shonuff@germainebrown.com` — MXroute via mail.germainebrown.com:2525 (SMTP) / :993 (IMAP) | ✅ Active | + +--- + +## 6. VoIP / RingLogix + +**Storage:** `~/.hermes/.env` + Hudu API assets + +| Credential | Hudu Asset | Storage | +|-----------|-----------|---------| +| **RingLogix Client ID** | [174] | `~/.hermes/.env` → `RINGLOGIX_CLIENT_ID` | +| **RingLogix Client Secret** | [175] | `~/.hermes/.env` → `RINGLOGIX_CLIENT_SECRET` | +| **RingLogix Username** | — | `~/.hermes/.env` → `RINGLOGIX_USERNAME` | +| **RingLogix Password** | — | `~/.hermes/.env` → `RINGLOGIX_PASSWORD` | +| **RingLogix Domain** | — | `~/.hermes/.env` → `RINGLOGIX_DOMAIN` | + +--- + +## 7. MSP / RMM / Security APIs + +**Storage:** All in `~/.hermes/.env` + Hudu API assets + +| Service | Hudu Asset | Storage | Status | +|---------|-----------|---------|--------| +| **SyncroMSP** | [168] Token + [169] API Key | `~/.hermes/.env` → `SYNCROMSP_API_TOKEN` + `SYNCROMSP_API_KEY` | ✅ Active | +| **Bitdefender GZ** | [172] | `~/.hermes/.env` → `BITDEFENDER_API_KEY` | ✅ Active | +| **VirusTotal** | [171] | `~/.hermes/.env` → `VIRUSTOTAL_API_KEY` | ✅ Active | +| **UISP/UNMS** | [173] | `~/.hermes/.env` → `UISP_API_KEY` | ✅ Active | + +--- + +## 8. Search & Data APIs + +| Service | Hudu Asset | Storage | Status | +|---------|-----------|---------|--------| +| **Firecrawl** | [164] | `~/.hermes/.env` → `FIRECRAWL_API_KEY` | ✅ Active | +| **Exa AI Search** | [163] | `~/.hermes/.env` → `EXA_API_KEY` | ✅ Active | + +--- + +## 9. Database Credentials + +| Database | Host | User | Password Location | Purpose | +|----------|------|------|-------------------|---------| +| **MySQL (apex track)** | 127.0.0.1:33060 (SSH tunnel from wphost02) | `apextrackexperience_1781549652` | `wp-config.php` on wphost02 | Apex Track Experience WordPress | +| **MySQL (CloudPanel)** | app3:3306 | `root` | `/root/.my.cnf` on app3 (also in Vaultwarden) | CloudPanel WordPress hosting | +| **LiteLLM Postgres** | app1 (Docker) | (in docker-compose) | `/root/docker/litellm/docker-compose.yml` on app1 | LiteLLM operational DB | + +--- + +## 10. Docker Services + +| Service | URL | Credential Location | Storage | +|---------|-----|--------------------|---------| +| **Vaultwarden** | vault.itpropartner.com / vault.iamgmb.com | Admin Token → `/root/docker/vaultwarden/.env` on Core | Docker env file | +| **DRE Portal** | portal.debtrecoveryexperts.com | Basic Auth (htpasswd) | `/etc/caddy/dre-passwd` | +| **SearXNG** | (internal, no public endpoint) | (none) | — | +| **DocuSeal** | sign.core.itpropartner.com / sign.iamgmb.com | (none / app-managed) | — | +| **Uptime Kuma** | uptimekuma.itpropartner.com | (app-managed) | — | +| **Open WebUI** | admin-ai.itpropartner.com | `admin@itpropartner.com` (password: ask Sho'Nuff) | Not in .env | +| **Mealie** | recipe.iamgmb.com | `G@germainebrown.com` (password → Vaultwarden) | Vaultwarden | +| **Ops Portal** | ops.itpropartner.com | `ippadmin` (password → `~/.hermes/.env`) | `~/.hermes/.env` | + +--- + +## 11. VPN & Network Keys + +### WireGuard (Core) + +| Item | Value | +|------|-------| +| **Interface** | `wg0` | +| **Core Private Key** | → `/etc/wireguard/wg0.conf` on Core | +| **Core IP** | `10.77.0.1/24` | +| **Listen Port** | `51821` | +| **Home Peer Public Key** | `1fPwdGQ20CxlZCQZQV134olDcE91hfp78yNDeaKJZzg=` | +| **Home Peer Endpoint** | `76.195.7.60:13231` | +| **Routed Networks** | `10.1.0.0/16`, `10.2.0.0/16`, `172.16.1.0/24`, `172.18.18.0/24` | + +### Tailscale + +| Node | IP | Type | Status | +|------|-----|------|--------| +| core | 100.71.155.7 | Linux | ✅ Online | +| app1 | 100.90.186.109 | Linux | ✅ Online | +| app2 | 100.117.164.66 | Linux | ✅ Online | +| app3 | 100.72.15.12 | Linux | ✅ Online | +| app1-bu | 100.112.23.21 | Linux | ⚠️ Offline (7d) | +| iphone-15-pro-max | 100.106.231.86 | iOS | ✅ Online | +| ipp-g-lap | 100.120.64.120 | macOS | ✅ Online | +| m4-mac-mini | 100.116.232.65 | macOS | ✅ Online | + +--- + +## 12. UniFi / UDM Pro Device Keys + +| Site | Key Location | Type | Status | +|------|-------------|------|--------| +| **Grand Lake Club** | UniFi Network Controller → Settings → API | Local Network API Key | ✅ Stored, pending direct verification | +| **Liberty Tire** | UniFi Network Controller → Settings → API | Local Network API Key | ✅ Stored, pending direct verification | + +--- + +## 13. Unknown / Not Found + +The following credentials are known to exist but were not found in the standard locations: + +| Item | Notes | +|------|-------| +| **Open WebUI admin password** | Recovery manual says "in .env or ask Sho'Nuff" — NOT in current .env. Must ask Germaine. | +| **Hudu API key** | In skill docs (`hudu-management`) — used programmatically, not in .env. | +| **Traccar/FleetTracker360 admin** | Not in .env. May be Docker env or app-managed. | +| **Twenty CRM credentials** | Docker on Core, env at `/root/docker/twenty/.env` (not read). | +| **WordPress site DB passwords** | Various sites, typically in `wp-config.php` on wphost02 or app3. | +| **app1-bu root password** | Hetzner CPX11 — accessed via itpp-infra SSH key only. | +| **ComfyUI / Z4** | GPU server allocated for TripFlow — credentials not yet documented. | +| **Home MikroTik admin** | SSH via `admin@10.77.0.2` with `wisp_rsa` key. RouterOS password in router config (not extracted). | + +--- + +## 14. Key Rotation & Audit Notes + +- **Last full audit:** 2026-07-23 +- **Last sanitization:** 2026-07-23 — all plaintext secrets removed; use Hudu + file paths for values +- **Hetzner token:** Rotated Jul 22 (old tokens in Hudu were invalid) +- **Twilio:** Live + test credentials both present in Hudu [184]/[185]/[186] +- **OpenRouter:** Fallback routing key — keep active even if not primary +- **admin-ai:** Primary gateway — all model calls route through this +- **Backups:** All .env + config files included in daily Hermes backup to S3 + +### Recovery Priority + +If Core is lost, you need these to rebuild (in order): +1. `/root/.ssh/itpp-infra` — SSH to all servers +2. `/root/.hermes/.env` — All API keys and secrets +3. `/root/.aws/credentials` — S3 access for backups +4. `/root/.hermes/config.yaml` — Full Hermes config +5. `/root/.config/himalaya/shonuff.pass` — Email access + +### Hudu API Assets (layout 49) + +All API keys are documented as Hudu assets. List them via: +``` +GET https://hudu.itpropartner.com/api/v1/companies/1/assets?page=1&per_page=25 +``` +Filter by `asset_layout_id == 49` to see all API keys with their Hudu asset IDs and storage locations. diff --git a/docs/model-chain.md b/docs/model-chain.md new file mode 100644 index 0000000..bfb0252 --- /dev/null +++ b/docs/model-chain.md @@ -0,0 +1,26 @@ +# AI Model Chain — IT Pro Partner + +**Updated:** July 21, 2026 +**Sanitized:** July 23, 2026 (plaintext keys removed) +**Gateways:** admin-ai (self-hosted LiteLLM), OpenRouter, DeepSeek, Google, xAI + +## Chain + +| # | Model | Provider | Key Storage | +|---|---|---|---| +| Primary | GPT-5.5 | admin-ai | Hudu [126] Hermes Primary Key | +| F1 | GPT-5.5 | OpenRouter | Hudu [153] | +| F2 | DeepSeek v4 Pro | DeepSeek | Hudu [152] | +| F3 | Gemini 3.5 Flash | Google | Hudu [161] / [151] | +| F4 | Grok 4.5 | xAI | Hudu [154] | + +## Admin-AI (LiteLLM) +- URL: `admin-ai.itpropartner.com` (app1, 152.53.36.131) +- Master key: `/root/docker/litellm/.env` on app1 (Hudu [178] LiteLLM Master Key) +- Hermes virtual key: Hudu [126] Hermes Primary Key (GPT-5.5 + DeepSeek v4 Pro routing) + +## Credential Storage +- Config: `/root/.hermes/config.yaml` +- Hudu: All API assets under layout 49 at https://hudu.itpropartner.com — search "AI Provider" or by model name +- Git: `itpp-infrastructure/docs/model-chain.md` (this file, sanitized) +- Full inventory: `itpp-infrastructure/docs/key-inventory.md` diff --git a/docs/ops-portal/CHANGELOG.md b/docs/ops-portal/CHANGELOG.md new file mode 100644 index 0000000..8ab3412 --- /dev/null +++ b/docs/ops-portal/CHANGELOG.md @@ -0,0 +1,39 @@ +# Ops Portal Changelog + +## 2026-07-20 — Major audit and fix session + +### Fixed +- **`/api/health` returning 404:** Caddy `handle_path` was stripping the path → changed to `handle`, port corrected to 8090 +- **`/api/servers` returning null:** Server list now returns all 5 servers with live ping health +- **Server IPs stale:** app1-bu updated to 5.161.225.131, legacy entries removed +- **Page titles inconsistent:** All 11 pages standardized to "X — IT Pro Partner Ops" format +- **Missing nav icons:** All 11 nav items now have SVG icons +- **FleetTracker360 missing from nav:** Added to navigation with car icon +- **Backups page no data:** `s3_buckets` → `s3_backups` key fix +- **FleetTracker360 page no nav:** Added ops.css, app.js, utils.js includes +- **Network page dark sidebar:** Replaced with standard top nav bar +- **Cache-busting broken:** All JS/CSS references now versioned with timestamps +- **Mobile nav broken:** `.nav-links.open` CSS rule missing → hamburger menu now toggles properly on iOS/Android +- **Auth guard race condition:** IIFE scripts replaced with DOMContentLoaded event listeners — pages now load data when user is authenticated +- **Cost page broken:** Missing `loadData` function → defined and wired +- **Dependency diagram 404:** File copied to static dir, link corrected +- **Logs page mangled title:** Triple-nested `` tags from sed accident → cleaned +- **Config page scripts directory:** Now populates directory listing when clicked +- **Services page:** Server column added showing "Core (152.53.192.33)" +- **Dashboard auto-refresh on tab focus:** Visibility API handler added +- **Critical service protection:** hermes, caddy, ops-portal restarts blocked via API + +### Removed +- Duplicate server entries: "app1 (AI Stack)" and "Docker Box (legacy)" +- Server count: 7 → 5 clean entries + +### Changed +- Admin credentials: germaine/itpp2026! → ippadmin (password → Vaultwarden) +- JWT_SECRET made persistent in /root/.hermes/.env to survive restarts +- Collector timeout: 20s → 90s to handle 94K-file S3 bucket scanning + +## Jul 17, 2026 — Initial deployment +- Ops portal deployed on Core as FastAPI app +- Caddy reverse proxy configured +- 10 HTML pages created +- Ops collector built for S3, system health, server status diff --git a/docs/ops-portal/README.md b/docs/ops-portal/README.md new file mode 100644 index 0000000..f4cc046 --- /dev/null +++ b/docs/ops-portal/README.md @@ -0,0 +1,55 @@ +# Ops Portal — ops.itpropartner.com + +## Architecture +- **Server:** Core (152.53.192.33, netcup RS 2000) +- **Backend:** FastAPI at `/opt/ops-portal/server.py` (port 8090) +- **Proxy:** Caddy → reverse_proxy to 127.0.0.1:8090 +- **Static files:** `/opt/ops-portal/static/` — 11 HTML pages, ops.css, app.js, utils.js +- **Auth:** JWT via `POST /api/auth/login`, token in localStorage +- **Data:** `/var/www/ops/data/ops-status.json` (5-min collector refresh) +- **Collector:** `/root/.hermes/scripts/ops-data-collector.py` — Wazuh, Bitdefender, S3, UISP, system health +- **Systemd:** `ops-portal.service`, env from `/root/.hermes/.env` +- **Credentials:** ippadmin (password → Vaultwarden / `~/.hermes/.env`) + +## Pages (11 total) +| Page | Path | Description | +|------|------|-------------| +| Dashboard | / | System health, widgets, audit log | +| Services | /services.html | Systemd service control, audit log, server column | +| Servers | /servers.html | 5 servers with ping health | +| Network | /network.html | UISP data (44 sites, 90 devices), DNS zones | +| Backups | /backups.html | S3 bucket status (6 buckets) | +| FleetTracker | /fleettracker360.html | Traccar device tracking | +| Cron Jobs | /cron.html | Hermes cron jobs with expandable scripts | +| Config | /config.html | Active configs, /root/.hermes/scripts/ listing | +| Logs | /logs.html | Aggregated log viewer | +| Audit | /audit.html | Full audit trail | +| Costs | /cost.html | API cost tracking by model | + +## API Endpoints +| Method | Path | Auth | Description | +|--------|------|------|-------------| +| POST | /api/auth/login | No | JWT authentication | +| GET | /api/health | No | Health check, DB status | +| GET | /api/status | JWT | Full dashboard data (17 sections) | +| GET | /api/servers | JWT | Server list with IPs | +| GET | /api/servers/health | JWT | Ping health (all 7 LIVE) | +| GET | /api/audit-log?limit=N | JWT | Audit trail entries | +| GET | /api/ft360/status | JWT | FleetTracker device data | + +## Critical Services (API restart blocked) +hermes, hermes-assistant, hermes-browser, caddy, ops-portal, mysql-tunnel + +## Dashboard Widgets +- System Health — Core metrics (jobs, disk, memory, S3, APIs) +- Wazuh Security — agent count, alerts +- Bitdefender GravityZone — 9 managed endpoints +- Alerts and Notifications — DR issues, backup failures, cron errors +- Quick Actions — Restart Ops Portal + +## Recovery +``` +systemctl restart ops-portal +systemctl reload caddy +python3 /root/.hermes/scripts/ops-data-collector.py +``` diff --git a/docs/project-log.md b/docs/project-log.md new file mode 100644 index 0000000..d5b1980 --- /dev/null +++ b/docs/project-log.md @@ -0,0 +1,68 @@ +# Project Log — All Completed Projects + +## 2026-07-20 + +### Ops Portal Audit and Overhaul +- Full audit of all 11 pages, 7 API endpoints, and 5 dashboard widgets +- Fixed 15 bugs: auth guards, cache-busting, mobile nav, page titles, missing icons, data keys +- Added 3 new widgets: Wazuh Security, Bitdefender GravityZone, Alerts and Notifications +- Standardized credentials: ippadmin (password → Vaultwarden / `~/.hermes/.env`) +- Added critical service protection (hermes/caddy/ops-portal restart blocked via API) +- Server list cleaned up (7→5), dependency diagram fixed, config page scripts listing + +### Backup-Restore Enhancements +- Added manual backup with domain dropdown and note field +- Added restore history logging with formatted 4-column table +- Fixed Caddy routing and timeouts (restore was returning 404 via proxy) +- Fixed mobile toggle on domain expansion cards +- 9 WordPress sites under daily backup (1 AM and 1 PM) + +### Docs Written +- `/root/projects/ops-portal/README.md` + `CHANGELOG.md` +- `/root/projects/backup-restore/README.md` + `CHANGELOG.md` + +--- + +## 2026-07-17 — Backup-Restore Initial Deployment +- Flask backup/restore app deployed on app3 (152.53.241.111) +- Daily snapshots scheduled at 1 AM and 1 PM +- Caddy reverse proxy from my.itpropartner.com +- 9 WordPress sites configured + +## 2026-07-21 — Home Lab Consolidation + +### Proxmox Migration +- vm-host-02 VMs migrated/destroyed: graylog, zabbix, fog, Ubuntu-Server +- vm-host-01 now hosts: docker-host-01, adguard-home +- vm-host-02 cleared for GPU installation (RTX 3090 pending verification) +- QNAP NFS shared storage created (2TB pool, mounted on both Proxmox hosts) + +### DNS Infrastructure +- Technitium DNS deployed on app2 (dns1.itpropartner.com) +- DoH upstreams: Quad9, Cloudflare, Google +- Home DNS chain: docker-host-01 AdGuard → dns1 Technitium → vm-host-01 AdGuard +- Firewall locked: port 53 restricted to 76.195.7.60 + +### Twilio +- Toll-free number verification submitted for IT Pro Partner +- Use case: customer notifications, appointment reminders, IVR + +### Mattermost +- Branding configured: IT Pro Partner NOC +- Channel structure designed (13 channels) +- Mobile push investigation: HPNS required for background notifications + +### Gift-a-Roast +- Domain giftaroast.com purchased, DNS live (Cloudflare → app1) +- ElevenLabs TTS + Deepgram STT keys verified +- Architecture: Twilio Voice → STT → AI → TTS → caller + +### Uptime Kuma +- Backed up (361MB, 25 monitors), updated to latest + +### IRS +- Name change letter drafted: CG Premier Transport LLC → IT Pro Partner LLC +- Georgia Secretary of State filing confirmed + +### Skills Updated +- 10 skills patched: docker-service-deployment, home-lab-*, server-architecture-plan, twilio-10dlc, vaultwarden-management, voip-portal, hudu, syncromsp, recurring-information-scout diff --git a/hermes-skills/systems-networks-team.md b/hermes-skills/systems-networks-team.md new file mode 100644 index 0000000..219fe56 --- /dev/null +++ b/hermes-skills/systems-networks-team.md @@ -0,0 +1,66 @@ +--- +name: systems-networks-team +description: Expert team for systems administration, network infrastructure, server deployment, backup/DR, and monitoring. Dispatch for any task involving servers, Docker, networking gear, cron jobs, security, or infrastructure automation. +category: delegation +--- + +# Systems & Networks Team + +You are a senior infrastructure engineer. You build and maintain systems that never fail, networks that never drop, and automation that eliminates toil. + +## Team Members (Skills) + +Always load these before beginning a sys/net task: + +- **docker-service-deployment** -- Deploy and document Docker services +- **server-provisioning-standard** -- ITPP base server deployment standards +- **hermes-backup** -- Hermes backup and restore procedures +- **disaster-recovery-audit** -- DR audit framework +- **mikrotik-onboarding** -- MikroTik router onboarding +- **reboot-with-health-check** -- Server reboots with verification +- **script-audit** -- Systematic script review +- **cloudflare-dns-and-domains** -- Cloudflare DNS management +- **cloudpanel-deployment** -- CloudPanel server deployment +- **status-page-deployment** -- Status page using Uptime Kuma + Caddy +- **vaultwarden-management** -- Vaultwarden deploy, SMTP, migration +- **tailscale-infrastructure-access** -- Tailscale private networking +- **model-failover-and-credit-tracking** -- Model failover chain management + +## Standing Orders + +1. **Verify before reporting.** Never claim a service is up or a config is applied until you've actually tested it. SSH in, run the command, read the output. + +2. **Idempotency by design.** Every script, every config, every deployment must survive being run twice. Check state before changing it. + +3. **Backup before mutation.** Before touching a production config, ensure a backup exists and is verified. + +4. **Document as you build.** Every change goes into the infrastructure Git repo. No undocumented changes to production systems. + +5. **Safe to fail.** Every change must have a rollback path. If you can't undo it, don't do it without explicit approval. + +## Server Inventory + +| Server | IP | Role | +|---|---|---| +| Core | 152.53.192.33 | Hermes, portals, monitoring | +| app1 | 152.53.36.131 | AI/services hub | +| app2 | 152.53.39.202 | Infrastructure server | +| app3 | 152.53.241.111 | Web hosting + backup | +| core-bu | 5.161.225.131 | Warm standby | +| wphost02 | 5.161.62.38 | Legacy RunCloud host (still live) | + +## Key Credentials + +- SSH key: `/root/.ssh/itpp-infra` (all servers, root access) +- CloudPanel MySQL root: password in Vaultwarden + `/root/.my.cnf` on app3, accessible on 127.0.0.1:3306 +- CloudPanel admin: panel.itpropartner.com (gmb — password → Vaultwarden) +- Wasabi S3: s3.us-east-1.wasabisys.com, bucket: hermes-vps-backups +- Cloudflare API: token in /root/.hermes/.env (CLOUDFLARE_API_TOKEN) + +## Quality Gates + +- [ ] Change tested in staging/dry-run before production +- [ ] Backup verified before mutation +- [ ] Health check passes after deployment +- [ ] Documentation committed to Git +- [ ] Rollback procedure documented diff --git a/sites/app3.md b/sites/app3.md new file mode 100644 index 0000000..590c5ea --- /dev/null +++ b/sites/app3.md @@ -0,0 +1,49 @@ +# App3 — Infrastructure Reference + +**Server:** netcup RS 4000 G12 +**IP:** 152.53.241.111 +**Provisioned:** July 16, 2026 +**Role:** Customer web hosting (WordPress sites) + +## Software Stack + +| Component | Version | +|---|---| +| Debian | 13 (Trixie) | +| Nginx | 2.x | +| PHP | 8.3 (FPM) | +| MySQL | Percona 8.4 | +| CloudPanel CE | Latest | +| Tailscale | Enabled | + +## Hosted Sites + +| Domain | Type | Status | +|---|---|---| +| voipsimplicity.com | WordPress | ✅ Live | +| support.voipsimplicity.com | DNS only | ✅ Created | + +## Issues Encountered & Fixed (Jul 16) + +1. **"Hello World" default page** — WordPress files were copied to `/home/voipsimplicity/htdocs/www/` but CloudPanel's Nginx root was `/home/voipsimplicity/htdocs/voipsimplicity.com/`. Fixed by rsyncing files to correct directory. + +2. **panel.itpropartner.com 404** — Manual Nginx config (`panel.conf`) was overriding CloudPanel's generated config. Removed manual config. CloudPanel's reverse proxy template had `try_files $uri =404` instead of `try_files $uri @reverse_proxy` — fixed. + +3. **CloudPanel MFA blocking login** — MFA was enabled on the `admin` user by default. Disabled via `clpctl user:disable:mfa`. + +4. **Let's Encrypt failure (unproxied DNS)** — DNS for `panel.itpropartner.com` was on SiteGround nameservers, not Cloudflare. Added manually via SiteGround panel. CloudPanel SSL via HTTP-01 challenge required `.well-known/acme-challenge` directory at reverse proxy site root. + +5. **"Error establishing a database connection"** — MySQL auth plugin mismatch. Site DB user was using `caching_sha2_password` but WordPress expects `mysql_native_password`. Fixed by converting auth plugin. + +6. **itpropartner.com not on Cloudflare** — Nameservers still at SiteGround. All DNS changes for `*.itpropartner.com` must be done at SiteGround manually. + +## Access + +- **CloudPanel:** https://panel.itpropartner.com — `info@itpropartner.com` (password → Vaultwarden) +- **SSH:** root@152.53.241.111 (itpp-infra key) +- **MySQL root:** `root` (password → Vaultwarden + `/root/.my.cnf` on app3) on `127.0.0.1:3306` + +## Pending + +- Migrate remaining WordPress sites from wphost02 to app3 +- Update all CloudPanel passwords (sanitized from plaintext in this doc 2026-07-23 — values in Vaultwarden)