Initial skills documentation — 25 categories, all SKILL.md + references + scripts

2026-07-15 17:42:20 -04:00
commit 1b4fcd4427
976 changed files with 188344 additions and 0 deletions
@@ -0,0 +1,60 @@
+# Debt Recovery Platform — Full Architecture
+
+## Overview
+
+Debt Recovery Experts (DRE) operates at debtrecoveryexperts.com. Contractors/owner-operators submit unpaid debt claims via the portal. DRE reviews, notarizes an LPOA, serves the debtor via certified mail, collects via ACH, deducts a fee, and disburses the balance.
+
+## Technology Stack
+
+| Component | Service | Status |
+|---|---|---|
+| **Signing** | DocuSeal (self-hosted, sign.itpropartner.com) | ✅ Live |
+| **Notarization** | Proof (formerly Notarize) — $25-35/session | 📋 Planned |
+| **Certified mail** | LetterStream — $8.34/letter | 📋 Planned |
+| **Payments** | Stripe ACH (0.8%, capped $5) or checks | 📋 Planned |
+| **AI Analysis** | Claude Opus 4.7 (via admin-ai) — document scoring | 📋 Planned |
+| **Portal** | Caddy reverse proxy on Core (152.53.192.33) | 🟡 Mockup done |
+
+## Domain Plan
+
+| Service | URL | DNS |
+|---|---|---|
+| **Signing** | sign.itpropartner.com | SiteGround A record → 152.53.192.33 (Caddy HTTPS) |
+| **DRE website** | debtrecoveryexperts.com | Cloudflare Zone |
+| **DRE portal** | portal.debtrecoveryexperts.com | Cloudflare A record → 152.53.192.33 |
+| **WordPress** | debtrecoveryexperts.com | wphost02 (RunCloud) |
+
+## Cost Per Claim
+
+- Notarization (RON): ~$25-35
+- Identity verification: ~$4
+- Certified mail: ~$8.34
+- ACH processing: ~$5 max
+- **Total: ~$42-52 per claim** (passed to debtor)
+
+## AI Claim Analysis Flow
+
+1. Customer uploads documents (contracts, invoices, correspondence)
+2. Portal sends docs to Claude Opus 4.7 for analysis
+3. Model scores: documentation quality, debtor solvency, TX legal standing, amount reasonableness
+4. Public case records (TX) are compared — similar case count, avg recovery rate, median time
+5. Internal DRE dashboard shows results — **never visible to customers**
+
+## Legal Documents (Drafts)
+
+- Terms of Service: `/root/TERMS_OF_SERVICE_DRAFT.md` ⚠️ Needs TX attorney review
+- Privacy Policy: `/root/PRIVACY_POLICY_DRAFT.md` ⚠️ Needs TX attorney review
+- Internal Compliance Manual: `/root/DRE_Compliance_Manual.md` ⚠️ Needs TX attorney review
+
+## Texas Law Key Findings (Jul 7, 2026 Research)
+
+| Topic | Finding |
+|---|---|
+| License/Bond | Likely required for third-party debt collectors in TX |
+| LPOA | Must be in writing, notarized, specifically enumerate debt collection powers |
+| Fee caps | No statutory cap in TX — set own %, must disclose in writing |
+| SOL (written contracts) | 4 years — reject claims past SOL |
+| Service of process | Certified mail (restricted delivery) is valid per TRCP 106(a)(2) |
+| FDCPA | Full compliance required: validation notice within 5 days, Mini-Miranda, cease comms |
+| Data breach | Must notify within 60 days per TX Bus. & Com. Code § 521 |
+| Debtor calling | Prior express consent for autodial; manual dial only; 3-call-per-7-day cap |
@@ -0,0 +1,20 @@
+Generated Jul 9, 2026 by research subagent. Full document at /root/.hermes/references/docker-management-research.md.
+
+## Tools compared
+
+| Tool | Multi-Host | REST API | Free | Web GUI |
+|------|-----------|----------|------|---------|
+| **Komodo** | ✅ Periphery Agent | ✅ REST + WS + 3 clients | ✅ 100% FOSS | ⭐⭐⭐⭐ |
+| **Arcane** | ✅ Agent (Direct+Edge) | ✅ OpenAPI 3.1 | ✅ 100% FOSS | ⭐⭐⭐⭐ |
+| **Portainer** | ✅ Agent | ✅ Full API | 3 nodes free | ⭐⭐⭐⭐⭐ |
+| Dockge | ✅ Agents (v1.4+) | ❌ No API | ✅ FOSS | ⭐⭐⭐⭐ |
+| Dockhand | ✅ Agent/TLS | ⏳ Roadmap | ✅ Homelab free | ⭐⭐⭐⭐⭐ |
+| Lazydocker | ❌ Single host | ❌ | ✅ FOSS | ❌ TUI only |
+
+## Recommendation
+
+1. **Komodo** — best API story, GPL-3.0, unlimited servers, built-in backup/restore CLI
+2. **Arcane** — BSD-3-Clause, beautiful UI, no MongoDB, Edge mode for NAT'd hosts
+3. **Portainer** — most mature, best documented, 3-node free tier
+
+**Ruled out:** Dockge (no API), Dockhand (BSL license, API is future), Lazydocker (single host).
@@ -0,0 +1,145 @@
+# Docker Multi-Host Management Tools — Comparison
+
+**Researched:** 2026-07-09
+**Context:** IT Pro Partner multi-server Docker Compose environment (Core, ai.itpropartner.com, docker, unms, etc.). Need single-pane visibility across hosts, REST API for Hermes scripting, and self-hosted.
+
+## Top 3 Recommendation
+
+### 🏆 1st: **Komodo** — API-first multi-host Docker management
+- **Architecture:** Core (web UI + API) + Periphery (agent on each host)
+- **Why:** Best API story — REST + WebSocket + CLI + Rust/npm client libraries. Perfect for Hermes scripting. GPL-3.0, truly free, unlimited servers. Built-in backup/restore, webhooks, GitOps.
+- **Trade-off:** Needs MongoDB (or FerretDB/Postgres). Heavier setup than alternatives.
+- **Setup:** Docker Compose (Core + Mongo) + Python install script for Periphery agents
+- **Docs:** https://komo.do/docs/intro
+- **GitHub:** https://github.com/moghtech/komodo
+
+### 🥈 2nd: **Arcane** — Modern, BSD-3-Clause, no-MongoDB option
+- **Architecture:** Manager (Go backend + SvelteKit UI) + Agent (Direct or Edge mode)
+- **Why:** 100% free BSD-3-Clause. Beautiful UI. OpenAPI 3.1 REST API. Edge mode handles NAT/firewall'd hosts (agents dial out). Built-in vulnerability scanning. iOS mobile app.
+- **Trade-off:** Newer project (6.3k stars), smaller community than Portainer.
+- **Setup:** Single Docker container for Manager, one per Agent
+- **Docs:** https://getarcane.app/docs/setup/installation
+- **GitHub:** https://github.com/getarcaneapp/arcane
+
+### 🥉 3rd: **Portainer** — Most mature, best documented
+- **Architecture:** Portainer Server + lightweight Go Agent on each host
+- **Why:** Industry standard. Extensive docs. Free BE tier covers 3 nodes with full features. Comprehensive REST API. Agent auto-updates, mTLS.
+- **Trade-off:** Beyond 3 nodes costs money. Heavier/fuller-featured than needed for Compose-only.
+- **Setup:** `docker run` agent command on each host, add via UI
+- **Docs:** https://docs.portainer.io/
+- **GitHub:** https://github.com/portainer/portainer
+
+## All Evaluated Tools
+
+| Tool | Multi-Host | REST API | Free | Web GUI | License | Stars | Setup |
+|---|---|---|---|---|---|---|---|
+| **Portainer** | ✅ Agent | ✅ Full API | 3 nodes free | ⭐⭐⭐⭐⭐ | Proprietary/MIT | ~30k | Easy |
+| **Arcane** | ✅ Agent (Direct+Edge) | ✅ OpenAPI 3.1 | ✅ 100% FOSS | ⭐⭐⭐⭐ | BSD-3-Clause | 6.3k | Easy |
+| **Komodo** 🏆 | ✅ Periphery Agent | ✅ REST + WS + 3 libs | ✅ 100% FOSS | ⭐⭐⭐⭐ | GPL-3.0 | ~3k | Medium |
+| **Dockge** | ✅ Agents (v1.4+) | ❌ No API | ✅ 100% FOSS | ⭐⭐⭐⭐ | MIT | 23.7k | Easy |
+| **Dockhand** | ✅ Agent/TLS/Hawser | ⏳ Roadmap (distant) | ✅ Homelab free | ⭐⭐⭐⭐⭐ | BSL 1.1 | 5.1k | Easy |
+| **Lazydocker** | ❌ Single host | ❌ No API | ✅ 100% FOSS | ❌ TUI only | MIT | 38.7k | Very Easy |
+
+## Why the Others Don't Fit
+
+- **Dockge:** No API, no webhooks, compose-only. Great for single-user compose management but can't be scripted by Hermes.
+- **Dockhand:** No API (roadmap item, "distant future"). BSL license (not truly open source). Paid for commercial use. Beautiful UI but unusable for automation.
+- **Lazydocker:** Single host only. Terminal-only. Good for quick SSH checks, not a management platform.
+
+## Quick Deploy Commands
+
+### Komodo (Correct v2 Env Vars — See references/komodo-deployment.md)
+
+**CRITICAL:** Komodo v2 uses `KOMODO_DATABASE_ADDRESS` (not `MONGO_CONNECTION_STRING`), `KOMODO_PORT` (not `KOMODO_SERVER_PORT`). Using wrong env vars causes startup failure with _"Failed to initialize database"_ errors.
+
+```yaml
+# Core (management server) — full reference in references/komodo-deployment.md
+services:
+  mongo:
+    image: mongo
+    restart: unless-stopped
+    command: --quiet --wiredTigerCacheSizeGB 0.25
+    volumes:
+      - ./data/mongo-data:/data/db
+  core:
+    image: ghcr.io/moghtech/komodo-core:2
+    init: true
+    restart: unless-stopped
+    ports:
+      - 9120:9120
+    depends_on:
+      - mongo
+    volumes:
+      - ./backups:/backups
+      - ./keys:/config/keys
+    environment:
+      - KOMODO_DATABASE_ADDRESS=mongo:27017
+      - KOMODO_PORT=9120
+      - KOMODO_HOST=https://komodo.itpropartner.com
+      - KOMODO_TITLE=Komodo
+      - KOMODO_LOCAL_AUTH=true
+      - KOMODO_INIT_ADMIN_USERNAME=admin
+      - KOMODO_INIT_ADMIN_PASSWORD=changeme
+      - KOMODO_FIRST_SERVER_NAME=core
+      - KOMODO_PERIPHERY_PUBLIC_KEY=file:/config/keys/periphery.pub
+      - KOMODO_DISABLE_CONFIRM_DIALOG=false
+      - KOMODO_DISABLE_INIT_RESOURCES=false
+      - KOMODO_JWT_SECRET=change-me
+      - KOMODO_JWT_TTL=1-day
+      - KOMODO_MONITORING_INTERVAL=15-sec
+      - KOMODO_RESOURCE_POLL_INTERVAL=1-hr
+```
+
+```bash
+# Periphery (each managed host)
+curl -sSL https://raw.githubusercontent.com/moghtech/komodo/main/scripts/setup-periphery.py \
+  | python3 - \
+  --core-address="https://komodo.itpropartner.com" \
+  --connect-as="$(hostname)" \
+  --onboarding-key="O-..."
+```
+
+### Arcane
+```bash
+# Manager
+docker run -d --name arcane -p 3552:3552 \
+  -v arcane_data:/data \
+  ghcr.io/getarcaneapp/manager:latest
+
+# Agent (each remote host)
+docker run -d --name arcane-agent -p 3553:3553 \
+  -v /var/run/docker.sock:/var/run/docker.sock \
+  -e AGENT_MODE=true \
+  -e AGENT_TOKEN=<token> \
+  -e MANAGER_API_URL=http://manager:3552 \
+  ghcr.io/getarcaneapp/agent:latest
+```
+
+### Portainer
+```bash
+# Agent on each remote host
+docker run -d -p 9001:9001 --name portainer_agent --restart=always \
+  -v /var/run/docker.sock:/var/run/docker.sock \
+  -v /var/lib/docker/volumes:/var/lib/docker/volumes \
+  portainer/agent:latest
+```
+
+## Key Links
+| Tool | Website | Docs | GitHub |
+|---|---|---|---|
+| Portainer | https://www.portainer.io/ | https://docs.portainer.io/ | https://github.com/portainer/portainer |
+| Arcane | https://getarcane.app/ | https://getarcane.app/docs/setup/installation | https://github.com/getarcaneapp/arcane |
+| Komodo | https://komo.do/ | https://komo.do/docs/intro | https://github.com/moghtech/komodo |
+| Dockge | https://github.com/louislam/dockge | README only | https://github.com/louislam/dockge |
+| Dockhand | https://dockhand.pro/ | https://dockhand.pro/manual/ | https://github.com/Finsys/dockhand |
+| Lazydocker | https://lazydocker.com/ | README only | https://github.com/jesseduffield/lazydocker |
+
+## Multi-Host Agent Architecture Patterns
+
+All multi-host tools use the same pattern: a central management server + lightweight agents on each Docker host.
+
+| Pattern | Used By | How It Works |
+|---|---|---|
+| **Agent polls out** | Arcane Edge, Dockhand Hawser | Agent behind NAT dials out to manager. No inbound port needed. |
+| **Agent listens** | Portainer Agent, Arcane Direct | Manager connects to agent on a TCP port. Requires inbound access. |
+| **Systemd agent** | Komodo Periphery | Agent runs as systemd service (or Docker). Bi-directional WebSocket to Core. |
@@ -0,0 +1,75 @@
+# Docker Volume Migration Pitfalls
+
+Common failures when moving Docker services between hosts.
+
+## Permission denied on data volume (n8n, Jul 10 2026)
+
+**Symptom:** Container starts but immediately crashes with `EACCES: permission denied, open '/home/node/.n8n/crash.journal'`
+
+**Root cause:** Docker named volumes copied from host A (`scp` the `_data` directory) carry the UID/GID from host A's filesystem. The container on host B runs as a different UID (n8n uses UID 1000, node user). The files are owned by `root` or `ippadmin` from the copy operation, not UID 1000.
+
+**Fix:**
+```bash
+VOL=$(docker volume inspect <volume_name> --format "{{.Mountpoint}}")
+chown -R 1000:1000 "$VOL"
+docker compose restart <service>
+```
+
+**For n8n specifically:** UID 1000 is the `node` user inside the container. Both `n8n_data` and `postgres_data` volumes may need this fix.
+
+**Prevention:** After copying volume data to a new host, always check ownership before starting containers:
+```bash
+ls -la /var/lib/docker/volumes/<volume>/_data/ | head -5
+```
+
+## Volume data owned by wrong user (general)
+
+Any Docker container that writes to a bind-mounted or named volume will have files owned by the container's internal UID. When copying between hosts:
+1. Identify the container's UID: `docker run --rm <image> id`
+2. Apply: `chown -R <UID>:<UID> <volume_path>`
+3. Then start the container
+
+## SSL redirect loop after migration (n8n, Jul 10 2026)
+
+**Symptom:** n8n returns 308 redirect loop when accessed through Caddy with HTTPS
+
+**Root cause:** n8n's `N8N_PROTOCOL` is set to `https` but Caddy terminates SSL. n8n sees the request came via HTTP (from Caddy's proxy) and redirects to HTTPS, creating an infinite loop.
+
+**Fix:** Set `N8N_PROTOCOL=http` in `.env` — Caddy handles SSL termination, n8n should trust the proxy.
+
+## Caddy `handle` fallback overrides path routes (Jul 10, 2026)
+
+**Symptom:** Browser shows "app1 — /n8n/ /webui/ /litellm/" instead of the app.
+
+**Root cause:** A `handle { respond "..." 200 }` at the bottom of the Caddyfile catches ALL requests — including `/n8n/signin` — because Caddy evaluates `handle` blocks before `handle_path` directives.
+
+**Fix:** Replace the `handle { respond ... }` with a bare `reverse_proxy` as the catch-all. Path-specific `handle_path` routes take priority over the catch-all reverse_proxy, but NOT over a catch-all `handle` block.
+
+**Lesson:** Only use `handle` blocks for specific path prefixes. For a true catch-all, use `reverse_proxy` directly without wrapping in `handle`. Never put `handle { respond }` as a fallback when you also have `handle_path` routes — it overrides them.
+
+**Symptom:** `app1.itpropartner.com/n8n/` shows the app but all assets (JS, CSS, images) 404
+
+**Root cause:** `handle_path /n8n/*` correctly strips the prefix before proxying, so n8n sees `/signin` instead of `/n8n/signin`. But n8n's HTML references assets at `/assets/...`, `/static/...` — these paths DON'T match `/n8n/*` and fall through to the default handler.
+
+**Fix:** Give the service its own subdomain instead of a sub-path. Path-based routing only works for apps that support a `baseUrl` config option. n8n does not.
+
+**Lesson:** Use sub-domain routing (`n8n.itpropartner.com`) for apps without baseUrl support, sub-path routing (`domain.com/app/`) only for apps that support it.
+
+## Let's Encrypt validation blocked by UFW / DNS (Jul 10, 2026)
+
+**Symptom:** Caddy log shows `authorization failed: HTTP 400 ... Timeout during connect (likely firewall problem)`. Cert never issues.
+
+**Root cause:** UFW default deny blocks HTTP-01 challenge from Let's Encrypt's servers, OR DNS still points to wrong IP (stale SiteGround records during Cloudflare migration).
+
+**Fix:** (1) Ensure `ufw allow 80/tcp` and `ufw allow 443/tcp` on the target server. (2) Verify DNS resolves to the correct IP from outside the server: `ping app1.itpropartner.com`. Core's DNS may be stale while your laptop's is fresh — test from an external source.
+
+## n8n Caddy deployment checklist (Jul 10, 2026)
+
+When deploying n8n behind Caddy on a new host:
+1. `N8N_PROTOCOL=http` — Caddy terminates SSL
+2. `N8N_HOST=<domain>` — match the public domain
+3. `WEBHOOK_URL=https://<domain>/` — for webhook callbacks
+4. Caddy: `reverse_proxy 127.0.0.1:5678 { flush_interval -1 }` — WebSocket support
+5. Caddy: `header { X-Forwarded-Proto https; X-Forwarded-Host {host} }` — proxy headers
+6. UFW: allow 80/tcp + 443/tcp for Let's Encrypt validation
+7. DNS must resolve to this server's IP BEFORE starting Caddy (otherwise cert issuance blocks on first request)
@@ -0,0 +1,47 @@
+# DocuSeal Deployment on Core
+
+Deployed Jul 7, 2026 via Docker on Core (netcup, 152.53.192.33).
+
+## Docker Compose
+
+```yaml
+services:
+  docuseal:
+    image: docuseal/docuseal:latest
+    container_name: docuseal
+    restart: always
+    ports:
+      - "127.0.0.1:3000:3000"
+    volumes:
+      - ./data:/data
+    environment:
+      - HOST=http://localhost:3000
+      - FORCE_SSL=false
+```
+
+Location: `/root/docker/docuseal/docker-compose.yml`
+Data: `/root/docker/docuseal/data/`
+
+## HTTPS
+
+Caddy reverse proxy at https://sign.itpropartner.com → 127.0.0.1:3000
+
+Caddy config at `/etc/caddy/Caddyfile`:
+```
+sign.itpropartner.com {
+    reverse_proxy 127.0.0.1:3000
+}
+```
+
+## API
+
+Token: created in DocuSeal UI (Settings → API Tokens). Linked to info@itpropartner.com.
+
+Test: `curl -s -H "X-Auth-Token: <token>" https://sign.itpropartner.com/api/user`
+
+## Key decisions
+
+- DocuSeal chosen over Documenso for this use case (AGPL, free embedding SDKs, Docker image 227MB, SQLite, no PostgreSQL needed)
+- Both self-hosted options are ESIGN Act / UETA compliant
+- Documenso would require ~$300/yr for embedding SDKs and PostgreSQL — overkill for this scale
+- Dual use: family via web UI, DRE portal via API
@@ -0,0 +1,38 @@
+# DRE Fee Structure
+## Proposed — July 2026
+## For discussion with Tony (DRE business partner)
+
+### Standard Fee Schedule
+
+**Tier 1 — Soft Touch** (automated email + payment link)
+| Claim Amount | DRE Fee | Client Receives |
+|---|---|---|
+| $1,000 – $5,000 | 25% (min $250) | 75% |
+| $5,000 – $15,000 | 22% | 78% |
+| $15,000+ | 20% | 80% |
+
+**Tier 2 — Formal Demand** (certified mail)
+- 30% flat across all amounts
+
+**Tier 3 — Escalation** (final notice + intensive contact)
+- 33% flat across all amounts
+
+**Tier 4 — Legal Action** (referral to partner law firm)
+- DRE: 10% referral fee
+- Law firm: 25% litigation fee
+- Client: 65%
+
+### Example: $15,000 claim
+
+| Scenario | DRE | Client | Timeline |
+|---|---|---|---|
+| Paid at Tier 1 | $3,300 (22%) | $11,700 | 2-14 days |
+| Paid at Tier 2 | $4,500 (30%) | $10,500 | 15-30 days |
+| Paid at Tier 3 | $4,950 (33%) | $10,050 | 30-60 days |
+| Resolved at Tier 4 | $1,500 (10%) + $3,750 (25% firm) | $9,750 | 60-180 days |
+
+### Out-of-pocket costs (passed to debtor or deducted)
+- Online notarization (LPOA): ~$25-35
+- Identity verification: ~$4
+- Certified mail: ~$8.34 per letter
+- Court filing fees: ~$250-400 (Tier 4 only)
@@ -0,0 +1,170 @@
+# Komodo Deployment — Reference
+
+**Installed:** 2026-07-09 on Core (netcup RS 2000)
+**Version:** Komodo v2.2.0
+**Purpose:** Multi-host Docker management across ITPP servers (Core, ai, docker, unms, etc.)
+**Architecture:** Core (web UI + API server, port 9120) + MongoDB → Periphery agents on each managed host
+
+## Directory Structure
+
+```
+/root/docker/komodo/
+├── docker-compose.yml      # Core + MongoDB
+├── data/
+│   ├── mongo-data/         # MongoDB persistent storage
+│   └── mongo-config/       # MongoDB config
+├── backups/                # Database backups
+└── keys/
+    ├── core.key            # Core private key (auto-generated)
+    └── core.pub            # Core public key (auto-generated)
+```
+
+## Docker Compose (Core)
+
+```yaml
+services:
+  mongo:
+    image: mongo
+    restart: unless-stopped
+    command: --quiet --wiredTigerCacheSizeGB 0.25
+    volumes:
+      - ./data/mongo-data:/data/db
+      - ./data/mongo-config:/data/configdb
+    networks:
+      - komodo-net
+    deploy:
+      resources:
+        limits:
+          memory: 1G
+
+  core:
+    image: ghcr.io/moghtech/komodo-core:2
+    init: true
+    restart: unless-stopped
+    ports:
+      - 9120:9120
+    depends_on:
+      - mongo
+    volumes:
+      - ./backups:/backups
+      - ./keys:/config/keys
+    environment:
+      # Database
+      - KOMODO_DATABASE_ADDRESS=mongo:27017
+      # Server
+      - KOMODO_PORT=9120
+      - KOMODO_HOST=https://komodo.itpropartner.com
+      - KOMODO_TITLE=Komodo
+      # Auth
+      - KOMODO_LOCAL_AUTH=true
+      - KOMODO_INIT_ADMIN_USERNAME=admin
+      - KOMODO_INIT_ADMIN_PASSWORD=changeme
+      # First server (this host)
+      - KOMODO_FIRST_SERVER_NAME=core
+      # Periphery auth
+      - KOMODO_PERIPHERY_PUBLIC_KEY=file:/config/keys/periphery.pub
+      # Secrets
+      - KOMODO_WEBHOOK_SECRET=komodo-webhook-secret
+      - KOMODO_JWT_SECRET=komodo-jwt-secret-change-me
+      - KOMODO_JWT_TTL=1-day
+      # Monitoring
+      - KOMODO_MONITORING_INTERVAL=15-sec
+      - KOMODO_RESOURCE_POLL_INTERVAL=1-hr
+      # UI
+      - KOMODO_DISABLE_CONFIRM_DIALOG=false
+      - KOMODO_TRANSPARENT_MODE=false
+    networks:
+      - komodo-net
+    deploy:
+      resources:
+        limits:
+          memory: 512M
+
+networks:
+  komodo-net:
+    driver: bridge
+```
+
+## Critical Env Variable Pitfalls (V2)
+
+- **Do NOT use `MONGO_CONNECTION_STRING`** — Komodo v2 uses `KOMODO_DATABASE_ADDRESS=mongo:27017`
+- **Do NOT use `KOMODO_SERVER_PORT`** — use `KOMODO_PORT=9120`
+- **Do NOT use `KOMODO_SERVER_HOST`** — use `KOMODO_BIND_IP=[::]` (default)
+- **`KOMODO_PERIPHERY_PUBLIC_KEY`** must be set with `file:/config/keys/periphery.pub` for auto-generation
+- **`init: true`** is essential on the core container (handles PID 1 zombie reaping properly)
+- **`KOMODO_HOST` must be set** to the URL users will access (used for OAuth redirects, webhook suggestions)
+
+## Startup Log Verification
+
+Successful startup shows:
+```
+Server starting on http://[::]:9120
+Creating init admin user...
+Successfully created init admin user.
+Creating initial system resources...
+```
+
+The admin user is created on first launch only. Core v2.2.0 auto-creates starting resources: "Backup Core Database" procedure and "Global Auto Update" procedure.
+
+## Admin Login (API)
+
+```bash
+# Get JWT token
+curl -s -X POST http://127.0.0.1:9120/auth/login \
+  -H "Content-Type: application/json" \
+  -d '{"type":"Local","username":"admin","password":"changeme"}'
+```
+
+Note: the login body requires `"type":"Local"` — missing this field returns: _"Failed to deserialize the JSON body into the target type: missing field `type`"_
+
+## Onboarding Key
+
+The onboarding key is available in the Komodo web UI after login:
+1. Go to the **Servers** page
+2. Click **Add Server / Onboard**
+3. Copy the onboarding key (starts with `O-`)
+
+Or retrieve via the API with a valid JWT:
+```bash
+curl -s http://127.0.0.1:9120/api/core -H "Authorization: Bearer <token>"
+# Extract onboarding_key from response
+```
+
+## Periphery Installation (on each managed host)
+
+```bash
+# Via systemd script (recommended for production)
+curl -sSL https://raw.githubusercontent.com/moghtech/komodo/main/scripts/setup-periphery.py \
+  | python3 - \
+  --core-address="https://komodo.itpropartner.com" \
+  --connect-as="$(hostname)" \
+  --onboarding-key="O-..."
+```
+
+The Periphery agent communicates with Core via WebSocket. It listens on port 8120 by default. The agent runs as a systemd service when installed via the Python script.
+
+## Docker Periphery (alternative, for containers)
+
+```yaml
+services:
+  periphery:
+    image: ghcr.io/moghtech/komodo-periphery:2
+    init: true
+    restart: unless-stopped
+    depends_on:
+      - core
+    env_file: ./compose.env
+    volumes:
+      - keys:/config/keys
+      - /var/run/docker.sock:/var/run/docker.sock
+      - /proc:/proc
+      - ${PERIPHERY_ROOT_DIRECTORY:-/etc/komodo}:${PERIPHERY_ROOT_DIRECTORY:-/etc/komodo}
+```
+
+## Public Key Authentication
+
+Core uses Noise protocol key pairs for Periphery authentication.
+
+- Core keys auto-generated at: `/config/keys/core.key` and `/config/keys/core.pub`
+- Periphery public key expected at: `/config/keys/periphery.pub` (auto-generated if `KOMODO_PERIPHERY_PUBLIC_KEY` is set and file doesn't exist)
+- Example public key: `-----BEGIN PUBLIC KEY-----\nMCowBQYDK2VuAyEAo+xXlRx+KcxbFkscAx3JcmGKudDwoID3AkGnzjFb1AE=\n-----END PUBLIC KEY-----`
@@ -0,0 +1,66 @@
+# LiteLLM Config Precedence Pitfalls
+
+## Master Key Config Override (Jul 2026)
+
+### The Problem
+
+LiteLLM reads `general_settings.master_key` from config.yaml when started with `--config /app/config.yaml`. This value takes precedence over the `LITELLM_MASTER_KEY` environment variable. If config.yaml has a stale/stub key, the stub silently becomes the UI login password.
+
+### How It Manifests
+
+- Admin UI login at `/v2/login` returns `401 Unauthorized`
+- Error: `"Invalid credentials used to access UI. Check 'UI_USERNAME', 'UI_PASSWORD' in .env file"`
+- Container logs: `ProxyException` at `login_utils.py` line 316 in `authenticate_user()`
+- The `/v1/chat/completions` endpoint may still work normally (API keys are separate from UI password)
+- User knows the correct master key (e.g., from `.env`) but login still fails
+
+### Root Cause
+
+In `login_utils.py`, the `authenticate_user()` function:
+
+1. Calls `get_ui_credentials(master_key)` which reads `UI_USERNAME` (default "admin") and `UI_PASSWORD` (env var or `master_key` parameter)
+2. The `master_key` parameter comes from LiteLLM's internal state, which loaded it from `general_settings.master_key` in config.yaml
+3. Compares the login form password against this config-file-derived key using `secrets.compare_digest()`
+4. If the comparison fails and no database user matches, throws the 401 error
+
+### Detection Workflow
+
+```bash
+# 1. Check container logs for login failures
+docker logs litellm --tail 50 2>&1 | grep -A 5 'login_v2\|401 Unauthorized'
+
+# 2. Compare config sources by hash and length (NOT raw value)
+ENV_KEY=$(grep LITELLM_MASTER_KEY .env | cut -d= -f2)
+CFG_KEY=$(grep 'master_key:' config.yaml | sed 's/.*master_key: //')
+echo "env:  len=${#ENV_KEY} sha=$(echo -n "$ENV_KEY" | sha256sum | cut -c1-16)"
+echo "cfg:  len=${#CFG_KEY} sha=$(echo -n "$CFG_KEY" | sha256sum | cut -c1-16)"
+
+# 3. Verify what the container actually sees
+docker exec litellm printenv | grep LITELLM_MASTER_KEY
+
+# 4. If lengths/hashes differ → config.yaml has a stale key
+# 5. Fix by removing master_key from config.yaml
+sed -i '/^  master_key:/d' config.yaml
+docker compose restart litellm
+
+# 6. Verify fix with a direct login test
+MASTER_KEY=$(grep LITELLM_MASTER_KEY .env | cut -d= -f2)
+curl -s -X POST http://127.0.0.1:4000/v2/login \
+  -H 'Content-Type: application/json' \
+  -d "{\"username\":\"admin\",\"password\":\"$MASTER_KEY\"}"
+# Expected: {"user_id":"...","key":"sk-...","user_role":"proxy_admin",...}
+```
+
+### Why API Still Works While Login Fails
+
+LiteLLM has two separate auth paths:
+- **API key auth** (`/v1/chat/completions`): Validates keys stored in `LiteLLM_VerificationToken` table, independent of the master_key
+- **UI login** (`/v2/login`): Compares against the in-memory `master_key` which came from config.yaml
+
+This is why a config.yaml with a stale key breaks UI login but leaves API traffic unaffected.
+
+### Prevention
+
+1. **Remove `master_key` from config.yaml** — rely on `LITELLM_MASTER_KEY` env var only
+2. **Audit dual-source configs** — any service with both config file AND env var for the same setting risks this category of mismatch
+3. **Use hash comparison for credentials** — never log or transmit raw secret values; compare by hash to detect mismatches without exposing secrets
@@ -0,0 +1,74 @@
+# LiteLLM env-key migration: encrypted DB rows after host move
+
+When migrating LiteLLM with `store_model_in_db: true`, `/v1/models` can return a healthy-looking catalog while `/chat/completions` fails or hangs. If logs show:
+
+```text
+Error decrypting value for key: api_key
+Did your master_key/salt key change recently?
+nacl.exceptions.CryptoError: Decryption failed
+```
+
+then the migrated Postgres DB contains encrypted model/provider params that the new container cannot decrypt.
+
+## Safe diagnosis
+
+Proceed read-only first. Do not print raw keys.
+
+1. Compare old and new container env fingerprints:
+   - `LITELLM_MASTER_KEY`
+   - `LITELLM_SALT_KEY`
+   - `DATABASE_URL`
+   - `STORE_MODEL_IN_DB`
+2. Compare by `sha256:<first16> len:<n>` only.
+3. Check model/credential/token table counts:
+   - `LiteLLM_ProxyModelTable`
+   - `LiteLLM_CredentialsTable`
+   - `LiteLLM_VerificationToken`
+4. Confirm public `/v1/models` and local `/health/readiness` separately from completions.
+
+## Minimal fix when fingerprints differ
+
+Scope the change to the new host only.
+
+1. Backup the new host `.env` first:
+   ```bash
+   cp /root/docker/litellm/.env /root/docker/litellm/.env.pre-keyfix-$(date -u +%Y%m%d-%H%M%S)
+   chmod 600 /root/docker/litellm/.env.pre-keyfix-*
+   ```
+2. Copy only these values from the old working host into the new host `.env`:
+   - `LITELLM_MASTER_KEY`
+   - `LITELLM_SALT_KEY`
+3. Verify the new host `.env` fingerprints match old.
+4. **Recreate the LiteLLM container; restart is not enough.** Docker container env is baked at create time:
+   ```bash
+   cd /root/docker/litellm
+   docker compose --env-file .env up -d --force-recreate --no-deps litellm
+   ```
+5. Verify the recreated container env fingerprints, not just the file.
+
+## Verification
+
+Run public HTTPS completion probes after readiness passes:
+
+```bash
+curl -sS https://admin-ai.example.com/v1/chat/completions \
+  -H "Authorization: Bearer $KEY" \
+  -H 'Content-Type: application/json' \
+  -d '{"model":"deepseek-chat","messages":[{"role":"user","content":"Reply with OK only."}],"max_tokens":128}'
+```
+
+Verify at least:
+- `deepseek-v4-pro`
+- `deepseek-chat`
+- `gemini-pro-latest`
+- `gemini-flash-latest`
+
+Use enough `max_tokens` for reasoning models. A too-small token budget can produce blank `content` even though the call succeeded.
+
+## Interpretation
+
+If non-OpenAI models work but `gpt-5.5` returns `429 insufficient_quota`, the migration/encryption issue is fixed; the remaining problem is upstream OpenAI billing/quota.
+
+## User handling standard
+
+For Germaine, migrations touching env/secrets/service restarts must be handled as: read-only compare first, report fingerprints and exact planned minimal change, get approval, change one scoped item, verify, then proceed. Do not bundle extra config changes or unrelated cleanup.
@@ -0,0 +1,32 @@
+# LiteLLM DB Migration Pitfall — Encrypted Models Don't Survive pg_dump
+
+**Date:** Jul 13, 2026
+**Root cause:** LiteLLM encrypts model parameters (api_key, model name, provider) in `LiteLLM_ProxyModelTable` using the `LITELLM_SALT_KEY`. A `pg_dump` + `pg_restore` to a new server will list the models in `/v1/models` but completions fail with "Invalid model name passed."
+
+**Symptom:** Same Docker image (v1.84.0), same SALT_KEY, same DB dump. Models appear in the listing but every completion returns `400: Invalid model name passed in model=X. Call /v1/models to view available models for your key.`
+
+**Why:** The encryption uses not just SALT_KEY but also some per-installation factor (possibly container hostname, Docker network IP, or a runtime-generated nonce) that differs between hosts. Identical SALT_KEY, identical DB, identical image — still can't decrypt.
+
+**Fix:** Do NOT rely on DB migration for models. Instead:
+1. Export the list of model names from the old box
+2. Recreate each model fresh on the new box via the `/model/new` endpoint
+3. Test each model with a completion
+
+```bash
+# List models on old box
+curl -s https://admin-ai.itpropartner.com/v1/models -H "Authorization: Bearer $KEY" | jq -r '.data[].id' > models.txt
+
+# Recreate on new box
+while read model; do
+  curl -s http://127.0.0.1:4000/model/new -X POST \
+    -H "Authorization: Bearer $MASTER_KEY" \
+    -H "Content-Type: application/json" \
+    -d "{\"model_name\":\"$model\",\"litellm_params\":{\"model\":\"$model\",\"custom_llm_provider\":\"openai\",\"api_base\":\"https://admin-ai.itpropartner.com/v1\",\"api_key\":\"$ADMIN_AI_KEY\"}}"
+done < models.txt
+```
+
+**Alternative:** If migrating TO a self-hosted LiteLLM that replaces the old admin-ai proxy, configure models to use OpenRouter directly instead of routing through admin-ai. This avoids the proxy-in-a-loop problem post-DNS-cutover.
+
+**Credentials table survives** — The `LiteLLM_CredentialsTable` decrypts fine after migration. Only `LiteLLM_ProxyModelTable` entries fail.
+
+**Post-migration config.yaml requirement:** The new LiteLLM needs a `config.yaml` with `database_url`, `store_model_in_db: true`, and `enforce_database: true`. Without it, models created via API are lost on restart.
@@ -0,0 +1,43 @@
+# n8n Migration: WEBHOOK_URL + Workflow Webhook Fix
+
+When migrating n8n from one domain to another, two things MUST be updated:
+
+## 1. Environment Variable — .env
+
+```
+WEBHOOK_URL=https://NEW_DOMAIN/
+```
+
+This controls the base URL for all webhook nodes. Restart n8n after changing.
+
+## 2. Existing Workflow Webhook URLs — PostgreSQL
+
+n8n stores webhook URLs in the `nodes` JSON column of `workflow_entity`. After changing `WEBHOOK_URL`, existing workflows still have old URLs baked in. Update them directly:
+
+```sql
+UPDATE workflow_entity 
+SET nodes = regexp_replace(
+    nodes::text, 
+    'https://old\.domain\.com', 
+    'https://new.domain.com', 
+    'g'
+)::json;
+```
+
+Then restart n8n to reload workflows from the database.
+
+## Full Migration Steps
+
+1. `docker compose down` on old server
+2. `tar czf n8n-data.tar.gz` Postgres volume + n8n data volume
+3. `scp` to new server
+4. Update `.env`: `N8N_HOST`, `N8N_PROTOCOL`, `WEBHOOK_URL` to new domain
+5. `docker compose up -d` on new server
+6. Fix volume permissions: `chown -R 1000:1000` on n8n data volume (node user inside container = UID 1000)
+7. Update workflow webhook URLs via SQL above
+8. Restart n8n: `docker compose restart n8n`
+9. Verify: `SELECT nodes::text LIKE '%old.domain.com%' FROM workflow_entity WHERE name LIKE '%workflow_name%';` → should return `f`
+
+## Common Pitfall
+
+Sub-path routing (e.g., `app1.domain.com/n8n/`) breaks n8n because its SPA uses absolute asset paths (`/assets/`, `/static/`). Always use a dedicated subdomain (`n8n.domain.com`).
@@ -0,0 +1,87 @@
+# Open WebUI ConfigVar Cheat Sheet
+
+Open WebUI persists connection settings in `webui.db` → `config` table as `ConfigVar` values. After first launch, env vars are **ignored** — the DB is authoritative. Changing `-e OPENAI_API_BASE_URL=...` and restarting does nothing.
+
+## Config Table Schema
+
+The `config` table is key-value with JSON values:
+```sql
+SELECT key, value FROM config WHERE key LIKE '%ollama%' OR key LIKE '%openai%';
+```
+
+## Connection Keys (as of Jul 2026)
+
+### Ollama Connection
+| Key | Type | Example |
+|-----|------|---------|
+| `ollama.enable` | bool | `true` / `false` |
+| `ollama.base_urls` | JSON array | `["http://ollama:11434"]` |
+| `ollama.api_configs` | JSON object | `{"0": {"enable": true, "tags": [], "prefix_id": "", "model_ids": [], "connection_type": "local", "auth_type": "bearer", "key": ""}}` |
+
+### OpenAI Connection (used for admin-ai / LiteLLM)
+| Key | Type | Example |
+|-----|------|---------|
+| `openai.enable` | bool | `true` / `false` |
+| `openai.api_base_urls` | JSON array | `["https://admin-ai.itpropartner.com/v1"]` |
+| `openai.api_keys` | JSON array | `["sk-lg_..."]` |
+| `openai.api_configs` | JSON object | `{"0": {"enable": true, "tags": [], "prefix_id": "", "model_ids": [], "connection_type": "external", "auth_type": "bearer"}}` |
+
+### RAG / Embeddings
+| Key | Type | Example |
+|-----|------|---------|
+| `rag.ollama.base_url` | string | `"http://host.docker.internal:11434"` |
+| `rag.ollama.api_key` | string | `""` |
+| `rag.openai.api_base_url` | string | `"http://litellm:4000/v1"` |
+| `rag.openai.api_key` | string | `"sk-..."` |
+
+### Audio (TTS/STT)
+| Key | Type | Example |
+|-----|------|---------|
+| `audio.tts.openai.api_base_url` | string | `"http://litellm:4000/v1"` |
+| `audio.tts.openai.api_key` | string | `"sk-..."` |
+| `audio.stt.openai.api_base_url` | string | `"http://litellm:4000/v1"` |
+| `audio.stt.openai.api_key` | string | `"sk-..."` |
+
+### Image Generation
+| Key | Type | Example |
+|-----|------|---------|
+| `image_generation.openai.api_base_url` | string | `"http://127.0.0.0:4000/v1"` |
+| `image_generation.openai.api_key` | string | `"sk-..."` |
+
+## Common Operations
+
+### Disable Ollama (stop pointing to Ollama)
+```bash
+sqlite3 /var/lib/docker/volumes/openwebui_data/_data/webui.db \
+  "UPDATE config SET value = 'false' WHERE key = 'ollama.enable';"
+sqlite3 /var/lib/docker/volumes/openwebui_data/_data/webui.db \
+  "UPDATE config SET value = json_set(value, '\$.\"0\".enable', json('false')) WHERE key = 'ollama.api_configs';"
+docker restart openwebui
+```
+
+### Change OpenAI base URL (switch admin-ai endpoint)
+```bash
+sqlite3 /var/lib/docker/volumes/openwebui_data/_data/webui.db \
+  "UPDATE config SET value = json_set(value, '\$[0]', 'https://new-url/v1') WHERE key = 'openai.api_base_urls';"
+docker restart openwebui
+```
+
+### Verify changes took effect
+```bash
+sqlite3 /var/lib/docker/volumes/openwebui_data/_data/webui.db \
+  "SELECT key, value FROM config WHERE key IN ('ollama.enable', 'openai.api_base_urls', 'ollama.api_configs');"
+```
+
+### Full audit — dump all connection-related config
+```bash
+sqlite3 /var/lib/docker/volumes/openwebui_data/_data/webui.db \
+  "SELECT key, value FROM config WHERE key LIKE '%ollama%' OR key LIKE '%openai%' ORDER BY key;"
+```
+
+## Pitfalls
+
+- **`OLLAMA_BASE_URL=/ollama` is baked into the image** — omitting the env var doesn't clear it; the image default applies. After first launch, it's persisted in `ollama.base_urls` in the DB.
+- **`ENABLE_PERSISTENT_CONFIG=false` disables ALL UI settings persistence** — use only for one-off fixes, not as permanent config.
+- **The container image doesn't include `sqlite3`** — install on the host: `apt-get install -y sqlite3`.
+- **After DB changes, `docker restart` is required** — the config is read at startup.
+- **JSON path syntax is strict** — `\$.\"0\".enable` for object keys that start with digits (SQLite's `json_set` requires double-quoted keys for numeric-looking keys).
@@ -0,0 +1,44 @@
+# SearXNG Self-Hosted Search on Core
+
+Deployed Jul 7, 2026 via Docker on Core. Port 8888 (127.0.0.1 only).
+
+## Docker Compose
+
+```yaml
+services:
+  searxng:
+    image: searxng/searxng:latest
+    container_name: searxng
+    restart: always
+    ports:
+      - "127.0.0.1:8888:8080"
+    volumes:
+      - ./searxng-data:/etc/searxng:rw
+      - ./searxng-themes:/usr/local/searxng/searx/static/themes:rw
+    environment:
+      - SEARXNG_BASE_URL=http://localhost:8080
+      - SEARXNG_SECRET_KEY=<auto-generated>
+```
+
+Location: `/root/docker/searxng/docker-compose.yml`
+Settings: `/root/docker/searxng/searxng-data/settings.yml` (added `search.formats: [html, json]` to enable JSON API)
+
+## Usage
+
+Search via JSON API:
+```
+curl -s "http://127.0.0.1:8888/search?q=query&format=json"
+```
+
+## Motivation
+
+- Hermes web tools (web_search, web_extract) require Firecrawl API key — not yet configured
+- SearXNG provides unlimited, self-hosted search as fallback
+- Used via terminal curl in Hermes — not as seamless as native tools but free and unlimited
+
+## Relation to Firecrawl
+
+SearXNG + Firecrawl complement each other:
+- SearXNG: unlimited search (finding URLs)
+- Firecrawl: page content extraction (getting text from URLs)
+- With both, Firecrawl only handles extraction — free tier lasts much longer
@@ -0,0 +1,42 @@
+# SearXNG — Self-Hosted Search Engine
+
+Deployed on **Core** (`/root/docker/searxng/`) via Docker Compose. Used as a local search engine when Firecrawl's API key is not configured.
+
+## Connection details
+- **URL:** `http://127.0.0.1:8888`
+- **JSON API:** `http://127.0.0.1:8888/search?q=QUERY&format=json`
+- **Container:** `searxng`
+- **Auto-start:** Docker restart policy
+- **Settings:** `/root/docker/searxng/searxng-data/settings.yml`
+
+## Configuration
+The settings.yml file enables both HTML and JSON formats:
+```yaml
+search:
+  formats:
+    - html
+    - json
+```
+
+## Usage from agent
+```bash
+curl -s "http://127.0.0.1:8888/search?q=QUERY&format=json" | python3 -c "
+import sys, json
+d = json.load(sys.stdin)
+for r in d.get('results',[])[:N]:
+    print(f'{r.get(\"title\",\"?\")} - {r.get(\"url\",\"?\")}')
+"
+```
+
+## Limitations
+- **No extraction.** SearXNG returns search results (title + URL + snippet) but does not extract full page content. For full page extraction, use Firecrawl or `curl` + HTML stripping.
+- **Local only.** Bound to 127.0.0.1 — not accessible from the network. Access via terminal on Core.
+- **Rate limited.** Default rate limit is 120 requests/hour from a single IP.
+
+## Relationship to Firecrawl
+| Capability | SearXNG | Firecrawl |
+|---|---|---|
+| Search (find links) | ✅ Unlimited | ⚠️ 500/mo free |
+| Extract (get page content) | ❌ | ✅ Clean markdown |
+| API key needed | No | Yes |
+| Self-hosted | Yes | No (cloud) |
@@ -0,0 +1,41 @@
+# Service Health Check (no_agent watchdog)
+
+A reusable pattern for no_agent cron jobs that monitor ALL critical services and only report on failure.
+
+## Design
+
+```bash
+#!/bin/bash
+# silent on success, report failures only
+
+CHECKS=()
+FAILED=0
+
+# Each check:
+command || { CHECKS+=("Service Name"); FAILED=1; }
+
+if [ $FAILED -eq 1 ]; then
+    echo "⚠️  Service Health Alert — $(date)"
+    for c in "${CHECKS[@]}"; do echo "  ❌ $c"; done
+    exit 1
+fi
+exit 0
+```
+
+## What to check
+
+- Systemd services (caddy, mysql-tunnel)
+- Docker containers (docker ps --format)
+- Local HTTP endpoints (curl -sf)
+- Remote reachability (ping through tunnel, SSH)
+- Database connectivity (mysql -e "SELECT 1")
+- API token validity (curl + verify response)
+
+## Pitfalls
+
+- **Must be no_agent: true** in cron config — LLM processing is wasted on binary health checks and causes typing indicators
+- **Include the exit code** — exit 0 = silent, exit 1 = output delivered. Without the right exit code, cron can't tell success from failure
+- **Test with `bash script.sh` and check `echo $?`** before deploying
+- **MySQL checks need `--skip-ssl`** for local tunnel connections
+- **Don't check too many things** — 10-12 checks is the sweet spot. More than 15 and the script gets brittle
+- **SSH checks need `-o BatchMode=yes -o ConnectTimeout=5`** to avoid hanging on unreachable hosts
@@ -0,0 +1,127 @@
+# TwentyCRM Docker Compose Deployment
+
+**Channel:** Core VPS, served at https://crm.debtrecoveryexperts.com  
+**Port:** 3001 (DocuSeal occupies 3000)  
+**Image:** `twentycrm/twenty:latest` (v2.20.0 pre-release as of 2026-07-07)  
+**Stack dir:** `~/docker/twenty/`
+
+## Services (4 containers)
+
+| Container | Image | Purpose | Depends on |
+|---|---|---|---|
+| `twenty-server-1` | `twentycrm/twenty:latest` | NestJS API + frontend SPA on port 3000 | db (healthy), redis (healthy) |
+| `twenty-worker-1` | `twentycrm/twenty:latest` | Background job processor (BullMQ) | db, server (healthy) |
+| `twenty-db-1` | `postgres:16` | PostgreSQL database | — |
+| `twenty-redis-1` | `redis` (noeviction policy) | Queue + cache | — |
+
+## docker-compose.yml
+
+Located at `~/docker/twenty/docker-compose.yml`. Based on upstream's `packages/twenty-docker/docker-compose.yml` with changes:
+
+- **Host port 3000 → 3001** to avoid conflict with DocuSeal.
+- **Added `REACT_APP_SERVER_BASE_URL: ${SERVER_URL}`** to server container environment.
+
+## .env
+
+Based on `packages/twenty-docker/.env.example`. Key values:
+
+```
+TAG=latest
+PG_DATABASE_USER=postgres
+PG_DATABASE_PASSWORD=<generated>
+PG_DATABASE_HOST=db
+PG_DATABASE_PORT=5432
+PG_DATABASE_NAME=default
+REDIS_URL=redis://redis:6379
+SERVER_URL=https://crm.debtrecoveryexperts.com
+ENCRYPTION_KEY=<openssl rand -base64 32>
+STORAGE_TYPE=local
+```
+
+**CRITICAL:** `SERVER_URL` must match the public domain exactly. Setting it to `http://localhost:3000` or `http://localhost:3001` will cause API calls to fail from the browser because the frontend uses `SERVER_URL` (via `REACT_APP_SERVER_BASE_URL`) for GraphQL requests.
+
+### Generate the encryption key
+
+```bash
+openssl rand -base64 32
+```
+
+**Warning:** Losing `ENCRYPTION_KEY` means losing access to every secret stored in the database (OAuth tokens, TOTP secrets, etc.). Back it up.
+
+## Initial deployment
+
+```bash
+git clone https://github.com/twentyhq/twenty.git ~/docker/twenty
+cp ~/docker/twenty/packages/twenty-docker/docker-compose.yml ~/docker/twenty/
+cp ~/docker/twenty/packages/twenty-docker/.env.example ~/docker/twenty/.env
+# Edit .env with encryption key + password + SERVER_URL
+# Adjust port in docker-compose.yml if needed
+# ADD REACT_APP_SERVER_BASE_URL: ${SERVER_URL} to server environment
+cd ~/docker/twenty && docker compose up -d
+```
+
+## PITFALL: REACT_APP_SERVER_BASE_URL required for reverse proxy
+
+TwentyCRM's Docker image serves both the React frontend and NestJS backend
+from the same container. The React build includes a runtime env injection step
+(`inject-runtime-env.sh`) that reads `REACT_APP_SERVER_BASE_URL` from the
+container environment.
+
+Without this env var, the frontend defaults API requests to `http://localhost:3000`,
+which breaks when accessed through Caddy/nginx because the browser sends API
+requests to `localhost` instead of the public domain.
+
+**Fix:** Add to docker-compose.yml server environment:
+```yaml
+environment:
+  NODE_PORT: 3000
+  PG_DATABASE_URL: ...
+  SERVER_URL: ${SERVER_URL}
+  REACT_APP_SERVER_BASE_URL: ${SERVER_URL}   # MUST ADD
+  REDIS_URL: ${REDIS_URL:-redis://redis:6379}
+```
+
+After adding, restart: `docker compose up -d server`
+
+## Caddy Reverse Proxy
+
+```caddy
+crm.debtrecoveryexperts.com {
+    reverse_proxy 127.0.0.1:3001
+}
+```
+
+## First-time setup (browser)
+
+1. Navigate to https://crm.debtrecoveryexperts.com/
+2. Click "Continue with Email"
+3. Create workspace admin account (sign-up form)
+4. Configure workspace name, timezone, currency
+5. Optionally set up Google/Microsoft calendar + email integrations
+
+## Verification
+
+```bash
+# All 4 containers healthy
+docker ps --filter name=twenty --format '{{.Names}} {{.Status}}'
+
+# Frontend serving
+curl -sI https://crm.debtrecoveryexperts.com/ | head -3
+
+# Backend health
+curl -s https://crm.debtrecoveryexperts.com/healthz
+
+# GraphQL API responding
+curl -s https://crm.debtrecoveryexperts.com/graphql \
+  -H 'Content-Type: application/json' \
+  -d '{"query":"{__typename}"}'
+
+# Console: check env config injected in HTML
+curl -sL https://crm.debtrecoveryexperts.com/ | grep -A3 'twenty-env-config'
+```
+
+## Backup
+
+```bash
+docker exec twenty-db-1 pg_dump -U postgres default > backup_$(date +%Y%m%d).sql
+```
@@ -0,0 +1,112 @@
+# TwentyCRM — DRE CRM Deployment Notes
+
+## Architecture (Docker Compose on Core)
+
+4 containers: `twenty-server` (NestJS API + React frontend), `twenty-worker` (BullMQ jobs), `twenty-db` (PostgreSQL 16), `twenty-redis` (cache/queue).
+
+## PITFALL: Frontend API calls go to localhost
+
+TwentyCRM's frontend SPA is served by the same NestJS server. At startup, the container runs `inject-runtime-env.sh` which reads `REACT_APP_SERVER_BASE_URL` from the environment and injects it into the HTML. If this env var is missing, the frontend defaults to `http://localhost:3000` for all API calls.
+
+**Fix:** Both `SERVER_URL` AND `REACT_APP_SERVER_BASE_URL` must be set in docker-compose.yml:
+```yaml
+environment:
+  NODE_PORT: 3000
+  SERVER_URL: ${SERVER_URL}
+  REACT_APP_SERVER_BASE_URL: ${SERVER_URL}    # ← Critical
+```
+
+**Test:** After restart, verify:
+```bash
+curl -s https://crm.debtrecoveryexperts.com/ | grep 'REACT_APP_SERVER_BASE_URL'
+# Should show the real domain, not localhost
+```
+
+## PITFALL: .env changes need full restart
+
+After changing `.env`, run `docker compose down && docker compose up -d`. Just restarting the server container is NOT enough — the frontend build is baked into the image and runs the inject script at container startup.
+
+## PITFALL: DNS propagation for new subdomains
+
+When creating a new subdomain (e.g., crm.debtrecoveryexperts.com), the initial A record (non-proxied) can take 5-15 minutes to propagate. If the user gets ERR_NAME_NOT_RESOLVED from their browser, switch to Cloudflare proxied (orange cloud) mode for instant resolution.
+
+## PITFALL: Port conflict with DocuSeal (port 3000)
+
+DocuSeal already uses port 3000. Map TwentyCRM to 3001:
+```yaml
+ports:
+  - "3001:3000"
+```
+
+## API Field Creation
+
+**REST endpoint for field creation (TEXT, NUMBER, CURRENCY, DATE, etc.):**
+```
+POST https://crm.debtrecoveryexperts.com/metadata
+Body: {"query":"mutation{createOneField(input:{field:{name:\"fieldName\",label:\"Field Label\",type:TEXT,objectMetadataId:\"...\"}}){id name}}"}
+```
+
+**RESERVED field names:** Avoid `address` as a field name — TwentyCRM rejects it with validation errors. Use `physicalAddress` or `mailingAddress` instead.
+
+**RELATION fields:** Creating relationships (foreign keys) via the REST API requires a `relationCreationPayload` parameter whose exact shape is not well documented. The UI is the reliable path for creating relationships.
+
+**Object IDs for DRE custom objects (July 7, 2026):**
+- Claims: `0c6808cf-b5e8-4f39-8c4f-6dc6b7d029f3`
+- Debtors: `bac887ba-22a9-46fe-b988-e1985ee19b29`
+- Payments: `bde74e7b-b1c2-48eb-be5b-f3ea2207873c`
+- Case Notes: `176736a7-6d3b-4129-ac03-67cea102d618`
+
+## DRE Custom Data Model
+
+### Claims Fields
+| Field | Type |
+|-------|------|
+| claimNumber | TEXT |
+| claimAmount | NUMBER |
+| debtType | TEXT |
+| currentTier | TEXT |
+| status | TEXT |
+| state | TEXT |
+| dateFiled | DATE |
+| aiScore | NUMBER |
+| description | TEXT |
+
+### Debtors Fields
+| Field | Type |
+|-------|------|
+| businessName | TEXT |
+| contactName | TEXT |
+| email | TEXT |
+| phone | TEXT |
+| address | TEXT |
+| businessType | TEXT |
+| notes | TEXT |
+
+### Payments Fields
+| Field | Type |
+|-------|------|
+| paymentAmount | NUMBER |
+| paymentDate | DATE |
+| paymentMethod | TEXT |
+| dreFee | NUMBER |
+| costsDeducted | NUMBER |
+| netToClient | NUMBER |
+
+### Case Notes Fields
+| Field | Type |
+|-------|------|
+| content | TEXT |
+| timestamp | DATE |
+
+## Default Workflows to Delete
+
+Twenty ships with sample workflows: "Create company when adding a new person" and "Quick Lead". Delete via REST:
+```bash
+for wid in "id1" "id2"; do
+  curl -X DELETE "https://crm.debtrecoveryexperts.com/rest/workflows/$wid" -H "Authorization: Bearer $TOKEN"
+done
+```
+
+Check remaining: `curl -s https://crm.debtrecoveryexperts.com/rest/workflows -H "Authorization: Bearer $TOKEN" | python3 -c "import sys,json; [print(w['name']) for w in json.load(sys.stdin)['data']['workflows']]"`
+
+Relationships still need to be set up via the UI: Claims → Debtor, Payments → Claim, Case Notes → Claim.
@@ -0,0 +1,18 @@
+# Vaultwarden Deployment Reference (Core via Caddy)
+
+**Server:** Core (netcup, 152.53.192.33)
+**URL:** https://core.itpropartner.com/vault/
+**Caddy:** handle_path /vault/* → reverse_proxy localhost:8080
+**Image:** vaultwarden/server:latest
+**Data volume:** vaultwarden-data
+
+
+## ⚠️ DOMAIN env var pitfall (Jul 12, 2026)
+
+**Setting `DOMAIN` causes Vaultwarden to return 404 on ALL routes.** Omit `DOMAIN` entirely. The Caddy reverse proxy handles URL routing transparently.
+
+## Pending
+
+- Replace plain-text ADMIN_TOKEN with Argon2 PHC string via `vaultwarden hash`
+- Add Tailscale Serve to any future service needing internal HTTPS
+- Set up SMTP if multi-user support is needed later
@@ -0,0 +1,122 @@
+# Web Scraping & Search Toolkit
+
+Technical reference for the three-tier web research stack configured on Core.
+
+## Tier 1 — SearXNG (self-hosted, unlimited)
+
+- **URL:** `http://127.0.0.1:8888`
+- **Docker:** `~/docker/searxng/`
+- **Port:** 8888 (localhost only)
+- **API:** `http://127.0.0.1:8888/search?q=query&format=json`
+- **Config:** `searxng-data/settings.yml` — must enable JSON format: `search.formats: [html, json]`
+- **Auth:** None (localhost only, no auth needed)
+- **Capacity:** Unlimited — self-hosted, no rate limits
+
+Used for: quick web lookups, private search, government site search (basic). Does NOT render JavaScript.
+
+## Tier 2 — Firecrawl (cloud, free tier)
+
+- **API key:** in `~/.hermes/.env` or per-profile `.env`
+- **Plan:** Starter (1,000 credits/mo)
+- **Endpoint:** `https://api.firecrawl.dev/v1/scrape`
+- **Native Hermes tools:** `web_search`, `web_extract` (loaded per session)
+- **Config:** `web.backend: firecrawl` in config.yaml
+
+Used for: content extraction from URLs, web search grounded in current data. Free tier good for ~30 lookups/day. Upgrade to Standard (~$50-100/mo) for 5,000 credits if needed.
+
+**Note:** The native `web_search` and `web_extract` tools require the API key to be present in `.env` at session START. Adding the key mid-session won't enable them until the next `/reset` or new Hermes invocation.
+
+## Tier 3 — ScrapingAnt (cloud, headless Chrome)
+
+- **API key:** in `~/.hermes/.env` or per-profile `.env`
+- **Plan:** Free (10,000 credits), then $19/mo for 100,000 credits
+- **Endpoint:** `https://api.scrapingant.com/v2/general?url=...&x-api-key=...`
+- **JS rendering:** `browser=true` parameter for headless Chrome execution
+
+Used for: JS-heavy sites (LinkedIn, Indeed, CBDriver, government portals, SPA apps). Essential for any site that serves blank HTML without JavaScript execution.
+
+## Usage Decision Matrix
+
+| Site type | Tool | Timeout |
+|---|---|---|
+| Simple HTML pages, blogs, docs | SearXNG (free, instant) | 5s |
+| JSON API documentation, content extraction | Firecrawl | 10s |
+| JavaScript-heavy, SPAs, login gates | ScrapingAnt (browser=true) | 30s |
+| Government sites (Texas Legislature, court records) | ScrapingAnt (browser=true) | 30s+ |
+| People search (CBDriver, Indeed) | ScrapingAnt | 15s |
+| LinkedIn public profiles | ScrapingAnt (proxy_country=US) | 30s |
+
+## Credit Tracking
+
+Usage logged at `~/.hermes/scripts/firecrawl-usage.json` via `track-firecrawl.py` cron. Daily check at 9 AM. Also tracked in portal at `capabilities.html`.
+
+## Sharing Across Profiles
+
+When another profile (e.g. Anita's) needs web tools, the API keys must be added to THEIR profile's `.env` — the main profile's `.env` is NOT inherited:
+
+```bash
+# ~/.hermes/profiles/anita/.env
+FIRECRAWL_API_KEY=...
+SCRAPINGANT_API_KEY=...
+SEARXNG_BASE_URL=http://127.0.0.1:8888
+```
+
+Also add to their `config.yaml`:
+```yaml
+web:
+  backend: firecrawl
+  use_gateway: true
+```
+
+Enable the web toolset: `hermes tools enable web --profile <name>`
+
+**CRITICAL: Profile Isolation — Config changes need gateway restart AND Telegram token**
+1. After config changes, the gateway must be RESTARTED — values are NOT hot-reloaded
+2. If the gateway restarts and comes up with NO messaging platforms connected, the Telegram bot token is likely missing from that profile's `.env`
+3. The main profile's Telegram token is NOT inherited by child profiles
+4. Each profile that needs Telegram must have `TELEGRAM_BOT_TOKEN`, `TELEGRAM_ALLOWED_USERS`, and `TELEGRAM_HOME_CHANNEL` in its own `.env`
+5. Anita's gateway died during config update because the restart cron killed the old process and spawned one with web API keys but no Telegram token
+6. Fix: copy Telegram env vars from main `.env` to the profile's `.env`, then kill the orphaned gateway PID and restart
+
+## Data Broker Removal
+
+Full action plan at `/root/data-broker-removal-action-plan.md`. Recommendation: use Kanary ($84/yr) for automated removal across 50+ sites. No tools needed beyond the search stack — the same ScrapingAnt + Firecrawl combo can scan for a person's presence on data broker sites.
+
+## Caddy Multi-Domain Reverse Proxy
+
+Caddy handles HTTPS termination and routing for multiple domains on the same server (Core, 152.53.192.33). Each domain gets a separate block in `/etc/caddy/Caddyfile`:
+
+```caddy
+sign.itpropartner.com {
+    reverse_proxy 127.0.0.1:3000
+}
+
+core.itpropartner.com {
+    header Access-Control-Allow-Origin "*"
+    @vehicles path /vehicles.json
+    handle @vehicles {
+        root * /var/www/static
+        file_server
+    }
+}
+
+app.itpropartner.com {
+    reverse_proxy 127.0.0.1:8081
+}
+```
+
+**Caddy commands:**
+- `caddy fmt --overwrite /etc/caddy/Caddyfile` — format the config file
+- `systemctl reload caddy` — hot-reload without downtime
+- `systemctl restart caddy` — full restart (needed when adding new domains)
+- Caddy auto-provisions Let's Encrypt certificates for each domain
+- After adding a new domain to the Caddyfile, Caddy needs ~10-15s to provision the certificate before the domain starts serving HTTPS
+
+**New domain DNS setup:**
+1. Add A record at DNS provider → 152.53.192.33
+2. Add `domain.com { ... }` block to Caddyfile
+3. `caddy fmt --overwrite` + `systemctl restart caddy`
+4. Wait ~15s for Let's Encrypt
+5. HTTPS works automatically
+
+**Port 443 note:** If Tailscale Serve is running, it holds port 443. Run `tailscale serve off` to free it before starting Caddy. This disables all Tailscale Serve routes.