Initial skills documentation — 25 categories, all SKILL.md + references + scripts

2026-07-15 17:42:20 -04:00
commit 1b4fcd4427
976 changed files with 188344 additions and 0 deletions
@@ -0,0 +1,122 @@
+# Web Scraping & Search Toolkit
+
+Technical reference for the three-tier web research stack configured on Core.
+
+## Tier 1 — SearXNG (self-hosted, unlimited)
+
+- **URL:** `http://127.0.0.1:8888`
+- **Docker:** `~/docker/searxng/`
+- **Port:** 8888 (localhost only)
+- **API:** `http://127.0.0.1:8888/search?q=query&format=json`
+- **Config:** `searxng-data/settings.yml` — must enable JSON format: `search.formats: [html, json]`
+- **Auth:** None (localhost only, no auth needed)
+- **Capacity:** Unlimited — self-hosted, no rate limits
+
+Used for: quick web lookups, private search, government site search (basic). Does NOT render JavaScript.
+
+## Tier 2 — Firecrawl (cloud, free tier)
+
+- **API key:** in `~/.hermes/.env` or per-profile `.env`
+- **Plan:** Starter (1,000 credits/mo)
+- **Endpoint:** `https://api.firecrawl.dev/v1/scrape`
+- **Native Hermes tools:** `web_search`, `web_extract` (loaded per session)
+- **Config:** `web.backend: firecrawl` in config.yaml
+
+Used for: content extraction from URLs, web search grounded in current data. Free tier good for ~30 lookups/day. Upgrade to Standard (~$50-100/mo) for 5,000 credits if needed.
+
+**Note:** The native `web_search` and `web_extract` tools require the API key to be present in `.env` at session START. Adding the key mid-session won't enable them until the next `/reset` or new Hermes invocation.
+
+## Tier 3 — ScrapingAnt (cloud, headless Chrome)
+
+- **API key:** in `~/.hermes/.env` or per-profile `.env`
+- **Plan:** Free (10,000 credits), then $19/mo for 100,000 credits
+- **Endpoint:** `https://api.scrapingant.com/v2/general?url=...&x-api-key=...`
+- **JS rendering:** `browser=true` parameter for headless Chrome execution
+
+Used for: JS-heavy sites (LinkedIn, Indeed, CBDriver, government portals, SPA apps). Essential for any site that serves blank HTML without JavaScript execution.
+
+## Usage Decision Matrix
+
+| Site type | Tool | Timeout |
+|---|---|---|
+| Simple HTML pages, blogs, docs | SearXNG (free, instant) | 5s |
+| JSON API documentation, content extraction | Firecrawl | 10s |
+| JavaScript-heavy, SPAs, login gates | ScrapingAnt (browser=true) | 30s |
+| Government sites (Texas Legislature, court records) | ScrapingAnt (browser=true) | 30s+ |
+| People search (CBDriver, Indeed) | ScrapingAnt | 15s |
+| LinkedIn public profiles | ScrapingAnt (proxy_country=US) | 30s |
+
+## Credit Tracking
+
+Usage logged at `~/.hermes/scripts/firecrawl-usage.json` via `track-firecrawl.py` cron. Daily check at 9 AM. Also tracked in portal at `capabilities.html`.
+
+## Sharing Across Profiles
+
+When another profile (e.g. Anita's) needs web tools, the API keys must be added to THEIR profile's `.env` — the main profile's `.env` is NOT inherited:
+
+```bash
+# ~/.hermes/profiles/anita/.env
+FIRECRAWL_API_KEY=...
+SCRAPINGANT_API_KEY=...
+SEARXNG_BASE_URL=http://127.0.0.1:8888
+```
+
+Also add to their `config.yaml`:
+```yaml
+web:
+  backend: firecrawl
+  use_gateway: true
+```
+
+Enable the web toolset: `hermes tools enable web --profile <name>`
+
+**CRITICAL: Profile Isolation — Config changes need gateway restart AND Telegram token**
+1. After config changes, the gateway must be RESTARTED — values are NOT hot-reloaded
+2. If the gateway restarts and comes up with NO messaging platforms connected, the Telegram bot token is likely missing from that profile's `.env`
+3. The main profile's Telegram token is NOT inherited by child profiles
+4. Each profile that needs Telegram must have `TELEGRAM_BOT_TOKEN`, `TELEGRAM_ALLOWED_USERS`, and `TELEGRAM_HOME_CHANNEL` in its own `.env`
+5. Anita's gateway died during config update because the restart cron killed the old process and spawned one with web API keys but no Telegram token
+6. Fix: copy Telegram env vars from main `.env` to the profile's `.env`, then kill the orphaned gateway PID and restart
+
+## Data Broker Removal
+
+Full action plan at `/root/data-broker-removal-action-plan.md`. Recommendation: use Kanary ($84/yr) for automated removal across 50+ sites. No tools needed beyond the search stack — the same ScrapingAnt + Firecrawl combo can scan for a person's presence on data broker sites.
+
+## Caddy Multi-Domain Reverse Proxy
+
+Caddy handles HTTPS termination and routing for multiple domains on the same server (Core, 152.53.192.33). Each domain gets a separate block in `/etc/caddy/Caddyfile`:
+
+```caddy
+sign.itpropartner.com {
+    reverse_proxy 127.0.0.1:3000
+}
+
+core.itpropartner.com {
+    header Access-Control-Allow-Origin "*"
+    @vehicles path /vehicles.json
+    handle @vehicles {
+        root * /var/www/static
+        file_server
+    }
+}
+
+app.itpropartner.com {
+    reverse_proxy 127.0.0.1:8081
+}
+```
+
+**Caddy commands:**
+- `caddy fmt --overwrite /etc/caddy/Caddyfile` — format the config file
+- `systemctl reload caddy` — hot-reload without downtime
+- `systemctl restart caddy` — full restart (needed when adding new domains)
+- Caddy auto-provisions Let's Encrypt certificates for each domain
+- After adding a new domain to the Caddyfile, Caddy needs ~10-15s to provision the certificate before the domain starts serving HTTPS
+
+**New domain DNS setup:**
+1. Add A record at DNS provider → 152.53.192.33
+2. Add `domain.com { ... }` block to Caddyfile
+3. `caddy fmt --overwrite` + `systemctl restart caddy`
+4. Wait ~15s for Let's Encrypt
+5. HTTPS works automatically
+
+**Port 443 note:** If Tailscale Serve is running, it holds port 443. Run `tailscale serve off` to free it before starting Caddy. This disables all Tailscale Serve routes.