hermes-skills/skills/devops/docker-service-deployment/references/web-scraping-toolkit.md

Web Scraping & Search Toolkit

Technical reference for the three-tier web research stack configured on Core.

Tier 1 — SearXNG (self-hosted, unlimited)

• URL: http://127.0.0.1:8888
• Docker: ~/docker/searxng/
• Port: 8888 (localhost only)
• API: http://127.0.0.1:8888/search?q=query&format=json
• Config: searxng-data/settings.yml — must enable JSON format: search.formats: [html, json]
• Auth: None (localhost only, no auth needed)
• Capacity: Unlimited — self-hosted, no rate limits

Used for: quick web lookups, private search, government site search (basic). Does NOT render JavaScript.

Tier 2 — Firecrawl (cloud, free tier)

• API key: in ~/.hermes/.env or per-profile .env
• Plan: Starter (1,000 credits/mo)
• Endpoint: https://api.firecrawl.dev/v1/scrape
• Native Hermes tools: web_search, web_extract (loaded per session)
• Config: web.backend: firecrawl in config.yaml

Used for: content extraction from URLs, web search grounded in current data. Free tier good for 30 lookups/day. Upgrade to Standard ($50-100/mo) for 5,000 credits if needed.

Note: The native web_search and web_extract tools require the API key to be present in .env at session START. Adding the key mid-session won't enable them until the next /reset or new Hermes invocation.

Tier 3 — ScrapingAnt (cloud, headless Chrome)

• API key: in ~/.hermes/.env or per-profile .env
• Plan: Free (10,000 credits), then $19/mo for 100,000 credits
• Endpoint: https://api.scrapingant.com/v2/general?url=...&x-api-key=...
• JS rendering: browser=true parameter for headless Chrome execution

Used for: JS-heavy sites (LinkedIn, Indeed, CBDriver, government portals, SPA apps). Essential for any site that serves blank HTML without JavaScript execution.

Usage Decision Matrix

Site type	Tool	Timeout
Simple HTML pages, blogs, docs	SearXNG (free, instant)	5s
JSON API documentation, content extraction	Firecrawl	10s
JavaScript-heavy, SPAs, login gates	ScrapingAnt (browser=true)	30s
Government sites (Texas Legislature, court records)	ScrapingAnt (browser=true)	30s+
People search (CBDriver, Indeed)	ScrapingAnt	15s
LinkedIn public profiles	ScrapingAnt (proxy_country=US)	30s

Credit Tracking

Usage logged at ~/.hermes/scripts/firecrawl-usage.json via track-firecrawl.py cron. Daily check at 9 AM. Also tracked in portal at capabilities.html.

Sharing Across Profiles

When another profile (e.g. Anita's) needs web tools, the API keys must be added to THEIR profile's .env — the main profile's .env is NOT inherited:

# ~/.hermes/profiles/anita/.env
FIRECRAWL_API_KEY=...
SCRAPINGANT_API_KEY=...
SEARXNG_BASE_URL=http://127.0.0.1:8888

Also add to their config.yaml:

web:
  backend: firecrawl
  use_gateway: true

Enable the web toolset: hermes tools enable web --profile <name>

CRITICAL: Profile Isolation — Config changes need gateway restart AND Telegram token

1. After config changes, the gateway must be RESTARTED — values are NOT hot-reloaded
2. If the gateway restarts and comes up with NO messaging platforms connected, the Telegram bot token is likely missing from that profile's .env
3. The main profile's Telegram token is NOT inherited by child profiles
4. Each profile that needs Telegram must have TELEGRAM_BOT_TOKEN, TELEGRAM_ALLOWED_USERS, and TELEGRAM_HOME_CHANNEL in its own .env
5. Anita's gateway died during config update because the restart cron killed the old process and spawned one with web API keys but no Telegram token
6. Fix: copy Telegram env vars from main .env to the profile's .env, then kill the orphaned gateway PID and restart

Data Broker Removal

Full action plan at /root/data-broker-removal-action-plan.md. Recommendation: use Kanary ($84/yr) for automated removal across 50+ sites. No tools needed beyond the search stack — the same ScrapingAnt + Firecrawl combo can scan for a person's presence on data broker sites.

Caddy Multi-Domain Reverse Proxy

Caddy handles HTTPS termination and routing for multiple domains on the same server (Core, 152.53.192.33). Each domain gets a separate block in /etc/caddy/Caddyfile:

sign.itpropartner.com {
    reverse_proxy 127.0.0.1:3000
}

core.itpropartner.com {
    header Access-Control-Allow-Origin "*"
    @vehicles path /vehicles.json
    handle @vehicles {
        root * /var/www/static
        file_server
    }
}

app.itpropartner.com {
    reverse_proxy 127.0.0.1:8081
}

Caddy commands:

• caddy fmt --overwrite /etc/caddy/Caddyfile — format the config file
• systemctl reload caddy — hot-reload without downtime
• systemctl restart caddy — full restart (needed when adding new domains)
• Caddy auto-provisions Let's Encrypt certificates for each domain
• After adding a new domain to the Caddyfile, Caddy needs ~10-15s to provision the certificate before the domain starts serving HTTPS

New domain DNS setup:

1. Add A record at DNS provider → 152.53.192.33
2. Add domain.com { ... } block to Caddyfile
3. caddy fmt --overwrite + systemctl restart caddy
4. Wait ~15s for Let's Encrypt
5. HTTPS works automatically

Port 443 note: If Tailscale Serve is running, it holds port 443. Run tailscale serve off to free it before starting Caddy. This disables all Tailscale Serve routes.