# Web Scraping & Search Toolkit

Technical reference for the three-tier web research stack configured on Core.

## Tier 1 — SearXNG (self-hosted, unlimited)

- **URL:** `http://127.0.0.1:8888`
- **Docker:** `~/docker/searxng/`
- **Port:** 8888 (localhost only)
- **API:** `http://127.0.0.1:8888/search?q=query&format=json`
- **Config:** `searxng-data/settings.yml` — must enable JSON format: `search.formats: [html, json]`
- **Auth:** None (localhost only, no auth needed)
- **Capacity:** Unlimited — self-hosted, no rate limits

Used for: quick web lookups, private search, government site search (basic). Does NOT render JavaScript.

## Tier 2 — Firecrawl (cloud, free tier)

- **API key:** in `~/.hermes/.env` or per-profile `.env`
- **Plan:** Starter (1,000 credits/mo)
- **Endpoint:** `https://api.firecrawl.dev/v1/scrape`
- **Native Hermes tools:** `web_search`, `web_extract` (loaded per session)
- **Config:** `web.backend: firecrawl` in config.yaml

Used for: content extraction from URLs, web search grounded in current data. Free tier good for ~30 lookups/day. Upgrade to Standard (~$50-100/mo) for 5,000 credits if needed.

**Note:** The native `web_search` and `web_extract` tools require the API key to be present in `.env` at session START. Adding the key mid-session won't enable them until the next `/reset` or new Hermes invocation.

## Tier 3 — ScrapingAnt (cloud, headless Chrome)

- **API key:** in `~/.hermes/.env` or per-profile `.env`
- **Plan:** Free (10,000 credits), then $19/mo for 100,000 credits
- **Endpoint:** `https://api.scrapingant.com/v2/general?url=...&x-api-key=...`
- **JS rendering:** `browser=true` parameter for headless Chrome execution

Used for: JS-heavy sites (LinkedIn, Indeed, CBDriver, government portals, SPA apps). Essential for any site that serves blank HTML without JavaScript execution.

## Usage Decision Matrix

| Site type | Tool | Timeout |
|---|---|---|
| Simple HTML pages, blogs, docs | SearXNG (free, instant) | 5s |
| JSON API documentation, content extraction | Firecrawl | 10s |
| JavaScript-heavy, SPAs, login gates | ScrapingAnt (browser=true) | 30s |
| Government sites (Texas Legislature, court records) | ScrapingAnt (browser=true) | 30s+ |
| People search (CBDriver, Indeed) | ScrapingAnt | 15s |
| LinkedIn public profiles | ScrapingAnt (proxy_country=US) | 30s |

## Credit Tracking

Usage logged at `~/.hermes/scripts/firecrawl-usage.json` via `track-firecrawl.py` cron. Daily check at 9 AM. Also tracked in portal at `capabilities.html`.

## Sharing Across Profiles

When another profile (e.g. Anita's) needs web tools, the API keys must be added to THEIR profile's `.env` — the main profile's `.env` is NOT inherited:

```bash
# ~/.hermes/profiles/anita/.env
FIRECRAWL_API_KEY=...
SCRAPINGANT_API_KEY=...
SEARXNG_BASE_URL=http://127.0.0.1:8888
```

Also add to their `config.yaml`:
```yaml
web:
  backend: firecrawl
  use_gateway: true
```

Enable the web toolset: `hermes tools enable web --profile <name>`

**CRITICAL: Profile Isolation — Config changes need gateway restart AND Telegram token**
1. After config changes, the gateway must be RESTARTED — values are NOT hot-reloaded
2. If the gateway restarts and comes up with NO messaging platforms connected, the Telegram bot token is likely missing from that profile's `.env`
3. The main profile's Telegram token is NOT inherited by child profiles
4. Each profile that needs Telegram must have `TELEGRAM_BOT_TOKEN`, `TELEGRAM_ALLOWED_USERS`, and `TELEGRAM_HOME_CHANNEL` in its own `.env`
5. Anita's gateway died during config update because the restart cron killed the old process and spawned one with web API keys but no Telegram token
6. Fix: copy Telegram env vars from main `.env` to the profile's `.env`, then kill the orphaned gateway PID and restart

## Data Broker Removal

Full action plan at `/root/data-broker-removal-action-plan.md`. Recommendation: use Kanary ($84/yr) for automated removal across 50+ sites. No tools needed beyond the search stack — the same ScrapingAnt + Firecrawl combo can scan for a person's presence on data broker sites.

## Caddy Multi-Domain Reverse Proxy

Caddy handles HTTPS termination and routing for multiple domains on the same server (Core, 152.53.192.33). Each domain gets a separate block in `/etc/caddy/Caddyfile`:

```caddy
sign.itpropartner.com {
    reverse_proxy 127.0.0.1:3000
}

core.itpropartner.com {
    header Access-Control-Allow-Origin "*"
    @vehicles path /vehicles.json
    handle @vehicles {
        root * /var/www/static
        file_server
    }
}

app.itpropartner.com {
    reverse_proxy 127.0.0.1:8081
}
```

**Caddy commands:**
- `caddy fmt --overwrite /etc/caddy/Caddyfile` — format the config file
- `systemctl reload caddy` — hot-reload without downtime
- `systemctl restart caddy` — full restart (needed when adding new domains)
- Caddy auto-provisions Let's Encrypt certificates for each domain
- After adding a new domain to the Caddyfile, Caddy needs ~10-15s to provision the certificate before the domain starts serving HTTPS

**New domain DNS setup:**
1. Add A record at DNS provider → 152.53.192.33
2. Add `domain.com { ... }` block to Caddyfile
3. `caddy fmt --overwrite` + `systemctl restart caddy`
4. Wait ~15s for Let's Encrypt
5. HTTPS works automatically

**Port 443 note:** If Tailscale Serve is running, it holds port 443. Run `tailscale serve off` to free it before starting Caddy. This disables all Tailscale Serve routes.