123 lines
5.3 KiB
Markdown
123 lines
5.3 KiB
Markdown
# Web Scraping & Search Toolkit
|
|
|
|
Technical reference for the three-tier web research stack configured on Core.
|
|
|
|
## Tier 1 — SearXNG (self-hosted, unlimited)
|
|
|
|
- **URL:** `http://127.0.0.1:8888`
|
|
- **Docker:** `~/docker/searxng/`
|
|
- **Port:** 8888 (localhost only)
|
|
- **API:** `http://127.0.0.1:8888/search?q=query&format=json`
|
|
- **Config:** `searxng-data/settings.yml` — must enable JSON format: `search.formats: [html, json]`
|
|
- **Auth:** None (localhost only, no auth needed)
|
|
- **Capacity:** Unlimited — self-hosted, no rate limits
|
|
|
|
Used for: quick web lookups, private search, government site search (basic). Does NOT render JavaScript.
|
|
|
|
## Tier 2 — Firecrawl (cloud, free tier)
|
|
|
|
- **API key:** in `~/.hermes/.env` or per-profile `.env`
|
|
- **Plan:** Starter (1,000 credits/mo)
|
|
- **Endpoint:** `https://api.firecrawl.dev/v1/scrape`
|
|
- **Native Hermes tools:** `web_search`, `web_extract` (loaded per session)
|
|
- **Config:** `web.backend: firecrawl` in config.yaml
|
|
|
|
Used for: content extraction from URLs, web search grounded in current data. Free tier good for ~30 lookups/day. Upgrade to Standard (~$50-100/mo) for 5,000 credits if needed.
|
|
|
|
**Note:** The native `web_search` and `web_extract` tools require the API key to be present in `.env` at session START. Adding the key mid-session won't enable them until the next `/reset` or new Hermes invocation.
|
|
|
|
## Tier 3 — ScrapingAnt (cloud, headless Chrome)
|
|
|
|
- **API key:** in `~/.hermes/.env` or per-profile `.env`
|
|
- **Plan:** Free (10,000 credits), then $19/mo for 100,000 credits
|
|
- **Endpoint:** `https://api.scrapingant.com/v2/general?url=...&x-api-key=...`
|
|
- **JS rendering:** `browser=true` parameter for headless Chrome execution
|
|
|
|
Used for: JS-heavy sites (LinkedIn, Indeed, CBDriver, government portals, SPA apps). Essential for any site that serves blank HTML without JavaScript execution.
|
|
|
|
## Usage Decision Matrix
|
|
|
|
| Site type | Tool | Timeout |
|
|
|---|---|---|
|
|
| Simple HTML pages, blogs, docs | SearXNG (free, instant) | 5s |
|
|
| JSON API documentation, content extraction | Firecrawl | 10s |
|
|
| JavaScript-heavy, SPAs, login gates | ScrapingAnt (browser=true) | 30s |
|
|
| Government sites (Texas Legislature, court records) | ScrapingAnt (browser=true) | 30s+ |
|
|
| People search (CBDriver, Indeed) | ScrapingAnt | 15s |
|
|
| LinkedIn public profiles | ScrapingAnt (proxy_country=US) | 30s |
|
|
|
|
## Credit Tracking
|
|
|
|
Usage logged at `~/.hermes/scripts/firecrawl-usage.json` via `track-firecrawl.py` cron. Daily check at 9 AM. Also tracked in portal at `capabilities.html`.
|
|
|
|
## Sharing Across Profiles
|
|
|
|
When another profile (e.g. Anita's) needs web tools, the API keys must be added to THEIR profile's `.env` — the main profile's `.env` is NOT inherited:
|
|
|
|
```bash
|
|
# ~/.hermes/profiles/anita/.env
|
|
FIRECRAWL_API_KEY=...
|
|
SCRAPINGANT_API_KEY=...
|
|
SEARXNG_BASE_URL=http://127.0.0.1:8888
|
|
```
|
|
|
|
Also add to their `config.yaml`:
|
|
```yaml
|
|
web:
|
|
backend: firecrawl
|
|
use_gateway: true
|
|
```
|
|
|
|
Enable the web toolset: `hermes tools enable web --profile <name>`
|
|
|
|
**CRITICAL: Profile Isolation — Config changes need gateway restart AND Telegram token**
|
|
1. After config changes, the gateway must be RESTARTED — values are NOT hot-reloaded
|
|
2. If the gateway restarts and comes up with NO messaging platforms connected, the Telegram bot token is likely missing from that profile's `.env`
|
|
3. The main profile's Telegram token is NOT inherited by child profiles
|
|
4. Each profile that needs Telegram must have `TELEGRAM_BOT_TOKEN`, `TELEGRAM_ALLOWED_USERS`, and `TELEGRAM_HOME_CHANNEL` in its own `.env`
|
|
5. Anita's gateway died during config update because the restart cron killed the old process and spawned one with web API keys but no Telegram token
|
|
6. Fix: copy Telegram env vars from main `.env` to the profile's `.env`, then kill the orphaned gateway PID and restart
|
|
|
|
## Data Broker Removal
|
|
|
|
Full action plan at `/root/data-broker-removal-action-plan.md`. Recommendation: use Kanary ($84/yr) for automated removal across 50+ sites. No tools needed beyond the search stack — the same ScrapingAnt + Firecrawl combo can scan for a person's presence on data broker sites.
|
|
|
|
## Caddy Multi-Domain Reverse Proxy
|
|
|
|
Caddy handles HTTPS termination and routing for multiple domains on the same server (Core, 152.53.192.33). Each domain gets a separate block in `/etc/caddy/Caddyfile`:
|
|
|
|
```caddy
|
|
sign.itpropartner.com {
|
|
reverse_proxy 127.0.0.1:3000
|
|
}
|
|
|
|
core.itpropartner.com {
|
|
header Access-Control-Allow-Origin "*"
|
|
@vehicles path /vehicles.json
|
|
handle @vehicles {
|
|
root * /var/www/static
|
|
file_server
|
|
}
|
|
}
|
|
|
|
app.itpropartner.com {
|
|
reverse_proxy 127.0.0.1:8081
|
|
}
|
|
```
|
|
|
|
**Caddy commands:**
|
|
- `caddy fmt --overwrite /etc/caddy/Caddyfile` — format the config file
|
|
- `systemctl reload caddy` — hot-reload without downtime
|
|
- `systemctl restart caddy` — full restart (needed when adding new domains)
|
|
- Caddy auto-provisions Let's Encrypt certificates for each domain
|
|
- After adding a new domain to the Caddyfile, Caddy needs ~10-15s to provision the certificate before the domain starts serving HTTPS
|
|
|
|
**New domain DNS setup:**
|
|
1. Add A record at DNS provider → 152.53.192.33
|
|
2. Add `domain.com { ... }` block to Caddyfile
|
|
3. `caddy fmt --overwrite` + `systemctl restart caddy`
|
|
4. Wait ~15s for Let's Encrypt
|
|
5. HTTPS works automatically
|
|
|
|
**Port 443 note:** If Tailscale Serve is running, it holds port 443. Run `tailscale serve off` to free it before starting Caddy. This disables all Tailscale Serve routes.
|