# Web Research & Scraping Stack

Configured Jul 7, 2026. Three-tier search/scraping capability for Hermes and Anita's profile.

## Current Stack

### Firecrawl (native web tools)
- **API Key:** fc-4f9...63c1 (in ~/.hermes/.env)
- **Plan:** Starter (1,000 credits/mo free)
- **Endpoint:** api.firecrawl.dev
- **Purpose:** Native `web_search` and `web_extract` tools in Hermes. Quick content extraction from URLs.
- **Auth:** Bearer token in Authorization header
- **Credits used per call:** 1 (cache hit) to 1+ (fresh scrape)
- **Credit tracking:** ~/.hermes/scripts/firecrawl-usage.json
- **Upgrade option:** ~$50-100/mo for 5,000 credits (not cost-effective vs ScrapingAnt — 20x less volume at higher cost)

### ScrapingAnt (JS-heavy sites)
- **API Key:** 8ab7...4af6 (in ~/.hermes/.env)
- **Plan:** Free tier (10,000 credits) → $19/mo for 100K+
- **Purpose:** JS-rendered pages, government sites (Texas Legislature), LinkedIn, Indeed, CBDriver
- **Endpoint:** api.scrapingant.com/v2/general?url=...&x-api-key=... (with optional &browser=true for JS render)
- **Features:** Headless Chrome, `browser=true` param for JS rendering
- **Cost per request:** varies by JS render usage

### SearXNG (self-hosted, unlimited)
- **URL:** http://127.0.0.1:8888
- **Purpose:** Private web search, unlimited queries, multi-engine aggregation
- **Format:** JSON API: `http://127.0.0.1:8888/search?q=<query>&format=json`
- **Deployment:** Docker container on Core (port 8888, localhost only)
- **Limits:** 120 req/min

## Anita's Profile Access

Anita's profile at ~/.hermes/profiles/anita/ was configured with:
- `.env` file with FIRECRAWL_API_KEY, SCRAPINGANT_API_KEY, SEARXNG_BASE_URL
- `web.backend: firecrawl` in config.yaml
- `web` toolset enabled
- Gateway restart required to pick up new config

## Use Cases

| Site | Tool | Method |
|------|------|--------|
| Government (TX Leg., court records) | ScrapingAnt | `browser=true` for JS rendering |
| CBDriver, Indeed | ScrapingAnt | Headless Chrome handles JS |
| LinkedIn (public profiles) | ScrapingAnt | Can pull public pages |
| LinkedIn (deep search) | Apify LinkedIn actor (~$30-50/mo) | Handles auth, rate limits, structured data |
| General web search | SearXNG (self-hosted) | Unlimited, private |
| Quick URL extract | Firecrawl (native tool) | Free credits |
| Debtor skip tracing | Sherlock + theHarvester | Username search / domain recon |

## OSINT Tools (installed on Core)

- **Sherlock** (`/opt/sherlock/`) — search 400+ sites by username. `cd /opt/sherlock && ./venv/bin/python sherlock <username>`
- **theHarvester** (`/opt/theharvester/`) — email/domain recon. `cd /opt/theharvester && ./venv/bin/python theHarvester -d <domain> -b google`

## Usage Tracking

`~/.hermes/scripts/track-firecrawl.py` tracks monthly usage across both services.
Cron: `firecrawl-usage-check` at 9 AM daily (no_agent).

## Cost Recommendation

Keep Firecrawl free tier for quick lookups (1K/mo free). Add ScrapingAnt $19/mo for JS-heavy sites (100K credits). Skip Firecrawl upgrade entirely. Add Apify LinkedIn actor (~$30-50/mo) only when deep LinkedIn people search is needed for DRE skip tracing.