# Web Research Tools Stack

Configured Jul 7, 2026 for IT Pro Partner infrastructure.

## Active Tools

| Tool | Type | Cost | Credits | Used for |
|------|------|------|---------|----------|
| **Firecrawl** | Web search + content extraction | Free tier | 1,000/mo | Quick web lookups, web_search and web_extract tools |
| **ScrapingAnt** | JS rendering via headless Chrome | Free tier | 10,000 free (no card) | JS-heavy sites, government pages, Texas Legislature, LinkedIn public pages |
| **SearXNG** | Self-hosted meta search (Docker) | $0 (runs on Core) | Unlimited | Private search, fallback when other tools unavailable |

## Configuration

### Firecrawl
- API key saved in `.env`: `FIRECRAWL_API_KEY=fc-4f9...63c1`
- Hermes config: `web.backend: firecrawl`, `web.use_gateway: true`
- Activate on next session /reset after adding key
- Free tier: 1,000 credits/mo, upgrade to 5K credits planned when needed
- Usage tracking: cron job `firecrawl-usage-check` runs daily at 9 AM, no_agent

### ScrapingAnt
- API key saved in `.env`: `SCRAPINGANT_API_KEY=8ab7...af6`
- Used via direct API calls from terminal (`curl` to scrapingant.com/v2/general with browser=true)
- 10,000 free credits/mo, no credit card required
- $19/mo for 100K credits (recommended upgrade when free tier runs out)

### SearXNG
- Deployed as Docker container on Core at port 8888 (localhost only)
- Accessed via `curl "http://127.0.0.1:8888/search?q=...&format=json"`
- JSON API returns structured results with titles, URLs, snippets
- Self-hosted, unlimited, no API key needed
- Auto-starts with Docker (compose file at ~/docker/searxng/)

## Multi-Profile Setup

When a second Hermes profile (e.g. Anita's `anita` profile) needs web tools:
1. Copy API keys to the profile's `.env` file
2. Enable web backend in the profile's `config.yaml`
3. Restart the profile's gateway to pick up changes
4. The user needs to /reset their session to see the new tools

## Why Both Firecrawl and SearXNG?

SearXNG handles unlimited search (finding the right article/result at no cost). Firecrawl handles content extraction (converting specific page URLs to clean markdown). Without SearXNG, Firecrawl would handle both search + extraction but burn credits on searches that SearXNG can do for free.