46 lines
2.2 KiB
Markdown
46 lines
2.2 KiB
Markdown
# Web Research Tools Stack
|
|
|
|
Configured Jul 7, 2026 for IT Pro Partner infrastructure.
|
|
|
|
## Active Tools
|
|
|
|
| Tool | Type | Cost | Credits | Used for |
|
|
|------|------|------|---------|----------|
|
|
| **Firecrawl** | Web search + content extraction | Free tier | 1,000/mo | Quick web lookups, web_search and web_extract tools |
|
|
| **ScrapingAnt** | JS rendering via headless Chrome | Free tier | 10,000 free (no card) | JS-heavy sites, government pages, Texas Legislature, LinkedIn public pages |
|
|
| **SearXNG** | Self-hosted meta search (Docker) | $0 (runs on Core) | Unlimited | Private search, fallback when other tools unavailable |
|
|
|
|
## Configuration
|
|
|
|
### Firecrawl
|
|
- API key saved in `.env`: `FIRECRAWL_API_KEY=fc-4f9...63c1`
|
|
- Hermes config: `web.backend: firecrawl`, `web.use_gateway: true`
|
|
- Activate on next session /reset after adding key
|
|
- Free tier: 1,000 credits/mo, upgrade to 5K credits planned when needed
|
|
- Usage tracking: cron job `firecrawl-usage-check` runs daily at 9 AM, no_agent
|
|
|
|
### ScrapingAnt
|
|
- API key saved in `.env`: `SCRAPINGANT_API_KEY=8ab7...af6`
|
|
- Used via direct API calls from terminal (`curl` to scrapingant.com/v2/general with browser=true)
|
|
- 10,000 free credits/mo, no credit card required
|
|
- $19/mo for 100K credits (recommended upgrade when free tier runs out)
|
|
|
|
### SearXNG
|
|
- Deployed as Docker container on Core at port 8888 (localhost only)
|
|
- Accessed via `curl "http://127.0.0.1:8888/search?q=...&format=json"`
|
|
- JSON API returns structured results with titles, URLs, snippets
|
|
- Self-hosted, unlimited, no API key needed
|
|
- Auto-starts with Docker (compose file at ~/docker/searxng/)
|
|
|
|
## Multi-Profile Setup
|
|
|
|
When a second Hermes profile (e.g. Anita's `anita` profile) needs web tools:
|
|
1. Copy API keys to the profile's `.env` file
|
|
2. Enable web backend in the profile's `config.yaml`
|
|
3. Restart the profile's gateway to pick up changes
|
|
4. The user needs to /reset their session to see the new tools
|
|
|
|
## Why Both Firecrawl and SearXNG?
|
|
|
|
SearXNG handles unlimited search (finding the right article/result at no cost). Firecrawl handles content extraction (converting specific page URLs to clean markdown). Without SearXNG, Firecrawl would handle both search + extraction but burn credits on searches that SearXNG can do for free.
|