Web Research & Scraping Stack

Configured Jul 7, 2026. Three-tier search/scraping capability for Hermes and Anita's profile.

Current Stack

Firecrawl (native web tools)

API Key: fc-4f9...63c1 (in ~/.hermes/.env)
Plan: Starter (1,000 credits/mo free)
Endpoint: api.firecrawl.dev
Purpose: Native web_search and web_extract tools in Hermes. Quick content extraction from URLs.
Auth: Bearer token in Authorization header
Credits used per call: 1 (cache hit) to 1+ (fresh scrape)
Credit tracking: ~/.hermes/scripts/firecrawl-usage.json
Upgrade option: ~$50-100/mo for 5,000 credits (not cost-effective vs ScrapingAnt — 20x less volume at higher cost)

ScrapingAnt (JS-heavy sites)

API Key: 8ab7...4af6 (in ~/.hermes/.env)
Plan: Free tier (10,000 credits) → $19/mo for 100K+
Purpose: JS-rendered pages, government sites (Texas Legislature), LinkedIn, Indeed, CBDriver
Endpoint: api.scrapingant.com/v2/general?url=...&x-api-key=... (with optional &browser=true for JS render)
Features: Headless Chrome, browser=true param for JS rendering
Cost per request: varies by JS render usage

SearXNG (self-hosted, unlimited)

URL: http://127.0.0.1:8888
Purpose: Private web search, unlimited queries, multi-engine aggregation
Format: JSON API: http://127.0.0.1:8888/search?q=<query>&format=json
Deployment: Docker container on Core (port 8888, localhost only)
Limits: 120 req/min

Anita's Profile Access

Anita's profile at ~/.hermes/profiles/anita/ was configured with:

.env file with FIRECRAWL_API_KEY, SCRAPINGANT_API_KEY, SEARXNG_BASE_URL
web.backend: firecrawl in config.yaml
web toolset enabled
Gateway restart required to pick up new config

Use Cases

Site	Tool	Method
Government (TX Leg., court records)	ScrapingAnt	`browser=true` for JS rendering
CBDriver, Indeed	ScrapingAnt	Headless Chrome handles JS
LinkedIn (public profiles)	ScrapingAnt	Can pull public pages
LinkedIn (deep search)	Apify LinkedIn actor (~$30-50/mo)	Handles auth, rate limits, structured data
General web search	SearXNG (self-hosted)	Unlimited, private
Quick URL extract	Firecrawl (native tool)	Free credits
Debtor skip tracing	Sherlock + theHarvester	Username search / domain recon

OSINT Tools (installed on Core)

Sherlock (/opt/sherlock/) — search 400+ sites by username. cd /opt/sherlock && ./venv/bin/python sherlock <username>
theHarvester (/opt/theharvester/) — email/domain recon. cd /opt/theharvester && ./venv/bin/python theHarvester -d <domain> -b google

Usage Tracking

~/.hermes/scripts/track-firecrawl.py tracks monthly usage across both services. Cron: firecrawl-usage-check at 9 AM daily (no_agent).

Cost Recommendation

Keep Firecrawl free tier for quick lookups (1K/mo free). Add ScrapingAnt $19/mo for JS-heavy sites (100K credits). Skip Firecrawl upgrade entirely. Add Apify LinkedIn actor (~$30-50/mo) only when deep LinkedIn people search is needed for DRE skip tracing.

3.1 KiB Raw Blame History