hermes-skills/skills/software-development/debt-recovery-platform/references/web-scraping-stack.md

Web Scraping & Search Stack

Current Services (July 2026)

Service	Credits/mo	JS Rendering	Cost	Best For
Firecrawl	1,000	Limited	Free	Quick web lookups, markdown extraction
ScrapingAnt	10,000 free / 100K paid	✅ Headless Chrome	$0 (free) / $19/mo (paid)	JS-heavy sites, gov pages, LinkedIn, Indeed
SearXNG	Unlimited	❌ No	Free (self-hosted)	Metadata search, query discovery

API Keys

• Firecrawl: fc-4f9...63c1 — saved in ~/.hermes/.env as FIRECRAWL_API_KEY
• ScrapingAnt: 8ab7...af6 — saved in ~/.hermes/.env as SCRAPINGANT_API_KEY
• SearXNG runs at http://127.0.0.1:8888 on Core (Docker, auto-start)

When to Use Which

Target	Tool	Why
General web page / article	Firecrawl (free)	Fast, clean markdown
JS-rendered gov site (TX Legislature)	ScrapingAnt + browser=true	Headless Chrome renders JS SPAs
LinkedIn public profiles	ScrapingAnt	Handles JS auth walls
Indeed / CBDriver / job sites	ScrapingAnt	Blocks raw HTTP scrapers
People search / skip tracing	ScrapingAnt + Sherlock	Combined: OSINT + JS rendering
Quick URL lookup (curl-friendly)	SearXNG	Unlimited, no API key needed

Credit Tracking

Daily cron at 8 AM: model-usage-tracker.sh also checks Firecrawl credits via header response.

Self-Hosted Search (SearXNG)

• Docker container at /root/docker/searxng/
• JSON API at http://127.0.0.1:8888/search?q=<query>&format=json
• Limited to 120 req/hr (configurable in settings.yml)
• Only accessible from localhost (not exposed externally)

Future Additions (Research Complete)

• Apify LinkedIn actor (~$30-50/mo) — structured LinkedIn people search for skip tracing
• Browserbase — interactive gov portal automation (form submission, login-gated searches)