# Web Scraping & Search Stack

## Current Services (July 2026)

| Service | Credits/mo | JS Rendering | Cost | Best For |
|---------|-----------|-------------|------|----------|
| **Firecrawl** | 1,000 | Limited | Free | Quick web lookups, markdown extraction |
| **ScrapingAnt** | 10,000 free / 100K paid | ✅ Headless Chrome | $0 (free) / $19/mo (paid) | JS-heavy sites, gov pages, LinkedIn, Indeed |
| **SearXNG** | Unlimited | ❌ No | Free (self-hosted) | Metadata search, query discovery |

## API Keys

- Firecrawl: `fc-4f9...63c1` — saved in ~/.hermes/.env as `FIRECRAWL_API_KEY`
- ScrapingAnt: `8ab7...af6` — saved in ~/.hermes/.env as `SCRAPINGANT_API_KEY`
- SearXNG runs at `http://127.0.0.1:8888` on Core (Docker, auto-start)

## When to Use Which

| Target | Tool | Why |
|--------|------|-----|
| General web page / article | Firecrawl (free) | Fast, clean markdown |
| JS-rendered gov site (TX Legislature) | ScrapingAnt + browser=true | Headless Chrome renders JS SPAs |
| LinkedIn public profiles | ScrapingAnt | Handles JS auth walls |
| Indeed / CBDriver / job sites | ScrapingAnt | Blocks raw HTTP scrapers |
| People search / skip tracing | ScrapingAnt + Sherlock | Combined: OSINT + JS rendering |
| Quick URL lookup (curl-friendly) | SearXNG | Unlimited, no API key needed |

## Credit Tracking

Daily cron at 8 AM: `model-usage-tracker.sh` also checks Firecrawl credits via header response.

## Self-Hosted Search (SearXNG)

- Docker container at `/root/docker/searxng/`
- JSON API at `http://127.0.0.1:8888/search?q=<query>&format=json`
- Limited to 120 req/hr (configurable in settings.yml)
- Only accessible from localhost (not exposed externally)

## Future Additions (Research Complete)

- **Apify LinkedIn actor** (~$30-50/mo) — structured LinkedIn people search for skip tracing
- **Browserbase** — interactive gov portal automation (form submission, login-gated searches)