43 lines
1.9 KiB
Markdown
43 lines
1.9 KiB
Markdown
# Web Scraping & Search Stack
|
|
|
|
## Current Services (July 2026)
|
|
|
|
| Service | Credits/mo | JS Rendering | Cost | Best For |
|
|
|---------|-----------|-------------|------|----------|
|
|
| **Firecrawl** | 1,000 | Limited | Free | Quick web lookups, markdown extraction |
|
|
| **ScrapingAnt** | 10,000 free / 100K paid | ✅ Headless Chrome | $0 (free) / $19/mo (paid) | JS-heavy sites, gov pages, LinkedIn, Indeed |
|
|
| **SearXNG** | Unlimited | ❌ No | Free (self-hosted) | Metadata search, query discovery |
|
|
|
|
## API Keys
|
|
|
|
- Firecrawl: `fc-4f9...63c1` — saved in ~/.hermes/.env as `FIRECRAWL_API_KEY`
|
|
- ScrapingAnt: `8ab7...af6` — saved in ~/.hermes/.env as `SCRAPINGANT_API_KEY`
|
|
- SearXNG runs at `http://127.0.0.1:8888` on Core (Docker, auto-start)
|
|
|
|
## When to Use Which
|
|
|
|
| Target | Tool | Why |
|
|
|--------|------|-----|
|
|
| General web page / article | Firecrawl (free) | Fast, clean markdown |
|
|
| JS-rendered gov site (TX Legislature) | ScrapingAnt + browser=true | Headless Chrome renders JS SPAs |
|
|
| LinkedIn public profiles | ScrapingAnt | Handles JS auth walls |
|
|
| Indeed / CBDriver / job sites | ScrapingAnt | Blocks raw HTTP scrapers |
|
|
| People search / skip tracing | ScrapingAnt + Sherlock | Combined: OSINT + JS rendering |
|
|
| Quick URL lookup (curl-friendly) | SearXNG | Unlimited, no API key needed |
|
|
|
|
## Credit Tracking
|
|
|
|
Daily cron at 8 AM: `model-usage-tracker.sh` also checks Firecrawl credits via header response.
|
|
|
|
## Self-Hosted Search (SearXNG)
|
|
|
|
- Docker container at `/root/docker/searxng/`
|
|
- JSON API at `http://127.0.0.1:8888/search?q=<query>&format=json`
|
|
- Limited to 120 req/hr (configurable in settings.yml)
|
|
- Only accessible from localhost (not exposed externally)
|
|
|
|
## Future Additions (Research Complete)
|
|
|
|
- **Apify LinkedIn actor** (~$30-50/mo) — structured LinkedIn people search for skip tracing
|
|
- **Browserbase** — interactive gov portal automation (form submission, login-gated searches)
|