65 lines
3.1 KiB
Markdown
65 lines
3.1 KiB
Markdown
# Web Research & Scraping Stack
|
|
|
|
Configured Jul 7, 2026. Three-tier search/scraping capability for Hermes and Anita's profile.
|
|
|
|
## Current Stack
|
|
|
|
### Firecrawl (native web tools)
|
|
- **API Key:** fc-4f9...63c1 (in ~/.hermes/.env)
|
|
- **Plan:** Starter (1,000 credits/mo free)
|
|
- **Endpoint:** api.firecrawl.dev
|
|
- **Purpose:** Native `web_search` and `web_extract` tools in Hermes. Quick content extraction from URLs.
|
|
- **Auth:** Bearer token in Authorization header
|
|
- **Credits used per call:** 1 (cache hit) to 1+ (fresh scrape)
|
|
- **Credit tracking:** ~/.hermes/scripts/firecrawl-usage.json
|
|
- **Upgrade option:** ~$50-100/mo for 5,000 credits (not cost-effective vs ScrapingAnt — 20x less volume at higher cost)
|
|
|
|
### ScrapingAnt (JS-heavy sites)
|
|
- **API Key:** 8ab7...4af6 (in ~/.hermes/.env)
|
|
- **Plan:** Free tier (10,000 credits) → $19/mo for 100K+
|
|
- **Purpose:** JS-rendered pages, government sites (Texas Legislature), LinkedIn, Indeed, CBDriver
|
|
- **Endpoint:** api.scrapingant.com/v2/general?url=...&x-api-key=... (with optional &browser=true for JS render)
|
|
- **Features:** Headless Chrome, `browser=true` param for JS rendering
|
|
- **Cost per request:** varies by JS render usage
|
|
|
|
### SearXNG (self-hosted, unlimited)
|
|
- **URL:** http://127.0.0.1:8888
|
|
- **Purpose:** Private web search, unlimited queries, multi-engine aggregation
|
|
- **Format:** JSON API: `http://127.0.0.1:8888/search?q=<query>&format=json`
|
|
- **Deployment:** Docker container on Core (port 8888, localhost only)
|
|
- **Limits:** 120 req/min
|
|
|
|
## Anita's Profile Access
|
|
|
|
Anita's profile at ~/.hermes/profiles/anita/ was configured with:
|
|
- `.env` file with FIRECRAWL_API_KEY, SCRAPINGANT_API_KEY, SEARXNG_BASE_URL
|
|
- `web.backend: firecrawl` in config.yaml
|
|
- `web` toolset enabled
|
|
- Gateway restart required to pick up new config
|
|
|
|
## Use Cases
|
|
|
|
| Site | Tool | Method |
|
|
|------|------|--------|
|
|
| Government (TX Leg., court records) | ScrapingAnt | `browser=true` for JS rendering |
|
|
| CBDriver, Indeed | ScrapingAnt | Headless Chrome handles JS |
|
|
| LinkedIn (public profiles) | ScrapingAnt | Can pull public pages |
|
|
| LinkedIn (deep search) | Apify LinkedIn actor (~$30-50/mo) | Handles auth, rate limits, structured data |
|
|
| General web search | SearXNG (self-hosted) | Unlimited, private |
|
|
| Quick URL extract | Firecrawl (native tool) | Free credits |
|
|
| Debtor skip tracing | Sherlock + theHarvester | Username search / domain recon |
|
|
|
|
## OSINT Tools (installed on Core)
|
|
|
|
- **Sherlock** (`/opt/sherlock/`) — search 400+ sites by username. `cd /opt/sherlock && ./venv/bin/python sherlock <username>`
|
|
- **theHarvester** (`/opt/theharvester/`) — email/domain recon. `cd /opt/theharvester && ./venv/bin/python theHarvester -d <domain> -b google`
|
|
|
|
## Usage Tracking
|
|
|
|
`~/.hermes/scripts/track-firecrawl.py` tracks monthly usage across both services.
|
|
Cron: `firecrawl-usage-check` at 9 AM daily (no_agent).
|
|
|
|
## Cost Recommendation
|
|
|
|
Keep Firecrawl free tier for quick lookups (1K/mo free). Add ScrapingAnt $19/mo for JS-heavy sites (100K credits). Skip Firecrawl upgrade entirely. Add Apify LinkedIn actor (~$30-50/mo) only when deep LinkedIn people search is needed for DRE skip tracing.
|