Initial skills documentation — 25 categories, all SKILL.md + references + scripts

2026-07-15 17:42:20 -04:00
commit 1b4fcd4427
976 changed files with 188344 additions and 0 deletions
@@ -0,0 +1,64 @@
+# Web Research & Scraping Stack
+
+Configured Jul 7, 2026. Three-tier search/scraping capability for Hermes and Anita's profile.
+
+## Current Stack
+
+### Firecrawl (native web tools)
+- **API Key:** fc-4f9...63c1 (in ~/.hermes/.env)
+- **Plan:** Starter (1,000 credits/mo free)
+- **Endpoint:** api.firecrawl.dev
+- **Purpose:** Native `web_search` and `web_extract` tools in Hermes. Quick content extraction from URLs.
+- **Auth:** Bearer token in Authorization header
+- **Credits used per call:** 1 (cache hit) to 1+ (fresh scrape)
+- **Credit tracking:** ~/.hermes/scripts/firecrawl-usage.json
+- **Upgrade option:** ~$50-100/mo for 5,000 credits (not cost-effective vs ScrapingAnt — 20x less volume at higher cost)
+
+### ScrapingAnt (JS-heavy sites)
+- **API Key:** 8ab7...4af6 (in ~/.hermes/.env)
+- **Plan:** Free tier (10,000 credits) → $19/mo for 100K+
+- **Purpose:** JS-rendered pages, government sites (Texas Legislature), LinkedIn, Indeed, CBDriver
+- **Endpoint:** api.scrapingant.com/v2/general?url=...&x-api-key=... (with optional &browser=true for JS render)
+- **Features:** Headless Chrome, `browser=true` param for JS rendering
+- **Cost per request:** varies by JS render usage
+
+### SearXNG (self-hosted, unlimited)
+- **URL:** http://127.0.0.1:8888
+- **Purpose:** Private web search, unlimited queries, multi-engine aggregation
+- **Format:** JSON API: `http://127.0.0.1:8888/search?q=<query>&format=json`
+- **Deployment:** Docker container on Core (port 8888, localhost only)
+- **Limits:** 120 req/min
+
+## Anita's Profile Access
+
+Anita's profile at ~/.hermes/profiles/anita/ was configured with:
+- `.env` file with FIRECRAWL_API_KEY, SCRAPINGANT_API_KEY, SEARXNG_BASE_URL
+- `web.backend: firecrawl` in config.yaml
+- `web` toolset enabled
+- Gateway restart required to pick up new config
+
+## Use Cases
+
+| Site | Tool | Method |
+|------|------|--------|
+| Government (TX Leg., court records) | ScrapingAnt | `browser=true` for JS rendering |
+| CBDriver, Indeed | ScrapingAnt | Headless Chrome handles JS |
+| LinkedIn (public profiles) | ScrapingAnt | Can pull public pages |
+| LinkedIn (deep search) | Apify LinkedIn actor (~$30-50/mo) | Handles auth, rate limits, structured data |
+| General web search | SearXNG (self-hosted) | Unlimited, private |
+| Quick URL extract | Firecrawl (native tool) | Free credits |
+| Debtor skip tracing | Sherlock + theHarvester | Username search / domain recon |
+
+## OSINT Tools (installed on Core)
+
+- **Sherlock** (`/opt/sherlock/`) — search 400+ sites by username. `cd /opt/sherlock && ./venv/bin/python sherlock <username>`
+- **theHarvester** (`/opt/theharvester/`) — email/domain recon. `cd /opt/theharvester && ./venv/bin/python theHarvester -d <domain> -b google`
+
+## Usage Tracking
+
+`~/.hermes/scripts/track-firecrawl.py` tracks monthly usage across both services.
+Cron: `firecrawl-usage-check` at 9 AM daily (no_agent).
+
+## Cost Recommendation
+
+Keep Firecrawl free tier for quick lookups (1K/mo free). Add ScrapingAnt $19/mo for JS-heavy sites (100K credits). Skip Firecrawl upgrade entirely. Add Apify LinkedIn actor (~$30-50/mo) only when deep LinkedIn people search is needed for DRE skip tracing.