commit 2967e21083abf3c28d49abcd4e5f656a82d852ef Author: root Date: Wed Jul 15 17:32:01 2026 -0400 Initial commit — 2026-07-15 diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..ce93d7e --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,11 @@ +# Changelog - OSINT People Search + +### 2026-07-10 +- **Documentation:** Migrated web scraping API comparison and tool evaluations. + +### 2026-07-02 +- **Research:** Completed comparison of web scraping APIs for OSINT data collection. + +### 2026-06-20 +- **API:** Drafted initial frontend UI and API integration plans. + diff --git a/README.md b/README.md new file mode 100644 index 0000000..a98daac --- /dev/null +++ b/README.md @@ -0,0 +1,11 @@ +# OSINT People Search + +An Open Source Intelligence tool for performing background checks, compiling data broker reports, and removing personal information. + +## Project Info + +- **Owner:** Germaine +- **Status:** IN DEVELOPMENT +- **Deployed:** Local / Testing +- **Key Contacts:** Germaine +- **Changelog:** [CHANGELOG.md](./CHANGELOG.md) diff --git a/research/web-scraping-api-comparison.md b/research/web-scraping-api-comparison.md new file mode 100644 index 0000000..6aa9196 --- /dev/null +++ b/research/web-scraping-api-comparison.md @@ -0,0 +1,241 @@ +# Web Scraping & Search API Comparison + +**Prepared for:** IT Pro Partner (MSP — debt recovery, WISP, trucking, track events) +**Current setup:** Firecrawl Starter (1,000 credits/mo), considering upgrade to 5,000 credits +**Goal:** Find complementary/replacement APIs for government site scraping, Google search, and JS-rendered pages + +--- + +## 1. Tavily — AI-Native Search API +**URL:** https://tavily.com | **Docs:** https://docs.tavily.com + +| Attribute | Detail | +|-----------|--------| +| **Problem it solves** | Real-time web search + content extraction optimized for AI agents/RAG. Complements Firecrawl's crawl/scrape with semantic search. | +| **Free tier** | **Researcher: Free** — 1,000 API credits/month, no credit card | +| **Paid tiers** | **Pay As You Go:** $0.008/credit. **Project:** $30/mo for 4,000 credits. **Enterprise:** Custom. | +| **APIs** | `/search`, `/extract`, `/crawl`, `/map`, `/research` | +| **JS rendering** | ❌ **No headless browser.** Cannot render JavaScript-heavy pages. Returns search-engine-level content. | +| **Rate limits** | Soft per-plan limits, credits-based | +| **API quality** | Excellent REST API with Python/JS SDKs, MCP server, 180ms p50, 99.99% SLA | +| **Best use case** | Grounding AI agents with real-time web search, RAG pipelines, semantic content extraction | +| **Firecrawl gap** | Tavily is a **search** engine, not a JS scraper. Complements Firecrawl for discovery phase. | + +--- + +## 2. SerpAPI — Google/Bing Search Results API +**URL:** https://serpapi.com + +| Attribute | Detail | +|-----------|--------| +| **Problem it solves** | Structured Google/Bing/etc. search results without getting blocked. 50+ specialized endpoints (Maps, Shopping, Flights, Hotels, Jobs, Scholar, Patents, Images, News, etc.). | +| **Free tier** | 100 searches/month | +| **Paid tiers** | Starts at ~$50/mo (5,000 searches). Higher tiers scale up. Enterprise available. | +| **JS rendering** | ✅ Yes — returns rendered search result pages. But **not** a general-purpose page scraper. | +| **API quality** | Mature REST API (since 2015), excellent docs, broad language support, proxy rotation built-in | +| **Best use case** | Getting clean Google/Bing SERP data for SEO, competitor research, local business data | +| **Firecrawl gap** | SerpAPI specializes in **search engine result pages** only. Use it when you need Google Maps reviews, Google Shopping prices, or Google Scholar data. Does NOT scrape arbitrary websites. | + +--- + +## 3. Serper — Google Search API +**URL:** https://serper.dev + +| Attribute | Detail | +|-----------|--------| +| **Problem it solves** | Fast, cheap Google search results. Lighter/faster alternative to SerpAPI. | +| **Free tier** | 2,500 searches/month free | +| **Paid tiers** | Starts at ~$50/mo for 50,000 searches. Scales up. | +| **JS rendering** | ❌ No — returns search results only | +| **API quality** | Simple REST API, very fast (<1s responses), good docs | +| **Best use case** | High-volume Google search lookups on a budget. Much cheaper than SerpAPI per-query. | +| **Firecrawl gap** | Pure search API. Cheaper than SerpAPI for basic Google searches. No page scraping. | + +--- + +## 4. ScrapingBee — Headless Browser Scraping API +**URL:** https://www.scrapingbee.com + +| Attribute | Detail | +|-----------|--------| +| **Problem it solves** | **Full JS rendering** via headless Chrome + rotating proxies + CAPTCHA handling. Direct Firecrawl competitor/complement for JS-heavy sites. | +| **Free tier** | 1,000 API credits (one-time trial, no card) | +| **Paid tiers** | **Freelance:** $49/mo (250K credits). **Startup:** $99/mo (1M credits). **Business:** $249/mo (3M credits). **Business+:** $599/mo (8M credits). **Custom:** Enterprise. | +| **Credit costs** | JS rendering = **5 credits/request**. Premium proxies = **10-25 credits/request**. Basic = 1 credit. | +| **Features** | JS rendering ✅, rotating proxies ✅, geotargeting ✅, screenshots ✅, extraction rules ✅, Google Search API ✅, Amazon/Walmart/YouTube dedicated scrapers ✅ | +| **Concurrency** | Freelance: 50, Startup: 100, Business: 200, Business+: 400 | +| **API quality** | Excellent REST API, well-documented, Python/JS/PHP SDKs. Dedicated scraping APIs for Amazon, Google, YouTube, Walmart, ChatGPT. | +| **Best use case** | JS-heavy government sites (Texas Legislature, court records), e-commerce scraping, any site that blocks simple HTTP requests | +| **Firecrawl gap** | ScrapingBee is a **direct Firecrawl alternative** for JS rendering. Cheaper per-credit than upgrading Firecrawl to 5,000 credits ($50-$100/mo unknown). Startup plan ($99/mo) gives 1M credits vs Firecrawl Starter's 1,000 credits — massive difference in volume. | + +**Verdict:** ✅ **RECOMMENDED** — Best all-around JS rendering alternative to Firecrawl. + +--- + +## 5. ScrapingAnt — Headless Chrome Scraping API +**URL:** https://scrapingant.com + +| Attribute | Detail | +|-----------|--------| +| **Problem it solves** | Headless Chrome + 3M+ rotating proxies + AI extraction + MCP server — all under one credit pool. Direct Firecrawl competitor. | +| **Free tier** | **10,000 credits/month** — no credit card required | +| **Paid tiers** | **100K credits:** $19/mo. **500K credits:** $49/mo. **3M credits:** $249/mo. **8M credits:** $599/mo. **10M+ credits:** $699+/mo. | +| **Credit costs** | Default (Chrome + standard proxy) = **10 credits/request**. Cheaper non-JS requests available. Residential proxies billed separately by GB. | +| **Features** | Headless Chrome ✅, 3M+ rotating proxies ✅ (residential + datacenter), AI data extraction ✅, LLM-ready markdown ✅, MCP server ✅ (Claude/Cursor/Windsurf), custom datasets ✅ | +| **Free tier** | Generous 10K free credits beats almost everyone | +| **API quality** | Clean REST API, MCP server for AI agents, docs at docs.scrapingant.com, Intercom support | +| **Best use case** | JS-heavy government sites, AI agent scraping (MCP integration), high-volume scraping on a budget | +| **Firecrawl gap** | **Direct Firecrawl competitor.** $19/mo for 100K credits vs Firecrawl Starter's 1,000 credits is **100x more volume** for a fraction of the cost. Also includes MCP server for AI agents. | + +**Verdict:** ✅ **STRONGLY RECOMMENDED** — Best value replacement for Firecrawl. 10K free credits, $19/mo for 100K. + +--- + +## 6. Jina AI Reader — URL-to-Markdown API +**URL:** https://jina.ai/reader + +| Attribute | Detail | +|-----------|--------| +| **Problem it solves** | Converts any URL to clean LLM-friendly markdown. Excellent for RAG pipelines and feeding web content to AI models. | +| **Free tier** | **10 million free tokens** to start. Free tier: 100 RPM, 100K TPM, 2 concurrent requests. | +| **Paid tiers** | **Paid:** 500 RPM, 2M TPM, 50 concurrent. **Premium:** 5,000 RPM, 50M TPM, 500 concurrent. Token-based billing. | +| **JS rendering** | ❌ No headless browser — reads server-rendered HTML only | +| **API quality** | Excellent token-based API. Also offers embeddings, reranker. Simple `GET` endpoint. | +| **Best use case** | Converting documentation, articles, and static pages to LLM-ready markdown. RAG content ingestion. | +| **Firecrawl gap** | Jina is **not a scraper** — it's a content transformer. If a page loads fine with HTTP, Jina gives you cleaner output than Firecrawl. But won't help with JS-rendered sites. | + +--- + +## 7. Brave Search API — Independent Search Index +**URL:** https://brave.com/search/api/ + +| Attribute | Detail | +|-----------|--------| +| **Problem it solves** | Google-independent search index with no API quotas like Google's. Own index, less SEO spam. | +| **Free tier** | **$5 in free credits/month** (approx 1,000 requests). Credit card required even for free plan (anti-fraud). | +| **Paid tiers** | **$5/1,000 requests** (standard). **$4/1,000 requests + $5/M tokens** (with AI summary). | +| **JS rendering** | ❌ No — search-only API | +| **API quality** | Clean REST API, websearch endpoint, summary endpoint, image search. Independent index — not just Google/Bing reseller. | +| **Best use case** | Search without Google dependency. Good for "we scrape everything" MSP use cases where you want result diversity. | +| **Firecrawl gap** | Pure search API. The independent index is its main differentiator vs Google-dependent alternatives. Very affordable ($5 covers ~1K searches). | + +--- + +## 8. Apify — Full Scraping Platform +**URL:** https://apify.com + +| Attribute | Detail | +|-----------|--------| +| **Problem it solves** | Full scraping platform with 2,000+ pre-built "Actors" (scrapers for any site). Web IDE, scheduling, proxy management, and compute platform all-in-one. | +| **Free tier** | **$5 free usage/month** ($0.20/compute unit), no credit card | +| **Paid tiers** | **Starter:** $29/mo ($29 spend). **Scale:** $199/mo ($199 spend). **Business:** $999/mo ($999 spend). Compute unit pricing drops at higher tiers ($0.20 → $0.16 → $0.13). | +| **JS rendering** | ✅ Yes — full headless Chrome via Crawlee/Playwright actors | +| **Features** | 2,000+ pre-built Actors ✅ (Website Content Crawler, Google scraper, LinkedIn scraper, Instagram, etc.), scheduling ✅, proxy rotation ✅, Crawlee library ✅, MCP server ✅, webhooks ✅ | +| **API quality** | Excellent. REST API, Crawlee SDK (Python/JS), webhooks, MCP integration. Open platform. | +| **Best use case** | "Set and forget" scraping. Use pre-built Actors for common sites. Build custom Actors for niche sites. Good for recurring scraping jobs. | +| **Firecrawl gap** | Apify is a **platform**, not just an API. The pre-built Actors mean you can scrape LinkedIn, Instagram, Google Maps etc. without writing code. More complex but far more powerful for ongoing scraping operations. | + +**Verdict:** ✅ **RECOMMENDED** — Best for recurring, complex scraping workflows with pre-built Actors. + +--- + +## 9. Browserbase — Cloud Browser Automation +**URL:** https://www.browserbase.com + +| Attribute | Detail | +|-----------|--------| +| **Problem it solves** | Cloud-hosted headless browsers with AI-powered automation (Stagehand SDK). Playwright/Puppeteer/Selenium compatible. Full browser control in the cloud. | +| **Free tier** | **Developer (free):** 3 agent runs/mo, 1,000 Fetch API calls, 1,000 Search API calls, $5 in model tokens, 7-day data retention | +| **Paid tiers** | **Startup:** 15 agent runs. **Team:** 50 agent runs. **Enterprise:** Custom. Fetch API: $1/1k calls (basic), $4/1k (with proxies). Extract: $4/1k, $7/1k (with proxies). Search: $7/1k requests. Proxies: $12/GB (overage). | +| **JS rendering** | ✅ **Yes** — full Playwright/Puppeteer control. This is a full browser in the cloud. | +| **Features** | Captcha solving ✅, Model Gateway ✅ (benchmark models against web workflows), stealth mode ✅, session recording ✅, custom proxies ✅, SOC 2 ✅, Stagehand AI SDK ✅ | +| **API quality** | Excellent. REST API + WebSocket, Stagehand SDK (TS/JS), connects with LangChain/CrewAI/Mastra. | +| **Best use case** | Agentic browser automation — when you need to **interact** with sites (click, fill forms, login). Government portal workflows. AI agent web tasks. | +| **Firecrawl gap** | Browserbase is **not a scraping API** — it's a **cloud browser**. Use it when you need to interact with JS-heavy sites dynamically (e.g., submit court record search forms, navigate multi-step government portals). Firecrawl can't do this at all. | + +**Verdict:** ✅ **RECOMMENDED** — Best for interactive browser automation (form fills, logins, multi-step government site workflows). + +--- + +## Comparison Matrix + +| Service | Free Tier | Entry Price | Volume (entry) | JS Rendering | Search | Scraping | Best For | +|---------|-----------|------------|----------------|-------------|--------|----------|----------| +| **Firecrawl** (current) | 500 credits | ? | 1,000/mo | ✅ | ✅ | ✅ | General web data — your existing pipeline | +| **Tavily** | 1,000 credits | $0.008/credit | 1,000/mo free | ❌ | ✅ | ✅ | AI agent search/RAG | +| **SerpAPI** | 100 searches | ~$50/mo | 5,000/mo | ✅ (SERP) | ✅ | ❌ | Google SERP data | +| **Serper** | 2,500 searches | ~$50/mo | 50,000/mo | ❌ | ✅ | ❌ | High-volume Google search | +| **ScrapingBee** | 1,000 credits | $49/mo | 250K/mo | ✅ (5cr/req) | ✅ | ✅ | **JS rendering — best Firecrawl alternative** | +| **ScrapingAnt** | **10,000 credits** | **$19/mo** | **100K/mo** | ✅ (10cr/req) | ✅ | ✅ | **Best value Firecrawl replacement** | +| **Jina AI Reader** | 10M tokens | Token-based | Free tier generous | ❌ | ❌ | ✅ (limited) | URL→Markdown for LLMs | +| **Brave Search** | $5/mo credit | $5/1K req | ~1K/mo free | ❌ | ✅ | ❌ | Independent search, low cost | +| **Apify** | $5 usage | $29/mo | Platform pricing | ✅ | ✅ | ✅ | Recurring/complex scraping workflows | +| **Browserbase** | 3 agent runs | Usage-based | 1K fetch free | ✅ **Full browser** | ✅ | ✅ | **Interactive browser automation** | + +--- + +## Strategic Recommendations + +### Tier 1: Critical Additions — Add Now + +**1. ScrapingAnt** — **Primary Firecrawl complement/replacement** +- **Why:** $19/mo for 100K credits vs Firecrawl's ~$50-100 for 5,000. 10,000 free credits/month to try. Headless Chrome + rotating proxies handle any JS-heavy government site. MCP server for AI agents is a bonus. +- **Use for:** Texas Legislature scraping, court records, any JS-heavy government/agency site +- **Cost:** Free to start, $19/mo for 100K credits + +**2. Browserbase** — **Interactive government portal automation** +- **Why:** Government sites often need form submission (search case number, enter parameters) — not just page loading. Browserbase gives full Playwright control in the cloud with captcha solving and stealth mode. +- **Use for:** Court record searches that require form submission, multi-step government workflows, login-gated agency portals +- **Cost:** Free tier (3 agent runs, 1K fetch) to start + +### Tier 2: Add When Needed + +**3. Brave Search API** — **Independent web search** +- **Why:** $5/mo free credits covers ~1K searches. Independent index avoids Google dependency. Complements Tavily/Firecrawl for content discovery. +- **Use for:** General web search for MSP research, finding resources across your business lines +- **Cost:** ~$5/mo effectively free + +**4. Apify** — **Recurring/complex scraping** +- **Why:** Pre-built Actors for LinkedIn, Google Maps, Instagram, etc. If you need to scrape these platforms regularly, Apify saves massive development time. +- **Use for:** Regular monitoring of competitors, directory scraping, social media data +- **Cost:** $29/mo to start + +### Tier 3: Nice-to-Have + +**5. Tavily** — **AI search (if you're building AI agents)** +- **Why:** Excellent for RAG/agent search. Free tier covers evaluation. If you're not building AI agents yet, skip this. +- **Use for:** AI-powered search within your ops portal +- **Cost:** Free tier available + +**6. Jina AI Reader** — **Clean markdown for LLMs** +- **Why:** If you're feeding scraped content into LLMs, Jina's markdown output is cleaner than raw HTML. Only useful for non-JS pages. +- **Use for:** LLM content ingestion pipeline +- **Cost:** 10M free tokens to start + +--- + +## Cost-Benefit Analysis: Firecrawl Upgrade vs Alternatives + +| Scenario | Firecrawl Upgrade (5K credits) | ScrapingAnt (100K credits) | Savings | +|----------|-------------------------------|---------------------------|---------| +| **Monthly volume** | 5,000 credits | 100,000 credits | **20x more volume** | +| **Estimated cost** | ~$50-100/mo | **$19/mo** | **60-80% cheaper** | +| **JS rendering** | ✅ | ✅ (Chrome included) | Same | +| **Free trial** | None mentioned | **10,000 credits free** | Risk-free trial | + +**Bottom line:** Adding ScrapingAnt ($19/mo for 100K credits) as a Firecrawl supplement gives you 20x more scraping capacity for less money. The 10K free credits let you test with zero risk. + +--- + +## Use Case Quick Reference + +| Use Case | Recommended API | Why | +|----------|----------------|-----| +| **Texas Legislature bill tracking** | ScrapingAnt or ScrapingBee | JS-heavy legislative sites need headless Chrome | +| **Court record searches (forms)** | Browserbase | Need to submit forms, handle redirects, solve CAPTCHAs | +| **Google Maps/MAPS data** | SerpAPI or Apify | Specialized Actors handle Maps data structure | +| **General web research for MSP** | Brave Search API | Independent index, $5/mo free | +| **RAG/LLM content ingestion** | Jina AI Reader | Cleanest markdown output for AI consumption | +| **High-volume Google searches** | Serper | Cheapest per-query Google search | +| **Recurring social media scraping** | Apify | Pre-built Actors for Instagram, LinkedIn, TikTok | +| **Simple page scraping (non-JS)** | Firecrawl (keep current) | Already works for basic sites |