Initial commit — 2026-07-15
This commit is contained in:
root
@@ -0,0 +1,241 @@
|
||||
# Web Scraping & Search API Comparison
|
||||
|
||||
**Prepared for:** IT Pro Partner (MSP — debt recovery, WISP, trucking, track events)
|
||||
**Current setup:** Firecrawl Starter (1,000 credits/mo), considering upgrade to 5,000 credits
|
||||
**Goal:** Find complementary/replacement APIs for government site scraping, Google search, and JS-rendered pages
|
||||
|
||||
---
|
||||
|
||||
## 1. Tavily — AI-Native Search API
|
||||
**URL:** https://tavily.com | **Docs:** https://docs.tavily.com
|
||||
|
||||
| Attribute | Detail |
|
||||
|-----------|--------|
|
||||
| **Problem it solves** | Real-time web search + content extraction optimized for AI agents/RAG. Complements Firecrawl's crawl/scrape with semantic search. |
|
||||
| **Free tier** | **Researcher: Free** — 1,000 API credits/month, no credit card |
|
||||
| **Paid tiers** | **Pay As You Go:** $0.008/credit. **Project:** $30/mo for 4,000 credits. **Enterprise:** Custom. |
|
||||
| **APIs** | `/search`, `/extract`, `/crawl`, `/map`, `/research` |
|
||||
| **JS rendering** | ❌ **No headless browser.** Cannot render JavaScript-heavy pages. Returns search-engine-level content. |
|
||||
| **Rate limits** | Soft per-plan limits, credits-based |
|
||||
| **API quality** | Excellent REST API with Python/JS SDKs, MCP server, 180ms p50, 99.99% SLA |
|
||||
| **Best use case** | Grounding AI agents with real-time web search, RAG pipelines, semantic content extraction |
|
||||
| **Firecrawl gap** | Tavily is a **search** engine, not a JS scraper. Complements Firecrawl for discovery phase. |
|
||||
|
||||
---
|
||||
|
||||
## 2. SerpAPI — Google/Bing Search Results API
|
||||
**URL:** https://serpapi.com
|
||||
|
||||
| Attribute | Detail |
|
||||
|-----------|--------|
|
||||
| **Problem it solves** | Structured Google/Bing/etc. search results without getting blocked. 50+ specialized endpoints (Maps, Shopping, Flights, Hotels, Jobs, Scholar, Patents, Images, News, etc.). |
|
||||
| **Free tier** | 100 searches/month |
|
||||
| **Paid tiers** | Starts at ~$50/mo (5,000 searches). Higher tiers scale up. Enterprise available. |
|
||||
| **JS rendering** | ✅ Yes — returns rendered search result pages. But **not** a general-purpose page scraper. |
|
||||
| **API quality** | Mature REST API (since 2015), excellent docs, broad language support, proxy rotation built-in |
|
||||
| **Best use case** | Getting clean Google/Bing SERP data for SEO, competitor research, local business data |
|
||||
| **Firecrawl gap** | SerpAPI specializes in **search engine result pages** only. Use it when you need Google Maps reviews, Google Shopping prices, or Google Scholar data. Does NOT scrape arbitrary websites. |
|
||||
|
||||
---
|
||||
|
||||
## 3. Serper — Google Search API
|
||||
**URL:** https://serper.dev
|
||||
|
||||
| Attribute | Detail |
|
||||
|-----------|--------|
|
||||
| **Problem it solves** | Fast, cheap Google search results. Lighter/faster alternative to SerpAPI. |
|
||||
| **Free tier** | 2,500 searches/month free |
|
||||
| **Paid tiers** | Starts at ~$50/mo for 50,000 searches. Scales up. |
|
||||
| **JS rendering** | ❌ No — returns search results only |
|
||||
| **API quality** | Simple REST API, very fast (<1s responses), good docs |
|
||||
| **Best use case** | High-volume Google search lookups on a budget. Much cheaper than SerpAPI per-query. |
|
||||
| **Firecrawl gap** | Pure search API. Cheaper than SerpAPI for basic Google searches. No page scraping. |
|
||||
|
||||
---
|
||||
|
||||
## 4. ScrapingBee — Headless Browser Scraping API
|
||||
**URL:** https://www.scrapingbee.com
|
||||
|
||||
| Attribute | Detail |
|
||||
|-----------|--------|
|
||||
| **Problem it solves** | **Full JS rendering** via headless Chrome + rotating proxies + CAPTCHA handling. Direct Firecrawl competitor/complement for JS-heavy sites. |
|
||||
| **Free tier** | 1,000 API credits (one-time trial, no card) |
|
||||
| **Paid tiers** | **Freelance:** $49/mo (250K credits). **Startup:** $99/mo (1M credits). **Business:** $249/mo (3M credits). **Business+:** $599/mo (8M credits). **Custom:** Enterprise. |
|
||||
| **Credit costs** | JS rendering = **5 credits/request**. Premium proxies = **10-25 credits/request**. Basic = 1 credit. |
|
||||
| **Features** | JS rendering ✅, rotating proxies ✅, geotargeting ✅, screenshots ✅, extraction rules ✅, Google Search API ✅, Amazon/Walmart/YouTube dedicated scrapers ✅ |
|
||||
| **Concurrency** | Freelance: 50, Startup: 100, Business: 200, Business+: 400 |
|
||||
| **API quality** | Excellent REST API, well-documented, Python/JS/PHP SDKs. Dedicated scraping APIs for Amazon, Google, YouTube, Walmart, ChatGPT. |
|
||||
| **Best use case** | JS-heavy government sites (Texas Legislature, court records), e-commerce scraping, any site that blocks simple HTTP requests |
|
||||
| **Firecrawl gap** | ScrapingBee is a **direct Firecrawl alternative** for JS rendering. Cheaper per-credit than upgrading Firecrawl to 5,000 credits ($50-$100/mo unknown). Startup plan ($99/mo) gives 1M credits vs Firecrawl Starter's 1,000 credits — massive difference in volume. |
|
||||
|
||||
**Verdict:** ✅ **RECOMMENDED** — Best all-around JS rendering alternative to Firecrawl.
|
||||
|
||||
---
|
||||
|
||||
## 5. ScrapingAnt — Headless Chrome Scraping API
|
||||
**URL:** https://scrapingant.com
|
||||
|
||||
| Attribute | Detail |
|
||||
|-----------|--------|
|
||||
| **Problem it solves** | Headless Chrome + 3M+ rotating proxies + AI extraction + MCP server — all under one credit pool. Direct Firecrawl competitor. |
|
||||
| **Free tier** | **10,000 credits/month** — no credit card required |
|
||||
| **Paid tiers** | **100K credits:** $19/mo. **500K credits:** $49/mo. **3M credits:** $249/mo. **8M credits:** $599/mo. **10M+ credits:** $699+/mo. |
|
||||
| **Credit costs** | Default (Chrome + standard proxy) = **10 credits/request**. Cheaper non-JS requests available. Residential proxies billed separately by GB. |
|
||||
| **Features** | Headless Chrome ✅, 3M+ rotating proxies ✅ (residential + datacenter), AI data extraction ✅, LLM-ready markdown ✅, MCP server ✅ (Claude/Cursor/Windsurf), custom datasets ✅ |
|
||||
| **Free tier** | Generous 10K free credits beats almost everyone |
|
||||
| **API quality** | Clean REST API, MCP server for AI agents, docs at docs.scrapingant.com, Intercom support |
|
||||
| **Best use case** | JS-heavy government sites, AI agent scraping (MCP integration), high-volume scraping on a budget |
|
||||
| **Firecrawl gap** | **Direct Firecrawl competitor.** $19/mo for 100K credits vs Firecrawl Starter's 1,000 credits is **100x more volume** for a fraction of the cost. Also includes MCP server for AI agents. |
|
||||
|
||||
**Verdict:** ✅ **STRONGLY RECOMMENDED** — Best value replacement for Firecrawl. 10K free credits, $19/mo for 100K.
|
||||
|
||||
---
|
||||
|
||||
## 6. Jina AI Reader — URL-to-Markdown API
|
||||
**URL:** https://jina.ai/reader
|
||||
|
||||
| Attribute | Detail |
|
||||
|-----------|--------|
|
||||
| **Problem it solves** | Converts any URL to clean LLM-friendly markdown. Excellent for RAG pipelines and feeding web content to AI models. |
|
||||
| **Free tier** | **10 million free tokens** to start. Free tier: 100 RPM, 100K TPM, 2 concurrent requests. |
|
||||
| **Paid tiers** | **Paid:** 500 RPM, 2M TPM, 50 concurrent. **Premium:** 5,000 RPM, 50M TPM, 500 concurrent. Token-based billing. |
|
||||
| **JS rendering** | ❌ No headless browser — reads server-rendered HTML only |
|
||||
| **API quality** | Excellent token-based API. Also offers embeddings, reranker. Simple `GET` endpoint. |
|
||||
| **Best use case** | Converting documentation, articles, and static pages to LLM-ready markdown. RAG content ingestion. |
|
||||
| **Firecrawl gap** | Jina is **not a scraper** — it's a content transformer. If a page loads fine with HTTP, Jina gives you cleaner output than Firecrawl. But won't help with JS-rendered sites. |
|
||||
|
||||
---
|
||||
|
||||
## 7. Brave Search API — Independent Search Index
|
||||
**URL:** https://brave.com/search/api/
|
||||
|
||||
| Attribute | Detail |
|
||||
|-----------|--------|
|
||||
| **Problem it solves** | Google-independent search index with no API quotas like Google's. Own index, less SEO spam. |
|
||||
| **Free tier** | **$5 in free credits/month** (approx 1,000 requests). Credit card required even for free plan (anti-fraud). |
|
||||
| **Paid tiers** | **$5/1,000 requests** (standard). **$4/1,000 requests + $5/M tokens** (with AI summary). |
|
||||
| **JS rendering** | ❌ No — search-only API |
|
||||
| **API quality** | Clean REST API, websearch endpoint, summary endpoint, image search. Independent index — not just Google/Bing reseller. |
|
||||
| **Best use case** | Search without Google dependency. Good for "we scrape everything" MSP use cases where you want result diversity. |
|
||||
| **Firecrawl gap** | Pure search API. The independent index is its main differentiator vs Google-dependent alternatives. Very affordable ($5 covers ~1K searches). |
|
||||
|
||||
---
|
||||
|
||||
## 8. Apify — Full Scraping Platform
|
||||
**URL:** https://apify.com
|
||||
|
||||
| Attribute | Detail |
|
||||
|-----------|--------|
|
||||
| **Problem it solves** | Full scraping platform with 2,000+ pre-built "Actors" (scrapers for any site). Web IDE, scheduling, proxy management, and compute platform all-in-one. |
|
||||
| **Free tier** | **$5 free usage/month** ($0.20/compute unit), no credit card |
|
||||
| **Paid tiers** | **Starter:** $29/mo ($29 spend). **Scale:** $199/mo ($199 spend). **Business:** $999/mo ($999 spend). Compute unit pricing drops at higher tiers ($0.20 → $0.16 → $0.13). |
|
||||
| **JS rendering** | ✅ Yes — full headless Chrome via Crawlee/Playwright actors |
|
||||
| **Features** | 2,000+ pre-built Actors ✅ (Website Content Crawler, Google scraper, LinkedIn scraper, Instagram, etc.), scheduling ✅, proxy rotation ✅, Crawlee library ✅, MCP server ✅, webhooks ✅ |
|
||||
| **API quality** | Excellent. REST API, Crawlee SDK (Python/JS), webhooks, MCP integration. Open platform. |
|
||||
| **Best use case** | "Set and forget" scraping. Use pre-built Actors for common sites. Build custom Actors for niche sites. Good for recurring scraping jobs. |
|
||||
| **Firecrawl gap** | Apify is a **platform**, not just an API. The pre-built Actors mean you can scrape LinkedIn, Instagram, Google Maps etc. without writing code. More complex but far more powerful for ongoing scraping operations. |
|
||||
|
||||
**Verdict:** ✅ **RECOMMENDED** — Best for recurring, complex scraping workflows with pre-built Actors.
|
||||
|
||||
---
|
||||
|
||||
## 9. Browserbase — Cloud Browser Automation
|
||||
**URL:** https://www.browserbase.com
|
||||
|
||||
| Attribute | Detail |
|
||||
|-----------|--------|
|
||||
| **Problem it solves** | Cloud-hosted headless browsers with AI-powered automation (Stagehand SDK). Playwright/Puppeteer/Selenium compatible. Full browser control in the cloud. |
|
||||
| **Free tier** | **Developer (free):** 3 agent runs/mo, 1,000 Fetch API calls, 1,000 Search API calls, $5 in model tokens, 7-day data retention |
|
||||
| **Paid tiers** | **Startup:** 15 agent runs. **Team:** 50 agent runs. **Enterprise:** Custom. Fetch API: $1/1k calls (basic), $4/1k (with proxies). Extract: $4/1k, $7/1k (with proxies). Search: $7/1k requests. Proxies: $12/GB (overage). |
|
||||
| **JS rendering** | ✅ **Yes** — full Playwright/Puppeteer control. This is a full browser in the cloud. |
|
||||
| **Features** | Captcha solving ✅, Model Gateway ✅ (benchmark models against web workflows), stealth mode ✅, session recording ✅, custom proxies ✅, SOC 2 ✅, Stagehand AI SDK ✅ |
|
||||
| **API quality** | Excellent. REST API + WebSocket, Stagehand SDK (TS/JS), connects with LangChain/CrewAI/Mastra. |
|
||||
| **Best use case** | Agentic browser automation — when you need to **interact** with sites (click, fill forms, login). Government portal workflows. AI agent web tasks. |
|
||||
| **Firecrawl gap** | Browserbase is **not a scraping API** — it's a **cloud browser**. Use it when you need to interact with JS-heavy sites dynamically (e.g., submit court record search forms, navigate multi-step government portals). Firecrawl can't do this at all. |
|
||||
|
||||
**Verdict:** ✅ **RECOMMENDED** — Best for interactive browser automation (form fills, logins, multi-step government site workflows).
|
||||
|
||||
---
|
||||
|
||||
## Comparison Matrix
|
||||
|
||||
| Service | Free Tier | Entry Price | Volume (entry) | JS Rendering | Search | Scraping | Best For |
|
||||
|---------|-----------|------------|----------------|-------------|--------|----------|----------|
|
||||
| **Firecrawl** (current) | 500 credits | ? | 1,000/mo | ✅ | ✅ | ✅ | General web data — your existing pipeline |
|
||||
| **Tavily** | 1,000 credits | $0.008/credit | 1,000/mo free | ❌ | ✅ | ✅ | AI agent search/RAG |
|
||||
| **SerpAPI** | 100 searches | ~$50/mo | 5,000/mo | ✅ (SERP) | ✅ | ❌ | Google SERP data |
|
||||
| **Serper** | 2,500 searches | ~$50/mo | 50,000/mo | ❌ | ✅ | ❌ | High-volume Google search |
|
||||
| **ScrapingBee** | 1,000 credits | $49/mo | 250K/mo | ✅ (5cr/req) | ✅ | ✅ | **JS rendering — best Firecrawl alternative** |
|
||||
| **ScrapingAnt** | **10,000 credits** | **$19/mo** | **100K/mo** | ✅ (10cr/req) | ✅ | ✅ | **Best value Firecrawl replacement** |
|
||||
| **Jina AI Reader** | 10M tokens | Token-based | Free tier generous | ❌ | ❌ | ✅ (limited) | URL→Markdown for LLMs |
|
||||
| **Brave Search** | $5/mo credit | $5/1K req | ~1K/mo free | ❌ | ✅ | ❌ | Independent search, low cost |
|
||||
| **Apify** | $5 usage | $29/mo | Platform pricing | ✅ | ✅ | ✅ | Recurring/complex scraping workflows |
|
||||
| **Browserbase** | 3 agent runs | Usage-based | 1K fetch free | ✅ **Full browser** | ✅ | ✅ | **Interactive browser automation** |
|
||||
|
||||
---
|
||||
|
||||
## Strategic Recommendations
|
||||
|
||||
### Tier 1: Critical Additions — Add Now
|
||||
|
||||
**1. ScrapingAnt** — **Primary Firecrawl complement/replacement**
|
||||
- **Why:** $19/mo for 100K credits vs Firecrawl's ~$50-100 for 5,000. 10,000 free credits/month to try. Headless Chrome + rotating proxies handle any JS-heavy government site. MCP server for AI agents is a bonus.
|
||||
- **Use for:** Texas Legislature scraping, court records, any JS-heavy government/agency site
|
||||
- **Cost:** Free to start, $19/mo for 100K credits
|
||||
|
||||
**2. Browserbase** — **Interactive government portal automation**
|
||||
- **Why:** Government sites often need form submission (search case number, enter parameters) — not just page loading. Browserbase gives full Playwright control in the cloud with captcha solving and stealth mode.
|
||||
- **Use for:** Court record searches that require form submission, multi-step government workflows, login-gated agency portals
|
||||
- **Cost:** Free tier (3 agent runs, 1K fetch) to start
|
||||
|
||||
### Tier 2: Add When Needed
|
||||
|
||||
**3. Brave Search API** — **Independent web search**
|
||||
- **Why:** $5/mo free credits covers ~1K searches. Independent index avoids Google dependency. Complements Tavily/Firecrawl for content discovery.
|
||||
- **Use for:** General web search for MSP research, finding resources across your business lines
|
||||
- **Cost:** ~$5/mo effectively free
|
||||
|
||||
**4. Apify** — **Recurring/complex scraping**
|
||||
- **Why:** Pre-built Actors for LinkedIn, Google Maps, Instagram, etc. If you need to scrape these platforms regularly, Apify saves massive development time.
|
||||
- **Use for:** Regular monitoring of competitors, directory scraping, social media data
|
||||
- **Cost:** $29/mo to start
|
||||
|
||||
### Tier 3: Nice-to-Have
|
||||
|
||||
**5. Tavily** — **AI search (if you're building AI agents)**
|
||||
- **Why:** Excellent for RAG/agent search. Free tier covers evaluation. If you're not building AI agents yet, skip this.
|
||||
- **Use for:** AI-powered search within your ops portal
|
||||
- **Cost:** Free tier available
|
||||
|
||||
**6. Jina AI Reader** — **Clean markdown for LLMs**
|
||||
- **Why:** If you're feeding scraped content into LLMs, Jina's markdown output is cleaner than raw HTML. Only useful for non-JS pages.
|
||||
- **Use for:** LLM content ingestion pipeline
|
||||
- **Cost:** 10M free tokens to start
|
||||
|
||||
---
|
||||
|
||||
## Cost-Benefit Analysis: Firecrawl Upgrade vs Alternatives
|
||||
|
||||
| Scenario | Firecrawl Upgrade (5K credits) | ScrapingAnt (100K credits) | Savings |
|
||||
|----------|-------------------------------|---------------------------|---------|
|
||||
| **Monthly volume** | 5,000 credits | 100,000 credits | **20x more volume** |
|
||||
| **Estimated cost** | ~$50-100/mo | **$19/mo** | **60-80% cheaper** |
|
||||
| **JS rendering** | ✅ | ✅ (Chrome included) | Same |
|
||||
| **Free trial** | None mentioned | **10,000 credits free** | Risk-free trial |
|
||||
|
||||
**Bottom line:** Adding ScrapingAnt ($19/mo for 100K credits) as a Firecrawl supplement gives you 20x more scraping capacity for less money. The 10K free credits let you test with zero risk.
|
||||
|
||||
---
|
||||
|
||||
## Use Case Quick Reference
|
||||
|
||||
| Use Case | Recommended API | Why |
|
||||
|----------|----------------|-----|
|
||||
| **Texas Legislature bill tracking** | ScrapingAnt or ScrapingBee | JS-heavy legislative sites need headless Chrome |
|
||||
| **Court record searches (forms)** | Browserbase | Need to submit forms, handle redirects, solve CAPTCHAs |
|
||||
| **Google Maps/MAPS data** | SerpAPI or Apify | Specialized Actors handle Maps data structure |
|
||||
| **General web research for MSP** | Brave Search API | Independent index, $5/mo free |
|
||||
| **RAG/LLM content ingestion** | Jina AI Reader | Cleanest markdown output for AI consumption |
|
||||
| **High-volume Google searches** | Serper | Cheapest per-query Google search |
|
||||
| **Recurring social media scraping** | Apify | Pre-built Actors for Instagram, LinkedIn, TikTok |
|
||||
| **Simple page scraping (non-JS)** | Firecrawl (keep current) | Already works for basic sites |
|
||||
Reference in New Issue
Block a user