Sync docs, audit artifacts, project notes, and VerdictTank proposal docs
- audit/phase-one + phase-two: security audit briefs, findings, credential-rotation plan, Docker-USER hardening scripts, rollback refs - disaster-recovery/restore-test-log.md + backup-dr-audit-2026-08-10.md - clients/ (modelortho SEO audit, ai-biz-dev competitive landscape), notes/ (tiktok strategy) - projects/: front-desk-voice-agent, seo-visibility-checker product plan, hotnow-savannah HTML, resend-transactional-email, backup-dashboard-enhancements, code-review-graph, seo-ci-architecture - proposals/verdicttank/: architecture v4.0, methodology, judge-pool review, consolidation reasoning, cross-check review - docs/super-search/firecrawl-provider-strategy.md - updates: CHANGELOG, model-chain, projects-master-readme, intelsight.io - .gitignore: exclude nested standalone repos (seo-tool, venturebuilt)
This commit is contained in:
@@ -0,0 +1,95 @@
|
||||
# Firecrawl — Provider Strategy & Configuration
|
||||
|
||||
**Created:** 2026-08-10 | **Status:** Active Burn Period → Hobby
|
||||
|
||||
---
|
||||
|
||||
## Plan Timeline
|
||||
|
||||
| Date | Event |
|
||||
|---|---|
|
||||
| 2026-08-10 | Standard plan (100k credits, $99/mo) — Firecrawl promoted to primary |
|
||||
| 2026-09-10 | Billing period ends — Standard plan credits expire |
|
||||
| 2026-09-11 | Downgrade to Hobby plan (5k credits, $16–19/mo) |
|
||||
|
||||
---
|
||||
|
||||
## Why We Burned the Standard Plan
|
||||
|
||||
Usage was ~1,400 credits/month — 1.4% of the Standard plan's 100k allocation. The $99/mo Standard plan was over-provisioned by 70x. Firecrawl was position #11 (dead last) in the search fallback chain, so it was almost never called. The decision: burn through Standard credits aggressively until the plan naturally expires on Sep 10, then let the Hobby downgrade take effect with 5k credits/month (~3.5x our actual needs).
|
||||
|
||||
---
|
||||
|
||||
## Rate Limit History
|
||||
|
||||
| When | Firecrawl Rate Limit | Context |
|
||||
|---|---|---|
|
||||
| Pre-Aug 10 | 5 tokens / 60s | Position #11 in search chain — rarely called |
|
||||
| Aug 10 (15:45) | 20 tokens / 60s | Promoted to primary in both chains |
|
||||
| Aug 10 (15:51) | 100 tokens / 5s | Maximized for burn period (20 req/s throughput) |
|
||||
|
||||
### Rate Limit Comparison (Pricing Page)
|
||||
|
||||
| Plan | Concurrent Requests | Credits/Month |
|
||||
|---|---|---|
|
||||
| Free | 2 | 1,000 |
|
||||
| Hobby | 5 | 5,000 |
|
||||
| Standard | 50 | 100,000 |
|
||||
| Growth | 100 | 500,000 |
|
||||
| Scale | 150 | 1,000,000 |
|
||||
|
||||
No per-second or per-minute rate caps published per tier. Concurrent request caps are the only documented limit, and Super Search runs calls sequentially — it never hits the 50 concurrent cap even on Standard.
|
||||
|
||||
---
|
||||
|
||||
## Provider Chain Order
|
||||
|
||||
### `web_search`
|
||||
```
|
||||
1. Firecrawl ← PRIMARY (promoted Aug 10 from #11)
|
||||
2. SearXNG (local)
|
||||
3. Exa
|
||||
4. OpenCorporates
|
||||
5. CourtListener
|
||||
6. DuckDuckGo
|
||||
7. Brave
|
||||
8. Serper
|
||||
9. Tavily
|
||||
10. Perplexity
|
||||
11. Parallel (Exa + Brave + DuckDuckGo)
|
||||
```
|
||||
|
||||
### `web_extract`
|
||||
```
|
||||
1. Firecrawl ← PRIMARY (promoted Aug 10 from #4)
|
||||
2. Jina Reader
|
||||
3. Trafilatura
|
||||
4. Browserless (CF bypass)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Files Modified
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| `/root/docker/super-search/server.py` | Firecrawl moved to position #1 in both search + extract chains; docstrings updated |
|
||||
| `/root/docker/super-search/ratelimit.py` | Firecrawl rate limit: 5/min → 20/min → 100/5s |
|
||||
|
||||
---
|
||||
|
||||
## Monitoring
|
||||
|
||||
| Job | Schedule | Description |
|
||||
|---|---|---|
|
||||
| `firecrawl-usage-check` (2a2f) | Every 2 hours | Queries Firecrawl API `/v1/team/credit-usage` — reports remaining credits to Telegram |
|
||||
| Script: `firecrawl-credit-check.sh` | — | Curl-based, no agent needed |
|
||||
|
||||
---
|
||||
|
||||
## Post-Burn Reversion Plan (Sep 11)
|
||||
|
||||
After Hobby downgrade:
|
||||
1. Revert rate limit to `{"tokens": 5, "interval": 60.0}` in `ratelimit.py`
|
||||
2. Consider moving Firecrawl back to fallback position (TBD based on credit consumption patterns during burn period)
|
||||
3. Keep 2-hour monitoring until burn period confirmed complete
|
||||
Reference in New Issue
Block a user