337 lines
11 KiB
Markdown
337 lines
11 KiB
Markdown
# Super Search MCP Enhancement Execution Plan
|
|
|
|
**Created:** 2026-08-07
|
|
**Source:** Super Search Enhancement Scanner (cron, Aug 7 2026)
|
|
**Status:** OPEN
|
|
|
|
---
|
|
|
|
## Overview
|
|
|
|
16 actionable enhancements identified for Super Search MCP (http://127.0.0.1:8899). Current stack: FastMCP 2.x, SearXNG Docker, Exa API, Firecrawl API, Trafilatura, DuckDuckGo fallback.
|
|
|
|
---
|
|
|
|
## HIGH Priority (Execute First -- Weeks 1-2)
|
|
|
|
### 1. Upgrade FastMCP 2.x -> 3.x
|
|
|
|
**Why:** Provider architecture, component versioning, OpenTelemetry, tool timeouts, concurrent execution. Current 2.x is aging out.
|
|
|
|
**Steps:**
|
|
- [ ] Pin current FastMCP version to freeze baseline
|
|
- [ ] Review breaking changes in FastMCP 3.x changelog (v3.0 Feb 2026 -> v3.3.0 May 2026)
|
|
- [ ] Upgrade in venv: `pip install --upgrade fastmcp`
|
|
- [ ] Test all 14 Super Search tools individually
|
|
- [ ] Test fallback chain behavior (SearXNG -> Exa -> DDG -> Firecrawl)
|
|
- [ ] Verify health_check and circuit_status still work
|
|
- [ ] Deploy and monitor for 48h
|
|
|
|
**Risk:** Medium -- API surface is largely compatible but component versioning may affect tool registration
|
|
**Effort:** 3-4 hours
|
|
**Dependencies:** None
|
|
|
|
---
|
|
|
|
### 2. Add Brave Search API to Fallback Chain
|
|
|
|
**Why:** Independent 40B+ page index (no Google/Bing dependency). $5/1K queries. LLM Context endpoint returns pre-formatted results for AI use.
|
|
|
|
**Steps:**
|
|
- [ ] Sign up for Brave Search API free tier (2,000 queries/month)
|
|
- [ ] Store API key in `.env`
|
|
- [ ] Add `search_brave(query, limit)` to server.py using Brave Web Search endpoint
|
|
- [ ] Insert between SearXNG and DuckDuckGo in fallback chain
|
|
- [ ] Add to circuit_status tool
|
|
- [ ] Add `web_search_llm_context` tool using Brave's LLM Context endpoint
|
|
- [ ] Test with 20 queries and compare result quality vs Exa/SearXNG
|
|
|
|
**Risk:** Low -- independent API, no shared infra
|
|
**Effort:** 2-3 hours
|
|
**Dependencies:** Brave API key (free signup)
|
|
|
|
---
|
|
|
|
### 3. Fix Exa API Deprecations
|
|
|
|
**Why:** Exa deprecated `pdf`, `github`, `tweet` categories and replaced `livecrawl` with `maxAgeHours`. `research paper` -> `publication`.
|
|
|
|
**Steps:**
|
|
- [ ] Audit server.py for all Exa category references
|
|
- [ ] Replace `category: "research paper"` -> `category: "publication"` in `web_search_academic`
|
|
- [ ] Remove `pdf`, `github`, `tweet` from category mapping logic
|
|
- [ ] Replace `livecrawl` -> `maxAgeHours` in `web_extract` Exa path
|
|
- [ ] Test academic search with new `publication` category (350M papers)
|
|
- [ ] Test extraction with `maxAgeHours` parameter
|
|
|
|
**Risk:** Low -- straightforward replacements, Exa docs are clear
|
|
**Effort:** 1 hour
|
|
**Dependencies:** None
|
|
|
|
---
|
|
|
|
### 4. Add Crawl4AI as Self-Hosted Extraction Backend
|
|
|
|
**Why:** Free, no rate limits, stealth mode (undetected browser), JS rendering, parallel crawling. 77K GitHub stars. Handles bot-protected pages that Trafilatura and Firecrawl can't reach.
|
|
|
|
**Steps:**
|
|
- [ ] Install Crawl4AI: `pip install crawl4ai`
|
|
- [ ] Add `web_extract_stealth(url)` tool -- uses Playwright stealth mode for JS-heavy/bot-protected pages
|
|
- [ ] Add `web_extract_bulk(urls)` tool -- parallel extraction for batch jobs
|
|
- [ ] Wire into fallback chain: Trafilatura -> Firecrawl -> Crawl4AI
|
|
- [ ] Test on known-bot-protected URLs (VRBO, Expedia, etc.)
|
|
- [ ] Document Playwright dependency (may need `playwright install chromium`)
|
|
|
|
**Risk:** Medium -- adds Chromium/Playwright dependency (~300MB), may increase RAM usage
|
|
**Effort:** 3-4 hours
|
|
**Dependencies:** `pip install crawl4ai playwright`, `playwright install chromium`
|
|
|
|
---
|
|
|
|
## MEDIUM Priority (Plan Next -- Weeks 3-4)
|
|
|
|
### 5. Migrate SSE -> Streamable HTTP Transport
|
|
|
|
**Why:** MCP spec (2026-07-28) deprecated SSE. Streamable HTTP works with standard CORS, auth, and load balancers.
|
|
|
|
**Steps:**
|
|
- [ ] Verify FastMCP 3.x supports Streamable HTTP natively (it does)
|
|
- [ ] Update server.py transport configuration
|
|
- [ ] Test with Hermes Agent as MCP client
|
|
- [ ] Verify Caddy reverse proxy still works
|
|
- [ ] Update any client configurations pointing to SSE endpoint
|
|
|
|
**Risk:** Medium -- transport change affects all MCP clients
|
|
**Effort:** 2 hours
|
|
**Dependencies:** **FastMCP 3.x upgrade (Item #1)**
|
|
|
|
---
|
|
|
|
### 6. Add `web_extract_document` Tool
|
|
|
|
**Why:** Currently Super Search only handles URLs. Firecrawl `/parse` handles PDFs, Word docs, spreadsheets up to 50MB -> clean markdown.
|
|
|
|
**Steps:**
|
|
- [ ] Add `web_extract_document(file_url)` tool wrapping Firecrawl `/parse`
|
|
- [ ] Support PDF, DOCX, XLSX, PPTX formats
|
|
- [ ] Return clean markdown with structured data where available
|
|
- [ ] Add file size validation (max 50MB)
|
|
- [ ] Test with sample PDF, Word doc, and spreadsheet
|
|
|
|
**Risk:** Low -- wraps existing Firecrawl endpoint
|
|
**Effort:** 1-2 hours
|
|
**Dependencies:** None
|
|
|
|
---
|
|
|
|
### 7. Add Firecrawl Lockdown Mode
|
|
|
|
**Why:** Zero-outbound-request extraction from Firecrawl cache. Critical for sensitive/sandboxed use cases.
|
|
|
|
**Steps:**
|
|
- [ ] Add `lockdown: true` parameter to `web_extract` when using Firecrawl
|
|
- [ ] Document that Lockdown Mode means no live outbound requests
|
|
- [ ] Test that Lockdown Mode returns only cached/indexed content
|
|
|
|
**Risk:** Low -- feature flag on existing Firecrawl API
|
|
**Effort:** 30 minutes
|
|
**Dependencies:** None
|
|
|
|
---
|
|
|
|
### 8. Add Exa Agent as `web_research_deep` Tool
|
|
|
|
**Why:** Multi-step agentic research for complex queries. Exa Agent does recursive search + extraction + synthesis.
|
|
|
|
**Steps:**
|
|
- [ ] Review Exa Agent API docs and pricing ($0.10/ACU + $0.005/search)
|
|
- [ ] Add `web_research_deep(query, effort="medium")` tool
|
|
- [ ] Support `outputSchema` for structured outputs
|
|
- [ ] Add cost estimation before execution (warn if >$0.50 estimated)
|
|
- [ ] Test with complex multi-step research query
|
|
|
|
**Risk:** Medium -- cost per query is higher, needs rate limiting
|
|
**Effort:** 2-3 hours
|
|
**Dependencies:** None
|
|
|
|
---
|
|
|
|
### 9. Add Result Deduplication Across Providers
|
|
|
|
**Why:** When multiple backends return the same URL, we serve duplicate results. Simple URL normalization + content hash dedup.
|
|
|
|
**Steps:**
|
|
- [ ] Implement URL normalization (strip tracking params, trailing slashes, www prefix)
|
|
- [ ] When merging results from multiple providers, hash URLs and deduplicate
|
|
- [ ] Keep the best snippet/metadata per unique URL (prefer richer provider)
|
|
- [ ] Add `dedup_summary` to response metadata (count of duplicates removed)
|
|
- [ ] Test with queries that hit multiple providers
|
|
|
|
**Risk:** Low -- purely additive, no breaking changes
|
|
**Effort:** 2 hours
|
|
**Dependencies:** None
|
|
|
|
---
|
|
|
|
### 10. Evaluate 4get-hijacked for SearXNG
|
|
|
|
**Why:** Community project that proxies ~30 search engines into SearXNG-compatible format. Sidesteps broken major engine scrapers.
|
|
|
|
**Steps:**
|
|
- [ ] Clone and review `cra88y/4get-hijacked` repo
|
|
- [ ] Test integration with our SearXNG Docker instance
|
|
- [ ] Benchmark result quality vs current engine pool
|
|
- [ ] Decide: add to search engine list or pass
|
|
|
|
**Risk:** Low -- evaluation only, no commitment
|
|
**Effort:** 1-2 hours
|
|
**Dependencies:** None
|
|
|
|
---
|
|
|
|
### 11. Add Tavily as `web_search_ai` Tool
|
|
|
|
**Why:** Purpose-built AI search with relevance scores. Not for general fallback chain (higher cost/latency) but excellent for AI-optimized results.
|
|
|
|
**Steps:**
|
|
- [ ] Sign up for Tavily free tier (1,000 queries/month)
|
|
- [ ] Add `web_search_ai(query, depth="advanced")` as standalone tool
|
|
- [ ] Return relevance-scored results with confidence markers
|
|
- [ ] Do NOT add to fallback chain (keep as separate tool for explicit use)
|
|
- [ ] Test against SearXNG/Exa for quality comparison
|
|
|
|
**Risk:** Low -- standalone tool, no chain impact
|
|
**Effort:** 1.5 hours
|
|
**Dependencies:** Tavily API key (free signup)
|
|
|
|
---
|
|
|
|
## LOW Priority (Nice to Have -- Weeks 5+)
|
|
|
|
### 12. Add Exa Monitors Integration
|
|
|
|
**Why:** Scheduled searches with webhook delivery, deduplicated against previous runs.
|
|
|
|
**Steps:**
|
|
- [ ] Review Exa Monitors API
|
|
- [ ] Add `web_monitor(query, schedule, webhook_url)` tool
|
|
- [ ] Could replace or augment custom monitoring scripts
|
|
|
|
**Effort:** 2 hours
|
|
**Dependencies:** Webhook endpoint for delivery
|
|
|
|
---
|
|
|
|
### 13. Add Brave Goggles for Custom Reranking
|
|
|
|
**Why:** Only search API that lets you boost/promote specific domains at query time.
|
|
|
|
**Steps:**
|
|
- [ ] Create Goggles config for IT Pro Partner preferred domains
|
|
- [ ] Add `goggles` parameter to Brave search calls
|
|
- [ ] Test domain boosting effectiveness
|
|
|
|
**Effort:** 1 hour
|
|
**Dependencies:** Brave Search API (Item #2)
|
|
|
|
---
|
|
|
|
### 14. Add LLM Metadata Enrichment
|
|
|
|
**Why:** Generate one-line semantic descriptions of extracted pages for better downstream RAG retrieval.
|
|
|
|
**Steps:**
|
|
- [ ] Add `enrich_metadata: true` option to web_extract
|
|
- [ ] Use cheap local model or Firecrawl's built-in summarization
|
|
- [ ] Tag results with semantic descriptions
|
|
- [ ] Benchmark retrieval improvement
|
|
|
|
**Effort:** 3-4 hours
|
|
**Dependencies:** None (can use Firecrawl's question format or local model)
|
|
|
|
---
|
|
|
|
### 15. Evaluate Kagi Search API
|
|
|
|
**Why:** Premium search quality. Worth a trial key for comparison benchmarking.
|
|
|
|
**Steps:**
|
|
- [ ] Sign up for Kagi trial
|
|
- [ ] Run 50 side-by-side comparisons: Kagi vs Exa vs Brave vs SearXNG
|
|
- [ ] Score relevance, freshness, and coverage
|
|
- [ ] Decide: add to chain or pass
|
|
|
|
**Effort:** 2 hours
|
|
**Dependencies:** Kagi API trial key
|
|
|
|
---
|
|
|
|
### 16. Add OpenTelemetry Tracing
|
|
|
|
**Why:** FastMCP 3.x has native OTEL -- spans for every search call, fallback path taken, extraction step.
|
|
|
|
**Steps:**
|
|
- [ ] Install OpenTelemetry packages
|
|
- [ ] Run with `opentelemetry-instrument fastmcp run server.py`
|
|
- [ ] Configure export to local collector or file
|
|
- [ ] Analyze fallback chain behavior
|
|
|
|
**Effort:** 1 hour
|
|
**Dependencies:** FastMCP 3.x upgrade (Item #1)
|
|
|
|
---
|
|
|
|
## Execution Order (Dependency-Aware)
|
|
|
|
```
|
|
Phase 1 (Week 1):
|
|
Day 1: Items #1 (FastMCP 3.0) + #3 (Exa deprecations) -- can run in parallel
|
|
Day 2: Item #2 (Brave Search API) -- independent
|
|
Day 3: Item #4 (Crawl4AI) -- independent, longest install
|
|
Day 4: Testing + burn-in of Phase 1 changes
|
|
|
|
Phase 2 (Week 2):
|
|
Item #5 (SSE -> Streamable HTTP) -- depends on #1
|
|
Items #6 + #7 (extract_document + Lockdown Mode) -- parallel, both Firecrawl
|
|
Item #9 (result dedup) -- independent
|
|
|
|
Phase 3 (Week 3):
|
|
Items #8 (Exa Agent) + #11 (Tavily) -- parallel, both new API integrations
|
|
Item #10 (4get-hijacked eval) -- independent
|
|
|
|
Phase 4 (Week 4+):
|
|
Items #12-#16 -- low priority, pick up as time allows
|
|
```
|
|
|
|
---
|
|
|
|
## Risk Register
|
|
|
|
| Risk | Likelihood | Impact | Mitigation |
|
|
|------|-----------|--------|------------|
|
|
| FastMCP 3.x breaking API changes | Medium | High | Pin 2.x, test exhaustively before deploy |
|
|
| Crawl4AI RAM usage with Chromium | Medium | Medium | Monitor RAM, consider Docker isolation |
|
|
| Exa Agent cost overruns | Low | Medium | Per-query cost estimate cap |
|
|
| Brave API rate limits | Low | Low | Free tier sufficient for testing |
|
|
| Streamable HTTP transport issues | Low | High | Test with Hermes Agent before cutting over |
|
|
|
|
---
|
|
|
|
## Success Metrics
|
|
|
|
- All 14 existing tools continue working post-upgrade
|
|
- Brave API adds independent fallback source (no Google/Bing dependency)
|
|
- Crawl4AI handles 3+ known-bot-protected sites that previously failed
|
|
- Result dedup eliminates >=80% of cross-provider duplicates
|
|
- Zero regressions in Hermes Agent's use of Super Search tools
|
|
|
|
---
|
|
|
|
## Reference
|
|
|
|
- Super Search server: `/root/docker/super-search/server.py`
|
|
- Systemd service: `super-search.service`
|
|
- Venv: `/root/docker/super-search/venv/`
|
|
- Health endpoint: `http://127.0.0.1:8899/health`
|
|
- Full audit: Aug 1, 2026 -- zero breaking patterns for FastMCP 4.0
|