diff --git a/docs/super-search-enhancement-plan.md b/docs/super-search-enhancement-plan.md new file mode 100644 index 0000000..8e8d841 --- /dev/null +++ b/docs/super-search-enhancement-plan.md @@ -0,0 +1,336 @@ +# Super Search MCP Enhancement Execution Plan + +**Created:** 2026-08-07 +**Source:** Super Search Enhancement Scanner (cron, Aug 7 2026) +**Status:** OPEN + +--- + +## Overview + +16 actionable enhancements identified for Super Search MCP (http://127.0.0.1:8899). Current stack: FastMCP 2.x, SearXNG Docker, Exa API, Firecrawl API, Trafilatura, DuckDuckGo fallback. + +--- + +## HIGH Priority (Execute First -- Weeks 1-2) + +### 1. Upgrade FastMCP 2.x -> 3.x + +**Why:** Provider architecture, component versioning, OpenTelemetry, tool timeouts, concurrent execution. Current 2.x is aging out. + +**Steps:** +- [ ] Pin current FastMCP version to freeze baseline +- [ ] Review breaking changes in FastMCP 3.x changelog (v3.0 Feb 2026 -> v3.3.0 May 2026) +- [ ] Upgrade in venv: `pip install --upgrade fastmcp` +- [ ] Test all 14 Super Search tools individually +- [ ] Test fallback chain behavior (SearXNG -> Exa -> DDG -> Firecrawl) +- [ ] Verify health_check and circuit_status still work +- [ ] Deploy and monitor for 48h + +**Risk:** Medium -- API surface is largely compatible but component versioning may affect tool registration +**Effort:** 3-4 hours +**Dependencies:** None + +--- + +### 2. Add Brave Search API to Fallback Chain + +**Why:** Independent 40B+ page index (no Google/Bing dependency). $5/1K queries. LLM Context endpoint returns pre-formatted results for AI use. + +**Steps:** +- [ ] Sign up for Brave Search API free tier (2,000 queries/month) +- [ ] Store API key in `.env` +- [ ] Add `search_brave(query, limit)` to server.py using Brave Web Search endpoint +- [ ] Insert between SearXNG and DuckDuckGo in fallback chain +- [ ] Add to circuit_status tool +- [ ] Add `web_search_llm_context` tool using Brave's LLM Context endpoint +- [ ] Test with 20 queries and compare result quality vs Exa/SearXNG + +**Risk:** Low -- independent API, no shared infra +**Effort:** 2-3 hours +**Dependencies:** Brave API key (free signup) + +--- + +### 3. Fix Exa API Deprecations + +**Why:** Exa deprecated `pdf`, `github`, `tweet` categories and replaced `livecrawl` with `maxAgeHours`. `research paper` -> `publication`. + +**Steps:** +- [ ] Audit server.py for all Exa category references +- [ ] Replace `category: "research paper"` -> `category: "publication"` in `web_search_academic` +- [ ] Remove `pdf`, `github`, `tweet` from category mapping logic +- [ ] Replace `livecrawl` -> `maxAgeHours` in `web_extract` Exa path +- [ ] Test academic search with new `publication` category (350M papers) +- [ ] Test extraction with `maxAgeHours` parameter + +**Risk:** Low -- straightforward replacements, Exa docs are clear +**Effort:** 1 hour +**Dependencies:** None + +--- + +### 4. Add Crawl4AI as Self-Hosted Extraction Backend + +**Why:** Free, no rate limits, stealth mode (undetected browser), JS rendering, parallel crawling. 77K GitHub stars. Handles bot-protected pages that Trafilatura and Firecrawl can't reach. + +**Steps:** +- [ ] Install Crawl4AI: `pip install crawl4ai` +- [ ] Add `web_extract_stealth(url)` tool -- uses Playwright stealth mode for JS-heavy/bot-protected pages +- [ ] Add `web_extract_bulk(urls)` tool -- parallel extraction for batch jobs +- [ ] Wire into fallback chain: Trafilatura -> Firecrawl -> Crawl4AI +- [ ] Test on known-bot-protected URLs (VRBO, Expedia, etc.) +- [ ] Document Playwright dependency (may need `playwright install chromium`) + +**Risk:** Medium -- adds Chromium/Playwright dependency (~300MB), may increase RAM usage +**Effort:** 3-4 hours +**Dependencies:** `pip install crawl4ai playwright`, `playwright install chromium` + +--- + +## MEDIUM Priority (Plan Next -- Weeks 3-4) + +### 5. Migrate SSE -> Streamable HTTP Transport + +**Why:** MCP spec (2026-07-28) deprecated SSE. Streamable HTTP works with standard CORS, auth, and load balancers. + +**Steps:** +- [ ] Verify FastMCP 3.x supports Streamable HTTP natively (it does) +- [ ] Update server.py transport configuration +- [ ] Test with Hermes Agent as MCP client +- [ ] Verify Caddy reverse proxy still works +- [ ] Update any client configurations pointing to SSE endpoint + +**Risk:** Medium -- transport change affects all MCP clients +**Effort:** 2 hours +**Dependencies:** **FastMCP 3.x upgrade (Item #1)** + +--- + +### 6. Add `web_extract_document` Tool + +**Why:** Currently Super Search only handles URLs. Firecrawl `/parse` handles PDFs, Word docs, spreadsheets up to 50MB -> clean markdown. + +**Steps:** +- [ ] Add `web_extract_document(file_url)` tool wrapping Firecrawl `/parse` +- [ ] Support PDF, DOCX, XLSX, PPTX formats +- [ ] Return clean markdown with structured data where available +- [ ] Add file size validation (max 50MB) +- [ ] Test with sample PDF, Word doc, and spreadsheet + +**Risk:** Low -- wraps existing Firecrawl endpoint +**Effort:** 1-2 hours +**Dependencies:** None + +--- + +### 7. Add Firecrawl Lockdown Mode + +**Why:** Zero-outbound-request extraction from Firecrawl cache. Critical for sensitive/sandboxed use cases. + +**Steps:** +- [ ] Add `lockdown: true` parameter to `web_extract` when using Firecrawl +- [ ] Document that Lockdown Mode means no live outbound requests +- [ ] Test that Lockdown Mode returns only cached/indexed content + +**Risk:** Low -- feature flag on existing Firecrawl API +**Effort:** 30 minutes +**Dependencies:** None + +--- + +### 8. Add Exa Agent as `web_research_deep` Tool + +**Why:** Multi-step agentic research for complex queries. Exa Agent does recursive search + extraction + synthesis. + +**Steps:** +- [ ] Review Exa Agent API docs and pricing ($0.10/ACU + $0.005/search) +- [ ] Add `web_research_deep(query, effort="medium")` tool +- [ ] Support `outputSchema` for structured outputs +- [ ] Add cost estimation before execution (warn if >$0.50 estimated) +- [ ] Test with complex multi-step research query + +**Risk:** Medium -- cost per query is higher, needs rate limiting +**Effort:** 2-3 hours +**Dependencies:** None + +--- + +### 9. Add Result Deduplication Across Providers + +**Why:** When multiple backends return the same URL, we serve duplicate results. Simple URL normalization + content hash dedup. + +**Steps:** +- [ ] Implement URL normalization (strip tracking params, trailing slashes, www prefix) +- [ ] When merging results from multiple providers, hash URLs and deduplicate +- [ ] Keep the best snippet/metadata per unique URL (prefer richer provider) +- [ ] Add `dedup_summary` to response metadata (count of duplicates removed) +- [ ] Test with queries that hit multiple providers + +**Risk:** Low -- purely additive, no breaking changes +**Effort:** 2 hours +**Dependencies:** None + +--- + +### 10. Evaluate 4get-hijacked for SearXNG + +**Why:** Community project that proxies ~30 search engines into SearXNG-compatible format. Sidesteps broken major engine scrapers. + +**Steps:** +- [ ] Clone and review `cra88y/4get-hijacked` repo +- [ ] Test integration with our SearXNG Docker instance +- [ ] Benchmark result quality vs current engine pool +- [ ] Decide: add to search engine list or pass + +**Risk:** Low -- evaluation only, no commitment +**Effort:** 1-2 hours +**Dependencies:** None + +--- + +### 11. Add Tavily as `web_search_ai` Tool + +**Why:** Purpose-built AI search with relevance scores. Not for general fallback chain (higher cost/latency) but excellent for AI-optimized results. + +**Steps:** +- [ ] Sign up for Tavily free tier (1,000 queries/month) +- [ ] Add `web_search_ai(query, depth="advanced")` as standalone tool +- [ ] Return relevance-scored results with confidence markers +- [ ] Do NOT add to fallback chain (keep as separate tool for explicit use) +- [ ] Test against SearXNG/Exa for quality comparison + +**Risk:** Low -- standalone tool, no chain impact +**Effort:** 1.5 hours +**Dependencies:** Tavily API key (free signup) + +--- + +## LOW Priority (Nice to Have -- Weeks 5+) + +### 12. Add Exa Monitors Integration + +**Why:** Scheduled searches with webhook delivery, deduplicated against previous runs. + +**Steps:** +- [ ] Review Exa Monitors API +- [ ] Add `web_monitor(query, schedule, webhook_url)` tool +- [ ] Could replace or augment custom monitoring scripts + +**Effort:** 2 hours +**Dependencies:** Webhook endpoint for delivery + +--- + +### 13. Add Brave Goggles for Custom Reranking + +**Why:** Only search API that lets you boost/promote specific domains at query time. + +**Steps:** +- [ ] Create Goggles config for IT Pro Partner preferred domains +- [ ] Add `goggles` parameter to Brave search calls +- [ ] Test domain boosting effectiveness + +**Effort:** 1 hour +**Dependencies:** Brave Search API (Item #2) + +--- + +### 14. Add LLM Metadata Enrichment + +**Why:** Generate one-line semantic descriptions of extracted pages for better downstream RAG retrieval. + +**Steps:** +- [ ] Add `enrich_metadata: true` option to web_extract +- [ ] Use cheap local model or Firecrawl's built-in summarization +- [ ] Tag results with semantic descriptions +- [ ] Benchmark retrieval improvement + +**Effort:** 3-4 hours +**Dependencies:** None (can use Firecrawl's question format or local model) + +--- + +### 15. Evaluate Kagi Search API + +**Why:** Premium search quality. Worth a trial key for comparison benchmarking. + +**Steps:** +- [ ] Sign up for Kagi trial +- [ ] Run 50 side-by-side comparisons: Kagi vs Exa vs Brave vs SearXNG +- [ ] Score relevance, freshness, and coverage +- [ ] Decide: add to chain or pass + +**Effort:** 2 hours +**Dependencies:** Kagi API trial key + +--- + +### 16. Add OpenTelemetry Tracing + +**Why:** FastMCP 3.x has native OTEL -- spans for every search call, fallback path taken, extraction step. + +**Steps:** +- [ ] Install OpenTelemetry packages +- [ ] Run with `opentelemetry-instrument fastmcp run server.py` +- [ ] Configure export to local collector or file +- [ ] Analyze fallback chain behavior + +**Effort:** 1 hour +**Dependencies:** FastMCP 3.x upgrade (Item #1) + +--- + +## Execution Order (Dependency-Aware) + +``` +Phase 1 (Week 1): + Day 1: Items #1 (FastMCP 3.0) + #3 (Exa deprecations) -- can run in parallel + Day 2: Item #2 (Brave Search API) -- independent + Day 3: Item #4 (Crawl4AI) -- independent, longest install + Day 4: Testing + burn-in of Phase 1 changes + +Phase 2 (Week 2): + Item #5 (SSE -> Streamable HTTP) -- depends on #1 + Items #6 + #7 (extract_document + Lockdown Mode) -- parallel, both Firecrawl + Item #9 (result dedup) -- independent + +Phase 3 (Week 3): + Items #8 (Exa Agent) + #11 (Tavily) -- parallel, both new API integrations + Item #10 (4get-hijacked eval) -- independent + +Phase 4 (Week 4+): + Items #12-#16 -- low priority, pick up as time allows +``` + +--- + +## Risk Register + +| Risk | Likelihood | Impact | Mitigation | +|------|-----------|--------|------------| +| FastMCP 3.x breaking API changes | Medium | High | Pin 2.x, test exhaustively before deploy | +| Crawl4AI RAM usage with Chromium | Medium | Medium | Monitor RAM, consider Docker isolation | +| Exa Agent cost overruns | Low | Medium | Per-query cost estimate cap | +| Brave API rate limits | Low | Low | Free tier sufficient for testing | +| Streamable HTTP transport issues | Low | High | Test with Hermes Agent before cutting over | + +--- + +## Success Metrics + +- All 14 existing tools continue working post-upgrade +- Brave API adds independent fallback source (no Google/Bing dependency) +- Crawl4AI handles 3+ known-bot-protected sites that previously failed +- Result dedup eliminates >=80% of cross-provider duplicates +- Zero regressions in Hermes Agent's use of Super Search tools + +--- + +## Reference + +- Super Search server: `/root/docker/super-search/server.py` +- Systemd service: `super-search.service` +- Venv: `/root/docker/super-search/venv/` +- Health endpoint: `http://127.0.0.1:8899/health` +- Full audit: Aug 1, 2026 -- zero breaking patterns for FastMCP 4.0