# Super Search MCP Enhancement Execution Plan

**Created:** 2026-08-07
**Source:** Super Search Enhancement Scanner (cron, Aug 7 2026)
**Status:** OPEN

---

## Overview

16 actionable enhancements identified for Super Search MCP (http://127.0.0.1:8899). Current stack: FastMCP 2.x, SearXNG Docker, Exa API, Firecrawl API, Trafilatura, DuckDuckGo fallback.

---

## HIGH Priority (Execute First -- Weeks 1-2)

### 1. Upgrade FastMCP 2.x -> 3.x

**Why:** Provider architecture, component versioning, OpenTelemetry, tool timeouts, concurrent execution. Current 2.x is aging out.

**Steps:**
- [ ] Pin current FastMCP version to freeze baseline
- [ ] Review breaking changes in FastMCP 3.x changelog (v3.0 Feb 2026 -> v3.3.0 May 2026)
- [ ] Upgrade in venv: `pip install --upgrade fastmcp`
- [ ] Test all 14 Super Search tools individually
- [ ] Test fallback chain behavior (SearXNG -> Exa -> DDG -> Firecrawl)
- [ ] Verify health_check and circuit_status still work
- [ ] Deploy and monitor for 48h

**Risk:** Medium -- API surface is largely compatible but component versioning may affect tool registration
**Effort:** 3-4 hours
**Dependencies:** None

---

### 2. Add Brave Search API to Fallback Chain

**Why:** Independent 40B+ page index (no Google/Bing dependency). $5/1K queries. LLM Context endpoint returns pre-formatted results for AI use.

**Steps:**
- [ ] Sign up for Brave Search API free tier (2,000 queries/month)
- [ ] Store API key in `.env`
- [ ] Add `search_brave(query, limit)` to server.py using Brave Web Search endpoint
- [ ] Insert between SearXNG and DuckDuckGo in fallback chain
- [ ] Add to circuit_status tool
- [ ] Add `web_search_llm_context` tool using Brave's LLM Context endpoint
- [ ] Test with 20 queries and compare result quality vs Exa/SearXNG

**Risk:** Low -- independent API, no shared infra
**Effort:** 2-3 hours
**Dependencies:** Brave API key (free signup)

---

### 3. Fix Exa API Deprecations

**Why:** Exa deprecated `pdf`, `github`, `tweet` categories and replaced `livecrawl` with `maxAgeHours`. `research paper` -> `publication`.

**Steps:**
- [ ] Audit server.py for all Exa category references
- [ ] Replace `category: "research paper"` -> `category: "publication"` in `web_search_academic`
- [ ] Remove `pdf`, `github`, `tweet` from category mapping logic
- [ ] Replace `livecrawl` -> `maxAgeHours` in `web_extract` Exa path
- [ ] Test academic search with new `publication` category (350M papers)
- [ ] Test extraction with `maxAgeHours` parameter

**Risk:** Low -- straightforward replacements, Exa docs are clear
**Effort:** 1 hour
**Dependencies:** None

---

### 4. Add Crawl4AI as Self-Hosted Extraction Backend

**Why:** Free, no rate limits, stealth mode (undetected browser), JS rendering, parallel crawling. 77K GitHub stars. Handles bot-protected pages that Trafilatura and Firecrawl can't reach.

**Steps:**
- [ ] Install Crawl4AI: `pip install crawl4ai`
- [ ] Add `web_extract_stealth(url)` tool -- uses Playwright stealth mode for JS-heavy/bot-protected pages
- [ ] Add `web_extract_bulk(urls)` tool -- parallel extraction for batch jobs
- [ ] Wire into fallback chain: Trafilatura -> Firecrawl -> Crawl4AI
- [ ] Test on known-bot-protected URLs (VRBO, Expedia, etc.)
- [ ] Document Playwright dependency (may need `playwright install chromium`)

**Risk:** Medium -- adds Chromium/Playwright dependency (~300MB), may increase RAM usage
**Effort:** 3-4 hours
**Dependencies:** `pip install crawl4ai playwright`, `playwright install chromium`

---

## MEDIUM Priority (Plan Next -- Weeks 3-4)

### 5. Migrate SSE -> Streamable HTTP Transport

**Why:** MCP spec (2026-07-28) deprecated SSE. Streamable HTTP works with standard CORS, auth, and load balancers.

**Steps:**
- [ ] Verify FastMCP 3.x supports Streamable HTTP natively (it does)
- [ ] Update server.py transport configuration
- [ ] Test with Hermes Agent as MCP client
- [ ] Verify Caddy reverse proxy still works
- [ ] Update any client configurations pointing to SSE endpoint

**Risk:** Medium -- transport change affects all MCP clients
**Effort:** 2 hours
**Dependencies:** **FastMCP 3.x upgrade (Item #1)**

---

### 6. Add `web_extract_document` Tool

**Why:** Currently Super Search only handles URLs. Firecrawl `/parse` handles PDFs, Word docs, spreadsheets up to 50MB -> clean markdown.

**Steps:**
- [ ] Add `web_extract_document(file_url)` tool wrapping Firecrawl `/parse`
- [ ] Support PDF, DOCX, XLSX, PPTX formats
- [ ] Return clean markdown with structured data where available
- [ ] Add file size validation (max 50MB)
- [ ] Test with sample PDF, Word doc, and spreadsheet

**Risk:** Low -- wraps existing Firecrawl endpoint
**Effort:** 1-2 hours
**Dependencies:** None

---

### 7. Add Firecrawl Lockdown Mode

**Why:** Zero-outbound-request extraction from Firecrawl cache. Critical for sensitive/sandboxed use cases.

**Steps:**
- [ ] Add `lockdown: true` parameter to `web_extract` when using Firecrawl
- [ ] Document that Lockdown Mode means no live outbound requests
- [ ] Test that Lockdown Mode returns only cached/indexed content

**Risk:** Low -- feature flag on existing Firecrawl API
**Effort:** 30 minutes
**Dependencies:** None

---

### 8. Add Exa Agent as `web_research_deep` Tool

**Why:** Multi-step agentic research for complex queries. Exa Agent does recursive search + extraction + synthesis.

**Steps:**
- [ ] Review Exa Agent API docs and pricing ($0.10/ACU + $0.005/search)
- [ ] Add `web_research_deep(query, effort="medium")` tool
- [ ] Support `outputSchema` for structured outputs
- [ ] Add cost estimation before execution (warn if >$0.50 estimated)
- [ ] Test with complex multi-step research query

**Risk:** Medium -- cost per query is higher, needs rate limiting
**Effort:** 2-3 hours
**Dependencies:** None

---

### 9. Add Result Deduplication Across Providers

**Why:** When multiple backends return the same URL, we serve duplicate results. Simple URL normalization + content hash dedup.

**Steps:**
- [ ] Implement URL normalization (strip tracking params, trailing slashes, www prefix)
- [ ] When merging results from multiple providers, hash URLs and deduplicate
- [ ] Keep the best snippet/metadata per unique URL (prefer richer provider)
- [ ] Add `dedup_summary` to response metadata (count of duplicates removed)
- [ ] Test with queries that hit multiple providers

**Risk:** Low -- purely additive, no breaking changes
**Effort:** 2 hours
**Dependencies:** None

---

### 10. Evaluate 4get-hijacked for SearXNG

**Why:** Community project that proxies ~30 search engines into SearXNG-compatible format. Sidesteps broken major engine scrapers.

**Steps:**
- [ ] Clone and review `cra88y/4get-hijacked` repo
- [ ] Test integration with our SearXNG Docker instance
- [ ] Benchmark result quality vs current engine pool
- [ ] Decide: add to search engine list or pass

**Risk:** Low -- evaluation only, no commitment
**Effort:** 1-2 hours
**Dependencies:** None

---

### 11. Add Tavily as `web_search_ai` Tool

**Why:** Purpose-built AI search with relevance scores. Not for general fallback chain (higher cost/latency) but excellent for AI-optimized results.

**Steps:**
- [ ] Sign up for Tavily free tier (1,000 queries/month)
- [ ] Add `web_search_ai(query, depth="advanced")` as standalone tool
- [ ] Return relevance-scored results with confidence markers
- [ ] Do NOT add to fallback chain (keep as separate tool for explicit use)
- [ ] Test against SearXNG/Exa for quality comparison

**Risk:** Low -- standalone tool, no chain impact
**Effort:** 1.5 hours
**Dependencies:** Tavily API key (free signup)

---

## LOW Priority (Nice to Have -- Weeks 5+)

### 12. Add Exa Monitors Integration

**Why:** Scheduled searches with webhook delivery, deduplicated against previous runs.

**Steps:**
- [ ] Review Exa Monitors API
- [ ] Add `web_monitor(query, schedule, webhook_url)` tool
- [ ] Could replace or augment custom monitoring scripts

**Effort:** 2 hours
**Dependencies:** Webhook endpoint for delivery

---

### 13. Add Brave Goggles for Custom Reranking

**Why:** Only search API that lets you boost/promote specific domains at query time.

**Steps:**
- [ ] Create Goggles config for IT Pro Partner preferred domains
- [ ] Add `goggles` parameter to Brave search calls
- [ ] Test domain boosting effectiveness

**Effort:** 1 hour
**Dependencies:** Brave Search API (Item #2)

---

### 14. Add LLM Metadata Enrichment

**Why:** Generate one-line semantic descriptions of extracted pages for better downstream RAG retrieval.

**Steps:**
- [ ] Add `enrich_metadata: true` option to web_extract
- [ ] Use cheap local model or Firecrawl's built-in summarization
- [ ] Tag results with semantic descriptions
- [ ] Benchmark retrieval improvement

**Effort:** 3-4 hours
**Dependencies:** None (can use Firecrawl's question format or local model)

---

### 15. Evaluate Kagi Search API

**Why:** Premium search quality. Worth a trial key for comparison benchmarking.

**Steps:**
- [ ] Sign up for Kagi trial
- [ ] Run 50 side-by-side comparisons: Kagi vs Exa vs Brave vs SearXNG
- [ ] Score relevance, freshness, and coverage
- [ ] Decide: add to chain or pass

**Effort:** 2 hours
**Dependencies:** Kagi API trial key

---

### 16. Add OpenTelemetry Tracing

**Why:** FastMCP 3.x has native OTEL -- spans for every search call, fallback path taken, extraction step.

**Steps:**
- [ ] Install OpenTelemetry packages
- [ ] Run with `opentelemetry-instrument fastmcp run server.py`
- [ ] Configure export to local collector or file
- [ ] Analyze fallback chain behavior

**Effort:** 1 hour
**Dependencies:** FastMCP 3.x upgrade (Item #1)

---

## Execution Order (Dependency-Aware)

```
Phase 1 (Week 1):
  Day 1: Items #1 (FastMCP 3.0) + #3 (Exa deprecations) -- can run in parallel
  Day 2: Item #2 (Brave Search API) -- independent
  Day 3: Item #4 (Crawl4AI) -- independent, longest install
  Day 4: Testing + burn-in of Phase 1 changes

Phase 2 (Week 2):
  Item #5 (SSE -> Streamable HTTP) -- depends on #1
  Items #6 + #7 (extract_document + Lockdown Mode) -- parallel, both Firecrawl
  Item #9 (result dedup) -- independent

Phase 3 (Week 3):
  Items #8 (Exa Agent) + #11 (Tavily) -- parallel, both new API integrations
  Item #10 (4get-hijacked eval) -- independent

Phase 4 (Week 4+):
  Items #12-#16 -- low priority, pick up as time allows
```

---

## Risk Register

| Risk | Likelihood | Impact | Mitigation |
|------|-----------|--------|------------|
| FastMCP 3.x breaking API changes | Medium | High | Pin 2.x, test exhaustively before deploy |
| Crawl4AI RAM usage with Chromium | Medium | Medium | Monitor RAM, consider Docker isolation |
| Exa Agent cost overruns | Low | Medium | Per-query cost estimate cap |
| Brave API rate limits | Low | Low | Free tier sufficient for testing |
| Streamable HTTP transport issues | Low | High | Test with Hermes Agent before cutting over |

---

## Success Metrics

- All 14 existing tools continue working post-upgrade
- Brave API adds independent fallback source (no Google/Bing dependency)
- Crawl4AI handles 3+ known-bot-protected sites that previously failed
- Result dedup eliminates >=80% of cross-provider duplicates
- Zero regressions in Hermes Agent's use of Super Search tools

---

## Reference

- Super Search server: `/root/docker/super-search/server.py`
- Systemd service: `super-search.service`
- Venv: `/root/docker/super-search/venv/`
- Health endpoint: `http://127.0.0.1:8899/health`
- Full audit: Aug 1, 2026 -- zero breaking patterns for FastMCP 4.0