ippadmin/itpp-infrastructure
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
5dc2619497a0952a3ed9eef5d8d9daace75868e7
itpp-infrastructure/docs/super-search/super-search-enhancement-plan.md
T

11 KiB
Raw Blame History

Super Search MCP Enhancement Execution Plan

Created: 2026-08-07 Source: Super Search Enhancement Scanner (cron, Aug 7 2026) Status: OPEN

Overview

16 actionable enhancements identified for Super Search MCP (http://127.0.0.1:8899). Current stack: FastMCP 2.x, SearXNG Docker, Exa API, Firecrawl API, Trafilatura, DuckDuckGo fallback.

HIGH Priority (Execute First -- Weeks 1-2)

1. Upgrade FastMCP 2.x -> 3.x

Why: Provider architecture, component versioning, OpenTelemetry, tool timeouts, concurrent execution. Current 2.x is aging out.

Steps:

  • Pin current FastMCP version to freeze baseline
  • Review breaking changes in FastMCP 3.x changelog (v3.0 Feb 2026 -> v3.3.0 May 2026)
  • Upgrade in venv: pip install --upgrade fastmcp
  • Test all 14 Super Search tools individually
  • Test fallback chain behavior (SearXNG -> Exa -> DDG -> Firecrawl)
  • Verify health_check and circuit_status still work
  • Deploy and monitor for 48h

Risk: Medium -- API surface is largely compatible but component versioning may affect tool registration Effort: 3-4 hours Dependencies: None

2. Add Brave Search API to Fallback Chain

Why: Independent 40B+ page index (no Google/Bing dependency). $5/1K queries. LLM Context endpoint returns pre-formatted results for AI use.

Steps:

  • Sign up for Brave Search API free tier (2,000 queries/month)
  • Store API key in .env
  • Add search_brave(query, limit) to server.py using Brave Web Search endpoint
  • Insert between SearXNG and DuckDuckGo in fallback chain
  • Add to circuit_status tool
  • Add web_search_llm_context tool using Brave's LLM Context endpoint
  • Test with 20 queries and compare result quality vs Exa/SearXNG

Risk: Low -- independent API, no shared infra Effort: 2-3 hours Dependencies: Brave API key (free signup)

3. Fix Exa API Deprecations

Why: Exa deprecated pdf, github, tweet categories and replaced livecrawl with maxAgeHours. research paper -> publication.

Steps:

  • Audit server.py for all Exa category references
  • Replace category: "research paper" -> category: "publication" in web_search_academic
  • Remove pdf, github, tweet from category mapping logic
  • Replace livecrawl -> maxAgeHours in web_extract Exa path
  • Test academic search with new publication category (350M papers)
  • Test extraction with maxAgeHours parameter

Risk: Low -- straightforward replacements, Exa docs are clear Effort: 1 hour Dependencies: None

4. Add Crawl4AI as Self-Hosted Extraction Backend

Why: Free, no rate limits, stealth mode (undetected browser), JS rendering, parallel crawling. 77K GitHub stars. Handles bot-protected pages that Trafilatura and Firecrawl can't reach.

Steps:

  • Install Crawl4AI: pip install crawl4ai
  • Add web_extract_stealth(url) tool -- uses Playwright stealth mode for JS-heavy/bot-protected pages
  • Add web_extract_bulk(urls) tool -- parallel extraction for batch jobs
  • Wire into fallback chain: Trafilatura -> Firecrawl -> Crawl4AI
  • Test on known-bot-protected URLs (VRBO, Expedia, etc.)
  • Document Playwright dependency (may need playwright install chromium)

Risk: Medium -- adds Chromium/Playwright dependency (~300MB), may increase RAM usage Effort: 3-4 hours Dependencies: pip install crawl4ai playwright, playwright install chromium

MEDIUM Priority (Plan Next -- Weeks 3-4)

5. Migrate SSE -> Streamable HTTP Transport

Why: MCP spec (2026-07-28) deprecated SSE. Streamable HTTP works with standard CORS, auth, and load balancers.

Steps:

  • Verify FastMCP 3.x supports Streamable HTTP natively (it does)
  • Update server.py transport configuration
  • Test with Hermes Agent as MCP client
  • Verify Caddy reverse proxy still works
  • Update any client configurations pointing to SSE endpoint

Risk: Medium -- transport change affects all MCP clients Effort: 2 hours Dependencies: FastMCP 3.x upgrade (Item #1)

6. Add web_extract_document Tool

Why: Currently Super Search only handles URLs. Firecrawl /parse handles PDFs, Word docs, spreadsheets up to 50MB -> clean markdown.

Steps:

  • Add web_extract_document(file_url) tool wrapping Firecrawl /parse
  • Support PDF, DOCX, XLSX, PPTX formats
  • Return clean markdown with structured data where available
  • Add file size validation (max 50MB)
  • Test with sample PDF, Word doc, and spreadsheet

Risk: Low -- wraps existing Firecrawl endpoint Effort: 1-2 hours Dependencies: None

7. Add Firecrawl Lockdown Mode

Why: Zero-outbound-request extraction from Firecrawl cache. Critical for sensitive/sandboxed use cases.

Steps:

  • Add lockdown: true parameter to web_extract when using Firecrawl
  • Document that Lockdown Mode means no live outbound requests
  • Test that Lockdown Mode returns only cached/indexed content

Risk: Low -- feature flag on existing Firecrawl API Effort: 30 minutes Dependencies: None

8. Add Exa Agent as web_research_deep Tool

Why: Multi-step agentic research for complex queries. Exa Agent does recursive search + extraction + synthesis.

Steps:

  • Review Exa Agent API docs and pricing ($0.10/ACU + $0.005/search)
  • Add web_research_deep(query, effort="medium") tool
  • Support outputSchema for structured outputs
  • Add cost estimation before execution (warn if >$0.50 estimated)
  • Test with complex multi-step research query

Risk: Medium -- cost per query is higher, needs rate limiting Effort: 2-3 hours Dependencies: None

9. Add Result Deduplication Across Providers

Why: When multiple backends return the same URL, we serve duplicate results. Simple URL normalization + content hash dedup.

Steps:

  • Implement URL normalization (strip tracking params, trailing slashes, www prefix)
  • When merging results from multiple providers, hash URLs and deduplicate
  • Keep the best snippet/metadata per unique URL (prefer richer provider)
  • Add dedup_summary to response metadata (count of duplicates removed)
  • Test with queries that hit multiple providers

Risk: Low -- purely additive, no breaking changes Effort: 2 hours Dependencies: None

10. Evaluate 4get-hijacked for SearXNG

Why: Community project that proxies ~30 search engines into SearXNG-compatible format. Sidesteps broken major engine scrapers.

Steps:

  • Clone and review cra88y/4get-hijacked repo
  • Test integration with our SearXNG Docker instance
  • Benchmark result quality vs current engine pool
  • Decide: add to search engine list or pass

Risk: Low -- evaluation only, no commitment Effort: 1-2 hours Dependencies: None

11. Add Tavily as web_search_ai Tool

Why: Purpose-built AI search with relevance scores. Not for general fallback chain (higher cost/latency) but excellent for AI-optimized results.

Steps:

  • Sign up for Tavily free tier (1,000 queries/month)
  • Add web_search_ai(query, depth="advanced") as standalone tool
  • Return relevance-scored results with confidence markers
  • Do NOT add to fallback chain (keep as separate tool for explicit use)
  • Test against SearXNG/Exa for quality comparison

Risk: Low -- standalone tool, no chain impact Effort: 1.5 hours Dependencies: Tavily API key (free signup)

LOW Priority (Nice to Have -- Weeks 5+)

12. Add Exa Monitors Integration

Why: Scheduled searches with webhook delivery, deduplicated against previous runs.

Steps:

  • Review Exa Monitors API
  • Add web_monitor(query, schedule, webhook_url) tool
  • Could replace or augment custom monitoring scripts

Effort: 2 hours Dependencies: Webhook endpoint for delivery

13. Add Brave Goggles for Custom Reranking

Why: Only search API that lets you boost/promote specific domains at query time.

Steps:

  • Create Goggles config for IT Pro Partner preferred domains
  • Add goggles parameter to Brave search calls
  • Test domain boosting effectiveness

Effort: 1 hour Dependencies: Brave Search API (Item #2)

14. Add LLM Metadata Enrichment

Why: Generate one-line semantic descriptions of extracted pages for better downstream RAG retrieval.

Steps:

  • Add enrich_metadata: true option to web_extract
  • Use cheap local model or Firecrawl's built-in summarization
  • Tag results with semantic descriptions
  • Benchmark retrieval improvement

Effort: 3-4 hours Dependencies: None (can use Firecrawl's question format or local model)

15. Evaluate Kagi Search API

Why: Premium search quality. Worth a trial key for comparison benchmarking.

Steps:

  • Sign up for Kagi trial
  • Run 50 side-by-side comparisons: Kagi vs Exa vs Brave vs SearXNG
  • Score relevance, freshness, and coverage
  • Decide: add to chain or pass

Effort: 2 hours Dependencies: Kagi API trial key

16. Add OpenTelemetry Tracing

Why: FastMCP 3.x has native OTEL -- spans for every search call, fallback path taken, extraction step.

Steps:

  • Install OpenTelemetry packages
  • Run with opentelemetry-instrument fastmcp run server.py
  • Configure export to local collector or file
  • Analyze fallback chain behavior

Effort: 1 hour Dependencies: FastMCP 3.x upgrade (Item #1)

Execution Order (Dependency-Aware)

Phase 1 (Week 1):
  Day 1: Items #1 (FastMCP 3.0) + #3 (Exa deprecations) -- can run in parallel
  Day 2: Item #2 (Brave Search API) -- independent
  Day 3: Item #4 (Crawl4AI) -- independent, longest install
  Day 4: Testing + burn-in of Phase 1 changes

Phase 2 (Week 2):
  Item #5 (SSE -> Streamable HTTP) -- depends on #1
  Items #6 + #7 (extract_document + Lockdown Mode) -- parallel, both Firecrawl
  Item #9 (result dedup) -- independent

Phase 3 (Week 3):
  Items #8 (Exa Agent) + #11 (Tavily) -- parallel, both new API integrations
  Item #10 (4get-hijacked eval) -- independent

Phase 4 (Week 4+):
  Items #12-#16 -- low priority, pick up as time allows

Risk Register

Risk Likelihood Impact Mitigation
FastMCP 3.x breaking API changes Medium High Pin 2.x, test exhaustively before deploy
Crawl4AI RAM usage with Chromium Medium Medium Monitor RAM, consider Docker isolation
Exa Agent cost overruns Low Medium Per-query cost estimate cap
Brave API rate limits Low Low Free tier sufficient for testing
Streamable HTTP transport issues Low High Test with Hermes Agent before cutting over

Success Metrics

  • All 14 existing tools continue working post-upgrade
  • Brave API adds independent fallback source (no Google/Bing dependency)
  • Crawl4AI handles 3+ known-bot-protected sites that previously failed
  • Result dedup eliminates >=80% of cross-provider duplicates
  • Zero regressions in Hermes Agent's use of Super Search tools

Reference

  • Super Search server: /root/docker/super-search/server.py
  • Systemd service: super-search.service
  • Venv: /root/docker/super-search/venv/
  • Health endpoint: http://127.0.0.1:8899/health
  • Full audit: Aug 1, 2026 -- zero breaking patterns for FastMCP 4.0
Reference in New Issue View Git Blame Copy Permalink