60 lines
1.9 KiB
Markdown
60 lines
1.9 KiB
Markdown
# Super Search — MCP Server Architecture Pattern
|
|
|
|
Updated: 2026-07-12
|
|
|
|
## Architecture
|
|
|
|
```
|
|
Client (Hermes) → localhost:8899/mcp (FastMCP HTTP)
|
|
|
|
Tools:
|
|
web_search(query, limit=5) — multi-provider chain
|
|
web_extract(url) — Trafilatura → Firecrawl
|
|
web_search_premium(query) — Exa direct (DRE-grade)
|
|
```
|
|
|
|
## Provider Chain with Rate Limiting
|
|
|
|
Each provider has a token bucket rate limiter, TTL cache, and the chain falls through on failure:
|
|
|
|
```
|
|
web_search → 1. SearXNG (local, 30 req/s, 2min cache)
|
|
→ 2. Exa (paid, 10 req/s, 5min cache)
|
|
→ 3. OpenCorporates (free, 1 req/s, 1hr cache) — company records
|
|
→ 4. CourtListener (free, ~100/min, 1hr cache) — court cases
|
|
→ 5. Firecrawl (paid, 5 req/min, 5min cache)
|
|
```
|
|
|
|
On 429/rate-limit: exponential backoff respecting Retry-After header, then fall to next provider. On 5xx: immediate fallback.
|
|
|
|
## Files
|
|
|
|
- `/root/docker/super-search/server.py` — FastMCP server (v1.2.0+)
|
|
- `/root/docker/super-search/ratelimit.py` — TokenBucket, TTLCache, retry helpers
|
|
- Systemd service: `super-search.service`
|
|
- Venv: `/root/docker/super-search/venv`
|
|
|
|
## Rate Limit Configuration
|
|
|
|
```python
|
|
RATE_LIMITS = {
|
|
"searxng": {"tokens": 30, "interval": 1.0},
|
|
"exa": {"tokens": 10, "interval": 1.0},
|
|
"firecrawl": {"tokens": 5, "interval": 60.0},
|
|
"opencorporates": {"tokens": 1, "interval": 1.0},
|
|
"courtlistener": {"tokens": 10, "interval": 6.0},
|
|
}
|
|
CACHE_TTL = {
|
|
"searxng": 120, "exa": 300, "firecrawl": 300,
|
|
"opencorporates": 3600, "courtlistener": 3600,
|
|
}
|
|
```
|
|
|
|
## Adding a New Provider
|
|
|
|
1. Add rate limit and cache TTL to config dicts in ratelimit.py
|
|
2. Add `_newprovider_search()` function decorated with `@rate_limited_search("newprovider")`
|
|
3. Add to the `web_search` fallback chain in order
|
|
4. Bump version, restart service: `systemctl restart super-search`
|
|
5. Verify: `hermes mcp test super-search`
|