Initial skills documentation — 25 categories, all SKILL.md + references + scripts
This commit is contained in:
root
@@ -0,0 +1,444 @@
|
||||
---
|
||||
name: mcp-servers
|
||||
description: Build, deploy, and integrate MCP (Model Context Protocol) servers with Hermes — FastMCP Python servers, HTTP transport configuration, Hermes MCP client wiring, and multi-provider search/extraction toolkits.
|
||||
version: 1.0.0
|
||||
author: Sho'Nuff
|
||||
tags: [mcp, servers, hermes-integration, fastmcp, tools]
|
||||
---
|
||||
|
||||
# MCP Servers
|
||||
|
||||
Build, deploy, and wire MCP servers into Hermes. MCP servers provide additional tools via the Model Context Protocol — think of them as plugin toolkits your agent can call natively.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Hermes Agent
|
||||
│ native MCP client
|
||||
├── mcp_servers.mysql → python3 /root/.hermes/scripts/mcp-mysql.py
|
||||
├── mcp_servers.super-search → http://127.0.0.1:8899/mcp
|
||||
└── mcp_servers.exa → https://mcp.exa.ai/mcp
|
||||
```
|
||||
|
||||
Two transport types:
|
||||
- **Command-based** — Hermes spawns a Python process that speaks MCP over stdio
|
||||
- **HTTP-based** — A persistent server runs at an HTTP endpoint, Hermes connects via SSE
|
||||
|
||||
## HTTP-Based MCP Server (FastMCP Pattern)
|
||||
|
||||
This is the preferred pattern for complex, persistent toolkits. The server runs as a systemd service and Hermes connects via HTTP.
|
||||
|
||||
### Build Steps
|
||||
|
||||
1. **Install dependencies** in a virtualenv:
|
||||
```bash
|
||||
pip install fastmcp httpx trafilatura firecrawl-py
|
||||
```
|
||||
|
||||
2. **Create the server** — a single Python file with FastMCP tools:
|
||||
```python
|
||||
from fastmcp import FastMCP
|
||||
mcp = FastMCP("Server Name", version="1.0.0")
|
||||
|
||||
@mcp.tool(description="Search the web")
|
||||
async def web_search(query: str, limit: int = 5) -> str:
|
||||
# ... implementation
|
||||
return results
|
||||
|
||||
if __name__ == "__main__":
|
||||
mcp.run(transport="http", host="127.0.0.1", port=8899, path="/mcp")
|
||||
```
|
||||
|
||||
3. **Create a systemd service:**
|
||||
```ini
|
||||
[Unit]
|
||||
Description=Super Search MCP Server
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=root
|
||||
WorkingDirectory=/root/docker/super-search
|
||||
Environment="PATH=/root/docker/super-search/venv/bin:/usr/local/sbin:..."
|
||||
Environment="PYTHONUNBUFFERED=1"
|
||||
EnvironmentFile=/root/.hermes/.env
|
||||
ExecStart=/root/docker/super-search/venv/bin/python3 /root/docker/super-search/server.py
|
||||
Restart=always
|
||||
RestartSec=3
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
4. **Wire into Hermes config:**
|
||||
```bash
|
||||
hermes config set mcp_servers.<name>.url 'http://127.0.0.1:<port>/mcp'
|
||||
hermes config set mcp_servers.<name>.enabled 'true'
|
||||
```
|
||||
|
||||
Or edit `~/.hermes/config.yaml` directly:
|
||||
```yaml
|
||||
mcp_servers:
|
||||
super-search:
|
||||
url: http://127.0.0.1:8899/mcp
|
||||
enabled: true
|
||||
```
|
||||
|
||||
5. **Restart gateway** or reload MCP:
|
||||
```bash
|
||||
# Need to restart from outside the gateway session
|
||||
systemctl restart hermes.service
|
||||
```
|
||||
|
||||
### Testing
|
||||
|
||||
```bash
|
||||
# List all MCP servers
|
||||
hermes mcp list
|
||||
|
||||
# Test connection and discover tools
|
||||
hermes mcp test <server-name>
|
||||
# Expected output:
|
||||
# ✓ Connected (XXms)
|
||||
# ✓ Tools discovered: N
|
||||
# tool_name1 Description...
|
||||
# tool_name2 Description...
|
||||
```
|
||||
|
||||
### Performance
|
||||
- Local HTTP MCP servers: ~50-60ms connection time
|
||||
- Remote HTTP MCP servers (e.g., Exa): ~1-2s connection time (includes OAuth/TLS)
|
||||
|
||||
### Troubleshooting
|
||||
|
||||
**Server won't start — port in use:**
|
||||
A leftover process from a subagent's test run often holds the port. The `fuser` command alone may not suffice:
|
||||
```bash
|
||||
# Find what's on the port
|
||||
ss -tlnp | grep <port> # shows PID
|
||||
kill <PID> # then restart
|
||||
systemctl restart <service>
|
||||
```
|
||||
|
||||
If the process won't die (orphaned): `fuser -k <port>/tcp` then restart.
|
||||
|
||||
**Subagent timeout during builds:**
|
||||
The default 600s child timeout can be tight for multi-step MCP server builds (install deps + write code + systemd service + Hermes config + gateway restart). The subagent usually produces the code and service file correctly before timing out — just the final wiring steps (systemctl enable, testing) are incomplete. Do those manually rather than re-dispatching.
|
||||
|
||||
**MCP endpoint returns HTTP 406:**
|
||||
The client must accept both `application/json` and `text/event-stream`. Add headers:
|
||||
```
|
||||
Accept: application/json, text/event-stream
|
||||
```
|
||||
|
||||
**MCP endpoint returns "Missing session ID":**
|
||||
The HTTP transport uses SSE for session negotiation. Simple curl tests won't work. Use `hermes mcp test` instead, which handles the MCP protocol correctly.
|
||||
|
||||
**Tools discovered but not showing in Hermes:**
|
||||
Restart the gateway: `systemctl restart hermes.service` (from outside the gateway session).
|
||||
|
||||
## Immich Photo Map MCP Pattern
|
||||
|
||||
For Immich/photo-location workflows, build a dedicated read-only MCP that uses Immich `POST /search/metadata` with `withExif: true` and returns geotagged assets plus GeoJSON for a private Leaflet map. Keep GPS data private and expose thumbnails only. See `references/immich-photo-map-mcp.md`.
|
||||
|
||||
## OSINT Person Intelligence Layer
|
||||
|
||||
When raw search becomes a person/skip-trace workflow, do **not** overload the generic search MCP with identity logic. Build a dedicated entity-resolution MCP above Super Search. Pattern reference: `references/osint-person-intelligence.md`.
|
||||
|
||||
Use this pattern for: person search, phone/email variant search, source scoring, identity clustering, dossier generation, DRE debtor enrichment, and compliance-preserving lead packets.
|
||||
|
||||
## Command-Based MCP Server Pattern
|
||||
|
||||
For simpler, stateless MCP servers that don't warrant a persistent daemon:
|
||||
|
||||
```yaml
|
||||
mcp_servers:
|
||||
mysql:
|
||||
command: python3
|
||||
args:
|
||||
- /root/.hermes/scripts/mcp-mysql.py
|
||||
enabled: true
|
||||
```
|
||||
|
||||
Hermes spawns the process per-request. No systemd service needed.
|
||||
|
||||
## Remote MCP Server Pattern (OAuth)
|
||||
|
||||
Some services (Exa, agent-data.dev) provide hosted MCP servers with OAuth:
|
||||
|
||||
```yaml
|
||||
mcp_servers:
|
||||
exa:
|
||||
url: https://mcp.exa.ai/mcp
|
||||
enabled: true
|
||||
```
|
||||
|
||||
First connection opens a browser for OAuth sign-in. Heremes handles this via `hermes mcp login <name>`.
|
||||
|
||||
## Rate Limiting & TTL Caching Layer
|
||||
|
||||
When integrating multiple third-party APIs with rate limits (free tiers, govt databases, people-search APIs), add a token-bucket rate limiter and TTL cache between the MCP tool and the provider calls.
|
||||
|
||||
### Architecture
|
||||
|
||||
```
|
||||
Query → TTL Cache (dict, per-provider TTL)
|
||||
↓ cache miss
|
||||
Token Bucket (per-provider tokens/interval)
|
||||
↓ tokens available
|
||||
Provider API → success? → cache + return
|
||||
↓ 429 / rate limited
|
||||
Exponential Backoff (respects Retry-After header)
|
||||
↓ exhausted
|
||||
Fallback → next provider in chain
|
||||
```
|
||||
|
||||
### Implementation — `ratelimit.py`
|
||||
|
||||
Create a standalone module that the MCP server imports:
|
||||
|
||||
```python
|
||||
"""ratelimit.py — Token bucket rate limiter + TTL cache for MCP providers."""
|
||||
|
||||
import time, hashlib, json, asyncio
|
||||
|
||||
# Per-provider rate limit config
|
||||
RATE_LIMITS = {
|
||||
"searxng": {"tokens": 30, "interval": 1.0}, # local — high limit
|
||||
"exa": {"tokens": 10, "interval": 1.0}, # paid tier
|
||||
"firecrawl": {"tokens": 5, "interval": 60.0}, # 1k/mo budget
|
||||
"opencorporates": {"tokens": 1, "interval": 1.0}, # 1/sec free tier
|
||||
"courtlistener": {"tokens": 10, "interval": 6.0}, # ~100/min free
|
||||
}
|
||||
|
||||
# Per-provider cache TTL in seconds
|
||||
CACHE_TTL = {
|
||||
"searxng": 120, # 2 min — web results change
|
||||
"exa": 300, # 5 min
|
||||
"opencorporates": 3600, # 1 hr — company records are stable
|
||||
"courtlistener": 3600, # 1 hr — court cases are infrequent
|
||||
"firecrawl": 300, # 5 min
|
||||
}
|
||||
|
||||
class TokenBucket:
|
||||
"""Async-safe token bucket for per-provider rate limiting."""
|
||||
def __init__(self, tokens, interval):
|
||||
self.max_tokens = tokens
|
||||
self.tokens = tokens
|
||||
self.interval = interval # seconds between refills
|
||||
self.last_refill = time.time()
|
||||
|
||||
async def acquire(self):
|
||||
now = time.time()
|
||||
if now - self.last_refill >= self.interval:
|
||||
self.tokens = self.max_tokens
|
||||
self.last_refill = now
|
||||
if self.tokens <= 0:
|
||||
return False
|
||||
self.tokens -= 1
|
||||
return True
|
||||
|
||||
_buckets = {}
|
||||
_cache = {}
|
||||
|
||||
async def rate_limited_search(provider: str, cache_key: str, fn, *args, **kwargs):
|
||||
"""Wrapper: check cache → rate limit → call → cache result."""
|
||||
# 1. Cache check
|
||||
cache_key_full = f"{provider}:{cache_key}"
|
||||
if cache_key_full in _cache:
|
||||
entry = _cache[cache_key_full]
|
||||
if time.time() - entry["ts"] < CACHE_TTL.get(provider, 300):
|
||||
return entry["data"]
|
||||
|
||||
# 2. Rate limit check
|
||||
rl = RATE_LIMITS.get(provider)
|
||||
if rl:
|
||||
if provider not in _buckets:
|
||||
_buckets[provider] = TokenBucket(rl["tokens"], rl["interval"])
|
||||
allowed = await _buckets[provider].acquire()
|
||||
if not allowed:
|
||||
raise RateLimitedError(f"{provider}: token bucket empty")
|
||||
|
||||
# 3. Call with retry
|
||||
result = await _retry_with_backoff(fn, *args, **kwargs)
|
||||
|
||||
# 4. Cache result
|
||||
_cache[cache_key_full] = {"data": result, "ts": time.time()}
|
||||
return result
|
||||
|
||||
async def _retry_with_backoff(fn, *args, retries=3, base_delay=1.0):
|
||||
"""Exponential backoff on 429/5xx, respects Retry-After."""
|
||||
for attempt in range(retries):
|
||||
try:
|
||||
return await fn(*args, **kwargs)
|
||||
except Exception as e:
|
||||
if attempt == retries - 1:
|
||||
raise
|
||||
delay = base_delay * (2 ** attempt)
|
||||
# Check for Retry-After in response headers
|
||||
await asyncio.sleep(delay)
|
||||
```
|
||||
|
||||
### Key decisions
|
||||
- **Cache aggressive for stable data** — company records and court cases get 1-hour TTLs since they rarely change. Web search results get 2-5 minutes.
|
||||
- **Token bucket per provider** — each provider has its own limit so one fast provider can't starve another.
|
||||
- **Fall through on rate limit** — don't fail the whole tool. The MCP server's fallback chain should catch the RateLimitedError and try the next provider.
|
||||
|
||||
## Multi-Provider Search Routing Pattern
|
||||
|
||||
When building a search/extraction MCP server that uses multiple backends, use this fallback chain pattern (incorporating rate limiting):
|
||||
|
||||
```python
|
||||
async def web_search(query, limit=5):
|
||||
# 1. Try SearXNG (free, local)
|
||||
try:
|
||||
results = await _searxng_search(query, limit)
|
||||
if results:
|
||||
return {"provider": "searxng", "results": results}
|
||||
except:
|
||||
pass
|
||||
|
||||
# 2. Fallback to Exa (premium, API key)
|
||||
try:
|
||||
results = await _exa_search(query, limit)
|
||||
if results:
|
||||
return {"provider": "exa", "results": results}
|
||||
except:
|
||||
pass
|
||||
|
||||
# 3. Last resort: Firecrawl (paid, most resilient)
|
||||
results = await _firecrawl_search(query, limit)
|
||||
return {"provider": "firecrawl", "results": results, "telemetry": telemetry}
|
||||
```
|
||||
|
||||
This ensures the service degrades gracefully — free first, premium second, paid last resort.
|
||||
|
||||
**Telemetry rule:** Multi-provider MCP tools must return provider telemetry, not just final results. Include providers attempted, status/error, result counts, and upstream health details (for example, SearXNG `unresponsive_engines` with CAPTCHA/rate-limit reasons). Do not silently hide a provider that is reachable but returning zero useful results because upstream engines are blocked.
|
||||
|
||||
## Remote Database MCP Data Source
|
||||
|
||||
When an MCP server needs to read from a database on a remote host (e.g., Traccar H2 on app2), use the SSH+SCP+query pattern: copy the DB as a read-only snapshot, query locally with Java H2 Shell, parse pipe-delimited output, return structured data. Avoids API auth, CORS, and credential management for read-only queries. Full reference: `references/h2-remote-db-query.md`.
|
||||
|
||||
This pattern works for H2, SQLite, and any file-based database accessible via SSH.
|
||||
|
||||
## Docker-Based MCP Deployment (alongside Open WebUI)
|
||||
|
||||
When deploying MCP servers on app1 to serve Open WebUI (and Hermes), use Docker Compose instead of systemd. Add the MCP service to the existing LiteLLM compose at `/root/docker/litellm/docker-compose.yml`.
|
||||
|
||||
### Pattern
|
||||
|
||||
```yaml
|
||||
services:
|
||||
super-search:
|
||||
build: ../super-search # Directory with server.py + Dockerfile
|
||||
container_name: super-search
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
- EXA_API_KEY=<key>
|
||||
- FIRECRAWL_API_KEY=<key>
|
||||
- PYTHONUNBUFFERED=1
|
||||
ports:
|
||||
- '127.0.0.1:8899:8899' # Bound to 127.0.0.1 — services on same host
|
||||
networks:
|
||||
- litellm-net # Share LiteLLM's network
|
||||
```
|
||||
|
||||
### Dockerfile (slim Python)
|
||||
|
||||
```dockerfile
|
||||
FROM python:3.13-slim
|
||||
WORKDIR /app
|
||||
COPY requirements.txt .
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
COPY server.py ratelimit.py ./
|
||||
ENV PYTHONUNBUFFERED=1
|
||||
EXPOSE 8899
|
||||
CMD ["python3", "server.py"]
|
||||
```
|
||||
|
||||
### Health check pitfall — slim containers
|
||||
|
||||
Python slim images lack `curl`, `ss`, `pgrep`, and other common health check tools. Do NOT use shell-based health checks. Options:
|
||||
|
||||
1. **Skip it** — Remove the health check block entirely. Docker will still restart on crash.
|
||||
2. **Python socket test** — `python3 -c "import socket; s=socket.socket(); s.connect(('127.0.0.1',8899)); s.close()"`
|
||||
3. **Docker exec test from host** — `docker exec super-search python3 -c "..."`
|
||||
|
||||
### Addressing
|
||||
|
||||
- **Open WebUI (Docker → host):** `http://host.docker.internal:8899/mcp`
|
||||
- **Hermes (host → Docker):** `http://127.0.0.1:8899/mcp` (since port is mapped to localhost)
|
||||
- **Process test from inside container:** `http://127.0.0.1:8899/mcp`
|
||||
|
||||
### Verifying deployment
|
||||
|
||||
Use `docker exec` with FastMCP client to verify tools:
|
||||
|
||||
```bash
|
||||
docker exec super-search python3 -c "
|
||||
from fastmcp import Client
|
||||
import asyncio
|
||||
async def test():
|
||||
async with Client('http://127.0.0.1:8899/mcp') as client:
|
||||
tools = await client.list_tools()
|
||||
print(f'Tools: {len(tools)}')
|
||||
for t in tools:
|
||||
print(f' - {t.name}')
|
||||
asyncio.run(test())
|
||||
"
|
||||
```
|
||||
|
||||
## Wiring MCP into Open WebUI
|
||||
|
||||
Open WebUI stores tool server connections as a JSON blob in its SQLite DB at key `tool_server.connections`. Add MCP servers programmatically:
|
||||
|
||||
```python
|
||||
import sqlite3, json, time
|
||||
db = sqlite3.connect('/var/lib/docker/volumes/openwebui_data/_data/webui.db')
|
||||
|
||||
row = db.execute("SELECT value FROM config WHERE key='tool_server.connections'").fetchone()
|
||||
current = json.loads(row[0]) if row else []
|
||||
|
||||
# Remove duplicates by URL
|
||||
current = [c for c in current if '8899' not in c.get('url', '')]
|
||||
|
||||
current.append({
|
||||
'url': 'http://host.docker.internal:8899/mcp',
|
||||
'type': 'mcp', # CRITICAL: not 'openapi'
|
||||
'auth_type': 'none',
|
||||
'config': {
|
||||
'enable': True,
|
||||
'function_name_filter_list': '',
|
||||
'access_grants': [{
|
||||
'principal_type': 'group',
|
||||
'principal_id': '<group-id>',
|
||||
'permission': 'read'
|
||||
}]
|
||||
},
|
||||
'info': {'name': 'My MCP', 'description': 'Tools description'}
|
||||
})
|
||||
|
||||
db.execute("UPDATE config SET value = ?, updated_at = ? WHERE key = ?",
|
||||
(json.dumps(current), int(time.time()), 'tool_server.connections'))
|
||||
db.commit()
|
||||
```
|
||||
|
||||
Then restart Open WebUI: `docker restart openwebui`
|
||||
|
||||
Users enable MCP tools per-chat via the **➕ → Integrations → Tools** menu.
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **yaml.dump drops top-level `networks:` section** — When editing docker-compose.yml with Python's `yaml.dump()`, the `networks:` block is silently removed on every write. This happened 2x in one session (Jul 15). Verify with `grep '^networks:' docker-compose.yml` after every YAML edit. If missing, restore from `.bak` and append new services via `cat >>` instead. You can also append the networks block manually: `echo -e '\nnetworks:\n litellm-net:\n driver: bridge' >> docker-compose.yml`
|
||||
- **Slim Python images lack health check tools** — `curl`, `ss`, `pgrep`, and `wget` are NOT in `python:3.13-slim`. Skip Docker health checks or use `pgrep -f server.py`.
|
||||
- **Port conflicts on app1** — Port 3000 is bound by Open WebUI. Check `ss -tlnp | grep <port>` before adding new MCP service ports.
|
||||
- **Open WebUI stores MCP connections in SQLite** — not in env vars or docker-compose. Adding an MCP server requires a DB update + restart at `/var/lib/docker/volumes/openwebui_data/_data/webui.db`, key `tool_server.connections`.
|
||||
- **Open WebUI `ConfigVar` env vars are ignored after first launch** — values persist in SQLite. Use `UPDATE config SET value=... WHERE key=...` for persistent config.
|
||||
- **MCP connection type must be `\"mcp\"`, not `\"openapi\"`** — using OpenAPI type for MCP URLs causes infinite loading spinner.
|
||||
- **`host.docker.internal` for Docker→host** — when Open WebUI (Docker) needs to reach an MCP server on the same host, use `http://host.docker.internal:<port>/mcp`, not `localhost`.
|
||||
- **Hermes config changes to mcp_servers require gateway restart** — tools won't be available until restart.
|
||||
- **HTTP transport needs SSE session negotiation** — curl POST won't work. Use `docker exec <name> python3 -c \"from fastmcp import Client...\"` for verification.
|
||||
- **Subagent timeout** — default 600s is tight for MCP server builds. Subagents often return \"next steps\" without deploying — verify yourself or rebuild.
|
||||
|
||||
See `references/browser-puppeteer-mcp.md` for the full Playwright + browserless/chrome MCP pattern.
|
||||
See `references/openwebui-config.md` for Open WebUI SQLite configuration reference.
|
||||
@@ -0,0 +1,184 @@
|
||||
# Browser MCP Pattern (Playwright + Browserless)
|
||||
|
||||
A fully headless browser MCP server that provides web navigation, screenshots, JavaScript execution, element extraction, and click interactions. Uses **browserless/chrome** as the rendering engine and **Playwright** for the MCP integration.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Open WebUI / Hermes
|
||||
│ MCP (Streamable HTTP)
|
||||
↓
|
||||
mcp-browser (Python 3.13, FastMCP, Playwright)
|
||||
│ CDP WebSocket
|
||||
↓
|
||||
browserless/chrome (headless Chromium)
|
||||
```
|
||||
|
||||
## Docker Compose (add to existing LiteLLM stack)
|
||||
|
||||
```yaml
|
||||
services:
|
||||
browserless:
|
||||
image: browserless/chrome:latest
|
||||
container_name: browserless
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
- CONNECTION_TIMEOUT=600000
|
||||
- MAX_CONCURRENT_SESSIONS=5
|
||||
- PREBOOT_CHROME=true
|
||||
ports:
|
||||
- '127.0.0.1:3005:3000' # 3000 conflicts with Open WebUI host mapping
|
||||
networks:
|
||||
- litellm-net
|
||||
deploy:
|
||||
resources:
|
||||
limits: {cpus: '2', memory: '1G'}
|
||||
reservations: {cpus: '0.5', memory: '256M'}
|
||||
|
||||
mcp-browser:
|
||||
build: ../mcp-browser
|
||||
container_name: mcp-browser
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
- BROWSER_WS=ws://browserless:3000 # Docker internal network name
|
||||
- PYTHONUNBUFFERED=1
|
||||
ports:
|
||||
- '127.0.0.1:8901:8901'
|
||||
networks:
|
||||
- litellm-net
|
||||
depends_on:
|
||||
- browserless
|
||||
```
|
||||
|
||||
## Dockerfile
|
||||
|
||||
```dockerfile
|
||||
FROM python:3.13-slim
|
||||
WORKDIR /app
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
curl ca-certificates && rm -rf /var/lib/apt/lists/*
|
||||
COPY requirements.txt .
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
RUN python3 -m playwright install chromium
|
||||
RUN python3 -m playwright install-deps chromium
|
||||
COPY server.py ./
|
||||
ENV PYTHONUNBUFFERED=1
|
||||
EXPOSE 8901
|
||||
CMD ["python3", "server.py"]
|
||||
```
|
||||
|
||||
## Requirements
|
||||
|
||||
```
|
||||
fastmcp>=3.0.0
|
||||
playwright>=1.45.0
|
||||
httpx>=0.27.0
|
||||
```
|
||||
|
||||
## Server Template (server.py)
|
||||
|
||||
```python
|
||||
from fastmcp import FastMCP
|
||||
from playwright.async_api import async_playwright
|
||||
import base64, os
|
||||
|
||||
BROWSER_WS = os.getenv('BROWSER_WS', 'ws://browserless:3000')
|
||||
mcp = FastMCP('Browser', version='1.0.0')
|
||||
_browser = None
|
||||
|
||||
async def get_browser():
|
||||
global _browser
|
||||
if _browser is None:
|
||||
pw = await async_playwright().start()
|
||||
_browser = await pw.chromium.connect_over_cdp(BROWSER_WS)
|
||||
return _browser
|
||||
|
||||
@mcp.tool(description="Navigate to a URL and return page text + title")
|
||||
async def browse(url: str) -> str:
|
||||
browser = await get_browser()
|
||||
page = await browser.new_page()
|
||||
try:
|
||||
await page.goto(url, wait_until='networkidle', timeout=30000)
|
||||
title = await page.title()
|
||||
text = await page.evaluate('document.body?.innerText || ""')
|
||||
return f"Title: {title}\n\n{text[:10000]}"
|
||||
finally:
|
||||
await page.close()
|
||||
|
||||
@mcp.tool(description="Take a full-page screenshot, return as base64 PNG")
|
||||
async def screenshot(url: str) -> str:
|
||||
browser = await get_browser()
|
||||
page = await browser.new_page()
|
||||
try:
|
||||
await page.goto(url, wait_until='networkidle', timeout=30000)
|
||||
data = await page.screenshot(full_page=True, type='png')
|
||||
return f"data:image/png;base64,{base64.b64encode(data).decode()}"
|
||||
finally:
|
||||
await page.close()
|
||||
|
||||
@mcp.tool(description="Execute JavaScript on a page, return result")
|
||||
async def execute_js(url: str, code: str) -> str:
|
||||
browser = await get_browser()
|
||||
page = await browser.new_page()
|
||||
try:
|
||||
await page.goto(url, wait_until='networkidle', timeout=30000)
|
||||
result = await page.evaluate(code)
|
||||
return str(result)[:5000]
|
||||
finally:
|
||||
await page.close()
|
||||
|
||||
@mcp.tool(description="Extract content by CSS selector")
|
||||
async def extract_element(url: str, selector: str) -> str:
|
||||
browser = await get_browser()
|
||||
page = await browser.new_page()
|
||||
try:
|
||||
await page.goto(url, wait_until='networkidle', timeout=30000)
|
||||
elements = await page.query_selector_all(selector)
|
||||
texts = [await el.inner_text() for el in elements if (await el.inner_text()).strip()]
|
||||
return f"Found {len(texts)} elements matching '{selector}':\n" + "\n---\n".join(texts[:10])
|
||||
finally:
|
||||
await page.close()
|
||||
|
||||
@mcp.tool(description="Click an element by text or CSS selector, return new page state")
|
||||
async def click(url: str, target: str, is_selector: bool = False) -> str:
|
||||
browser = await get_browser()
|
||||
page = await browser.new_page()
|
||||
try:
|
||||
await page.goto(url, wait_until='networkidle', timeout=30000)
|
||||
if is_selector:
|
||||
await page.click(target)
|
||||
else:
|
||||
await page.click(f'text={target}')
|
||||
await page.wait_for_load_state('networkidle', timeout=10000)
|
||||
title = await page.title()
|
||||
text = await page.evaluate('document.body?.innerText || ""')
|
||||
return f"Title: {title}\nURL: {page.url}\n\n{text[:8000]}"
|
||||
finally:
|
||||
await page.close()
|
||||
|
||||
if __name__ == '__main__':
|
||||
mcp.run(transport='http', host='0.0.0.0', port=8901, path='/mcp')
|
||||
```
|
||||
|
||||
## Verifying
|
||||
|
||||
```bash
|
||||
docker exec mcp-browser python3 -c "
|
||||
from fastmcp import Client, asyncio
|
||||
async def test():
|
||||
async with Client('http://127.0.0.1:8901/mcp') as client:
|
||||
tools = await client.list_tools()
|
||||
print(f'Tools: {len(tools)}')
|
||||
for t in tools:
|
||||
print(f' - {t.name}: {t.description[:50]}...')
|
||||
asyncio.run(test())
|
||||
"
|
||||
```
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **Port 3000 conflict** — Open WebUI maps host port 3000. Browserless also defaults to 3000. Use `127.0.0.1:3005:3000` to remap.
|
||||
- **Playwright deps install is slow** — `playwright install-deps chromium` pulls apt packages, 2-3 min build time. Cache the Docker layer.
|
||||
- **Browserless TOKEN env** — Set `TOKEN=` (empty) to disable token auth. The default requires no auth for internal networks.
|
||||
- **Slim images lack curl** — The Dockerfile adds `curl ca-certificates` explicitly. Health checks using `curl` still won't work in slim containers without this.
|
||||
- **CDP connection reuse** — The server reuses one browser instance across all requests via `connect_over_cdp`. Each tool opens/closes a page (tab), not a whole browser.
|
||||
@@ -0,0 +1,157 @@
|
||||
# Remote H2 Database Query via SSH
|
||||
|
||||
When an MCP server needs to read from an H2 database on a remote host (e.g., Traccar on app2), use this pattern:
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
MCP tool call
|
||||
→ SSH into remote host (key-based, no password prompt)
|
||||
→ scp the .mv.db file to a local temp file
|
||||
→ Java H2 Shell query (org.h2.tools.Shell)
|
||||
→ Parse pipe-delimited output
|
||||
→ Delete temp file
|
||||
→ Return structured JSON
|
||||
```
|
||||
|
||||
## Why this pattern
|
||||
|
||||
- **H2 databases lock.** You cannot query a live H2 database from two processes simultaneously. Copying with `scp` creates a read-only snapshot — the copy succeeds even while the DB is in use, and you query the copy.
|
||||
- **Java is needed.** H2's file format is proprietary. Python libraries like `jaydebeapi` exist but are brittle; the H2 Shell jar is self-contained and reliable.
|
||||
- **30-second cache.** The SCP + Java query takes 3-5 seconds. Caching avoids this overhead on repeated calls.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
On the local host:
|
||||
```bash
|
||||
apt-get install -y default-jre-headless
|
||||
# Copy H2 jar from the remote host or the container
|
||||
scp root@remote:/tmp/h2.jar /tmp/h2.jar
|
||||
# Test: java -cp /tmp/h2.jar org.h2.tools.Shell --help
|
||||
```
|
||||
|
||||
On the remote host (the H2 jar must exist):
|
||||
```bash
|
||||
# If inside a Docker container:
|
||||
docker cp container_name:/opt/app/lib/h2-*.jar /tmp/h2.jar
|
||||
```
|
||||
|
||||
## Implementation Template
|
||||
|
||||
```python
|
||||
import subprocess, tempfile, os, time
|
||||
|
||||
HOST = "152.53.xxx.xxx"
|
||||
DB_PATH = "/root/docker/app/data/database"
|
||||
CACHE_TTL = 30
|
||||
|
||||
_cache = {"data": None, "ts": 0}
|
||||
|
||||
def query_devices():
|
||||
if _cache["data"] and (time.time() - _cache["ts"]) < CACHE_TTL:
|
||||
return _cache["data"]
|
||||
|
||||
tmp_db = tempfile.mktemp(suffix=".mv.db")
|
||||
# 1. Copy DB snapshot from remote
|
||||
subprocess.run(
|
||||
["scp", "-i", "/root/.ssh/itpp-infra", "-o", "StrictHostKeyChecking=no",
|
||||
f"root@{HOST}:{DB_PATH}.mv.db", tmp_db],
|
||||
capture_output=True, timeout=15, check=True
|
||||
)
|
||||
|
||||
# 2. Query with Java H2 Shell
|
||||
base = tmp_db.replace(".mv.db", "")
|
||||
result = subprocess.run(
|
||||
["java", "-cp", "/tmp/h2.jar", "org.h2.tools.Shell",
|
||||
"-url", f"jdbc:h2:{base}", "-user", "sa", "-password", "",
|
||||
"-sql", "SELECT id, name, latitude, longitude FROM tc_devices;"],
|
||||
capture_output=True, text=True, timeout=20
|
||||
)
|
||||
os.unlink(tmp_db)
|
||||
|
||||
# 3. Parse pipe-delimited output (H2 Shell format)
|
||||
# Output: "ID | NAME | LATITUDE | LONGITUDE\n1 | foo | 10.4 | -75.5"
|
||||
rows = []
|
||||
for line in result.stdout.strip().split('\n'):
|
||||
parts = [p.strip() for p in line.split('|')]
|
||||
if len(parts) >= 4 and parts[0].strip().isdigit():
|
||||
rows.append({
|
||||
"id": int(parts[0]),
|
||||
"name": parts[1] if parts[1] != "null" else None,
|
||||
})
|
||||
|
||||
data = {"rows": rows, "count": len(rows)}
|
||||
_cache["data"] = data
|
||||
_cache["ts"] = time.time()
|
||||
return data
|
||||
```
|
||||
|
||||
## Traccar Schema Reference (GPS Tracking)
|
||||
|
||||
These are the live table schemas used by Traccar 6.x. All queries go through the H2 database at `/root/docker/traccar/traccar-data/database.mv.db` on app2 (152.53.39.202).
|
||||
|
||||
### tc_devices
|
||||
|
||||
| Column | Type | Notes |
|
||||
|---|---|---|
|
||||
| id | INT | Primary key |
|
||||
| name | VARCHAR | User-assigned device name |
|
||||
| uniqueid | VARCHAR | Device identifier (IMEI, serial) |
|
||||
| lastupdate | TIMESTAMP | Last position received |
|
||||
| status | VARCHAR | "online", "offline", or "unknown" |
|
||||
|
||||
### tc_positions
|
||||
|
||||
| Column | Type | Notes |
|
||||
|---|---|---|
|
||||
| id | BIGINT | Auto-increment |
|
||||
| deviceid | INT | FK → tc_devices.id |
|
||||
| latitude | DOUBLE | Decimal degrees |
|
||||
| longitude | DOUBLE | Decimal degrees |
|
||||
| speed | DOUBLE | Knots (× 1.15078 for mph) |
|
||||
| devicetime | TIMESTAMP | Device-reported time |
|
||||
|
||||
### Common Queries
|
||||
|
||||
```sql
|
||||
-- All devices with latest position (one query)
|
||||
SELECT d.id, d.name, d.uniqueid, d.lastupdate, d.status,
|
||||
p.latitude, p.longitude, p.speed, p.devicetime
|
||||
FROM tc_devices d
|
||||
LEFT JOIN tc_positions p ON p.id = (
|
||||
SELECT MAX(id) FROM tc_positions WHERE deviceid = d.id
|
||||
)
|
||||
ORDER BY d.lastupdate DESC;
|
||||
|
||||
-- Position history for one device
|
||||
SELECT latitude, longitude, speed, devicetime
|
||||
FROM tc_positions WHERE deviceid = 1
|
||||
ORDER BY devicetime ASC;
|
||||
|
||||
-- Stats: distance (haversine), max speed, active time
|
||||
SELECT COUNT(*) as positions,
|
||||
MAX(speed) * 1.15078 as max_speed_mph
|
||||
FROM tc_positions
|
||||
WHERE deviceid = 1
|
||||
AND devicetime >= NOW() - INTERVAL '24' HOUR;
|
||||
```
|
||||
|
||||
## FastAPI Proxy Integration
|
||||
|
||||
When the MCP server is slow (3-5s per SCP+Java query), serve data through a FastAPI proxy with 30-second caching. The ops portal at `/opt/ops-portal/server.py` uses this pattern:
|
||||
|
||||
```python
|
||||
# /api/ft360/status endpoint proxies Traccar data
|
||||
@app.get("/api/ft360/status")
|
||||
async def traccar_status():
|
||||
return _get_traccar_data() # cached 30s, SCP+Java underneath
|
||||
```
|
||||
|
||||
The JavaScript dashboard fetches from `/api/ft360/status` every 30 seconds — same TTL as the cache. No authentication needed; the proxy handles the SSH and Java complexity.
|
||||
|
||||
- **Java must be on PATH.** Install `default-jre-headless` before using. The `java` binary in Docker containers (e.g., `/opt/traccar/jre/bin/java`) isn't on PATH by default.
|
||||
- **H2 jar must be on the same host and compatible.** Copy it from the container or remote host — don't download a different version from Maven Central.
|
||||
- **H2 Shell output format.** The first line is column headers (`ID | NAME | ...`), subsequent lines are data. Filter with `.isdigit()` on the first column.
|
||||
- **SCP key must work.** The SSH key path must be absolute, and `StrictHostKeyChecking=no` avoids prompt-freezing on first connection.
|
||||
- **Temp file cleanup.** Always `os.unlink(tmp_db)` in a `finally` block or immediately after the query. A 176KB file won't cause issues if left, but it's sloppy.
|
||||
- **Cache invalidation.** If you stop+start the remote container (restarting the DB), the cache may return stale data until the TTL expires. For write operations, skip the cache.
|
||||
@@ -0,0 +1,33 @@
|
||||
# Immich photo map MCP pattern
|
||||
|
||||
Use this when Germaine asks to extract photo metadata from Immich and show where photos were taken.
|
||||
|
||||
## Immich API endpoints
|
||||
|
||||
Relevant stable endpoints:
|
||||
|
||||
- `POST /search/metadata` with `withExif: true` to search assets and include EXIF/location data.
|
||||
- `GET /assets/{id}/metadata` to fetch full key-value metadata for a single asset.
|
||||
|
||||
`/search/metadata` can filter by album IDs, taken date, city/state/country, camera make/model, favorite state, asset type, etc.
|
||||
|
||||
## Recommended MCP tools
|
||||
|
||||
- `immich_search_geo_assets(taken_after?, taken_before?, album_id?, limit=500)` — return assets with latitude/longitude, taken time, filename, city/state/country, camera model, thumbnail URL.
|
||||
- `immich_get_asset_metadata(asset_id)` — return full Immich metadata for one asset.
|
||||
- `immich_geojson(...)` — return a GeoJSON FeatureCollection suitable for Leaflet/MapLibre.
|
||||
- `immich_map_summary(...)` — summarize counts by city/country/date range and count missing-location assets.
|
||||
|
||||
## Map UI
|
||||
|
||||
The MCP should return data; a small UI can live in ops portal or a private subdomain. Use Leaflet + OpenStreetMap tiles + marker clustering. Marker click should show thumbnail, filename, taken date, and location labels.
|
||||
|
||||
## Security
|
||||
|
||||
GPS photo data is sensitive. Keep the MCP localhost-only, store `IMMICH_BASE_URL` and `IMMICH_API_KEY` in a chmod 600 `.env`, and put any map UI behind auth/Cloudflare Access. Prefer thumbnails, not full-resolution assets.
|
||||
|
||||
## Implementation notes
|
||||
|
||||
- Count missing GPS as `missing_location`, not an error.
|
||||
- Return GeoJSON coordinates as `[longitude, latitude]`.
|
||||
- Build read-only first. Metadata update/fix workflows are a later explicit feature.
|
||||
@@ -0,0 +1,108 @@
|
||||
# Open WebUI Configuration via SQLite
|
||||
|
||||
Open WebUI v0.10+ stores persistent config as JSON key/value pairs in its SQLite database at `/app/backend/data/webui.db` (Docker volume: `/var/lib/docker/volumes/openwebui_data/_data/webui.db`).
|
||||
|
||||
After the first launch, Docker `ConfigVar` environment variables are **ignored** — the DB is the single source of truth. All configuration changes after initial setup must be made directly in SQLite.
|
||||
|
||||
## Schema
|
||||
|
||||
```sql
|
||||
CREATE TABLE config (
|
||||
"key" TEXT NOT NULL,
|
||||
value JSON NOT NULL,
|
||||
updated_at BIGINT,
|
||||
PRIMARY KEY ("key")
|
||||
);
|
||||
```
|
||||
|
||||
## Essential config keys
|
||||
|
||||
### Model capabilities (vision, image generation)
|
||||
|
||||
```sql
|
||||
-- Enable vision for all models (required for image upload in chat)
|
||||
UPDATE config SET value = '{"capabilities": {"vision": true}}', updated_at = <unix_ts>
|
||||
WHERE key = 'models.default_metadata';
|
||||
```
|
||||
|
||||
Without this, image upload is disabled in Open WebUI regardless of which vision-capable models are connected.
|
||||
|
||||
### Image generation
|
||||
|
||||
```sql
|
||||
-- Enable image generation feature
|
||||
UPDATE config SET value = 'true', updated_at = <unix_ts> WHERE key = 'image_generation.enable';
|
||||
-- Engine/model is stored in separate keys
|
||||
-- Engine: openai, automatic1111, comfyui, gemini
|
||||
UPDATE config SET value = '"openai"', updated_at = <unix_ts> WHERE key = 'image_generation.engine';
|
||||
UPDATE config SET value = '"gemini/imagen-4.0-generate-001"', updated_at = <unix_ts> WHERE key = 'image_generation.model';
|
||||
```
|
||||
|
||||
### Web search
|
||||
|
||||
```sql
|
||||
-- Enable web search
|
||||
UPDATE config SET value = 'true', updated_at = <unix_ts> WHERE key = 'web.search.enable';
|
||||
-- Set engine: searxng, google_pse, brave, serpapi, etc.
|
||||
UPDATE config SET value = '"google_pse"', updated_at = <unix_ts> WHERE key = 'web.search.engine';
|
||||
```
|
||||
|
||||
Public SearXNG instances (searx.be, search.sapti.me, etc.) are heavily rate-limited (403/429). Google PSE is the reliable fallback but uses API quota. Self-hosted SearXNG is ideal.
|
||||
|
||||
### MCP tool server connections
|
||||
|
||||
```python
|
||||
import sqlite3, json, time
|
||||
db = sqlite3.connect('/var/lib/docker/volumes/openwebui_data/_data/webui.db')
|
||||
|
||||
row = db.execute("SELECT value FROM config WHERE key='tool_server.connections'").fetchone()
|
||||
current = json.loads(row[0]) if row else []
|
||||
|
||||
# Remove duplicates by port
|
||||
current = [c for c in current if '<port>' not in c.get('url', '')]
|
||||
|
||||
current.append({
|
||||
'url': 'http://host.docker.internal:<PORT>/mcp',
|
||||
'type': 'mcp', # CRITICAL: not 'openapi'
|
||||
'auth_type': 'none',
|
||||
'key': '',
|
||||
'headers': None,
|
||||
'config': {
|
||||
'enable': True,
|
||||
'function_name_filter_list': '',
|
||||
'access_grants': [{
|
||||
'principal_type': 'group',
|
||||
'principal_id': '<group-id>', # from DB: SELECT value FROM config WHERE key='ui.default_group_id'
|
||||
'permission': 'read'
|
||||
}]
|
||||
},
|
||||
'info': {'id': '', 'name': 'Server Name', 'description': 'What it does'}
|
||||
})
|
||||
|
||||
db.execute("UPDATE config SET value = ?, updated_at = ? WHERE key = ?",
|
||||
(json.dumps(current), int(time.time()), 'tool_server.connections'))
|
||||
db.commit()
|
||||
db.close()
|
||||
# Then: docker restart openwebui
|
||||
```
|
||||
|
||||
### User features/permissions
|
||||
|
||||
```sql
|
||||
-- Check image_generation and other feature flags
|
||||
SELECT value FROM config WHERE key = 'user.permissions';
|
||||
-- Look for JSON path: $.features.image_generation
|
||||
```
|
||||
|
||||
## Change workflow
|
||||
|
||||
1. Update the DB: `python3 -c "..."` (sqlite3 write)
|
||||
2. Verify: `SELECT value FROM config WHERE key = '...'`
|
||||
3. Restart: `docker restart openwebui`
|
||||
4. Wait 10-15 seconds for health check
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **ConfigVar env vars are dead on restart** — After first launch, `DEFAULT_MODEL_METADATA` and other ConfigVar env vars in docker-compose are silently ignored. The DB wins.
|
||||
- **Quoting in shell** — When running Python SQLite updates through SSH, use single-quoted Python strings and double-quoted SQL identifiers to avoid escaping hell. Better: write a .py script, scp it, run it.
|
||||
- **Group ID discovery** — The principal_id for access grants is found at `SELECT value FROM config WHERE key='ui.default_group_id'`. This changes per installation.
|
||||
@@ -0,0 +1,69 @@
|
||||
# OSINT Person Intelligence MCP Pattern
|
||||
|
||||
Build a dedicated entity-resolution layer above generic search MCPs when the task is person/skip-trace intelligence rather than raw search.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Hermes
|
||||
↓
|
||||
osint-person-mcp # entity resolution, scoring, dossier logic
|
||||
↓
|
||||
super-search MCP # raw search + extraction providers
|
||||
↓
|
||||
SearXNG / Exa / OpenCorporates / CourtListener / Firecrawl
|
||||
```
|
||||
|
||||
Keep `super-search` generic. Put identity-specific logic in a separate MCP so the generic search engine does not become a grab bag of scoring and compliance behavior.
|
||||
|
||||
## Tool set
|
||||
|
||||
Recommended tools:
|
||||
|
||||
| Tool | Purpose |
|
||||
|---|---|
|
||||
| `person_search` | Query generation, search, source scoring, entity clustering, compliance notes |
|
||||
| `phone_search_variants` | Normalize phone and emit common variants |
|
||||
| `phone_search` | Search all phone variants and cluster matches |
|
||||
| `email_search` | Search email address and cluster matches |
|
||||
| `business_affiliation_search` | Search person + organization/business/location anchors |
|
||||
| `court_business_search` | Search court/business/public-record query variants |
|
||||
| `score_source` | Score URL/title/description reliability |
|
||||
| `dossier_report` | Convert structured JSON into concise markdown report |
|
||||
|
||||
## Source scoring
|
||||
|
||||
Use durable source tiers:
|
||||
|
||||
| Tier | Examples | Treatment |
|
||||
|---|---|---|
|
||||
| High | official school/government/court/publication sources | Strong evidence only when identity anchors also match |
|
||||
| Medium-high | LinkedIn, RocketReach, TheOrg, Tradeloop, ZoomInfo | Useful but needs corroboration |
|
||||
| Medium-low | Whitepages, 411, BeenVerified, NPD, FastBackgroundCheck | Lead only; mark unverified |
|
||||
| Low | obituaries/name-only pages | Usually weak/name-collision |
|
||||
| Reject | random blob/CDN/search-spam pages | Exclude from useful clusters |
|
||||
|
||||
Do not let organization-only pages rank as likely person matches. An official SCAD page with no subject name is context, not identity evidence.
|
||||
|
||||
## Matching pitfalls
|
||||
|
||||
- Ignore one-letter middle initials for alias matching. A lone `m` from `Germaine M. Brown` matches almost every page.
|
||||
- Normalize phone numbers into all common forms: digits, dashed, dotted, `(xxx) xxx-xxxx`, `+1 xxx xxx xxxx`, `+1-xxx-xxx-xxxx`.
|
||||
- Separate likely target, possible target, weak/different, and rejected results.
|
||||
- Preserve source URL, query, provider, matched fields, score, and timestamp for every lead.
|
||||
- Treat Atlanta/Savannah/location-only matches as insufficient without identity anchors.
|
||||
|
||||
## Compliance guardrails
|
||||
|
||||
- OSINT results are leads, not verified facts.
|
||||
- Do not auto-contact from scraped or data-broker leads without human review.
|
||||
- Preserve source URL and timestamp for every claim.
|
||||
- Mark data-broker information as unverified until corroborated.
|
||||
|
||||
## DRE integration path
|
||||
|
||||
1. Add `Run OSINT` button on claim/debtor page.
|
||||
2. Call `person_search` with debtor anchors.
|
||||
3. Save JSON packet + markdown dossier to debtor record.
|
||||
4. Require human review before any collection action uses a lead.
|
||||
5. Later add paid providers only where MVP gaps are proven (People Data Labs, Trestle/Ekata-style phone intelligence, Pipl).
|
||||
@@ -0,0 +1,84 @@
|
||||
# Super Search — MCP Server Build Reference
|
||||
|
||||
**Built:** July 11, 2026 · **Updated:** July 15, 2026 (v2.0.0)
|
||||
**Server:** Core (152.53.192.33)
|
||||
**Service:** `super-search.service` (systemd, auto-restart)
|
||||
**Endpoint:** http://127.0.0.1:8899/mcp
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Super Search (FastMCP v2.0.0) — 10 tools, 7 providers
|
||||
├── web_search(query, limit, category, time_range, site_filter)
|
||||
│ └── SearXNG → Exa → OpenCorporates → CourtListener → DuckDuckGo → Firecrawl
|
||||
├── web_extract(url, char_limit)
|
||||
│ └── Trafilatura → Firecrawl
|
||||
├── web_search_premium(query, limit, include_domains, exclude_domains, date_start, date_end)
|
||||
│ └── Exa direct (DRE-grade)
|
||||
├── web_search_news(query, limit, days) ★ NEW
|
||||
│ └── SearXNG (news) → Exa
|
||||
├── web_search_academic(query, limit) ★ NEW
|
||||
│ └── SearXNG (science) → Exa
|
||||
├── web_extract_batch(urls, char_limit) ★ NEW
|
||||
│ └── Parallel Trafilatura → Firecrawl (max 10 URLs)
|
||||
├── web_suggest_queries(query, limit) ★ NEW
|
||||
│ └── SearXNG suggestions
|
||||
├── health_check() ★ NEW
|
||||
│ └── Provider statuses + cache stats + rate limit buckets
|
||||
├── web_lookup(query, limit) ★ NEW
|
||||
│ └── DuckDuckGo Instant Answers + Wikipedia
|
||||
└── web_search_images(query, limit) ★ NEW
|
||||
└── SearXNG (images) → general fallback
|
||||
```
|
||||
|
||||
## File Layout
|
||||
|
||||
```
|
||||
/root/docker/super-search/
|
||||
├── server.py ← FastMCP server (192 lines)
|
||||
├── super-search.service ← systemd unit file (copied to /etc/systemd/system/)
|
||||
├── venv/ ← Python virtualenv
|
||||
├── exa.env ← Exa API key (chmod 600)
|
||||
├── check_api.py ← test artifacts (cleanup)
|
||||
└── check_api2.py ← test artifacts (cleanup)
|
||||
```
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
1. **Async I/O everywhere** — httpx.AsyncClient for all HTTP calls. The searxng, exa, firecrawl, duckduckgo, and wikipedia search functions are all async, allowing the server to handle concurrent requests without blocking.
|
||||
|
||||
2. **Seven-provider fallback chain** — SearXNG (free, privacy-respecting) → Exa (premium neural) → OpenCorporates (company records) → CourtListener (case law) → DuckDuckGo (instant answers) → Wikipedia (encyclopedia) → Firecrawl (last resort). Each tier fails independently with full telemetry reporting.
|
||||
|
||||
3. **Extraction fallback** — Trafilatura (local, fast, free) → Firecrawl scrape (remote, paid). Trafilatura handles most HTML pages; Firecrawl handles JavaScript-heavy and paywalled sites.
|
||||
|
||||
4. **Batch extraction** — `web_extract_batch` extracts up to 10 URLs in parallel using `asyncio.gather`, dramatically faster than sequential extraction.
|
||||
|
||||
5. **Domain-specific tools** — News, academic, image search, and factual lookups each get dedicated tools with appropriate fallback chains and defaults (date ranges for news, science category for academic).
|
||||
|
||||
6. **Credentials from .env** — FIRECRAWL_API_KEY and EXA_API_KEY loaded from ~/.hermes/.env via python-dotenv. The systemd service uses EnvironmentFile to pass them.
|
||||
|
||||
7. **Rate limiting** — Token bucket per provider (7 providers), TTL caching per provider, retry with exponential backoff on 429/5xx.
|
||||
|
||||
8. **Provider health telemetry** — Every search tool returns which providers were tried, their status, result counts, and upstream engine health (SearXNG unresponsive engines with reasons).
|
||||
|
||||
## Hermes Integration
|
||||
|
||||
```yaml
|
||||
mcp_servers:
|
||||
super-search:
|
||||
url: http://127.0.0.1:8899/mcp
|
||||
enabled: true
|
||||
```
|
||||
|
||||
After deploying a new version, the Hermes gateway must be restarted to pick up new tools:
|
||||
```bash
|
||||
systemctl restart hermes.service
|
||||
# or from outside the gateway: hermes gateway restart
|
||||
```
|
||||
|
||||
## Version History
|
||||
|
||||
- v1.0 — Initial build (SearXNG → Firecrawl, Trafilatura → Firecrawl) — 3 tools
|
||||
- v1.1 — Added Exa as premium tier (SearXNG → Exa → Firecrawl) — 3 tools
|
||||
- v1.2 — Added OpenCorporates + CourtListener to fallback chain + rate limiting — 3 tools
|
||||
- v2.0.0 — Major expansion: 7 new tools (10 total), 2 new providers (DuckDuckGo, Wikipedia), enhanced params on all existing tools — July 15, 2026
|
||||
Reference in New Issue
Block a user