Initial skills documentation — 25 categories, all SKILL.md + references + scripts
This commit is contained in:
root
@@ -0,0 +1,444 @@
|
||||
---
|
||||
name: mcp-servers
|
||||
description: Build, deploy, and integrate MCP (Model Context Protocol) servers with Hermes — FastMCP Python servers, HTTP transport configuration, Hermes MCP client wiring, and multi-provider search/extraction toolkits.
|
||||
version: 1.0.0
|
||||
author: Sho'Nuff
|
||||
tags: [mcp, servers, hermes-integration, fastmcp, tools]
|
||||
---
|
||||
|
||||
# MCP Servers
|
||||
|
||||
Build, deploy, and wire MCP servers into Hermes. MCP servers provide additional tools via the Model Context Protocol — think of them as plugin toolkits your agent can call natively.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Hermes Agent
|
||||
│ native MCP client
|
||||
├── mcp_servers.mysql → python3 /root/.hermes/scripts/mcp-mysql.py
|
||||
├── mcp_servers.super-search → http://127.0.0.1:8899/mcp
|
||||
└── mcp_servers.exa → https://mcp.exa.ai/mcp
|
||||
```
|
||||
|
||||
Two transport types:
|
||||
- **Command-based** — Hermes spawns a Python process that speaks MCP over stdio
|
||||
- **HTTP-based** — A persistent server runs at an HTTP endpoint, Hermes connects via SSE
|
||||
|
||||
## HTTP-Based MCP Server (FastMCP Pattern)
|
||||
|
||||
This is the preferred pattern for complex, persistent toolkits. The server runs as a systemd service and Hermes connects via HTTP.
|
||||
|
||||
### Build Steps
|
||||
|
||||
1. **Install dependencies** in a virtualenv:
|
||||
```bash
|
||||
pip install fastmcp httpx trafilatura firecrawl-py
|
||||
```
|
||||
|
||||
2. **Create the server** — a single Python file with FastMCP tools:
|
||||
```python
|
||||
from fastmcp import FastMCP
|
||||
mcp = FastMCP("Server Name", version="1.0.0")
|
||||
|
||||
@mcp.tool(description="Search the web")
|
||||
async def web_search(query: str, limit: int = 5) -> str:
|
||||
# ... implementation
|
||||
return results
|
||||
|
||||
if __name__ == "__main__":
|
||||
mcp.run(transport="http", host="127.0.0.1", port=8899, path="/mcp")
|
||||
```
|
||||
|
||||
3. **Create a systemd service:**
|
||||
```ini
|
||||
[Unit]
|
||||
Description=Super Search MCP Server
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=root
|
||||
WorkingDirectory=/root/docker/super-search
|
||||
Environment="PATH=/root/docker/super-search/venv/bin:/usr/local/sbin:..."
|
||||
Environment="PYTHONUNBUFFERED=1"
|
||||
EnvironmentFile=/root/.hermes/.env
|
||||
ExecStart=/root/docker/super-search/venv/bin/python3 /root/docker/super-search/server.py
|
||||
Restart=always
|
||||
RestartSec=3
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
4. **Wire into Hermes config:**
|
||||
```bash
|
||||
hermes config set mcp_servers.<name>.url 'http://127.0.0.1:<port>/mcp'
|
||||
hermes config set mcp_servers.<name>.enabled 'true'
|
||||
```
|
||||
|
||||
Or edit `~/.hermes/config.yaml` directly:
|
||||
```yaml
|
||||
mcp_servers:
|
||||
super-search:
|
||||
url: http://127.0.0.1:8899/mcp
|
||||
enabled: true
|
||||
```
|
||||
|
||||
5. **Restart gateway** or reload MCP:
|
||||
```bash
|
||||
# Need to restart from outside the gateway session
|
||||
systemctl restart hermes.service
|
||||
```
|
||||
|
||||
### Testing
|
||||
|
||||
```bash
|
||||
# List all MCP servers
|
||||
hermes mcp list
|
||||
|
||||
# Test connection and discover tools
|
||||
hermes mcp test <server-name>
|
||||
# Expected output:
|
||||
# ✓ Connected (XXms)
|
||||
# ✓ Tools discovered: N
|
||||
# tool_name1 Description...
|
||||
# tool_name2 Description...
|
||||
```
|
||||
|
||||
### Performance
|
||||
- Local HTTP MCP servers: ~50-60ms connection time
|
||||
- Remote HTTP MCP servers (e.g., Exa): ~1-2s connection time (includes OAuth/TLS)
|
||||
|
||||
### Troubleshooting
|
||||
|
||||
**Server won't start — port in use:**
|
||||
A leftover process from a subagent's test run often holds the port. The `fuser` command alone may not suffice:
|
||||
```bash
|
||||
# Find what's on the port
|
||||
ss -tlnp | grep <port> # shows PID
|
||||
kill <PID> # then restart
|
||||
systemctl restart <service>
|
||||
```
|
||||
|
||||
If the process won't die (orphaned): `fuser -k <port>/tcp` then restart.
|
||||
|
||||
**Subagent timeout during builds:**
|
||||
The default 600s child timeout can be tight for multi-step MCP server builds (install deps + write code + systemd service + Hermes config + gateway restart). The subagent usually produces the code and service file correctly before timing out — just the final wiring steps (systemctl enable, testing) are incomplete. Do those manually rather than re-dispatching.
|
||||
|
||||
**MCP endpoint returns HTTP 406:**
|
||||
The client must accept both `application/json` and `text/event-stream`. Add headers:
|
||||
```
|
||||
Accept: application/json, text/event-stream
|
||||
```
|
||||
|
||||
**MCP endpoint returns "Missing session ID":**
|
||||
The HTTP transport uses SSE for session negotiation. Simple curl tests won't work. Use `hermes mcp test` instead, which handles the MCP protocol correctly.
|
||||
|
||||
**Tools discovered but not showing in Hermes:**
|
||||
Restart the gateway: `systemctl restart hermes.service` (from outside the gateway session).
|
||||
|
||||
## Immich Photo Map MCP Pattern
|
||||
|
||||
For Immich/photo-location workflows, build a dedicated read-only MCP that uses Immich `POST /search/metadata` with `withExif: true` and returns geotagged assets plus GeoJSON for a private Leaflet map. Keep GPS data private and expose thumbnails only. See `references/immich-photo-map-mcp.md`.
|
||||
|
||||
## OSINT Person Intelligence Layer
|
||||
|
||||
When raw search becomes a person/skip-trace workflow, do **not** overload the generic search MCP with identity logic. Build a dedicated entity-resolution MCP above Super Search. Pattern reference: `references/osint-person-intelligence.md`.
|
||||
|
||||
Use this pattern for: person search, phone/email variant search, source scoring, identity clustering, dossier generation, DRE debtor enrichment, and compliance-preserving lead packets.
|
||||
|
||||
## Command-Based MCP Server Pattern
|
||||
|
||||
For simpler, stateless MCP servers that don't warrant a persistent daemon:
|
||||
|
||||
```yaml
|
||||
mcp_servers:
|
||||
mysql:
|
||||
command: python3
|
||||
args:
|
||||
- /root/.hermes/scripts/mcp-mysql.py
|
||||
enabled: true
|
||||
```
|
||||
|
||||
Hermes spawns the process per-request. No systemd service needed.
|
||||
|
||||
## Remote MCP Server Pattern (OAuth)
|
||||
|
||||
Some services (Exa, agent-data.dev) provide hosted MCP servers with OAuth:
|
||||
|
||||
```yaml
|
||||
mcp_servers:
|
||||
exa:
|
||||
url: https://mcp.exa.ai/mcp
|
||||
enabled: true
|
||||
```
|
||||
|
||||
First connection opens a browser for OAuth sign-in. Heremes handles this via `hermes mcp login <name>`.
|
||||
|
||||
## Rate Limiting & TTL Caching Layer
|
||||
|
||||
When integrating multiple third-party APIs with rate limits (free tiers, govt databases, people-search APIs), add a token-bucket rate limiter and TTL cache between the MCP tool and the provider calls.
|
||||
|
||||
### Architecture
|
||||
|
||||
```
|
||||
Query → TTL Cache (dict, per-provider TTL)
|
||||
↓ cache miss
|
||||
Token Bucket (per-provider tokens/interval)
|
||||
↓ tokens available
|
||||
Provider API → success? → cache + return
|
||||
↓ 429 / rate limited
|
||||
Exponential Backoff (respects Retry-After header)
|
||||
↓ exhausted
|
||||
Fallback → next provider in chain
|
||||
```
|
||||
|
||||
### Implementation — `ratelimit.py`
|
||||
|
||||
Create a standalone module that the MCP server imports:
|
||||
|
||||
```python
|
||||
"""ratelimit.py — Token bucket rate limiter + TTL cache for MCP providers."""
|
||||
|
||||
import time, hashlib, json, asyncio
|
||||
|
||||
# Per-provider rate limit config
|
||||
RATE_LIMITS = {
|
||||
"searxng": {"tokens": 30, "interval": 1.0}, # local — high limit
|
||||
"exa": {"tokens": 10, "interval": 1.0}, # paid tier
|
||||
"firecrawl": {"tokens": 5, "interval": 60.0}, # 1k/mo budget
|
||||
"opencorporates": {"tokens": 1, "interval": 1.0}, # 1/sec free tier
|
||||
"courtlistener": {"tokens": 10, "interval": 6.0}, # ~100/min free
|
||||
}
|
||||
|
||||
# Per-provider cache TTL in seconds
|
||||
CACHE_TTL = {
|
||||
"searxng": 120, # 2 min — web results change
|
||||
"exa": 300, # 5 min
|
||||
"opencorporates": 3600, # 1 hr — company records are stable
|
||||
"courtlistener": 3600, # 1 hr — court cases are infrequent
|
||||
"firecrawl": 300, # 5 min
|
||||
}
|
||||
|
||||
class TokenBucket:
|
||||
"""Async-safe token bucket for per-provider rate limiting."""
|
||||
def __init__(self, tokens, interval):
|
||||
self.max_tokens = tokens
|
||||
self.tokens = tokens
|
||||
self.interval = interval # seconds between refills
|
||||
self.last_refill = time.time()
|
||||
|
||||
async def acquire(self):
|
||||
now = time.time()
|
||||
if now - self.last_refill >= self.interval:
|
||||
self.tokens = self.max_tokens
|
||||
self.last_refill = now
|
||||
if self.tokens <= 0:
|
||||
return False
|
||||
self.tokens -= 1
|
||||
return True
|
||||
|
||||
_buckets = {}
|
||||
_cache = {}
|
||||
|
||||
async def rate_limited_search(provider: str, cache_key: str, fn, *args, **kwargs):
|
||||
"""Wrapper: check cache → rate limit → call → cache result."""
|
||||
# 1. Cache check
|
||||
cache_key_full = f"{provider}:{cache_key}"
|
||||
if cache_key_full in _cache:
|
||||
entry = _cache[cache_key_full]
|
||||
if time.time() - entry["ts"] < CACHE_TTL.get(provider, 300):
|
||||
return entry["data"]
|
||||
|
||||
# 2. Rate limit check
|
||||
rl = RATE_LIMITS.get(provider)
|
||||
if rl:
|
||||
if provider not in _buckets:
|
||||
_buckets[provider] = TokenBucket(rl["tokens"], rl["interval"])
|
||||
allowed = await _buckets[provider].acquire()
|
||||
if not allowed:
|
||||
raise RateLimitedError(f"{provider}: token bucket empty")
|
||||
|
||||
# 3. Call with retry
|
||||
result = await _retry_with_backoff(fn, *args, **kwargs)
|
||||
|
||||
# 4. Cache result
|
||||
_cache[cache_key_full] = {"data": result, "ts": time.time()}
|
||||
return result
|
||||
|
||||
async def _retry_with_backoff(fn, *args, retries=3, base_delay=1.0):
|
||||
"""Exponential backoff on 429/5xx, respects Retry-After."""
|
||||
for attempt in range(retries):
|
||||
try:
|
||||
return await fn(*args, **kwargs)
|
||||
except Exception as e:
|
||||
if attempt == retries - 1:
|
||||
raise
|
||||
delay = base_delay * (2 ** attempt)
|
||||
# Check for Retry-After in response headers
|
||||
await asyncio.sleep(delay)
|
||||
```
|
||||
|
||||
### Key decisions
|
||||
- **Cache aggressive for stable data** — company records and court cases get 1-hour TTLs since they rarely change. Web search results get 2-5 minutes.
|
||||
- **Token bucket per provider** — each provider has its own limit so one fast provider can't starve another.
|
||||
- **Fall through on rate limit** — don't fail the whole tool. The MCP server's fallback chain should catch the RateLimitedError and try the next provider.
|
||||
|
||||
## Multi-Provider Search Routing Pattern
|
||||
|
||||
When building a search/extraction MCP server that uses multiple backends, use this fallback chain pattern (incorporating rate limiting):
|
||||
|
||||
```python
|
||||
async def web_search(query, limit=5):
|
||||
# 1. Try SearXNG (free, local)
|
||||
try:
|
||||
results = await _searxng_search(query, limit)
|
||||
if results:
|
||||
return {"provider": "searxng", "results": results}
|
||||
except:
|
||||
pass
|
||||
|
||||
# 2. Fallback to Exa (premium, API key)
|
||||
try:
|
||||
results = await _exa_search(query, limit)
|
||||
if results:
|
||||
return {"provider": "exa", "results": results}
|
||||
except:
|
||||
pass
|
||||
|
||||
# 3. Last resort: Firecrawl (paid, most resilient)
|
||||
results = await _firecrawl_search(query, limit)
|
||||
return {"provider": "firecrawl", "results": results, "telemetry": telemetry}
|
||||
```
|
||||
|
||||
This ensures the service degrades gracefully — free first, premium second, paid last resort.
|
||||
|
||||
**Telemetry rule:** Multi-provider MCP tools must return provider telemetry, not just final results. Include providers attempted, status/error, result counts, and upstream health details (for example, SearXNG `unresponsive_engines` with CAPTCHA/rate-limit reasons). Do not silently hide a provider that is reachable but returning zero useful results because upstream engines are blocked.
|
||||
|
||||
## Remote Database MCP Data Source
|
||||
|
||||
When an MCP server needs to read from a database on a remote host (e.g., Traccar H2 on app2), use the SSH+SCP+query pattern: copy the DB as a read-only snapshot, query locally with Java H2 Shell, parse pipe-delimited output, return structured data. Avoids API auth, CORS, and credential management for read-only queries. Full reference: `references/h2-remote-db-query.md`.
|
||||
|
||||
This pattern works for H2, SQLite, and any file-based database accessible via SSH.
|
||||
|
||||
## Docker-Based MCP Deployment (alongside Open WebUI)
|
||||
|
||||
When deploying MCP servers on app1 to serve Open WebUI (and Hermes), use Docker Compose instead of systemd. Add the MCP service to the existing LiteLLM compose at `/root/docker/litellm/docker-compose.yml`.
|
||||
|
||||
### Pattern
|
||||
|
||||
```yaml
|
||||
services:
|
||||
super-search:
|
||||
build: ../super-search # Directory with server.py + Dockerfile
|
||||
container_name: super-search
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
- EXA_API_KEY=<key>
|
||||
- FIRECRAWL_API_KEY=<key>
|
||||
- PYTHONUNBUFFERED=1
|
||||
ports:
|
||||
- '127.0.0.1:8899:8899' # Bound to 127.0.0.1 — services on same host
|
||||
networks:
|
||||
- litellm-net # Share LiteLLM's network
|
||||
```
|
||||
|
||||
### Dockerfile (slim Python)
|
||||
|
||||
```dockerfile
|
||||
FROM python:3.13-slim
|
||||
WORKDIR /app
|
||||
COPY requirements.txt .
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
COPY server.py ratelimit.py ./
|
||||
ENV PYTHONUNBUFFERED=1
|
||||
EXPOSE 8899
|
||||
CMD ["python3", "server.py"]
|
||||
```
|
||||
|
||||
### Health check pitfall — slim containers
|
||||
|
||||
Python slim images lack `curl`, `ss`, `pgrep`, and other common health check tools. Do NOT use shell-based health checks. Options:
|
||||
|
||||
1. **Skip it** — Remove the health check block entirely. Docker will still restart on crash.
|
||||
2. **Python socket test** — `python3 -c "import socket; s=socket.socket(); s.connect(('127.0.0.1',8899)); s.close()"`
|
||||
3. **Docker exec test from host** — `docker exec super-search python3 -c "..."`
|
||||
|
||||
### Addressing
|
||||
|
||||
- **Open WebUI (Docker → host):** `http://host.docker.internal:8899/mcp`
|
||||
- **Hermes (host → Docker):** `http://127.0.0.1:8899/mcp` (since port is mapped to localhost)
|
||||
- **Process test from inside container:** `http://127.0.0.1:8899/mcp`
|
||||
|
||||
### Verifying deployment
|
||||
|
||||
Use `docker exec` with FastMCP client to verify tools:
|
||||
|
||||
```bash
|
||||
docker exec super-search python3 -c "
|
||||
from fastmcp import Client
|
||||
import asyncio
|
||||
async def test():
|
||||
async with Client('http://127.0.0.1:8899/mcp') as client:
|
||||
tools = await client.list_tools()
|
||||
print(f'Tools: {len(tools)}')
|
||||
for t in tools:
|
||||
print(f' - {t.name}')
|
||||
asyncio.run(test())
|
||||
"
|
||||
```
|
||||
|
||||
## Wiring MCP into Open WebUI
|
||||
|
||||
Open WebUI stores tool server connections as a JSON blob in its SQLite DB at key `tool_server.connections`. Add MCP servers programmatically:
|
||||
|
||||
```python
|
||||
import sqlite3, json, time
|
||||
db = sqlite3.connect('/var/lib/docker/volumes/openwebui_data/_data/webui.db')
|
||||
|
||||
row = db.execute("SELECT value FROM config WHERE key='tool_server.connections'").fetchone()
|
||||
current = json.loads(row[0]) if row else []
|
||||
|
||||
# Remove duplicates by URL
|
||||
current = [c for c in current if '8899' not in c.get('url', '')]
|
||||
|
||||
current.append({
|
||||
'url': 'http://host.docker.internal:8899/mcp',
|
||||
'type': 'mcp', # CRITICAL: not 'openapi'
|
||||
'auth_type': 'none',
|
||||
'config': {
|
||||
'enable': True,
|
||||
'function_name_filter_list': '',
|
||||
'access_grants': [{
|
||||
'principal_type': 'group',
|
||||
'principal_id': '<group-id>',
|
||||
'permission': 'read'
|
||||
}]
|
||||
},
|
||||
'info': {'name': 'My MCP', 'description': 'Tools description'}
|
||||
})
|
||||
|
||||
db.execute("UPDATE config SET value = ?, updated_at = ? WHERE key = ?",
|
||||
(json.dumps(current), int(time.time()), 'tool_server.connections'))
|
||||
db.commit()
|
||||
```
|
||||
|
||||
Then restart Open WebUI: `docker restart openwebui`
|
||||
|
||||
Users enable MCP tools per-chat via the **➕ → Integrations → Tools** menu.
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **yaml.dump drops top-level `networks:` section** — When editing docker-compose.yml with Python's `yaml.dump()`, the `networks:` block is silently removed on every write. This happened 2x in one session (Jul 15). Verify with `grep '^networks:' docker-compose.yml` after every YAML edit. If missing, restore from `.bak` and append new services via `cat >>` instead. You can also append the networks block manually: `echo -e '\nnetworks:\n litellm-net:\n driver: bridge' >> docker-compose.yml`
|
||||
- **Slim Python images lack health check tools** — `curl`, `ss`, `pgrep`, and `wget` are NOT in `python:3.13-slim`. Skip Docker health checks or use `pgrep -f server.py`.
|
||||
- **Port conflicts on app1** — Port 3000 is bound by Open WebUI. Check `ss -tlnp | grep <port>` before adding new MCP service ports.
|
||||
- **Open WebUI stores MCP connections in SQLite** — not in env vars or docker-compose. Adding an MCP server requires a DB update + restart at `/var/lib/docker/volumes/openwebui_data/_data/webui.db`, key `tool_server.connections`.
|
||||
- **Open WebUI `ConfigVar` env vars are ignored after first launch** — values persist in SQLite. Use `UPDATE config SET value=... WHERE key=...` for persistent config.
|
||||
- **MCP connection type must be `\"mcp\"`, not `\"openapi\"`** — using OpenAPI type for MCP URLs causes infinite loading spinner.
|
||||
- **`host.docker.internal` for Docker→host** — when Open WebUI (Docker) needs to reach an MCP server on the same host, use `http://host.docker.internal:<port>/mcp`, not `localhost`.
|
||||
- **Hermes config changes to mcp_servers require gateway restart** — tools won't be available until restart.
|
||||
- **HTTP transport needs SSE session negotiation** — curl POST won't work. Use `docker exec <name> python3 -c \"from fastmcp import Client...\"` for verification.
|
||||
- **Subagent timeout** — default 600s is tight for MCP server builds. Subagents often return \"next steps\" without deploying — verify yourself or rebuild.
|
||||
|
||||
See `references/browser-puppeteer-mcp.md` for the full Playwright + browserless/chrome MCP pattern.
|
||||
See `references/openwebui-config.md` for Open WebUI SQLite configuration reference.
|
||||
Reference in New Issue
Block a user