hermes-skills/skills/devops/mcp-servers/SKILL.md

---
name: mcp-servers
description: Build, deploy, and integrate MCP (Model Context Protocol) servers with Hermes — FastMCP Python servers, HTTP transport configuration, Hermes MCP client wiring, and multi-provider search/extraction toolkits.
version: 1.0.0
author: Sho'Nuff
tags: [mcp, servers, hermes-integration, fastmcp, tools]
---

# MCP Servers

Build, deploy, and wire MCP servers into Hermes. MCP servers provide additional tools via the Model Context Protocol — think of them as plugin toolkits your agent can call natively.

## Architecture

```
Hermes Agent
  │ native MCP client
  ├── mcp_servers.mysql     → python3 /root/.hermes/scripts/mcp-mysql.py
  ├── mcp_servers.super-search → http://127.0.0.1:8899/mcp
  └── mcp_servers.exa       → https://mcp.exa.ai/mcp
```

Two transport types:
- **Command-based** — Hermes spawns a Python process that speaks MCP over stdio
- **HTTP-based** — A persistent server runs at an HTTP endpoint, Hermes connects via SSE

## HTTP-Based MCP Server (FastMCP Pattern)

This is the preferred pattern for complex, persistent toolkits. The server runs as a systemd service and Hermes connects via HTTP.

### Build Steps

1. **Install dependencies** in a virtualenv:
   ```bash
   pip install fastmcp httpx trafilatura firecrawl-py
   ```

2. **Create the server** — a single Python file with FastMCP tools:
   ```python
   from fastmcp import FastMCP
   mcp = FastMCP("Server Name", version="1.0.0")

   @mcp.tool(description="Search the web")
   async def web_search(query: str, limit: int = 5) -> str:
       # ... implementation
       return results

   if __name__ == "__main__":
       mcp.run(transport="http", host="127.0.0.1", port=8899, path="/mcp")
   ```

3. **Create a systemd service:**
   ```ini
   [Unit]
   Description=Super Search MCP Server
   After=network.target
   
   [Service]
   Type=simple
   User=root
   WorkingDirectory=/root/docker/super-search
   Environment="PATH=/root/docker/super-search/venv/bin:/usr/local/sbin:..."
   Environment="PYTHONUNBUFFERED=1"
   EnvironmentFile=/root/.hermes/.env
   ExecStart=/root/docker/super-search/venv/bin/python3 /root/docker/super-search/server.py
   Restart=always
   RestartSec=3
   
   [Install]
   WantedBy=multi-user.target
   ```

4. **Wire into Hermes config:**
   ```bash
   hermes config set mcp_servers.<name>.url 'http://127.0.0.1:<port>/mcp'
   hermes config set mcp_servers.<name>.enabled 'true'
   ```
   
   Or edit `~/.hermes/config.yaml` directly:
   ```yaml
   mcp_servers:
     super-search:
       url: http://127.0.0.1:8899/mcp
       enabled: true
   ```

5. **Restart gateway** or reload MCP:
   ```bash
   # Need to restart from outside the gateway session
   systemctl restart hermes.service
   ```

### Testing

```bash
# List all MCP servers
hermes mcp list

# Test connection and discover tools
hermes mcp test <server-name>
# Expected output:
#   ✓ Connected (XXms)
#   ✓ Tools discovered: N
#     tool_name1     Description...
#     tool_name2     Description...
```

### Performance
- Local HTTP MCP servers: ~50-60ms connection time
- Remote HTTP MCP servers (e.g., Exa): ~1-2s connection time (includes OAuth/TLS)

### Troubleshooting

**Server won't start — port in use:**
A leftover process from a subagent's test run often holds the port. The `fuser` command alone may not suffice:
```bash
# Find what's on the port
ss -tlnp | grep <port>   # shows PID
kill <PID>               # then restart
systemctl restart <service>
```

If the process won't die (orphaned): `fuser -k <port>/tcp` then restart.

**Subagent timeout during builds:**
The default 600s child timeout can be tight for multi-step MCP server builds (install deps + write code + systemd service + Hermes config + gateway restart). The subagent usually produces the code and service file correctly before timing out — just the final wiring steps (systemctl enable, testing) are incomplete. Do those manually rather than re-dispatching.

**MCP endpoint returns HTTP 406:**
The client must accept both `application/json` and `text/event-stream`. Add headers:
```
Accept: application/json, text/event-stream
```

**MCP endpoint returns "Missing session ID":**
The HTTP transport uses SSE for session negotiation. Simple curl tests won't work. Use `hermes mcp test` instead, which handles the MCP protocol correctly.

**Tools discovered but not showing in Hermes:**
Restart the gateway: `systemctl restart hermes.service` (from outside the gateway session).

## Immich Photo Map MCP Pattern

For Immich/photo-location workflows, build a dedicated read-only MCP that uses Immich `POST /search/metadata` with `withExif: true` and returns geotagged assets plus GeoJSON for a private Leaflet map. Keep GPS data private and expose thumbnails only. See `references/immich-photo-map-mcp.md`.

## OSINT Person Intelligence Layer

When raw search becomes a person/skip-trace workflow, do **not** overload the generic search MCP with identity logic. Build a dedicated entity-resolution MCP above Super Search. Pattern reference: `references/osint-person-intelligence.md`.

Use this pattern for: person search, phone/email variant search, source scoring, identity clustering, dossier generation, DRE debtor enrichment, and compliance-preserving lead packets.

## Command-Based MCP Server Pattern

For simpler, stateless MCP servers that don't warrant a persistent daemon:

```yaml
mcp_servers:
  mysql:
    command: python3
    args:
      - /root/.hermes/scripts/mcp-mysql.py
    enabled: true
```

Hermes spawns the process per-request. No systemd service needed.

## Remote MCP Server Pattern (OAuth)

Some services (Exa, agent-data.dev) provide hosted MCP servers with OAuth:

```yaml
mcp_servers:
  exa:
    url: https://mcp.exa.ai/mcp
    enabled: true
```

First connection opens a browser for OAuth sign-in. Heremes handles this via `hermes mcp login <name>`.

## Rate Limiting & TTL Caching Layer

When integrating multiple third-party APIs with rate limits (free tiers, govt databases, people-search APIs), add a token-bucket rate limiter and TTL cache between the MCP tool and the provider calls.

### Architecture

```
Query → TTL Cache (dict, per-provider TTL)
          ↓ cache miss
       Token Bucket (per-provider tokens/interval)
          ↓ tokens available
       Provider API → success? → cache + return
          ↓ 429 / rate limited
       Exponential Backoff (respects Retry-After header)
          ↓ exhausted
       Fallback → next provider in chain
```

### Implementation — `ratelimit.py`

Create a standalone module that the MCP server imports:

```python
"""ratelimit.py — Token bucket rate limiter + TTL cache for MCP providers."""

import time, hashlib, json, asyncio

# Per-provider rate limit config
RATE_LIMITS = {
    "searxng":        {"tokens": 30, "interval": 1.0},   # local — high limit
    "exa":            {"tokens": 10, "interval": 1.0},   # paid tier
    "firecrawl":      {"tokens": 5,  "interval": 60.0},  # 1k/mo budget
    "opencorporates": {"tokens": 1,  "interval": 1.0},  # 1/sec free tier
    "courtlistener":  {"tokens": 10, "interval": 6.0},  # ~100/min free
}

# Per-provider cache TTL in seconds
CACHE_TTL = {
    "searxng":        120,    # 2 min — web results change
    "exa":            300,    # 5 min
    "opencorporates": 3600,  # 1 hr — company records are stable
    "courtlistener":  3600,  # 1 hr — court cases are infrequent
    "firecrawl":      300,   # 5 min
}

class TokenBucket:
    """Async-safe token bucket for per-provider rate limiting."""
    def __init__(self, tokens, interval):
        self.max_tokens = tokens
        self.tokens = tokens
        self.interval = interval  # seconds between refills
        self.last_refill = time.time()

    async def acquire(self):
        now = time.time()
        if now - self.last_refill >= self.interval:
            self.tokens = self.max_tokens
            self.last_refill = now
        if self.tokens <= 0:
            return False
        self.tokens -= 1
        return True

_buckets = {}
_cache = {}

async def rate_limited_search(provider: str, cache_key: str, fn, *args, **kwargs):
    """Wrapper: check cache → rate limit → call → cache result."""
    # 1. Cache check
    cache_key_full = f"{provider}:{cache_key}"
    if cache_key_full in _cache:
        entry = _cache[cache_key_full]
        if time.time() - entry["ts"] < CACHE_TTL.get(provider, 300):
            return entry["data"]

    # 2. Rate limit check
    rl = RATE_LIMITS.get(provider)
    if rl:
        if provider not in _buckets:
            _buckets[provider] = TokenBucket(rl["tokens"], rl["interval"])
        allowed = await _buckets[provider].acquire()
        if not allowed:
            raise RateLimitedError(f"{provider}: token bucket empty")

    # 3. Call with retry
    result = await _retry_with_backoff(fn, *args, **kwargs)

    # 4. Cache result
    _cache[cache_key_full] = {"data": result, "ts": time.time()}
    return result

async def _retry_with_backoff(fn, *args, retries=3, base_delay=1.0):
    """Exponential backoff on 429/5xx, respects Retry-After."""
    for attempt in range(retries):
        try:
            return await fn(*args, **kwargs)
        except Exception as e:
            if attempt == retries - 1:
                raise
            delay = base_delay * (2 ** attempt)
            # Check for Retry-After in response headers
            await asyncio.sleep(delay)
```

### Key decisions
- **Cache aggressive for stable data** — company records and court cases get 1-hour TTLs since they rarely change. Web search results get 2-5 minutes.
- **Token bucket per provider** — each provider has its own limit so one fast provider can't starve another.
- **Fall through on rate limit** — don't fail the whole tool. The MCP server's fallback chain should catch the RateLimitedError and try the next provider.

## Multi-Provider Search Routing Pattern

When building a search/extraction MCP server that uses multiple backends, use this fallback chain pattern (incorporating rate limiting):

```python
async def web_search(query, limit=5):
    # 1. Try SearXNG (free, local)
    try:
        results = await _searxng_search(query, limit)
        if results:
            return {"provider": "searxng", "results": results}
    except:
        pass
    
    # 2. Fallback to Exa (premium, API key)
    try:
        results = await _exa_search(query, limit)
        if results:
            return {"provider": "exa", "results": results}
    except:
        pass
    
    # 3. Last resort: Firecrawl (paid, most resilient)
    results = await _firecrawl_search(query, limit)
    return {"provider": "firecrawl", "results": results, "telemetry": telemetry}
```

This ensures the service degrades gracefully — free first, premium second, paid last resort.

**Telemetry rule:** Multi-provider MCP tools must return provider telemetry, not just final results. Include providers attempted, status/error, result counts, and upstream health details (for example, SearXNG `unresponsive_engines` with CAPTCHA/rate-limit reasons). Do not silently hide a provider that is reachable but returning zero useful results because upstream engines are blocked.

## Remote Database MCP Data Source

When an MCP server needs to read from a database on a remote host (e.g., Traccar H2 on app2), use the SSH+SCP+query pattern: copy the DB as a read-only snapshot, query locally with Java H2 Shell, parse pipe-delimited output, return structured data. Avoids API auth, CORS, and credential management for read-only queries. Full reference: `references/h2-remote-db-query.md`.

This pattern works for H2, SQLite, and any file-based database accessible via SSH.

## Docker-Based MCP Deployment (alongside Open WebUI)

When deploying MCP servers on app1 to serve Open WebUI (and Hermes), use Docker Compose instead of systemd. Add the MCP service to the existing LiteLLM compose at `/root/docker/litellm/docker-compose.yml`.

### Pattern

```yaml
services:
  super-search:
    build: ../super-search          # Directory with server.py + Dockerfile
    container_name: super-search
    restart: unless-stopped
    environment:
      - EXA_API_KEY=<key>
      - FIRECRAWL_API_KEY=<key>
      - PYTHONUNBUFFERED=1
    ports:
      - '127.0.0.1:8899:8899'       # Bound to 127.0.0.1 — services on same host
    networks:
      - litellm-net                  # Share LiteLLM's network
```

### Dockerfile (slim Python)

```dockerfile
FROM python:3.13-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY server.py ratelimit.py ./
ENV PYTHONUNBUFFERED=1
EXPOSE 8899
CMD ["python3", "server.py"]
```

### Health check pitfall — slim containers

Python slim images lack `curl`, `ss`, `pgrep`, and other common health check tools. Do NOT use shell-based health checks. Options:

1. **Skip it** — Remove the health check block entirely. Docker will still restart on crash.
2. **Python socket test** — `python3 -c "import socket; s=socket.socket(); s.connect(('127.0.0.1',8899)); s.close()"`
3. **Docker exec test from host** — `docker exec super-search python3 -c "..."`

### Addressing

- **Open WebUI (Docker → host):** `http://host.docker.internal:8899/mcp`
- **Hermes (host → Docker):** `http://127.0.0.1:8899/mcp` (since port is mapped to localhost)
- **Process test from inside container:** `http://127.0.0.1:8899/mcp`

### Verifying deployment

Use `docker exec` with FastMCP client to verify tools:

```bash
docker exec super-search python3 -c "
from fastmcp import Client
import asyncio
async def test():
    async with Client('http://127.0.0.1:8899/mcp') as client:
        tools = await client.list_tools()
        print(f'Tools: {len(tools)}')
        for t in tools:
            print(f'  - {t.name}')
asyncio.run(test())
"
```

## Wiring MCP into Open WebUI

Open WebUI stores tool server connections as a JSON blob in its SQLite DB at key `tool_server.connections`. Add MCP servers programmatically:

```python
import sqlite3, json, time
db = sqlite3.connect('/var/lib/docker/volumes/openwebui_data/_data/webui.db')

row = db.execute("SELECT value FROM config WHERE key='tool_server.connections'").fetchone()
current = json.loads(row[0]) if row else []

# Remove duplicates by URL
current = [c for c in current if '8899' not in c.get('url', '')]

current.append({
    'url': 'http://host.docker.internal:8899/mcp',
    'type': 'mcp',                      # CRITICAL: not 'openapi'
    'auth_type': 'none',
    'config': {
        'enable': True,
        'function_name_filter_list': '',
        'access_grants': [{
            'principal_type': 'group',
            'principal_id': '<group-id>',
            'permission': 'read'
        }]
    },
    'info': {'name': 'My MCP', 'description': 'Tools description'}
})

db.execute("UPDATE config SET value = ?, updated_at = ? WHERE key = ?",
    (json.dumps(current), int(time.time()), 'tool_server.connections'))
db.commit()
```

Then restart Open WebUI: `docker restart openwebui`

Users enable MCP tools per-chat via the **➕ → Integrations → Tools** menu.

## Pitfalls

- **yaml.dump drops top-level `networks:` section** — When editing docker-compose.yml with Python's `yaml.dump()`, the `networks:` block is silently removed on every write. This happened 2x in one session (Jul 15). Verify with `grep '^networks:' docker-compose.yml` after every YAML edit. If missing, restore from `.bak` and append new services via `cat >>` instead. You can also append the networks block manually: `echo -e '\nnetworks:\n  litellm-net:\n    driver: bridge' >> docker-compose.yml`
- **Slim Python images lack health check tools** — `curl`, `ss`, `pgrep`, and `wget` are NOT in `python:3.13-slim`. Skip Docker health checks or use `pgrep -f server.py`.
- **Port conflicts on app1** — Port 3000 is bound by Open WebUI. Check `ss -tlnp | grep <port>` before adding new MCP service ports.
- **Open WebUI stores MCP connections in SQLite** — not in env vars or docker-compose. Adding an MCP server requires a DB update + restart at `/var/lib/docker/volumes/openwebui_data/_data/webui.db`, key `tool_server.connections`.
- **Open WebUI `ConfigVar` env vars are ignored after first launch** — values persist in SQLite. Use `UPDATE config SET value=... WHERE key=...` for persistent config.
- **MCP connection type must be `\"mcp\"`, not `\"openapi\"`** — using OpenAPI type for MCP URLs causes infinite loading spinner.
- **`host.docker.internal` for Docker→host** — when Open WebUI (Docker) needs to reach an MCP server on the same host, use `http://host.docker.internal:<port>/mcp`, not `localhost`.
- **Hermes config changes to mcp_servers require gateway restart** — tools won't be available until restart.
- **HTTP transport needs SSE session negotiation** — curl POST won't work. Use `docker exec <name> python3 -c \"from fastmcp import Client...\"` for verification.
- **Subagent timeout** — default 600s is tight for MCP server builds. Subagents often return \"next steps\" without deploying — verify yourself or rebuild.

See `references/browser-puppeteer-mcp.md` for the full Playwright + browserless/chrome MCP pattern.
See `references/openwebui-config.md` for Open WebUI SQLite configuration reference.