Initial skills documentation — 25 categories, all SKILL.md + references + scripts
This commit is contained in:
root
+86
@@ -0,0 +1,86 @@
|
||||
# H2-backed MCP snapshot refresh pitfalls
|
||||
|
||||
Use this for Traccar/FT360-style MCP servers that read a remote H2 database by copying `database.mv.db` locally and querying with Java H2 Shell.
|
||||
|
||||
## Parser pitfall: H2 header rows
|
||||
|
||||
H2 Shell output includes a header row like:
|
||||
|
||||
```text
|
||||
ID | NAME | UNIQUEID | ...
|
||||
```
|
||||
|
||||
If the MCP parser returns that as data, code like `int(row[0])` fails with `invalid literal for int() with base 10: 'ID'`.
|
||||
|
||||
Parser must skip header rows:
|
||||
|
||||
```python
|
||||
if cells[0].upper() in {"ID", "DEVICEID", "LATITUDE", "NAME"}:
|
||||
continue
|
||||
```
|
||||
|
||||
Also skip summary rows beginning with `(` and separator rows beginning with `---`.
|
||||
|
||||
## Snapshot freshness pitfall
|
||||
|
||||
Do not copy the remote H2 DB only when the local file is missing. That causes stale dashboard/MCP data for hours or days. If the tool has its own short response cache, refresh the local DB snapshot on every actual query after the cache expires:
|
||||
|
||||
```python
|
||||
def _query_db(sql):
|
||||
db_path = _scp_db() # always refresh snapshot when query runs
|
||||
if db_path is None:
|
||||
return []
|
||||
...
|
||||
```
|
||||
|
||||
A 30-second in-memory cache is enough to avoid excessive SCPs while keeping live tracking data fresh.
|
||||
|
||||
## Contact freshness vs GPS freshness
|
||||
|
||||
For Traccar-like data, do not equate device `online` or `lastupdate` with a fresh GPS fix. Traccar can update device contact time when a mobile client sends a heartbeat/notification-token payload without new coordinates.
|
||||
|
||||
Expose both concepts separately:
|
||||
|
||||
| Field | Meaning |
|
||||
|---|---|
|
||||
| `last_contact_time` / `last_contact_age_minutes` | Device/server contact freshness (`tc_devices.lastupdate`) |
|
||||
| `gps_fix_time` / `gps_age_minutes` | Real GPS fix age (`tc_positions.fixtime`, fallback to `devicetime`) |
|
||||
| `is_contact_fresh` | `last_contact_age_minutes <= threshold` |
|
||||
| `is_location_fresh` | `gps_age_minutes <= threshold` |
|
||||
|
||||
A valid OsmAnd GPS payload looks like:
|
||||
|
||||
```text
|
||||
id=612982&lat=10.417...&lon=-75.551...×tamp=...&accuracy=...
|
||||
```
|
||||
|
||||
A heartbeat/notification-token payload may look like:
|
||||
|
||||
```text
|
||||
id=612982¬ificationToken=...
|
||||
```
|
||||
|
||||
The latter can cause Traccar to keep the device online while reusing the last known coordinates. Dashboard wording should say `contact live / GPS stale` or equivalent, not `live location`.
|
||||
|
||||
## Static dashboard export pattern
|
||||
|
||||
For a static ops page, generate a JSON file from the MCP/backend on a short `no_agent=True` cron:
|
||||
|
||||
```text
|
||||
/root/.hermes/scripts/ft360-export.py -> /var/www/ops/data/ft360-devices.json
|
||||
schedule: every 1m
|
||||
no_agent: true
|
||||
```
|
||||
|
||||
If the exporter imports MCP code that depends on a venv (for example FastMCP in `/opt/ops-portal/venv`), either run the exporter with the venv Python or add that venv's `site-packages` before importing:
|
||||
|
||||
```python
|
||||
import site
|
||||
site.addsitedir('/opt/ops-portal/venv/lib/python3.13/site-packages')
|
||||
```
|
||||
|
||||
The page should fetch the JSON with `cache: 'no-store'` and display contact age separately from GPS fix age.
|
||||
|
||||
## Verification
|
||||
|
||||
After patching, restart the MCP service and call the MCP tool, not just `hermes mcp test`. `hermes mcp test` verifies tool discovery only; a real `get_devices` call verifies parser and freshness.
|
||||
+165
@@ -0,0 +1,165 @@
|
||||
# H2 Traccar Database Integration
|
||||
|
||||
Reference for MCP servers that query a remote Traccar H2 database via SSH + Java subprocess.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
MCP Server (on Core)
|
||||
└── on each tool call:
|
||||
1. SSH cat from app2 → /tmp/ft360_db/database.mv.db (local file cache)
|
||||
2. java -cp /tmp/h2.jar org.h2.tools.Shell → query
|
||||
3. Parse pipe-delimited output
|
||||
4. Return JSON
|
||||
```
|
||||
|
||||
## Database Copy (SSH cat — avoid scp scanner block)
|
||||
|
||||
```bash
|
||||
# scp is often blocked by raw-IP security scanner; use ssh cat instead:
|
||||
ssh -o StrictHostKeyChecking=no -o ConnectTimeout=15 \
|
||||
-i /root/.ssh/itpp-infra root@152.53.39.202 \
|
||||
"cat /root/docker/traccar/traccar-data/database.mv.db" \
|
||||
> /tmp/ft360_db/database.mv.db
|
||||
```
|
||||
|
||||
In Python:
|
||||
```python
|
||||
cmd = ["ssh", "-o", "StrictHostKeyChecking=no", "-i", SSH_KEY,
|
||||
f"root@{APP2_HOST}", f"cat {REMOTE_DB}"]
|
||||
with open(LOCAL_DB_PATH, "wb") as f:
|
||||
subprocess.run(cmd, stdout=f, timeout=30)
|
||||
```
|
||||
|
||||
## H2 JDBC URL
|
||||
|
||||
The H2 MVStore file is `database.mv.db`. Strip the `.mv.db` extension for the JDBC URL:
|
||||
|
||||
```
|
||||
jdbc:h2:file:/tmp/ft360_db/database;IFEXISTS=TRUE
|
||||
```
|
||||
|
||||
## Query via H2 Shell
|
||||
|
||||
```bash
|
||||
java -cp /tmp/h2.jar org.h2.tools.Shell \
|
||||
-url "jdbc:h2:file:/tmp/ft360_db/database;IFEXISTS=TRUE" \
|
||||
-user sa -password "" \
|
||||
-sql "SELECT id, name, uniqueid FROM tc_devices"
|
||||
```
|
||||
|
||||
Default credentials: user=`sa`, password=`""` (empty).
|
||||
|
||||
## Output Parsing
|
||||
|
||||
H2 Shell produces pipe-delimited tabular output:
|
||||
|
||||
```
|
||||
ID | NAME | UNIQUEID | LASTUPDATE | STATUS
|
||||
1 | GMB-iPhone | 612982 | 2026-07-13 19:34:27.841 | offline
|
||||
(1 row, 15 ms)
|
||||
```
|
||||
|
||||
Parse with three mandatory safety checks:
|
||||
|
||||
```python
|
||||
def parse_h2_output(output: str) -> list[list[str]]:
|
||||
rows = []
|
||||
for line in output.strip().split("\n"):
|
||||
stripped = line.strip()
|
||||
if not stripped or stripped.startswith("(") or stripped.startswith("---"):
|
||||
continue
|
||||
cells = [c.strip() for c in stripped.split("|")]
|
||||
if all(c == "" for c in cells):
|
||||
continue
|
||||
# (1) MANDATORY: skip column header rows like "ID | NAME | ..."
|
||||
# H2 Shell repeats headers in every result set. Passing these
|
||||
# to int("ID") or float("NAME") crashes the parser at runtime.
|
||||
if cells[0].upper() in {"ID", "DEVICEID", "LATITUDE", "NAME", "COUNT(*)"}:
|
||||
continue
|
||||
rows.append(cells)
|
||||
return rows
|
||||
```
|
||||
|
||||
- **Header row crash** — H2 Shell repeats `ID | NAME | DEVICEID | ...` headers in every result set. Passing them directly to `int("ID")` or `float("LATITUDE")` crashes the parser with `ValueError`. Skip rows whose first cell matches case-folded known header names. The `all(c == "")` check above is not sufficient: header rows have real content that passes it.
|
||||
- **Stale snapshot** — Always SCP the remote DB before each tool call. An `if not exists` guard (copy once, cache forever) silently returns hours/days-old data while the live DB has fresh positions. The MCP cache TTL controls tool response caching, not DB snapshot age. Use `_scp_db()` unconditionally before `_query_db()`.
|
||||
- **Server-local timestamps, not UTC** — H2 timestamps from a Traccar deployment on app2 are stored in the container's local timezone (Eastern, matching the server's `date`), not UTC. Age calculations using `datetime.utcnow()` will be off by the timezone offset (4 hours in this case). Use `datetime.now()` for naive timestamps stored in server-local time.
|
||||
|
||||
## Schema
|
||||
|
||||
```sql
|
||||
-- tc_devices
|
||||
SELECT id, name, uniqueid, lastupdate, status FROM tc_devices;
|
||||
|
||||
-- tc_positions
|
||||
SELECT id, deviceid, latitude, longitude, speed, devicetime FROM tc_positions;
|
||||
```
|
||||
|
||||
- `speed` is in **knots**. Convert to mph: `speed_mph = speed_kn * 1.15078`.
|
||||
- `devicetime` format: `2026-07-13 19:34:27.841` (fractional seconds optional).
|
||||
|
||||
## Key Queries
|
||||
|
||||
### Devices with latest position
|
||||
```sql
|
||||
SELECT d.id, d.name, d.uniqueid, d.lastupdate, d.status,
|
||||
p.latitude, p.longitude, p.speed, p.devicetime
|
||||
FROM tc_devices d
|
||||
LEFT JOIN (
|
||||
SELECT deviceid, latitude, longitude, speed, devicetime,
|
||||
ROW_NUMBER() OVER (PARTITION BY deviceid ORDER BY devicetime DESC) rn
|
||||
FROM tc_positions
|
||||
) p ON d.id = p.deviceid AND p.rn = 1
|
||||
ORDER BY d.id
|
||||
```
|
||||
|
||||
### Position history (last N hours)
|
||||
```sql
|
||||
SELECT latitude, longitude, speed, devicetime
|
||||
FROM tc_positions
|
||||
WHERE deviceid = 1
|
||||
AND devicetime >= DATEADD('HOUR', -6, NOW())
|
||||
ORDER BY devicetime ASC
|
||||
```
|
||||
|
||||
## Distance Calculation (Haversine)
|
||||
|
||||
```python
|
||||
import math
|
||||
|
||||
def haversine_miles(lat1, lon1, lat2, lon2):
|
||||
R = 3958.8 # Earth radius in miles
|
||||
dlat = math.radians(lat2 - lat1)
|
||||
dlon = math.radians(lon2 - lon1)
|
||||
a = (math.sin(dlat / 2) ** 2 +
|
||||
math.cos(math.radians(lat1)) * math.cos(math.radians(lat2)) *
|
||||
math.sin(dlon / 2) ** 2)
|
||||
c = 2 * math.atan2(math.sqrt(a), math.sqrt(1 - a))
|
||||
return R * c
|
||||
```
|
||||
|
||||
Total trip distance = sum of haversine distances between consecutive positions.
|
||||
|
||||
## Caching
|
||||
|
||||
30-second TTL in-memory dict cache per tool/key avoids redundant SSH + H2 calls:
|
||||
|
||||
```python
|
||||
_cache: dict[str, tuple[float, str]] = {}
|
||||
CACHE_TTL = 30
|
||||
|
||||
def _cache_get(key: str) -> str | None:
|
||||
entry = _cache.get(key)
|
||||
if entry is None: return None
|
||||
ts, value = entry
|
||||
if time.time() - ts > CACHE_TTL:
|
||||
del _cache[key]
|
||||
return None
|
||||
return value
|
||||
```
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **H2 URL without `;IFEXISTS=TRUE`** → the Shell tool creates a new empty database if the file doesn't exist, masking copy failures. Always use `IFEXISTS=TRUE`.
|
||||
- **Timestamp parsing** — `devicetime` may or may not include fractional seconds (`.841`). Use `timestamp_string[:19]` when parsing with `datetime.strptime`.
|
||||
- **Database locked by running Traccar** — the MVStore format supports reading while Traccar is running, but concurrent writes during the `ssh cat` may produce a slightly stale snapshot. Acceptable for read-only analytics at 30s cache TTL.
|
||||
@@ -0,0 +1,93 @@
|
||||
# Immich photo map MCP pattern
|
||||
|
||||
## Use case
|
||||
|
||||
Build a read-only Immich MCP that extracts asset metadata and exposes geotagged photos as structured data for maps and summaries.
|
||||
|
||||
## Immich endpoints
|
||||
|
||||
Useful Immich API endpoints:
|
||||
|
||||
```text
|
||||
POST /search/metadata
|
||||
GET /assets/{id}/metadata
|
||||
```
|
||||
|
||||
For asset searches, request EXIF in results:
|
||||
|
||||
```json
|
||||
{
|
||||
"withExif": true,
|
||||
"size": 500,
|
||||
"takenAfter": "2026-01-01T00:00:00Z",
|
||||
"takenBefore": "2026-12-31T23:59:59Z"
|
||||
}
|
||||
```
|
||||
|
||||
## MCP tool shape
|
||||
|
||||
Recommended tools:
|
||||
|
||||
- `immich_search_geo_assets(taken_after, taken_before, album_id=None, limit=500)` — returns assets with latitude/longitude, taken date, location labels, camera info, and thumbnail URL.
|
||||
- `immich_get_asset_metadata(asset_id)` — returns full metadata for one asset.
|
||||
- `immich_geojson(...)` — returns a GeoJSON `FeatureCollection` for Leaflet/OpenStreetMap.
|
||||
- `immich_map_summary(...)` — returns top locations, missing-GPS count, and date-range summary.
|
||||
|
||||
## Output contracts
|
||||
|
||||
Asset record:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "asset-uuid",
|
||||
"filename": "IMG_1234.jpeg",
|
||||
"taken_at": "2026-07-14T10:31:00",
|
||||
"latitude": 32.0809,
|
||||
"longitude": -81.0912,
|
||||
"city": "Savannah",
|
||||
"state": "Georgia",
|
||||
"country": "United States",
|
||||
"camera_make": "Apple",
|
||||
"camera_model": "iPhone 15 Pro",
|
||||
"thumb_url": "/api/assets/{id}/thumbnail"
|
||||
}
|
||||
```
|
||||
|
||||
GeoJSON feature:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "Feature",
|
||||
"geometry": {"type": "Point", "coordinates": [-81.0912, 32.0809]},
|
||||
"properties": {"asset_id": "uuid", "filename": "IMG_1234.jpeg", "taken_at": "..."}
|
||||
}
|
||||
```
|
||||
|
||||
## Map UI
|
||||
|
||||
The MCP provides data; the web UI can be Leaflet.js with OpenStreetMap tiles, marker clustering, album/date filters, and thumbnail popups.
|
||||
|
||||
Recommended internal paths:
|
||||
|
||||
```text
|
||||
ops.itpropartner.com/immich-map.html
|
||||
or
|
||||
photos.itpropartner.com/map
|
||||
```
|
||||
|
||||
## Security notes
|
||||
|
||||
Photo GPS data is sensitive.
|
||||
|
||||
- Keep MCP localhost-only.
|
||||
- Store `IMMICH_BASE_URL` and `IMMICH_API_KEY` in a chmod 600 `.env` / systemd `EnvironmentFile`.
|
||||
- Put any map UI behind existing auth/Cloudflare Access.
|
||||
- Prefer thumbnails over full-resolution images.
|
||||
- Count missing GPS as `missing_location`, not as errors.
|
||||
|
||||
## Build order
|
||||
|
||||
1. Read-only MCP first: search, metadata, GeoJSON, summaries.
|
||||
2. Verify against a small date range.
|
||||
3. Build map UI.
|
||||
4. Only later consider metadata update/fix tools.
|
||||
+197
@@ -0,0 +1,197 @@
|
||||
# Super Search — Rate Limiting & Caching Reference
|
||||
|
||||
Full implementation of `ratelimit.py` used in the Super Search MCP server (`/root/docker/super-search/ratelimit.py`).
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─ Cache HIT? ──→ return cached result
|
||||
│
|
||||
Caller ──→ rate_limited_search(provider)
|
||||
│
|
||||
├─ Cache MISS ──→ acquire rate-limit token (TokenBucket)
|
||||
│ │
|
||||
│ ┌──────┘
|
||||
│ ├─ 200 OK ──→ cache.set(TTL) ──→ return
|
||||
│ ├─ 429/5xx ──→ _retry_with_backoff (max 3)
|
||||
│ └─ permanent failure ──→ raise
|
||||
```
|
||||
|
||||
## TokenBucket
|
||||
|
||||
Async-aware token bucket with `acquire()` that blocks until a token is available:
|
||||
|
||||
```python
|
||||
class TokenBucket:
|
||||
def __init__(self, tokens: float, interval: float) -> None:
|
||||
self._capacity = float(tokens)
|
||||
self._tokens = float(tokens)
|
||||
self._interval = float(interval)
|
||||
self._refill_rate = self._capacity / self._interval
|
||||
self._last_refill = time.monotonic()
|
||||
self._lock = asyncio.Lock()
|
||||
|
||||
def _refill(self) -> None:
|
||||
now = time.monotonic()
|
||||
elapsed = now - self._last_refill
|
||||
self._tokens = min(self._capacity, self._tokens + elapsed * self._refill_rate)
|
||||
self._last_refill = now
|
||||
|
||||
async def acquire(self) -> None:
|
||||
async with self._lock:
|
||||
self._refill()
|
||||
if self._tokens >= 1.0:
|
||||
self._tokens -= 1.0
|
||||
return
|
||||
wait = (1.0 - self._tokens) / self._refill_rate
|
||||
await asyncio.sleep(wait)
|
||||
async with self._lock:
|
||||
self._refill()
|
||||
self._tokens -= 1.0
|
||||
```
|
||||
|
||||
## Provider Rate Limits
|
||||
|
||||
Configured per provider based on API tier and pricing:
|
||||
|
||||
| Provider | Tokens | Interval | Effective Rate | Rationale |
|
||||
|---------------|--------|----------|---------------|-------------------------------|
|
||||
| searxng | 30 | 1.0s | 30/sec | Local instance, virtually free |
|
||||
| exa | 10 | 1.0s | 10/sec | Paid tier, moderate allowance |
|
||||
| firecrawl | 5 | 60.0s | ~0.08/sec | 1k/month free tier, conserve |
|
||||
| opencorporates| 1 | 1.0s | ~1/sec | Free tier, no API key |
|
||||
| courtlistener | 10 | 6.0s | ~1.67/sec | Free tier, ~100/min max |
|
||||
|
||||
## TTLCache
|
||||
|
||||
Dict-based with async lock and per-key expiration:
|
||||
|
||||
```python
|
||||
class TTLCache:
|
||||
def __init__(self) -> None:
|
||||
self._cache: dict[str, tuple[Any, float]] = {} # key → (value, expires_at)
|
||||
self._lock = asyncio.Lock()
|
||||
|
||||
async def get(self, key: str) -> Optional[Any]:
|
||||
async with self._lock:
|
||||
entry = self._cache.get(key)
|
||||
if entry is None:
|
||||
return None
|
||||
value, expires_at = entry
|
||||
if time.monotonic() >= expires_at:
|
||||
del self._cache[key]
|
||||
return None
|
||||
return value
|
||||
|
||||
async def set(self, key: str, value: Any, ttl: int) -> None:
|
||||
async with self._lock:
|
||||
self._cache[key] = (value, time.monotonic() + ttl)
|
||||
```
|
||||
|
||||
## Cache TTLs
|
||||
|
||||
| Provider | TTL | Rationale |
|
||||
|---------------|--------|--------------------------------------------------|
|
||||
| searxng | 120s | Web results change frequently |
|
||||
| exa | 300s | Neural search — moderate freshness |
|
||||
| opencorporates| 3600s | Company records don't change often |
|
||||
| courtlistener | 3600s | Court opinions are infrequent additions |
|
||||
|
||||
Cache key format: `{provider}:{query.strip().lower()}:{limit}`
|
||||
|
||||
## Retry with Backoff
|
||||
|
||||
Handles transient failures (429, 503, 502, 504) with exponential backoff:
|
||||
|
||||
```python
|
||||
RETRYABLE_STATUSES: set[int] = {429, 503, 502, 504}
|
||||
|
||||
async def _retry_with_backoff(
|
||||
coro_factory: Callable[[], Any],
|
||||
provider: str,
|
||||
max_retries: int = 3,
|
||||
base_delay: float = 1.0,
|
||||
) -> Any:
|
||||
for attempt in range(max_retries + 1):
|
||||
try:
|
||||
result = await coro_factory()
|
||||
return result
|
||||
except httpx.HTTPStatusError as e:
|
||||
status = e.response.status_code
|
||||
if status not in RETRYABLE_STATUSES or attempt == max_retries:
|
||||
raise
|
||||
retry_after = e.response.headers.get("Retry-After", "")
|
||||
if retry_after and retry_after.isdigit():
|
||||
delay = float(retry_after)
|
||||
else:
|
||||
delay = base_delay * (2 ** attempt)
|
||||
logger.warning(
|
||||
"Provider %s returned %d (attempt %d/%d). Retrying in %.1fs…",
|
||||
provider, status, attempt + 1, max_retries, delay,
|
||||
)
|
||||
await asyncio.sleep(delay)
|
||||
```
|
||||
|
||||
Delays: 1s → 2s → 4s → 8s. Respects `Retry-After` header when present.
|
||||
|
||||
## rate_limited_search Decorator
|
||||
|
||||
Composite decorator: cache-check → rate-limit → retry → cache-store:
|
||||
|
||||
```python
|
||||
def rate_limited_search(provider: str):
|
||||
def decorator(func):
|
||||
@wraps(func)
|
||||
async def wrapper(query: str, limit: int = 5, **kwargs):
|
||||
# 1. Check cache
|
||||
key = cache_key(provider, query, limit)
|
||||
cached = await _cache.get(key)
|
||||
if cached is not None:
|
||||
return cached
|
||||
|
||||
# 2. Acquire rate-limit token
|
||||
bucket = get_bucket(provider)
|
||||
await bucket.acquire()
|
||||
|
||||
# 3. Execute with retry
|
||||
ttl = CACHE_TTL.get(provider, 300)
|
||||
|
||||
async def _call():
|
||||
return await func(query, limit=limit, **kwargs)
|
||||
|
||||
result = await _retry_with_backoff(_call, provider)
|
||||
|
||||
# 4. Cache result (only on success)
|
||||
if result:
|
||||
await _cache.set(key, result, ttl)
|
||||
|
||||
return result
|
||||
return wrapper
|
||||
return decorator
|
||||
```
|
||||
|
||||
## Usage in server.py
|
||||
|
||||
```python
|
||||
from ratelimit import rate_limited_search
|
||||
|
||||
@rate_limited_search("opencorporates")
|
||||
async def _opencorporates_search(query: str, limit: int = 5) -> list[dict]:
|
||||
"""Search company records via OpenCorporates API."""
|
||||
# ... implementation — rate limiting + caching handled by decorator
|
||||
|
||||
@rate_limited_search("courtlistener")
|
||||
async def _courtlistener_search(query: str, limit: int = 5) -> list[dict]:
|
||||
"""Search US court opinions via CourtListener API."""
|
||||
# ... implementation
|
||||
```
|
||||
|
||||
## Dependencies
|
||||
|
||||
- `asyncio` (stdlib)
|
||||
- `time` (stdlib)
|
||||
- `logging` (stdlib)
|
||||
- `functools.wraps` (stdlib)
|
||||
- `httpx` (for HTTPStatusError type in retry handler only)
|
||||
|
||||
No external dependencies required for the rate-limiting module itself.
|
||||
Reference in New Issue
Block a user