hermes-skills/skills/devops/ops-portal-and-collector/references/fastapi-backend-pattern.md

# FastAPI Ops Portal Backend Pattern

Reference architecture for the FastAPI backend built Jul 9, 2026 at `/opt/ops-portal/server.py`.

## Route ordering (CRITICAL)

FastAPI processes routes in definition order. The route list must be:

1. **API routes first** (`/api/auth/login`, `/api/health`, `/api/status`, ...)
2. **Metrics** (`/metrics`)
3. **Static assets** (`/css/{path}`, `/js/{path}`)
4. **Page routes** (`/`, `/{page}.html`)

If a catch-all `/{page}.html` comes before `/api/*`, ALL API calls return "Page not found".

## Static file serving pattern

```python
# Do NOT use app.mount("/", StaticFiles(...)) — this swallows API routes.
# Instead, serve explicitly:

STATIC_DIR = Path(__file__).parent / "static"

@app.get("/css/{file_path:path}")
async def serve_css(file_path: str):
    fpath = STATIC_DIR / "css" / file_path
    if fpath.exists() and fpath.is_file():
        return FileResponse(str(fpath))
    raise HTTPException(status_code=404, detail="Not found")

@app.get("/js/{file_path:path}")
async def serve_js(file_path: str):
    fpath = STATIC_DIR / "js" / file_path
    if fpath.exists() and fpath.is_file():
        return FileResponse(str(fpath))
    raise HTTPException(status_code=404, detail="Not found")

@app.get("/")
async def serve_index():
    index_path = STATIC_DIR / "index.html"
    if index_path.exists():
        return FileResponse(str(index_path))
    return JSONResponse({"status": "ok", "message": "Ops Portal API ready"})

@app.get("/{page}.html")
async def serve_html_page(page: str):
    fpath = STATIC_DIR / f"{page}.html"
    if fpath.exists():
        return FileResponse(str(fpath))
    raise HTTPException(status_code=404, detail="Page not found")
```

## JWT auth pattern

```python
from jose import jwt, JWTError
from fastapi import Depends, HTTPException, Request as FastAPIRequest
from datetime import datetime, timezone, timedelta

ADMIN_USERNAME = "germaine"
ADMIN_PASSWORD = "..."  # from .env
JWT_SECRET = "..."      # from .env
JWT_ALGORITHM = "HS256"
JWT_EXPIRE_HOURS = 24

def create_jwt(username: str) -> str:
    payload = {
        "sub": username,
        "iat": datetime.now(timezone.utc),
        "exp": datetime.now(timezone.utc) + timedelta(hours=JWT_EXPIRE_HOURS),
        "jti": secrets.token_hex(16),
    }
    return jwt.encode(payload, JWT_SECRET, algorithm=JWT_ALGORITHM)

async def get_current_user(request: FastAPIRequest) -> str:
    auth = request.headers.get("Authorization", "")
    if not auth.startswith("Bearer "):
        raise HTTPException(status_code=401, detail="Missing or invalid Authorization header")
    token = auth[7:]
    payload = jwt.decode(token, JWT_SECRET, algorithms=[JWT_ALGORITHM])
    return payload["sub"]

# Usage:
@app.get("/api/status")
async def get_status(user: str = Depends(get_current_user)):
    ...
```

## Audit log pattern (SQLite)

```python
import sqlite3

DB_PATH = "/opt/ops-portal/ops.db"

def get_db():
    conn = sqlite3.connect(DB_PATH)
    conn.row_factory = sqlite3.Row
    conn.execute("PRAGMA journal_mode=WAL")
    return conn

def init_db():
    conn = get_db()
    conn.executescript("""
        CREATE TABLE IF NOT EXISTS audit_log (
            id INTEGER PRIMARY KEY AUTOINCREMENT,
            timestamp TEXT NOT NULL DEFAULT (datetime('now')),
            user TEXT NOT NULL,
            action TEXT NOT NULL,
            target TEXT,
            result TEXT NOT NULL,
            detail TEXT
        );
        CREATE TABLE IF NOT EXISTS api_tokens (
            id INTEGER PRIMARY KEY AUTOINCREMENT,
            user TEXT NOT NULL,
            token_hash TEXT NOT NULL UNIQUE,
            created TEXT NOT NULL DEFAULT (datetime('now')),
            last_used TEXT
        );
    """)
    conn.commit()
    conn.close()

def log_audit(user: str, action: str, target, result: str, detail=None):
    conn = get_db()
    conn.execute(
        "INSERT INTO audit_log (user, action, target, result, detail) VALUES (?, ?, ?, ?, ?)",
        (user, action, target, result, detail)
    )
    conn.commit()
    conn.close()
```

## Service control pattern (systemd)

```python
import subprocess
from fastapi import HTTPException

@app.post("/api/services/{name}/restart")
async def restart_service(name: str, user: str = Depends(get_current_user)):
    svc = name if name.endswith(".service") else f"{name}.service"
    try:
        result = subprocess.run(
            ["systemctl", "restart", svc],
            capture_output=True, text=True, timeout=30
        )
        success = result.returncode == 0
        log_audit(user, "restart_service", name, "ok" if success else "error", result.stderr[:500])
        if success:
            return {"status": "ok", "message": f"Service {name} restarted"}
        return JSONResponse(status_code=500, content={"status": "error", "message": result.stderr[:500]})
    except subprocess.TimeoutExpired:
        raise HTTPException(status_code=504, detail="Service restart timed out")
```

## Prometheus counters

```python
from prometheus_client import Counter, generate_latest, CONTENT_TYPE_LATEST
from fastapi.responses import JSONResponse

HTTP_REQUESTS = Counter("ops_portal_http_requests_total", "Total HTTP requests", ["method", "endpoint"])
SERVICE_ACTIONS = Counter("ops_portal_service_actions_total", "Service actions", ["action", "service"])
BACKUP_TRIGGERS = Counter("ops_portal_backup_triggers_total", "Backup triggers", ["bucket"])

@app.get("/metrics")
async def metrics():
    return JSONResponse(
        content=generate_latest().decode(),
        media_type=CONTENT_TYPE_LATEST
    )
```

## .env file layout

```
# /opt/ops-portal/.env — chmod 600
ADMIN_USERNAME=germaine
ADMIN_PASSWORD=...
JWT_SECRET=...
CLOUDFLARE_API_TOKEN=...
CLOUDFLARE_EMAIL=...
HETZNER_API_TOKEN=...
```

## Systemd service

```ini
[Unit]
Description=ITPP Ops Portal Backend
After=network.target

[Service]
Type=simple
User=root
WorkingDirectory=/opt/ops-portal
ExecStart=/opt/awscli-venv/bin/uvicorn server:app --host 127.0.0.1 --port 8090
Restart=always
RestartSec=5
StandardOutput=journal
StandardError=journal
EnvironmentFile=/opt/ops-portal/.env

[Install]
WantedBy=multi-user.target
```

## Caddy routing

```caddy
ops.itpropartner.com {
    reverse_proxy 127.0.0.1:8090
    encode gzip
    log {
        output file /var/log/caddy/ops.log
    }
}
```

The entire Caddy block is just a reverse proxy — no static file serving, no API matchers. The FastAPI backend handles everything.