# Ops Dashboard Pattern

## When to use

Build an infrastructure operations dashboard when you need a single pane of glass showing cron health, service status, backup freshness, API reachability, server inventory, and system resource usage -- all updating automatically.

## Architecture

```
Python data collector (every 5m via no_agent cron)
        |
        v writes JSON to
   /var/www/<app>/data/ops-status.json
        |
        v served by Caddy
   <domain>/data/ops-status.json
        |
        v fetched by
   index.html (dark theme dashboard)
        |
        v auto-refresh every 30s
   dashboard stays live
```

## Key pieces

### 1. Data collector script

- Self-contained (stdlib only: subprocess, urllib, json, pathlib, os, datetime, socket)
- Each data source wrapped in safe() -- one failure never crashes the whole cycle
- Sources: Hermes cron jobs, systemd services, disk/memory, S3 backups (aws CLI), API health checks, versions, Hetzner server list, netcup server info
- Output: /var/www/ops/data/ops-status.json (~6KB JSON)
- Error log: /var/log/ops-collector.log
- Runs via cron: --schedule "*/5 * * * *" --no_agent --script ops-data-collector.py

### 2. Dashboard HTML

- Dark theme: bg #0f172a, cards #1e293b, amber accents #f59e0b
- NO external dependencies (no Tailwind, no CDN, no frameworks) -- inline CSS + JS
- Mobile-first responsive, PWA meta tags for iPhone Safari
- Sections: Overall Health Bar, Scheduled Jobs, Services, S3 Backups, API Health, Server Inventory, Routers (placeholder), Versions
- Fetch /data/ops-status.json on load, auto-refresh every 30s
- If fetch fails: red banner "Dashboard data not available -- collector may be offline"

### 3. Caddy serving

```
ops.itpropartner.com {
    root * /var/www/ops
    encode gzip
    file_server
    log { output file /var/log/caddy/ops.log }
}
```

### 4. CORS for the data file

```
header /data/* {
    Access-Control-Allow-Origin "*"
}
```

## Pitfalls

- S3 auth requires endpoint-url for Wasabi: aws s3 ls --recursive --endpoint-url https://s3.us-east-1.wasabisys.com
- Datetime timezone mismatch: S3 ls returns naive datetimes. The collector must .replace(tzinfo=timezone.utc) before subtracting.
- Python 3.13 deprecation: datetime.utcnow() is deprecated. Use datetime.now(timezone.utc) instead.
- AWS CLI path: must activate /opt/awscli-venv/bin/activate before calling aws commands
- Placeholder sections: Show amber cards with "Coming soon" for empty sections