Ops Dashboard Pattern

When to use

Build an infrastructure operations dashboard when you need a single pane of glass showing cron health, service status, backup freshness, API reachability, server inventory, and system resource usage -- all updating automatically.

Architecture

Python data collector (every 5m via no_agent cron)
        |
        v writes JSON to
   /var/www/<app>/data/ops-status.json
        |
        v served by Caddy
   <domain>/data/ops-status.json
        |
        v fetched by
   index.html (dark theme dashboard)
        |
        v auto-refresh every 30s
   dashboard stays live

Key pieces

1. Data collector script

Self-contained (stdlib only: subprocess, urllib, json, pathlib, os, datetime, socket)
Each data source wrapped in safe() -- one failure never crashes the whole cycle
Sources: Hermes cron jobs, systemd services, disk/memory, S3 backups (aws CLI), API health checks, versions, Hetzner server list, netcup server info
Output: /var/www/ops/data/ops-status.json (~6KB JSON)
Error log: /var/log/ops-collector.log
Runs via cron: --schedule "*/5 * * * *" --no_agent --script ops-data-collector.py

2. Dashboard HTML

Dark theme: bg #0f172a, cards #1e293b, amber accents #f59e0b
NO external dependencies (no Tailwind, no CDN, no frameworks) -- inline CSS + JS
Mobile-first responsive, PWA meta tags for iPhone Safari
Sections: Overall Health Bar, Scheduled Jobs, Services, S3 Backups, API Health, Server Inventory, Routers (placeholder), Versions
Fetch /data/ops-status.json on load, auto-refresh every 30s
If fetch fails: red banner "Dashboard data not available -- collector may be offline"

3. Caddy serving

ops.itpropartner.com {
    root * /var/www/ops
    encode gzip
    file_server
    log { output file /var/log/caddy/ops.log }
}

4. CORS for the data file

header /data/* {
    Access-Control-Allow-Origin "*"
}

Pitfalls

S3 auth requires endpoint-url for Wasabi: aws s3 ls --recursive --endpoint-url https://s3.us-east-1.wasabisys.com
Datetime timezone mismatch: S3 ls returns naive datetimes. The collector must .replace(tzinfo=timezone.utc) before subtracting.
Python 3.13 deprecation: datetime.utcnow() is deprecated. Use datetime.now(timezone.utc) instead.
AWS CLI path: must activate /opt/awscli-venv/bin/activate before calling aws commands
Placeholder sections: Show amber cards with "Coming soon" for empty sections

2.4 KiB Raw Permalink Blame History