71 lines
2.4 KiB
Markdown
71 lines
2.4 KiB
Markdown
# Ops Dashboard Pattern
|
|
|
|
## When to use
|
|
|
|
Build an infrastructure operations dashboard when you need a single pane of glass showing cron health, service status, backup freshness, API reachability, server inventory, and system resource usage -- all updating automatically.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
Python data collector (every 5m via no_agent cron)
|
|
|
|
|
v writes JSON to
|
|
/var/www/<app>/data/ops-status.json
|
|
|
|
|
v served by Caddy
|
|
<domain>/data/ops-status.json
|
|
|
|
|
v fetched by
|
|
index.html (dark theme dashboard)
|
|
|
|
|
v auto-refresh every 30s
|
|
dashboard stays live
|
|
```
|
|
|
|
## Key pieces
|
|
|
|
### 1. Data collector script
|
|
|
|
- Self-contained (stdlib only: subprocess, urllib, json, pathlib, os, datetime, socket)
|
|
- Each data source wrapped in safe() -- one failure never crashes the whole cycle
|
|
- Sources: Hermes cron jobs, systemd services, disk/memory, S3 backups (aws CLI), API health checks, versions, Hetzner server list, netcup server info
|
|
- Output: /var/www/ops/data/ops-status.json (~6KB JSON)
|
|
- Error log: /var/log/ops-collector.log
|
|
- Runs via cron: --schedule "*/5 * * * *" --no_agent --script ops-data-collector.py
|
|
|
|
### 2. Dashboard HTML
|
|
|
|
- Dark theme: bg #0f172a, cards #1e293b, amber accents #f59e0b
|
|
- NO external dependencies (no Tailwind, no CDN, no frameworks) -- inline CSS + JS
|
|
- Mobile-first responsive, PWA meta tags for iPhone Safari
|
|
- Sections: Overall Health Bar, Scheduled Jobs, Services, S3 Backups, API Health, Server Inventory, Routers (placeholder), Versions
|
|
- Fetch /data/ops-status.json on load, auto-refresh every 30s
|
|
- If fetch fails: red banner "Dashboard data not available -- collector may be offline"
|
|
|
|
### 3. Caddy serving
|
|
|
|
```
|
|
ops.itpropartner.com {
|
|
root * /var/www/ops
|
|
encode gzip
|
|
file_server
|
|
log { output file /var/log/caddy/ops.log }
|
|
}
|
|
```
|
|
|
|
### 4. CORS for the data file
|
|
|
|
```
|
|
header /data/* {
|
|
Access-Control-Allow-Origin "*"
|
|
}
|
|
```
|
|
|
|
## Pitfalls
|
|
|
|
- S3 auth requires endpoint-url for Wasabi: aws s3 ls --recursive --endpoint-url https://s3.us-east-1.wasabisys.com
|
|
- Datetime timezone mismatch: S3 ls returns naive datetimes. The collector must .replace(tzinfo=timezone.utc) before subtracting.
|
|
- Python 3.13 deprecation: datetime.utcnow() is deprecated. Use datetime.now(timezone.utc) instead.
|
|
- AWS CLI path: must activate /opt/awscli-venv/bin/activate before calling aws commands
|
|
- Placeholder sections: Show amber cards with "Coming soon" for empty sections
|