Initial skills documentation — 25 categories, all SKILL.md + references + scripts

2026-07-15 17:42:20 -04:00
commit 1b4fcd4427
976 changed files with 188344 additions and 0 deletions
@@ -0,0 +1,62 @@
+# Multi-Server Resource Threshold Alerting
+
+Monitors RAM, disk, and CPU usage across all VPS servers, alerting via Telegram when thresholds are crossed. Runs as a Hermes no-agent cron job every 15 minutes.
+
+## Architecture
+
+```
+Bash script (vps-threshold-check.sh)
+  └─ Server manifest (hostname|ip|local_flag)
+      ├─ ssh to each remote server (itpp-infra key, 5s timeout)
+      └─ local commands for core server
+  ├─ Collect: df -h / (disk %), free (RAM %), top -bn1 (CPU %), nproc
+  ├─ Check 3 thresholds (80%, 90%, 95%) per metric per server
+  ├─ 24h re-alert suppression via JSON state file
+  └─ stdout on alert → no-agent cron → Telegram delivery
+```
+
+## Script: `/root/.hermes/scripts/vps-threshold-check.sh`
+
+### Server Manifest Format
+```
+"hostname|ip_address|is_local_flag"
+```
+- `is_local_flag=1` — run commands directly (core server)
+- `is_local_flag=0` — SSH via `/root/.ssh/itpp-infra` key
+
+### Alert State File
+`/root/.hermes/data/threshold-alerts.json` — JSON object keyed by `hostname_metric_threshold` (e.g. `ai_disk_90`), value is ISO timestamp of last alert.
+
+### Threshold Logic
+- **Per threshold**: 80%, 90%, and 95% — each tracked independently
+- **Alert on first crossing**: records timestamp, outputs alert to stdout
+- **Suppress re-alerts**: 86,400 seconds (24h) cooldown per key
+- **Reset on drop**: when metric falls below a threshold, that alert key is removed from state — so it can re-alert immediately on the next spike
+- **No output when healthy**: silent stdout = no Telegram noise (cron skips delivery)
+
+## Cron Job
+
+```
+Name:      vps-threshold-check
+Schedule:  every 15m
+Script:    vps-threshold-check.sh
+Mode:      no-agent (script stdout delivered directly)
+Deliver:   telegram
+```
+
+Uses Hermes' no-agent cron pattern: script writes alert text to stdout → cron scheduler delivers to Telegram via the running gateway's `TELEGRAM_BOT_TOKEN` env var.
+
+## Threshold Design Decisions
+
+- **3-tier thresholds** rather than a single number: gives escalating urgency (80% ≈ warning, 90% ≈ critical, 95% ≈ emergency)
+- **Per-threshold tracking** not per-metric: if disk jumps from 70% to 92%, both the 80% and 90% alerts fire once (not every 15 min)
+- **24h re-alert window** prevents notification fatigue but ensures admins still hear about sustained issues
+- **keyed suppression** rather than last-N-hours per server: handles the case where a metric oscillates across a threshold
+
+## Known Pitfalls
+
+- **CPU measurement**: `top -bn1` reads the first sample which may show 0% on idle systems. For accurate CPU under load, use `top -bn2 | awk '/Cpu\(s\)/ {print $2}' | tail -1` (second sample picks up between-snapshot delta). The current script uses the first sample which is fine for sustained high-CPU scenarios.
+- **SSH timeouts**: each server has a 5s ConnectTimeout. If a server is unreachable, the `ssh` output is empty and metrics default to 0 — no false alarms.
+- **Alert state persistence**: if the JSON file is deleted, all thresholds re-alert on the next run (resets the 24h clock). This is intentional — better to re-alert than stay silent.
+- **Telegram env vars**: delivery requires the gateway process to have `TELEGRAM_BOT_TOKEN`, `TELEGRAM_ALLOWED_USERS`, and `TELEGRAM_HOME_CHANNEL` in its environment (set at `hermes gateway run` launch). These are inherited from the parent shell or `.env`.
+- **Gateway not running**: if the gateway is down, the cron scheduler logs failed delivery but the script still runs fine — the JSON state file is updated regardless.