Multi-Server Resource Threshold Alerting

Monitors RAM, disk, and CPU usage across all VPS servers, alerting via Telegram when thresholds are crossed. Runs as a Hermes no-agent cron job every 15 minutes.

Architecture

Bash script (vps-threshold-check.sh)
  └─ Server manifest (hostname|ip|local_flag)
      ├─ ssh to each remote server (itpp-infra key, 5s timeout)
      └─ local commands for core server
  ├─ Collect: df -h / (disk %), free (RAM %), top -bn1 (CPU %), nproc
  ├─ Check 3 thresholds (80%, 90%, 95%) per metric per server
  ├─ 24h re-alert suppression via JSON state file
  └─ stdout on alert → no-agent cron → Telegram delivery

Script: `/root/.hermes/scripts/vps-threshold-check.sh`

Server Manifest Format

"hostname|ip_address|is_local_flag"

is_local_flag=1 — run commands directly (core server)
is_local_flag=0 — SSH via /root/.ssh/itpp-infra key

Alert State File

/root/.hermes/data/threshold-alerts.json — JSON object keyed by hostname_metric_threshold (e.g. ai_disk_90), value is ISO timestamp of last alert.

Threshold Logic

Per threshold: 80%, 90%, and 95% — each tracked independently
Alert on first crossing: records timestamp, outputs alert to stdout
Suppress re-alerts: 86,400 seconds (24h) cooldown per key
Reset on drop: when metric falls below a threshold, that alert key is removed from state — so it can re-alert immediately on the next spike
No output when healthy: silent stdout = no Telegram noise (cron skips delivery)

Cron Job

Name:      vps-threshold-check
Schedule:  every 15m
Script:    vps-threshold-check.sh
Mode:      no-agent (script stdout delivered directly)
Deliver:   telegram

Uses Hermes' no-agent cron pattern: script writes alert text to stdout → cron scheduler delivers to Telegram via the running gateway's TELEGRAM_BOT_TOKEN env var.

Threshold Design Decisions

3-tier thresholds rather than a single number: gives escalating urgency (80% ≈ warning, 90% ≈ critical, 95% ≈ emergency)
Per-threshold tracking not per-metric: if disk jumps from 70% to 92%, both the 80% and 90% alerts fire once (not every 15 min)
24h re-alert window prevents notification fatigue but ensures admins still hear about sustained issues
keyed suppression rather than last-N-hours per server: handles the case where a metric oscillates across a threshold

Known Pitfalls

CPU measurement: top -bn1 reads the first sample which may show 0% on idle systems. For accurate CPU under load, use top -bn2 | awk '/Cpu$s$/ {print $2}' | tail -1 (second sample picks up between-snapshot delta). The current script uses the first sample which is fine for sustained high-CPU scenarios.
SSH timeouts: each server has a 5s ConnectTimeout. If a server is unreachable, the ssh output is empty and metrics default to 0 — no false alarms.
Alert state persistence: if the JSON file is deleted, all thresholds re-alert on the next run (resets the 24h clock). This is intentional — better to re-alert than stay silent.
Telegram env vars: delivery requires the gateway process to have TELEGRAM_BOT_TOKEN, TELEGRAM_ALLOWED_USERS, and TELEGRAM_HOME_CHANNEL in its environment (set at hermes gateway run launch). These are inherited from the parent shell or .env.
Gateway not running: if the gateway is down, the cron scheduler logs failed delivery but the script still runs fine — the JSON state file is updated regardless.

3.5 KiB Raw Blame History