# Multi-Server Resource Threshold Alerting

Monitors RAM, disk, and CPU usage across all VPS servers, alerting via Telegram when thresholds are crossed. Runs as a Hermes no-agent cron job every 15 minutes.

## Architecture

```
Bash script (vps-threshold-check.sh)
  └─ Server manifest (hostname|ip|local_flag)
      ├─ ssh to each remote server (itpp-infra key, 5s timeout)
      └─ local commands for core server
  ├─ Collect: df -h / (disk %), free (RAM %), top -bn1 (CPU %), nproc
  ├─ Check 3 thresholds (80%, 90%, 95%) per metric per server
  ├─ 24h re-alert suppression via JSON state file
  └─ stdout on alert → no-agent cron → Telegram delivery
```

## Script: `/root/.hermes/scripts/vps-threshold-check.sh`

### Server Manifest Format
```
"hostname|ip_address|is_local_flag"
```
- `is_local_flag=1` — run commands directly (core server)
- `is_local_flag=0` — SSH via `/root/.ssh/itpp-infra` key

### Alert State File
`/root/.hermes/data/threshold-alerts.json` — JSON object keyed by `hostname_metric_threshold` (e.g. `ai_disk_90`), value is ISO timestamp of last alert.

### Threshold Logic
- **Per threshold**: 80%, 90%, and 95% — each tracked independently
- **Alert on first crossing**: records timestamp, outputs alert to stdout
- **Suppress re-alerts**: 86,400 seconds (24h) cooldown per key
- **Reset on drop**: when metric falls below a threshold, that alert key is removed from state — so it can re-alert immediately on the next spike
- **No output when healthy**: silent stdout = no Telegram noise (cron skips delivery)

## Cron Job

```
Name:      vps-threshold-check
Schedule:  every 15m
Script:    vps-threshold-check.sh
Mode:      no-agent (script stdout delivered directly)
Deliver:   telegram
```

Uses Hermes' no-agent cron pattern: script writes alert text to stdout → cron scheduler delivers to Telegram via the running gateway's `TELEGRAM_BOT_TOKEN` env var.

## Threshold Design Decisions

- **3-tier thresholds** rather than a single number: gives escalating urgency (80% ≈ warning, 90% ≈ critical, 95% ≈ emergency)
- **Per-threshold tracking** not per-metric: if disk jumps from 70% to 92%, both the 80% and 90% alerts fire once (not every 15 min)
- **24h re-alert window** prevents notification fatigue but ensures admins still hear about sustained issues
- **keyed suppression** rather than last-N-hours per server: handles the case where a metric oscillates across a threshold

## Known Pitfalls

- **CPU measurement**: `top -bn1` reads the first sample which may show 0% on idle systems. For accurate CPU under load, use `top -bn2 | awk '/Cpu\(s\)/ {print $2}' | tail -1` (second sample picks up between-snapshot delta). The current script uses the first sample which is fine for sustained high-CPU scenarios.
- **SSH timeouts**: each server has a 5s ConnectTimeout. If a server is unreachable, the `ssh` output is empty and metrics default to 0 — no false alarms.
- **Alert state persistence**: if the JSON file is deleted, all thresholds re-alert on the next run (resets the 24h clock). This is intentional — better to re-alert than stay silent.
- **Telegram env vars**: delivery requires the gateway process to have `TELEGRAM_BOT_TOKEN`, `TELEGRAM_ALLOWED_USERS`, and `TELEGRAM_HOME_CHANNEL` in its environment (set at `hermes gateway run` launch). These are inherited from the parent shell or `.env`.
- **Gateway not running**: if the gateway is down, the cron scheduler logs failed delivery but the script still runs fine — the JSON state file is updated regardless.