42 lines
1.3 KiB
Markdown
42 lines
1.3 KiB
Markdown
# Service Health Check (no_agent watchdog)
|
|
|
|
A reusable pattern for no_agent cron jobs that monitor ALL critical services and only report on failure.
|
|
|
|
## Design
|
|
|
|
```bash
|
|
#!/bin/bash
|
|
# silent on success, report failures only
|
|
|
|
CHECKS=()
|
|
FAILED=0
|
|
|
|
# Each check:
|
|
command || { CHECKS+=("Service Name"); FAILED=1; }
|
|
|
|
if [ $FAILED -eq 1 ]; then
|
|
echo "⚠️ Service Health Alert — $(date)"
|
|
for c in "${CHECKS[@]}"; do echo " ❌ $c"; done
|
|
exit 1
|
|
fi
|
|
exit 0
|
|
```
|
|
|
|
## What to check
|
|
|
|
- Systemd services (caddy, mysql-tunnel)
|
|
- Docker containers (docker ps --format)
|
|
- Local HTTP endpoints (curl -sf)
|
|
- Remote reachability (ping through tunnel, SSH)
|
|
- Database connectivity (mysql -e "SELECT 1")
|
|
- API token validity (curl + verify response)
|
|
|
|
## Pitfalls
|
|
|
|
- **Must be no_agent: true** in cron config — LLM processing is wasted on binary health checks and causes typing indicators
|
|
- **Include the exit code** — exit 0 = silent, exit 1 = output delivered. Without the right exit code, cron can't tell success from failure
|
|
- **Test with `bash script.sh` and check `echo $?`** before deploying
|
|
- **MySQL checks need `--skip-ssl`** for local tunnel connections
|
|
- **Don't check too many things** — 10-12 checks is the sweet spot. More than 15 and the script gets brittle
|
|
- **SSH checks need `-o BatchMode=yes -o ConnectTimeout=5`** to avoid hanging on unreachable hosts
|