Service Health Check Cron

Pattern

A no_agent cron that checks all critical services every 5 minutes. Silent on success. Reports only when something is down. Implemented at ~/.hermes/scripts/service-health-check.sh.

What it checks

#	Check	Method
1	Caddy (sign.itpropartner.com)	`systemctl is-active`
2	MySQL tunnel (wphost02)	`systemctl is-active`
3	SearXNG container	`docker ps --format`
4	DocuSeal container	`docker ps --format`
5	Portal mockup (port 8081)	`curl -sf`
6	DocuSeal web (port 3000)	`curl -sf`
7	SearXNG search (port 8888)	`curl -sf` with search query
8	WireGuard (home router)	`ping 10.77.0.2`
9	MySQL through tunnel	`mysql --skip-ssl -e "SELECT 1"`
10	wphost02 SSH	`ssh -o ConnectTimeout=5 hostname`
11	Hermes gateway process	`ps aux
12	Cloudflare API token	`curl -X GET /tokens/verify`

Cron setup

hermes cron create \
  --name "service-health-check" \
  --schedule "every 5m" \
  --no_agent \
  --script service-health-check.sh

Script design

exit 0 when all services healthy — no output, no delivery
exit 1 when any service fails — prints a bullet list of failed checks
Failed services array tracks which checks failed
Each check is independent — one failure doesn't abort the script
Check count shown in alert message

Pitfalls

Sensitive info in error output: The MySQL connection string appears in the script. If a service fails, the error output may include the full password. The cron delivery is Telegram (visible only to user), not email.
False negatives on restart: During a container restart, the health check may fire before the service comes back up. The 5-minute interval means this resolves within one cycle.
Token expiry: The Cloudflare API token check is a direct API call. If the token expires, ALL Cloudflare checks fail. The script does not differentiate between expired token and Cloudflare outage.
MySQL credential in plaintext: The password is in the script file and .env. If the cron output is ever delivered to the wrong channel, credentials leak. Consider moving MySQL check to a separate script that sources credentials from .env with tighter permissions.

⚠️ Fixed Bug: Bearer token sent as literal asterisks

Root cause discovered Jul 7, 2026: Line 38 of the script reads the token from .env into a variable, but line 39 sends Authorization: Bearer *** (literal asterisks) instead of Authorization: Bearer $TOKEN. This caused the health check to report the Cloudflare API token as "unhealthy" every 5 minutes, even though the actual token was valid.

Before (broken):

TOKEN=$(grep CLOUDFLARE_API_TOKEN ~/.hermes/.env | cut -d= -f2-)
curl -sf -X GET "https://api.cloudflare.com/client/v4/user/tokens/verify" \
  -H "Authorization: Bearer ***" | ...

After (fixed):

CLOUD_TOKEN=$(grep CLOUDFLARE_API_TOKEN ~/.hermes/.env | cut -d= -f2-)
curl -sf -X GET "https://api.cloudflare.com/client/v4/user/tokens/verify" \
  -H "Authorization: Bearer $CLOUD_TOKEN" | ...

Variable name was changed from $TOKEN to $CLOUD_TOKEN to avoid any confusion with other use of TOKEN in the script or environment.

3.2 KiB Raw Permalink Blame History